Skip to content

Datasets for RAG Benchmarking

You cannot improve what you cannot measure. Production-grade RAG requires rigorous evaluation against ground-truth datasets. This list covers general-purpose benchmarks and domain-specific corpora.

The Leaderboards

Before choosing a model, check these live leaderboards:

General Knowledge (Open Domain QA)

  • MS MARCO - 1M+ Queries.
  • Making AI the first truly conversational search engine.
  • (Best For: Retrieval)
  • HotpotQA - 113k pairs.
  • Question answering requiring multi-hop reasoning.
  • (Best For: Reasoning)
  • Natural Questions (NQ) - 300k+.
  • Real user queries issued to Google Search.
  • (Best For: Realism)
  • TriviaQA - 95k.
  • Reading comprehension dataset containing triples of (question, answer, evidence).
  • (Best For: Factuality)

Long-Context & Document Understanding

  • SQuAD 2.0 - Stanford Question Answering Dataset.
  • Includes unanswerable questions.
  • (Best For: Hallucination Detection)
  • Qasper - Question answering over NLP papers.
  • (Best For: Technical/Scientific)
  • NarrativeQA - QA over collected stories (books and movie scripts).
  • (Best For: Long Context)

BEIR Benchmark Suite

The de facto zero-shot retrieval benchmark — 18 heterogeneous datasets, 9 task types — spanning bio-medical IR, fact-checking, argument retrieval, and more.

  • BEIR - 18 datasets, 9 task types.
  • Heterogeneous benchmark for zero-shot evaluation of dense and sparse retrievers across domains.
  • (Best For: Zero-Shot Retrieval)
  • MIRACL - 18 languages, 700k+ queries.
  • Multilingual retrieval benchmark over Wikipedia in typologically diverse languages.
  • (Best For: Multilingual Retrieval)
  • SciFact - 1.4k scientific claims.
  • Claim verification against a corpus of research literature.
  • (Best For: Scientific Faithfulness)
  • CUAD - 13k+ labels, 510 contracts.
  • Expert-annotated commercial contracts across 41 clause categories.
  • (Best For: Clause Extraction)
  • LegalBench - 162 tasks.
  • Collaborative benchmark for legal reasoning, built with practicing lawyers.
  • (Best For: Legal Reasoning)
  • CaseHOLD - 53k+ holdings.
  • Multiple-choice case-holding identification over US legal opinions.
  • (Best For: Case Law Retrieval)
  • Open Australian Legal Corpus - 200k+ documents.
  • Largest open, permissively licensed (CC-BY) legal corpus suitable for pretraining and fine-tuning.
  • (Best For: Legal Pretraining)

Medical & Biomedical

  • PubMedQA - 1k expert + 211k auto.
  • Yes/No/Maybe QA over biomedical research abstracts.
  • (Best For: Biomedical QA)
  • BioASQ - Annual challenge.
  • Large-scale biomedical semantic indexing and question answering (registration required).
  • (Best For: Semantic Indexing)
  • MedQA (USMLE) - 12k+ MCQs.
  • Medical board exam questions across English, Chinese, and Taiwanese licensure.
  • (Best For: Clinical Reasoning)
  • MIMIC-IV - De-identified clinical notes.
  • ICU and ED records from Beth Israel Deaconess; PhysioNet credentialing required.
  • (Best For: Clinical Notes)

Finance & Markets

  • FinQA - 8k+ QA pairs.
  • Numerical reasoning over S&P 500 earnings reports (text + tables).
  • (Best For: Numerical Reasoning)
  • TAT-QA - 16k+ QA pairs.
  • Hybrid tabular and textual QA over real-world financial reports.
  • (Best For: Tabular Reasoning)
  • ConvFinQA - Conversational extension.
  • Multi-turn numerical reasoning over financial documents.
  • (Best For: Conversational Finance)

Synthetic Data Generation

Don't have a dataset? Generate one from your own internal documents.

(back to main resource)