RAG evaluation is the practice of measuring how well a retrieval-augmented generation (RAG) system finds the right information and turns it into a correct, grounded answer. A solid evaluation scores two layers separately: retrieval (did the system fetch the passages that contain the answer?) and generation (is the answer faithful to those passages and relevant to the question?). Teams then add operational checks—latency and cost—before shipping to production.

That split matters because a RAG app can fail in two very different ways. If the retriever misses the right document, even the best model will guess. If retrieval is perfect but the model ignores or distorts the context, you get a confident hallucination. Evaluating only the final answer hides which part broke. If you need a refresher on the architecture itself, start with our explainer on what retrieval-augmented generation is and how it works.

Why RAG evaluation matters now

Building a RAG demo takes an afternoon: chunk some documents, embed them, store them in a vector database and prompt a model. Making it reliable is the hard part. Small changes—a new chunk size, a different embedding model, a reranker, a prompt tweak—can quietly improve one type of question and break another. Without a repeatable evaluation set, teams end up judging changes by “vibes” on a handful of examples.

That is why evaluation has become a core AI engineering skill rather than an afterthought. Frameworks such as Ragas and cloud platforms such as Microsoft Azure AI Foundry’s RAG evaluators now ship ready-made metrics for retrieval and groundedness, and the broader topic of testing LLM systems is covered in our guide to AI evals and how companies test agents.

Layer 1: How to evaluate retrieval quality

Retrieval evaluation borrows decades of methods from classic information retrieval. You need a set of test questions and, for each one, the passages (or documents) that are known to be relevant. Then you compare what your retriever returns in its top k results against that ground truth. Stanford’s Introduction to Information Retrieval is still the standard reference for these metrics.

Recall@k

Of all the relevant passages for a question, what share appears in the top k results? Recall@k is usually the most important retrieval metric for RAG, because the model can only use what it receives. If recall@5 is low, no prompt engineering will fix the answers.

Precision@k

Of the k passages returned, how many are actually relevant? Low precision means the context window is filled with noise, which raises cost and can distract the model.

MRR (Mean Reciprocal Rank)

MRR looks at the position of the first relevant result. A relevant passage in position 1 scores 1, in position 2 scores 0.5, in position 3 scores 0.33, and so on, averaged across questions. It is useful when one good passage is usually enough.

nDCG (normalized Discounted Cumulative Gain)

nDCG rewards putting the most relevant passages at the top and supports graded relevance (for example “highly relevant” vs. “partially relevant”). It is the main metric in the BEIR benchmark, which compares retrieval models across many domains, and it is the natural metric for judging whether a reranking stage is worth its added latency.

These metrics are also how you compare retrieval strategies fairly: dense vector search vs. keyword search (BM25) vs. hybrid search, different chunk sizes, or different embedding models. Run the same question set through each configuration and keep the one that wins on recall and nDCG at an acceptable latency.

Layer 2: How to evaluate generation quality

Once retrieval is measured, you evaluate whether the model used the context well. The RAGAS paper popularized a reference-free set of metrics that many teams now use:

• Faithfulness (groundedness): are the claims in the answer supported by the retrieved context? This is your main hallucination signal.

• Answer relevance: does the answer actually address the question, without padding or drifting off topic?

• Context precision and context recall: LLM-judged versions of retrieval metrics that check whether the useful context was retrieved and ranked high.

• Answer correctness: when you have a reference answer, how closely does the generated answer match it?

Most of these are computed with an LLM-as-a-judge: a second model reads the question, context and answer and scores them against a rubric. It scales well, but judges have biases, so calibrate them by comparing their scores with human labels on a sample before trusting them.

Layer 3: Latency, cost and production signals

A system that scores well offline can still fail users. Track end-to-end latency (p50 and p95), tokens and cost per query, and the share of questions where the system correctly says “I don’t know.” In production, add observability: log the query, the retrieved chunks, the prompt and the answer so you can trace any bad response back to the layer that caused it, and feed real failures back into your test set.

A step-by-step RAG evaluation workflow

1. Build a golden dataset. Collect 50–200 realistic questions from users, support tickets or domain experts. Label the relevant passages and, where possible, a reference answer. Include hard cases: ambiguous questions, multi-hop questions and questions the corpus cannot answer.

2. Measure retrieval first. Compute recall@k, MRR and nDCG for your current retriever. Fix retrieval before touching prompts.

3. Measure generation. Score faithfulness, answer relevance and correctness with an LLM judge, spot-checked by humans.

4. Change one variable at a time. Chunk size, embedding model, hybrid search, reranker, prompt. Re-run the full set after each change.

5. Add it to CI. Treat the evaluation like a test suite so regressions are caught before deployment.

6. Monitor and refresh. Sample production traffic, label failures and grow the dataset over time.

Common RAG evaluation mistakes

• Only scoring the final answer. You will not know whether to fix the retriever or the prompt.

• Tiny or synthetic-only test sets. Synthetic questions are a good start, but they rarely capture how real users phrase things.

• Trusting an uncalibrated LLM judge. Always validate judge scores against human review.

• Ignoring unanswerable questions. A good RAG system should refuse when the context does not contain the answer.

• Optimizing quality while ignoring latency and cost. A reranker that adds a second of latency may not be worth a small nDCG gain.

FAQ

What is the most important metric for evaluating RAG?

There is no single metric, but for most teams the two to watch first are retrieval recall@k (did the right context reach the model?) and faithfulness (is the answer grounded in that context?). Together they tell you whether a failure comes from search or from generation.

How is RAG evaluation different from general LLM evaluation?

General LLM evaluation scores a model’s outputs. RAG evaluation adds a retrieval layer, so you also measure search quality with information retrieval metrics such as recall, MRR and nDCG, and you check whether the answer stays faithful to the retrieved documents.

Do I need labeled data to evaluate a RAG system?

Reference-free metrics like faithfulness and answer relevance can run without labels, which makes them a good starting point. For reliable retrieval metrics and answer correctness, you still need a labeled golden dataset, even a small one.

Which tools are used for RAG evaluation?

Common options include open-source frameworks such as Ragas, evaluation and tracing platforms such as LangSmith, and cloud evaluators such as those in Azure AI Foundry. The tool matters less than having a stable, realistic test set you run on every change.

Where can I learn to build and evaluate RAG applications hands-on?

Look for a program where you build a retrieval app end to end, evaluate it with real metrics and get feedback from an instructor. Evaluation and observability are covered as a full module in Coderhouse’s live AI Engineering course, below.

Keep learning

RAG evaluation sits at the intersection of AI engineering and data analysis: you design experiments, measure results and make decisions from the numbers. If you want to go deeper with live classes, a cohort and human feedback, explore these programs:

• AI Engineering Course

• Data Analytics Course