Retrieval-Augmented Generation (RAG) has become a foundational pattern for grounding large language models in external knowledge. But a working RAG pipeline is not the same as a reliable one. A system that retrieves irrelevant passages, hallucinates answers not supported by its sources, or misses critical context will fail in production even if it works on a handful of test queries. This article provides a practical framework for evaluating RAG systems using measurable metrics: context precision, context recall, faithfulness, and answer relevance. It is written for engineers who need to move beyond ad-hoc testing to systematic evaluation.
The Evaluation Gap in RAG
Most teams start by checking whether the system returns an answer that "looks right" on a few queries. This approach has three problems. First, it conflates retrieval quality with generation quality—a fluent answer built on wrong context is worse than no answer. Second, it misses systematic failure modes that only appear at scale: retrieval drift as the corpus grows, or faithfulness degradation when prompts change. Third, it provides no basis for comparing alternative configurations (chunk size, embedding model, top-k, reranker) because "looks right" is not a comparable signal.
A proper RAG evaluation separates the pipeline into independently measurable stages: retrieval (did we find the right passages?), generation (did we use them correctly?), and the joint behavior (does the final answer reflect the retrieved evidence?). Each stage requires its own metrics.
Core Metrics Defined
Context Precision
Context precision measures whether the retrieved passages that are relevant appear at the top of the ranked list. It is the precision-at-k applied to the retrieval stage. If the retriever returns ten passages but only three are relevant, and those three are ranked positions 1, 4, and 8, the precision at 10 is 0.3—but the precision at 3 is 1.0. For RAG, high precision at low k matters more than average precision because the generator typically sees only the top few passages.
Formally, for a query with a set of relevant passage IDs R and a ranked list of retrieved passage IDs P, precision at k is:
Precision@k = |{p ∈ P[1:k] : p ∈ R}| / k
A practical implementation requires a labeled set of queries with known relevant passages. Building this set is the primary upfront investment.
Context Recall
Context recall measures whether the retriever finds all relevant passages for a query. If there are five truly relevant passages in the corpus and the retriever returns three of them in the top k, recall at k is 0.6. Low recall means the generator never sees critical information, making faithful answers impossible.
Recall@k = |{p ∈ P[1:k] : p ∈ R}| / |R|
In practice, recall is harder to measure than precision because it requires knowing the complete set of relevant passages for each query—essentially, exhaustive annotation. A common compromise is to use a high-recall retrieval method (e.g., BM25 with a large k, or an ensemble of retrievers) to construct a candidate pool, then have annotators label relevance within that pool.
Faithfulness
Faithfulness (sometimes called groundedness or attribution) measures whether the generated answer is supported by the retrieved context. An unfaithful answer introduces claims not present in the provided passages, or contradicts them. This is the metric that catches hallucination in the RAG context.
Faithfulness is typically evaluated by decomposing the answer into atomic claims and verifying each claim against the retrieved passages. A simple approach:
- Split the answer into sentences or propositional claims.
- For each claim, check whether it is entailed by, contradicted by, or unaddressed by the retrieved context.
- Faithfulness score = (entailed claims) / (total claims).
LLM-based evaluators can automate this decomposition and verification, but they require careful prompting and calibration against human judgments. A known limitation: an LLM judge may be lenient toward plausible-sounding but unsupported claims.
Answer Relevance
Answer relevance measures whether the generated answer actually addresses the user's question, regardless of correctness. An answer can be faithful (fully supported by context) but irrelevant (answering a different question). Relevance is typically assessed by comparing the answer to the query, often using an LLM judge prompted to rate on a Likert scale or binary yes/no.
Answer relevance and faithfulness are distinct: a system that always replies "I don't know" has perfect faithfulness but zero relevance. A system that hallucinates a detailed but wrong answer has high relevance (it looks like an answer) but low faithfulness.
Building an Evaluation Workflow
Step 1: Create a Labeled Query Set
Start with 50–100 representative queries covering your expected query distribution. For each query, identify the ground-truth relevant passages in your corpus. This is the most labor-intensive step but cannot be skipped. If you lack resources for full annotation, begin with a smaller set and expand iteratively.
Structure your dataset as:
{
"query": "What is the refund policy for digital goods?",
"relevant_passage_ids": ["doc-123-chunk-2", "doc-456-chunk-1"],
"expected_answer": "Digital goods are eligible for refund within 14 days of purchase if not downloaded..."
}
Include edge cases: ambiguous queries, queries requiring multi-hop reasoning, queries with no answer in the corpus.
Step 2: Evaluate Retrieval in Isolation
Before testing the full pipeline, measure the retriever alone. Compute precision@k and recall@k for k ∈ {1, 3, 5, 10, 20}. Plot the precision-recall curve. This reveals whether your embedding model, chunking strategy, and index configuration are adequate before you introduce generator variability.
Test variations systematically:
- Embedding models (e.g., text-embedding-3-small vs. bge-large-en-v1.5)
- Chunk sizes (256, 512, 1024 tokens) and overlap strategies
- Dense vs. sparse vs. hybrid retrieval
- Reranker impact (cross-encoder reranking top-50 to top-10)
Record latency and cost alongside quality metrics.
Step 3: Evaluate Generation with Fixed Context
To isolate generator behavior, feed the generator the gold relevant passages (from your labeled set) and measure faithfulness and answer relevance. This tells you the generator's ceiling performance when retrieval is perfect. If faithfulness is low even with gold context, the problem is in the prompt, model, or generation parameters—not the retriever.
Test prompt variations:
- Explicit citation instructions ("Cite passages using [doc-id] format")
- Chain-of-thought prompting for multi-step reasoning
- Negative constraints ("If the context does not contain the answer, say so")
Step 4: End-to-End Evaluation
Run the full pipeline (retrieval + generation) on your labeled queries. Compute all four metrics. The end-to-end faithfulness score will be lower than the fixed-context score because retrieval errors propagate. The gap between fixed-context and end-to-end faithfulness quantifies retrieval's impact on generation quality.
For each query, record:
- Retrieved passage IDs and scores
- Generated answer
- Per-claim faithfulness judgments
- Relevance judgment
- Latency and token usage
Step 5: Error Analysis and Iteration
Categorize failures:
- Missed retrieval: Relevant passages not in top-k (recall failure)
- Noisy retrieval: Irrelevant passages ranked high (precision failure)
- Ungrounded generation: Claims not supported by retrieved context (faithfulness failure)
- Non-responsive generation: Answer does not address query (relevance failure)
- Partial answers: Correct but incomplete (often a recall or prompt issue)
Prioritize fixes by frequency and impact. A common pattern: improving recall (e.g., via hybrid retrieval) yields larger gains than prompt engineering when recall is the bottleneck.
Practical Implementation Considerations
LLM-as-Judge Calibration
LLM evaluators for faithfulness and relevance require calibration. Use a small human-labeled subset (20–30 examples) to measure agreement between your LLM judge and human annotators. Adjust the judge prompt until Cohen's kappa exceeds 0.7. Re-calibrate when you change the generator model or prompt.
Handling "No Answer" Cases
Queries with no relevant passages in the corpus should be in your test set. A good RAG system should recognize this and refuse to answer (or say "I don't know"). Evaluate this separately: measure the false positive rate (answering when no ground-truth passages exist) and false negative rate (refusing when ground-truth passages exist).
Continuous Evaluation
RAG systems degrade over time as the corpus grows, user query distributions shift, or models are updated. Schedule weekly evaluation runs on a held-out query set. Alert on metric regressions exceeding a threshold (e.g., faithfulness drop > 5 percentage points).
Cost-Quality Tradeoffs
Track cost per query (embedding tokens + generation tokens + reranker calls) alongside quality. A configuration with 2% higher faithfulness at 5x cost may not be the right choice for your use case. Plot quality vs. cost to find the Pareto frontier.
Limitations and Open Questions
This framework has known limitations:
- Label dependency: All metrics require human-annotated relevance judgments. Building and maintaining this dataset is ongoing work.
- Claim decomposition granularity: Faithfulness scores vary significantly depending on whether claims are split at the sentence, clause, or propositional level. Standardize your decomposition method.
- Subjectivity in relevance: Answer relevance judgments can vary between annotators. Use multiple annotators and measure agreement.
- Static corpus assumption: Relevance labels can depend on a particular corpus version. In streaming-update scenarios, relevance labels become stale.
- Single-turn focus: This framework evaluates single-query interactions. Multi-turn conversational RAG requires additional metrics (context consistency, conversation coherence).
Conclusion
Use systematic evaluation before deciding whether a RAG configuration meets your application requirements. The four metrics—context precision, context recall, faithfulness, and answer relevance—provide a shared vocabulary for diagnosing failures and comparing configurations. The workflow described here (labeled queries → isolated retrieval evaluation → fixed-context generation evaluation → end-to-end evaluation → error analysis) is a practical starting point. The upfront investment in a labeled query set pays off every time you test a new embedding model, chunking strategy, or prompt template. Without it, you are guessing. With it, you are engineering.
Use the fine-tuning versus RAG worksheet to choose the next experiment, and the classifier evaluation guide to record an inspectable error analysis.
Sources
- Ragas metric documentation, including context precision, context recall, response relevancy and faithfulness. Exact formulas and required inputs depend on the chosen metric implementation.
- Hugging Face Evaluate quick tour, for evaluation-library terminology. The workflow recommendations are original editorial guidance.