RAG Evaluation

RAG evaluation is the practice of scoring a retrieval-augmented generation system with metrics that separate retrieval quality from generation quality, so that a bad answer can be traced to the component that produced it. The metric set most teams standardize on comes from the open-source Ragas framework and appears in near-identical form in LlamaIndex, TruLens, and DeepEval: faithfulness, answer relevance, context precision, and context recall.

Each metric localizes a different failure. Faithfulness measures what proportion of the claims in the generated answer are supported by the retrieved context; a low score means the model is asserting things its sources do not say, which is a generation-stage problem. Answer relevance measures whether the response actually addresses the question asked rather than adjacent material, also generation-stage. Context precision measures how much of the retrieved context was relevant and whether the relevant chunks ranked near the top; a low score means the retriever is returning noise and burning context window. Context recall measures whether everything needed to answer the question was retrieved at all; a low score means the retriever missed, and no amount of prompt engineering downstream will recover the missing fact. The diagnostic value is in the combinations. High recall with low precision is a ranking problem. High precision with low faithfulness is a model or prompt problem, not a data problem. Low recall with high faithfulness produces an answer that is honest about knowing nothing, which is a retrieval sourcing problem.

For a team past the prototype stage, the practical use of these metrics is proving that a vendor swap actually helped. The method is to hold everything else fixed: freeze a question set drawn from real product traffic, freeze the generation model, freeze the prompt, and change only the retrieval layer. Context recall and context precision should move. Faithfulness and answer relevance should stay roughly flat, and if they do not, something other than retrieval changed. Two cautions apply. Most of these metrics are computed by an LLM acting as judge, which introduces variance run to run, so compare distributions across repeated runs rather than single decimals, and treat a two-point difference as noise unless it survives repetition. And a question set below a few hundred items will not distinguish vendors that are genuinely close.

For the specific case of web-sourced retrieval, Vals AI publishes a web search evaluation that scores how much different search tools lift agent accuracy on research tasks, which is the closest third-party signal to the question buyers of Exa, Tavily, and their peers are actually asking. Use those published numbers as a starting shortlist, then run your own evaluation on your own questions. Retrieval quality is domain-dependent in a way that a general benchmark cannot capture: a vendor that wins on open-web research questions can lose badly on a corpus of regulatory filings or non-English product pages.