RAG Evaluation: Context, Faithfulness, Answer

What to Measure

• Context relevance - Did retrieval find the right notes?
• Faithfulness - Does the answer stick to the context?
• Answer relevance - Does the answer address the question?

Build an Eval Set

Collect 30-60 real questions with the note each one should cite. Run every change against this fixed set so scoring is repeatable.

Score Retrieval and Generation Separately

def retrieval_hit(pairs, top_k=3):
    hits = 0
    for question, expected in pairs:
        found = retrieve(question, top_k)
        hits += int(expected in found)
    return hits / len(pairs)

def generation_pass(pairs):
    passed = 0
    for question, expected in pairs:
        answer = answer_rag(question)
        passed += int(expected not in answer)   # check faithfulness
    return passed / len(pairs)

Separate scores tell you which stage to fix: retrieval or generation.

Operational Metrics

Also track end-to-end latency, cache hit rate, and the share of answers that cite a chunk. A knowledge base scoring well in offline evals can still drift online.

Key Points

• Score retrieval and generation separately.
• A fixed eval set makes tuning measurable.
• Faithfulness catches hallucination early.
• Watch online drift, not just offline scores.