Evaluation
| # | File | Covers |
|---|---|---|
| 01 | Why evaluating LLM systems is hard | why accuracy doesn’t apply; the layers to measure at |
| 02 | Building eval sets | where cases come from, sizing, statistical honesty |
| 03 | LLM as judge | the biases, pairwise over scoring, validating the judge |
| 04 | Online evaluation and experiments | shadow deployment, A/B testing, why offline and online diverge |
| 05 | The eval and observability tooling | Langfuse, RAGAS, DeepEval, promptfoo — and what “we have evals” should mean |
Retrieval-specific evaluation lives in Agentic RAG and evaluating retrieval.
Three things to have ready
Deterministic checks before judges. Schema validity, required strings, citation presence and refusal behaviour are all free and catch most regressions. A judge is the fallback for what code cannot verify.
Pairwise beats scoring. Absolute scores drift and cluster around 7-8. Run both orderings and treat disagreement as a tie — that converts position bias into an honest tie rather than a coin flip.
85% to 88% on 100 cases is noise. The confidence interval is roughly ±7 points. Report intervals, or use paired comparison which has far more power.