The eval and observability tooling

Job ads name these tools directly, and being able to say what each one is separates someone who has run an eval from someone who has read about one. The concepts are in the rest of this folder; this is the landscape.

Note: versions verified 2026-08-12 — Langfuse 4.14.4, DeepEval 4.1.8, Ragas 0.4.3. This space moves fast; the categories below outlive the versions.

The two jobs, and why tools blur them

They started as separate products and converged, because the feedback loop between them is the whole point: a bad production trace becomes an eval case. Most tools now do both, which is why the category names are muddled.

Observability

Langfuse is the one to name in a regulated context, because self-hosting means prompts and completions — which contain PII — never leave your boundary. It also ships datasets, experiments, LLM-as-judge scoring and prompt management, so for many teams it is the whole stack.

LangSmith is the path of least resistance if you are already on LangChain/LangGraph: tracing appears with no instrumentation.

The vendor-neutral option is OpenTelemetry with the GenAI semantic conventions, exporting into whatever you already run. The argument for it: an LLM call is one span in a request that also touched Postgres and two services, and keeping them in one trace beats correlating across systems by timestamp. See LLM observability.

Evaluation

RAGAS — the RAG-specific one

The four metrics worth memorising, because they map onto the two halves of a RAG pipeline:

from ragas import EvaluationDataset, evaluate
from ragas.metrics import (
    ContextRecall, Faithfulness,
)

report = evaluate(
    EvaluationDataset.from_list(samples),
    metrics=[ContextRecall(), Faithfulness()],
)

The first two grade retrieval, the last two grade generation. That split is why RAGAS is worth naming: it forces the diagnosis this folder keeps insisting on — if faithfulness is high and context recall is low, prompting is not your problem.

DeepEval — pytest for LLMs

from deepeval import assert_test
from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase

def test_grounded():
    case = LLMTestCase(
        input=q, actual_output=answer,
        retrieval_context=chunks,
    )
    metric = FaithfulnessMetric(threshold=0.8)
    assert_test(case, [metric])

The framing is the selling point: evals as unit tests, run by pytest, so they live in CI next to everything else rather than in a notebook someone runs occasionally.

Its agentic metrics are the ones ads are starting to name — Task Completion, Tool Correctness, Step Efficiency, Plan Adherence — plus G-Eval for custom criteria and DAG for decision-tree scoring. If you are asked how to measure an agent rather than an answer, that list is the vocabulary.

The rest, briefly

• promptfoo — declarative YAML test cases, good for prompt A/B comparison and red-team scans, language-agnostic.
• TruLens — feedback functions over traces, research-flavoured.
• Giskard — leans toward scanning for vulnerabilities and bias.
• OpenAI Evals — provider-tied, fine if you are single-vendor.

Choosing, in one paragraph

Start with one observability tool wired to traces, because you cannot evaluate what you cannot see. Add RAGAS if you run RAG, since its four metrics answer the diagnostic question directly. Add DeepEval when you want evals gating CI, because pytest integration is the thing that makes them actually run. Reach for promptfoo when the task is comparing prompt variants rather than scoring a pipeline.

Gotcha: every LLM-judged metric here has the biases in LLM as judge — position, verbosity, self-preference — and they cost tokens per evaluation. A 200-case suite run on every PR is a real bill, and a judge scoring its own family’s output is a real bias. Calibrate against human labels on a subset.

What “we have evals” should mean

The tool is the least interesting part. The claim is credible when:

1. A golden set from real queries exists and is versioned.
2. It runs in CI, gating merges on regression, not on a laptop.
3. Retrieval and generation are scored separately.
4. Production failures flow back into the set.
5. Someone can say what the numbers were last month.

A team with a spreadsheet and those five properties is ahead of a team with Langfuse and none of them — which is the honest thing to say when asked which tool you prefer.

Related

• Why evaluating LLM systems is hard — why this is not ordinary testing
• Building eval sets — the golden set these tools consume
• LLM as judge — the biases in the scoring
• LLM observability — the tracing half