Determinism and reproducible LLM tests

Every team that writes its first LLM test hits the same wall: the same input produces a different string, so assert response == expected fails on the second run. The instinct is to hunt for the setting that makes it stop. There isn’t one, and the useful skill is building tests that don’t need it.

The mechanism — temperature 0 is greedy decoding, but GPU batching changes floating-point reduction order, so identical requests can still diverge — is in LLM inference fundamentals.

What the knobs actually promise

temperature=0 removes deliberate randomness. It does not remove non-determinism, because the source is arithmetic, not sampling: your request gets batched with whatever else arrived, batch shape changes the reduction order in the matmuls, and a tie between two near-equal logits breaks differently. One token differs, and because generation is autoregressive, the rest of the response diverges from there.

seed fixes the pseudo-random draw used when sampling. Combined with temperature=0 it is best effort, which is the word providers use. It is worth setting and worth not relying on.

system_fingerprint is the honest one. It identifies the backend configuration, and providers return it so that when your output changes you can tell whether the model or the infrastructure moved underneath you. Log it with every call; when an eval score shifts overnight and nothing in your repo changed, this is the field that explains it.

Two different problems

Conflating them is the common mistake.

Reproducing a specific past response — for a bug report, a regression diff, a demo. Solved by recording, not by settings. Cache the response, or record it VCR-style and replay it in tests.

# The test does not call the provider. It exercises your code around it.
@pytest.fixture
def llm(monkeypatch):
    cassette = json.loads(Path("tests/cassettes/refund.json").read_text())
    monkeypatch.setattr(client.chat.completions, "create",
                        lambda **kw: FakeResponse(cassette))

This is the right default for unit tests. Most of what you want to test — prompt assembly, tool dispatch, parsing, error handling, retry logic — is your code, and your code is deterministic. Do not spend provider tokens proving that json.loads works.

Detecting that behaviour changed — the eval problem. Here reproducibility is not the goal at all, because a system that only works on one exact string is already broken. You want a stable measurement of a distribution.

Testing a non-deterministic system

Assert on properties, not on strings. In rough order of how often they earn their place:

def test_extraction(llm):
    out = extract_invoice(SAMPLE_PDF_TEXT)

    assert Invoice.model_validate(out)          # schema: exact
    assert out["total"] == Decimal("1240.50")   # extracted value: exact
    assert out["currency"] in {"EUR", "USD"}    # constrained: exact
    assert_cites_source(out, SAMPLE_PDF_TEXT)   # grounding: mechanical

Schema validity, extracted values, citation presence, refusal behaviour and forbidden content are all deterministic assertions about non-deterministic output. They catch the failures that matter and they never flake. Reach for a judge only for the qualities that genuinely resist mechanical checking, and know that a judge is itself a non-deterministic component — see LLM as judge.

Sample more than once. A single call tells you about one draw. If a behaviour matters, run it n times and assert on the rate:

@pytest.mark.parametrize("run", range(5))
def test_abstains_when_context_is_empty(run):
    assert answer("who is the CFO?", context=[]).abstained

Five passes is weak evidence, but it is much stronger than one, and a behaviour that fails one run in five is a real bug you would otherwise ship.

Separate the tiers. Unit tests replay cassettes and run on every commit. The eval suite calls the real provider, costs money, runs on a schedule and on changes to prompts or retrieval, and reports a score rather than pass/fail until it crosses a threshold you set in advance.

Where it breaks in production

• A pinned model version is not pinned behaviour. Providers patch serving stacks. system_fingerprint moving with no version change is the tell.
• Cassettes rot silently. A recorded response keeps your tests green through a real API change. Re-record on a schedule and diff.
• Flaky tests get retried instead of fixed. A test that passes on retry is measuring a rate you have not written down. Convert it into an explicit n-of-m assertion.
• Caching hides drift. A semantic cache makes repeat calls look deterministic in staging and then diverge in production on a cache miss.
• Chasing exact reproduction. Teams spend weeks on it. The output was never the contract; the properties are.

Related

• LLM inference fundamentals - sampling parameters and why temperature 0 is not deterministic.
• The LLM gateway and caching - exact, prefix and semantic caching as separate layers.
• Building eval sets
  • the eval tier, and paired comparison over absolute scores.
• LLM as judge - judges are non-deterministic components too.
• LLM JSON Validation — Common Interview Questions and Answers
  • validating structured output mechanically.