LLM regression testing: a practical guide

LLM regression testing is the practice of running a fixed set of inputs through your LLM feature on every change and checking that quality didn't get worse. Instead of asserting exact text, each case is scored by an evaluator, compared with a baseline you previously approved, and flagged when the score drops past a threshold. Run in CI, it turns "the prompt seems fine" into a pass or fail on the pull request.

This guide covers how to build one that people trust: the dataset, the evaluators, baselines and thresholds, CI gating, noise, and what to watch after you ship. Examples use Drift-CI, the open-source tool we build, but the ideas carry over to any setup. If you're still choosing a tool, we've compared the main LLM evaluation tools for CI use.

What LLM regression testing is

Every LLM feature has a contract, even if nobody wrote it down. The classifier returns one of four labels. The extractor returns JSON with these fields. The support bot declines legal questions and never invents a refund policy. A regression is any change that breaks part of that contract: a prompt edit, a model swap, an SDK bump, or the provider changing the model behind the same name.

A regression test pins the contract to concrete examples. Each test case has an input, optionally a system prompt, and one or more evaluators that turn the model's response into a score between 0 and 1. A suite is a list of those cases. You run it, record the scores as a baseline, and from then on compare every run against that baseline.

This is different from a one-off eval or a leaderboard benchmark. The question isn't "how good is this model?" but "did my change make my feature worse on the inputs I care about?" That narrower question is what makes it cheap enough to run on every pull request.

Why exact-match tests fail for LLMs

The obvious first attempt is a unit test: send an input, compare the response to a stored string. It breaks almost immediately, for two reasons.

Outputs are not deterministic. The same prompt can produce different text on different calls. Temperature 0 narrows this a lot, but it doesn't guarantee identical output, and some newer models don't accept a temperature parameter at all.

Correct answers can be phrased many ways. "Your refund has been processed" and "We've processed your refund" are both right. A string comparison fails one of them, and a test that fails on harmless rewording gets ignored, then deleted.

The fix is to score properties of the output instead of its exact text, and to compare scores rather than strings. Exact match still has a place, but only for outputs where there is exactly one right answer.

Build a golden dataset

The golden dataset is the set of inputs your suite runs. It matters more than any tool choice.

• Start small and real. Ten to twenty inputs taken from real usage beat two hundred synthetic ones. Cover the main paths through your feature and the edge cases you already know about.
• Add every production failure. When a user reports a bad answer, turn that input into a case before you fix the prompt. The suite then proves the fix and guards against the bug coming back.
• Keep expected outputs reviewed. Where a case has an expected answer, someone who understands the product should have agreed it's correct. A suite that encodes wrong answers only protects the wrong behaviour.
• Separate concerns. One case per behaviour makes failures easy to read. A case that checks format, tone and accuracy at once tells you something broke but not what.

In Drift-CI the dataset lives in .drift/suite.yaml, in the same repository as the prompt, so changes to both are reviewed together. This case from our own dogfood suite checks a ticket classifier:

- id: classify-billing-label
  tags: [classification, exact-match]
  systemPrompt: |
    You are a support-ticket classifier. Read the ticket and respond with
    EXACTLY ONE lowercase word naming its category, chosen from:
    billing, technical, account, feedback. Output only that word.
  input: "I was charged twice for my subscription this month. Please refund the duplicate."
  evaluators: [exact-match]
  expected: "billing"

Choose evaluators

Pick the cheapest evaluator that captures what you care about for each case. Drift-CI ships seven, and a case can combine several into one weighted score.

Exact match for deterministic fields

When there is one right answer, such as a label, an ID or a yes/no, use exact-match. Drift-CI compares the whole response with expected after trimming whitespace and lowercasing both. It's fast, free and unambiguous, which makes it the best evaluator you have whenever it applies.

JSON schema for structure

If code parses the output, test what the code needs. The json-schema evaluator parses the raw response and validates it against a schema defined on the case: required keys, types, enums, ranges. Invalid JSON or a schema violation scores 0. This catches the most common structured-output failures, such as a model that starts wrapping JSON in a code fence. Testing LLM JSON output covers it in depth.

Semantic similarity

cosine-similarity embeds the response and the expected answer and scores how close they are. It runs a small embedding model locally, so it needs no API key. It suits answers where the meaning should stay stable but the wording can move. It's a blunt instrument: two answers can be close in embedding space and still differ on the one fact that matters.

LLM as a judge and rubric checklists

For qualities only a reader can judge, use a model as the grader. llm-judge asks a judge model to score the response from 0 to 1 against your criteria. rubric-checklist goes further: you list between two and twenty specific requirements, each graded strictly (pass or fail) or leniently (a partial score), and the case score is their weighted sum. Rubrics are much more stable than a single "rate this answer" prompt, because each item is a narrow question.

Two rules make judges trustworthy:

• Use a different judge model from the model under test. Models tend to rate their own output generously. Drift-CI's judge.allowSelfBias setting defaults to false: llm-judge warns when the judge and the tested model are the same, and rubric-checklist refuses to run. For rubric checklists you can also configure several named judges and require a majority or unanimous verdict.
• Write criteria a stranger could apply. "Is the answer good?" produces noise. "Declines the refund because more than 14 days have passed" produces a clear verdict.

Drift-CI also has refusal-detection, which scores 0 when the response opens with a refusal phrase, and safety-classifier, which sends the response to a moderation model.

Baselines and thresholds

A score on its own doesn't tell you much. A judge might give your best answer 0.92 forever. What matters is the change from a known-good state, which is what a baseline records.

Compare to a baseline, not to perfection. Once you've looked at a run and agreed the behaviour is right, store each case's score and output as its baseline. Future runs are judged by their delta against it. A case that scores 0.85 every day is fine; a case that drops from 0.95 to 0.70 is a regression.

Keep baselines in version control. Drift-CI writes one JSON file per case to .drift/baseline/, committed with the code. Accepting a behaviour change means committing new baseline files in the same pull request, so reviewers see the old and new outputs side by side. If you edit a case definition, Drift-CI marks its baseline as stale instead of reporting a regression, because the old score no longer describes the same test.

Use thresholds to absorb noise. A regression is a drop larger than the threshold. Drift-CI defaults to 0.10, set in .drift/config.yaml, and any case can override it. Judge-scored cases wobble a little between runs, so they need some slack; binary evaluators score 0 or 1, so any change crosses the threshold.

Use temperature 0 where the model allows it. Drift-CI sends temperature: 0 on every case, except to Claude models from the 4.7 generation on, which reject sampling parameters.

Gate pull requests in CI

LLM regression testing only works if it runs without anyone remembering to run it. Put it in CI, on every pull request that can affect model behaviour:

• Fail the check on regression. A red check with branch protection is the only gate people can't skip.
• Post the results where reviewers look. Drift-CI posts one PR comment, updated on each push, with each case's score, delta and latency, and instructions for accepting intentional changes.
• Export machine-readable results. The Drift-CI GitHub Action writes a JUnit XML report on every run (the CLI does too, with --junit-path), so CI systems that understand JUnit can show per-case results.

The GitHub Actions tutorial sets this up step by step, and the prompt regression testing use case shows what the PR experience looks like.

Handle flakiness and outages

Failures that aren't real teach people to ignore red checks. Three sources of noise are worth handling explicitly.

Provider errors. Rate limits, timeouts and network errors say nothing about your prompt. Drift-CI retries rate-limited calls with exponential backoff, gives the remaining failures their own statuses, and never counts them as regressions. If enough cases fail that way in one run (three cases or 20% of the suite, whichever is larger), the run aborts with a message that the provider is likely the problem.

Evaluator errors. Sometimes a case can't be scored at all, for example when a judge call fails. Those cases get no score and are excluded from the comparison, instead of counting as a drop to zero.

Genuine variance. Some cases will always move a little. Tighten the prompt, switch to a more deterministic evaluator, or give that case a looser threshold. Don't raise the global threshold to silence one noisy case.

Track quality over time

CI catches the changes you make. It doesn't catch the provider changing the model under you, or slow decay across many small changes that each stayed inside the threshold.

• Run the suite on a schedule against a pinned model. A failure with no code change means the model moved. LLM drift monitoring shows a nightly workflow.
• Keep run history. Per-case score charts and output diffs between runs show what changed and when.
• Track regressions as issues. On drift-ci.com's Team plan, a regressed case opens an issue that reopens if the case regresses again after it was resolved.
• Watch cost alongside quality. On the Team plan, the drift-ci.com dashboard estimates spend from each run's token usage, which matters most when you're weighing a cheaper model. Testing an LLM model upgrade walks through that comparison.

Common mistakes

A few patterns show up again and again when teams start testing LLM features.

Grading everything with one judge prompt. A single "score this answer from 0 to 1" judge is easy to set up and hard to trust. Its scores drift with wording you didn't change, and a low score doesn't tell you what went wrong. Split the question into rubric items, and use deterministic evaluators for anything code can check.

Accepting baselines without reading them. Accepting new baselines is a review decision, not a chore. If a pull request accepts twenty new outputs at once and nobody reads the diff, the suite has just learned whatever the model happens to say today.

Testing only the happy path. The inputs that break are rarely the ones in the demo. Ambiguous requests, out-of-scope questions, hostile users, empty fields and very long inputs belong in the suite from the start.

Letting the suite rot. Cases that test removed features, or that everyone knows are flaky, teach people to ignore red checks. Delete or fix them quickly; a smaller suite people trust beats a larger one they skip.

Treating a green run as proof of quality. A passing suite means nothing you tested got worse. It says nothing about inputs you haven't written down yet, which is why production failures keep feeding the dataset.

A checklist

• Write down the contract your LLM feature must keep.
• Build a golden dataset from real inputs, starting with ten to twenty cases.
• Add a case for every production failure before fixing it.
• Use exact-match or JSON schema wherever a deterministic check exists.
• Use rubric checklists with a different judge model for everything else.
• Commit baselines to the repository and review changes to them.
• Set a regression threshold, and override it per case instead of globally.
• Run the suite on every pull request and fail the check on regression.
• Treat provider errors as infrastructure problems, not regressions.
• Run the suite on a schedule to catch drift you didn't cause.