To test an LLM model upgrade, run your existing eval suite against the candidate model and compare every case with the baseline you captured on the current model. With Drift-CI you test LLM model upgrade candidates with a one-line change in a pull request: the PR comment shows which cases regressed and which improved, and the check fails if anything dropped past your threshold.
Why model upgrades break prompts
A prompt is tuned against one model's habits, often without anyone noticing. A newer or cheaper model brings different ones:
- Formatting. One model returns bare JSON when asked; another wraps it in a code fence or adds a sentence of preamble. A parser that worked yesterday now throws.
- Instruction following. Smaller models are more likely to drift from "output only that word" or "no prose".
- Refusals. Models disagree about what to decline, in both directions.
- Sampling parameters. Some newer models reject parameters that older ones accepted. Claude models from the 4.7 generation on, for example, return a 400 if you send
temperature, so Drift-CI stops sending it to them.
None of this shows up in a changelog for your use case. You have to run your own inputs.
Test LLM model upgrade candidates in a pull request
Open a pull request that changes the model, and let the suite run. In our dogfood repo we tried two candidates for a support-ticket triage agent pinned to claude-sonnet-4-6.
The first PR changed the model in .drift/config.yaml:
provider:
name: anthropic
model: claude-haiku-4-5The second left the config alone and overrode the model on the GitHub Action instead:
- uses: Drift-CI/drift-ci@v1
with:
provider: anthropic
model: claude-sonnet-4-5 # candidate model for this PR
api-key: ${{ secrets.ANTHROPIC_API_KEY }}Both PRs failed the check, and in both the regressions landed on the same kind of case: the structured classification cases, which ask for a bare JSON object and validate it against a schema (both structured cases on Haiku, one on Sonnet 4.5). The bare-label classification cases held on both models. That's the useful part. Instead of "the new model seems fine", you get a per-case answer, and it points at the prompts that need work before you can switch.
The Haiku run also surfaced a configuration issue. The suite's judge model was claude-haiku-4-5, so once the model under test was Haiku too, the rubric-checklist cases refused to let a model grade its own output and came back as evaluator errors rather than scores. When you change the model under test, check that it isn't also your judge.
A prompt change is not a model change
Drift-CI separates these on purpose. Each baseline stores a hash of the case definition: input, system prompt, expected output, evaluators, schema and threshold. If you edit a case, the baseline is reported as stale and you're asked to re-accept it. A score drop there is expected, so it isn't counted as a regression.
The model isn't part of that hash. Swapping models with the cases unchanged compares the new model directly against baselines captured on the old one, so every drop is reported as a real regression. That's the comparison you want when deciding whether to switch.
If you go ahead, run the suite locally on the new model, accept the new baselines with npx @drift-ci/cli baseline accept .drift/suite.yaml --all, and commit them in the same PR so reviewers see exactly which outputs changed.
Comparing models over time in the dashboard
A single PR run tells you how the candidate does today. When runs sync to drift-ci.com, the project page lists every run with its provider and model, so you can watch a candidate across several runs before committing to it.
On the Team plan, org analytics adds a model comparison table: runs, average score, pass rate, average latency and estimated cost per run and per case for each model, plus a daily quality trend.
Cost matters here as much as quality. A cheaper model that holds every case is an easy switch; one that saves money but fails your structured-output cases isn't, at least not until you've fixed those prompts.
When the upgrade isn't the model
Upgrades also arrive through SDKs and API versions. A provider SDK bump can change request defaults or which parameters an API version accepts, while your mocked unit tests stay green. A live eval run on the dependency-bump PR is what catches that class of problem. We hit exactly this with an Anthropic SDK major version; the post-mortem on "temperature: Extra inputs are not permitted" walks through it.
Once you've switched, keep a scheduled run pointed at the pinned model. Providers can change behaviour behind a model name too; LLM drift monitoring covers that. For the basics of gating prompt edits the same way, see prompt regression testing, and for choosing evaluators and thresholds, the LLM regression testing guide.