Vibe Checks Ship Broken AI: Designing Evals You Can Trust
Stop breaking production with casual AI testing. Design evals that catch model drift and quality issues before they cost you customers and revenue.
Most teams testing an AI feature do the same thing: open a playground, type five prompts, nod, ship. Mitesh Shah calls this out precisely in How to Test AI Agents — "a vibe check with a deployment pipeline attached." That phrase deserves to be on a wall somewhere. It is also the honest answer to anyone asking how to start vibe coding responsibly: the coding part is fast now, the verification part is where teams still get burned.
The problem is not laziness. It is that traditional software testing gives a binary answer and AI systems do not. As the Dust blog puts it: no two outputs are the same, there is no agreed definition of "correct," and quality can drift without warning after a model or prompt change. You can ship the exact same code on Tuesday and get worse results than Monday, because the model underneath moved.
Why this matters commercially, not just technically
An AI feature that fails silently is worse than one that crashes. A crash generates a ticket. A subtly wrong answer generates a confidently wrong invoice, a hallucinated policy quote, or a support conversation that quietly erodes trust with a customer who never complains — they just leave.
Anthropic's engineering team framed the root cause well in Demystifying evals for AI agents (January 2026): the capabilities that make agents useful — autonomy, intelligence, flexibility — are exactly what make them difficult to evaluate. An agent that can find a creative path to a goal will also find creative paths around your test assertions.
Their example is instructive. Opus 4.5 solved a τ2-bench flight-booking problem by discovering a loophole in the stated policy. The eval marked it as a failure. The model had actually found a better outcome for the user. The eval was wrong, not the agent.
Step 1: Define what "good" means before writing a single test
The prerequisite step is unglamorous and skipped constantly: sit down and answer what a successful interaction actually looks like. Be specific. "Helpful and accurate" is not a spec. "Cites at least one retrieved document, never quotes a price not present in the catalog, escalates to a human when the user mentions a refund" — that is a spec.
The mindset shift matters here. The question is not "does this match the expected output?" It is "is this output good enough, safe enough, and grounded enough for production?" Those are different questions and they require different measurement.
Anthropic draws a distinction worth internalising: the transcript versus the outcome. An agent might say "your flight is booked." The outcome is whether a reservation actually exists in the SQL database. Grade the database, not the prose. Anywhere a real state change exists — a record written, an email queued, a ticket status flipped — check the state, not the narration.
Step 2: Start small. Twenty to thirty cases.
Teams either build no evals or attempt a 500-case benchmark that dies half-finished. Dust's recommendation is the practical middle: twenty to thirty well-chosen test cases often provide enough signal to catch the most common reliability issues before real users hit them.
Real numbers: thirty cases is roughly an afternoon of work for a team that already knows its domain. That is a trivially small investment against the cost of a production incident.
Tricentis makes the same argument in its step-by-step guide to testing AI agents: start with a small, focused benchmark, build from what human review surfaces, automate the checks that are well understood and stable, and keep humans in the loop for the rest.
Where do those thirty cases come from? Three buckets, deliberately unbalanced:
- The five or six most common real requests (your happy path — necessary, but the least informative)
- The awkward middle: ambiguous phrasing, missing context, users who ask two things at once
- The genuinely nasty: adversarial prompts, out-of-scope requests, cases where the correct answer is "I don't know"
Tricentis names testing only happy-path cases as a primary failure mode. If the benchmark does not reflect the real distribution of inputs, the evals are optimistic in a way that eventually costs money.
Step 3: Choose your scoring method honestly
Three approaches, and most serious systems use all three.
Deterministic checks
Did the JSON parse? Is the ID in the valid set? Does the price match the catalog? These are cheap, fast, and never argue. Use them for everything that can be checked exactly. They belong in CI like any other unit test.
Semantic similarity
Compare the output to a reference answer in embedding space. Useful when phrasing varies but meaning should not. Weaker when there are many legitimately different good answers.
LLM-as-judge
A language model grades the response against a rubric — for example, whether an answer is grounded in the retrieved context (faithfulness) or whether it actually completed the task. Harness AI Evals combines LLM-as-judge with deterministic checks and semantic similarity, and lets teams write rubrics in natural language or custom scoring logic in Python. Their platform offers over 50 built-in metrics covering deterministic checks, safety and policy compliance, prompt injection detection, and trajectory evaluation for multi-step agents.
Honest take: LLM-as-judge is powerful and it drifts. Swivl documented exactly this failure in its M2 rubric scoring write-up: the system returned scores clustered in the middle of the scale, producing mostly 2s and 3s regardless of actual quality. A judge that always says "3" is a random number generator with good manners.
The fix is calibration against human graders. M2's scoring is explicitly calibrated to match how trained evaluators apply rubric criteria, and continuously monitored to stay that way. The stated design goal is temporal consistency: a score of 3 on Monday should mean the same thing as a 3 two weeks later.
The rubric-based auto-grading framework from 8allocate generalises this into five layers of trust: standards-based integration, rubric calibration, bias controls, human oversight, and compliance infrastructure. The hybrid model is the operative recommendation — AI handles routine scoring at scale, humans keep control over the calls that carry consequences. Calibration is what minimises drift; training samples get periodically refreshed as expectations change.
Step 4: Version everything, or you're flying blind
Tricentis names testing without version tracking as a distinct failure mode, and the phrasing is blunt: if you cannot determine whether performance on a given eval improved or regressed between two model updates, you are navigating without instruments.
The minimum bar is low. Even basic tagging of eval runs with model version and date gives you something to reason about. The audit trail matters most at exactly the moment things break and someone needs to reconstruct the sequence of events.
Put simply: an eval score without a version tag is a number with no story attached.
Step 5: Treat production as your best test-case factory
This is where evals stop being a checkbox and start compounding.
Arthur's argument in Evaluating AI Agents in Production is that pre-deployment test suites are simply not enough. Agents are non-deterministic, multi-step and tool-using, and production inputs are far more diverse than any handwritten test set can cover. Their model is a connected loop: observability provides the traces, continuous evals surface failures from those traces, guardrails intercept problems in real time, and experiments turn failures into durable improvements.
The mechanics of the loop are simple. Monitor conversations. Flag interesting failures. Add the failure case to the dataset with the expected correct behaviour. Run the eval suite to verify the fix. Repeat.
Every production failure that becomes a test case makes the system permanently harder to break in that exact way. Yesterday's incident becomes tomorrow's boring green check. The framing that changes team behaviour: treat every production incident as evidence of an eval gap, not just a support ticket to close.
That dataset then does double duty. Arthur notes the same production-derived dataset works as a regression suite, re-run before every deployment. Harness makes this explicit at the tooling level — offline and online evaluation share the same metrics and datasets, so there is no drift between what was tested pre-deploy and what is measured in production.
Step 6: Iterate at the cheapest layer first
Arthur separates experiments into three types, ordered by cost:
- Prompt experiments — run a prompt against a dataset without the full agent. The fastest iteration loop.
- RAG experiments — verify that retrieval is returning the right context. If retrieval is broken, no prompt fixes it.
- Agent experiments — run the full agent end to end, evaluating both the final output and the intermediate trajectory.
The recommended workflow: start narrow, iterate at the prompt or RAG level, then validate end to end before promoting. Most teams do the opposite, debugging a full multi-step agent when the actual bug is a retrieval query returning the wrong three documents. The distinction between prompt-level and context-level fixes is itself a discipline — worth reading alongside what changed when context engineering replaced prompt engineering.
Step 7: Wire evals into the pipeline as a gate
An eval suite that runs when someone remembers is not a quality control. Harness structures quality bars around four building blocks — Target, Dataset, Metrics, and Threshold — where thresholds are either all-must-pass (blocking) or weighted composite (advisory). Evals compose into Suites that attach to the release pipeline as a native quality gate.
Their evaluation SDK is open source under Apache 2.0, so scoring logic runs in any CI system, notebook, or test suite. The commercial platform layers pipeline gating, an AI asset registry, online evaluation and governance on top. AI Evals is currently in beta, developed with design partners running agents in production.
Here is what we recommend on gating strategy: make safety and policy checks blocking, and make quality scores advisory at first. A blocking quality threshold set before you trust your judge will train the team to disable the gate. Earn the right to block.
Multiple trials, because one run proves nothing
Anthropic defines a task as a single test with defined inputs and success criteria — and notes that multiple trials are run per task because model outputs vary between runs. A single pass on a single run is not evidence. If a case passes three times out of five, that is a 60% pass rate, not a pass.
This changes budgeting. Thirty cases × three trials × a judge call per trial is a real cost per eval run. Deterministic checks help here: run them on every trial, reserve the expensive LLM judge for the cases that need judgment.
Failure modes to avoid
Compiled from Tricentis and the broader practice:
- Happy-path-only benchmarks. Optimistic in a way that costs money later.
- No version tracking. No instruments, no navigation.
- Grading the transcript instead of the outcome. The agent says booked; the database says nothing.
- An uncalibrated judge. Middle-clustered scores that look like measurement but carry no signal.
- Evals that never change. Static suites go stale as the product and users move.
Key takeaway for business
Key takeaway for business: three things determine whether AI evals are worth the engineering time.
- Start at thirty cases, not five hundred. Twenty to thirty well-chosen cases catch the most common reliability issues. The suite that exists beats the suite that was planned.
- Make production failures permanent. Every incident converted into a test case is a class of bug that cannot recur silently. That compounding is the whole return on investment.
- Version and gate. Tag every run with model version and date, then attach the suite to the release pipeline. Without both, you have opinions about quality rather than measurements.
The teams building reliable AI products are not the ones with the best prompts. They are the ones who can answer, with a number and a date, whether last week's change made things better. And that discipline sits on top of ordinary engineering judgment — which is why coding still matters when building AI agents.
Frequently Asked Questions
How do you prevent metric gaming, where optimizing one measure degrades the system?
Use composite thresholds rather than a single score, and keep at least one metric the team is not optimizing against as a control. Harness supports weighted composite thresholds for exactly this reason. Also keep human review in the loop for judgment calls — as Tricentis recommends, automate only the checks that are well understood and stable.
How do you evaluate performance when real-world data drifts from your test set?
Production monitoring is the answer, not a bigger initial test set. Arthur's loop treats continuous evals on production traces as the source of new dataset entries, so the benchmark tracks the real input distribution instead of the one you imagined at launch. Refresh calibration samples periodically, as the rubric-scoring practice recommends.
What should differ between evaluating decision-making agents and evaluating language models?
For language models, output quality is the thing being measured. For agents that act, the final state in the environment is what counts — Anthropic's distinction between outcome and transcript. Agent evaluation also needs trajectory checks: whether the intermediate steps and tool calls were sound, not just whether the ending looked acceptable.
How do you make trustworthy-AI principles concrete rather than theoretical?
Turn each principle into a metric with a threshold in the pipeline. Faithfulness becomes a grounding check against retrieved context; safety becomes policy-compliance and prompt-injection metrics; oversight becomes a defined set of cases routed to humans. If a principle has no failing test case attached to it, it is not yet operational.
This article is based on publicly available sources and may contain inaccuracies.


