You test an AI system before production with three layers of evals, not manual spot-checking: unit evals that score individual prompt behavior against a fixed dataset, integration evals that run the full pipeline end to end, and regression evals that re-run both automatically every time you change a model, prompt, or dependency. If you're only testing by reading outputs and thinking "yeah, that looks right," you don't have a test suite. You have a vibe, and vibes don't catch the failure that shows up three weeks after launch on the one input pattern nobody thought to try.
The common wrong approach: reading outputs and calling it QA
Most teams test an LLM feature the way they'd test a UI: run it a few times, eyeball the responses, ship if nothing looks broken. This feels reasonable because the failures are often subtle — not a stack trace, just a slightly wrong summary, a JSON field that's technically valid but semantically off, a classification that's confident and incorrect. Nothing crashes, so nothing gets caught.
It breaks the moment you touch anything. Bump the model version, tweak a system prompt to fix one bug, add a new tool — and you have no way to know what else moved. We've seen a one-line prompt change meant to fix a formatting issue quietly drop the model's tool-calling accuracy on edge-case inputs by 15%. Nobody noticed until a customer did. Manual testing samples a handful of cases you already thought of. It cannot tell you about the 200 cases you didn't.
The better approach: three layers of evals
Unit evals test one prompt or one model call in isolation, against a fixed dataset of inputs with known-good outputs or scoring criteria. For Email Triage, a Dainty project that classifies inbound email into action categories, this means a dataset of 150+ real (anonymized) emails with human-labeled correct categories, run through the classifier on every change, scored for exact match and near-miss. We use promptfoo or a thin custom harness on top of pytest — the tool matters less than having a dataset that's actually representative of production traffic, including the annoying edge cases: forwarded threads, auto-replies, emails in a second language.
For outputs that aren't exact-match — summaries, generated copy, extracted structured data with some tolerance — we score with an LLM-as-judge: a separate model call that grades the output against a rubric (correctness, completeness, tone) and returns a 1–5 score plus a reason. This is not perfect, but it's consistent, and consistency is what you need to detect regressions, not a philosophical guarantee of correctness.
Integration evals run the whole pipeline — retrieval, tool calls, multi-step agent loops, the works — against realistic scenarios, and check the final outcome, not just one step. For Ghost Writer, our AI drafting tool, that means feeding in a full brief-to-draft workflow and checking the final draft against style and factual-accuracy criteria, not just whether the first LLM call parsed correctly. Integration evals catch the failures unit evals structurally can't: a retrieval step that returns stale context, a tool call that succeeds but with the wrong arguments, state that leaks between steps in a multi-turn agent.
Regression evals are the first two, wired into CI, run on every prompt change, every model version bump, every dependency update. This is the layer that actually earns its keep. Anthropic and OpenAI ship model updates that change behavior in ways their release notes don't fully capture. Without a regression suite, you find out from your users. With one, you find out in a PR check, before it ships.
Where this breaks
Eval suites cost real time to build and maintain, and for a two-person team shipping a first AI feature, a full three-layer stack is probably overkill on day one. Start with unit evals on your highest-risk prompt — the one where a wrong answer costs money or trust — before building the rest out. LLM-as-judge scoring also drifts: the judge model updates too, and its grading criteria can shift under you, so re-baseline it periodically against human review, don't treat its scores as ground truth forever. And evals only test what's in your dataset — if your production traffic shifts (new user segment, new input format), your eval set goes stale and gives you false confidence. Budget time to refresh it quarterly, not once at launch.
What to do this week
Pick your single highest-stakes prompt — the one where being wrong is expensive — and write 20 real examples with expected outputs or a clear pass/fail rubric. Run it through promptfoo or a 50-line pytest script. That's your first unit eval. It won't cover everything, but it will catch the next regression before your users do, and it's the seed that the integration and regression layers grow from. If you want help building this out for a production system, start a project with us — this is exactly the kind of gap we get called in to close, most recently on a structured-extraction pipeline for CV Matcher where the eval suite caught a model-update regression before it ever reached a customer.