The uncomfortable truth about building with language models is that you often can't tell whether a change made things better. Swap a prompt, upgrade a model, adjust a retrieval step, and the output is different — but is it better? For traditional software you write a test that asserts an exact value. For a system whose correct output is "a good summary" or "a helpful answer," exact-match assertions are useless, and the temptation is to skip evaluation entirely and ship on vibes. That's how you end up with a system nobody can safely change. Here's how to evaluate one properly, and how to keep the evaluation itself from becoming a liability.
Start with a golden dataset
Everything begins with a set of examples where you know what good looks like. A golden dataset is a curated collection of inputs paired with reference outputs or explicit acceptance criteria — the fixtures of the LLM world. It doesn't need to be huge; a few dozen carefully chosen cases that cover your real traffic, the tricky edge cases, and the failure modes you've already been burned by will out-perform thousands of random samples.
The discipline is in the curation. Pull real inputs from production, include the cases that broke last time, and write down what "correct" means for each one before you look at any model's output — otherwise you'll rationalize whatever the current system produces as the target. This dataset is the asset. Models and prompts are cheap and swappable; a well-built golden set is what lets you swap them without fear.
Offline versus online evals
There are two moments to measure, and you need both.
Offline evals run against the golden dataset in your development loop, before anything ships. They're fast, repeatable, and cheap enough to run on every change, and they answer "did this change regress anything I already care about." Their blind spot is that they only test what's in the dataset — they can't tell you about inputs you didn't think to include.
Online evals measure the live system on real traffic: user thumbs-up/down, downstream conversion, escalation rates, or a judge scoring a sample of production responses. They catch the distribution shift that offline evals miss — the questions real users actually ask, which never match your imagination. The cost is latency, spend, and the delay of waiting for real interactions. Use offline evals as the gate for shipping and online evals as the ground truth that feeds new cases back into the golden set. The two form a loop, not a choice.
The LLM-as-judge pattern
For outputs with no single correct answer, the scalable move is to have a second model score the first. You give a judge model the input, the response, and a rubric ("rate factual accuracy from 1-5; a 5 cites only information present in the source"), and it returns a score with a rationale. Done well, this correlates with human judgment far better than any string-matching metric, at a fraction of the cost and latency of human review.
Done carelessly, it launders bias into a number that looks objective. A judge is a language model, with all the same failure modes as the model it's grading, and you have to design around them.
The biases that will bite you
Judges have systematic, well-documented tilts. Ignore them and your eval measures the judge's quirks instead of your system's quality.
- Position bias. When comparing two responses, judges favor whichever one they see first (or, in some models, last) regardless of content. Always run the comparison both ways — A-then-B and B-then-A — and only count it as a win if the verdict is consistent. The disagreement rate between the two orders is itself a useful signal of how confident the judge really is.
- Verbosity bias. Judges reward longer, more detailed answers even when the extra length adds nothing correct. If your rubric doesn't explicitly value concision, you'll drift toward padded outputs. Control for length, or tell the judge to.
- Self-preference bias. A model tends to rate outputs from its own family higher than those from other model families. If you use the same model to generate and to judge, you've baked in a thumb on the scale. Using a different model as judge than the one under test is the cheap mitigation.
None of these are reasons to abandon LLM judges. They're reasons to validate the judge against human labels on a slice of your data before you trust it — measure the judge's agreement with humans, and if it's poor, fix the rubric before you fix the system.
Pairwise versus pointwise scoring
Two ways to ask the judge for a verdict, with different reliability.
Pointwise scoring rates one response in isolation on an absolute scale ("give this answer a 1-5"). It's simple and it aggregates cleanly, but absolute scores are noisy — a judge's notion of "4 out of 5" drifts across runs and rubric wordings, and small quality differences vanish into that noise.
Pairwise scoring shows the judge two responses and asks which is better. Models are markedly more reliable at relative judgments than absolute ones, the same way a person can reliably say which of two coffees is better long before agreeing on a 1-10 scale. Pairwise is the stronger tool for "is version B better than version A," which is the question you usually actually have. The cost is that it doesn't give you an absolute quality level and it scales combinatorially if you compare everything to everything — so anchor new versions against a fixed baseline rather than running a full tournament.
Pick the cheapest method that works
Not every check needs a model. Match the tool to the property you're measuring, cheapest first:
- Exact-match or regex for anything with a deterministic correct answer — a classification label, a JSON field, an extracted number, a required disclaimer. These are free, instant, and unambiguous. Reach for them wherever the output space is closed.
- Semantic similarity (embedding distance to a reference) for "did it convey roughly the right content" without demanding the exact words. Cheap, fast, and good for catching gross regressions, though blunt about nuance.
- LLM-as-judge only for the genuinely open-ended qualities — helpfulness, tone, faithfulness, reasoning quality — where the first two can't reach. It's the most expensive and least deterministic option, so use it where it's the only thing that works, not as a default.
A well-built eval suite is mostly cheap deterministic checks with a thin layer of judge calls on top, not a wall of expensive model invocations.
Keeping eval cost sane
Evaluation runs on every change, so its cost compounds. A few practices keep it from ballooning:
- Right-size the judge. You rarely need your most expensive model as the grader. A mid-tier or small model, validated against human labels, is often a perfectly good judge at a fraction of the price — and cheaper judges mean you can afford to run the two-way position-swap that catches position bias.
- Sample online, don't grade everything. Judging every production response is wasteful; a representative sample tracks quality trends at a small fraction of the cost.
- Batch offline runs. Evals are latency-insensitive by nature, which makes them a textbook fit for asynchronous batch APIs that process requests at a discount — often half price — for the tolerance of a slower turnaround. There is no reason to pay real-time rates to grade a fixed dataset overnight.
- Cache the stable parts. A judge prompt with a long fixed rubric is exactly the repeated-prefix pattern prompt caching was built for; put the rubric behind a cache breakpoint and pay full price for it once.
The goal isn't a perfect measurement of quality, which doesn't exist. It's a measurement consistent and honest enough that you can change your system and know which direction you moved — cheaply enough that you'll actually run it every time. A skeptical, well-instrumented eval is the thing that turns "we think this is better" into "we checked."