Three months after shipping a customer support agent, we pushed a system prompt update to improve tone. Seven days later, our escalation rate climbed 14%. Nobody noticed until a customer sent a screenshot showing the agent confidently giving wrong refund policy information, the kind it had handled correctly for weeks.
We had staging. We had manual QA. We had a senior engineer review the prompt diff. What we didn't have was a test suite that could catch a regression in refund-policy accuracy while measuring tone improvement at the same time.
That incident is where I learned that LLM evaluation is not optional for production systems. It's the thing that keeps a 3am system prompt tweak from becoming a Monday incident review.
This post is a practical guide to building evals that work: not as a checklist exercise, but as an engineering discipline that catches the failures you care about before they reach users.
Why Manual QA Fails at Scale
The problem with manual LLM testing is that language model behavior is probabilistic, multi-dimensional, and context-sensitive. A human reviewer checking ten sample outputs will miss the edge case that appears 0.3% of the time. At 100,000 turns per day, that's 300 failures. Per day.
When we audited our pre-incident QA process, we found three structural problems:
Coverage is sparse by design. Our QA reviewer checked 20-30 outputs per release. In our experience, production distributions span hundreds of distinct intent categories. We were sampling less than 8% of the space.
Reviewers anchor on the change. When a prompt is modified to improve tone, reviewers evaluate tone. They don't systematically check whether factual accuracy, policy compliance, or escalation behavior changed. The changed dimension crowds out the unchanged ones.
There's no baseline. Without a recorded baseline, "does this output look right?" is the full evaluation. A regression from 94% accuracy to 87% accuracy on policy questions is invisible to the human eye when reviewing individual samples.
The fix is to stop treating LLM testing as QA and start treating it as engineering: codify your quality criteria, measure them programmatically, and run them on every change.
# What we had before: ad hoc manual review
def review_output(prompt, response):
print(f"Prompt: {prompt}")
print(f"Response: {response}")
rating = input("Rate 1-5: ")
return int(rating)
# What we needed: an eval harness
def run_eval_suite(model_fn, test_cases, evaluators):
results = []
for case in test_cases:
response = model_fn(case["prompt"])
scores = {
name: evaluator(case, response)
for name, evaluator in evaluators.items()
}
results.append({
"case_id": case["id"],
"response": response,
"scores": scores,
"passed": all(v >= case.get("threshold", {}).get(k, 0.8)
for k, v in scores.items())
})
return results
The Three Layers of LLM Evaluation
A production eval suite has three distinct layers. Each catches different failure modes. Skipping any one of them leaves a gap.
Layer 1: Deterministic Evals
Deterministic evals check things you can verify with code: format compliance, required field presence, length bounds, prohibited string patterns, JSON schema validity. These run in milliseconds, cost nothing, and should be your first gate.
import re
import json
def eval_format_compliance(case, response):
"""Check that response meets structural requirements."""
checks = []
# JSON output when required
if case.get("requires_json"):
try:
json.loads(response)
checks.append(1.0)
except json.JSONDecodeError:
checks.append(0.0)
# Length bounds
if "max_words" in case:
word_count = len(response.split())
checks.append(1.0 if word_count <= case["max_words"] else 0.0)
# Prohibited phrases (legal/brand compliance)
prohibited = case.get("prohibited_phrases", [])
for phrase in prohibited:
if phrase.lower() in response.lower():
checks.append(0.0)
break
else:
if prohibited:
checks.append(1.0)
return sum(checks) / len(checks) if checks else 1.0
def eval_required_elements(case, response):
"""Check that required elements appear in the response."""
required = case.get("required_elements", [])
if not required:
return 1.0
found = sum(1 for elem in required if elem.lower() in response.lower())
return found / len(required)
Deterministic evals are also where you catch safety regressions fast. If your model should never output a phone number, a credit card pattern, or a competitor's name: that's a regex check, not an LLM-as-judge call.
In our case, we had seventeen deterministic checks covering format, prohibited phrases, required policy disclosures, and response length bounds. These ran on every pull request and caught around 40% of regressions without spending a single inference token.
Layer 2: LLM-as-Judge
LLM-as-judge uses a separate, typically stronger model to evaluate response quality on dimensions that resist algorithmic measurement: factual correctness, helpfulness, tone, reasoning quality, and policy compliance.
The key insight is that the judge model doesn't need to be the same model under test. We use Claude claude-opus-4-8 as a judge for outputs from a smaller, faster model; the judge has better calibration and can reason about nuanced quality dimensions.
import anthropic
client = anthropic.Anthropic()
JUDGE_PROMPT = """You are evaluating an AI assistant response for quality and correctness.
Question asked: {question}
Expected criteria: {criteria}
Response to evaluate: {response}
Score the response on each criterion from 0.0 to 1.0.
Return a JSON object with keys matching the criteria names.
Be strict. A score of 0.8 means "mostly correct with minor issues."
A score of 1.0 means "completely correct and appropriately detailed."
A score below 0.5 means the response has a significant problem."""
def llm_judge(case, response):
"""Use Claude as a judge to evaluate response quality."""
criteria = case.get("judge_criteria", {
"accuracy": "Is the information factually correct?",
"helpfulness": "Does the response actually help the user?",
"tone": "Is the tone appropriate for a customer support context?"
})
judge_response = client.messages.create(
model="claude-opus-4-8",
max_tokens=512,
messages=[{
"role": "user",
"content": JUDGE_PROMPT.format(
question=case["prompt"],
criteria="\n".join(f"- {k}: {v}" for k, v in criteria.items()),
response=response
)
}]
)
import json
try:
scores = json.loads(judge_response.content[0].text)
return scores
except (json.JSONDecodeError, KeyError, IndexError):
return {k: 0.5 for k in criteria}
The common failure mode with LLM-as-judge is prompt ambiguity. We measured this: our initial judge prompt produced inter-judge agreement of only 61% (two different judge prompt variants scoring the same outputs). After standardizing scoring rubrics and adding few-shot calibration examples, we reached 89% agreement, per our internal calibration runs across 500 scored pairs.
Critical rules for LLM-as-judge:
- Anchor the scale with examples. "Score 0.0 to 1.0" means nothing without calibration examples showing what a 0.3 looks like versus a 0.9.
- Separate dimensions. Don't ask the judge to produce a single score; ask for factual accuracy separately from tone separately from completeness.
- Validate judge calibration. Periodically take outputs your team has manually scored and check whether the judge agrees. If agreement drops, the judge prompt has drifted.
def calibrate_judge(judge_fn, human_scored_cases, threshold=0.85):
"""Check judge agreement with human scores on calibration set."""
agreements = []
for case in human_scored_cases:
judge_scores = judge_fn(case, case["human_reviewed_response"])
for dimension, human_score in case["human_scores"].items():
judge_score = judge_scores.get(dimension, 0.5)
# Agreement = within 0.15 of human score
agreements.append(abs(judge_score - human_score) <= 0.15)
agreement_rate = sum(agreements) / len(agreements)
print(f"Judge calibration: {agreement_rate:.1%} agreement")
if agreement_rate < threshold:
print("WARNING: Judge calibration below threshold. Review judge prompt.")
return agreement_rate
Layer 3: End-to-End Scenario Tests
End-to-end scenario tests simulate complete multi-turn conversations against your production system prompt. These catch the failures that only appear in context: a model that handles each individual turn correctly but loses track of a key fact across three turns; an agent that correctly identifies a tool to call but fails when that tool returns an unexpected response format.
def run_scenario(scenario, model_fn):
"""Run a complete multi-turn scenario and evaluate the final state."""
conversation = []
for turn in scenario["turns"]:
conversation.append({"role": "user", "content": turn["user"]})
response = model_fn(conversation)
conversation.append({"role": "assistant", "content": response})
# Mid-turn assertions (optional: check invariants at each step)
for assertion in turn.get("assertions", []):
result = assertion["fn"](response)
if not result and assertion.get("required", True):
return {
"passed": False,
"failure_turn": turn["id"],
"failure_assertion": assertion["name"],
"conversation": conversation
}
# Final state evaluation
final_response = conversation[-1]["content"]
final_scores = {}
for evaluator_name, evaluator_fn in scenario["final_evaluators"].items():
final_scores[evaluator_name] = evaluator_fn(scenario, final_response)
return {
"passed": all(v >= 0.8 for v in final_scores.values()),
"scores": final_scores,
"conversation": conversation
}
We have 47 end-to-end scenarios covering our most common and highest-stakes conversation flows. These are expensive to run (full model inference for each turn, plus LLM-as-judge on the final output), so they run on merge to main, not on every PR. In our experience, median scenario runtime sits around four seconds.
Building a Golden Dataset
A golden dataset is a curated set of (input, expected criteria) pairs that represents your production distribution and captures your known failure modes. It's the foundation of meaningful regression detection.
Building it well requires intentionality. A golden dataset built entirely from easy cases will give you 97% pass rates and zero useful signal.
from dataclasses import dataclass
from typing import Callable, Optional
import json
@dataclass
class EvalCase:
id: str
prompt: str
conversation_context: list # prior turns if multi-turn
judge_criteria: dict # dimension -> description
thresholds: dict # dimension -> minimum score
required_elements: list # must appear in response
prohibited_phrases: list # must not appear
tags: list # for filtering/analysis
source: str # "production", "synthetic", "edge_case"
def build_golden_dataset():
"""Framework for golden dataset construction."""
cases = []
# 1. Sample from production logs (real distribution)
production_samples = sample_production_logs(
n=200,
stratify_by="intent_category", # even coverage across intents
filter_fn=lambda x: x.get("escalated") or x.get("low_rating")
)
# 2. Synthesize adversarial cases
adversarial = synthesize_adversarial_cases(
seed_cases=production_samples[:20],
perturbations=["rephrase", "add_noise", "boundary_condition"]
)
# 3. Add regression cases from past incidents
regression_cases = load_known_failure_cases("incidents/")
cases.extend(production_samples)
cases.extend(adversarial)
cases.extend(regression_cases)
return cases
Three rules for golden dataset quality:
Stratify by production distribution, not by what you think matters. Pull real intent distribution data from your logs and ensure your test cases match it proportionally. If 30% of your production traffic is refund questions, 30% of your test cases should be refund questions.
Weight failure modes heavily. Cases that caused past incidents, edge cases from user feedback, and boundary conditions around policy rules deserve disproportionate representation. Your golden dataset isn't random sampling; it's risk-weighted sampling.
Annotate with source. Every case should record whether it came from production logs, synthetic generation, or a past incident. This lets you analyze pass rates by source and identify whether your synthetic generation is representative.
Regression Detection and CI/CD Integration
An eval suite is only useful if you run it continuously and act on the results. Regression detection requires establishing baselines and alerting when scores drop below threshold.
import json
import os
from pathlib import Path
import statistics
def run_eval_with_regression_check(
model_fn,
test_cases,
evaluators,
baseline_file="eval-baselines/current.json",
regression_threshold=0.03, # alert if any dimension drops > 3%
):
"""Run eval suite and check for regressions against baseline."""
# Run current eval
results = run_eval_suite(model_fn, test_cases, evaluators)
# Compute aggregate scores per dimension
current_scores = {}
for evaluator_name in evaluators:
dimension_scores = [
r["scores"][evaluator_name]
for r in results
if evaluator_name in r["scores"]
]
current_scores[evaluator_name] = statistics.mean(dimension_scores)
# Load and compare baseline
baseline_path = Path(baseline_file)
regressions = []
if baseline_path.exists():
baseline = json.loads(baseline_path.read_text())
for dimension, current_score in current_scores.items():
baseline_score = baseline.get(dimension)
if baseline_score is None:
continue
drop = baseline_score - current_score
if drop > regression_threshold:
regressions.append({
"dimension": dimension,
"baseline": baseline_score,
"current": current_score,
"drop": drop
})
return {
"current_scores": current_scores,
"regressions": regressions,
"passed": len(regressions) == 0,
"raw_results": results
}
def update_baseline(scores, baseline_file="eval-baselines/current.json"):
"""Update baseline after human sign-off on new scores."""
path = Path(baseline_file)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(scores, indent=2))
print(f"Baseline updated: {scores}")
The hardest part of baseline management is deciding when to update the baseline. Our rule: if a score drops, investigate before updating. If a score improves, update the baseline automatically after a 48-hour soak. This prevents score inflation from gradual drift while capturing genuine improvements.
For CI/CD integration, we use a GitHub Actions workflow that runs the deterministic and LLM-as-judge layers on every PR. The end-to-end layer runs nightly and on merges to main.
# .github/workflows/eval.yml
name: LLM Eval Suite
on:
pull_request:
push:
branches: [main]
jobs:
deterministic-evals:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run deterministic evals
run: python scripts/run_evals.py --layer deterministic
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
llm-judge-evals:
runs-on: ubuntu-latest
needs: deterministic-evals
steps:
- uses: actions/checkout@v4
- name: Run LLM-as-judge on sample
run: python scripts/run_evals.py --layer judge --sample-rate 0.3
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
The Metric Worth Tracking from Day One
The single most useful eval metric to log from day one is pass rate by test case category, not aggregate pass rate.
An aggregate pass rate of 91% can hide the fact that your refund-policy category is at 73% and your escalation-detection category is at 61%. Both of those are production fires in slow motion.
def aggregate_results_by_category(results, test_cases):
"""Compute pass rates broken down by case tag/category."""
by_category = {}
for result in results:
case = next(c for c in test_cases if c.id == result["case_id"])
for tag in case.tags:
if tag not in by_category:
by_category[tag] = {"passed": 0, "total": 0}
by_category[tag]["total"] += 1
if result["passed"]:
by_category[tag]["passed"] += 1
return {
category: {
"pass_rate": stats["passed"] / stats["total"],
"n": stats["total"]
}
for category, stats in by_category.items()
}
In our system, we emit these per-category metrics to Prometheus and display them in Grafana. When a specific category drops, we know exactly which eval cases to examine, pointing us directly to which prompt section or which tool behavior regressed.
Production Considerations
Eval cost scales with quality. Deterministic evals cost nothing. LLM-as-judge costs inference tokens. End-to-end scenarios cost the most. Structure your CI pipeline to gate on cheap evals first so you only pay for expensive evals when the cheap gates pass.
Don't eval with the same model you're testing. If you use Claude claude-sonnet-5 as your production model and Claude claude-sonnet-5 as your judge, the judge will be biased toward the same failure modes as the production model. Use a larger or different model as judge.
Synthetic test case generation degrades. Synthetic cases generated by an LLM will cluster around modes the LLM finds natural. Over time, your golden dataset will underrepresent the long tail of real production inputs. Schedule periodic reviews to inject new cases from production logs.
Version your evals alongside your prompts. An eval suite that tests last month's prompt spec is worse than no eval suite; it gives false confidence. Store evals in the same repository as your prompts and tag them together.
Golden dataset contamination is real. If your production model was trained on data that included outputs similar to your golden dataset, your evals will overstate performance. This is especially relevant if you're fine-tuning. Test on held-out data that wasn't in any training pipeline.
Conclusion
LLM evaluation is the engineering discipline that separates teams that discover regressions from users from teams that discover them in incident reviews. The three layers (deterministic evals, LLM-as-judge, and end-to-end scenarios) cover different failure modes and run at different costs. Starting with deterministic evals costs nothing and catches a surprising fraction of bugs. Adding LLM-as-judge with careful calibration catches quality regressions across multiple dimensions. End-to-end scenarios catch the failures that only appear across multi-turn context.
The investment pays back within weeks. The incident that prompted all this work for our team would have been caught by a twelve-case golden dataset and a single LLM-as-judge check on refund-policy accuracy. Twelve cases, run on every PR, would have blocked the change.
Build the eval suite before you need it. You will need it.
Get the next one
Every week: one production LLM bug, debugged, plus the companion code for each deep-dive.
Subscribe to AI Engineering Weekly — no spam, unsubscribe anytime.
Can you catch a tone regression without breaking accuracy? That's the eval problem. What's the hardest quality dimension you've had to measure in production?
Sources
- Anthropic — Building Effective Agents: Evals — official guidance on evaluation methodology for Claude-based systems
- Hamel Husain — Your AI Product Needs Evals — practitioner guide on building evaluation pipelines for production LLM applications
- Brinkmann et al. — LLM-as-a-Judge: A Survey — comprehensive survey of LLM-as-judge approaches, calibration methods, and known failure modes
About the Author
Toc Am
Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.
Published: 2026-07-05 · Written with AI assistance, reviewed by Toc Am.
Get These In Your Inbox
Weekly deep-dives on AI engineering, no fluff. Join the newsletter →
Or grab the book ($39, ~100 pages) · Buy me a coffee
☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter
No comments:
Post a Comment