Your AI Evaluation Set Is Probably Too Clean to Catch the Failures Users See
Build a durable LLM evaluation set from real tasks, edge cases, abstentions, and production failures without leaking sensitive data.
A benchmark made of easy examples tells a comforting story
Teams often begin evaluating a language model with a handful of carefully written prompts. The model performs well, so the feature ships. Then users ask incomplete questions, paste messy documents, switch languages, or rely on a subtle exception, and quality falls apart. A clean set can test whether a demo works; it cannot establish that a product is ready for the long tail.
Evaluation sets also decay. Product behavior changes, source data changes, and new failure modes appear. A static collection that was representative six months ago may now overstate performance. The task is to make the set a maintained engineering asset rather than a one-time launch artifact.
Start with the decisions the system supports. Define what a correct response looks like, what evidence is allowed, when the model should ask a question, and when it should refuse or abstain. Without those criteria, reviewers will score style and confidence instead of usefulness.
Sample real work responsibly
Collect task patterns from support queues, search logs, code reviews, and user research. Remove or transform personal data, credentials, and confidential details. Keep a documented lawful basis and access policy for any retained examples. Where raw examples cannot be retained, preserve the structure of the failure in a synthetic or redacted case reviewed by a domain expert.
Stratify by task type and risk. Include common requests, rare but costly cases, ambiguous inputs, long context, multilingual phrasing, malformed data, no-answer questions, and adversarial instructions. Make sure each important product segment has examples; a global average can hide a severe failure for one language or customer tier.
For each example, store the expected behavior, accepted alternatives, required evidence, severity, and reviewer notes. Some tasks have several valid answers. Define constraints and key facts instead of requiring exact wording. Keep a holdout set that prompt authors do not tune against, or your score will reward memorization of the evaluation examples.
Separate metrics by failure mode
Do not collapse every property into one score. Track factual correctness, evidence support, citation quality, instruction following, schema validity, refusal appropriateness, latency, and cost. For agent workflows, add tool selection, argument validity, permission compliance, successful verification, and recovery after tool failure.
Use deterministic tests for deterministic requirements. A JSON schema, permission boundary, or calculation should be checked in code. Human review is better for tone, usefulness, and whether a nuanced explanation meets the user's need. Model-based graders can scale review, but calibrate them against human judgments and inspect disagreements. A grader using a similar model may share the same blind spots as the system being evaluated.
Report confidence intervals or sample counts when comparing candidates. Small differences on a small set may be noise. Track results by category, not just as a single percentage. Decide in advance which regressions block release and which can be accepted with mitigation.
Turn production failures into regression cases
When a user reports a failure, classify it, remove sensitive details, and add a representative test if doing so is safe and useful. Preserve the failure condition: stale version, hidden exception, hostile document, ambiguous entity, or unsupported claim. Do not add only the exact prompt and answer if the important issue is a general behavior that should be tested across paraphrases.
Version the data, labels, prompts, model identifiers, and grading rules. Record why an example was added and who reviewed it. Retire cases only when the product behavior has changed or the example is no longer valid; keep the history so trend comparisons remain interpretable.
Run evaluations where they can change a decision
Use offline evaluations before changing models, prompts, retrieval, or tool schemas. Run shadow comparisons on eligible traffic, then a controlled canary. Keep human review for high-risk outputs and sample lower-risk outputs to detect drift. Monitor production correction rate, abandonment, escalation, and user reports alongside benchmark scores.
Protect the evaluation set itself. Restrict access to hidden cases, especially if they contain sensitive patterns or security tests. Avoid sending data to an external grader without approval. Keep clear retention limits and a deletion process.
An evaluation set is not a trophy that proves a model is good. It is a map of what you have measured and what remains uncertain. Build it from real work, label the failure that matters, preserve hard examples, and connect scores to release decisions. The value is not a perfect number; it is fewer surprises when a change reaches users.