How many examples does an eval need?
Start with 20 to 50 realistic cases that cover common tasks and known failure modes. Grow it every time you find a new bug. A small eval you actually run beats a large one you never get around to.
A repeatable test set with a scoring method, used to measure whether an AI system does its job and catch regressions.
An eval, short for evaluation, is a repeatable test for an AI system. It is a set of example inputs plus a way to score the outputs. Scoring can be exact checks, like whether code passes its tests or a field matches the expected value, human ratings, or another model acting as a grader against a written rubric. You run the eval before and after a change, such as a new prompt, model or instructions file, and compare the scores. Because model outputs vary, evals usually run many examples, sometimes several times each, to see a real difference rather than luck.
Without evals, changes to AI systems are judged by trying a few examples and going with a feeling, which hides regressions. Even a small eval of 30 to 50 realistic cases catches a lot. Public benchmark scores are a kind of eval too, but they measure the vendor chosen tasks, not yours, so they say little about how a model will do on your work.
To test a new version of an AGENTS.md file, you pick 20 past tasks from your repository, run a coding agent on each with the old file and the new one, and record how many finish with passing tests and no rule violations. If the new file scores 17 of 20 and the old one 14 of 20 across a couple of runs, the change probably helped.
A high benchmark score does not mean a model is best for your task. Benchmarks can be saturated, leaked into training data, or simply unlike your work. A small eval built from your own cases is more informative.
All 28 terms are in the AI glossary. For models side by side, see frontier vs open-weight.
Start with 20 to 50 realistic cases that cover common tasks and known failure modes. Grow it every time you find a new bug. A small eval you actually run beats a large one you never get around to.
Yes, this is often called model-graded evaluation or LLM as a judge. It works best with a clear written rubric and spot checks by a person, because the grading model has its own biases and errors.
A benchmark is a published, shared eval used to compare models. An eval is any repeatable test, and the most useful ones are private, built from your own data and tasks.