EVAL

A repeatable test set with a scoring method, used to measure whether an AI system does its job and catch regressions.

Zero retention: everything you type stays in this browser. Verify it yourself in DevTools.
Plain-language definition

What Does Eval Mean?

An eval, short for evaluation, is a repeatable test for an AI system. It is a set of example inputs plus a way to score the outputs. Scoring can be exact checks, like whether code passes its tests or a field matches the expected value, human ratings, or another model acting as a grader against a written rubric. You run the eval before and after a change, such as a new prompt, model or instructions file, and compare the scores. Because model outputs vary, evals usually run many examples, sometimes several times each, to see a real difference rather than luck.

Why Does Eval Matter?

Without evals, changes to AI systems are judged by trying a few examples and going with a feeling, which hides regressions. Even a small eval of 30 to 50 realistic cases catches a lot. Public benchmark scores are a kind of eval too, but they measure the vendor chosen tasks, not yours, so they say little about how a model will do on your work.

What Does Eval Look Like in Practice?

To test a new version of an AGENTS.md file, you pick 20 past tasks from your repository, run a coding agent on each with the old file and the new one, and record how many finish with passing tests and no rule violations. If the new file scores 17 of 20 and the old one 14 of 20 across a couple of runs, the change probably helped.

What Do People Get Wrong About Eval?

A high benchmark score does not mean a model is best for your task. Benchmarks can be saturated, leaked into training data, or simply unlike your work. A small eval built from your own cases is more informative.

Which Terms Are Related?

  • Guardrail: A rule or check, in the instructions or in code, that keeps an AI system from doing what it should not.
  • Hallucination: When a model confidently produces something false or made up, such as an API that does not exist.
  • Frontier model: One of the most capable AI models at a given moment, typically closed and served by a large lab through an API.
  • Recursive self-improvement: The idea of AI systems helping to build better AI, which then builds better AI again. Partly real, mostly still speculative.

All 28 terms are in the AI glossary. For models side by side, see frontier vs open-weight.

Frequently Asked Questions

How many examples does an eval need?

Start with 20 to 50 realistic cases that cover common tasks and known failure modes. Grow it every time you find a new bug. A small eval you actually run beats a large one you never get around to.

Can an AI model grade another model?

Yes, this is often called model-graded evaluation or LLM as a judge. It works best with a clear written rubric and spot checks by a person, because the grading model has its own biases and errors.

What is the difference between an eval and a benchmark?

A benchmark is a published, shared eval used to compare models. An eval is any repeatable test, and the most useful ones are private, built from your own data and tasks.

Was this helpful?