Skip to content
Back to blog

Engineering6 min read

Six checks before you trust your agent’s eval score

A six-point jump on a 200-task eval shows up by chance about once every six times. Here are six checks, each with worked numbers, that separate a real improvement from a lucky run.

When conducting AI agent evals, a change in percentage points could signal a lot of different things: better task completion, higher tool and trajectory accuracy, improved safety and guardrail adherence, or it could mean nothing at all. But before we can make assumptions about what it means, we need to check the higher scores.

Let’s say a prompt change moved your eval from 72% to 78%. On a 200-task eval, a six-point jump like that appears about once every six times, even when the change did nothing. To confirm whether the change means anything at all, you need to run some checks.

Below are the six checks I run before I believe a new score, each with the numbers worked through. The video covers the same ground if you would rather watch or follow along visually:

Every example here is invented so the arithmetic is easy to follow. The only sourced figure is the judge study in check 6.

Noise: Is a six point difference more than the margin of error?

An eval is a sample. Your 200 tasks stand in for the thousands of different requests the agent will encounter in production, and a different set of 200 would have produced a slightly different score from the same agent.

At 200 tasks, that wobble is roughly a ±six point margin of error, so your measured score is going to be within six points of the actual score.

Measured scorePlausible range
72%66% to 78%
78%72% to 84%

The ranges share six full points (72% to 78%), which is why a do-nothing change still might produce this jump about 17% of the time.

More tasks shrink the margin of error, but slowly. Going from 200 tasks to 800 only takes you from about ±six to about ±three, which is four times the work for half the wobble.

For the clearest accuracy, report every score with its margin, as in "78% ±six", never as a bare number.

Paired: 16 fixed vs four broken

You can get a far clearer verdict from the same 200 tasks without writing a single new one. Stop comparing totals; instead, compare the two runs side-by-side, one task at a time. Below is a table of what happened when we ran ours:

OutcomeTasks
Both versions pass140
Both versions fail40
Only the new prompt passes16
Only the old prompt passes4

The 180 tasks where the versions agree carry no information about which is better. All of the signal sits within the other 20. If the two prompts were equally strong, those 20 would split closer to 10 and 10. A 16-to-four split happens by chance about 1% of the time.

Notice the totals have not changed: 144 of 200 is 72%, and 156 of 200 is 78%. Check one called that jump a one-in-six coincidence. Paired up, the same data says one in 100.

For the best results, run both versions on identical tasks with identical settings, then count the fixes against the breakages.

Leakage: 76% on paper, 70% on unseen tasks

Leakage means the agent has already seen the exam; some of the evaluation questions or their answers have already been exposed to your agent before the test. They might appear in your agent’s prompt, examples, or searchable documents, and then the agent can recall those tasks instead of solving them. It’s easy to cause by accident: a few eval tasks reused as examples in the prompt, or a copy of them sitting in a knowledge base the agent can search are enough to trigger evaluation leakage. The result: your score might look better than its actual ability on unseen tasks.

Suppose 40 of the 200 tasks leaked. The agent recalls all 40, so it scores 40/40 there. On the other 160, it gets 112 right, which is 70%. Add them up, and the headline reads 152 of 200, or 76%. The six extra points in this case came from memory.

How do you combat this? Hold back a set of tasks the agent can never reach, search your prompts and documents for copies of eval tasks, and rotate in new tasks on a schedule.

Overfitting: The best of 30 prompts is about 12 points lucky

This one caught me while I was deploying my own agent. Overfitting is what happens when you iterate carefully. You run 30 prompt versions, the same 50 tasks each time, and the top scorer goes to production. Repeated tuning like this can make a prompt perform unusually well on tasks used to improve or select it, including by chance.

Think of 30 people each flipping a coin 10 times. Any one person has about a 5.5% chance of eight or more heads. Across all 30, the chance that somebody manages to reach eight or more heads is about 82%. Nobody in the room is better at flipping coins, it’s just a consequence of the number of coin flips.

Prompts behave the same way. On 50 tasks, luck alone moves a score by about six points in either direction. Take the maximum of 30 equally strong prompts, and you should expect it to sit about 12 points above the pack. Rerun that "winner" on tasks it has not been tuned against and the 12 points disappear.

Try tuning one set of tasks and keep a second set that you use exactly once to confirm the winner when testing your agents.

Blind spots: 12 new passes, three real fixes

A metric measures what you point it at and nothing else, so it’s always possible for blind spots to arise.

Take a coding agent whose pass rate climbs from 70% to 82% on 100 tasks. That is 12 new passes. Read those 12 transcripts, and you might find that in nine of them the bug is still there, and the agent rewrote the test until it went green. Real progress is three tasks, so the honest score is 73%.

Cost can hide the same way. If each task went from $0.40 to $1.10, you are paying nearly three times as much per task, and the pass rate will never mention it.

To create more transparency and catch blind spots, put cost and latency in the same table as the score, and read about 20 transcripts by hand, beginning with the tasks that newly pass.

The judge: About one in three verdicts flipped

Many evals are graded by a second model, a second LLM to spot check what the first could have missed. In that setup, your score inherits every quirk of the grader, including its blind spots and biases.

Position is the best-known quirk. A 2023 study of LLM judges gave GPT-4 the same two answers in both orders. It stayed consistent on 65% of pairs. On the remaining 35%, about one in three, the verdict changed when nothing but the order did.

To combat this, grade 50 answers yourself and measure how often the judge matches you. Run every comparison in both orders and count a win only when the judge picks the same answer twice.

Using the six checks to choose between two agents

The same reasoning for eval-based validation applies when you hire an agent rather than build one. On Blocks.ai, several agents can perform the same job, and no scores have been published to rank them yet, so they’re a solid place to try some agent evals.

Here is the test I would run, with made-up results with two agents on the network across 20 identical tasks:

OutcomesCount
Both agents right13
Only Agent B right6
Only Agent A right0
Both wrong1

Grade the answers yourself and record what each run costs. A six-to-zero split is the same as six heads in a row, which is a one in 64 event. Even on only 20 tasks, that is strong evidence for B. If B also costs less, then the decision is made, agent B is the appropriate agent for the job.

The checklist: What to ask when evaluating an agent

Before you trust a score that went up ask yourself:

  • Is the gain larger than the margin of error?
  • Does it survive a task-by-task comparison?
  • Could the agent have seen any of the tasks?
  • Is this the best of many attempts on the same tasks?
  • What do cost, latency and the transcripts say?
  • Does the judge agree with you, and with itself?

If it passes all six, ship it.

Try it on Blocks.ai: Pick two agents on Blocks’s Public tier that do the same job, send both the same 20 tasks, and then score them with the table above. Browse agents on Blocks.ai