Anthropic
Blogcast

Alex: If you have ever shipped an A I feature and watched it drift from brilliant to baffling, this one is for you. Anthropic's developers just published a piece on how they actually evaluate what they build, and it ships inside Claude Code as two new commands. Jordan, walk us through it.

Jordan: The article is called Automating eval design and hillclimbing with Claude, on the claude dot dev blog from Anthropic's Lance Martin. The core idea is simple to say and hard to do: an evaluation is the signal that tells you whether your app, your prompt, or your skill is getting better or worse, and designing that evaluation, then improving against it without fooling yourself, is one of the hardest parts of building with these models.

Alex: And the headline deliverable is the claude-api skill now shipping two commands. You run slash claude-api build-eval to build an evaluation inside your codebase, and slash claude-api hillclimb to improve your application against it, one change at a time, with a held-out set of examples to catch overfitting. That held-out set is the detail that matters, because without it you are just tuning until the score looks good.

Jordan: Exactly. The article opens with four principles of a well-designed evaluation. First, eval tasks should mirror production. Sample the tasks you actually care about, in the setting where the capability will be used. Second, performance should improve with stronger models and more thinking. If you swap in a better model and the score does not move, the eval is broken, usually ambiguous tasks or a bad grader. Third, there has to be passable headroom at the frontier. The best model at the highest effort should sit well below one hundred percent, or you cannot tell whether a change helped. But the gap must not come from impossible tasks: a task that fails on every single run is a warning sign, not a challenge. Fourth, low run-to-run variance. If the same model gives wildly different scores on repeat runs, you cannot attribute any score change to your change.

Alex: Then comes the sharpest idea in the piece: adversarial sampling.

Jordan: Model capability is jagged. If you pick test cases by finding things today's model fails at, you are sampling the valleys of one model's capability surface. The eval ends up measuring that model's failure fingerprint, not what is intrinsically hard or valuable. When the next model arrives with a smoother surface, your eval stops meaning what you thought it meant. The fix: pick hard cases because a person judged them hard. You should be able to say why a task is hard before you include it. Pull from production traffic, bug reports, and tickets, but do not blindly trust user traffic, because users try what they expect to work, so traffic can skew easy.

Alex: That is the theory. Now build-eval turns it into a guided workflow. In Claude Code, Claude interviews you, builds the eval inside your codebase, and pauses for approval at key points. The sampling order is deliberate: production transcripts first, then bug reports and tickets, then five to ten cases you write by hand, then cases synthesized from your codebase. It shows every input on a review page and waits until you confirm they are representative.

Jordan: Then comes the grader, where most homemade evals die. The skill proposes the cheapest grader that fits your output. Constrained output gets programmatic checks: exact match, a label from a fixed set, JSON matching a schema, tests that pass. Open-ended output gets L L M as judge: a second model reads the input, the output, and a rubric written as checkable claims, not a one-to-five scale, and returns a score with reasoning. And the judge model should not be the model you are testing.

Alex: Then it validates the grader before trusting it. It grades a handful of cases and asks whether you would have scored any differently. It runs the grader twice on the same output and reports whether the verdict changed. It checks plumbing so timeouts and cut-off answers do not masquerade as model variance. And it checks headroom: if the baseline already scores around ninety-five percent or higher, it warns you and suggests the hillclimb target cost or latency instead of quality. When it is done, you get the cases, the grader, the runner, one JSON line and one full transcript per case, and a results page with the baseline and a confidence interval.

Jordan: Now hillclimbing. Three criteria for where to apply it. First, cheap iteration. It should be inexpensive to modify the surface you tune. Prompts and skills are the sweet spot, cheap to change and revert. Second, attributable. Score changes should be attributable to the surface you modify, like skill triggering, where the trigger rate is directly coupled to the skill description being edited. Third, a well-scoped objective. Do not ask for vague gains on a saturated eval. And cost is almost always legitimate: even a saturated eval leaves room to reduce cost while performance holds.

Alex: Then the big caution: overfitting. The danger of hillclimbing is getting better at the eval while getting worse at reality.

Jordan: The mechanism is precise. An eval rarely matches production, and the eval can leak into your harness, the code around the model, including the prompts, tools, and loop. The article's example: a task that benefits from optical character recognition when production tasks almost never do. The harness adds an O C R tool, the score goes up, production sees nothing. Three defenses. Split the cases into a train set the hillclimber may read and a test set that is never seen. If train improves while test stays flat, that is the classic overfitting warning. Never paste failures into the prompt. And keep the answers structurally out of the model's reach.

Alex: The hillclimb command automates this. You choose what it may change: the system prompt, skills, tool descriptions, model and effort level, or harness code. You state the goal: performance, or cost while performance holds. It splits the set into test and train, and first checks that the eval's noise is smaller than the smallest improvement you would act on. Each round, Claude reads the previous round's train transcripts and proposes one change as a patch, fixing failing behavior at its root rather than rewording a line. Then it runs the evaluation. Train up and test flat, it suspects overfitting and reverts. Regression, revert. Both improve, keep the patch. When the score stalls for two or three rounds, Claude stops editing and sorts each remaining train failure by cause: ambiguous cases, harness errors, or run-to-run variance. When the run completes, your code stays at the version that did best on the test set, and Claude reports the test result against the baseline with confidence intervals. If the gain is within noise, it says so and recommends against merging.

Jordan: Two worked examples. First, cost. An internal customer support benchmark: forty-four tickets, thirty used for the search, fourteen held out. It started on Opus four point eight at high effort, seventy-four point four percent accuracy, at four point six cents per ticket. The hillclimb audited the prompt, removing tool-call rituals, a scratchpad step, and contradictory rules. Then Opus five point five on low effort hit eighty-seven point eight percent at one point nine cents per ticket, partly because its input and output tokens cost twenty percent less and cache reads cost sixty percent less. Then it stepped down a tier: Sonnet five on low effort scored eighty-eight point nine percent at about one cent per ticket, and better routing rules brought it to ninety-eight point nine percent at about the same cost. On the fourteen held-out tickets the search never saw, the final config scored ninety point five percent against the original seventy-eight point six, at about one fifth of the cost.

Alex: Better accuracy, one fifth the cost, held on unseen tickets. The second example was performance on the claude-api skill itself, evaluated on a set derived from Anthropic's docs. It started at sixty-six percent. The hillclimber found the skill was missing coverage of eight features; adding them took it to seventy-four. Fixed C sharp and Java type tables brought it to seventy-seven. Then the score stalled, and the reflection step earned its keep. The skill content was present, but the model kept writing older A P I shapes from its trained priors. The fix was a table near the top mapping remembered A P I forms to current ones, like fixed token budgets to adaptive thinking. That reached eighty percent. Then failure sorting found the eval itself was flawed: one grader demanded three errors on a task asking for one, and another grader contradicted the docs, which testing the real A P I proved correct. Fixing the eval bugs plus more skill edits brought it to about eighty-eight percent.

Jordan: The lesson there: sometimes the failures are not your code, they are your test. Getting started is simple. First run claude update, since the skill ships inside Claude Code. Then run slash claude-api build-eval to generate an evaluation set, and slash claude-api hillclimb to improve against it. You can steer build-eval with examples like traces, and Claude asks you to approve the examples and the grader first.

Alex: The bottom line: evals are not a chore you tack on after building. They are the instrument panel. Build them from production reality, grade them cheaply but honestly, keep a test set the optimizer never sees, and let one measured change per round tell you the truth.

Jordan: And the tooling is now in your terminal. Thanks for listening.

Alex: We will catch you on the next one.