Evals belong in CI, not in a notebook

If an eval only runs when someone remembers to open a notebook, it isn’t a test. We moved ours into the pull request, and here is the setup.

Why notebooks fail

Notebooks are great for exploring. They are bad at being the last line of defence, because they depend on someone remembering, having the right environment, and reading the output carefully at the end of a long day.

The CI setup

1. Store eval cases as plain files next to the agent code.

2. Run the suite on every pull request that touches prompts, tools or model config.

3. Report a pass rate per suite, not one global score.

4. Block the merge when a suite drops below its threshold.

Thresholds, not perfection

Agents are not deterministic, so a hundred percent pass rate is the wrong goal. Set the threshold a few points under your current baseline and raise it as the suite stabilises.

What we measure

• Task success: did the agent reach the right outcome?

• Path quality: how many steps, retries and tokens did it take?

• Policy: did it stay inside its tools and spend limits?

A regression in path quality is a bug, even when the final answer is right.

What changed

Prompt regressions caught before merge went from roughly one in five to almost all of them. Review got faster too, because reviewers read a pass-rate diff instead of guessing.