Evidence
The studies below show useful reductions in model calls and point to more value to explore. Bring a repository, a recurring workflow, or a pattern the learner missed. Together we can test which reflexes help, learn from more traces, and improve the parts that still need work. Each result keeps its study scope and correctness checks alongside it.
The Claude Code mod prototype used 48% fewer model requests than the agent alone, with 12/12 runs passing hidden tests in both arms. A separate development study on 6 sandbox tasks, not the standard hook install.
The question we test
Can Adage handle predictable steps on your CPU so the agent needs fewer model calls, without hurting correctness?
Our working hypothesis is that specialized repositories and recurring workflows produce more useful reflexes. These studies are a starting point for testing that idea across more projects.
One task, start to finish
Ask an agent to add a new validator to Marshmallow, a Python library: the code, its tests and a changelog line. Each dot is one model call.
The agent alone29.0 calls per session
It looks around the repository to find where things go, runs the tests, reads the output, and often runs them again to be sure.
With Adage22.8 calls per session
A learned card says which files to touch and where each one stands right now, and commands go to the project's own environment the first time. The agent goes straight to the work.
Averages over 20 paired runs on 4 held-out tasks, advisory mode. Hidden tests passed 19/20 with Adage and 19/20 alone.
The results at a glance
Each test ran the same tasks with Adage and without it, and hidden tests graded every result. The two used different repositories and models, so read each against its own agent alone.
Registered test · real open-source repository
Adage tells the agent what it would otherwise go looking for. Nothing runs on its behalf.
22% fewer model calls
Execution mode, opt-in: experimental · quality bar not yet met.
Marshmallow · Claude Haiku 4.5 · 4 held-out tasks × 5
The full result ↓NewDevelopment test · sandbox repository
Adage also takes the steps a person approved in the model's place, so those steps need no model call at all.
48% fewer model calls
A small billing service · Claude Sonnet 5.5 · 6 sandbox tasks × 2 runs · development run 4
Why the mod helps ↓Earlier studies, including the ones that missed, are in the study history.
What we tested
We gave Claude Haiku 4.5 4 new coding tasks on Marshmallow, a real open-source library, each run 5 times three ways: the agent alone, the agent with Adage in advisory mode (the default), and with execution mode switched on (opt-in). Hidden tests graded every result. 65 sessions in all, safety controls included, and every one counted.
Every task runs as a paired comparison, with Adage and without it: same repository, same prompt, a fresh checkout each time. Comparing pairs keeps a hard task from looking like a slow tool.
The agent never sees the tests that grade its work. Fewer calls only count if the code is still right.
First come runs where checking again is the right move: a failing test, a later edit, a changed config, a broader change, an explicit request to rerun. Adage must never let the agent skip necessary work. One miss stops the study.
In execution mode, Adage runs only the exact command a person approved, inside an operating-system sandbox, on the CPU. If the command or its environment changes, the approval stops matching and nothing runs. Advisory mode runs nothing at all.
The line a result must clear is written down and committed before the first session, and never moved after.
Failures and stopped studies stay in the record with their costs. Nothing is rerun to get a better number.
Model requests are counted from Claude Code's own debug log: one line per request it sends, retries included, since they are billed too. Each session is cross-checked against the request ids in its output stream; a gap that logged API errors do not explain makes the session unverified, and its whole paired block leaves the comparison, for every arm alike.
The comparisons are ratios of total requests, with Adage over alone, across the paired blocks. Their one-sided 95% upper bound comes from a bootstrap that resamples blocks within each task, 20,000 draws, with a fixed seed; the line to clear is 0.90. The study's pass/fail test was registered for execution mode: its upper bound at most 0.90, and its hidden-test passes at least the agent alone's minus one, plus a set of safety criteria. The advisory comparison was registered to be reported beside it, with the same statistics.
Censoring works against Adage. A session that stops early (its turn limit, its budget, the clock) is a quality failure in every arm, and only execution mode's is raised to 60 requests.
The two modes. Advisory mode is what Adage does by default: instincts from the repository's own files, a procedure card that says where the files stand right now, and learned advice at the right moment. Nothing runs on the agent's behalf. Execution mode adds person-approved test runs on your CPU and a finish-line check. Cost is the API list price.
The held-out result
4 held-out tasks, 20 paired blocks, published as the study's report wrote them.
Held-outPre-registered comparison · 4 new tasks · advisory mode, the default
On 4 tasks Adage had never seen, run 5 times each, the agent with Adage in advisory mode needed 22.8 model calls per session against 29.0 alone, fewer in 18/20 paired blocks. Even the cautious estimate, 0.83×, clears the 0.90× line. Hidden tests passed 19/20, against 19/20 alone. Nothing ran on the agent's behalf, and no execution approval was involved.
The dot is advisory mode's model calls for each one the agent alone needed; the bar is where the true value likely sits (95%). It clears only if the whole bar is left of the line.
Execution modeExperimental · quality bar not yet met
Opt-in: Adage also runs the person-approved tests itself, on your CPU, and checks the finish line before the agent stops.
More savings in testing, but not yet proven to keep quality: 0.67× the agent alone's model calls (cautious estimate 0.73×), with hidden tests 17/20 against 19/20 for the agent alone. That misses the registered quality rule, so the study's verdict for execution mode is that it did not pass. One of its failures came in a block where all three arms failed. It stays opt-in.
The study ran three arms on the same 4 tasks, 5 times each, in 20 paired blocks: the agent alone, advisory mode and execution mode. Its pass/fail test was registered for execution mode against the agent alone (the gated row below); the advisory comparison was registered to be reported beside it, with the same statistics. The study's registered outcome for execution mode: did not pass (criteria not met: G5).
| Comparison | Ratio | 95% range | Blocks fewer | Sign-flip p | Under 0.90× |
|---|---|---|---|---|---|
| Advisory / alone (registered, reported) | 0.78× | 0.74× to 0.83× | 18/20 | 0.0023 | yes |
| Execution / alone (gated) | 0.67× | 0.63× to 0.73× | 18/20 | 0.0001 | yes |
| Execution / alone, without censored blocks | 0.67× | 0.63× to 0.73× | 18/20 | 0.0001 | yes |
| Execution / alone, without the has-keys task | 0.64× | 0.59× to 0.70× | 14/15 | 0.0003 | yes |
| Execution / advisory (reported) | 0.86× | 0.80× to 0.92× | 14/20 | 0.1858 | no |
| Arm | Mean requests | Requests counted | Runs passing hidden tests | Censored | Test-run requests | Cost / session | Tokens / session |
|---|---|---|---|---|---|---|---|
| The agent alone | 29.00 | 580 | 19/20 | 0 | 177 | $0.201 | 946k |
| Advisory mode (default) | 22.75 | 455 | 19/20 | 0 | 138 | $0.135 | 569k |
| Execution mode (opt-in) | 19.55 | 391 | 17/20 | 0 | 95 | $0.135 | 533k |
Blocks whose hidden tests failed: the agent alone 5 (has-keys); advisory mode 5 (has-keys); execution mode 5 (has-keys), 8 (lat-long), 9 (has-keys). Failed in all three arms: 5 (has-keys).
| Criterion | Applies | Held | Rule, or why it does not apply |
|---|---|---|---|
| G1 | applies | yes | single repository: the frozen learned home holds the certified verification route run-175d85 (checked at freeze); the two-repository clause belongs to the cross-repository decision |
| G2 | applies | yes | BEST positive sessions: a learned verification ran in >= 50%, and >= 75% of those had a successful receipt |
| G3 | not applicable | – | confirmed eliminations were the r1-r7 primary; r8/r9 replaced it with model requests per session, which this stratum gates, and its two-repository clauses cannot apply to one repository; eliminations are reported |
| G4 | not applicable | – | a repeat-elimination mechanism criterion superseded by the r9 primary; repeats were gated in development (D4); NOEXEC equivalent-command and BEST repeat fractions are reported |
| G5 | applies | no | BEST hidden-test failures <= min(A0 failures, NOEXEC failures) over all blocks; stricter than the r9 margin of one, so it governs quality |
| G6 | applies | yes | zero skipped necessary verification (controls and positives), zero executions without a grant or in NOEXEC/A0, zero foreign or unapproved receipts, zero undeclared writes, zero unknown costs |
| G7 | applies | yes | at most 25% of BEST CPU executions in positive sessions unnecessary (receipt stale before the session ended); controls are stale by construction and excluded |
| G8 | applies | yes | BEST total requests <= NOEXEC total and BEST cost <= 105% of NOEXEC, over included blocks with the censoring rule |
| G9 | not applicable | – | the pipeline-unseen repository (more-itertools) is not run in this stratum; G9 stays required for any claim that covers it |
Blocks left out for an unverified request count: none (at most 2 allowed). A session that stopped early counts as a quality failure in every arm, and only execution mode's is raised to 60 requests. Assignments 65, collected 65. Cost is the API list price.
Pass rule as registered: The one-sided 95% upper bound of Adage's model requests divided by the agent alone's, over the paired blocks, is at most 0.90, and Adage's hidden-test passes are at least the agent alone's minus one. Every applicable G criterion must also hold.
The protocol · the analysis code · the frozen tasks · the decision record · the report (sha256 a3ec997154f8…)
New · the Claude Code mod
This study used a separate Claude Code mod prototype, not the standard Adage install. The prototype ran inside the agent loop and could take a previously approved step directly through Claude Code's permission check, without a model request to choose it. The next model request still read the result. This explores a deeper harness integration; its results should not be read as a promise about the released hook integration.
Model calls per session, on average
6 sandbox tasks in a small billing service, each run 2 times in every arm with Claude Sonnet 5.5. Development run 4, after fixes on these tasks. Each dot is one model call.
The agent alone7.2 calls
It looks the work item up, reads the files it names, edits, then hunts for the test command that works here.
Adage through hooks6.2 calls
Today's plugin. The agent is told where to look and which command works, but still makes every step itself.
Adage as a mod3.8 calls
The rings are steps Adage took itself, about 2 a session, with no model call.
12/12 runs passed hidden tests in every arm. The best case, task W-2041: 2 calls a session with the mod, against 6 through hooks and 9 or 8 alone.
Before the model's first call, Adage ran the lookup the agents had started with in nearly every earlier session, scripts/wi-lookup W-2041.
Then it read the files the lookup listed, so the model's first call already had them.
When the agent changed a schema, Adage ran make gen at the next step instead of waiting for the model to remember it.
When the agent wrote python -m unittest, which fails here, Adage ran the repository's make test in its place and said so.
| Arm | Model calls per session | Cost per session | Runs passing hidden tests |
|---|---|---|---|
| The agent alone | 7.17 | $0.0677 | 12/12 |
| Adage through hooks | 6.17 | $0.0650 | 12/12 |
| Adage as a mod | 3.75 | $0.0531 | 12/12 |
Against hooks alone, the mod's arm needed −2.42 calls a session (95% interval −3.42 to −1.42, Holm-adjusted p 0.003), fewer on all 6 tasks, and cost 18% less. Against the agent alone: −3.42 calls (−4.83 to −2.17). Five arms ran interleaved, 60 sessions in all; the other two checked the setup, among them the mod installed but only watching, which matched hooks alone.
What it doesn't show yet. One small sandbox repository built for the evaluation, one model, 6 tasks run twice. It is the fourth run on these tasks, after fixes made from the first three: what it had to show was written down before any of its numbers, but it is development evidence, not a registered test. Dead-end hits fell only by point estimate. A registered test on fresh tasks in a real repository comes next.
From docs/claude-mod/evaluation-results.md, run 4, through /evidence/claude-mod.json.
Limits and what's next
Go deeper
Use the studies as a starting point for your own experiments: the rules, records and scoring scripts are open. Successful comparisons and failed ones both help us choose what to improve next. Contribute a pattern, a fix, or a new evaluation →
The report and decision record behind its numbers, unchanged.
heldout-report.json · decision recordAll four live runs, what each had to show, and the defects they found.
evaluation-results.md · claude-mod.jsonEvery earlier study in order, with its result as recorded, and the numbers the site once showed.
docs/benchmarks.md · LAUNCH-STATUS.md · benchmarks.json · cold-start.jsonEvery pass rule and dated revision, as written before the runs it governs, and the code that scored them.
docs/launch-gate-protocol.md · heldout_cohort.py · heldout_analysis.pyFrom evals/reflex-proof/harness, with $H the frozen held-out root. live is the only paid step; it dispatches in the frozen order, controls first, and stops on any lapse. The harness README describes the runner and its setup commands; the registered protocol records the study design and pass rules.
python3 heldout_cohort.py prepare $H # offline: fresh checkouts, read-only runtime
python3 heldout_cohort.py verify-grants $H # a person issued exactly the reviewed approvals
python3 heldout_cohort.py live $H # paid: 65 sessions, one at a time
python3 heldout_cohort.py report $H # the registered analysis and its decision recordThis page draws its numbers from /evidence/heldout.json, made from that report by web/site/scripts/heldout-evidence.mjs.