Tool response token cost

Measured 2026-09-05 on darwin 25.5.0 / arm64, trace-mcp 3.18.0 (94dbaf70) — the build stamp travels with the figure to every surface that quotes it, and the preregistration states the bar and the verdict (this run publishes as a miss, on the reduction half of the bar). Against trace-mcp’s own repo (2 159 files, 11 134 symbols) over a real stdio tools/call round-trip. TRA-880, extended to the tail by TRA-945. Reproduce with:

pnpm run build && npx tsx scripts/bench-response-tokens.ts [repoPath]

Token column is the median of three runs, a real o200k_base count of the response text, not an estimate. Call volume is this machine’s ~/.trace/savings.json (20359 calls since the store was created) — real usage, one machine, never an average user.

What was wrong

The advertised surface side of the token story has been measured and guarded for weeks (preset-surface-budget.test.ts). The response side never was.

src/savings.ts scored a call before the tool ran: recordCall(name) took a hand-written RAW_COST_ESTIMATES[name], multiplied it by a flat COMPRESSION_RATIO = 0.15, and booked the difference as saved. The gate (src/server/tool-gate-helpers.ts) was the only caller and never passed a real count. So tokens_saved was calls x constant — arithmetically confirmable in the store: 5 123 search_text calls, 13 063 650 saved, exactly 2 550 each.

That number is not internal. It is the counter on the homepage and in the README (docs/_data/savings.yml), and calls/tokens_saved ride the usage ping.

The measurement

24 tools, covering 97.2% of recorded call volume. Ratio is measured response ÷ the raw Read/Grep the tool is credited with replacing; above 1.00 means the tool costs more than what it stands in for.

Those 18319 calls cost 21904545 measured tokens against a 27729800-token baseline — 21% fewer, or 31.5% if you floor the losing tools at zero the way the corrected counter does. That is the figure the homepage and the README quote in place of the old “~40–50% on average” (TRA-904). It is generated into docs/_data/response_tokens.json by npx tsx scripts/gen-response-tokens-data.ts from this table’s two inputs, so no surface can retype it. The baseline half is still an estimate — see the last section.

Three things the table says:

1. 0.15 is wrong on every tool that matters. The four busiest (88% of all calls) measure 0.37–1.54. The assumption is off by 2.5x on the best of them.
2. 8 of the 22 cost more than the baseline they replace — and, before the counter was corrected, were still booking a positive number on every call. It was ten of twenty-two until TRA-952 reshaped the three worst (below); the ones left are led by get_complexity_report and get_dead_code at 2.27x. Each is a response-shaping defect: a default depth/limit too generous for what the caller asked.
3. A few are far better than claimed — get_context_bundle at 0.02 and get_env_vars at 0.03 were being under-credited by an order of magnitude.

The tail, and the tools with nothing to compare against

TRA-880 measured twelve tools (88.4% of calls) and published as a miss on coverage. Measuring the remaining twelve found something the head could not show: some tools have no baseline at all.

A savings figure is “what a Read/Grep would have cost, minus what we returned”. register_edit is a notification that a file changed; reindex rebuilds an index. There is no file read an agent could have run instead, so that subtraction has no left-hand side. DEFAULT_RAW_COST = 500 was supplying one anyway — and register_edit is the fourth busiest tool on this machine, 1 289 calls. Across the whole store, 1 731 calls to mutating tools had booked ~736 000 tokens of savings that never existed, 3.0% of everything the counter had ever claimed.

Fixed in src/savings.ts: NO_BASELINE_TOOLS credits zero. The response is still counted on the spend side, because the agent still paid for it:

That is 1473 calls and 455561 tokens of pure overhead — real cost with no counterfactual. Counting it on the spend side and nothing on the baseline side gives the all-in number: 19.4%. That is what a session costs; the 21% above is what a lookup costs. Neither is wrong; they answer different questions, and the lower one is the one to plan a budget against.

src/tools/register/__tests__/no-baseline-tools.test.ts fails CI if a tool that describes itself as mutating is left out of the set, so the next one cannot quietly start booking savings again.

What closing the tail did to the headline

The two left-hand columns are frozen literals: they record what those runs measured, so a later re-measurement cannot rewrite them. Only the TRA-952 column is live.

The tail was more expensive than the head, in both directions: it contained the worst per-call ratios in the product and the calls that should never have been scored. Fixing the coverage miss produced a reduction miss.

The fix

SavingsTracker.recordActualTokens(tool, tokens) corrects the pre-call guess once the response exists. recordCall stays where it is, before the tool runs, because budget clamping and dedup both read the session totals first — this is a two-phase estimate-then-reconcile, not a move.

Four things the correction has to get right, all found in review and guarded in tests/tools/savings.test.ts:

• A response bigger than its baseline credits zero, not a fat positive.
• A failed call credits zero (recordFailedCall). Scored as payload, a 4-token error from get_task_context would have booked 7 996 saved — more than any real answer to the same call. Applies to error responses and to throws alike.
• batch is corrected too. It dispatches handlers directly and never goes through the gate, so every batched call would otherwise have kept the guess.
• Measured last, on the wire bytes. enrichResponse adds fields and applyWireFormat can re-encode into a denser format; measuring before either books a number the client never receives. An empty response is a measured zero, not a missing one.

How the numbers are collected

Three runs per tool, median published, min and max printed. That is not ceremony: a single sample recorded get_task_context at 5 383 tokens and then at 8 357 minutes later on the same commit. The spread turned out not to be variance but a degraded surface — when a daemon is already running, the stdio session proxies to it and the session’s own --preset is ignored, so twelve of the twenty-four tools answer Tool X disabled and the bench was about to publish those error strings as measurements. The harness now aborts on any errored call rather than writing it to the artifact.

Within one healthy session the responses are near-deterministic: every tool above has a min–max spread of 0–3 tokens.

What is still an estimate, and what to do next

RAW_COST_ESTIMATES — “what a Read/Grep would have cost instead” — is still hand-written and unvalidated, so the savings baseline remains a guess even though the response side is now measured. That is the next measurement, not this one: it needs a real counterfactual (the same question answered with Read/Grep, tokens counted), which is what benchmarks/pr-context-benchmark does for PR context and nothing does for tool calls.

One caveat on get_outline: it read 1 427 tokens against 1 056 in TRA-880, on the same target file, because the change that added NO_BASELINE_TOOLS grew src/savings.ts by ~70 lines. The bench measures a live repository, so its own commits move its numbers. That is a property of the corpus, not noise, and it is why the corpus size is stated at the top.

What shaping the three worst tools did (TRA-952)

The first three tools on that follow-up list have been reshaped, and the table above is the after. What each one was returning that nobody asked for:

No response field was dropped and no schema changed: the subprojects list is still available on request, the call graph still reaches the same depth, and a deeper dead-code page is still one limit away.

And it moved the headline by 0.1 points. Those three tools are 82 of 18,319 recorded calls. The weighted figure is decided by search_text, get_outline and search, which are 78% of call volume between them — search at 1.54x and get_outline at 1.19x are now the whole of the negative block that matters. Worst-ratio-first was the right order for finding defects and the wrong one for moving the number; the next response-shaping issue should be search, on volume.

get_dead_code is left at 2.27x on purpose. Its baseline is 1,200 tokens — “what a Read/Grep would have cost instead” for a whole-repo dead-code sweep over 3,412 exports, which is not a credible 1,200 tokens. Cutting the tool further would buy the ratio by answering less; the honest correction there is on the baseline half, which is still an estimate.

Three rows moved for reasons that are not this change

get_changed_symbols (521 → 1,224), find_usages (1,122 → 975) and search_text (1,659 → 1,722) were not touched. get_changed_symbols reports the diff of whatever working tree it runs on, so the two runs asked it different questions and its row is not comparable between them at all. The other two track the corpus: the repo gained files and symbols between the two measurements. Same caveat as get_outline above, and the reason the per-tool ratios are the durable part of this page and the aggregate is not.

Last updated: September 5, 2026