TOON Token Savings — Measured

This document captures real-world token measurements for the TOON output format wired across trace-mcp tools, plus the independent search_text grouping: "by_file" reshape.

TL;DR — which tools support TOON

After benchmarking, TOON is wired only on the five tools where it is a clear net win on representative payloads:

For every other tool TOON is off by default and the parameter is not accepted in the schema.

Why we removed TOON from 4 tools

@toon-format/toon remains a project dependency because the five keepers above still rely on it.

Internal mechanism — table mode vs list mode

TOON has two output shapes (driven entirely by the encoder, not by the caller):

• Table mode ([N]{col1, col2, col3}:) — emitted only when every row in an array contains the same scalar-only fields (string, number, bool, null). One header amortizes over all rows; each row collapses to a single CSV-like line. This is where TOON wins.
• List mode (YAML-style nested keys) — emitted when any row has a nested object, an inner array, or differs in field set. Each row pays its own field labels; the header amortization disappears. TOON loses here vs JSON because JSON’s {"k": is already terse.

The scripts/toon-diagnostic-2.ts script reproduces the breakeven curve. Sample output (n = rows per array, Δ% = TOON savings over JSON):

N  | scalar-only (table)   | with array tags         | with nested obj
   | json   toon  Δ%   mode| json   toon  Δ%    mode | json   toon  Δ%    mode
---|----------------------|------------------------|------------------------
 10|  185    130  +29.7  T |  255   333  -30.6   L  |  354   443  -25.1   L
 20|  365    250  +31.5  T |  505   663  -31.3   L  |  704   883  -25.4   L
 50|  905    610  +32.6  T | 1255  1653  -31.7   L  | 1754  2203  -25.6   L
100| 1805   1210  +33.0  T | 2505  3303  -31.9   L  | 3504  4403  -25.7   L

Key observations:

• A single inner tags: [...] array per row collapses the win into a ~30% regression — the encoder falls out of table mode.
• A single nested symbol: {...} object per row is similarly fatal (~25% regression).
• Table-mode wins grow with column count: for 20 scalar columns × 20 rows the encoder reaches +47.4% vs JSON. This matches what query_decisions hits in production.

The four loser tools all land in list mode by construction: find_usages has nested symbol{}, search_text has context[], get_artifacts row shapes vary per kind, get_context_bundle payloads are deeply nested and small.

Methodology

• Script: scripts/bench-toon.ts for the per-tool numbers; scripts/toon-diagnostic-2.ts for the table-vs-list-mode curve.
• Invocation pattern: each registered MCP tool’s closure is captured via a fake server.tool(...). This bypasses the MCP transport but exercises the identical handler code that ships in production.
• Corpus for live-DB scenarios: a snapshot copy of the trace-mcp self-index (1,501 indexed files, 9,467 symbols).
• Corpus for fixture-only scenarios (query_decisions, get_changed_symbols): in-memory stores seeded with realistic shapes.
• Tokenizer: gpt-tokenizer with the cl100k_base encoding. This is a GPT-4 / Claude family proxy. Anthropic’s tokenizer is not publicly released; cl100k_base is the closest open approximation.
• Roundtrip assertion: every TOON output is decoded via @toon-format/toon’s decode() and structurally compared to the JSON payload, with a ~1e-6 relative tolerance for float precision.

Run it:

pnpm exec tsx scripts/bench-toon.ts
pnpm exec tsx scripts/toon-diagnostic-2.ts

Overall — JSON vs TOON (original 9-tool sweep)

search_text — flat vs by_file (TOON removed, grouping kept)

All percentages are computed against the flat-json baseline (2160 tokens / 7850 bytes for the same 50-hit corpus). output_format: "toon" is no longer accepted on this tool; grouping: "by_file" stays and is the recommended optimisation.

grouping: "by_file" is a pure structural reshape — it deduplicates file paths under nested files[].hits[] buckets. It is lossless and independent of output_format.

Caveats

1. Tokenizer: cl100k_base is a proxy for Anthropic’s tokenizer. Real Claude-side savings can differ by ±2-3 percentage points.
2. Float precision: TOON rounds high-precision floats below ~8 significant digits (e.g. PageRank scores). For human-readable outputs this is invisible; for callers that hash JSON output, it is a real semantic difference.
3. Single-sample bench: each scenario runs once. The retrieval tools are deterministic given a fixed DB, so this is fine for token counts.
4. Corpus is one repo: trace-mcp’s self-index. Results will shift on other codebases — wider repos with longer file paths boost by_file more; repos with sparser fields and shorter strings boost TOON more.

Recommendation

• TOON is enabled (and worth selecting via output_format: "toon") on exactly five tools: query_decisions, get_outline, get_changed_symbols, search, get_feature_context.
• For everything else, JSON is the default and the only accepted output format. The encoder regression on heterogeneous / nested / small payloads outweighs any benefit.
• For search_text, prefer grouping: "by_file" when paths repeat across hits — that is a clean +20.8% lossless win and is unrelated to TOON.
• Document the float-precision caveat for callers that hash or diff responses.

Wave 2 candidates — measured

Forecasted savings if the same output_format: "toon" switch were wired into each of the candidates below. These tools are NOT wired — this is a measurement-only pass to decide which to wire in a follow-up.

Each row encodes the JSON payload returned by the production handler, then re-encodes the same payload via encodeResponse(..., "toon"). Tokenised with cl100k_base, roundtripped via @toon-format/toon’s decode() against the parsed JSON with the loose-float comparator. Sorted by savings_pct descending.

Skipped on this corpus (no bundles installed locally):

• list_bundles — empty payload
• search_bundles — empty payload

Wave 2 recommendation

Wire output_format: "toon" into these 10 tools — every one is in table mode with the loose-float roundtrip clean, and savings cross the +15% cutoff we established for the original five keepers:

Do not wire these — savings below the +15% cutoff or list-mode regression:

Wave 2 surprises

1. predict_bugs landed in table mode at +21.5%. Prediction said the per-row signals: string[] array would force list mode (-25%-ish, like find_usages). What actually happens: in this corpus most predictions have an empty signals array, and the TOON encoder keeps table mode when every row’s array field has the same length. The win is real but could degrade on a repo where signals is densely populated — re-measure before wiring on a high-churn repo.
2. list_pins (2 seeded rows) only saves 19.3%. The fixed TOON preamble is ~10-15 tokens; on a 2-row payload it eats most of the column-header amortization. The savings curve crosses +30% somewhere around 8-10 pins.
3. get_pagerank saves +36.0% on just two columns (file, score). The win is driven entirely by the repetitive file paths — TOON strips the per-row key labels, JSON repeats "file": for every row.
4. get_implementations regressed even though no row’s via was an array in this run (33 implementors of LanguagePlugin, all with single extends). The encoder still went list mode — likely because of the union-typed field shape across encoder probing, or because some rows have signature: null. Investigate before wiring.
5. analyze_perf is tied for first place at +43.2% — that is the strongest predicted candidate by a wide margin, despite being seeded with synthetic latency stats. A real persistent-telemetry payload (24h/7d windows) would benefit even more.

Last updated: August 30, 2026