Preregistration — PR review quality benchmark

This file was written before the run. It was committed on 2026-09-06 with verdict: PENDING, ahead of the 180 model calls it describes. The token arm’s preregistration was retrospective and said so — this one is the correction of that.

Everything below Question is as registered and has not been edited. Only the verdict has been rewritten, once: the first run was invalidated by a defect in the harness’s trace arm, the bars did not move, and both the struck numbers and the re-run’s are below.

Verdict: MET, on both bars — after the first run was found invalid

Two runs sit under this preregistration, and the first one does not count.

The 2026-09-06 run published MISSED on both bars: the trace arm understood the change 50% of the time against the naive arm’s 65% (a 15 point loss where the bar allowed 10) and produced 1.20 false positives per PR against 0.65 (+0.55 where the bar allowed +0.50). On 2026-09-07, TRA-1090 diagnosed the 13 pull requests behind that gap and found they had nothing in common except this: the trace arm’s context contained no source code at all. get_context_bundle read symbol bodies through a bare require('node:fs'), which throws under ESM and was swallowed by a catch, so the harness — which imports src/ as real ESM — assembled signatures only. The shipped build was never affected. The diagnosis has the account.

That run therefore measured a context the product does not serve. It is struck, not defended.

The re-run, same 60 pull requests, same bars, unchanged:

The trace arm comes out 67% against 65% — non-inferiority met with the sign in our favour, which the bar did not require and which a 60-PR sample cannot make significant. Read it as parity, not as a win. False positives are +0.22, inside the 0.5 allowed and close to the +0.2–0.5 the prediction below registered. Per-PR agreement: both arms understood 36, only the naive arm 3 (13 in the struck run), only trace-mcp 4, neither 17.

The baseline arm scored 65% in both runs — the same 65% it scored against the broken trace arm. It was never touched by the defect, and that is the control this correction rests on.

Latency is the column that changed hands. The struck run had trace at 74.5 s against 90.0 s, which was the speed of a context with the code removed. With bodies restored the two arms are level: 92.9 s against 93.0 s. The saving is in input tokens, not in wall clock.

Question

The token benchmark showed that trace-mcp’s review context costs about a tenth of the input tokens of loading the diff plus every file it touches. It did not show that a review written from that thinner context is worth reading. This run asks exactly that: on the same 60 pull requests, does a model reviewing the trace-mcp context understand the change as well as one reviewing the full files, and does it invent fewer or more problems?

Metrics

Per pull request, both arms, judged by a model given the PR’s own diff as ground truth:

• understood — did the review name the defect the diff fixes, or correctly describe what the change does and why? Binary. Reported as understood_rate across PRs. This is deliberately not “found a latent bug”: the corpus is merged bug-fix PRs, so the defect is visible in the diff and the honest question is comprehension, not discovery. See Limits.
• false_positives — claims the review makes about the code that are wrong about the code shown: a bug that is not there, a call site that does not exist, an already-handled edge case. Stylistic nitpicks and speculative “consider…” suggestions do not count. Reported as a per-PR mean. This is the number that decides whether a review bot is usable at all; a cheap context that produces confident nonsense is not a win.
• api_ms — model latency per review, median. Secondary, but it gets asked.

Emitted by scripts/bench-pr-quality.ts into benchmarks/pr-context/quality.json and, preformatted for the site, into docs/_data/pr_context_quality.json.

Corpus

The same 60 pull requests as the token arm, and — this is the point — the same prompts. scripts/bench-pr-context.ts --dump-prompts writes each arm’s assembled context to disk during the token run, and this script reads those files. The texts scored here are byte-for-byte the texts that were token-counted there; neither arm is re-assembled or re-worded for the quality run.

Method

Every call is made through the same function at the same settings — one named model (claude-sonnet-4-5), default temperature, no tools, no MCP servers, no project or user settings, no dynamic system-prompt sections. Only the system prompt and the user text differ between reviewer and judge.

The judge sees the PR title, the diff, and the two reviews blind and in randomised order — it is never told which arm is which, and a fixed A/B order would let position bias ride along with the result.

Pass bar

• Primary (non-inferiority): trace.understood_rate ≥ baseline.understood_rate − 10 percentage points. The claim this benchmark supports is “a tenth of the tokens for the same review”, so the quality arm has to show no meaningful loss, not a gain.
• Secondary: trace.false_positives_per_pr ≤ baseline.false_positives_per_pr + 0.5. A thinner context that makes the model guess more is a real cost even if comprehension holds.

Unadjustable after seeing data. If the trace arm lands 15 points down, it publishes as MISSED at 15 points, and the next issue is fixing pack_context or the bundle budget — not moving the bar to 20.

Prediction

Registered before the run: the trace arm loses a small amount of comprehension — 0 to 8 points — because the baseline arm can read surrounding code the bundle omits, and it produces slightly more false positives for the same reason, on the order of +0.2 to +0.5 per PR. We expect the primary bar to be met and the secondary to be the close one.

Control

The baseline arm is a real control: same PRs, same model, same settings, same judge, context assembled by loading the diff plus every file it touches. A miss is therefore a result about trace-mcp’s context, not about a guessed baseline.

Release gate (TRA-1100)

The 180 model calls behind this measurement go through the claude CLI on a subscription, not an API key (see Limits), take roughly 90 minutes on 8 parallel workers, and need a checkout with an index built — none of which fits a GitHub Actions job on every PR. The bar above is still meant to be enforced, just not there: run it by hand, or from the Releaser autopilot, before cutting a release that touched retrieval, ranking, or context assembly (src/scoring/, src/tools/navigation/context-bundle.ts, src/tools/analysis/impact.ts):

tsx scripts/bench-pr-context.ts --dump-prompts benchmarks/pr-context/prompts
tsx scripts/bench-pr-quality.ts
node scripts/check-pr-quality-thresholds.mjs

The third command reads benchmarks/pr-context/quality.json — the second command’s raw output, not the pre-rounded docs/_data/pr_context_quality.json this page renders from — and exits non-zero if either bar above is missed, or if the run itself isn’t the one the bar was set against: fewer than the registered 60 PRs, any failed model call, or the wrong reviewer/judge model. Code review on the first version of this checker (2026-09-07) found it read only four rounded display strings and would report MET on a 1-row smoke run or a 60-row attempt where 59 calls failed, as long as the rows that did complete looked fine; both are covered by tests/scripts/check-pr-quality-thresholds.test.ts now. The comparison itself is kept in one place (scripts/check-pr-quality-thresholds.mjs) so a release call does not depend on someone re-reading percentages correctly. A MISSED run blocks the release; it does not get a threshold adjustment to pass (see Pass bar, “unadjustable after seeing data”).

What runs on every PR instead, cheaply, is the structural half of this gate: tests/ci/context-bundle-body-coverage.test.ts asserts that a generously-budgeted get_context_bundle result actually carries a body for every symbol it lists, using the same detail field this fix added to the tool’s JSON output. It cannot tell you the reviewer understood the change — only the full harness above can — but it catches the exact defect that made this preregistration fail once already (TRA-1090: a bundle that lists symbols without their bodies) on every commit, in milliseconds, without a model call.

Limits

Stated in advance so they are not read as excuses afterwards:

• The ground truth is weak. These are merged bug-fix PRs; the defect is in the diff. A reviewer that reads the diff carefully can score understood without any surrounding context at all, which compresses the gap between the arms in the trace arm’s favour. A harder corpus — PRs followed by a revert or an explicit regression fix, where the reviewer should have caught something and did not — would separate the arms better and is a separate issue. This run keeps the pinned set so the quality numbers sit on the same rows as the token numbers.
• The judge is a model. false_positives in particular is a judgement call. Per-PR reviews and judgements are committed alongside the aggregates so the scoring is auditable rather than asserted.
• The transport is the claude CLI in headless mode, because this runtime has no API key. That adds a constant ~16k tokens of tool definitions to every call’s system prompt, identical in both arms and in the judge, and no tool is callable. It does not affect the comparison; it does mean the token and cost figures on this page are the CLI’s and not the benchmark’s. Read results.json for those.