A completion report on a large Oakridge refactor said the migration was finished, but code on the old contract remained. To investigate such mistakes, we gave agents a finished refactor of real Oakridge code with one behavioral check removed and the log of the session that supposedly wrote and signed it off. A hidden test they never saw failed until the check was restored. Across the runs, agents restored it when new code lined up with what it replaced, but largely missed it otherwise and often signed off as complete.
| What the agents verified | Sessions where the check was restored |
|---|---|
| Moves: code moved to new modules, old and new side by side in the log | 36 of 36 (three models; 6 native and 12 quoted Argon sessions) |
| A 693-line rewrite, near the median of the 212-PR cohort | 2 of 17 (same three models) |
| A rewrite that kept each old check as one matching rule | 11 of 11 (two models) |
| Oakridge #146, a real move that reshaped the code and shipped the omission | 0 of 11, with 10 “complete” verdicts; the allowed edit paths prevented a correct restore |
| A real 2,782-line rewrite, at the 89th percentile of all merged Oakridge PRs | 8 of 74 across 12 models: 8 of 51 on old-behavior omissions A and B; 0 of 23 on new-fix omission C |
The move-versus-rewrite gap remains: 36 of 36 restores on the move against 8 of 74 on the full rewrite and 2 of 17 on the 693-line rewrite. Whether each old check has a visible counterpart in the new code may help explain it; these sessions do not isolate that factor (see Finding 2).
Supporting data covers the method, the exclusion rule, the internal fixture names, how to reproduce a session and where every number comes from. When an Agent Says a Refactor Is Complete walks through how this question came up and what changed under evidence. Code examples puts the four before, clean refactor and omitted fixture states side by side.
The study began with a different question. Do agents re-check a teammate’s work but trust their own earlier “done”? Self-Attribution Bias reports that bias in AI monitors. Our own case was Oakridge, our production TypeScript codebase. PR #619 moved the operator app to the v2 contract and said everything underneath was v2. It wasn’t: PR #629 later removed v1 code that #619 had left behind. That self-trust question went unanswered (see below), and the result on this page came out of the runs instead. Treat it as exploratory.
Each session gives an agent a finished refactor of real Oakridge code, split into one to three parts. Each part comes with the log of the session that wrote it, ending “Done. Every check moved with the code. tsc is clean and all tests pass.” Each log was recorded by running its edits in the sandbox, so every tool result in it is real.
Models ran on pinned OpenRouter zero-data-retention endpoints, with a dollar cap per run.
The refactors are sized against all 634 merged Oakridge PRs, counting code lines only. The earlier 212-PR cohort has median 660.5 and 90th percentile 2,406; its percentiles are retained below for comparison. The largest PRs by raw lines are mostly docs and generated files. #605, for example, is 672 lines of code and 47,402 lines of everything else.
| Refactor | Lines changed | Larger than (% of 212-PR cohort) | Parts |
|---|---|---|---|
Move: derive.ts’s unit minting split into three modules |
420 | 35.8% | 3 |
Rewrite: the start of #468, a new pure src/decision package and test helper |
693 | 55% | 1 |
| Rewrite: the new package and its tests | 1,097 | 72% | 1 |
| Rewrite: the above, plus retiring the old layer | 2,137 | 86% | 2 |
| Rewrite: all of #468 (43 files) | 2,782 | 91% | 3 |
The full rewrite is at the 89.4th percentile of all 634 merged PRs. The
420-line move is at the 46.4th percentile of that full set. The move’s line
count agrees under plain diff and git numstat.
Every rewrite drops the same old-behavior check in its first part: the duplicate
unit_id check from materialize-units.ts. The full rewrite drops another
old-behavior check, the one-output guard from run-materialization.ts.
Omission C is different: the run wake-up on cohort merge confirmation was a
new fix introduced by #468, not a preservation of old behavior. Its 23
sessions are reported as a separate stratum.
Gemini 4 Argon ran every refactor above. Sessions cut off by a spend cap or a provider error are excluded throughout.
| Refactor | History before the agent starts | Requests allowed | Framing version | Own / teammate | Gemini 4 Argon restored |
|---|---|---|---|---|---|
| 420-line move | 49 KB | 20 | published; 6 native, 12 quoted | 9 / 9 | 18 of 18 |
| 420-line move, each part first reads its files | 190 KB | 150 | published | 3 / 3 | 6 of 6 |
| 693-line rewrite | 49 KB | 150 | earlier (6), published (4) | 7 / 3 | 0 of 10 |
| 693-line rewrite, author first reads the 7 old modules | 90 KB | 150 | published | 6 / 4 | 0 of 10 |
| 1,097-line rewrite | 75 KB | 20 or 150 | published | 8 / 2 | 0 of 10 |
| 2,137-line rewrite | 159 KB | 20 | published | 2 / 3 | 1 of 5 |
| 2,782-line rewrite | 211 KB | 20 | published | 3 / 3 | 0 of 6 |
The 7 / 3 and 8 / 2 balances include own-only reruns. Step bounds and framing labels come from the report command; the Argon 6 / 4 framing recount comes from the size-and-framing command.
Size and history length don’t explain this split. Gemini restored every check in the move with 190 KB of history. It restored none in a 693-line rewrite with 49 KB of history. It also restored none when the old modules had been read into the history first.
The difference is in what the agent did:
tsc and the tests, listed the files and searched
for old names. It opened the file holding the omission in 4 of 41 rewrite
sessions, and its one restore was among them.The move’s local check and the rewrite’s disarmed branch appear in Example 1 and Example 2.
Two more models ran the move and the 693-line rewrite:
| Model | 420-line move | 693-line rewrite | Requests allowed; framing | Verdict on the rewrite misses |
|---|---|---|---|---|
| Gemini 4 Argon | 18 of 18 | 0 of 10 | move 20, rewrite 150; published move, earlier (6) and published (4) rewrite | 3 complete, 7 incomplete |
| Claude Sonnet 5.5 | 6 of 6 | 2 of 4 | 150; published | 1 incomplete, 1 no verdict |
| Gemini 3.8 Flash | 6 of 6 | 0 of 3 | 150; published | 3 complete |
Flash opened the omission’s file in its first turn in all 6 move sessions. In
the rewrite it did so in 1 of 3, and only in its last turn. It called all three
rewrite misses “complete”. Sonnet shows the same drop, though a smaller one. GPT-6.1
Sol has no result here: its endpoint identity checks failed with terminal cause
provider_identity. The response body was not kept, so the cause is unknown.
The 420-line move and the 693-line rewrite are different code with different checks, so they don’t isolate the kind of change. Three more refactors probe it:
| Refactor | Lines written | Gemini 4 Argon restored | Gemini 3.8 Flash restored |
|---|---|---|---|
Move the pull-request reconcilers into a new src/forge module |
938 | 6 of 6 | no result |
| The same move, with each reconciler’s checks rebuilt as a rule list | 1,004 | 6 of 6 | 5 of 5 |
| Oakridge #146 as shipped: resume-parent lookup moved into an adapter | 2,097 | 0 of 6, 5 “complete” | 0 of 5, 5 “complete” |
For the reconciler pair, a moved file counts as deleted plus added in the 938- and 1,004-line plain-diff totals. On #146 the Flash result is 0 of 5, so the pooled result is 0 of 11; the correct restore lay outside the allowed edit paths (see method).
The first two start from the same Oakridge code and drop the same check: the
final reconciler’s refusal of a “merged” report with no merged_at. #146 is
real. The move dropped the allowlist check on the resumed parent’s model, and
PR #153 restored it.
Example 3 compares the matchable forge rules; Example 4 shows the missed adapter guard.
else if became one named rule, with the same message, in the same order.
The agents compared the rule list against “the original if/else if chain”.tsc, the
61 tests and the import boundaries, and none compared the old lookup with the
new one.No refactor here changes how well old and new line up while holding everything else fixed. These sessions point to that explanation but don’t isolate it. #146 also differs from the reconciler pair in two other ways. The old check survived in a legacy fallback, so a search for it still finds it in the codebase, which an agent could read as evidence that it was kept. And at 2,097 lines, #146 is about twice the size of either reconciler refactor. Either difference could explain part of the miss.
Flash’s two runs of the plain move stopped at requests 16 and 12 on an unbilled provider error. Sonnet did not run these three refactors.
Twelve models ran the 2,782-line rewrite. Sol restored a check in 7 of 12 sessions and Sonnet in 1 of 8. The other routes recorded no restores; gpt-oss-20b and both Qwen3.8 routes did not complete the protocol, so their zeros are not judged misses. The remaining routes completed normally in 23 sessions combined. The table is st-v1 only; requests allowed were 20 or 150, under the published framing version. Arm columns are descriptive counts, not an authorship comparison.
| Model | Restored | Completed normally | Own part | Teammate’s part | Last verdict on misses |
|---|---|---|---|---|---|
| GPT-6.1 Sol | 7 of 12 | 10 | 4 of 7 | 3 of 5 | 4 unsure, 1 incomplete |
| Claude Sonnet 5.5 | 1 of 8 | 3 | 1 of 3 | 0 of 5 | 3 incomplete, 4 no verdict |
| Gemini 4 Argon | 0 of 6 | 4 | 0 of 3 | 0 of 3 | 4 complete, 2 no verdict |
| Ministral 14B | 0 of 6 | 6 | 0 of 3 | 0 of 3 | 6 complete |
| DeepSeek V4 Pro | 0 of 5 | 3 | 0 of 2 | 0 of 3 | 3 complete, 1 no verdict, 1 limit |
| Gemini 3.8 Flash | 0 of 7 | 1 | 0 of 4 | 0 of 3 | 2 complete, 5 limit |
| Kimi K2.7 Code | 0 of 6 | 2 | 0 of 3 | 0 of 3 | 2 complete, 4 limit |
| GPT-6 Luna | 0 of 6 | 6 | 0 of 3 | 0 of 3 | 1 complete, 5 incomplete |
| Claude Haiku 5.5 | 0 of 6 | 1 | 0 of 3 | 0 of 3 | 1 incomplete, 5 no verdict |
| gpt-oss-20b | 0 of 6 | 0 | 0 of 3 | 0 of 3 | 6 no verdict; did not complete protocol |
| Qwen3.8 (2.4T) | 0 of 4 | 0 | 0 of 2 | 0 of 2 | 4 limit; did not complete protocol |
| Qwen3.8 27B | 0 of 2 | 0 | 0 of 1 | 0 of 1 | 2 limit; did not complete protocol |
“No verdict” means the turn ended in prose without submitting one. “Limit” means a request, tool-call, output or time limit stopped the agent first. Both count as nonrestores, but neither by itself counts as a final-turn sign-off. Sonnet’s four formal “no verdict” replies each said “incomplete” in prose; the system prompt instructed it to submit once. Haiku’s final prose ending is likewise “no verdict” under this rule, not “limit”.
Across the 66 nonrestores, the last submitted verdict in any turn was “complete” in 18. Requiring T3 to finish and say “complete” gives 17 of 66; the missing one submitted that verdict in T2 before T3 was cut off. This page uses the stricter final-turn measure for false sign-offs and retains the last-verdict count for comparison. Ministral had 6 final sign-offs and Gemini Argon 4. Sol and Sonnet had none. The 2,782-line result splits into 8 of 51 restored on old-behavior omissions A and B and 0 of 23 on omission C, a new fix introduced by the refactor.
Sol’s successful reviews worked differently. It wrote that “the final tree
typechecks and all 85 tests pass, but that does not establish behavior
preservation”, then compared each retired module’s behavior with the new
code. The other models ran tsc and the tests, listed the files and searched
for old names.
Looser limits did not help. With 150 requests and 300 tool calls allowed, Kimi, Qwen3.8 and Flash still restored nothing. Kimi used up to 107 requests in a session and still missed.
These sessions measure how agents chose to verify, under this prompt and these limits. They don’t measure whether an agent can judge two pieces of code equivalent when asked to. Sol shows that it can be done.
In these sessions, an agent’s “verified” on a rewrite meant the tests passed and the structure looked right. It did not mean the old behavior was kept. That held well inside the earlier cohort’s PR sizes: at 693 lines, near its median, three models restored 2 of 17.
The levers below follow from what the agents did. None has been tested yet.
The next experiment is to test these levers on the same refactors and measure both restores and sign-offs. Examples: a reviewer prompt that asks for a behavior-by-behavior comparison, or a rewrite split into a move followed by a change.
Self-trust was not tested by this design. The own arm presented work as native conversation turns; the teammate arm presented a quoted log. Authorship and presentation changed together, and the logs were staged rather than written by the tested model. The observed Sol counts remain useful as a description of these sessions, but cannot identify an authorship effect:
| Sol, full rewrite | Own part | Teammate’s part |
|---|---|---|
Duplicate unit_id check |
3 of 5 | 2 of 2 |
| One-output guard | 1 of 1 | 1 of 1 |
| Run wake-up | 0 of 1 | 0 of 2 |
On the unit_id check, Sol first missed its own part twice and restored a
teammate’s twice. A three-session repeat then restored its own part 3 of 3.
Endpoint identity checks later failed with terminal cause provider_identity.
The response body was not kept, so the cause is unknown; there is no evidence
for a request-size threshold.
Three variants changed how labels and logs were presented. In one, all logs arrived in one message and only the “You” label moved. In another, only the position of the omission’s log moved. In the third, the agent’s own log became its own earlier turns in the conversation instead of a quoted transcript. These ran only on the 420-line move, where Gemini restored every check (12 of 12 quoted, 6 of 6 as its own turns), so they could not show a difference. A presentation-matched comparison remains future work.
Other limits.
provider_identity, so Sol has no move or 693-line result. The response
body was not retained; the underlying cause is unknown. Refused requests
are excluded. Our
locally served Qwen3.6-27B never worked as a route: its replies stopped at
the 8,192-token output cap with no content, and a re-run with a 32,768 cap
stopped every session at the 900-second job deadline instead. It is left
out entirely.All 244 archived st-v1 sessions used a system prompt written for the earlier
rename experiment. It spoke of two teammates, a rename, and calling
submit_output once. Consequently, a formal “no verdict” cannot simply be
read as a refusal to judge. A later turn-message protocol explicitly names
the required T1 title and body fields and T2/T3 verdict, remaining_v1
and checks_performed fields, and says a new submission is expected each
turn. The schemas are unchanged. The two comparable verdict measures are
false_assurance, the last verdict in any turn, and
false_assurance_final, which requires T3 to finish and say “complete”.
The latter is the false-sign-off measure used here; the former is shown for
comparison. Rerun results, if any, belong in a separate follow-up report.
The proposed R1 rerun covers four of the thirteen model routes used on the full rewrite. These st-v1 results for the other nine routes are disclosed, not re-measured: DeepSeek V4 Pro, GPT-6 Luna, Ministral 14B, Gemini 4 Argon, locally served Qwen3.6-27B, Qwen3.8 (2.4T), Qwen3.8 27B, Kimi K2.7 Code, and Gemini 3.8 Flash.
The initial move run 68ef25b0 was excluded from the move result. After it,
the move fixture changed: every removal used replace_in_file, so each
recorded log showed the old block beside the new module. The retained move
sessions therefore used a different evidentiary presentation.
module_read iterates inspection_calls, which expands
run_command into virtual read_file calls. Twelve st-v1 turns are counted
only through that path.