bench — a prototype verification loop

The goal of this tool is to better inform your review of the diff. It does that via a prediction the person shipping writes before anything runs, to check that they understand what they're shipping. The human produces the claims and the machine records the reality. Below you can find a real example from building this tool, and the prediction missed. Can you spot where?

Prediction — written by a person, locked before the code ran

RouteSpec should run against both the logic within TaskService and the DoobieTaskRepo, via PostgresSuite. I expect all 14 cases to pass. Its possible an edge case exists around the FSM, with (Assigned, Assign) => Assign yielding unclear but unexpected behavior.

locked 2026-08-09 15:54:46Z · write-once · no edits after this point

Captured run — recorded by the machine, not described by the author

$ sbt test exit 0 10.5s · 18 passed

started 15:56:00.561Z · ended 15:56:11.045Z · 50 lines stdout · 0 bytes stderr

85 seconds after the prediction was locked. The gap is enforced by the server against its own clock, not by the client.

The delta — where the run did not match the claim

Predicted

Ran

Unaccounted for

The 4 TaskTransitionSpec — the suite whose entire job is exhaustively testing FSM transitions. The prediction never mentions it.

The prediction's own uncertainty “Its possible an edge case exists around the FSM.”

The gap in the map sits exactly where the author's confidence was lowest. Those are the same four tests.

The rest of the record — submitted, ordered, write-once, one click away

Captured output — key lines, ANSI stripped

Test run verificationloop.domain.TaskTransitionSpec finished: 0 failed, 0 ignored, 4 total 0.018s
Test run verificationloop.http.RoutesSpec finished: 0 failed, 0 ignored, 14 total 2.99s
[info] Passed: Total 18, Failed 0, Errors 0, Passed 18
[success] Total time: 8 s

Excerpted from a 50-line capture. The stored payload keeps the full output and the ANSI escapes; stripping happens at render time.

Check — submitted 16:15:41Z

The TaskTransitionSpec is clear about the FSM, and has coverage for exactly the ambiguous/error prone case I called out.

Induced failure — submitted 16:18:39Z

I set the status to Ok instead of Forbidden in .rejection: only a junior may claim a task, it broke because the test correctly returns 403 (the logic dictates it only allow juniors to claim tasks)

Shown as it was recorded. This step has since been redesigned: the check and the mutation that should break it now run in one capture, with a three-valued result. What is above is what the loop asked for at the time, and it is left unretouched.

→

The diff

The reviewer proceeds to the diff with the above in hand.

The diff is where this ends, not what it replaces.