bench — a prototype verification loop

The goal of this tool is to better inform your review of the diff. It does that via a prediction the person shipping writes before anything runs, to check that they understand what they're shipping. The human produces the claims and the machine records the reality. Below you can find a real example from building this tool, and the prediction missed. Can you spot where?

Prediction — written by a person, locked before the code ran

RouteSpec should run against both the logic within TaskService and the DoobieTaskRepo, via PostgresSuite. I expect all 14 cases to pass. Its possible an edge case exists around the FSM, with (Assigned, Assign) => Assign yielding unclear but unexpected behavior.

locked 2026-08-09 15:54:46Z · write-once · no edits after this point

Captured run — recorded by the machine, not described by the author

$ sbt test exit 0 10.5s · 18 passed

started 15:56:00.561Z · ended 15:56:11.045Z · 50 lines stdout · 0 bytes stderr

85 seconds after the prediction was locked. The gap is enforced by the server against its own clock, not by the client.

The delta — where the run did not match the claim

Predicted

Ran

Unaccounted for

The 4 TaskTransitionSpec — the suite whose entire job is exhaustively testing FSM transitions. The prediction never mentions it.

The prediction's own uncertainty “Its possible an edge case exists around the FSM…”

The gap in the map sits exactly where the author's confidence was lowest. Those are the same four tests.

The rest of the record — submitted, ordered, write-once, one click away

Captured output — key lines, ANSI stripped

Test run verificationloop.domain.TaskTransitionSpec finished: 0 failed, 0 ignored, 4 total 0.018s
Test run verificationloop.http.RoutesSpec finished: 0 failed, 0 ignored, 14 total 2.99s
[info] Passed: Total 18, Failed 0, Errors 0, Passed 18
[success] Total time: 8 s

Excerpted from a 50-line capture. The stored payload keeps the full output and the ANSI escapes; stripping happens at render time.

Check — submitted 16:15:41Z

The TaskTransitionSpec is clear about the FSM, and has coverage for exactly the ambiguous/error prone case I called out.

Induced failure — superseded, and what replaced it · 2026-08-23

The fourth step above asked the author to break the check by hand and describe what happened. That is now a machine's job: one command runs the check clean, applies a mutation that should break it, runs it again, and restores the tree — three captures, one verdict. Below is the first time it was pointed at this project's own test suite, on a later task.

The mutation — delete the role check that stops a senior claiming a junior's task:

-      case Right((_, actor)) if actor.role != Role.Junior =>
-        ServiceError.Forbidden("only a junior may claim a task").asLeft[Task].pure[ConnectionIO]

The prediction, locked before the run — it commits to an answer and names how it could be wrong:

Prediction, committed: B is red. Stated risk, committed: if B is green, it is the assignedTo guard covering for the role guard, and the test needs isolating rather than the code needing fixing.

What the run recorded — the risk, not the prediction:

clean     exit 0   Test run verificationloop.http.RoutesSpec finished: 0 failed, 0 ignored, 25 total 3.989s
mutated   exit 0   Test run verificationloop.http.RoutesSpec finished: 0 failed, 0 ignored, 25 total 3.882s
restored  exit 0   Test run verificationloop.http.RoutesSpec finished: 0 failed, 0 ignored, 25 total 3.912s

The guard was deleted and the suite did not notice: 96/96 either way. Verdict: fail — not the code failing, the check failing. The review said so and sent it back:

The control came back fail: B green, 96/96, the check did not notice the role check being deleted. That is a real gap, not a flaky run — RoutesSpec asserts the 403 status but not which guard produced it, and assignTask means a senior is never the assignee, so the assignment guard answers with the same status. Assert the message and run the control again.

Then the same mutation, against the strengthened test — one assertion added, comparing which guard answered:

==> X verificationloop.http.RoutesSpec.rejection: only a junior may claim a task 0.035s munit.ComparisonFailException: src/test/scala/verificationloop/http/RoutesSpec.scala:391
…
values are not the same
=> Obtained
task is not assigned to this junior
=> Diff (- expected, + obtained)
-only a junior may claim a task
+task is not assigned to this junior

Red at one line in one suite, green again once the guard is restored. Verdict: pass. The mechanism submitted with it states the limit rather than the win:

The limit of the claim: this proves the suite now detects the deletion of the role check. It does not prove the ordering of the two guards is itself pinned, and no test here exercises a third role, because the seed data has two people.

A different task from the bundle above, recorded the same way and shown for the same reason: the first run of this control on the project's own code found a test that could not see what it claimed to test. Nothing was wrong with the code. A green suite could not have told anyone that.

→

The diff

The reviewer proceeds to the diff with the above in hand.

The diff is where this ends, not what it replaces.

Will the check catch it?

Four edits to a small project, four real runs. Decide whether the check goes red, then open each one. Every exit code, every line of output and every verdict below was recorded by a bench control run; nothing here is illustrative, and one of the four does not get an answer at all.

The checksrc/test/scala/CalcSpec.scala

class CalcSpec extends munit.FunSuite:
  test("add sums its arguments"):
    assertEquals(Calc.add(2, 2), 4)

The whole of what it testssrc/main/scala/Calc.scala

object Calc:
  def add(a: Int, b: Int): Int = a + b