Eval Gates

$ pit-harness gate receipt.json --baseline evals/baseline.json --mode block
python -m pit_harness.gate  mode=block  tolerance=defaults
  baseline   evals/baseline.json
             run bf78382b9021b3be  sha256:4580ffd9...  bundle sha256:bc924991...
  candidate  receipt.json
             run 9ba0269e2c5cd3b7  sha256:5a14bfb1...  bundle sha256:bc924991...

  CHECK             SUBJECT                           BASELINE            CANDIDATE           CHANGE     TOLERANCE
  positive_control  shifted@pit                       0.100000            0.000000            -0.100000  0.000000   pass
  timing            mean_sessions_pit_ahead_of_dump   0.000000            0.011700            +0.011700  0.050000   pass
  timing            mean_sessions_leaky_ahead_of_pit  1.000000            0.988300            -0.011700  0.050000   pass
  timing            mean_leaky_lookahead_hours        0.000000            6.661600            +6.661600  0.500000   FAIL
  timing            max_leaky_lookahead_hours         0.000000            8.026700            +8.026700  0.500000   FAIL
  timing            acceptance_fallback_rate          1.000000            0.939427            -0.060573  0.050000   FAIL
  integrity         baseline body_sha256              sha256:4580ffd9...  sha256:4580ffd9...  n/a        n/a        pass
  integrity         candidate body_sha256             sha256:5a14bfb1...  sha256:5a14bfb1...  n/a        n/a        pass
  integrity         candidate run valid               n/a                 True                n/a        n/a        pass
  comparability     bundle_content_digest             sha256:bc924991...  sha256:bc924991...  n/a        n/a        pass
  comparability     config_sha256                     sha256:e846d10f...  sha256:38983f39...  n/a        n/a        WARN
  comparability     harness version                   0.1.0               0.1.0               n/a        n/a        pass

  3 failures, 1 warning, 12 checks.
  FAIL timing mean_leaky_lookahead_hours: moved +6.661600 hours, past the 0.500000 allowed, so the two runs saw different clocks
  FAIL timing max_leaky_lookahead_hours: moved +8.026700 hours, past the 0.500000 allowed, so the two runs saw different clocks
  FAIL timing acceptance_fallback_rate: moved -0.060573 fraction, past the 0.050000 allowed, so the two runs saw different clocks
  WARN comparability config_sha256: the run configuration changed, so a moved number may be the config rather than the agent
  Blocking on 3 failing checks, exit 1.

Run on 2026-08-27. The whole table is printed, passes included, and the failures are repeated underneath with the numbers behind them. --json prints the same report machine-readably, and --report-json PATH writes it to a file while the table still goes to the log.

Read the three failures back and they say something specific: the March window carries acceptance stamps the ten-day window did not, so the PIT clock stopped falling back to the filer clock, and the naive arm's lookahead became measurable at 6.66 hours on average and 8.03 at worst. Those numbers moved because the two runs read different clocks. Comparing contamination between them would be comparing different populations, which is exactly what the gate is for.

A join gets rewritten, a helper starts reading a current-ticker table, a cache warms from a later snapshot. The tests pass, because the change is correct in every way except the clock, and the eval score goes up, because the agent now sees things it could not have seen. Nothing in the diff points at a clock, so the higher score reads as progress and the commit lands.

The gate turns that into a red build on the commit that introduced it. It compares the gap between the agent's score and what each control arm reproduces, and it only counts growth: a delta that shrank, meaning the main arm moved closer to its controls, passes.

It also checks the check. If the baseline said mask_holds and the candidate says mask_leaks, the build fails at any tolerance, including one set deliberately wide. The shifted arm's delta measures memorisation only while the mask holds, so once the probe recovers the true date or the issuer, that delta is measuring something else and reading low for a reason unrelated to the agent. A tolerance there would apply to the check that qualifies every contamination number above it.

acceptance_fallback_rate is the one worth knowing about: it is the share of events that fell back to the filer clock because the row carried no acceptance stamp. It rises when a rebuild lost stamps, which silently collapses the PIT clock onto the dump clock and makes two arms identical without anything looking broken.

Exit 0 on no failures, or on --mode report whatever it found. Exit 1 in block mode with at least one failure. Exit 2 when a receipt could not be read at all, in both modes, so a broken file never reads as a pass.

- uses: actions/checkout@v5
- run: pit-harness run --config run.json --receipt receipt.json
- uses: ./harness/action
  with:
    receipt: receipt.json
    baseline_path: evals/baseline.json
    tolerance: "0.005"
    mode: block

harness/action/ is a composite action written with bash and python3 and no other action. It installs the package sitting next to it, runs the gate, appends the table to the job summary, and exits 1 in block mode. Its outputs are status, failures, warnings and exit_code, so a later step can comment on the pull request or upload the JSON report without re-reading the log.

Establishing a baseline is two commands: run the harness, commit the receipt. Read that baseline's own probe verdict before you commit it, because a baseline that was already leaking will pass every candidate that leaks the same amount.

$ pit-harness run --config run.json --receipt receipt.json
$ cp receipt.json evals/baseline.json && git add evals/baseline.json

Ships a gate

Langfuse, whose experiment action landed 2026-05-25 and fails on regression by default, and Confident AI, whose DeepEval ships a pull-request eval gate with a regression tolerance and a governance command.

Ships something weaker

Braintrust's action takes no threshold and fails only on an exception. LangSmith alerts without blocking.

Documents a do-it-yourself gate

Arize, Galileo, HoneyHive, Maxim and Opik point at an assert or a sys.exit on their CI pages.

This gate compares the gap between the agent's score and what a control arm reproduces, and then checks that the control still held. Measuring leakage needs a control arm and a probe against it; the vendors above ship neither, so what their gates compare is the score.

The baseline sets the floor

If the run you committed was already scoring on memorised knowledge, every candidate that leaks the same amount compares clean.

A changed release resets the comparison

The deltas and the timing aggregates are properties of a run over one release. Regenerate the baseline when the release changes; the comparability warning tells you it happened, and --fail-on-warn makes it stop the build.

It reads the probe's verdict and nothing more

A model that recovers the period and says nothing about it passes the probe, and therefore passes this check.

Calibration is not gated

The harness marks that metric a stub, and the gate does not compare it.

It compares two runs at a time

The gate keeps no state between runs. Tracking a number across many commits is something your CI does with the receipt files.

Evaluation Harness

The runner that writes the receipts this reads. The gate has no other input.

Benchmark Datasets

The release both runs are scored against, and the digest the comparability check watches.

Contamination Certificates

The same evidence signed for an outside reader, rather than checked for your build.

Decision Audit Logs

The production counterpart: the same discipline applied to decisions rather than to commits.