Runtime

Deterministic replay

Re-run a recorded agent session with every tool answer served back out of the recording, byte for byte. The world is held still, so a changed outcome is a change in your agent.

BUILDING It ships in the harness package as pit_harness.replay and runs today against the fixture release in the repository. The harness is Apache-2.0 and installs from source; the PyPI release is still pending.

pip
python -m pit_harness.replay
Apache-2.0
offline

The artifact

A dry replay, reproduced on this machine

One stub-agent run over the committed fixture release, then a dry replay of it. Both were run on 2026-08-27, and the hashes are reproducible on any machine that holds the same fixture.

python -m pit_harness.replayexit 0

replay dry  of sha256:649ac976e2b70b1d0cfd67765f00086812289e0ab7194a4edebb80fd915b99fd
  recording verified against receipt  turns 94/94  calls matched 90/90
  replay trajectory sha256:649ac976e2b70b1d0cfd67765f00086812289e0ab7194a4edebb80fd915b99fd  identical=True
  no divergence
  wrote replay/trajectory.jsonl, replay/receipt.json, replay/divergence.json

terminalno output means no difference

$ cmp trajectory.jsonl replay/trajectory.jsonl

94 decision points, 90 recorded tool calls, every one matched on its position, its name and a sha256 over its canonical arguments. The replay wrote the same trajectory bytes the run did.

The failure it prevents

A score that moved for an unknown reason

You change a prompt and the number changes. It could have been the agent, the harness, or the corpus underneath both, and without a recording of the last run there is no way to separate them.

Mode	The agent	The tools	What it tells you
`full`	runs live	answered from the recording	whether this agent, today, behaves as it did over the same world
`dry`	answered from the recording	answered from the recording	whether the loop itself is still deterministic, at no model cost

Run full after changing a prompt, a model or your agent code. The tool answers are held to the recorded ones, so a changed result points at your change. Run dry to separate a harness change from an agent change: it calls no model, re-issues the recorded calls in order, and puts the whole event stream back through the runner. If dry stops reproducing the recording, the harness moved.

Where the dry mode sits against other products

We read the published documentation of the products that use the word “replay” on 2026-08-27. One of them answers a recorded agent's tool calls from the recording rather than from the live system: Kitaru, from ZenML, shipped 2026-08-18, whose history tool policy takes on_miss="fail", the same fail-closed rule used here. Its replay re-runs the model every time, and its documentation says a replay runs “from the top”, so it has no policy that serves the model's own answers back. Dry replay is a zero-cost determinism check of the loop itself that, as of 2026-08-27, no other vendor ships.

Several products use the word “replay” for something else: LangGraph time travel re-executes nodes including the LLM calls, Microsoft Foundry trace replay and AgentOps session replay play a recording back in an interface without re-executing, and vcrpy and its relatives record at the HTTP layer as libraries you wire up yourself.

How it works

The recording is the world, and it is hashed first

Three things have to match before an answer is handed back

A recorded turn is one prompt, the tool calls the agent made after it, and the answer it gave. Replay walks the turns in the order the runner produced them and matches each call three ways: its position in the turn, its tool name, and a sha256 over its canonical arguments. When all three match, the recorded result is handed back unchanged, down to known_at and the withheld-row count.

an unmatched call, and the only thing it gets

{"error": "replay_divergence", "rows": [],
 "detail": "the recording holds no pit_coverage call at this position with
            these arguments, and a replay serves nothing it did not record"}

The replay never reads the release through a tool, so an unmatched call cannot quietly succeed against live data and hide the divergence that produced it.

A tampered recording is refused before a byte is served

The recording is hashed before anything is served from it and hashed again immediately before the first byte goes out. Passing --receipt checks the receipt's own body_sha256, the trajectory's sha256 against binding.trajectory_sha256, and the canonical form of every line. Changing one character inside one served title is enough:

one byte edited in a served rowrefused

refused: tampered.jsonl hashes to sha256:79def7382e884a80d65eeb100b585a03db96cd54e3968e59dfc748f7675a62d9,
but the receipt binds sha256:649ac976e2b70b1d0cfd67765f00086812289e0ab7194a4edebb80fd915b99fd;
the recording has changed since the run

A recording with nothing to check it against is refused by default. --unverified replays one anyway, and both the report and the receipt then say the recording was unverified and the replay invalid.

Where a divergence is attributed

Kind	Means	Attributed to
`tool_call_mismatch`	The call at this position had a different name or different arguments.	agent
`unrecorded_call`	The agent kept calling after the recorded turn ran out.	agent
`missing_call`	The recording holds a call the agent did not make.	agent
`response_mismatch`	Every call matched and the agent still answered differently.	agent
`prompt_mismatch`	The runner rendered a prompt the recording does not hold.	harness
`unrecorded_turn`	The replay reached a decision point the recording has no turn for.	harness

The first four are attributed to the agent because the tool answers were identical by construction. The last two mean the replay is running a different configuration, not a different agent.

Integration

Run it against a run you already have

Only the paths come from the command line. The arms, the clocks, the window, the task, the declared model and the frictions are rebuilt from the recording's own run_start event. The identity map is the exception, because the receipt records how many entries it had rather than what they were.

terminalexit 0 clean · 1 diverged · 2 refused

$ pip install -e harness/
$ python -m pit_harness.replay \
    --trajectory trajectory.jsonl --receipt receipt.json \
    --bundle dist/bundle/pit-eval-sec-2022-11_2023-03 \
    --prices prices.csv --out replay/

A replay is itself a run, so it writes an ordinary run receipt with two fields added: body.replay_of holds the original receipt's body_sha256, and body.replay holds the mode, the call and turn counts, the divergence list, and whether the trajectory came out identical.

What it does not establish

It does not show you the world as it is now. Every tool answer comes from the file, so a release rebuilt since, or a filing corrected since, still replays as it was served. Run the harness again when you want the current world.
It does not score the answers. A clean replay says the agent made the same calls and gave the same answers over the same bytes, and the metrics and the control arms measure whether those answers were any good.
It does not establish where a recording came from. The hashes show the trajectory has not changed since the receipt was written, and a recording and a receipt forged together would agree with each other. Catching that needs a signature, which is what a contamination certificate adds.
It does not remove a model's own nondeterminism. In full mode a model above temperature zero will diverge on a world that did not move, which is the variation that mode measures, and dry is the control for it.

Upstream