The artifact

A run, start to finish, with no model wired up

--agent defaults to a stub that needs no model: it acts on every filing the window has just made knowable. That is enough to watch the machinery work before you spend a token on it.

pit-harness run · 2023-03-01 → 2023-03-10 · 8-K 2.7s · exit 0

$ pit-harness run --config run.json \
       --trajectory trajectory.jsonl --receipt receipt.json
run bf78382b9021b3be  valid=True  receipt sha256:4580ffd92e09f045e71d4e63069ce200c9d14f865aef5a6ef88026410718c395
trajectory sha256:43e4a26c953d2a238a75c86bdafbe16dc2a5003130c13117d80f135fb898cc60  events 266
  main      leaky  total_return      n/a  trades    0  reachable=False
  main      dump   total_return      n/a  trades    0  reachable=True
  main      pit    total_return      n/a  trades    0  reachable=True
  redacted  leaky  total_return      n/a  trades    0  reachable=False
  redacted  dump   total_return      n/a  trades    0  reachable=True
  redacted  pit    total_return      n/a  trades    0  reachable=True
  shifted   leaky  total_return      n/a  trades    0  reachable=False
  shifted   dump   total_return      n/a  trades    0  reachable=True
  shifted   pit    total_return      n/a  trades    0  reachable=True
  probe shifted   mask_leaks  recovered 1/10
  protocol gaps: P1_model_version, P2_training_cutoffs

Run on 2026-08-27 against the pit-eval-sec-2022-11_2023-03 release, reading the JSONL arms, on a laptop.

Three lines of that output need explaining.

`total_return n/a`

The harness ships no market data and fetches none. This run passed no prices, so the nine cells score no returns and still produce a complete receipt. You supply the bars and state where they came from, and the receipt records that statement without checking it.

`mask_leaks 1/10`

The positive control asks the agent, on the shifted arm, what the real date and issuer were, then scores the answer text against the answer key. The stub does not answer the question — it replies with the same sentence it gives the trading task, and that sentence contains the session's own timestamp. One probe item happened to be dated 2023-03-10, the substring matched, and the verdict flipped. Widen the window to all of March and the same stub scores 0/10 and mask_holds. A leaking probe on an agent that never read the question is the scorer being literal, and it is visible because the receipt carries every per-item score.

`protocol gaps`

P1 and P2 are the model version and its training cutoff, which the harness cannot observe. They are declarations you make in RunConfig, and leaving them empty lists them as gaps rather than filling them with a guess.

The artifact, continued

The receipt, and checking it on your own machine

receipt.json · body, trimmed sha256:4580ffd9…

{
  "receipt_version": 1,
  "run_id": "bf78382b9021b3be",
  "harness": {"name": "pit-harness", "version": "0.1.0"},
  "valid": true,
  "invalidated_by": [],
  "binding": {
    "bundle_content_digest": "sha256:bc924991d6ea64984dd533874014e83dac8a52b9bd442f08f89b60a5f055c15c",
    "config_sha256":         "sha256:e846d10fbd126ff9a249c7d2ff8dac0cd1f8b74d458943334c175ecd21fa443f",
    "trajectory_sha256":     "sha256:43e4a26c953d2a238a75c86bdafbe16dc2a5003130c13117d80f135fb898cc60"
  },
  "network": {
    "guard": "socket-monkeypatch",
    "policy": "fail_closed",
    "clean": true,
    "egress_attempts": 0,
    "scope": "in-process; a subprocess or a model provider's own weights are out of scope"
  },
  "trajectory": {"path": "trajectory.jsonl", "events": 266, "sha256": "sha256:43e4a26c…"},
  "protocol_complete": false,
  "protocol_gaps": ["P1_model_version", "P2_training_cutoffs"]
}

verify recomputes all three hashes on the reader's machine, against files the reader already holds. It needs no network and no trust in us.

pit-harness verify exit 0

$ pit-harness verify receipt.json \
       --trajectory trajectory.jsonl --bundle pit-eval-sec-2022-11_2023-03
body_sha256 ok  sha256:4580ffd92e09f045e71d4e63069ce200c9d14f865aef5a6ef88026410718c395
run_valid True  protocol_complete False
trajectory ok  sha256:43e4a26c953d2a238a75c86bdafbe16dc2a5003130c13117d80f135fb898cc60
bundle content_digest ok  sha256:bc924991d6ea64984dd533874014e83dac8a52b9bd442f08f89b60a5f055c15c

Two runs of the same configuration over the same release write the same receipt body. We ran the command twice, three minutes apart, into different directories:

"body_sha256": "sha256:4580ffd92e09f045e71d4e63069ce200c9d14f865aef5a6ef88026410718c395" - "emitted_at": "2026-08-27T18:50:35Z" + "emitted_at": "2026-08-27T18:53:36Z"

emitted_at is the only wall-clock field and it sits outside the hash, so the run has one identity and the file has a timestamp. Keys sort at every level, floats are fixed at six decimals, -0.0 is normalised, and probe items are chosen by a fixed stride rather than a sample.

The failure it prevents

Two ways an eval score is not about the agent

It looked it up

While a run is active the harness replaces the socket entry points with ones that raise: INET socket creation, connect, create_connection, DNS resolution, and TLS wrapping. AF_UNIX sockets keep working because they cannot leave the machine. There is no allowlist and no override flag. Every attempt is recorded with the call site that made it, and the receipt reports the run invalid whether the policy was to abort at the first attempt or to carry on. The guard covers this process, which is where a tool, an SDK or a stray urlopen lives; a subprocess the agent spawns needs a network namespace around the whole run.

It already knew

No guard reaches a model that read about March 2023 in training. The benchmark release therefore ships three arms over the same rows: as served, with issuer identity removed, and with every date moved ten years forward. The score that survives the shifted arm came from the rows; the score that vanishes came from memory. The gap between them is reported per clock as the contamination delta.

And the mask itself can leak

A shifted arm is only a control while the mask holds, and naive masking is breakable. So the same agent is asked, on the shifted arm and against rows it was actually served, what the real date and issuer were. date_recovered needs a full date rendering; a bare year is recorded separately as weaker evidence. verdict sits in the receipt next to the arm delta it qualifies, because a delta that collapsed because the mask failed reads exactly like an agent that behaved.

For the size of the effect, we ran the same three clocks over 88 8-K filings from twelve issuers between 2022-11-01 and 2023-03-31 with the clock as the only variable: +5.13% on the naive filed-date join, −2.02% waiting for the nightly dump, and +0.18% on the acceptance receipt. 47 of the 64 filings that carry a receipt were bought at an auction that had already opened when EDGAR accepted them. The full method is on benchmarks.

How it works

Nine cells, four tools, one calendar

The grid

A run is arms by clocks: three arms from the release, three clock policies, nine cells. For each cell the runner walks the session calendar and calls your agent once per session with its tools bound to that session's opening bell.

The three clocks

leaky reads the partition date at 00:00:00Z and enters at the opening auction of the filing's own index day, which is unreachable and labelled so. dump reads published_at at 23:59:59Z and enters the next morning. pit reads acceptance_at where the row carries one and published_at otherwise, so an intraday receipt lands the entry on the same morning when it arrives before 09:30 and the next one when it arrives after. Daylight saving is resolved through America/New_York.

The tools

pit_list, pit_search, pit_read and pit_coverage. Every answer carries known_at, clock, arm and session_date, and every call goes into the trajectory with its arguments and its whole result. Reading a row whose clock has not passed returns not_yet_knowable and the key, never the row.

Row keys, not tickers

Picks are row_key values, the one identifier present and identical in all three arms, so the same agent code runs against every arm and the deltas compare like for like. Control arms are scored on the reference arm's clocks, joined on row_key, because the shifted arm's own clocks sit ten years forward.

P1 to P6

The receipt fills the six reporting slots The Alpha Illusion asks for — model version, training cutoff, retrieval timestamps, costs and frictions, window provenance, disaggregation — from the run rather than from prose. Three are measured, two are declared, and one is declared and then checked: P2 compares your stated cutoff against the release window and returns window_ends_before_cutoff when the window is inside the model's training data, which means the main arm's score is not evidence of forecasting.

Integration

Wrap the function you already have

a plain callable python

from pit_harness import Bundle, RunConfig, load_prices, run
from pit_harness.adapters.plain import from_callable

bundle = Bundle.load("pit-eval-sec-2022-11_2023-03",
                     forms=["8-K"], start="2023-03-01", end="2023-03-10")
prices = load_prices("prices.csv", source="your vendor", pulled_at="2026-08-27")

def my_agent(prompt, tools):                      # your model goes here
    return [r["row_key"] for r in tools.pit_list(limit=20)["rows"]]

result = run(from_callable(my_agent),
             RunConfig(bundle_path=bundle.root, forms=("8-K",),
                       start="2023-03-01", end="2023-03-10",
                       trajectory_path="trajectory.jsonl",
                       receipt_path="receipt.json"),
             bundle, prices)
print(result.receipt["body_sha256"], result.total_return("main", "pit"))

from_tool_loop(step) takes the shape most provider SDKs already have and runs the loop for you. pit_tools(toolbox) exposes the four window tools as Inspect tools, and from_inspect_solver runs an Inspect solver inside the guard, the trajectory and the receipt. For anything else, export writes the decision points as JSONL that Inspect reads through json_dataset(), promptfoo through file://samples.jsonl, and lm-evaluation-harness through a datasets loader.

export decision points sh

$ pit-harness export --bundle <release> --arm shifted --clock pit --out samples.jsonl

The samples carry the prompt and the metadata but no tools, so a run driven from the exported file still reads the window through the harness.

Limits

What this does not do yet

No filing text: The built release carries metadata only, and the title is a mechanical company-and-form concatenation. A language model has almost nothing to read, so the contamination delta measures what it remembers about a date and an issuer name rather than how it reads a filing. The rebuild that adds text is the same command under the same release name.
Sparse acceptance stamps: 3,124 of 287,929 rows in that release carry acceptance_at. Rows without one fall back to published_at on the PIT clock and the run counts the fallback; a window in 2022-11 has no stamps at all, so its PIT and dump arms are identical by construction.
What the probe misses: It catches a model that writes the true date or names the issuer. A model that recovers the period and keeps it to itself passes, so mask_holds says the answer text carried nothing, and says nothing about what was inferred silently.
You supply the prices: No market data ships and none is fetched. The receipt records your statement about where the bars came from under P4 without checking it.
One asset class, one entry rule: US equity daily bars, long at the next opening auction, held a fixed number of sessions, no costs, no slippage, no shorts. Absolute returns belong to the basket and the window; only the difference between cells is attributable to the input that changed.

Benchmark Datasets: The release a run reads: the frozen window, the three arms, and the receipts the harness pins in bundle_content_digest.
Eval Gates: The same receipt, compared against a committed baseline on every commit, so leakage fails a build the day it appears.
Post-Cutoff Holdouts: Windows dated after any current model's training data, where the contamination question is settled by the calendar.
Contamination Certificates: A signed verdict over one run, built from this receipt and the reasoning audit beside it.

The harness is Apache-2.0 and free, and it runs against a release you already hold. Measuring your own agent needs nothing else on this page.

Evaluation Harness