The artifact

The problem, stated by a run receipt

Declare your model's training cutoff and the harness compares it against the window you are evaluating on. Here is that slot from a real run over the frozen 2022–2023 release, with a cutoff of 2026-05-01 declared:

receipt.json · body.protocol.P2_training_cutoffs protocol_complete: true

{
  "training_cutoff": "2026-05-01",
  "evaluation_window": {"start": "2022-11-01", "end": "2023-03-31"},
  "verdict": "window_ends_before_cutoff",
  "note": "window_ends_before_cutoff means the main arm is inside the model's
           training data and its score is not evidence of forecasting"
}

Run on 2026-08-27. Leave the cutoff empty and the slot reports undeclared, which lands in protocol_gaps.

A holdout window flips that verdict, because the window starts later than the cutoff. The corpus's forward edge is public and takes no key:

GET /v1/meta 200 · read 2026-08-27

$ curl -s https://api.pit.aqx.llc/v1/meta \
       | jq '.results[0].corpus.sources[] | {source_id, last_partition_date}'
{"source_id": "sec.edgar",                 "last_partition_date": "2026-08-27"}
{"source_id": "us.cftc.cot",               "last_partition_date": "2023-03-28"}
{"source_id": "us.fda.enforcement.device", "last_partition_date": "2023-03-12"}
{"source_id": "us.fda.enforcement.drug",   "last_partition_date": "2023-03-12"}
{"source_id": "us.fda.enforcement.food",   "last_partition_date": "2023-03-12"}
{"source_id": "us.federal_register.pi",    "last_partition_date": "2026-08-27"}
{"source_id": "us.ofac.sdn",               "last_partition_date": "2024-08-06"}

Two of the seven sources carry a forward edge, because those are the two the tape polls. Run the same call tomorrow and both dates have moved.

The failure it prevents

Where redaction and date-shifting stop working

Redaction removes the issuer. Date-shifting moves the calendar. Both are masks over events that already happened, and masks can be recovered: an agent that reads the period back out of wording, a brand name the scrubber missed, or plain recall will score the control arm on the same knowledge as the main arm, and the delta between them collapses for a reason that has nothing to do with the agent behaving. That is why the harness ships a probe whose only job is to test whether the mask held.

A window that begins after a model's training cutoff needs no mask. There is no earlier version of these records anywhere, because the events had not happened when the model was trained. A control arm argues that the score came from the rows; a post-cutoff window settles the same claim by comparing two dates.

The rows also carry the stamp that makes the claim checkable. available_at is set to the minute our poller fetched the bytes, with availability_basis: local_first_seen. A backfill cannot produce that stamp, and neither can a later crawl of any archive, so a holdout row proves it was observed rather than reconstructed.

How it works

The tape ages into an eval set

Capture

A scheduled job has read the SEC current feed and the Federal Register public-inspection list every fifteen minutes since 2026-08-26, hashing what came back and stamping each new row with the minute it arrived. A tick writes whatever the record published in that window; on a quiet evening that is nothing at all, and the day's certificate records the window as read rather than leaving a hole.

Cutting a window

A holdout is a date range over the forward_first_seen lane, cut the same way a benchmark release is cut and shipped with the same three arms and the same day-by-day receipts. It reads through the same tools, so an agent written against a frozen window runs against a holdout without an edit.

The verdict

You declare the cutoff your provider states, in YYYY-MM-DD or YYYY-MM, and the receipt records it beside the window and returns window_ends_after_cutoff. That verdict travels with the run, so a reader six months later does not have to take your word for which side of the line the evaluation sat on.

It only grows forward

A week the tape does not run has no first-seen stamps in it, and nothing produces them afterwards. Someone starting the same job tomorrow starts a holdout that begins tomorrow.

Integration

Declaring the cutoff

The verdict comes from one block in the run config. This is the config behind the receipt at the top of this page, with the window swapped for a holdout range:

run.json json

{
  "bundle_path": "<your holdout release>",
  "forms": ["8-K"],
  "start": "2026-08-26",
  "end":   "2026-08-27",
  "model": {
    "name": "your-model",
    "version": "1",
    "provider": "your provider",
    "training_cutoff": "2026-05-01"
  }
}

Run it and P2_training_cutoffs reports window_ends_after_cutoff, and protocol_complete turns true once P1 is declared alongside it. Leave model out and both slots land in protocol_gaps, which is a finding rather than a default.

A holdout reads through the same four tools as a frozen window, so an agent already wired to the harness needs no change beyond the path.

Limits

What a post-cutoff window does and does not settle

It is young: The tape started on 2026-08-26. A holdout is as wide as the record since then, which on 2026-08-27 is two partition days. For a long evaluation window today, the frozen 2022–2023 release with its control arms is the larger instrument.
Two of the seven sources: The tape polls SEC EDGAR and the Federal Register public-inspection list. The other five sources in the corpus are backfill and stop in 2023 or 2024.
The cutoff is a declaration: The verdict compares your window against the date you stated. A provider that trained past its published cutoff, or a model refreshed after you declared, moves the line without moving the receipt. What the receipt fixes is the claim, so a reader can check it against the provider rather than against you.
An agent can still know another way: An agent with a live tool, a retrieval index or a system prompt written after the window can still know the answer. The harness's network guard closes the first, and the run is reported invalid if anything tried to leave the machine.

Live Feed: The job that writes these rows, and the stream that carries them as they arrive.
Benchmark Datasets: The frozen window with control arms. A holdout is the same shape over a later range, and it is included with the release.
Evaluation Harness: Where the cutoff verdict is written, next to the network report and the arm deltas.
Forward-only arenas: Why a live arena cannot backtest, and where a recorded forward tape sits between the two.

Post-Cutoff Holdouts