Test · evaluation
A frozen evaluation window shipped as files, with matched control arms: the rows as served, the same rows with the issuer removed, and the same rows with every date moved ten years forward. Whatever score survives the shifted arm came from the rows.
Licensed apart from the plans, so it starts with an email rather than a checkout. The text-complete edition is days away.
The artifact
pit-harness bundle reads a release and prints what it
is. Every field below is measured from the files in the release.
{
"release": "pit-eval-sec-2022-11_2023-03",
"source_id": "sec.edgar",
"window": {"start": "2022-11-01", "end": "2023-03-31"},
"content_digest": "sha256:bc924991d6ea64984dd533874014e83dac8a52b9bd442f08f89b60a5f055c15c",
"corpus_version": "sha256:29336a245255fdc7be9b5b0b1a6e90053e958a78a58166dbefeaeedce05eea7a",
"manifest_rows": 287929,
"distinct_ids": 287925,
"arms_loaded": {"main": 287929, "redacted": 287929, "shifted": 287929},
"partitions": {"days": 151, "complete": 103, "partial": 0, "missing": 48, "empty": 0},
"clocks": {"event_at": 0, "available_at": 0, "acceptance_at": 3124},
"text": {"status": "absent", "rows_with_text": 0, "days_with_text": 0}
}
Read on 2026-08-27. Trimmed; the note under text is quoted in the limits below.
The three arms hold the same rows in the same order, joined on
row_key. This is DuckDB reading two of them side by side,
with no PIT code in the process:
┌────────────┬──────────────┬──────────────────────────────┬───────────────────────┐ │ main_date │ shifted_date │ main_title │ shifted_title │ ├────────────┼──────────────┼──────────────────────────────┼───────────────────────┤ │ 2023-03-10 │ 2033-03-10 │ ALEXANDERS INC 8-K │ [issuer redacted] 8-K │ │ 2023-03-10 │ 2033-03-10 │ SKYWORKS SOLUTIONS, INC. 8-K │ [issuer redacted] 8-K │ │ 2023-03-10 │ 2033-03-10 │ BALL Corp 8-K │ [issuer redacted] 8-K │ └────────────┴──────────────┴──────────────────────────────┴───────────────────────┘
Ten years forward preserves month, day and time of day, and moves the weekday. Ten is chosen so the window lands past today.
Alongside the arms, receipts.jsonl carries one line per
calendar day in the window, holes included — 151 lines for this
release. Each line holds that day's coverage certificate, the parquet
partition it came from, and the publisher bytes we fetched.
{
"source_id": "sec.edgar",
"partition_date": "2022-11-01",
"status": "complete",
"row_count": 2128,
"scope": {"scope_id": "sec.edgar.forms.v1", "selection": "form_allowlist",
"forms": ["10-K", "10-Q", "4", "8-K", "SC 13D", …],
"scope_sha256": "d17340d6247e3e26ea015c6bcbf5ad5a6f5fa8881f689508455e0b9985e5e2af"},
"partition": {"path": "sec.edgar/2022/11/01.parquet",
"sha256": "77d950955af9e013b804c6292321a69e3b6bb438f4b262fb67cd5628427e2148",
"bytes": 664337, "row_count": 2128},
"publisher": [{"artifact": "master.idx",
"url": "https://www.sec.gov/Archives/edgar/daily-index/2022/QTR4/master.20221101.idx",
"sha256": "2ab0092d…", "bytes": 462090,
"received_at": "2026-08-25T12:22:18Z"}]
}
The publisher receipt is the line that cannot be rebuilt later. SEC rewrites its index files on Saturday mornings and folds post-acceptance corrections into them, so the file at that URL is not necessarily the file that was there when it was fetched.
The failure it prevents
Run a model over 2023 filings and it does well. The question the number cannot answer on its own is whether it did well because it read the rows or because it read about March 2023 during training. A single-arm benchmark has no way to tell those apart, and the usual response is a paragraph of caveats.
Three arms turn the question into a measurement. The redacted arm keeps every clock and removes issuer identity, so a model that still names the event is working from the date. The shifted arm moves the dates too, so a model that still scores is working from something in the rows. Subtracting a control arm's return from the main arm's, per clock, gives the contamination delta: the part of the score the control could not reproduce.
For scale, the same rows under three clock policies over 88 8-K filings from twelve issuers: +5.13% when the joins hand over tomorrow's filings, −2.02% when they wait for the nightly dump, and +0.18% on the acceptance receipt. 47 of the 64 filings that carry a receipt were bought at an opening auction that had happened before EDGAR accepted them.
The basket rule, the two excluded tickers and the per-event timing are on benchmarks.
How it works
2022-11-01 → 2023-03-31
287,929 rows of sec.edgar across 151 calendar days,
103 of them complete and 48 with no filings to read. 287,925
distinct ids; the four repeats are known duplicates in EDGAR's own
index.
Release name is stable across rebuilds. A different window is a different release.
209 MB parquet · 579 MB JSONL
Each arm ships twice, as parquet and as JSONL holding the same rows
in the same order with the same fields. The JSONL path is stdlib
only, so a machine with no pyarrow still runs a whole
evaluation.
Plus MANIFEST.json, SCHEMA.json, receipts.jsonl and a generated README.
corpus_version is the hash of the served corpus manifest
the rows were cut from, and /v1/meta returns the same
string while that corpus is current.
content_digest is a hash over every data file's path and
hash, sorted, which ignores the build clock and the README. Two
builds of the same corpus produce the same
content_digest, which is why it is the value a run
receipt pins.
accession, cik,
content_sha256 and source_locator become
empty strings; title becomes
[issuer redacted] <form>; id becomes
the pseudonymous row_key. Every clock keeps its real
value in the redacted arm, and only the shifted arm moves them.
Only the four optional clocks and supersedes are
nullable. Every string column is required and carries
'' when empty, so where cik is null matches
nothing in the redacted arm and where cik = '' matches
all of it. JSONL follows the same rule.
pandas.read_parquet,
datasets.load_dataset("parquet", …), DuckDB over the
file path, Inspect through json_dataset(), promptfoo
through file://data/main.jsonl. No SDK, no key, no
network.
Integration
No SDK, no key, no network. The files are ordinary parquet and ordinary JSONL.
import duckdb
con = duckdb.connect()
con.sql("set timezone='UTC'")
con.sql(
"select m.partition_date, s.partition_date, m.title, s.title "
"from 'data/main.parquet' m "
"join 'data/shifted.parquet' s using (row_key) "
"where m.form = '8-K' and m.partition_date = '2023-03-10'"
).show()
# or, with no pyarrow installed at all:
import pandas as pd
df = pd.read_json("data/main.jsonl", lines=True)
To score an agent rather than to read rows, hand the release to the
harness and it pins the content_digest into the receipt
for you:
$ pit-harness run --bundle pit-eval-sec-2022-11_2023-03 \
--agent mypkg.agents:trader --receipt receipt.json
Inspect reads the JSONL through json_dataset(),
promptfoo through file://data/main.jsonl, and
lm-evaluation-harness through a datasets loader.
Limits
MANIFEST.json says "text": {"status": "absent"} and the title is a mechanical company-and-form concatenation of the index line. A language model has almost nothing to read here and will answer from its own memory of the period, so on this edition the contamination delta measures memory of a date and a name. The rebuild that adds text is one command and keeps the release name; content_digest changes.acceptance_at. On a window with none, the PIT and dump clocks are identical by construction, and the harness counts the fallback so a run cannot hide it.sec.edgar only. A different window or a different source is a different release.Related
content_digest, and writes a receipt.