Measured results
Benchmarks
What we have measured, with the inputs published beside it. One ablation puts a number on a clock error, and one method separates a score that came from the rows from a score that came from the model's memory.
Every figure on this page is recomputable from
ablation.json, which the run
commits and go run ./cmd/ablation regenerates. The
full method, with both worked filings and the exclusions, is on
the methods page.
- 88 8-K filings
- 12 issuers
- 105 sessions
- 2022-11-01 → 2023-03-31
The clock ablation
One strategy, one basket, three clocks
Every arm reads the same corpus rows, the same twelve issuers and the same daily bars. One rule turns a visibility instant into a trade: enter at the first opening auction at or after that instant, hold five sessions, exit at the close of the fifth. The arms differ only in which instant they are handed.
| Arm | Instant it treats the filing as knowable | First auction it can reach | Reachable |
|---|---|---|---|
leaky |
partition date at 00:00:00Z, which is what a join on the filed date anchors to |
the filing's own index day | no |
dump |
published_at, the day-end bound at 23:59:59Z |
the next session | yes |
pit |
acceptance_at where the tape has one, published_at otherwise |
whichever session follows the receipt | yes |
| Arm | Total return | Max drawdown | Trades |
|---|---|---|---|
leaky | +5.13% | −13.83% | 88 |
dump | −2.02% | −17.58% | 88 |
pit | +0.18% | −16.91% | 88 |
+4.95
points, leaky minus pit. The file calls it lookahead_inflation.
+7.15
points, leaky minus dump.
+2.20
points, pit minus dump, over the 17 filings where the two arms could differ at all.
Corpus window 2022-11-01 to 2023-03-31, 287,929 rows scanned, 88 tradable 8-K events from 12 issuers, 105 sessions on the curve from 2022-10-31, prices pulled 2026-08-27. Absolute returns belong to this basket and this window. The measurement is the size of a timing error, not a return anyone could have earned, and nothing here is advice.
Reading the numbers
The two gaps rest on different kinds of evidence
The leaky gap rests on mechanics
Midnight UTC is 19:00 or 20:00 the previous evening in New York, so the leaky arm's instant always precedes the same calendar day's bell and it buys that morning. It enters exactly one session ahead of the dump arm on all 88 filings. On 47 of the 64 filings that carry an acceptance receipt, its entry auction opened before EDGAR had accepted the filing, by a mean of 6.05 hours and a maximum of 7.88 hours.
One row in full: 0000066740-22-000080,
3M, with an ACCEPTANCE-DATETIME of 2022-11-14 17:22:43
ET. The leaky arm bought at the 2022-11-14 opening auction, 09:30
ET, on a filing EDGAR had not yet received. What five months of
that head start is worth in points depends on the basket and the
window, but the head start itself is on every filing in the sample
rather than on a lucky subset.
The +2.20 points rests on 17 trades
The PIT and dump arms enter on a different session for 17 of the 88 filings, and only those 17 can differ at all. Across them the per-trade difference averages +0.95% with a standard deviation of 2.92%, and 9 of the 17 go the PIT arm's way. The mean sits about 1.3 standard errors from zero (t = 1.34 on 17 paired trades), so the sample sizes the effect and leaves its sign open. The +2.20 points describes this basket over this window.
64 of the 88 events carry an acceptance stamp and 24 fall back to the day-end bound. The acceptance backfill is still stamping rows, so the PIT arm's entries move earlier as it runs and the two reachable arms will separate further. That shows up as a real diff in the committed file.
What the basket is, and why it cannot pick its own result
The universe is the Dow Jones Industrial Average as constituted for the whole window: it took its 2020-08-31 form and held it until 2024-02-26, so one membership list covers the window with no substitution. Of those thirty, keep the members the price vendor still carries, rank them by how many 8-Ks they filed inside the window, and take the top twelve, breaking ties on ticker. The rank is computed from filing counts, which are independent of every price in the run.
The twelve are AMGN, AXP, BA, CRM, CSCO, GS, HON, JPM, MMM, PG, V and WMT. The basket is identical in all three arms, so it cannot produce the difference between them. It does produce the absolute level of all three, which is why the absolute numbers are not the claim.
An event any arm cannot fill is dropped from all three and recorded
in exclusions, so the arms are always scored on the
same event list. SIVB and WBA are excluded because the price source
carries no bars for them. Prices are Yahoo Finance daily bars,
fetched once at build time and cached outside the repository; only
derived values reach the published file, so no OHLC row is
redistributed.
The control-arm method
Which part of a score came from the rows
The ablation above puts a number on a clock error. The method below separates the part of a score that came from the rows from the part that came from the model already knowing the period. A benchmark release ships the same window three times and a run scores all three, so the arms differ in what the agent can recognise rather than in how much it can read.
main
The rows as served
The window exactly as the record holds it. This is the score everyone quotes.
redacted
Issuers removed
The same rows with the identity taken out. A score that survives came from the content rather than the name.
shifted
Dates moved ten years forward
The same rows relocated in time. A score that survives came from the rows rather than from remembering the period.
The contamination delta, and the probe that qualifies it
The delta is the main arm's return minus each control arm's, per clock. A large positive delta is the part of the main arm's score that the control arm could not reproduce, which is the part that depended on identity or on the date rather than on the rows.
A shifted arm is a control only while the mask holds, and naive
masking is breakable (arXiv 2504.14765). An agent that reconstructs
“this is really March 2023, and this is SVB” from
wording, a brand name the scrubber missed, or plain memory scores
the shifted arm on the same knowledge as the main arm, and the
delta collapses toward zero for a reason that has nothing to do
with the agent being honest. The harness therefore puts a
date-recoverability question to the same agent on the shifted arm,
against rows it was actually served, and scores the free-text
answer against the main arm. The verdict is
mask_holds while the recovery rate stays at or below
the threshold, and it sits in the receipt next to the delta it
qualifies.
The probe reads the agent's answer text, so a model that recovers the date and says nothing about it passes. That is a limit of the check, and it is why the reasoning audit reads the written rationale as well.
The network is off, and the receipt says so
While a run is active the harness replaces the socket entry points
with ones that raise: INET socket creation, connect,
create_connection, DNS resolution and TLS wrapping.
Unix sockets keep working, because they cannot leave the machine.
The guard takes no allowlist and no override flag. Every attempt is
recorded with the call site that made it, and the receipt reports
the run invalid.
The guard works inside the harness process, where it stops a tool,
an SDK or a stray urlopen. It does not follow a
subprocess the agent spawns, and it has no effect on a provider
that already holds the answer in its weights. Run the harness
inside a network-namespace jail for the first, and read the delta
and the probe for the second.
What a run receipt records
| Slot | Holds | Declared or measured |
|---|---|---|
P1_model_version | name, version, provider, fingerprint, temperature, parameters | declared |
P2_training_cutoffs | the declared cutoff against the release window, with a verdict either way | declared, checked |
P3_retrieval_timestamps | tool-call count, decision points, first and last known_at, the clock policies | measured |
P4_costs_and_frictions | the declared frictions, the network statement, the price provenance | declared, plus measured |
P5_window_provenance | the release's content digest, corpus version, window, row and partition counts, clock coverage | measured |
P6_disaggregation | every arm-by-clock cell, the contamination delta, the timing aggregates | measured |
Nothing is averaged across cells. The receipt reports every arm-by-clock cell separately, and a contamination certificate restates those cells rather than recomputing them.
Run it yourself
The inputs, and the two products that produce them
Benchmark Datasets
The frozen eval windows with the three arms in them, plus a manifest carrying a sha256 for every file and the corpus version the release was cut from.
Evaluation Harness
The open-source runner that scores an agent against one of those windows with the network off, and writes the receipt the numbers above come out of.
$ go run ./cmd/ablation # fetch what is missing, write the JSON $ go run ./cmd/ablation -offline # cache only, no network
The output file is committed, so its diff is empty when nothing changed. Object keys sort at every level, arrays sort on an explicit key, floats are fixed-width, the pull date comes from the price cache rather than the wall clock, and no field records when the file was generated. Two inputs still move underneath it: the acceptance backfill stamps rows as it runs, and a price vendor can revise a bar. Both show up as a real diff, which is the intent.