Backtesting an LLM on filings without lookahead

Every SEC title in this corpus is built the same way, as the company name from the daily index followed by the form type, so all 86 SIVB rows in the 10 March 2023 cut collapse to six distinct strings. Accession 0001193125-23-064680 on 8 March, the capital raise, and accession 0001193125-23-067777 on 10 March, the receivership, both read SVB FINANCIAL GROUP 8-K.

A model reading a cut therefore writes mostly from its own memory of the period. Our four-arm run over SVB week measured that: given only SVB FINANCIAL GROUP 8-K, the model produced a specific dollar figure for the capital raise and a description of the securities sale, neither of which was in the payload. With the issuer name withheld and every date shifted forward ten years, the same model returned a neutral view on every day of that week, receivership day included. The signal was coming from training data, and training data extends past your cut instant.

Filing text is being ingested now. Until it lands, fetch the document from source_locator yourself when the run depends on what a filing says, and treat a model’s narrative about a PIT row as unattributed to the row.

The four above are retrieval leaks, in that they govern which rows reach your model. The fifth runs through the model’s own weights: it was trained on text from after your cut instant, and cutting the retrieval side does not remove that. A prompt that fixes the date to 9 March 2023 is an instruction to the model, and the weights carry that week either way.

That channel matters more here than it usually would. The rows carry no filing text yet, so the model’s own memory is the only thing in the run carrying detail. Our control arm withholds the issuer name and shifts every date forward ten years, which is the cheapest way to measure how much of a result depends on that memory. If your numbers survive the shift, the payload was doing the work.

$ curl -s "$PIT/v1/sample/news?example=svb&as_of=2023-03-09T20:00:00Z"
$ curl -s "$PIT/v1/sample/news?example=svb&as_of=2023-03-10T23:59:59Z"
$ curl -s "$PIT/v1/sample/coverage?example=svb"

The first returns count 0 with both sources certified complete for 9 March. The second returns the 8-K on the default clock. The same instant asked with visible_by=available_at returns HTTP 409, because that clock is null on SEC rows and the strict cut refuses to guess.

• PIT closes the retrieval channel. A model’s own memory of the period stays open, and with no filing text on the rows, that memory carries most of the signal.
• status: ok with an empty coverage.missing says the days behind the cut were covered, and says nothing about what the model did with them.
• The public sample serves three fixed examples from pinned fixtures, so treat it as a demo rather than a small paid plan.
• Every certified_pit row is parsed from a stored publisher document, so nothing generated is ever served back as a certified row.
• The certified backfill ends 2023-03-31, and the live tape picks up at 2026-08-26, so 2024 and 2025 are outside the periods you can test.

The full policy is on clocks, coverage, and lookahead bias. Plans and what the corpus covers.