Ingest · data

Live Feed

A job reads the public record every fifteen minutes and stamps each new row with the minute we first saw it. A question about 14:15 is answered from a row that was written at 14:15, rather than from an archive assembled afterwards.

LIVE
SSE
REST

Included on every paid plan.

Get access API reference

The artifact

Checking the tape from outside

/v1/meta takes no key. The last partition date is whatever the tape wrote most recently, so this call is the cheapest check that the feed is running.

GET /v1/meta 200 · read 2026-08-27

$ curl -s https://api.pit.aqx.llc/v1/meta | jq '.results[0].corpus'
{
  "first_partition_date": "2008-09-01",
  "last_partition_date": "2026-08-27",
  "rows": 587748,
  "sources": [
    {
      "source_id": "sec.edgar",
      "first_partition_date": "2008-09-01",
      "last_partition_date": "2026-08-27",
      "complete_days": 204,
      "partial_days": 3,
      "missing_days": 99,
      "rows": 564518
    },
    {
      "source_id": "us.federal_register.pi",
      "first_partition_date": "2020-02-01",
      "last_partition_date": "2026-08-27",
      "complete_days": 182,
      "partial_days": 5,
      "missing_days": 91,
      "rows": 21931
    }
  ]
}

Trimmed to two of the seven sources. The tape has written forward since 2026-08-26; the 2008 end of that range is backfill.

The stream needs a key. Without one it answers 401.

GET /v1/stream · no key 401 · unauthorized

$ curl -s https://api.pit.aqx.llc/v1/stream
{"status":"error","results":[],"count":null,
 "error":{"code":"unauthorized","message":"API key missing or invalid.","param":null}}

With a key it is text/event-stream. Each frame names the event, carries the row id so a dropped connection can resume from it, and puts one news row in data.

frame shape · internal/httpapi/stream.go event: news

event: news
id: <row id, echoed back as Last-Event-ID on reconnect>
data: {"id": …, "source_id": …, "lane": "forward_first_seen",
       "available_at": …, "availability_basis": "local_first_seen", …}

:                      <- comment heartbeat, every 15 seconds

The failure it prevents

A first-seen stamp cannot be recovered later

available_at is set to the current minute only when this job fetched the bytes itself, and availability_basis records that as local_first_seen. A backfill of a 2020 filing leaves available_at null, because no poller of ours watched those bytes appear.

The stream applies the same rule at the door: a row reaches a subscriber only when it carries a real available_at and a basis other than unknown. What arrives on the feed is therefore the part of the record we watched appear, and the backfilled part stays on the other lane.

Somebody starting the same job tomorrow gets tomorrow's stamps, and no later crawl of any archive recovers when a row first became fetchable, so the August 2026 stamps exist only for whoever was polling in August 2026.

A poll we miss is written down as partial or missing coverage for that window, so a gap in the feed shows up in coverage.missing on any cut that crosses it.

How it works

One scheduled job, four writes

The tick

Cloud Scheduler runs cmd/ingest live -once as a Cloud Run job. Each tick reads the SEC current feed and the Federal Register public-inspection list, parses what came back, hashes it, appends a revision, writes the parquet partition and that day's coverage certificate, and updates the manifest. committed_at is the moment of the write. No model runs anywhere on this path.

Cadence

Production runs every fifteen minutes (local.ingest_schedule in infra/terraform/envs/prod/main.tf). The ingester's own default interval is one minute, and the production schedule has not been changed to match it. A tick used to cost 171 seconds on 2 vCPU because every publish ended with a full scan of all 415 partitions; that scan is gone, and a tick that fetches nothing now costs a bucket listing. Raising the cadence needs a fresh tick measured on the deployed image, since the headroom is a property of that binary.

Lanes

Rows the tape observed carry lane: forward_first_seen. Backfilled rows carry certified_pit. A single answer holds one lane, and a backtest that mixes them is comparing a witnessed clock against an asserted one.

The connection

GET /v1/stream filters on ticker, cik and lane. A comment heartbeat goes out every fifteen seconds so an idle proxy does not close the socket, and sending the last id you saw as Last-Event-ID resumes from there. WebSocket comes later with the same payload.

Integration

Two lines to watch it

subscribe sh

$ curl -N -H "Authorization: Bearer $PIT_API_KEY" \
       'https://api.pit.aqx.llc/v1/stream?ticker=AAPL'

Every data: line is one news row with the same fields the REST envelope returns, so a consumer written against /v1/news reads the stream without a second parser. To replay a window instead of following it, ask /v1/news/diff for the rows that became knowable between two instants.

The stream is on every paid plan. Rate limits are per plan and are listed on the pricing page.

Point-in-Time API: The query surface over the same rows, including the ones this feed wrote a minute ago.
Post-Cutoff Holdouts: What this feed becomes after a day: evaluation windows dated later than any current model's training data.
Flat Files: The partitions this job writes, downloadable with a sha256 for each one.
Status: A browser-side read of /health and /v1/meta, which is where you check the tape from outside.