The artifact

One partition, and a query over it

/v1/files lists every partition with its hash, its byte count and a time-limited signed GCS URL.

GET /v1/files?source=sec.edgar 200 · one FileObject

$ curl -s -H "Authorization: Bearer $PIT_API_KEY" \
       'https://api.pit.aqx.llc/v1/files?source=sec.edgar'
{
  "id": "sec.edgar/2023/03/10.parquet",
  "source": "sec.edgar",
  "partition": "2023-03-10",
  "bytes": 998952,
  "sha256": "95db284f431d7dccf15a9cf0539eae379d22aff5097b539a3911ce7f7d4267bd",
  "storage": "gcs",
  "gcs_uri": "gs://pit-aqx-corpus-prod/sec.edgar/2023/03/10.parquet",
  "url": "https://storage.googleapis.com/… (signed, time-limited)",
  "expires_at": "…",
  "corpus_version": "sha256:16cb590e…"
}

One entry from the listing. bytes and sha256 are this build's real values for that partition; url and expires_at are minted per request and are shown here as their shape.

Once it is on disk it is an ordinary parquet file. This is DuckDB reading the partition above with no PIT code in the process at all.

duckdb · sec.edgar/2023/03/10.parquet 3,124 rows · 998,952 bytes

$ duckdb -c "select form, count(*) as rows
             from 'sec.edgar/2023/03/10.parquet' group by 1 order by 2 desc limit 5"
┌──────────┬───────┐
│   form   │ rows  │
│ varchar  │ int64 │
├──────────┼───────┤
│ 4        │  2071 │
│ 8-K      │   383 │
│ SC 13G/A │   215 │
│ 10-K     │   151 │
│ 6-K      │   111 │
└──────────┴───────┘

$ shasum -a 256 sec.edgar/2023/03/10.parquet
95db284f431d7dccf15a9cf0539eae379d22aff5097b539a3911ce7f7d4267bd

Run on 2026-08-27 against the corpus in this build. The hash is the one the manifest and the FileObject above both carry, so the three agree or the download is wrong.

The failure it prevents

Re-running a result six months later

A result computed against a live endpoint depends on what that endpoint held the day you ran it. The corpus moves under you: the tape adds a day every day, and SEC rewrites its own index files on Saturday mornings, folding in corrections taken after acceptance. Six months later the same query can return a different answer, and nothing in the old result says which corpus produced it.

Every file here carries a sha256, and every listing carries the corpus_version the files were cut from. Pin that string in your run and the comparison holds: two runs over the same corpus_version read the same bytes, and a run over a different one is a different experiment that says so up front.

The API path has the same property in a weaker form. Reading a whole window through /v1/news means a few thousand paginated requests against a corpus that can publish underneath you, which is why a cursor is bound to corpus_version and expires when the corpus does. For a full-window read the files are the cheaper and more stable path.

How it works

Partitions, hashes, and a manifest

Layout

One parquet file per source per day, keyed <source>/<yyyy>/<mm>/<dd>.parquet. The corpus in this build is 415 such files across seven sources. The columns are the served row, field for field, so a query written against the API's JSON reads the parquet without a translation layer.

The manifest

MANIFEST.json lists every file with its sha256, bytes and row_count. Its own hash is the corpus_version that /v1/meta reports and that every API response repeats, so the same string identifies the corpus whether you queried it or downloaded it.

Delivery

url is a signed HTTPS GET against Google Cloud Storage and gcs_uri is the same object for a client that speaks gs://. There is no S3 mirror and no packed archive format. On a local backend there is no bucket, so url, gcs_uri and expires_at are JSON null and id is the locator; the server's own filesystem path is never published.

Timestamps

The four optional clocks — event_at, available_at, acceptance_at and supersedes — are the only nullable columns. Every string column is required and carries '' when it holds nothing, so where cik is null matches no rows and where cik = '' matches the ones you meant.

Access

Power and Desk plans read /v1/files. A Researcher key gets 403 with plan_required rather than a partial listing.

Integration

pandas, DuckDB, or the harness

read a window python

import duckdb

con = duckdb.connect()
con.sql("set timezone='UTC'")
con.sql("""
  select form, count(*) as rows
  from 'sec.edgar/2023/03/*.parquet'
  where published_at <= timestamp '2023-03-10 23:59:59+00'
  group by 1 order by 2 desc
""").show()

The as-of rule is the same one the API applies: a row is in the cut when the clock you chose is at or before your instant. What the files do not carry is the coverage certificate, so a query over a day the corpus never read returns zero rows instead of refusing. Read /v1/coverage alongside the files, or take the certificates that ship inside a benchmark release, where they are a line per calendar day.

pandas.read_parquet and datasets.load_dataset("parquet", …) both read these files directly. Nothing in the path needs a PIT SDK.

Point-in-Time API: The same rows through a query, with a coverage certificate and a refusal where the corpus has a hole.
Benchmark Datasets: A frozen window of these partitions with control arms, day-by-day receipts, and a content digest to pin.
Live Feed: The job that writes today's partition, every fifteen minutes.
Coverage reference: What a certificate asserts, and how a partial day differs from a missing one.

Flat Files