Run an agent with the network off

$ python3 boundary.py

leaky  served  SVB FINANCIAL GROUP 8-K  acceptance_at=2023-03-10T22:23:03Z
dump   not_yet_knowable  This row exists in the release but its clock has not passed at 2023-03-10T14:30:00Z under the dump policy.
pit    not_yet_knowable  This row exists in the release but its clock has not passed at 2023-03-10T14:30:00Z under the pit policy.

The same filing gets three different answers at the same instant. At the opening auction on 10 March 2023 the leaky policy hands an agent SVB Financial Group's receivership 8-K. EDGAR stamped that filing as accepted at 22:23:03 UTC, nearly eight hours after the bell. The other two policies refuse the row and name the clock that had not passed.

A backtest that joins filings on their index date is running the first column. This module builds the run that measures the difference and writes the measurement into a file a second reader can recompute from their own copy of the data.

What a run is

A run is a grid: every arm of the release crossed with every clock policy. For each cell the harness walks a session calendar and calls your agent once per session, with four tools bound to that session's opening bell — pit_list, pit_search, pit_read and pit_coverage. A row reaches the agent only when its clock under that cell's policy has passed. Ask for one that has not and the tool returns not_yet_knowable and the row's key, with none of the row's fields.

Midnight UTC is seven or eight in the evening in New York, so a join on the filed date acts on filings hours before they existed. 23:59:59Z is 18:59:59 in New York, after the close, so the dump arm waits for the next morning. An acceptance receipt is an intra-day timestamp, which puts the entry on the same morning when it arrives before 09:30 and on the next one when it arrives after.

Before you start

You need Python 3.11 or newer, a shell, and the release directory pit-eval-sec-2022-11_2023-03. It needs no key, no model and no market data. pit-harness has no runtime dependencies, and every arm ships as JSONL beside its parquet twin, so the whole module runs on the standard library.

$ python3.11 -m venv .venv && . .venv/bin/activate
$ pip install ./pit-harness

Successfully installed pit-harness-0.1.0

pit-harness is Apache-2.0 and is not on PyPI yet, so today it installs from a source directory. Mail operations@aqx.llc for that directory and the release together. Work from the directory that holds the release, or give an absolute path. The receipt records the basename either way, so both produce the same hashes.

Step 1 — read the release before you run against it

$ pit-harness bundle pit-eval-sec-2022-11_2023-03

{
  "arms_loaded": {
    "main": 287929,
    "redacted": 287929,
    "shifted": 287929
  },
  "builder": "cmd/bundle",
  "bundle_schema_version": 1,
  "clocks": {
    "acceptance_at": 3124,
    "available_at": 0,
    "event_at": 0
  },
  "content_digest": "sha256:bc924991d6ea64984dd533874014e83dac8a52b9bd442f08f89b60a5f055c15c",
  "corpus_version": "sha256:29336a245255fdc7be9b5b0b1a6e90053e958a78a58166dbefeaeedce05eea7a",
  "distinct_ids": 287925,
  "manifest_rows": 287929,
  "partitions": {
    "complete": 103,
    "days": 151,
    "empty": 0,
    "missing": 48,
    "partial": 0
  },
  "read_format": "jsonl",
  "release": "pit-eval-sec-2022-11_2023-03",
  "root": "pit-eval-sec-2022-11_2023-03",
  "source_id": "sec.edgar",
  "text": {
    "days_with_text": 0,
    "note": "…",
    "rows_with_text": 0,
    "status": "absent"
  },
  "window": {
    "end": "2023-03-31",
    "start": "2022-11-01"
  }
}

One value is cut: text.note, a paragraph saying the same thing status does. Everything else is verbatim.

Four numbers here set the limits of what the rest of the module can claim. available_at is zero, so no cut in this release can be bounded by proven availability. acceptance_at is 3,124 out of 287,929, and all 3,124 sit on 2023-03-10, which is why the window below ends where it does. Anywhere else in the release the pit and dump arms produce identical results. partitions.missing is 48 of 151 days, which is weekends and holidays rather than lost data. And text.status is absent: a title is the company name and the form type joined together, so a model reading these rows has almost nothing to read and will answer from memory. Control arms, in module 3, are how that last one gets measured.

content_digest is the value to pin. It covers the data files and ignores the build clock, so two builds of the same corpus give the same digest. Every hash printed further down was produced against the build with digest sha256:bc924991….

Step 2 — run it

The config is one JSON file. This one asks for the main arm only, all three clocks, 8-K filings, and the first two weeks of March 2023. The window runs to the 14th rather than the 10th so that the filings of the 10th — the day with the acceptance stamps — have a session to enter on.

{
  "bundle_path": "pit-eval-sec-2022-11_2023-03",
  "arms": ["main"],
  "clocks": ["leaky", "dump", "pit"],
  "forms": ["8-K"],
  "start": "2023-03-01",
  "end": "2023-03-14",
  "bundle_format": "jsonl",
  "session_source": "partitions",
  "probe": false
}

Two settings are worth naming. bundle_format is pinned to jsonl because the format the run read goes into the receipt, so pinning it means a reader with pyarrow installed and a reader without it get the same hashes. session_source is partitions because you have no price file: the calendar becomes the days the corpus holds rows for, and the entry rule is unchanged.

$ pit-harness run --config run.json \
    --trajectory trajectory.jsonl --receipt receipt.json

run b04a6eb930ae9c0f  valid=True  receipt sha256:07502a39f9c197e77f117f16b9ae65220d5144c77af3281c9f849a0a7be7f9b6
trajectory sha256:659cd5d3a8eaccd49f5469062fbb46294de432b31971f30bf38367daa167112b  events 98
  main      leaky  total_return      n/a  trades    0  reachable=False
  main      dump   total_return      n/a  trades    0  reachable=True
  main      pit    total_return      n/a  trades    0  reachable=True
  protocol gaps: P1_model_version, P2_training_cutoffs

The agent that just ran is stub_agent, the default. It needs no model. It acts on every filing the window has just made knowable, reads no text and forms no view, so any difference between its cells comes from the clock. Use it to check the machinery. Replacing it with your own agent is one call to from_callable, which is what the later modules do.

total_return is n/a because the harness ships no market data and fetches none. You supply the bars, and module 5 is where that happens. The trajectory, the timing distribution and the receipt are written without them.

Step 3 — read what the clocks did

The trajectory is one JSON object per line, in the order things happened, with no wall clock anywhere in it. Each prompt line records how many rows were knowable at that session's bell.

import json
from collections import defaultdict

rows = defaultdict(dict)
for line in open("trajectory.jsonl"):
    event = json.loads(line)
    if event["kind"] == "prompt":
        rows[event["session_date"]][event["clock"]] = event["payload"]["knowable_rows"]

print(f"{'session':<12}{'leaky':>8}{'dump':>8}{'pit':>8}")
for session, seen in sorted(rows.items()):
    print(f"{session:<12}{seen['leaky']:>8}{seen['dump']:>8}{seen['pit']:>8}")

$ python3 sessions.py

session        leaky    dump     pit
2023-03-01       413       0       0
2023-03-02       760     413     413
2023-03-03      1014     760     760
2023-03-06      1322    1014    1014
2023-03-07      1627    1322    1322
2023-03-08      1866    1627    1627
2023-03-09      2234    1866    1866
2023-03-10      2617    2234    2308
2023-03-13      3059    2617    2617
2023-03-14      3357    3059    3059

On the first session the leaky column already holds 413 filings that nobody outside the filer had seen at 09:30. It stays exactly one session ahead for the whole window, which is the size of the error a date join introduces.

2023-03-10 is the one day in this release that carries acceptance receipts, and it is the only session where pit and dump disagree: 2,308 rows against 2,234. The 74 in between are filings EDGAR accepted before the opening bell that morning, which a day-end bound would make you wait another session for. Acting on them early is backed by evidence rather than by an assumption, because the receipt puts the filing in EDGAR's hands before the auction.

The receipt carries the same gap as an aggregate.

import json

timing = json.load(open("receipt.json"))["body"]["metrics"]["timing_distribution"]
for field in ("events", "acceptance_stamped", "leaky_entries_before_acceptance",
              "mean_leaky_lookahead_hours", "max_leaky_lookahead_hours"):
    print(f"{field:<34}{timing[field]}")
ahead = {bucket["value"]: bucket["count"] for bucket in timing["sessions_pit_ahead_of_dump"]}
print(f"{'events pit gained a session on':<34}{ahead.get(1, 0)}")

$ python3 timing.py

events                            3059
acceptance_stamped                383
leaky_entries_before_acceptance   309
mean_leaky_lookahead_hours        6.6616
max_leaky_lookahead_hours         8.0267
events pit gained a session on    74

383 of these 8-Ks carry an acceptance receipt. On 309 of them the leaky clock entered before that receipt existed, by a mean of 6.66 hours and at worst 8.03, hours during which the filing was not yet public and the position was already open.

Step 4 — verify the receipt

The receipt is one JSON file with a body and a hash over that body. emitted_at is the only wall-clock field and it sits outside the hash, so two runs of the same config over the same release produce byte-identical bodies.

$ pit-harness verify receipt.json \
    --trajectory trajectory.jsonl --bundle pit-eval-sec-2022-11_2023-03

body_sha256 ok  sha256:07502a39f9c197e77f117f16b9ae65220d5144c77af3281c9f849a0a7be7f9b6
run_valid True  protocol_complete False
trajectory ok  sha256:659cd5d3a8eaccd49f5469062fbb46294de432b31971f30bf38367daa167112b
bundle content_digest ok  sha256:bc924991d6ea64984dd533874014e83dac8a52b9bd442f08f89b60a5f055c15c

Three of those lines are recomputations: the body against its own hash, the trajectory file's bytes against the hash the receipt bound, and the release's digest against the one the run read. All three run on your machine against files you already hold, so the check does not depend on anything we serve.

Run the whole thing a second time into a fresh directory and both hashes come back identical. Object keys sort at every level, floats are fixed at six decimals, and nothing derived from the clock on the wall goes inside the body.

Step 5 — make the guard fire

While a run is active the harness replaces the socket entry points with ones that raise: INET socket creation, connect, create_connection, DNS resolution and TLS wrapping. The guard takes no allowlist and no override flag, so the way to watch it work is to hand it an agent that tries.

"""An agent that reaches for the internet in the middle of a scored run."""

from urllib.request import urlopen

from pit_harness.adapters.plain import from_callable


def _look_it_up(prompt, tools):
    rows = tools.pit_list(limit=5)["rows"]
    urlopen("https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany")
    return [r["row_key"] for r in rows]


trader = from_callable(_look_it_up)

Copy run.json to run-leaky.json and add "on_egress": "continue", which records each attempt and carries on instead of stopping at the first one. Either setting marks the run invalid, and this one leaves you a receipt to read.

$ PYTHONPATH=. pit-harness run --config run-leaky.json \
    --agent leaky_agent:trader --trajectory leaky.jsonl --receipt leaky-receipt.json

run 14f160d394e01c21  valid=False  receipt sha256:d9f6d004c5188f50dab2ed9a52ae721f4442a2dea180e6dfcef6f4630591035e
trajectory sha256:c6acd2155c4494df0418ac6e8c6a593e3f54965ff215cc1f33bbb96c2f9380b3  events 128
  main      leaky  total_return      n/a  trades    0  reachable=False
  main      dump   total_return      n/a  trades    0  reachable=True
  main      pit    total_return      n/a  trades    0  reachable=True
  protocol gaps: P1_model_version, P2_training_cutoffs
  INVALID: network egress attempted while the guard was armed; the agent attempted network egress

The receipt's network block holds egress_attempts: 30 — one per decision point — with the call site of each, and invalidated_by names both the guard and the agent. The trajectory carries an egress_blocked line where each one happened.

Exercise — close the protocol gaps

Every run so far has ended on the same two names: P1_model_version and P2_training_cutoffs. The receipt fills six protocol slots, and four of them it fills from the run itself. These two are declarations you make, and leaving them empty lists them in protocol_gaps rather than filling them with a guess.

Add a model block to run.json and declare them. Use the model you would actually run here — for the stub agent, declare the stub and your own environment's cutoff.

  "model": {
    "name": "stub-agent",
    "version": "pit-harness-0.1.0",
    "provider": "none",
    "training_cutoff": "2026-05",
    "declared_by": "you@example.com"
  }

$ pit-harness run --config run-declared.json \
    --trajectory declared.jsonl --receipt declared-receipt.json

run 21b2bf285801a658  valid=True  receipt sha256:ba408bbacabec74ae568bbacb63fd7c9390f6375704609f706e0e588ac907422
trajectory sha256:72aa246b1845410b29bbfa66c42088b420c3ecf2ead418a9fec249fb997e1a9a  events 98

The gaps line is gone. Read what filling them in produced.

{
  "evaluation_window": {
    "end": "2023-03-31",
    "start": "2022-11-01"
  },
  "note": "window_ends_before_cutoff means the main arm is inside the model's training data and its score is not evidence of forecasting",
  "training_cutoff": "2026-05",
  "verdict": "window_ends_before_cutoff"
}

Declaring the model changed nothing about what the agent did. It surfaced a fact that was already true of the run: the window ends in March 2023 and the declared cutoff is May 2026, so the model had three years to read about this period before the run started. The guard closed the retrieval channel; this verdict describes the other one. Modules 3 and 5 measure it, with control arms and the recoverability probe.

Check your work

$ pit-harness verify declared-receipt.json \
    --trajectory declared.jsonl --bundle pit-eval-sec-2022-11_2023-03

body_sha256 ok  sha256:ba408bbacabec74ae568bbacb63fd7c9390f6375704609f706e0e588ac907422
run_valid True  protocol_complete True
trajectory ok  sha256:72aa246b1845410b29bbfa66c42088b420c3ecf2ead418a9fec249fb997e1a9a
bundle content_digest ok  sha256:bc924991d6ea64984dd533874014e83dac8a52b9bd442f08f89b60a5f055c15c

Three oks, run_valid True and protocol_complete True, which is the end of the module.

Your body_sha256 will match the value above if your release carries content digest sha256:bc924991… and you used these filenames. A rebuild that adds filing text keeps the release name and changes the digest, and every hash on this page moves with it. The three oks hold either way, because each one recomputes a hash from files you have rather than comparing against a value we publish.

What this run does not prove

Where the lead artifact came from

The three-clock output at the top of this page runs against the same release, using the row key of SVB's receivership 8-K, accession 0001193125-23-067777.

from datetime import datetime

from pit_harness import Bundle, WindowWorld

SVB = "row_e0f932ec9387facd"        # SVB Financial Group 8-K, accession 0001193125-23-067777
BELL = datetime.fromisoformat("2023-03-10T14:30:00+00:00")   # 09:30 in New York, 10 March

bundle = Bundle.load("pit-eval-sec-2022-11_2023-03", arms=["main"], forms=["8-K"],
                     start="2023-03-01", end="2023-03-14", format="jsonl")

for clock in ("leaky", "dump", "pit"):
    tools = WindowWorld(bundle.rows("main"), clock=clock).toolbox(BELL, session_date="2023-03-10")
    answer = tools.pit_read(SVB)
    if "row" in answer:
        print(f"{clock:<6} served  {answer['row']['title']}  acceptance_at={answer['row']['acceptance_at']}")
    else:
        print(f"{clock:<6} {answer['error']}  {answer['detail']}")

Ask the pit world again at the next session's bell, 2023-03-13T13:30:00+00:00, and the row comes back with its accession, its acceptance stamp and text: ''. The bell moved an hour in UTC between those two dates because US daylight saving started on 12 March, which the calendar resolves through America/New_York rather than a fixed offset.

Next

Module 2 covers the five timestamp fields behind these three policies, which three of them a query can be bounded by, and what the API does when a day has no certificate. It has no date yet. Timestamps and coverage are the reference pages under it, and both are written and live today.