Audit

Reasoning audits

Every specific in the agent's written rationale, checked against the rows that run actually served. A figure, a name, a date or an event that appears in none of them is quoted back with its position and scored.

BUILDING The package is pit-audit, Apache-2.0, Python 3.11 or newer, with no runtime dependencies. It installs from the repository and runs offline; metered pricing per decision audited is not live yet.

• pip
• pit-audit run
• CLI · Python
• exit 1 on unsupported

unsupported: 9 flagged of 12 claims over 5 rationales (5 at or above entity), worst number
  2023-03-08 main/pit number[number] unmatched '$2.25B'
  2023-03-08 main/pit number[number] unmatched '$2.25 billion'
  2023-03-08 main/pit event[entity] unmatched 'Form 8-K (2023-03-08) announcing large …'
  2023-03-07 main/pit vague[vague] unmatched 'The recent 10-K highlights mounting …'
  cell main/pit: unsupported, 6 flagged, worst number
  cell redacted/pit: grounded, 1 flagged, worst vague
  cell shifted/pit: grounded, 2 flagged, worst vague

The finding is the part that makes the verdict checkable rather than asserted. It reports how many served values were considered, names the nearest one, and gives the gap:

{
  "claim_id": "940eadc8ea9c109c",
  "context": "Form 8-K (2023-03-08) announcing large realized portfolio losses and an urgent $2.25B capital raise",
  "decision": {
    "arm": "main",
    "clock": "pit",
    "known_at": "2023-03-08T23:59:59Z",
    "seq": 9,
    "session_date": "2023-03-08"
  },
  "detail": {
    "quantity": {
      "currency": "USD",
      "kind": "money",
      "raw": "2.25",
      "scale": "b",
      "value": 2250000000.0
    }
  },
  "end": 85,
  "extractor": "rules",
  "field": "meta.key_signal",
  "flagged": true,
  "kind": "number",
  "match": {
    "candidates_considered": 966,
    "claimed_value": 2250000000.0,
    "matched": false,
    "nearest_served": {
      "absolute_gap": 405447766.0,
      "raw": "1844552234",
      "relative_gap": 0.180199,
      "value": 1844552234.0
    },
    "rung": "none"
  },
  "quote": "$2.25B",
  "severity": "number",
  "start": 79,
  "unmatched": true
}

966 candidate values were considered for that cell. The closest was 1,844,552,234, a magnitude derived from an accession number, 18.0% away and outside the loosest 5% rung. The redacted and shifted arms read the same 86 rows with the issuer removed and with every date moved ten years forward, and neither produced an unmatched number or entity; their findings are all at vague.

$ cd audit && uv venv && uv pip install -e '.[dev]'
$ pit-audit run --trajectory trajectory.jsonl \
    --out findings.json --certificate audit.json
$ pit-audit calibrate          # grade the extractor on nine findings a person made

--certificate audit.json writes the document that fills the reasoning-audit slot in a contamination certificate. pit_harness.certify owns that block and recomputes its counts from the findings rather than copying them, so a certificate cannot state a total its own findings do not support.

Extraction can be model-assisted, and matching never is. A model chooses spans; typing, matching and severity are computed here. Every returned quote must be a verbatim substring of the rationale or it is dropped and counted, and a provider error falls back to the rule extractor and is counted too.

What it measures, and what it does not

• A cleared rung means the run served a close-enough value. Whether that row was right, and whether the agent read it correctly, are outside what this checks.
• Vague findings are noisy by construction. An agent stating what it cannot see scores badly against what it was shown, so those findings are reported and left below the default threshold rather than dropped.
• A run with no written rationale cannot be audited. The report says so in scope.decisions_with_rationale rather than reading as clean.
• Contamination that never reached the written text does not show up here. The control arms and the positive-control probe cover that.