Monthly Trust Report #1 · 2026-09

Every number is machine-recomputed or signature-verifiable; where we could not verify, we say so. The markdown original of this report is signed — docs/reports/TRUST_REPORT_2026-09_EN.md(.sig) in our repository.

1 · Who we are

Nautilus Assay independently recomputes published AI-agent performance claims from public artifacts, issues three-state verdicts (agree / disagree / not_computable), and signs every receipt with ed25519. Protocol published CC-BY — anyone can implement it or verify our receipts without trusting us. Operator: 伊洛科技有限公司.

2 · External claims verified this month

Target	Claim	Recompute	Verdict
Aider — SWE-bench Lite	26.3%	79/300, subset relations verified	AGREE
Terminal-Bench v1 — OpenHands	41.25% ± 0.69pp	165/400 = 0.4125 exact	AGREE
NanoJev (embodied)	6 claims	incl. trajectory replay	AGREE

Disclosures: Terminal-Bench's leaderboard excludes 5 incomplete trials from the denominator (defensible, undocumented until now); the OpenHands row was self-flagged verified: false — third-party recompute is what such flags await.

3 · Our own house

verdict-bus pilot: 10/10 NOT recomputable — inputs only in session logs, producer-set flags, zero token counts. This started our criteria catalog.
Judge incident: 47/47 self-inconsistent → batch voided.
First-party exam trend: 1/5 → 2/5 → 3/5 → 3.5/5 → 3.5/5, published at measurement time, failures included. Latest datapoint a disagree (wiring-class failure; directional hints only).

4 · Criteria catalog (open)

15 registered criteria, ratcheted (tighten-only with evidence chains): verdict recomputability, external-verified provenance, four adversarial gaming classes, paradigm corroboration chains, and calibration-claim verification — calibration claims of probability-output controllers are safety claims, recomputable on public holdouts.

5 · Tooling released (free layer, permanent)

assay-verify on PyPI — keygen/attest/verify in three lines, zero dependencies, with a built-in MCP stdio server.
VerifyPack v0.3 — sealed evidence packs, episode/trajectory checks (frame-level invariants), receipts with subject fingerprints and TTL: expired = UNVERIFIABLE until re-verified. Continuous trust, like certificate renewal.

6 · BC1 cohort preview

30 items · 4 dimensions · public/holdout 18/12 · seed committed before generation. Release 2026-09-27 on the wall.

7 · Liability

Misjudgment = OVERTURNED verdict of the same rigor + refund + challenge costs + track-record reset; 90-day window; anonymous verifications treated as UNVERIFIABLE (protocol §5b).

8 · Get verified (free)

Open a recompute request (EN/中文). Self-attestation free forever; independent verification is what we sell.

Signing pubkey f7554b8709b7fe36f5a63e7f76cf2a31f827aee5724ff8dd4f11772d1aa3e8be · Next report: 2026-10 · ← The Wall