Measure the claim you actually made.

Committed instruction-context footprint

Using tiktoken 0.13.0 with the o200k_base encoding, the published v3.6.0 measurement records:

Comet avoids 5,501 tokens (53.8%) compared with eager full-bundle loading. Comet loads the core skill and mode/risk reference only; it does not load work templates or routing/preflight instructions.

These are static instruction-context measurements. They support progressive context loading; they are not task quality, latency, price, or total-run tokens, and they do not prove that every multi-agent run uses fewer total tokens than a single Sol run.

The committed JSON evidence includes hashes, byte, word, and token counts. The stable historical keys remain intentional even though the public mode names changed.

Public evidence status

The current repository verification passed 100+ automated tests and package verification. That validates behavior and contracts; it does not prove Astral beats single-Sol or establish a valid outcome comparison.

The first end-to-end pilot remains invalid exploratory evidence because a fresh review found protocol defects. The invalid-pilot disclosure preserves raw artifacts for audit but not comparison. No valid outcome comparison exists. Astral does not publish outcome, token, time, or quality numbers from that pilot.

Local outcome scorecard

The standard-library scorecard does not call Codex, send data anywhere, or manufacture a result. Run the same frozen cases under a single-Sol control and Astral, record at least two JSONL trials per strategy and case, then run:

python3 plugins/astral-orchestrator/scripts/benchmark-scorecard.py benchmarks/trials.jsonl
python3 plugins/astral-orchestrator/scripts/benchmark-scorecard.py --format json benchmarks/trials.jsonl

The scorecard requires the same case fingerprint, matching repetitions, identical acceptance checks, and fixed observed route role/model/effort within each strategy. Strategies may intentionally differ. It reports success, first-pass acceptance, rework, route correctness, wall time, and model calls; tokens and a 0–100 blind quality score are optional.

Read the benchmark guide and JSONL schema before collecting trials.

Interpret results conservatively

The output describes the recorded cases and settings. It cannot prove Astral is generally faster, better, or cheaper, establish causation from a small sample, or repair an unequal comparison. A route-correctness warning, non-blind quality score, wrong route, unequal checks, or small and unrepresentative case set is a reason to investigate before making a product claim.

Tested workflow contracts include exact pinned routes, objective checks plus fresh review, and the local privacy/no-analytics runtime posture—not outcome superiority.

Method attribution

The reproducible pinned/state/evidence method was inspired by OpenRouter's Ori Eval and its spawn-ori-eval skill. Astral does not run Ori or OpenRouter and has no OpenRouter runtime or API dependency.