scout

Evaluations

Scout’s Phase 1 evaluation system lives in src/scout/evals/phase1/. It runs the checked-in YAML corpus through the production relevance, drafting, critic, and dossier-safety paths without granting write access to a dossier source.

Production evidence

The Phase 1 exit audit accepts fixed UTC half-open boundaries and emits a schema-v3 canonical JSON report plus Markdown rendered from that same object. Only production and live scans qualify. Historical drafts are replayed through the pure content verifier, so current author-rate rows cannot alter a historical outcome. Approval references and tamper-evident bundles contain no deployment-specific procedure.

Corpus execution

scout.evals.phase1.loader.load_phase1_corpus validates case shape, distribution, the pinned dossier, stored grades, and unique IDs before constructing an immutable Phase1Registry.

ScoutPipelineAdapter runs scout_response cases through the real pipeline. It sends dossier_bad_write cases to a pure read-only rail evaluator, which reports automated_write_forbidden and never performs the proposed operation.

ScoutPhase1Grader scores three binary dimensions:

• outcome_semantics checks the exact terminal status and applicable posture.
• content_safety reruns draft verification or checks the expected rail violations.
• grade_contract validates the stored grade against the shared schema.

The export adapter projects eligible production grades into the same Jig case shape and a seven-dimension human score vector. It is pure and read-only; Scout’s tables remain the mutable authority.

Hermetic and live runs

CI scripts all inner model phases and uses generalized fixture dossiers in a temporary git repository. It makes no paid model calls. A seeded regression confirms that Jig’s regression detection notices a known score drop.

A live model sweep is operator-only and never runs in CI:

uv run scout eval phase1 \
  --configs eval-phase1-sweep.yaml \
  --dossier-root /path/to/dossier-source \
  --output phase1_sweep_result.json

Example configuration:

concurrency: 4
seeds: 1
threshold: 0.05
baseline:
  name: baseline
  relevance_model: claude-sonnet-4-6
  reply_draft_model: claude-sonnet-4-6
  critic_model: claude-sonnet-4-6
candidates:
  - name: candidate-haiku-draft
    relevance_model: claude-sonnet-4-6
    reply_draft_model: claude-haiku-4-6
    critic_model: claude-sonnet-4-6

The command prints rollups and regression alerts, optionally writes JSON, and exits nonzero when a run fails or a regression crosses the configured threshold.

For replaying recorded production phases against candidate prompts or models, see Offline replay.

Typesafe shadow visibility

When schema v47 shadow data is available, each review evaluation shows the latest shadow relevance verdict beside the author-class badge. The badge is advisory: it includes eligibility, uncertainty, the stored reason, and the band distribution from the decision details without affecting review state. Older databases continue to render without the badge.

Operators can compare the latest shadow verdict per evaluation with the LLM relevance result and any finalized human grade:

uv run scout typesafe report --since 2026-09-01T00:00:00Z
uv run scout typesafe report --scan-id 42 --json

Human agreement includes only the latest grade revision when both it and the current grade row are finalized under the current schema; an invalidating revision does not fall back to an older label. --since is an inclusive instant cutoff and requires an explicit timezone (Z or a numeric offset). Equivalent offsets are normalized to UTC.

The summary includes agreement counts and an offline replay of the checked-in placeholder acceptance fixture, making catalogue or decision-mapping drift visible. If shadow_relevance_runs is absent, the command explains that the database predates schema v47 and exits successfully.

The review badge and this report are observational views of the same stored shadow rows; neither gates an evaluation. A train-only fitted report can be produced with scout typesafe fit once a task pins a FrozenPartition. Its September evidence directory contains the predeclared PLAN.md, observed RESULTS.md, a per-evaluation inventory.json, checksums.json, and the versioned weight-set.json. The current pinned September agent-ops task has partition all and no digest, so real-row fitting remains deliberately blocked; placeholder-backend rows over a synthesized partition exercise the command.

This site is open source. Improve this page.