Results

SAGA has working source-change, blocked-claim and calibration examples. The evidence below supports a prototype; human usefulness and savings remain unmeasured.

Where AI helps

The classifier labelled categories correctly on 73/79 records (92.41%), versus 44/79 (55.70%) for the keyword baseline on the same AI-labelled validation set. That is evidence for category classification on this dataset. Structured source-change detection tied its rules-only comparator at 4/4. Neither result establishes production effectiveness.

The busy replay's 52 inputs and 11 review records demonstrate the workflow. They are curated counts, not a measurement of autonomous detection, recall or time saved.

Classifier

Frozen validation contains 79 AI-labelled records. The test split remains sealed.

Only five categories appear in validation. Category accuracy's 95% interval is 84.4–96.5%. The small safety sample and missing classes limit broader claims. Safety recall of 2/5 is the main technical evaluation limitation. Deterministic safeguards and qualified review provide additional controls; they do not establish how many safety records the full system might miss.

Improving this requires independently reviewed safety examples and a fresh evaluation, not tuning against these five records. Current classifier results, v1.1 promotion comparison, and system results retain the detailed comparisons.

Coverage experiment and workflow audit

A fixed broader-training candidate was rejected: category agreement fell to 69/79, scope recall to 42/45 and safety recall to 1/5. Disease agreement rose to 43/45. The preregistered release gates failed, so v1.1 remains deployed. Training and promotion details make the negative result inspectable.

The AI walkthrough inspected seven current renderer outputs and three in-memory adversarial controls. Seven case checks were observed, all three controls passed, and no render errors occurred across 18 current decisions. The walkthrough includes a verified historical transition and explicitly simulated calibration. It does not measure human usefulness, review duration or downstream benefit.

The new review-task selector can choose only among questions constructed from validated source fields, or abstain. Its current examples concern an estimated milestone, termination rationale and application history. This is a constrained source-specific capability; the previous finding that free-form interpretations were withheld remains valid.

Source changes

A protocol fixed six registry IDs and up to four consecutive versions before collection. The 21 returned versions contain 15 adjacent pairs: four field changes and eleven non-changes. Both the learned and rules-only arms surfaced 4/4 changes, suppressed 11/11 non-changes and 21/21 repeated observations. This chronology comparison establishes no AI advantage; it does not negate the separate category-classification improvement.

The first run hid one safe status change because another field was sensitive. That was repaired; the original miss remains recorded. The final result is development evidence, not untouched holdout performance or human relevance. The selected cohort comes from existing registry sources, not a random prospective sample.

Drafting and calibration

Four live GPT-5.6 Luna drafts completed; 11/11 quotes matched their source. The semantic verifier accepted 4/4 factual headlines, while the exact verifier accepted 0/4 paraphrases. Neither retained any of the eight generated interpretation fields. Fallback remains necessary. Both passed five small known-answer controls. Model acceptance is not human factual validation.

The calibration example adds Medical to a new synthetic Access source while leaving a guideline control unchanged. It demonstrates rule transfer, not measured stakeholder agreement.

Human evidence still needed

False-discard performance is not measured by a real human-reviewed sample. The app now freezes stratified samples across terminal filtering reasons, with source-observation units, recorded inclusion fractions, persistent sample membership and separate human/simulated outcomes. Field replacements on surviving records are excluded from that terminal population.

These controls make an audit reproducible; they do not supply a measured system-wide miss rate. The pilot must adjudicate the sample. Keeping the test split sealed protects the evaluation; it is not a missing MVP feature.

Reproduce

uv run python scripts/evaluate_chronology.py
uv run python scripts/evaluate_live_drafting.py --recheck-recorded
uv run python scripts/summarize_incremental_evidence.py

These checks use isolated databases. The drafting recheck uses saved responses and makes no hosted generation calls; semantic model weights may need an initial download.

Protocol · Chronology · Drafting · Calibration · Pilot

Comparison scope

The keyword baseline is our own rules implementation. These results do not establish superiority over commercial intelligence products or human reviewers. See the existing approaches comparison for sourced product context and the proposed user study.