GitHub links require repository access. # Results SAGA has working source-change, blocked-claim and calibration examples. The evidence below supports a prototype; human usefulness and savings remain unmeasured. ## Where AI helps The classifier labelled categories correctly on 73/79 records (92.41%), versus 44/79 (55.70%) for the keyword baseline on the same AI-labelled validation set. That is evidence for category classification on this dataset. Structured source-change detection tied its rules-only comparator at 4/4. Neither result establishes production effectiveness. The busy replay's 52 inputs and 11 review records demonstrate the workflow. They are curated counts, not a measurement of autonomous detection, recall or time saved. ## Classifier Frozen validation contains 79 AI-labelled records. The test split remains sealed. | Measure | Result | |---|---:| | Category accuracy | 73/79 · 92.41% | | Scope precision / recall / F1 | 0.9556 / 0.9556 / 0.9556 | | Disease accuracy | 84.44% | | Supported-class macro F1 | 0.8695 | | Fixed eleven-class macro F1 | 0.3952 | | Safety recall | 2/5 | Only five categories appear in validation. Category accuracy's 95% interval is 84.4–96.5%. The small safety sample and missing classes limit broader claims. Safety recall of 2/5 is the main technical evaluation limitation. Deterministic safeguards and qualified review provide additional controls; they do not establish how many safety records the full system might miss. Improving this requires independently reviewed safety examples and a fresh evaluation, not tuning against these five records. [Current classifier results](../artifacts/evaluation/classifier-source-conditioned-v1-fixed-validation.json), [v1.1 promotion comparison](../artifacts/evaluation/classifier-v1.1-promotion.json), and [system results](evaluation/system-evaluation-v1.json) retain the detailed comparisons. ## Coverage experiment and workflow audit A fixed broader-training candidate was rejected: category agreement fell to 69/79, scope recall to 42/45 and safety recall to 1/5. Disease agreement rose to 43/45. The preregistered release gates failed, so v1.1 remains deployed. [Training and promotion details](models.md#broader-coverage-experiment-september-2026) make the negative result inspectable. The [AI walkthrough](https://github.com/wysh3/saga/blob/main/docs/evaluation/simulated-pilot-v2.md) inspected seven current renderer outputs and three in-memory adversarial controls. Seven case checks were observed, all three controls passed, and no render errors occurred across 18 current decisions. The walkthrough includes a verified historical transition and explicitly simulated calibration. It does not measure human usefulness, review duration or downstream benefit. The new review-task selector can choose only among questions constructed from validated source fields, or abstain. Its current examples concern an estimated milestone, termination rationale and application history. This is a constrained source-specific capability; the previous finding that free-form interpretations were withheld remains valid. ## Source changes A protocol fixed six registry IDs and up to four consecutive versions before collection. The 21 returned versions contain 15 adjacent pairs: four field changes and eleven non-changes. Both the learned and rules-only arms surfaced 4/4 changes, suppressed 11/11 non-changes and 21/21 repeated observations. This chronology comparison establishes no AI advantage; it does not negate the separate category-classification improvement. The first run hid one safe status change because another field was sensitive. That was repaired; [the original miss](https://github.com/wysh3/saga/blob/main/docs/evaluation/chronology-v1-before-repair.json) remains recorded. The final result is development evidence, not untouched holdout performance or human relevance. The selected cohort comes from existing registry sources, not a random prospective sample. ## Drafting and calibration Four live GPT-5.6 Luna drafts completed; 11/11 quotes matched their source. The semantic verifier accepted 4/4 factual headlines, while the exact verifier accepted 0/4 paraphrases. Neither retained any of the eight generated interpretation fields. Fallback remains necessary. Both passed five small known-answer controls. Model acceptance is not human factual validation. The calibration example adds Medical to a new synthetic Access source while leaving a guideline control unchanged. It demonstrates rule transfer, not measured stakeholder agreement. ## Human evidence still needed False-discard performance is not measured by a real human-reviewed sample. The app now freezes stratified samples across terminal filtering reasons, with source-observation units, recorded inclusion fractions, persistent sample membership and separate human/simulated outcomes. Field replacements on surviving records are excluded from that terminal population. These controls make an audit reproducible; they do not supply a measured system-wide miss rate. The pilot must adjudicate the sample. Keeping the test split sealed protects the evaluation; it is not a missing MVP feature. ## Reproduce ```sh uv run python scripts/evaluate_chronology.py uv run python scripts/evaluate_live_drafting.py --recheck-recorded uv run python scripts/summarize_incremental_evidence.py ``` These checks use isolated databases. The drafting recheck uses saved responses and makes no hosted generation calls; semantic model weights may need an initial download. [Protocol](https://github.com/wysh3/saga/blob/main/data/evaluation/chronology-v1/protocol.json) · [Chronology](https://github.com/wysh3/saga/blob/main/docs/evaluation/chronology-v1.json) · [Drafting](https://github.com/wysh3/saga/blob/main/docs/evaluation/drafting-v1.json) · [Calibration](https://github.com/wysh3/saga/blob/main/docs/evaluation/calibration-transfer-v1.json) · [Pilot](https://github.com/wysh3/saga/blob/main/docs/pilot.md) ## Comparison scope The keyword baseline is our own rules implementation. These results do not establish superiority over commercial intelligence products or human reviewers. See the [existing approaches comparison](https://github.com/wysh3/saga/blob/main/docs/pilot.md#existing-approaches-and-sagas-focus) for sourced product context and the proposed user study.