Evaluations · fixture support-rag-py (F1)
Evaluators and labels
Deterministic checks are tested against seeded failures. A semantic judge is only trusted after it agrees with human labels on a held-out split, so labels come first.
Deterministic checks on seeded failures
Mutation operators with a known effect on each check's specification, applied to real F1 answers and the 34 documented answers. This verifies detection, not agreement with human judgment.
| check | recall | precision | FPR | seeded failures | clean controls |
|---|---|---|---|---|---|
| citation_required | 1.000 | 1.000 | 0.000 | 122 | 78 |
| duplicate_action | 1.000 | 1.000 | 0.000 | 2 | 2 |
| forbidden_pattern | 1.000 | 1.000 | 0.000 | 200 | 45 |
| prompt_prefix_stable | 1.000 | 1.000 | 0.000 | 80 | 80 |
| side_effect_requires_approval | 1.000 | 1.000 | 0.000 | 3 | 3 |
Blind spot: an answer that cites a retrieved but wrong document passes citation_required in 31/31 seeded cases. That needs the judge.
Source: bench/results/2026-10-06/rq2.json. After labeling, run uv run poe bench-rq2 to recompute.
Labeling queue
60 real answers sampled by model and question kind. Keys: j/k move, p/f grounded yes/no, c/x/- correct yes/no/n.a., n note.
The labeling queue needs the API, which this recorded-only site does not have.