Loading results from the API… If the hosted API was asleep, this can take up to a minute.
Loading results from the API… If the hosted API was asleep, this can take up to a minute.
Guard · fixture support-rag-py (F1)
Each row is a scripted pull request against F1. Guard maps the diff onto the behavior-to-code graph and runs only the checks the change can affect. To measure what that costs in missed regressions, the full suite was also run on every PR; its failures are the ground truth.
Recorded under the perf gate in use at the time (a p95 point rule with no warmup). That gate was later found biased and replaced; the current gate's results are on the R1 page and in the corrections. Which checks were selected does not depend on that gate; every perf verdict in this table does.
| pull request | change | checks run | Guard | full suite | missed by targeting | time, s |
|---|---|---|---|---|---|---|
| docs_only README wording change | docs | 1/8 | success | success | none | 73 → 2 |
| prompt_wording Tone instruction reworded in the system prompt | prompt | 7/8 | success | success | none | 72 → 71 |
| r1_dynamic_head R1: request id and timestamp prepended to the system prompt 'for debugging' | code, imports, prompt | 7/8 | failure | failurebench:chat_perf_gate, check:prompt_prefix_stable | none | 85 → 84 |
| r2_context_bloat R2: retrieve 12 chunks and quadruple the context budget 'to improve recall' | retrieval, retrieval_config | 5/8 | success | success | none | 74 → 67 |
| r3_citation_strip R3: prompt told to drop citation tags 'because the UI shows sources separately' | prompt | 7/8 | success | success | none | 71 → 70 |
| r3b_chunk_regex_strip Per-chunk regex meant to strip citation tags from the stream (ineffective: tags span chunks) | code, imports | 3/8 | success | success | none | 73 → 12 |
| r4_approval_removed R4: approval check removed from ticket creation | code, tool | 1/8 | failure | failuresecurity:approval_gate | bench:chat_perf_gate (warn) | 72 → 2 |
| r5_prefix_caching_off R5: prefix caching disabled in the serving config | serving_config | 3/8 | neutral | neutral | none | 72 → 57 |
| max_tokens_up Generation budget raised from 256 to 1024 tokens | generation_config | 4/8 | success | success | none | 71 → 57 |
| model_swap Serving model changed to the 1.5B AWQ checkpoint | model, serving_config | 4/8 | success | success | none | 72 → 57 |
| test_only New retriever unit test | tests | 2/8 | success | success | none | 72 → 3 |
| dependency_bump Pin a newer FastAPI | dependency | 3/8 | success | success | none | 86 → 19 |
| health_payload Health endpoint returns a version field | code_off_llm_path | 1/8 | success | success | none | 72 → 2 |
Source: bench/results/2026-10-06/rq3.json. The judge item is SKIP in every run (no judge key configured); serving-config PRs cannot fail because every revision runs on the same endpoint.
# No regressions in 1 targeted check conclusion: success **Ran 1 of 8 suite items** selected from the behavior-to-code graph. Change categories: docs. ### Security | check | result | detail | |---|---|---| | `security:approval_gate` | **PASS** | 1 approval test(s) (generated from the base revision): 1 passed in 0.17s | <details><summary>Not run for this change</summary> - `unit:tests/test_retriever.py`: exercises nothing this change touches - `unit:tests/test_prompts.py`: exercises nothing this change touches - `check:citation_required`: exercises nothing this change touches - `judge:ungrounded_answer`: exercises nothing this change touches - `check:prompt_prefix_stable`: exercises nothing this change touches - `check:context_budget`: exercises nothing this change touches - `bench:chat_perf_gate`: exercises nothing this change touches </details> ### What changed in the graph
# No regressions in 7 targeted checks conclusion: success **Ran 7 of 8 suite items** selected from the behavior-to-code graph. Change categories: prompt. ### Quality | check | result | detail | |---|---|---| | `check:citation_required` | **PASS** | citation contract met in 8/34 answers on the PR vs 5/34 on base (+8.8 pts; budget -5 pts); via full answer path (incl. post-processing) | | `judge:ungrounded_answer` | **SKIP** | no judge model key configured (BYOK); judge not run | ### Performance | check | result | detail | |---|---|---| | `check:prompt_prefix_stable` | **PASS** | requests share 3964 leading characters (base 3954, need 3558) | | `check:context_budget` | **PASS** | largest prompt 982 tokens (base 980) against a budget of 3840; median 966 vs 964 | | `bench:chat_perf_gate` | **PASS** | median p95 TTFT 232.8 -> 218.5 ms (-6.2%, pass; 3 runs each, runs overlap); prefix-cache hit 98% -> 98% | ### Security | check | result | detail | |---|---|---| | `security:approval_gate` | **PASS** | 1 approval test(s) (generated from the base revision): 1 passed in 0.25s | ### Tests | check | result | detail | |---|---|---| | `unit:tests/test_prompts.py` | **PASS** | 2 passed in 0.41s | <details><summary>Not run for this change</summary> - `unit:tests/test_retriever.py`: exercises nothing this change touches </details> ### What changed in the graph - `prompt:app/prompts.py::SYSTEM_PROMPT`: `static_chars` 3927 -> 3937; `approx_tokens` 982 -> 984 - `prompt:app/prompts.py::build_messages#system`: `static_prefix_chars` 3927 -> 3937; `static_chars` 3927 -> 3937
# 2 regressions found: prompt_prefix_stable, chat_perf_gate conclusion: failure **Ran 7 of 8 suite items** selected from the behavior-to-code graph. Change categories: code, imports, prompt. ### Quality | check | result | detail | |---|---|---| | `check:citation_required` | **PASS** | citation contract met in 6/34 answers on the PR vs 5/34 on base (+2.9 pts; budget -5 pts); via full answer path (incl. post-processing) | | `judge:ungrounded_answer` | **SKIP** | no judge model key configured (BYOK); judge not run | ### Performance | check | result | detail | |---|---|---| | `check:prompt_prefix_stable` | **FAIL** | requests share 15 leading characters (base 3954, need 3558) | | `check:context_budget` | **PASS** | largest prompt 1024 tokens (base 980) against a budget of 3840; median 1007 vs 964 | | `bench:chat_perf_gate` | **FAIL** | median p95 TTFT 624.1 -> 1500.2 ms (+140.4%, block; 3 runs each, runs separated); prefix-cache hit 85% -> 0% | ### Security | check | result | detail | |---|---|---| | `security:approval_gate` | **PASS** | 1 approval test(s) (generated from the base revision): 1 passed in 0.23s | ### Tests | check | result | detail | |---|---|---| | `unit:tests/test_prompts.py` | **PASS** | 2 passed in 0.34s | <details><summary>Not run for this change</summary> - `unit:tests/test_retriever.py`: exercises nothing this change touches </details> ### What changed in the graph - `prompt:app/prompts.py::build_messages#system`: `static_prefix_chars` 3927 -> 8; `static_chars` 3927 -> 3940; `dynamic_head` False -> True
# No regressions in 5 targeted checks conclusion: success **Ran 5 of 8 suite items** selected from the behavior-to-code graph. Change categories: retrieval, retrieval_config. ### Quality | check | result | detail | |---|---|---| | `check:citation_required` | **PASS** | citation contract met in 5/34 answers on the PR vs 6/34 on base (-2.9 pts; budget -5 pts) | | `judge:ungrounded_answer` | **SKIP** | no judge model key configured (BYOK); judge not run | ### Performance | check | result | detail | |---|---|---| | `check:context_budget` | **PASS** | largest prompt 1264 tokens (base 980) against a budget of 3840; median 1228 vs 964 | | `bench:chat_perf_gate` | **PASS** | p95 TTFT 437.9 -> 351.5 ms (-19.7%, pass); prefix-cache hit 96% -> 90% | ### Security | check | result | detail | |---|---|---| | `security:approval_gate` | **SKIP** | no approval-gate test installed (run Forge to install one) | <details><summary>Not run for this change</summary> - `unit:tests/test_retriever.py`: exercises component:app/retriever.py::retrieve, but the change (retrieval, retrieval_config) is not one it detects - `unit:tests/test_prompts.py`: exercises nothing this change touches - `check:prompt_prefix_stable`: exercises nothing this change touches </details> ### What changed in the graph - `config_key:config/rag.yaml#retrieval.max_context_chars`: `value` 6000 -> 24000 - `config_key:config/rag.yaml#retrieval.top_k`: `value` 3 -> 12 - `retriever:app/retriever.py::retrieve`: `top_k` 3 -> 12
# 1 regression found: citation_required conclusion: failure **Ran 7 of 8 suite items** selected from the behavior-to-code graph. Change categories: prompt. ### Quality | check | result | detail | |---|---|---| | `check:citation_required` | **FAIL** | citation contract met in 2/34 answers on the PR vs 5/34 on base (-8.8 pts; budget -5 pts); via full answer path (incl. post-processing) | | `judge:ungrounded_answer` | **SKIP** | no judge model key configured (BYOK); judge not run | ### Performance | check | result | detail | |---|---|---| | `check:prompt_prefix_stable` | **PASS** | requests share 3875 leading characters (base 3954, need 3558) | | `check:context_budget` | **PASS** | largest prompt 959 tokens (base 980) against a budget of 3840; median 943 vs 964 | | `bench:chat_perf_gate` | **PASS** | median p95 TTFT 204.5 -> 211.1 ms (+3.2%, pass; 3 runs each, runs overlap); prefix-cache hit 98% -> 98% | ### Security | check | result | detail | |---|---|---| | `security:approval_gate` | **PASS** | 1 approval test(s) (generated from the base revision): 1 passed in 0.20s | ### Tests | check | result | detail | |---|---|---| | `unit:tests/test_prompts.py` | **PASS** | 2 passed in 0.46s | <details><summary>Not run for this change</summary> - `unit:tests/test_retriever.py`: exercises nothing this change touches </details> ### What changed in the graph - `prompt:app/prompts.py::SYSTEM_PROMPT`: `static_chars` 3927 -> 3848; `approx_tokens` 982 -> 962 - `prompt:app/prompts.py::build_messages#system`: `static_prefix_chars` 3927 -> 3848; `static_chars` 3927 -> 3848
# 1 regression found: approval_gate conclusion: failure **Ran 1 of 8 suite items** selected from the behavior-to-code graph. Change categories: code, tool. ### Security | check | result | detail | |---|---|---| | `security:approval_gate` | **FAIL** | 1 approval test(s) (generated from the base revision): 1 failed in 0.20s | > assert not isinstance(exc.value, _NoOutbound), "side effect executed without approval" | <details><summary>Not run for this change</summary> - `unit:tests/test_retriever.py`: exercises nothing this change touches - `unit:tests/test_prompts.py`: exercises nothing this change touches - `check:citation_required`: exercises nothing this change touches - `judge:ungrounded_answer`: exercises nothing this change touches - `check:prompt_prefix_stable`: exercises nothing this change touches - `check:context_budget`: exercises nothing this change touches - `bench:chat_perf_gate`: exercises nothing this change touches </details> ### What changed in the graph - `security_boundary:app/tickets.py::create_ticket#approval`: `exists` True -> False - `tool:app/tickets.py::create_ticket`: `approval_gate` True -> False