Demo · recorded

One pull request, start to finish.

Everything on this page is a recording, not a live run. The numbers come from the stored results of three repeated trials. Live runs happen on the owner’s runner; this site is not connected to one, so it shows recordings only.

Silent screen recording of the end-to-end rehearsal, 2026-10-14, on a laptop: the landing and this page, a public GitHub scan, the graph, a live benchmark, and a live Guard run that flags the regression (about two minutes of it is the run itself). Narration follows the three-minute script.

The pull request, step by step

The exact change, the measured effect on the model server, the gate’s decision, the fix Furnace drafted, and the human review it waits for. Switch between the three trials; each one is compared inside its own interleaved runs.

RECORDED RUN · NOT LIVEtrial

Scripted pull request into main · fixture app F1 (support assistant)

“Add request id and timestamp to the system prompt for debugging”

The change app/prompts.py · build_messages

@@ -1 +1,4 @@
+import uuid
+from datetime import UTC, datetime
+
 from app.retriever import Chunk
@@ -59,3 +62,6 @@
     return [
−        {"role": "system", "content": SYSTEM_PROMPT},
+        {
+            "role": "system",
+            "content": f"Request {uuid.uuid4().hex[:8]} at {datetime.now(UTC).isoformat()}\n" + SYSTEM_PROMPT,
+        },
         {"role": "user", "content": f"CONTEXT:\n{context}\n\nQUESTION: {question}"},

Answers stay the same, so output-quality evals pass. But every request now starts with different text, so the server can no longer reuse its cached prompt prefix.

2
Measured on the model server geometric-mean TTFT
main
68.6 ms · hit 98%
this PR
494.2 ms · hit 0%
+620.7% 90% interval +590.3% to +652.5%
3
Decision block
The whole interval is above the 10% tolerance and the change exceeds the 25% block threshold. The check run fails; nothing is merged.

Draft repair verified

Tried first and rejected by the same gate: move_dynamic_to_suffix (kept the values in the prompt). Then move_dynamic_to_log:

app/prompts.py +5 −1 · show the repair diff

@@ -1,3 +1,4 @@
+import logging
 import uuid
 from datetime import UTC, datetime
 
@@ -59,10 +60,13 @@
 
 def build_messages(question: str, chunks: list[Chunk], max_chars: int) -> list[dict[str, str]]:
     context = build_context(chunks, max_chars)
+    logging.getLogger(__name__).info(
+        "per-request values (logged, not sent in the prompt): %s", f"Request {uuid.uuid4().hex[:8]} at {datetime.now(UTC).isoformat()}\n"
+    )
     return [
         {
             "role": "system",
−            "content": f"Request {uuid.uuid4().hex[:8]} at {datetime.now(UTC).isoformat()}\n" + SYSTEM_PROMPT,
+            "content": SYSTEM_PROMPT,
         },
         {"role": "user", "content": f"CONTEXT:\n{context}\n\nQUESTION: {question}"},
     ]

repair

71.5 ms · hit 98%

vs this PR: -85.5% interval -86.2% to -84.9% · must be below 0%
vs main: +4.2% worst case +9.5% · must stay below +25%

✓ regression test written first: fails on this PR, passes on main
✓ every executed check passed on the repaired tree · LLM judge skipped (no key), not counted as a pass

5
Human review pending
Opened as a draft pull request into the PR’s branch. Furnace never merges. The repair logs the request id instead of sending it, so the reviewer decides whether that is acceptable.

Trial 1 of 3 repeated trials of this one scenario (fixture F1, regression R1). Laptop RTX 3050 Ti, vLLM, Qwen2.5-0.5B, concurrency 8, 3 interleaved runs per revision, campaign 2026-10-12-gate-validation. Raw results

On real GitHub

The same flow, on a public repository

Recorded 2026-10-14 on dilipna/furnace-demo-f1. One live trial on a laptop GPU, not a rate. An independent check of GitHub afterwards passed 10 of 10 items (permissions, token scope, commit, drafts, untouched branches, webhooks).

01
Forge
draft PR with the checks, validated in a sandbox first
Open on GitHub
02
The regression
a pull request adding a request id to the system prompt
Open on GitHub
03
Guard
check run on the PR’s head commit: failure, “2 regressions found: prompt_prefix_stable, chat_perf_gate”
Open on GitHub
04
Repair
draft fix: -1.6% vs main (90% interval -9.1% to +6.6%), while the PR measured +593.5%
Open on GitHub
05
Webhook
a new commit (49224d2) on a fresh PR: GitHub’s signed event queued a new Guard run, whose check landed on that exact commit (failure); redeliveries of the event changed nothing
Open on GitHub

What the evidence supports, and where it stops

Every campaign and correction →

Every figure is read from a result file in the repository. One laptop GPU, one small model, two fixture apps.

0/36

false alarms on unchanged code

Perf gate, fresh validation after its design was frozen: PASS 29/36, the rest INCONCLUSIVE (2026-10-12-gate-validation)

6/6

times the prompt-cache regression was blocked

Regression R1 on fixture F1, median change +613% geometric-mean TTFT (2026-10-12-gate-validation)

3/3

repairs verified, repeated trials of one scenario

Fixture F1, regression R1, rule-based. Not a rate over apps or regression types (2026-10-12-gate-validation)

+433%

p95 time to first token with prefix caching off

Concurrency 16, real F1 workload, laptop RTX 3050 Ti at 780 MHz, 3 paired repeats (2026-10-06)

Limits, measured the same way

15/18

small regressions left undecided

Graded changes measured +4% to +7% (set medians), below what they were sized for. Catching regressions just above the 10% tolerance is not shown

32/48

LLM call sites found in five unseen apps

Held-out first scan, labels committed before it. Misses: unsupported SDKs and agent runners

39/77

scanner findings that were right

Same apps; recall 39/64. Scanner findings are advisory, never a gate

@@ -1 +1,4 @@

import uuid

from datetime import UTC, datetime

from app.retriever import Chunk

@@ -59,3 +62,6 @@

return [

{"role": "system", "content": SYSTEM_PROMPT},

{

"role": "system",

"content": f"Request {uuid.uuid4().hex[:8]} at {datetime.now(UTC).isoformat()}\n" + SYSTEM_PROMPT,

{"role": "user", "content": f"CONTEXT:\n{context}\n\nQUESTION: {question}"},

@@ -1,3 +1,4 @@

import logging

import uuid

from datetime import UTC, datetime

@@ -59,10 +60,13 @@

def build_messages(question: str, chunks: list[Chunk], max_chars: int) -> list[dict[str, str]]:

context = build_context(chunks, max_chars)

logging.getLogger(__name__).info(

"per-request values (logged, not sent in the prompt): %s", f"Request {uuid.uuid4().hex[:8]} at {datetime.now(UTC).isoformat()}\n"

)

return [

{

"role": "system",

"content": f"Request {uuid.uuid4().hex[:8]} at {datetime.now(UTC).isoformat()}\n" + SYSTEM_PROMPT,

"content": SYSTEM_PROMPT,

{"role": "user", "content": f"CONTEXT:\n{context}\n\nQUESTION: {question}"},

]

One pull request, start to finish.

Screen recording

The pull request, step by step

The same flow, on a public repository

What the evidence supports, and where it stops

Limits, measured the same way

One pull request, start to finish.

Screen recording

The pull request, step by step

The same flow, on a public repository

What the evidence supports, and where it stops

Limits, measured the same way