FurnaceBench

How well does Furnace work?

Five research questions, measured with reproducible commands and reported with their negative results. The report below is generated from result files only (uv run poe bench-report).

Corrections and updates, 2026-10-09. This campaign's report below is generated from its result files and is left as it was.

1. The perf gate was biased, so RQ5's "3/3 verified" does not hold. Without a warmup wave, the first requests after the GPU sat idle took ~200 ms instead of ~65 ms and set each run's p95; the base revision, measured first, read ~2.4x slower. An A/A test (main vs itself) showed differences up to 60%; with a warmup wave the A/A noise is at most 9.7% (2026-10-09/gate_noise.md). Re-measured, R1 costs +905% to +1,004% p95 TTFT (not +336% to +414%), and the repair reported below ("move to the end of the user message") is +40% to +50% over main and is blocked: 0/3 (2026-10-09-gate-warmup/).

2. Current result (2026-10-09-gate-warmup-v2/): the repair loop now tries a second strategy, logging the per-request values instead of sending them: 3/3 verified at -2.0% to +8.5% vs main, prefix-cache hit 98% -> 98%, 3/3 full-suite audits clean.

3. "Capped GPU clock" was the wrong word. The laptop GPU reaches 1,850 MHz under a saturating load; under this serving load the driver holds ~780 MHz with no throttle reason active (NOTES.md, 2026-10-09 addendum).

RQ1, RQ2 and RQ4 do not use the perf gate. RQ3's perf-gate noise figures came from the biased gate; its regression recall did not depend on them.

4. RQ1 update (new results, not a correction). A new held-out set 3 (Gemini, Mistral, Replicate, Bedrock, Cohere apps; labeled before any scan) measured the SDK-coverage change: LLM call sites 1/9 -> 8/9 at precision 1.00, Streamlit/CLI workflows 0/9 -> 6/9. Set 2, below, is no longer held-out (it was seen while building the change): 1/5 -> 5/5. A scorer bug that never matched call sites inside class methods was fixed and both scanners were re-scored (2026-10-09/RQ1_NOTES.md, 2026-10-09/rq1.md).

5. Update, 2026-10-11: "A/A within 9.7%" was a sample, not a bound. A second A/A on 2026-10-11 at the same 780 MHz clock, on AC, measured up to +13.4% between identical revisions (12 trials; 2026-10-11-gate-ci/gate_noise.md). With the WARN-above-10% point rule, 2 of those 12 would have been false WARNs. The gate now decides on a 95% interval (hierarchical bootstrap over runs and requests): on the same 12 trials, 0 false WARN or BLOCK, 7 PASS, 5 INCONCLUSIVE (reported as such, never as a pass), and the interval contained the true difference in 12/12 (docs/furnacebench.md, "The perf gate's decision rule"). Item 2's repair figures (-2.0% to +8.5% vs main) were judged with the point rule; repair under the interval rule has not been re-measured yet.

Since this report

Later studies, newest first. Each is the committed NOTES.md of its directory in bench/results/, next to its raw JSON.

2026-10-14-live-webhook

Fresh-commit webhook run, 2026-10-14 (18:45 UTC): PASS, first attempt

Closes the gap left by the live GitHub run of the same day (2026-10-14-live-github/), where the webhook-queued job found its head already checked and reused that check run. Here the head commit was new, so nothing could be reused. Recorded by scripts/live_webhook_run.py (code eeb419a) in webhook_run.{json,md}; this was the only attempt.

Setup: API on :8010 with the App's webhook secret, the App's deliveries forwarded byte for byte by smee-client (channel URL not recorded), the Guard runner and the lab vLLM on the laptop (AC power; GPU clock not pinned: a demonstration, not a measurement campaign). Repair consent off (FURNACE_GUARD_AUTO_REPAIR unset).

step	what was observed (read from GitHub or the database)
commit	branch `webhook-recheck-20261014-1845` from `r1-request-id` (0a637d7) plus one README line, head 49224d282d101806e2f43873a7f22ca15a839e8a, pushed with the owner's git credentials
pr	https://github.com/dilipna/furnace-demo-f1/pull/4 (non-draft, into `main`), opened with the App's PR token as the live run opened PR #2
delivery	GitHub delivery `61c93440-c7ff-11f1-8f9e-466f538a0d5c` (`pull_request`/`opened`, 18:45:11Z), answered 200; row in `webhook_deliveries`: source `push` (HMAC-verified), outcome `queued`, job `d09d0db6-8ae5-491a-af9e-1d97a56deafb`
redelivery while running	GitHub redelivered the same GUID while the job was `running`: answered 200; delivery rows for the GUID 1 → 1, jobs for this head 1 → 1, all `guard.pr` jobs 2 → 2
job	claimed by the laptop runner, `running` → `succeeded`, 1 attempt, 102.9 s from enqueue to finish
check	exactly one Furnace check run on 49224d2: https://github.com/dilipna/furnace-demo-f1/runs/116037356084, by `furnace-dev-dilipna`, `head_sha` equal to the new commit, started 18:45:15Z, completed 18:46:53Z, conclusion failure ("2 regressions found: prompt_prefix_stable, chat_perf_gate"), 4 annotations
redelivery after completion	same GUID again: answered 200, rows unchanged (1, 1, 2), still one check run on the head

Guard's verdict, from the check run (7 of 8 suite items selected; categories code, docs, imports, prompt): perf gate geometric-mean TTFT 73.6 → 496.9 ms (+575.4%, 90% interval +509.1% to +649.0%), BLOCK; prefix-cache hit 98% → 0%; p95 advisory +820.2%; prompt_prefix_stable FAIL; citation, context budget, approval gate and unit tests PASS; judge SKIP (no key, not a pass). The branch carries the R1 regression, so "failure" was the expected verdict; it agrees with the check on PR #2's head (+590.0%, 2026-10-14-live-github/). No repair job was enqueued (consent off; signatures: [prefix_instability]).

What this shows: GitHub's signed delivery for a new head → HMAC check → exactly one queued job → runner → a fresh check run attached to that exact head, and GitHub redeliveries of the same GUID (during and after the job) change nothing. What it does not show: the dedupe of two *different* events for one head (e.g. opened then ready_for_review), which is covered by tests only; behaviour behind a hosted API (the receiver ran on the laptop behind smee).

2026-10-14-live-github

Live GitHub runs, 2026-10-14

Repository dilipna/furnace-demo-f1 (public), App furnace-dev-dilipna (id in .env, installation 171233745, selected repositories: this one only). Every attempt is kept.

Attempt 1 (05:08 UTC, code eb5f8a4): INCOMPLETE, stopped at Forge

attempt-1/live_run.{json,md}. App check and branch heads passed. Forge validated its change set in the sandbox (13 files), then opening the draft PR failed: POST /git/trees 403 "Resource not accessible by integration". Nothing was written to the repository.

Cause, confirmed with a probe using the same PR-scoped token (tree objects only, unreferenced, no branch or ref changed): a normal file → 201; a file under .github/workflows/ → 403. GitHub needs the separate workflows permission for those paths. The mock GitHub used in the tests did not model it, so this was first seen live.

Fix (40d41a3): Furnace still does not request workflows; Forge commits the other files and puts the workflow file in the PR description for the reviewer to add.

Attempt 2

Recorded in live_run.{json,md} beside this file, after the fix, same repository and App. (05:11 UTC, code 3456656, clean): PASS, every step.

step	result
Forge	draft PR https://github.com/dilipna/furnace-demo-f1/pull/1 (`furnace/forge-bb4d918` → main), 13 files validated in the sandbox; the workflow file is in the PR description, not committed
R1 PR	https://github.com/dilipna/furnace-demo-f1/pull/2 (`r1-request-id` → main, head 0a637d7)
Guard	check run 115692708323 on 0a637d7: failure, "2 regressions found: prompt_prefix_stable, chat_perf_gate". 7 of 8 checks selected. Perf gate: geometric-mean TTFT 72.7 → 501.7 ms (+590.0%, 90% interval +557.4 to +624.3), BLOCK; prefix-cache hit 98% → 0%; p95 advisory +851.8%. Judge SKIP (no key, not a pass); citation check 1/34 vs 0/34 on base (at its floor, says nothing about quality). 97 s
Repair	first strategy (`move_dynamic_to_suffix`) rejected: +36.1% vs main (90% interval +27.8 to +45.1), BLOCK. Second (`move_dynamic_to_log`) accepted: -1.6% vs main (90% interval -9.1 to +6.6), with the PR at +593.5% (72.4 → 502.1 ms) in the same interleaved runs; draft PR https://github.com/dilipna/furnace-demo-f1/pull/3 (`furnace/repair-pr2-0a637d7` → `r1-request-id`). 210 s
Replays	Forge and repair run again on the same commits returned PRs #1 and #3 (`reused=True`). The repair replay re-measured: first strategy +46.7% (+44.4 to +49.1) BLOCK, accepted fix -0.7% (-4.5 to +3.4)

One live trial (plus its replay) on a laptop GPU, not a rate; each figure comes from its own step's interleaved runs (live_run.json, steps guard, repair, replay_repair). The GPU clock was not pinned for this run (a demonstration, not a measurement campaign).

Independent verification (05:20 UTC): PASS, 10/10

verify.{json,md}, read from GitHub and the local receiver, not from the run record: permissions equal the manifest and the installation covers only this repository; Scan and Guard tokens get 403 on writes; the check run is on the PR's head commit, by this App, with a start, a completion and annotations on app/prompts.py (each changed hunk is annotated once per failing check, so the two lines appear twice); both PRs are drafts by furnace-dev-dilipna[bot] from furnace/* branches; main and r1-request-id unchanged; one open Furnace PR per base; 3 real pull_request deliveries reached the receiver over HMAC through the smee relay (outcomes: ignored, queued); a GitHub redelivery changed no rows; unsigned, wrong-secret and tampered requests got 401.

The Guard runner was stopped during the run so the webhook-triggered job would not compete for the GPU with the run's own Guard; that job was queued (outcome "queued").

The webhook-queued job, run afterwards (05:2x UTC)

The real pull_request delivery had queued one guard.pr job (6e6731cc) for PR #2's head 0a637d7. With the runner started (Guard settings in its .env), it claimed the job and succeeded: it found Furnace's completed check run 115692708323 on that head commit and reused it (reused: true, conclusion failure) instead of opening a second one. So the event path GitHub delivery → HMAC → job → runner is shown; a fresh Guard run triggered by a webhook is not (the head had already been checked). signatures: [] in the reused result, so no repair job was enqueued (repair consent was also off).

2026-10-13-set6

Held-out set 6: results and review (protocol docs/scanner-eval-set6.md)

Scanner frozen at 41ca529; five apps chosen by one pre-declared GitHub search (selection.md); labels and expected findings committed in ebd5824 before any scan. Labels by the developer agent, not independent. Three quantities, never combined:

quantity	result	file
LLM call sites (first scan)	found 32/48 (67%, exact 95% CI 52-80%), precision 32/32 (CI 89-100%)	`rq1.md` (column "held-out set 6")
Blueprint findings	precision 39/77 (51%, CI 39-62%), recall 39/64 (61%, CI 48-73%)	`set6/findings.md`
Evidence links	148/148 resolve (6 file-level)	`set6/citations.md`

Per app, call sites: Soul-of-Waifu 24/27, BYO-LLM-WIKI 5/12, audiobook-creator 3/5, mdflux 0/3, ps-fuzz 0/1. The pooled 32/48 is dominated by Soul-of-Waifu's 27 sites, which are one pattern repeated (nine provider classes x three methods); the mean of per-app recall is 0.38. Compared with sets 1-5 (first scans pooled 4/34), set 6 is higher; the sets are different apps and are reported side by side, not pooled.

The headline point: every evidence link resolved while half of the emitted findings were wrong. Link validity says nothing about finding correctness.

Review of every disagreement (after the result; the numbers above are not changed)

Judged by the developer agent after seeing the output. "Label" = the pre-registered label or its definition was wrong or too narrow; "scanner" = the scanner was wrong; "open" = defensible either way.

Missed call sites (16), all scanner: raw HTTP chat requests (mdflux 3, Soul-of-Waifu's Anthropic provider 3); LangChain chat models received as a function parameter or held in self.client (BYO 7, ps-fuzz 1); Pydantic AI Agent(...).run (audiobook 2).

Emitted, not expected (38):

Scanner, 32: sec.tool_approval_boundary on 28 functions that no model can call (ffmpeg,

file saves, config writes, release scripts: audiobook 23, mdflux 4, ps-fuzz 1). The rule treats any side-effecting function as a tool. eval.grounding_judge on mdflux (no retrieval). BYO: rel.fallback_gateway and eval.regression_suite (it has explicit max_retries=2 and Ragas evals, neither recognized). audiobook: inf.prefix_instability on a system message whose leading value is fixed per deployment (static analysis cannot tell; under the pre-declared definition the label stands).

Label, 4: prompt constants whose names lack "prompt"/"instruction"/"system"

(_CROSSDOC_JUDGE, _REQUIREMENTS, _SKILL_JSON_SHAPE in BYO; ethical_compliance_template in ps-fuzz). The naming rule in the labels was too narrow; the scanner was right.

Open, 2: audiobook inf.serving_baseline (its default endpoint is a local LM Studio

server: self-hosted, but not in the rule's engine list the labels mirrored); ps-fuzz eval.grounding_judge (its RAG-poisoning attack does retrieve; labeled as not RAG because it does not ground answers).

Expected, not emitted (25):

Cascade from missed call sites, 8: every app-level rule needs an LLM call, so mdflux (0/3

call sites) and ps-fuzz (0/1) lost timeout, fallback, eval, tracing and serving findings.

Tools not recognized, 8 (BYO): LangChain @tool functions, and the MCP write path's

approval gate.

Prompts built inside functions, 8 (audiobook 7: f-string system prompts inside functions

and Pydantic AI system_prompt=, and the prefix rules that depend on them) and 1 (BYO: the deep agent's system_prompt= constant).

audiobook obs.genai_tracing, 1: scanner.

Re-scored with the four label corrections only (for orientation, not a result): precision 43/77, recall 43/68. The pre-registered numbers stand.

What this says about the scanner

It is accurate where it fires on call sites (no false positive in 48 labeled sites) but misses common constructs (raw HTTP, injected LangChain models, Pydantic AI, LangChain tools), and each missed call site silently removes the app-level findings that depend on it. Its tool rule is wrong in kind: it flags side-effecting functions that no model can invoke. Neither is fixed here; a fix would be measured on a new held-out set (set 6 is now spent).

2026-10-13-fallback

Does uncertain or unsupported analysis widen the checks, or silently skip them? (2026-10-13)

Question (owner, 2026-10-13): when Guard's impact analysis cannot account for a change, does it fall back to broader checks, or does it skip checks that matter? And does Scan say what it could not read?

Guard: verified by tests written before the fix

The documented rule was "an application-code change that maps to no fact selects the full suite". It held only for *modified Python lines*. Seven tests were written against the code at d2df18d (packages/furnace/tests/test_impact.py, section "uncertain or unsupported analysis"); six failed (no fallback, and in five of them no category either, so nothing but the always-on checks ran):

change in the PR (on F1)	before (d2df18d)	after
an article in `docs/` that `app/retriever.py` loads (`(ROOT / "docs").glob("*.md")`)	category `docs`, nothing selected	full suite: "read by application code (app/retriever.py:28)"
a TypeScript file added under `app/`	skipped, no category	full suite: "source the analysis does not parse"
an unmodelled data file in the app (`app/data/thresholds.csv`)	skipped, no category	full suite: "a file the analysis does not model"
an added Python module that maps to no graph node	skipped (added files have no hunks)	full suite
a removed Python module that maps to no graph node	skipped	full suite
an app module that no longer parses	treated as removed nodes	full suite: "does not parse as Python"
CI workflow, an image, CHANGELOG (inert)	no fallback	no fallback (unchanged, the seventh test)

A template the scanner models (.j2) is still targeted, not a fallback (its prompt node is touched). Rule now (guardian/impact.py): a changed file is either mapped to graph nodes, or known to be inert (tests, CI, repository metadata, images, docs nothing reads), or it selects the full suite with the reason. "Read by application code" is a string-literal heuristic on the file name or its directory; it errs toward a fallback, which only widens.

Effect on results already reported

RQ3 (13 F1 PR scenarios, 2026-10-06/rq3/): selection recomputed under the new rule is

identical in 13/13 scenarios (docs_only edits the README, which nothing reads). RQ3 never exercised the cases above, so its "targeted ran 46% of checks" is unchanged and says nothing about them.

Held-out set 4 synthetic PRs (external_validation.md here vs 2026-10-12-reconcile/): the

Guard table is identical, 9/9 correct.

Scan: what it could not read

Unchanged and verified: a repository whose source is in a language other than Python gets a coverage box on the Blueprint naming the languages and their share, and an empty Blueprint says "a coverage limit, not a clean bill of health". New: Python files the extractor skips (syntax errors, Python 2, files over 2 MB) were silently dropped; they are now listed in the scan's stats (python_not_analyzed) and in the same coverage box.

2026-10-12-optimizer

Optimizer grid and independent confirmation (2026-10-13)

Design and decision rules fixed before any measurement (docs/optimizer.md; confirmation rules added before the grid). Run by bench/run_remaining_gpu.sh after RQ5, 18:37-19:21 UTC. Pre-run conditions conditions/2026-10-13T183741Z.json: on AC, vLLM image sha256:8a69ffad..., measurement code clean at 6efebfd, 780 MHz under the probe load. (The 03:15 UTC attempt refused to start on battery, conditions/2026-10-13T031547Z.json; no grid result exists from it.)

Result: no change is recommended

Grid (optimizer/optimizer.md): 5 of 8 configs feasible. Highest median SLO goodput:

0.5B, prefix caching on, max-num-seqs 8, 8.34 req/s (range 7.37-8.34), against the baseline (0.5B, caching on, seqs 32) at 8.30 (7.36-8.37). The ranges overlap, so by the pre-declared rule the recommendation is keep the baseline. Infeasible: both 1.5B-AWQ configs with caching off (p95 TTFT about 1.1 s > 500 ms SLO) and 0.5B caching off seqs 8 (facts quality not shown within the 5-point tolerance: -1.6, interval -7.8 to +4.7).

Independent confirmation (optimizer/confirm.md; 8 fresh launches A B B A A B B A, new

seeds): inconclusive. Goodput B vs A +3.8% (90% interval -8.8 to +18.2); p95 TTFT +1.0% (-6.7 to +9.5); quality facts +0.0 pts (-4.0 to +4.0), citations +0.0. No improvement is claimed.

What the grid does show, as measured on this laptop: turning prefix caching off roughly

halves goodput for the 0.5B model (4.4-4.7 vs 8.3 req/s) and pushes the 1.5B-AWQ model past the 500 ms p95 SLO; 1.5B-AWQ scores higher on both deterministic quality checks (facts 85.7% vs 74.6%, citations 12.7% vs 0.0%) at about 77% of the 0.5B goodput with caching on.

Clocks (not set; chosen by the driver)

config	SM MHz, 3 runs
0.5B caching on, seqs 8 / 32 (pick, baseline)	780 / 780
1.5B-AWQ caching on, seqs 8 / 32	781-782
0.5B caching off, seqs 8 / 32	796-847 / 791-810
1.5B-AWQ caching off, seqs 8 / 32	1,421-1,644 / 1,349-1,621

All eight confirmation launches ran at 780 MHz. The driver raised the clock only for the heaviest (caching-off) configs, which makes them look faster than they would at 780 MHz; they were slower or infeasible anyway, so no conclusion depends on it. The pick and the baseline were compared at the same clock, in the grid and in the confirmation.

Limits

One GPU, one workload (W1), concurrency 8; quality is two deterministic checks, not judged answer quality; the grid's three runs per config are consecutive on one container. Cost is N/A (no $/GPU-hour on a self-hosted laptop).

Operator activity during the run

File reads, greps, git status/git log, a directory comparison and one python -c that printed a stored JSON field (about 0.1 s), between 18:40 and 19:10 UTC. No test, lint, build or analysis script ran.

2026-10-12-gate-validation

Gate validation (2026-10-12/13): conditions, interruptions, affected trials

Protocol and thresholds: docs/gate-revision.md (frozen 6e90c99, implemented 26fa8eb, graded sizes 9d80db8). Nothing in this file changes the statistic, the interval, the run count or a threshold. Report: validation/report.md (generated by bench/gate_validation_report.py).

Conditions intended for every trial

Laptop on AC; RTX 3050 Ti; lab vLLM image pinned by digest (sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90), model Qwen2.5-0.5B-Instruct served as lab; c=8, 34 requests per run, 3 interleaved runs per side; no other compute on the machine while a trial runs.

Code. What decides a verdict (perf_gate.compare, the Welch interval, tdist.py) and what measures (furnace-bench runner and metrics, the harness, the F1 fixture) is as implemented in 26fa8eb. git diff 26fa8eb on the measurement paths shows four later changes, none of which touches a measured or decided quantity: perf_gate.py gained metric_label and a reworded repair summary string (95caf29); docker_sandbox.py stages dependency files with read permissions for the sandbox user, which only matters on Linux (ad295ec); furnace_bench gained correlation.py, which the gate does not import (7badfa2); bench/common.py writes JSON through a temporary file (2026-10-13). Trials 1-2 ran before ad295ec, trials 3-14 at ad295ec with a dirty tree (other files were being edited; the manifest records 95caf29 because it is rewritten after each trial). From trial 15 on, bench/conditions.py refuses to start unless the measurement paths are committed, and records the commit.

Interruptions (kept trials are never rerun)

V1 after trial 2 (~05:22 UTC Oct 12): the machine slept or Docker Desktop stopped; resumed

19:42 UTC on AC at 780 MHz.

V1 after trial 14 (19:54 UTC Oct 12): Windows went to sleep (System log, Kernel-Power 42);

every process died. Trial 15 had not finished and left no record. Resumed through the keep-awake launcher (scripts/run_awake.ps1), recorded in validation/v1_aa.json resumed.

A trial is the experimental unit: it is written to <set>.jsonl only after it finished, so a kill loses at most the trial in progress, which is then run again. A line cut off mid-write is kept in <set>.jsonl.cut and never counted.

Affected trials: the rule

Added 2026-10-13 ~00:55 UTC at the owner's direction, after trials 1-14 had been seen and before trial 15 ran. It is a validity rule, not an analysis change.

1. Definition. A trial is *affected* if competing compute ran on the machine during the trial's window: a test run, lint/format/type check, build, Docker or sandbox job, benchmark, or analysis script that is not part of the trial. File reads, searches, text edits, git and status checks of about 2 s or less are not competing compute. 2. Evidence, not results. Whether a trial is affected is decided from a timestamped log of what ran (the session's command log, with each command's start and end), compared with the trial's start time and duration. The trial's verdict, change or interval is never consulted. 3. Handling. An affected trial stays in the record with its result (validation/<set>.json, affected lists it with the reason and evidence). It is left out of the primary tally and replaced by one more trial at the end of the same set, so the set reaches its planned count of unaffected trials. The report shows a second row with the affected trials included, so their effect on every figure is visible. 4. Going forward. No competing compute is run by the operator during the remaining trials, and each trial records the machine's CPU busy share while it ran (mean and largest 1 s value) as supporting data; no rule uses that number.

Affected trials in V1 (audit of trials 1-14)

Every kept trial's window was compared with the previous session's command log (f87e7c7f-...jsonl, start and end of each shell command):

trial	window (UTC, Oct 12)	verdict	competing compute in the window
6	19:45:39-19:46:29	PASS	`ruff check --fix packages` + `ruff check` (19:45:43-47); `ruff format` + `bench/correlation.py` analysis run (19:46:07-10)
9	19:48:14-19:49:06	PASS	`pytest test_repair test_gate_stats` with the Docker sandbox, 19:48:22-19:49:13 (26 passed in 35.69 s): about 44 s of the trial
10	19:49:08-19:49:58	PASS	the same test run, its last ~5 s (19:49:08-19:49:13)

The handoff note written during the run named only trial 10, from an estimated trial start (~19:48:46). The recorded start times show the test run overlapped trial 9 for most of its duration and trial 10 only briefly; both, and trial 6, meet the definition above. Trials 1-5, 7-8 and 11-14 had only reads, edits, greps and git in their windows.

All three affected trials were PASS, so leaving them out cannot make the primary tally look better; the 'incl. affected' row shows the difference. Not quantifiable: the editor's own lint/type checking ran in the background at unknown times during trials 1-14 (the editor was open on the repository). That applies to every one of those trials and is a limitation of trials 1-14, not grounds to mark any one of them.

Launch on 2026-10-13 (02:04 UTC), stopped before any trial

The first launch never reached a trial: the lab vLLM container hung on Hugging Face Hub lookups ("Network is unreachable": it is also attached to the internal sandbox network) and the wrapper's readiness poll had no per-request timeout. Stopped by hand at 02:23 UTC; no trial, discard or record was written. Fix: HF_HUB_OFFLINE=1 for the vLLM service (the same cached model files; serving flags unchanged) and a 5 s timeout on the poll. The image digest and every measured setting are as before; the conditions check runs again.

Result (all sets complete, 2026-10-13 03:10 UTC; validation/report.md)

All 63 counted-or-affected trials ran on AC (checked after every trial; no discard, no stop) at 780-828 MHz. Nothing was retuned.

Unchanged code: false FAIL 0/36 (V1 24 + V2 12; exact 95% CI 0-9.7%), PASS 29/36

(V1 18/24 = 75%, V2 11/12 = 92%). Including the affected V1 trials: 0/39, PASS 32/39. Pre-declared targets (FAIL <= 10%, PASS >= 70%) met in both sets.

R1 (known large regression): BLOCK 6/6 (geometric mean +608.9% to +620.6%), target met.
Graded sets: the calibrated sizes were not reproduced. Before any trial,

calibrate-topk measured depth 4 at +12.9%, depth 8 at +18.2% and depth 12 (R2) at +30.2% (3 interleaved runs per variant, five variants in one rotation). Inside the gate trials the same changes measured medians of +4.3%, +6.5% and +7.2% (ranges +2.2..+6.5, +4.4..+33.0, +1.1..+54.5). The gate's verdicts are consistent with its own measurements (15/18 INCONCLUSIVE, 3/18 PASS, 0 FAIL), so the trials do not test detection of a regression above the 10% tolerance at these sizes. Whether the 3 PASS verdicts are wrong depends on which measurement of the true size is right; these data cannot decide it. Stated as a finding about the calibration, not tuned away; a separate calibration replication would be needed (exploratory, not part of this protocol).

Tail (p95, advisory): R1 +941% median; on unchanged code the p95 interval ruled out a

>10% tail regression in only 13/36 trials (width ~22-27 points), which is why p95 does not decide.

Observation, no rule attached: the first trial after each vLLM (re)start was the noisiest

(trial 3: -15.0%, interval -46.6..+35.3; trial 15: -12.0%, interval -51.1..+58.5; both INCONCLUSIVE, neither a false FAIL).

Reconciliation (2026-10-13, after all sets; every row in validation/trials.md)

Why 63 trials when the sets total 60. The planned sets are V1 24 + V2 12 + V3a 6 + V3b 6 + V3c 6 + 6 = 60 counted trials. V1 trials 6, 9 and 10 were *affected* (tests, lint or an analysis script ran during them; rule and evidence above). They are kept with their results, left out of the counted 60 and replaced by three extra V1 trials, so V1 recorded 27 trials: 60 counted + 3 affected = 63 recorded. V1 trial 15's first attempt, cut off by Windows Sleep before it finished, wrote no record and is in no count.

Why unchanged-code validation is "36", not "24". It was 36 from the start. The protocol frozen in 6e90c99 (before any validation trial) defines two unchanged-code sets, V1 fresh A/A (24) and V2 restored baseline (12), and states "Unchanged-code trials V1+V2 give 36" with the 0/36 Clopper-Pearson bound. Progress notes during the run ("14 of 24 trials so far") counted V1 alone, which made 24 look like the cohort. Nothing was added after outcomes were seen; the only change during the run was the affected-trial rule (section 8 of the protocol), which removed three PASS trials from the tally and replaced them. Both sets are also reported separately: V1 PASS 18/24, V2 PASS 11/12, false FAIL 0 in each.

GPU clock: configured versus observed. No clock is set: the laptop's driver chooses it for the load (780 MHz for this light serving load on Oct 6-9 and 12-13, ~1,960 MHz on Oct 11 after a reboot; 2026-10-11-live-clock/). What is configured is a pre-run condition, bench/conditions.py: on AC and a median SM clock of 780 MHz +/- 80 under a probe load, or the set does not start (passed at 02:24 and 18:23 UTC, conditions/). Observed per run (378 runs): 780-828 MHz, 359 runs at exactly 780. Every unchanged-code trial ran at 780 MHz in all six runs. The higher clocks (up to 828 MHz) occurred only in the R1 trials (all six) and one R2 trial, i.e. under the heavier prefill load of a regressed revision; a faster clock on that side can only make a regression measure smaller, so it does not inflate any BLOCK. No result here is compared with runs at another clock.

Every unchanged-code outcome. 36 counted: 29 PASS, 7 INCONCLUSIVE, 0 FAIL; the 3 affected trials are PASS. The seven that did not pass (none is a FAIL; an INCONCLUSIVE is reported as "the data cannot say", not as a pass):

set	trial	change	90% interval	note
V1	1	+4.3%	-1.3 to +10.1	upper bound 0.1 pt over the tolerance
V1	3	-15.0%	-46.6 to +35.3	first trial after a vLLM restart
V1	7	+4.5%	-7.0 to +17.4
V1	12	+5.4%	-3.9 to +15.6
V1	15	-12.0%	-51.1 to +58.5	first trial after a vLLM restart
V1	20	+3.5%	-5.1 to +13.0
V2	3	-1.0%	-11.5 to +10.8

RQ5 under the frozen gate: first attempt lost AC

Attempt 0 (started 03:10:10 UTC) lost AC at 03:10:38 (System log, Kernel-Power 105: the charger was disconnected). The attempt finished on battery and is kept, not counted, under rq5/discarded/ (it would have verified the second strategy at -2.7%; that number is not used). The optimizer's conditions check then refused to start (battery, 1,125 MHz). At 03:30 Windows Sleep was chosen (Kernel-Power 187, winlogon), after the run had ended. RQ5 and the optimizer were relaunched on AC at 18:21 UTC; the conditions check is in gpu.log.

RQ5 under the frozen gate: result (2026-10-13 18:23-18:37 UTC; rq5.md, rq5.json, rq5/)

What "3/3" is. Three repeated verification trials of one supported repair scenario: fixture F1 (support-rag-py), regression R1 (a request id and timestamp prepended to the system prompt "for debugging", fixtures/scenarios/f1_scenarios.py), the two rule strategies in their fixed order. It is not a success rate over apps, regressions or strategies. No LLM strategy exists.

Conditions and versions. Pre-run conditions check conditions/2026-10-13T182330Z.json: on AC, vLLM image sha256:8a69ffad..., measurement code clean at 6efebfd, 780 MHz under load; power re-checked after the run (gpu.log: on AC, 80%). Gate G3 (design frozen 6e90c99, implemented 26fa8eb; repair acceptance perf_gate.accept_repair: the candidate must be faster than the PR head with 95% confidence and its interval upper bound against base must stay below the 25% block threshold). Same F1 workload as V1-V3: c=8, 34 requests per run, 3 interleaved runs per revision.

repeat	strategy 1 `move_dynamic_to_suffix` (vs base)	strategy 2 `move_dynamic_to_log` (vs base)	PR head vs base	outcome
0	+9.5%, 90% interval -26.0..+62.1, INCONCLUSIVE: rejected	+4.2%, -0.8..+9.5, PASS: accepted	+620.7%, +590.3..+652.5, BLOCK	verified
1	+37.8%, +28.6..+47.6, BLOCK: rejected	+0.9%, -2.2..+4.1, PASS: accepted	+601.8%, +577.9..+626.5, BLOCK	verified
2	+39.6%, +33.7..+45.8, BLOCK: rejected	+1.7%, -3.6..+7.2, PASS: accepted	+626.1%, +608.8..+643.8, BLOCK	verified

Geometric-mean TTFT. Each strategy is measured in its own interleaved block with its own base runs (strategy 1's repeat-0 base was 91.0 ms against 68.6-71.3 ms elsewhere, the first block after the vLLM restart). In every repeat the generated regression test failed on the PR head and passed on base, and the regression test and existing tests passed on the accepted candidate. Prefix-cache hit rate: 98% on base, 98% on the repair.

Full-suite audit of each repaired tree (the whole F1 suite with the repair as head, run after acceptance; it does not decide "verified"): every executed item passed in 3/3 (unit tests, citation_required, prompt_prefix_stable, context_budget, security:approval_gate, bench:chat_perf_gate). judge:ungrounded_answer was skipped in 3/3 (no judge key). Two limits of that audit: the citation check passed at 0/34 vs 0/34 (Qwen2.5-0.5B never meets the citation contract on base either, the known floor effect), so it cannot detect a quality change here; and the judge did not run.

Is the skipped judge optional under the frozen criteria? Yes. "Verified" is decided in guardian/repair/loop.py by the regression test, the existing tests and the perf gate only; the plan's RQ5 criteria are regression + existing tests + perf budget, with the full-suite audit reporting new failures; and the plan makes an uncalibrated judge advisory (§ LLM judges: "Otherwise the judge stays calibrating and its results show as advisory"). The judge is uncalibrated (no human labels), so even a run would have been advisory. No released claim depends on it.

Is removing the value from the prompt behavior-preserving? The moved expression is f"Request {uuid.uuid4().hex[:8]} at {datetime.now(UTC).isoformat()}\n": a random id and the current UTC time, generated in the function and derived from no input. The PR's stated purpose is debugging, so it is diagnostic metadata. The accepted patch (patch in rq5/r1_dynamic_head_r*.json) restores the system message to exactly SYSTEM_PROMPT and leaves the user message untouched; the repaired app/prompts.py differs from base only by imports and the one logging call (the system dict is also split over four lines, same content). Checked by rendering after the run (bench/rq5_prompt_identity.py, rq5/prompt_identity.md): for each of the 3 accepted patches, 34/34 documented questions render byte-identical chat messages on base and repair. The F1 application (pricing, plans, invoices, refunds from its documentation) has no question that needs the time or a request id, and the base revision never sent them. Trade-off, stated in the repair PR: the model no longer sees these values.

Does the new logging expose sensitive values? Not in this scenario: the log line is logging.getLogger(__name__).info("per-request values (logged, not sent in the prompt): %s", <the id and timestamp>), which writes a random 8-hex id and a timestamp, no question text, user data or credential. In general the strategy logs whatever expressions it moves out of the prompt and does not inspect them; in another app those could be user identifiers. That is a limit of the strategy, stated in the repair PR's explanation (2026-10-13), not something it checks.

Kept, not counted, not hidden:

the three rejected strategy-1 candidates, one per repeat, with their measurements above

(INCONCLUSIVE once, BLOCK twice);

R2 (context bloat), attempted once: rejected in 0.16 s, "no system message changed from

prefix-stable to dynamic-head" (rq5/r2_context_bloat_r0.json): outside the one supported regression family, so the loop refuses instead of guessing;

the interrupted attempt of 03:10 UTC (AC lost), rq5/discarded/.

Operator activity during the run (rule above: reads, searches, edits, git and status checks of about 2 s or less are not competing compute): nvidia-smi, docker ps, docker stats --no-stream, one PowerShell process listing, curl of a public URL, file reads, greps, git status/git diff, all around 18:24-18:33 UTC. No test, lint, build or analysis script ran.

2026-10-12-gate-ci

2026-10-12: repair (RQ5) under the interval rule, and A/A with 5 runs per side

Code b13cf73 (clean). Lab vLLM, Qwen2.5-0.5B-Instruct, RTX 3050 Ti laptop, on AC at start (pre-flight 03:48 UTC) and at the end of both runs (manifests: on AC, 80 W limit); SM clock 780 MHz in every run, the same as 2026-10-09 and 2026-10-11. Both drivers check power after each attempt/trial (none was discarded).

rq5.md: 1/3 verified (interval acceptance rule)

Every repeat: the regression test failed on the PR head and passed on base; R1 measured

+830% to +865% p95 TTFT vs main (BLOCK, intervals entirely above +630%).

Strategy 1 (move the request id to the end of the user message): rejected 3/3 (point

+20.7% to +46.7%; twice WARN/BLOCK with the whole interval above the tolerance).

Strategy 2 (log the request id instead of sending it): point estimate +2.0%, +8.6%, +1.7%

vs main, prefix-cache hit 98% -> 98%, faster than the PR head every time. Accepted in repeat 0 (interval upper bound +12.5%) and rejected in repeats 1 and 2 because the interval's upper bound (+29.7%, +33.3%) was not below the 25% block threshold. The rule refused to accept a candidate it could not bound; the candidate was very likely fine.

3/3 full-suite audits of the repaired trees without a FAIL.

Reading: the interval rule does not accept unproven candidates (no false accept), but at 3 runs x 34 requests it lacks the power to confirm a good repair most of the time. Under the 2026-10-09 point rule the same measurements would have been 3/3 verified.

gate_noise_r5.md: A/A with 5 interleaved runs per side (8 trials)

0 false WARN or BLOCK; PASS 1/8, INCONCLUSIVE 7/8; the interval contained 0 in 8/8; median width 27.7 points (3 runs per side on 2026-10-11: 21.4). More runs per side did not narrow the interval: the statistic (median across runs of each run's p95 over 34 requests) is itself noisy, so the gate's decision statistic, not the number of runs, is the limit.

2026-10-11-live-clock

Same live run, two GPU clocks (laptop RTX 3050 Ti, lab vLLM, Qwen2.5-0.5B)

Two "Run it now" runs (POST /api/live/bench: c = 1, 2, 4, 8 x 20 requests, F1 workload fingerprint, prefix caching on), identical code and endpoint configuration. The only difference observed is the SM clock the driver chose under this load, after a reboot in between.

run	started (UTC)	SM clock	p95 TTFT c=1	c=2	c=4	c=8	prefix-cache hit
`runs/6e4ec05557ae`	2026-10-09 02:16	780 MHz	54 ms	66 ms	106 ms	137 ms	88-89%
`runs/bd1286537d74`	2026-10-11 03:25	1,957-1,971 MHz	31 ms	45 ms	55 ms	91 ms	89-91%

p95 TTFT is 32% to 48% lower at the higher clock. Both runs are single runs (no repeats), so this is an observation, not an effect estimate. It is why every GPU result records its clock and why comparisons are only made inside a run set that shares one: the FurnaceBench RQ3, RQ4 and RQ5 numbers were all measured at ~780 MHz, and the /lab reference ticks say so.

2026-10-11-gate-ci

2026-10-11: perf gate with 95% intervals (bench/run_gate_ci.sh)

Code: commit 6ee6cc1 (interval-based gate from 5e7c066), working tree clean. Lab vLLM, Qwen2.5-0.5B-Instruct, RTX 3050 Ti laptop; SM clock 780 MHz in every run (the same clock as the 2026-10-09 A/A, so the two are comparable). Unrelated containers (wc26-mlops-*, cal-api) were running, as in earlier campaigns.

gate_noise.md: VALID (12 trials, 34 requests per run)

On AC at start (pre-flight 22:58 UTC: "Best performance", not throttled) and at the end (manifest: on AC, 90%, 80 W GPU limit); every trial took 52-55 s.

Verdicts: PASS 7/12, INCONCLUSIVE 5/12, false WARN 0/12, false BLOCK 0/12.
The 95% interval contained the true difference (0) in 12/12; median width 21.4 points.
Largest A/A change: +13.4% (trial 2), and +13.0% (trial 10). The 2026-10-09 A/A

reported "within 9.7%" from 12 trials; that was one sample, not a bound. Under the point rule in use until 2026-10-11 (WARN above 10%), trials 2 and 10 would have been false WARNs (2/12). Under the interval rule both are INCONCLUSIVE.

gate_noise_x4.md: INVALID, kept as recorded (do not use)

Meant to test whether 4x the requests per run (136) narrows the interval. The laptop lost AC power during this run: the manifest at the end says on battery (78%, GPU limit 30 W), the battery fell from 90% to 78% between 23:12 and 23:26 UTC, and trials 3-6 took 148-157 s instead of 131-132 s. When AC was lost cannot be determined, so no trial is usable. The SM clock read 780 MHz throughout, but the power limit changed (80 W to 30 W). gate_noise.py checked power only at start; since commit 767c2bf it checks after every trial and stops.

Observation that motivated a re-run (not a result): the spread between runs of the same side, not the number of requests in a run, dominates the interval, so more runs rather than more requests per run is the likely fix. To be measured on AC.

rq5: NOT RUN

rq5.py refused to start: "refusing to benchmark on battery". Repair under the interval acceptance rule is therefore not measured yet; the most recent repair result remains 2026-10-09-gate-warmup-v2/rq5.md (point rule: 3/3 verified at -2.0% to +8.5% vs main).

2026-10-11-external

2026-10-11: held-out set 4, Scan + Forge + Guard on three independent apps

Labels and the validation driver (with its expected outcomes) were committed in f0fce9d before any scan. No scanner or reconstruction code changed between that commit and the first scan, so rq1_set4_first_scan.* and external_validation_first_run.* are held-out results. Apps (MIT, pinned commits in bench/ground_truth/): rag-postgres-openai-python (FastAPI, pgvector hybrid RAG, OpenAI Agents SDK), tech-trends-chatbot (FastAPI + SSE, Redis vector store, tool calling; a different author), openai-chat-backend-fastapi (minimal Responses API backend).

Scan (RQ1, rq1_set4_first_scan.md)

category	held-out set 4	for comparison: set 3 (2026-10-09)
LLM call sites	2/9 recall, precision 1.00	8/9, precision 1.00
routes	10/13, precision 1.00
serving engine	2/3
models	0/6
retrievers, tools	0/2, 0/2
attribute accuracy	25/26 (empty scan: 15/21)

Why calls were missed (each is a construct the extractor does not resolve today):

rag-postgres: all five generation calls go through the OpenAI Agents SDK

(Runner.run / Runner.run_streamed with Agent(instructions=...)).

openai-chat-backend: the client is held in a module-level dict

(clients["openai"].responses.create).

Models: values come from os.getenv(..., default) branches and pydantic-settings field

defaults; none were read.

Prompts score low partly by labeling convention: the scanner reported the prompt files (prompts/answer.txt) and per-call system slots; the labels name the Python attributes that load the files. By the matching rule fixed before scoring these count as misses and false positives; the labels were not changed after the scan. The report table was re-rendered after a header fix (set 4 column was unlabeled); the numbers were verified identical.

Source links, Forge, Guard (external_validation*.md)

Source links: every file evidence of every finding resolves to the file and line range

and contains its excerpt: 10 checked, 0 broken.

Guard on synthetic PRs (expected = correct behavior, fixed before the run): docs-only and

unrelated-code changes selected nothing, correct in 6/6. The prompt change (a per-request value prepended to the system prompt) was fully correct in 0/3: the prefix check and the prefix-instability signature were found in 2/3 apps, but the perf gate was not selected (no workflow reaches the prompt: the call sites above are missed, or the route reaches the LLM through asyncio.create_task); in rag-postgres the Agents SDK instructions were not recognized as a system prompt at all.

Repair applicability: both strategies apply on openai-chat-backend and make the prompt

prefix-stable again (re-analyzed); in tech-trends the system message is built in __init__ and used elsewhere, which the strategies do not handle.

Forge: plans 7 files on tech-trends (timeout test, prompt extraction, CI gate); 0 on the

others, each with the reason (no harness: control flow before the messages are built / no route reaches the system-message builder; timeouts not installable on clients created inside factory functions). Forge refuses rather than guesses.

Sandbox validation (run 2, _sandbox_before_deps_fix): failed on all three because the

sandbox only installed a root requirements.txt; these apps keep requirements in src/, src/backend/ (included from a root requirements-dev.txt) or Poetry only. Fix (Furnace change, so later Forge-validation numbers on set 4 are "after fixes"): the sandbox now installs the root requirements-dev.txt or requirements.txt with every -r/-c include that stays inside the repository (escapes refused, tested).

Run 3 (external_validation.md, after the fix): openai-chat-backend's own test suite now

runs in the sandbox, 5 passed and 1 failed (test_chat_stream, a snapshot mismatch on the unmodified code at the pinned commit: pre-existing, with openai>=1.108.1 unpinned); Forge requires a green baseline, so no PR. rag-postgres: dependency install fails building psycopg2 from source in the slim image, and its tests need a live Postgres, which a network-less sandbox cannot provide. tech-trends: Poetry-only dependencies are not installed; no PR.

What this says

Furnace generalizes where code calls an LLM SDK directly (set 3: 8/9) and degrades sharply on indirection that is common in newer apps (agent frameworks, clients in containers, background tasks): 2/9. It fails safe: no false positives on call sites or routes, no wrong selections on unrelated PRs, Forge refusing with reasons, and no PR without a green sandbox run. The next scanner work, measured on a new held-out set afterwards: Agents SDK calls and instructions, subscripted clients, create_task edges, env/settings defaults.

After fixes (set 4 is no longer held-out for these)

1. Sandbox dependency install (above): external_validation_after_deps_fix.*. 2. Guard selection: a detected prefix-instability signature now selects the benchmarks whose triggers include prompt changes even when no traced workflow reaches the prompt (guardian/impact.py, lost_stable_prefix). external_validation.md: the prompt scenario is correct in 2/3 apps (was 0/3), all scenarios 8/9; the remaining miss is rag-postgres, whose Agents SDK instructions are not recognized as a system prompt (a scanner gap, not a selection gap). F1's impact tests are unchanged.

Scanner change for set-4 misses, measured on held-out set 5

Change (code_intel/python_ast.py): Runner.run / run_streamed / run_sync are LLM calls when Runner comes from the agents package (OpenAI Agents SDK), with the model read from the module-level Agent(model=...) passed to it (also across modules and for Agent[Ctx](...)); a client held in a subscript (clients["openai"].responses.create) is matched by API suffix. Negative tests: a local Runner class and an unrelated registry["x"].get() are not calls.

Set 5 (openai/openai-cs-agents-demo, labels committed in f038d20 before the change; baseline with the unchanged scanner in rq1_set5_baseline_before_change.md): LLM call sites 0/3 -> 3/3 at precision 1.00, models 0/2 -> 1/2 (the streamed call's agent is chosen at runtime). Still missed: agent instructions are not extracted as prompts (0/8), and the ChatKit server's route does not trace to the call (workflows 0/1). One app with three call sites: evidence that the change generalizes beyond set 4, not a measurement of recall.

Regression check (rq1_after_agents_sdk.json vs 2026-10-09/rq1.json): F1 and sets 1-3 unchanged in every category. Set 4 (no longer held-out): call sites 2/9 -> 9/9.

Agent instructions as system prompts (after fixes; set 4 and 5 no longer held-out for this)

Agent(instructions=<text>) from the Agents SDK is analyzed like a system message (prefix stability; one key per agent, <scope>@<name>#system); instruction *functions* are skipped; module-level agents are included (module-level message dicts still are not). self.X / cls.X in a prompt is shared text when X is a class attribute somewhere in the repository (evaluated once at import). Results (external_validation.md, rq1_after_agent_prompts.md): Guard on set 4 synthetic PRs now 9/9 correct, including rag-postgres's prompt regression; the repair strategies do not handle agent instructions ("not applicable", reported), so that app gets a correct diagnosis but no automated fix. Prompt P/R on sets 4-5 stays low by the labeling convention noted above. F1 and sets 1-3 unchanged in every RQ1 category. The driver now treats file-level evidence (excerpt = the path) as a file-existence check; before, it reported one such link as broken.

2026-10-09-gate-warmup-v2

RQ5 with the warmed-up gate and two repair strategies (2026-10-09)

Perf gate: one unmeasured warmup wave before every measured run (A/A noise after the fix:

max |change| 9.7%, ../2026-10-09/gate_noise.md). Laptop RTX 3050 Ti on AC, SM clock ~780 MHz under this load (driver choice, not throttling: ../2026-10-06/NOTES.md, 2026-10-09 addendum).

Strategies tried in order. In all 3 attempts move_dynamic_to_suffix (values kept in the prompt)

was blocked at +44% to +50% p95 TTFT; move_dynamic_to_log (values logged instead of sent) passed at -2.0% to +8.5%, inside the measured noise, prefix-cache hit 98% -> 98%, full-suite audit clean.

Trade-off of the proposed fix, stated in the repair PR: the model no longer sees the request id

and timestamp. For R1 ("for debugging") that is the intent; for a value the model needs, a human would choose the suffix candidate, whose measured cost is listed beside it.

The LLM judge item is SKIPPED (no key configured), as in every campaign.

2026-10-09-gate-warmup

RQ5 rerun with the warmed-up perf gate (2026-10-09), suffix strategy only

Same RQ5 procedure as 2026-10-06, with one change to the perf gate: each measured run is now preceded by an unmeasured warmup wave (see ../2026-10-09/gate_noise*.md: without it the first wave after idle set the p95, and the base revision, always measured first, read ~2.4x slower).

Result: the only repair strategy then (move_dynamic_to_suffix) removes ~95% of R1's cost (p95 TTFT ~910 -> ~125 ms) but stays +40-47% above base (~87 ms): the gate blocks it, 0/3 verified. The 2026-10-06 "3/3 verified" was measured with the cold-start-biased gate. Kept unchanged as evidence; the follow-up with a second strategy is ../2026-10-09-gate-warmup-v2/.

Campaign conditions (read first)

Campaign notes, 2026-10-06 (directory pinned with FURNACE_BENCH_DATE=2026-10-06)

Power. The first RQ4 sweep ran on battery and was discarded (rq4_battery_invalid/NOTES.md).

A second sweep on AC was cut short by a session end and discarded (rq4_capped_partial/NOTES.md).

GPU clock cap. On AC, before and after a reboot, at 14% and at 56% battery, the laptop held

the RTX 3050 Ti at ~780-830 MHz mean SM clock under load (spot samples: P0, ~30-34 W, 80 W enforced power limit, no NVML throttle reason active; RQ4 telemetry recorded power peaks up to 64 W). The GPU's maximum is 2,100 MHz; the 2026-10-04 runs averaged 1,948 MHz and 72 W. This is a platform policy (OEM performance mode, Windows power mode or charger wattage), not something Furnace controls; it was not changed for this campaign.

Consequence. The final RQ3, RQ4 and RQ5 results in this directory ran at that capped clock.

Comparisons inside each RQ (configs, base vs PR vs repair) share the clock and are valid; absolute latencies are not comparable to the 2026-10-04 results. Each RQ4 run records its SM clock and AC state (rq4/rq4.json -> clocks, clock_spread_pct).

Host. Development continued on the same machine during the GPU runs: editor, a Next.js dev

server, the FastAPI dev server, the test suite (poe check, including short Docker sandbox tests) and Playwright screenshots. This adds CPU noise to client-side timing; repeats and revisions are interleaved so it spreads across configs rather than biasing one.

Addendum 2026-10-09: the clock was not a power cap

Measured on the same laptop on AC, Windows power mode "Best performance", driver 592.82:

A saturating load (fp16 4096x4096 matmul loop for 20 s in the vLLM image) ran at

1,830-1,852 MHz and 80 W (24.7 TFLOPS); the only active clock-event reason was the normal 80 W software power limit.

The lab vLLM serving Qwen2.5-0.5B under 16-48 concurrent requests ran at 780 MHz, ~30 W,

utilization ~80%, with "Idle" as the only active clock-event reason: no power, thermal or power-brake throttling.

So "capped" above is the wrong word: the driver chose ~780 MHz for this light, bursty serving load. The 2026-10-04 vLLM runs averaged 1,948 MHz; what made the driver boost then and not now is not identified. The consequence paragraph stands unchanged: comparisons inside each RQ share the clock; absolute latencies are at this clock. scripts/preflight.py --load reports the clock and the active reasons before a demo.

2026-10-11 (UTC), after a reboot: under the same vLLM load the driver chose 1,957-1,972 MHz at up to 60 W (only the "Idle" reason active). The clock this load gets is a driver state that changed across reboots. A live run at that clock was 32-48% faster at p95 TTFT than the same run at 780 MHz (../2026-10-11-live-clock/NOTES.md).

FurnaceBench report (2026-10-06)

Generated by uv run poe bench-report from the files in bench/results/2026-10-06 at commit 77528b5c76a33d78414c195e7a81519969586161 (working tree dirty). Hardware: NVIDIA GeForce RTX 3050 Ti Laptop GPU (4096 MB), driver 592.82, Windows-11-10.0.26300-SP0. Protocols: docs/furnacebench.md. Every number below is copied from a results file named in its section. Read first: campaign conditions in bench/results/2026-10-06/NOTES.md.

RQ1 Reconstruction

Sources: bench/results/2026-10-06/rq1_first_scan.md, bench/results/2026-10-06/rq1_set2_old_scanner.md, bench/results/2026-10-06/rq1.md.

category (P / R (tp/truth))	F1 (dev fixture)	set 1, first scan	set 1 after fixes (not held-out)	set 2, old scanner	set 2, fixed scanner
llm_call_sites	1.00 / 1.00 (1/1)	– / 0.00 (0/8)	1.00 / 1.00 (8/8)	1.00 / 0.20 (1/5)	1.00 / 0.20 (1/5)
models	1.00 / 1.00 (1/1)	– / 0.00 (0/4)	1.00 / 1.00 (4/4)	– / 0.00 (0/2)	– / 0.00 (0/2)
serving_engines	1.00 / 1.00 (1/1)	– / 0.00 (0/4)	1.00 / 1.00 (4/4)	– / 0.00 (0/3)	– / 0.00 (0/3)
retrievers	1.00 / 1.00 (1/1)	1.00 / 0.50 (1/2)	1.00 / 1.00 (2/2)	n/a	n/a
tools	1.00 / 1.00 (1/1)	– / 0.00 (0/1)	– / 0.00 (0/1)	n/a	n/a
prompts	1.00 / 1.00 (1/1)	0.20 / 0.33 (1/3)	1.00 / 0.33 (1/3)	0.00 / 0.00 (0/4)	0.00 / 0.00 (0/4)
workflows	1.00 / 1.00 (2/2)	– / 0.00 (0/9)	– / 0.00 (0/9)	1.00 / 0.20 (1/5)	1.00 / 0.20 (1/5)
contradictions	1.00 / 1.00 (1/1)	– / 0.00 (0/1)	– / 0.00 (0/1)	0.00 / 0.00 (0/1)	0.00 / 0.00 (0/1)

attribute accuracy	F1 (dev fixture)	set 1, first scan	set 1 after fixes (not held-out)	set 2, old scanner	set 2, fixed scanner
scanner	13/13	19/22	46/46	15/16	15/16
empty scan (baseline)	4/7	15/20	15/20	13/14	13/14

Confidence ECE, first scan (old scanner; dev + set 1): 0.082 over n = 12 decidable claims.

Confidence ECE, current scanner (all apps): 0.054 over n = 45 decidable claims.

RQ2 Evaluators

Source: bench/results/2026-10-06/rq2.md.

check (seeded)	recall	positives / controls
citation_required	1.000	122 / 78
duplicate_action	1.000	2 / 2
forbidden_pattern	1.000	200 / 45
prompt_prefix_stable	1.000	80 / 80
side_effect_requires_approval	1.000	3 / 3

Seeded results verify each check against its own specification; they are expected to be perfect for deterministic checks and say nothing about agreement with human judgment.

RQ3 Impact-aware selection

Source: bench/results/2026-10-06/rq3.md.

metric	value
scenarios	13
suite items executed (targeted)	46%
wall time full → targeted (s)	964 → 502
endpoint-busy time full → targeted (s)	839 → 444
regression recall (full-suite FAILs caught)	3/3 = 1.00
warning recall	1/2

Perf-gate noise on 6 PRs that cannot affect serving (full-suite runs): p95 TTFT change -17.6% to +10.7% (docs_only -9.6%, r3b_chunk_regex_strip -17.6%, r4_approval_removed +10.7%, test_only -0.7%, dependency_bump -6.0%, health_payload +2.5%).

RQ4 Inference (laptop GPU)

Source: bench/results/2026-10-06/rq4/rq4.md (raw runs in rq4/runs/).

max-num-seqs	c	TTFT p95 change with prefix cache off, median [min, max] %	goodput lost, req/s	pairs	same sign
8	8	+319.8 [+291.6, +376.0]	+3.59 [+2.84, +4.10]	3	yes
8	16	+95.6 [+64.8, +97.4]	+0.80 [+0.75, +1.05]	3	yes
32	8	+303.6 [+287.9, +319.0]	+3.76 [+2.74, +4.30]	3	yes
32	16	+432.7 [+395.9, +520.5]	+5.21 [+4.30, +5.51]	3	yes
128	8	+312.0 [+300.9, +371.8]	+3.63 [+2.63, +4.26]	3	yes
128	16	+411.8 [+394.7, +416.2]	+4.58 [+4.28, +4.71]	3	yes

Prefix-cache hit rate with caching on: 88.1–89.3% measured vs 89.0% predicted from the workload's shared prefix.

RQ5 Repair

Source: bench/results/2026-10-06/rq5.md (current rule strategy).

repeat	status	p95 TTFT base → PR, ms	repair p95 TTFT, ms	repair hit rate	full-suite audit
0	verified	203.1 → 1043.1 (+414%)	312.9 (+54.1%, warn)	98% → 94%	neutral
1	verified	231.6 → 1009.9 (+336%)	269.8 (+16.5%, warn)	98% → 94%	neutral
2	verified	238.2 → 1071.3 (+350%)	275.7 (+15.7%, warn)	98% → 94%	neutral

current strategy: 3/3 verified; 3/3 audits without a FAIL.

First run (bench/results/2026-10-06/rq5_rule_v1.md): the rule moved the per-request value to the end of the *system* message, which still sits before the retrieved context. Kept for the record:

repeat	status	p95 TTFT base → PR, ms	repair p95 TTFT, ms	repair hit rate	full-suite audit
0	rejected	244.1 → 1100.2 (+351%)	436.4 (+78.7%, block)	98% → 83%	failure
1	rejected	257.0 → 1076.5 (+319%)	445.2 (+73.2%, block)	98% → 83%	failure
2	verified	235.1 → 1052.2 (+348%)	437.9 (+86.2%, warn)	98% → 83%	failure

first strategy: 1/3 verified; 0/3 audits without a FAIL.

Negative results

RQ1: the first scan of held-out set 1 (3 OSS apps) found 0/8 LLM call sites. Probes (minimal files, same scanner) confirmed two causes: SDK calls were extracted inside functions but not at module level (Streamlit scripts), and LangChain wrappers were not extracted as call sites at all. Both were fixed afterwards; set 1 is therefore no longer held-out, and set 2 (labeled before the fixes) is the clean test.
RQ1 (set 2): the scanner fixes motivated by set 1 changed nothing on set 2 (LLM call sites 1/5 before and after). Set 2 uses SDKs the scanner has no rules for (Replicate, llama.cpp) and Azure OpenAI behind a custom base_url: coverage is bounded by hand-written SDK rules, so every unseen SDK is a miss.
RQ1 (set 2, llama2): false positives: model: ['Llama 2', 'Llama2-7B', 'llama2-chatbot', 'llama2.streamlitapp.com', 'Llama2-13B', 'Llama2-70B']. Left unfixed so set 2 stays a clean test.
RQ1 (set 2, openai-chat-app-quickstart): false positives: tests/conftest.py::mock_openai_responses_stream.mock_stream#system; tests/test_app.py::test_chat_stream_text#system; tests/test_app.py::test_chat_stream_text_history#system; wrong attributes: rag.present=True. Left unfixed so set 2 stays a clean test.
RQ1 (set 1 after fixes): recall below 0.5 in: tools (0/1), prompts (1/3), workflows (0/9), contradictions (0/1).
RQ1 (set 2): recall below 0.5 in: llm_call_sites (1/5), models (0/2), serving_engines (0/3), prompts (0/4), workflows (1/5), contradictions (0/1).
RQ2: citation_required passes answers that cite a retrieved but wrong document (31/31 seeded cases): it is a format/provenance check, not a groundedness check.
RQ3: seeded regression r2_context_bloat (R2 context bloat) produced no FAIL even in the full suite; the suite cannot see it on this setup.
RQ3: seeded regression r3_citation_strip (R3 citations dropped) produced no FAIL even in the full suite; the suite cannot see it on this setup.
RQ3: seeded regression r3b_chunk_regex_strip (R3b ineffective tag strip) produced no FAIL even in the full suite; the suite cannot see it on this setup.
RQ3: seeded regression r5_prefix_caching_off (R5 prefix caching off (serving config)) produced no FAIL even in the full suite (WARN: check:citation_required); the suite cannot see it on this setup.
RQ3: in the R3 run the base revision met the citation contract in 0/34 answers, so a citation regression could not show: a floor effect, not a pass. Across the runs the base rate varied from 0/34 to 2/34 with identical base code (serving nondeterminism); the citation check warned on r5_prefix_caching_off, which does not change prompts. The 2026-10-04 live Guard run (bench/results/2026-10-04-guard, serving model not recorded there) had 5/34 on base.
RQ3: the perf gate's +10% WARN threshold is inside run-to-run noise on this host (-17.6% to +10.7% on PRs that cannot affect serving); it warned on r4_approval_removed. None of these PRs was blocked: BLOCK also requires separated runs.
RQ5 (current strategy): in repeat 0 the gate passed the repair as WARN at +54.1% p95 TTFT because its runs overlapped the base runs; a BLOCK needs separated runs, so a large but noisy regression can pass.
RQ5: r2_context_bloat: no repair strategy (status rejected).
RQ5 (first strategy): 2/3 R1 repair attempts were rejected by the gate.
RQ5 (first strategy): the full-suite audit of the repaired tree failed in 3/3 repeats (bench:chat_perf_gate).
RQ5 (first strategy): in repeat 2 the gate passed the repair as WARN at +86.2% p95 TTFT because its runs overlapped the base runs; a BLOCK needs separated runs, so a large but noisy regression can pass.

Not run

RQ2 human-labeled agreement: queue of 60 real answers is ready (bench/labels/f1_queue.jsonl), not labeled yet.
RQ2 LLM judge TPR/TNR: no Groq/OpenRouter key configured.
RQ1 F2 (TypeScript agent app): no TypeScript extractor yet.
RQ1 +LLM synthesis ablation: LLM synthesis step not built.
RQ4 Kaggle T4 (Qwen2.5-1.5B fp16 vs AWQ) and Ollama comparison: not run.
RQ4 vllm bench serve cross-check: not run.
RQ5 LLM-patch strategy: not built.

Earlier single-run results (2026-10-04, not repeated here)

bench/results/2026-10-04-r1-derisk: removing a 1,000-token shared prefix, c=16 TTFT p95 752 → 1,577 ms (1 repeat).
bench/results/2026-10-04-f1-quality: citation contract met 3/89 (0.5B) and 9/88 (1.5B AWQ) on real F1 traffic.
bench/results/2026-10-04-guard: live Guard checks for R1-R4 and controls (1 run each).