Loading results from the API… If the hosted API was asleep, this can take up to a minute.
Loading results from the API… If the hosted API was asleep, this can take up to a minute.
Inference Lab · fixture support-rag-py (F1)
F1's traffic was fingerprinted from its own call log, then replayed against vLLM with prefix caching on and off and three scheduler sizes. Every number on this page is read from bench/results/2026-10-06; nothing is simulated except where labeled.
Loading the recorded run…
4 concurrency levels × 20 requests on the F1 workload fingerprint, against the lab endpoint. One run at a time.
Live runs need the lab GPU, which this deployment does not have; the reference ticks are the recorded RQ4 results. The live demo runs on the laptop.
Reference ticks: median over 3 paired repeats, max-num-seqs 32, bench/results/2026-10-06/rq4, GPU SM clock 780–830 MHz. The live run uses the endpoint as it is configured now (vLLM default: prefix caching on); its measured hit rate and GPU clock are in the table. Latency follows the clock the driver picks, so compare the ticks with the dots only at similar clocks.
NVIDIA GeForce RTX 3050 Ti Laptop GPU (4096 MB) · vllm 0.30.0 · Qwen/Qwen2.5-0.5B-Instruct · image sha256:8a69ffad015f · commit 3dfa766fb1
GPU SM clock during runs 780–830 MHz (spread 6.0%), all runs on AC: yes. Comparisons between configs share this clock; absolute latencies depend on it.
# Campaign notes, 2026-10-06 (directory pinned with FURNACE_BENCH_DATE=2026-10-06) - **Power.** The first RQ4 sweep ran on battery and was discarded (`rq4_battery_invalid/NOTES.md`). A second sweep on AC was cut short by a session end and discarded (`rq4_capped_partial/NOTES.md`). - **GPU clock cap.** On AC, before and after a reboot, at 14% and at 56% battery, the laptop held the RTX 3050 Ti at **~780-830 MHz mean SM clock** under load (spot samples: P0, ~30-34 W, 80 W enforced power limit, no NVML throttle reason active; RQ4 telemetry recorded power peaks up to 64 W). The GPU's maximum is 2,100 MHz; the 2026-10-04 runs averaged 1,948 MHz and 72 W. This is a platform policy (OEM performance mode, Windows power mode or charger wattage), not something Furnace controls; it was not changed for this campaign. - **Consequence.** The final RQ3, RQ4 and RQ5 results in this directory ran at that capped clock. Comparisons inside each RQ (configs, base vs PR vs repair) share the clock and are valid; absolute latencies are not comparable to the 2026-10-04 results. Each RQ4 run records its SM clock and AC state (`rq4/rq4.json` -> `clocks`, `clock_spread_pct`). - **Host.** Development continued on the same machine during the GPU runs: editor, a Next.js dev server, the FastAPI dev server, the test suite (`poe check`, including short Docker sandbox tests) and Playwright screenshots. This adds CPU noise to client-side timing; repeats and revisions are interleaved so it spreads across configs rather than biasing one. ## Addendum 2026-10-09: the clock was not a power cap Measured on the same laptop on AC, Windows power mode "Best performance", driver 592.82: - A saturating load (fp16 4096x4096 matmul loop for 20 s in the vLLM image) ran at **1,830-1,852 MHz and 80 W** (24.7 TFLOPS); the only active clock-event reason was the normal 80 W software power limit. - The lab vLLM serving Qwen2.5-0.5B under 16-48 concurrent requests ran at **780 MHz, ~30 W**, utilization ~80%, with **"Idle" as the only active clock-event reason**: no power, thermal or power-brake throttling. So "capped" above is the wrong word: the driver chose ~780 MHz for this light, bursty serving load. The 2026-10-04 vLLM runs averaged 1,948 MHz; what made the driver boost then and not now is not identified. The consequence paragraph stands unchanged: comparisons inside each RQ share the clock; absolute latencies are at this clock. `scripts/preflight.py --load` reports the clock and the active reasons before a demo. 2026-10-11 (UTC), after a reboot: under the same vLLM load the driver chose **1,957-1,972 MHz at up to 60 W** (only the "Idle" reason active). The clock this load gets is a driver state that changed across reboots. A live run at that clock was 32-48% faster at p95 TTFT than the same run at 780 MHz (`../2026-10-11-live-clock/NOTES.md`).
The replay generator draws prompt lengths and output lengths from this fingerprint (outputs are fixed per request with ignore_eos), so load shape is synthetic while every distribution is measured. Source: bench/results/2026-10-04-f1-quality/workload-f1-traces-qwen05b.yaml
Line: median across repeats. Band: min-max across repeats.
Line: median across repeats. Band: min-max across repeats.
Line: median across repeats. Band: min-max across repeats.
Line: median across repeats. Band: min-max across repeats.
Same seed (same request sequence) within each repeat; median [min, max] over repeats.
| max-num-seqs | concurrency | TTFT p95 change, % | goodput lost, req/s | pairs | same sign in every pair |
|---|---|---|---|---|---|
| 8 | 1 | +184 [176, 187] | +0.53 [0.25, 0.59] | 3 | yes |
| 8 | 2 | +314 [265, 323] | +0.82 [0.53, 0.92] | 3 | yes |
| 8 | 4 | +317 [237, 344] | +2.01 [1.34, 2.40] | 3 | yes |
| 8 | 8 | +320 [292, 376] | +3.59 [2.84, 4.10] | 3 | yes |
| 8 | 16 | +96 [65, 97] | +0.80 [0.75, 1.05] | 3 | yes |
| 32 | 1 | +193 [179, 201] | +0.57 [0.28, 0.63] | 3 | yes |
| 32 | 2 | +304 [296, 314] | +0.72 [0.52, 0.86] | 3 | yes |
| 32 | 4 | +261 [234, 311] | +2.11 [1.30, 2.46] | 3 | yes |
| 32 | 8 | +304 [288, 319] | +3.76 [2.74, 4.30] | 3 | yes |
| 32 | 16 | +433 [396, 521] | +5.21 [4.30, 5.51] | 3 | yes |
| 128 | 1 | +181 [177, 197] | +0.53 [0.25, 0.64] | 3 | yes |
| 128 | 2 | +310 [303, 313] | +0.66 [0.57, 0.85] | 3 | yes |
| 128 | 4 | +241 [216, 330] | +1.83 [1.20, 2.48] | 3 | yes |
| 128 | 8 | +312 [301, 372] | +3.63 [2.63, 4.26] | 3 | yes |
| 128 | 16 | +412 [395, 416] | +4.58 [4.28, 4.71] | 3 | yes |
| config | c | TTFT p50 | TTFT p95 | TTFT p99 | TPOT p50 | E2E p95 | req/s | goodput | tok/s | hit rate | failures |
|---|---|---|---|---|---|---|---|---|---|---|---|
| pc-off_seqs-128 | 1 | 138 | 140 | 143 | 9.6 | 1307 | 2.10 | 2.10 | 76 | – | 0.0% |
| pc-off_seqs-128 | 2 | 152 | 261 | 273 | 16.0 | 2302 | 2.44 | 2.44 | 99 | – | 0.0% |
| pc-off_seqs-128 | 4 | 165 | 285 | 435 | 22.6 | 3295 | 3.50 | 3.50 | 148 | – | 0.0% |
| pc-off_seqs-128 | 8 | 175 | 656 | 923 | 34.5 | 4990 | 4.37 | 4.00 | 185 | – | 0.0% |
| pc-off_seqs-128 | 16 | 248 | 1547 | 1766 | 45.1 | 6682 | 5.90 | 4.62 | 265 | – | 0.0% |
| pc-off_seqs-32 | 1 | 139 | 143 | 145 | 9.7 | 1320 | 2.12 | 2.12 | 76 | – | 0.0% |
| pc-off_seqs-32 | 2 | 152 | 260 | 268 | 15.2 | 2387 | 2.41 | 2.41 | 98 | – | 0.0% |
| pc-off_seqs-32 | 4 | 165 | 293 | 444 | 22.8 | 3273 | 3.50 | 3.44 | 148 | – | 0.0% |
| pc-off_seqs-32 | 8 | 181 | 652 | 904 | 34.8 | 4935 | 4.33 | 3.90 | 183 | – | 0.0% |
| pc-off_seqs-32 | 16 | 282 | 1568 | 1786 | 50.9 | 6995 | 5.38 | 4.20 | 239 | – | 0.0% |
| pc-off_seqs-8 | 1 | 138 | 140 | 142 | 9.5 | 1275 | 2.13 | 2.13 | 77 | – | 0.0% |
| pc-off_seqs-8 | 2 | 152 | 263 | 271 | 15.6 | 2470 | 2.39 | 2.39 | 97 | – | 0.0% |
| pc-off_seqs-8 | 4 | 164 | 384 | 446 | 22.6 | 3210 | 3.53 | 3.47 | 149 | – | 0.0% |
| pc-off_seqs-8 | 8 | 176 | 655 | 911 | 34.1 | 4704 | 4.34 | 3.98 | 190 | – | 0.0% |
| pc-off_seqs-8 | 16 | 1824 | 2244 | 2564 | 32.1 | 6813 | 4.31 | 0.22 | 194 | – | 0.0% |
| pc-on_seqs-128 | 1 | 35 | 50 | 50 | 9.7 | 1270 | 2.67 | 2.67 | 96 | 88.7% | 0.0% |
| pc-on_seqs-128 | 2 | 48 | 64 | 72 | 14.5 | 1798 | 3.10 | 3.10 | 129 | 88.7% | 0.0% |
| pc-on_seqs-128 | 4 | 62 | 89 | 104 | 16.2 | 2461 | 5.33 | 5.33 | 218 | 88.7% | 0.0% |
| pc-on_seqs-128 | 8 | 77 | 158 | 165 | 18.5 | 2479 | 7.63 | 7.63 | 323 | 89.2% | 0.0% |
| pc-on_seqs-128 | 16 | 115 | 303 | 303 | 26.2 | 3714 | 9.27 | 9.27 | 419 | 88.8% | 0.0% |
| pc-on_seqs-32 | 1 | 35 | 49 | 50 | 9.6 | 1229 | 2.70 | 2.70 | 96 | 88.7% | 0.0% |
| pc-on_seqs-32 | 2 | 48 | 64 | 69 | 14.6 | 1819 | 3.13 | 3.13 | 127 | 88.9% | 0.0% |
| pc-on_seqs-32 | 4 | 62 | 87 | 98 | 16.1 | 2344 | 5.56 | 5.56 | 217 | 88.7% | 0.0% |
| pc-on_seqs-32 | 8 | 76 | 159 | 160 | 18.6 | 2431 | 7.66 | 7.66 | 325 | 89.2% | 0.0% |
| pc-on_seqs-32 | 16 | 113 | 294 | 304 | 26.1 | 3623 | 9.41 | 9.41 | 427 | 88.8% | 0.0% |
| pc-on_seqs-8 | 1 | 35 | 49 | 50 | 9.8 | 1183 | 2.66 | 2.66 | 96 | 88.7% | 0.0% |
| pc-on_seqs-8 | 2 | 48 | 64 | 70 | 14.6 | 1883 | 3.25 | 3.25 | 128 | 88.9% | 0.0% |
| pc-on_seqs-8 | 4 | 61 | 86 | 103 | 16.0 | 2379 | 5.48 | 5.48 | 218 | 88.7% | 0.0% |
| pc-on_seqs-8 | 8 | 79 | 154 | 170 | 18.5 | 2452 | 7.57 | 7.57 | 324 | 89.2% | 0.0% |
| pc-on_seqs-8 | 16 | 892 | 1137 | 1157 | 19.0 | 3622 | 7.64 | 1.02 | 328 | 88.8% | 0.0% |