Loading results from the API… If the hosted API was asleep, this can take up to a minute.
Loading results from the API… If the hosted API was asleep, this can take up to a minute.
Furnace tests every pull request on your own model server, blocks the ones that make responses slower, and drafts a tested fix. Your team just reviews it.
Shown end to end on a public GitHub repository: the check run and the draft fix
Pull request · fixture support app
Add request id and timestamp to the system prompt
Time to first token on the model server
Blocked before merge
+621%, 90% interval +590% to +653%
Fix drafted and tested
72 ms · cache 98% · awaiting review
Trial 1 of 3 repeated trials of one scenario, laptop RTX 3050 Ti, vLLM, Qwen2.5-0.5B. Full walkthrough
The problem
A request ID added to the top of a prompt. The answers didn’t change, so every answer check passed. But the model server could no longer reuse its cached work, so every request waited longer.
98% → 0%
of the prompt reused from the server’s cache
69 → 494 ms
time to first token (geometric mean)
Recorded on a fixture app, laptop RTX 3050 Ti, vLLM, Qwen2.5-0.5B (trial 1 of 3).
How it works
GitHub App · installed on one repository
your-org / support-assistant
Never merges. Never pushes to your default branch.
Evidence
0/36
false alarms on unchanged code
6/6
cache regressions blocked
3/3
fixes verified, one scenario repeated
15/18
small slowdowns left undecided
Two fixture apps, one laptop GPU, one small model. Every figure is read from a result file in the repository. The GitHub flow (check run, draft fix, independent verification) was shown once on a public repository; see the Demo page.
FAQ
Often, yes. promptfoo can fail a test above a per-call latency threshold, and LangSmith records first-token percentiles. Furnace is narrower. It compares two revisions on your own server in interleaved runs, decides with an interval instead of a fixed threshold, reads the server’s cache hit rate, points to the line that changed, and for one kind of regression drafts a fix. It does not judge answer quality, so keep your eval tool for that.
The decision rule was frozen before it was validated. On unchanged code it raised 0 false alarms in 36 trials. It blocked the known cache regression in 6 of 6. It is not perfect: smaller slowdowns were left inconclusive in 15 of 18 trials, so catching regressions just above the 10% tolerance is not shown. Validation report
It works for Python apps that build prompts in code and call a model you serve (measured on vLLM). It needs a runner on a machine with Docker and a GPU. It can repair one regression family: per-request values at the start of a prompt. Not yet: TypeScript, latency on hosted APIs, and other repairs, which it refuses rather than guesses. The GitHub flow ran end to end once on a public repository (a check run on the right commit, a draft fix, real webhooks); that is one live trial, not a track record.
On your machine. The runner downloads your repository with a token scoped to that one repository and runs your tests in a container with no network, read-only, without root. Prompts go only to your own model server. Security controls and their tests
Not yet. Furnace is an early prototype, verified on two fixture apps on one laptop GPU. We are looking for a few design partners who serve their own models to try it on one repository, for free.
This is untested. The leading idea is a per-repository subscription for the hosted control plane, with the runner on the customer’s own hardware. The pilots exist partly to learn whether teams would pay, and for what.
A free pilot, set up with you in one session. You keep everything it finds.