What breaks when there is no upstream
A cloud assistant is allowed to be lazy in ways that are invisible until you take the network away. It can call a bigger model when it is unsure, fetch a fresh index, look something up, and lean on a fleet where a slow request just goes to another node. Put the entire system on a single machine in a building with no outbound connection and every one of those affordances disappears at once.
We built this because the customers who most need an assistant are frequently the ones least able to send their data anywhere. A county attorney handling privileged material and a clinic handling patient records have the same objection, and it is not solved by a compliance certificate. It is solved by the data not leaving.
The four things that had to change
- No escalation to a larger model, so the local model has to be right the first time
- No shared cache across tenants, so cold-start behaviour has to be good
- Speech, reasoning and retrieval share one accelerator instead of separate fleets
- Updates arrive on a schedule, not continuously, so the system must be stable between them
Sharing one accelerator between speech and thought
In the cloud, transcription, synthesis and reasoning run on separate hardware and nobody thinks about it. On a single appliance they contend, and the contention shows up exactly where it hurts most: a caller is speaking, transcription needs the accelerator now, and a long reasoning request is halfway through occupying it.
The scheduler treats live audio as strictly preemptive. Speech work interrupts reasoning work rather than queueing behind it, and interrupted reasoning resumes from its last checkpoint instead of restarting. This costs throughput on batch work, which is the right trade: nobody notices a report finishing ninety seconds later, and everybody notices a pause on a phone call.
Quantization where it does not show
Getting a useful model onto one machine means quantizing, and the naive approach degrades exactly the capabilities that matter for this work. We quantize non-uniformly, keeping higher precision in the layers that carry instruction-following and tool-use behaviour, and accepting more loss in layers whose degradation shows up as slightly less elegant prose.
Measured on OpsBench, the on-premises configuration scores within a few points of the hosted configuration on task completion and escalation correctness, and noticeably lower on summary fluency. That is the trade we chose deliberately. A summary that reads a little flatter is acceptable; an assistant that books the wrong appointment is not.
Operating something nobody can reach
The hardest part was not inference. It was everything an operations team normally does over a network. You cannot ship a hotfix, tail a log, or roll back from a laptop. Whatever is on that machine has to survive until someone physically visits, and it has to fail in ways the customer’s own staff can understand without calling anyone.
Diagnostics the customer owns
The appliance keeps a full local record of what it did and why, readable by the customer’s own staff without a vendor account. When something goes wrong, the people in the building can see the failure rather than filing a ticket into the dark. This was originally a procurement requirement and turned out to be a better operational model than the one we had.
| Hosted | On-premises appliance | |
|---|---|---|
| Task completion (OpsBench) | 0.91 | 0.88 |
| Escalation precision | 0.89 | 0.87 |
| Median first audio | 240 ms | 290 ms |
| Data leaving the building | n/a | none |
The last row is the entire point, and it is the row that cannot be achieved with a stronger certification, a better contract, or a regional datacenter. Either the material crosses your network boundary or it does not, and for a category of customer that distinction decides whether they can adopt anything at all.
What we would tell anyone attempting this
Budget for the operations problem, not the inference problem. Getting a capable model running on one machine is a known quantity now. Making that machine supportable by people who have never met you, in a building you cannot reach, updated four times a year, is the work that actually takes the time, and it is the work that determines whether the deployment survives its second year.
What the first installations taught us
Some of the most useful lessons had nothing to do with models. Delivering hardware into a building, connecting it to systems that were configured a decade ago by someone who has since retired, and handing it to staff who did not ask for it, is a different discipline from shipping software, and we were not initially good at it.
The integration is the project
Every deployment spends the majority of its calendar time on connecting existing systems, not on the assistant. Records live in formats chosen long ago, permissions are encoded in conventions nobody wrote down, and the person who understands the exceptions is one specific member of staff. This is normal, it is not a sign of a badly run organisation, and any plan that treats it as a two-day task is wrong.
We now scope integration first and the assistant second, which sounds obvious and inverts how the conversation usually starts. Buyers ask what the AI can do. The useful question is what it will be allowed to see, and the answer to that determines almost everything about what the deployment is worth.
Training the people, not the model
- Staff need to know what it will not do as much as what it will
- The first fortnight generates the same questions in the same order everywhere
- One person becomes the internal owner whether or not you plan for it
- Written policy about what may be delegated matters more than any setting in the product
The deployments that go well are the ones where somebody inside the organisation takes ownership early. The deployments that stall are the ones where the system is installed correctly, works correctly, and nobody has decided what it is for. That is not a technical failure and it cannot be fixed with a better model.
Being honest about the trade
On-premises is not strictly better. It is slower to update, it costs more per user at small scale, and it gives up the ability to escalate to a larger model when the local one is unsure. What it gives in exchange is that the data does not move, and for a meaningful set of organisations that single property is the difference between adopting something and adopting nothing.