The number that decides whether a call feels human
A person on a phone call will tolerate roughly a third of a second of silence before they start talking again. Past that, they assume the line dropped, they repeat themselves, or they hang up. Every assistant that answers a real phone lives or dies on that budget, and the budget is not negotiable: it is set by human conversational reflex, not by what a model is capable of.
The awkward part is that the budget has to cover everything. Speech has to be transcribed, the transcript has to be understood, whatever the caller asked for has to be looked up in a real system, an answer has to be composed, and that answer has to be turned back into audio. A model that reasons beautifully in four seconds is useless here. A model that answers in eighty milliseconds and gets the address wrong is worse than useless.
Ruby Voice runs the full loop at a median of 240 milliseconds from end of caller speech to first audio out. That figure is measured on production traffic over real carrier connections, not in a datacenter with a synthetic waveform. It includes the network, which is the part most benchmarks quietly leave out.
Where the time actually goes
Breaking the budget down was the first useful thing we did. Once we had per-stage timings on live calls rather than estimates, it became obvious that the model was not the dominant cost. Transcription finalization and the round trip to whatever system held the answer were together larger than inference, and both had been treated as fixed.
- Speech finalization: 60 to 90 ms, dominated by waiting to confirm the caller had actually stopped
- Retrieval from a connected system: 40 to 120 ms, almost entirely network and query planning
- First token from the model: 55 to 80 ms
- Speech synthesis to first audio frame: 30 to 45 ms
Stated that way, the optimization targets stop being mysterious. The model was already fast enough. What was slow was everything arranged around it, and most of that was arranged serially for no reason other than that it was easier to write.
Speculating on the caller, not just the tokens
Speculative decoding is standard practice for making a large model emit tokens faster: a small model guesses ahead, the large model verifies in a batch, and you keep whatever survives. We use it, and it accounts for a meaningful share of the token-rate improvement in Ruby Voice. But token rate was never our binding constraint, so on its own it moved the number very little.
The larger win came from speculating one layer up. Long before a caller finishes a sentence, the shape of what they need is usually clear. Someone who has said "I wanted to check on" is going to ask about an order, an appointment or an invoice, and all three require a lookup we could already be running. So we start running it.
Retrieval that begins mid-sentence
While the caller is still speaking, a small classifier proposes the two or three most likely intents and their associated lookups, and those queries are issued immediately against the connected systems. By the time the transcript finalizes, the answer is usually already in memory. If the guess was wrong we discard the result and pay the normal cost, but we were going to pay that cost anyway.
The economics work because reads are cheap and latency is expensive. Roughly seventy per cent of speculative lookups are used. The thirty per cent that are discarded cost a database read that nobody waits on. Trading wasted reads for saved milliseconds is a bargain when the milliseconds are the thing the caller can feel.
This does mean the system must never take a speculative action. Reads run ahead; writes never do. Nothing is booked, cancelled, charged or sent on the strength of a half-finished sentence, and the boundary between the two is enforced in the tool layer rather than left to the model to respect.
Reasoning without making the caller wait for it
The harder cases are the ones that genuinely need thought: a scheduling conflict across three calendars, a quote that depends on rules with exceptions, a caller whose account state contradicts what they are describing. These are exactly the calls where an assistant earns its place, and they are exactly the calls where a reasoning model will blow through a 300 millisecond budget.
The resolution is that a conversation is not a single response. Ruby Voice answers immediately with what it already knows while Ruby-Horizon keeps reasoning behind it, in the same way a competent person says "let me pull that up" and then actually pulls it up. The first audio goes out inside the budget; the substantive answer arrives a beat later, inside speech that was already flowing.
Held turns, not filler
We were careful to make this a real mechanism rather than a stalling phrase. The model emits a short grounded acknowledgement that references something specific the caller said, which buys 700 to 1,100 milliseconds of natural speech. Reasoning continues against that window. If it completes early the assistant moves on; if it needs longer it says so honestly rather than looping.
Callers do not experience this as latency. In blind listening tests, held turns were rated as more natural than the low-latency responses they replaced, because a small, specific pause before a considered answer is what a competent human does. The version that answered instantly and shallowly was rated worst, which was not the result we expected when we started.
What it costs
Reasoning inside a held turn is not free, and we cap it. A turn that cannot resolve within its window escalates rather than extending, and escalation is a first-class outcome instead of a failure path. The assistant tells the caller it is bringing in a person, and hands over with the conversation summarized so the caller does not start again.
| Previous generation | Ruby Voice | |
|---|---|---|
| Median first audio | 610 ms | 240 ms |
| 95th percentile first audio | 1,340 ms | 520 ms |
| Turns requiring a held turn | n/a | 11% |
| Escalations to a person | 9.4% | 6.1% |
The escalation figure is the one we care about most. A faster assistant that hands off more often has not improved anything. Handing off less while answering faster is the only combination that means the underlying work actually got better rather than just quicker.
Latency is a systems property
One thing we would emphasise to anyone measuring this: latency is a property of the whole system, and the model is usually not the largest term. We spent an embarrassing amount of early effort making inference faster while a synchronous call to a customer scheduling system sat in the critical path costing four times as much. The profile told us that immediately; our intuition had told us the opposite for months.
The general lesson is that the components you built yourself are the ones you instrument, and the components you integrated are the ones you assume are fast. In practice the integrations are where the time goes, because they were designed for a web form where 400 milliseconds is imperceptible rather than for a conversation where it is most of the budget.
Measuring on the carrier, not the loopback
We now measure exclusively on real carrier paths. Benchmarks run against a local audio file will happily report figures that no caller will ever experience, because they omit jitter buffering, packet loss recovery and the codec negotiation that a real telephone connection imposes. Those add between 40 and 90 milliseconds that are entirely outside the model and entirely inside the caller experience.
This is why we publish median and 95th percentile rather than a single number. The median is what a demo shows. The 95th percentile is what a receptionist experiences on the fortieth call of the morning, and it is the figure that determines whether they trust the system enough to stop hovering over it.
What we are working on next
- Cutting speech finalization further by predicting turn ends from prosody rather than silence duration
- Running the intent classifier and the first retrieval on the same forward pass
- Extending held turns to multi-step tasks without them becoming stalling
- Reducing the 95th percentile, which is where the remaining discomfort lives
None of these will move the median much. All of them should narrow the distribution, and a narrow distribution is what makes a system feel dependable rather than occasionally impressive. Consistency has turned out to matter more to the people using this every day than the headline figure does.