Interruption is not an error condition
Most voice systems treat being interrupted as something to detect and recover from: the caller talked over the prompt, so stop the audio and start listening. That framing is why so many of them feel wrong. In real conversation, interruption is not a fault. It is one of the main ways people steer, and roughly a third of the turns in our production call sample contain one.
Once we started transcribing what callers actually said when they cut in, the variety was the surprise. Only a minority were corrections. Many were agreement, encouragement to continue, a request to slow down, or an answer supplied before the question finished. Treating all of these identically, stop talking and start listening, throws away the information in the interruption itself.
Four kinds, and they want different things
- Backchannel: "mm-hm", "right", "sure". The caller wants you to keep going. Stopping is the wrong response.
- Correction: "no, the other address". Stop immediately and discard the current plan, not just the current sentence.
- Completion: the caller answers before you finish asking. Stop, accept the answer, do not re-ask.
- Redirection: "actually, before that". Stop, push the current thread, and be able to return to it.
An assistant that cannot tell these apart produces a specific and familiar failure: it stops dead when someone says "mm-hm", loses its place, and asks the question again. Callers read this as not listening, which is precisely backwards: the system stopped because it heard them.
Classifying the interruption before reacting to it
Ruby Voice classifies an interruption within about 120 milliseconds of onset, using the first phonemes together with the state of the assistant’s own utterance. Position matters enormously: the same sound at the start of a sentence and three-quarters of the way through it usually mean different things, and a classifier that ignores where it landed will get backchannels wrong most of the time.
Backchannels get no reaction at all beyond a small prosodic acknowledgement: the assistant keeps talking, slightly warmer. Corrections and redirections stop speech within 80 milliseconds. Completions stop speech and mark the pending question as answered, so the assistant does not immediately ask a thing it was just told.
Stopping cleanly is its own problem
Cutting audio mid-word sounds broken even when the decision was right, so speech halts at the nearest phoneme boundary and the buffered remainder is discarded rather than played out. More importantly, the model is told what it had already said aloud rather than what it had generated. Those two diverge during an interruption, and a model that believes it finished a sentence the caller never heard will build its next turn on something that did not happen.
That distinction between generated and delivered turned out to be the single largest source of confusion in the previous generation. It produced the maddening pattern where an assistant refers back to information it never actually spoke, and the caller has no idea what it is talking about.
What changed on real calls
We measured on production traffic across a mixed sample of trades, clinical and front-office deployments, comparing the previous generation against Ruby Voice on the same call types. The headline is not the classification accuracy, which is a proxy. It is that callers stopped repeating themselves, which is the behaviour that tells you a system is genuinely listening.
| Previous generation | Ruby Voice | |
|---|---|---|
| Interruption handled as intended | 61% | 94% |
| Caller repeated themselves in a turn | 18% | 4% |
| Question re-asked after being answered | 12% | 1.6% |
| Median stop latency | 310 ms | 80 ms |
The re-asking figure is the one worth dwelling on. Being asked a question you have just answered is the clearest possible signal that the thing on the other end is not listening, and it was the single most common complaint in transcripts before this work. Getting it to under two per cent changed how callers described the system more than any latency improvement did.
What we did not fix
Two callers speaking simultaneously on a speakerphone remains hard, and we currently degrade to asking for one voice at a time rather than guessing. Heavy background noise in vehicles still produces false interruption detections at a rate we are not satisfied with. Both are being worked on, and neither is solved by the classifier described here.
Interruption handling is not a feature anyone asks for by name. Nobody has ever requested better barge-in. But it is most of the difference between a system people talk to and a system people endure, and it lives almost entirely in decisions that never surface in a product description.
Why this is a model problem, not an audio one
The instinct when barge-in behaves badly is to reach for the audio pipeline: tune the voice activity detector, adjust the energy threshold, shorten the endpointing window. We spent a long time there and got modest results, because the pipeline was mostly doing its job. It was correctly detecting that a human had made a sound. The failure was in what we concluded from that.
Voice activity detection answers is someone speaking. That is a necessary input and almost no part of the actual question, which is what does this person want me to do differently. Answering the second question requires knowing what was being said, how far through it was, what was asked, and what the caller has already told you. None of that is available in the audio pipeline, and all of it is available to the model.
The information that only the model has
- Whether the assistant had already asked a question that the interruption might be answering
- Whether the interrupted sentence was essential or a courtesy phrase
- Whether this caller has interrupted before, and how those resolved
- Whether the topic being introduced relates to the current thread or replaces it
Once the decision moved into the model with that context available, the same audio events started producing sensible outcomes. The energy thresholds we had spent weeks tuning became close to irrelevant, which was a frustrating result to arrive at and a clarifying one to have.
A note on evaluating this
Interruption handling resists offline evaluation, because whether a response was right depends on what the caller did next, information that only exists if you actually responded. Our offline set is therefore a lower bound, and the production comparison in the previous section is the number we trust. Anyone reporting a barge-in benchmark from recorded audio alone is measuring detection, not handling.
That distinction matters for buyers. Detection accuracy is easy to report and easy to make look excellent. Handling quality requires live traffic and shows up as whether callers repeat themselves, which is the metric we would ask any vendor for and rarely see offered.