One endpoint, several frontiers
No single model is the right choice for every request a working assistant handles in a day. Summarizing a two-minute call, drafting a contract clause, deciding whether a caller is describing an emergency, and extracting four fields from a scanned invoice have almost nothing in common except that they all arrive through the same product. Pinning all of them to one model means overpaying for the easy ones and underserving the hard ones.
Heyno One is a single endpoint that routes each request to the model best suited to it, across our own models and the frontier APIs we have access to. Callers of the API do not choose. They describe the task and the constraints, and the routing happens underneath without a code change when the available models shift.
What the router actually sees
- The task class, inferred from the request shape rather than declared
- Latency budget, which for voice work is the hard constraint
- Whether the output feeds a side-effecting action or is read by a person
- Context size, and whether the prompt prefix is cacheable
- Cost ceiling for the account and the current period
The last two matter more than they sound. A request whose prefix is stable across thousands of calls is dramatically cheaper on a model with prompt caching, and the router will prefer that model even when a marginally stronger one exists, because the quality difference on that task class does not justify the multiple.
Knowing when the route was wrong
Routing is the easy half. The hard half is noticing that a route was a mistake, because the failure is silent: a cheaper model returns a plausible answer, nobody complains, and quality erodes in a way no error rate captures. A router without a feedback path is just a cost optimizer that degrades your product slowly enough that you do not attribute it.
Shadow evaluation on a sample
A small percentage of requests are run against a stronger model in parallel and the two outputs are compared offline. Where they disagree materially, the task class is flagged and its routing threshold moves. This costs a few per cent in additional inference and is the only reason we trust the router at all, because it converts silent degradation into a measurable signal.
Disagreement is judged on the thing that matters for that task class, not on textual similarity. For extraction, the field values either match or they do not. For a summary that will drive a follow-up action, the comparison is whether the same action would be taken. Two summaries can be worded entirely differently and be equivalent for our purposes, and a similarity score would call that a regression.
Escalating mid-request
When a routed model returns low confidence or produces output that fails a cheap structural check, the request is re-run on a stronger model rather than returned. This happens on about 3% of requests and is invisible to the caller apart from latency. It is also the reason the aggregate quality of the cheap route can be held at the level of the expensive one while most traffic never touches it.
What it looks like in practice
Across a month of production traffic, the majority of requests resolve on a mid-tier model, a small tail requires the frontier tier, and a meaningful share of high-volume extraction work runs on a small model that would be embarrassing to use for anything else. The distribution is heavily skewed, and that skew is the entire economic argument for routing.
| Task class | Share of requests | Routed to |
|---|---|---|
| Field extraction | 38% | small |
| Call summarization | 24% | mid |
| Drafting for review | 19% | mid |
| Judgement with side effects | 12% | frontier |
| Long-context reasoning | 7% | frontier |
The practical benefit is not the cost line, though that is real. It is that model choice stops being an architectural commitment. When a new model ships, it enters the routing table and earns traffic on measured performance for specific task classes. No integration changes, no migration, and no quarter spent re-testing prompts that were tuned against a model that is no longer the best option.
The part that is not automatic
Accounts can pin a task class to a specific tier, and some should. A regulated workflow that has been validated against one model should not silently move to another because the router found it cheaper. Pinning is a first-class setting rather than an escape hatch, and pinned classes are excluded from shadow evaluation so the validation stays meaningful.
What routing does not solve
It is worth being direct about the limits, because routing is easy to oversell. It does not make a weak model strong, it does not remove the need to evaluate, and it introduces a component that can itself be wrong in ways that are harder to debug than a fixed model choice would be.
Debuggability costs
When every request goes to the same model, reproducing a bug is straightforward. When the router is involved, the same input can take different paths depending on load, cost state and cache warmth. We therefore record the full routing decision with every request: the class inferred, the candidates considered, the reason for the winner. Any support conversation starts there.
Without that record, routing would be a net negative for anyone operating the system. The performance and cost benefits are real, and they would not be worth a class of bug that cannot be reproduced. The decision log is not a nice-to-have; it is the thing that makes the architecture defensible.
Version pinning and reproducibility
Regulated workflows need to know that the same input produces comparable output over time. Pinned task classes are frozen against a specific model version and are excluded from routing changes entirely, including the automatic escalation described earlier. That costs some quality on hard cases and buys reproducibility, and for the accounts that ask for it, that is the correct trade.
- Pinned classes never move, including across model deprecations, until the account moves them
- Routing decisions are retained with each request for the account retention period
- Shadow evaluation is disabled on pinned classes so validation stays meaningful
- Deprecation of a pinned model triggers a notice rather than a silent migration
The general principle is that automatic behaviour should be the default and never the only option. A router that cannot be overridden is a router that will eventually make a decision on your behalf that you needed to make yourself, and the fact that it was usually right will not help in that conversation.