A router that sits in front of several model providers has one job that looks trivial and is not: when a provider starts failing, send the next request somewhere healthy. If your client waits for a complete response, that is a retry. If your client is reading a stream, you have already committed to a provider by the time the first byte leaves.
The problem with retrying a stream
Most failover implementations check the status code before streaming starts. A 429 or a 503 is easy: pick the next route, send the same payload, and the client never knows. The failures that hurt are the ones that arrive after the connection is open and the response has begun: a provider that returns headers immediately and then stalls, or one that emits two tokens and dies.
By then the client has received a partial answer. Restarting the request at a different provider means the client sees duplicated text, or a response that contradicts itself halfway through. Silently appending a second provider’s answer to the first is worse than failing loudly.
Peeking the first chunk
The approach that worked was to treat the first chunk as the real commitment point rather than the response headers. The executor opens the upstream stream, pulls exactly one chunk, and only then hands the stream to the client.
That single chunk is the decision. If it arrives within the timeout, the route is healthy and the rest of the stream is forwarded as it comes. If the connection errors, or nothing arrives inside the budget, the executor discards the partial stream, marks that route as degraded, and moves to the next tier, with a payload the client has never seen a byte of.
let mut upstream = provider.stream(&payload).await?;
let first = timeout(budget, upstream.next()).await??;
// Only now is the response committed to this route.The cost is one chunk of latency on every request, and the benefit is that failover stops being visible. Ordering matters too: tiers are walked in the order the key defines, so a cheap-but-flaky provider never gets picked ahead of the primary just because it answered faster.
Measured savings, not claimed ones
The same principle applies to the token-saving step. A saver that reports its own savings is grading its own homework: it counts what it removed, not what the model actually billed.
Running the savers as a deterministic pipeline and then re-measuring the rewritten messages gives a number that can be trusted, because it is derived from the payload that was actually sent. If a saver fails, the pipeline fails open: the request goes out unmodified rather than not going out at all.
The general lesson is worth repeating. Any number a system reports about itself should be recomputed from the artifact it produced, not accumulated from the intentions of the code that produced it.
