LLM streaming fallback: after the first token, the client owns the failure

A model gateway can switch providers cleanly until it has sent something to your client. After that point, every option changes an answer someone has already started reading. Gateways document that point in different places, and "first byte", "first frame" and "first token" are not the same moment in a stream.

Our position: for agent workloads, the default after any content reaches the client should be an explicit incomplete state, with any retry run by the application. Gateway-side continuation is a reasonable choice for prose and a poor one for tool calls. Whichever you pick, write it down as a contract and test the gateway against it, because documentation does not tell you what your client actually receives.

A Hacker News commenter asked: "Do you buffer or stream and then switch? This impacts correctness, especially for partial outputs or tool calls." (comment). One person's checklist, but the right question.

Five things that get called "the first byte"

1. Response headers. Once a 200 with text/event-stream goes out, the HTTP status is fixed. Later failures travel inside the body or as a dropped connection. In Anthropic's streaming docs, an overloaded_error can arrive as an event: error that "would normally correspond to an HTTP 529 in a non-streaming context".
2. The first body byte. It may belong to a ping or a metadata event with no content.
3. The first SSE event. A Messages API stream starts with message_start, "a Message object with empty content", and ping events can appear anywhere.
4. The first content chunk. A text_delta, or an input_json_delta holding a fragment of tool arguments.
5. Model tokens. Chunks do not map one-to-one to tokens, and message_delta usage counts are cumulative.

So "no fallback after the first token" and "no fallback after the first byte" describe different gateways. Three implementations, as documented on 4 November 2026 (not measured for this article):

Implementation	Documented boundary	After the boundary
Worksi gateway	Up to two retries with backoff on connection errors, 429 and 5xx, then your fallback models. "Never after the first streamed byte."	The failure reaches the client.
DeltaLLM	Fallback "only before the first downstream response frame". Role and metadata events wait in a bounded pre-commit buffer.	No new provider attempt. A classified stop after partial output completes as content_filter or length. Other broken streams close without [DONE] (OpenAI-compatible) or get one sanitized event: error frame (Anthropic Messages).
Continuum Router	Before the stream begins, an eligible failure hops to the next permitted chain target. On the OpenAI-compatible relay, fallback.mid_stream_enabled (default true) buffers the SSE response so it can recover mid-stream. On /anthropic/v1/messages and /v1/responses, the boundary is the first successful provider handshake.	OpenAI-compatible relay: streaming.mid_stream_fallback.enabled: true continues from accumulated output when enough exists; false restarts on the fallback model. Mid-stream hops skip native-protocol and Unix-socket targets. Messages and Responses: no mid-stream recovery. With mid_stream_enabled: false, a failure after output begins is a stream error.

DeltaLLM documents holding role events back, so a provider that dies right after one can still be retried there. A gateway whose boundary is the first byte may already have forwarded that event. The "Headers, then nothing" test below checks exactly this.

What the client should do, by failure point

The options are the same at every point: propagate an incomplete or error response, restart at the application level, or continue from what was already generated. What changes is the cost. The table assumes your application executes tools itself, so a failed model call has run no tool.

Failure point	What the client has	Propagate incomplete	Application restart	Continuation	Default we would choose
Before headers	Nothing	Only if the chain is exhausted	No delivered text to duplicate	Not applicable	Let the gateway fall back, subject to the checks below
After headers, before content	A 200, maybe metadata or pings	Acceptable	No delivered text to duplicate	Not applicable	Retry the failed model call
After partial text	Text a person may have read	Honest; the user sees a cut-off answer and a retry option	The UI must replace, not append; the new answer may contradict the half already read	One answer from two generations, possibly two models	Propagate incomplete; restart on request
During tool-call output	A fragment of JSON arguments	Required	Discard the partial block, regenerate the turn	A second generation completing arguments the first one chose	Never execute; regenerate the turn
After a tool ran	A real side effect	Required for the failed call	Resend the stored tool result, not the user message	Same risks as above	Checkpoint at turn boundaries

"No delivered text to duplicate" is the whole claim in those cells. It does not mean nothing happened upstream. A missing response does not prove a remote operation did not run, which is why Stripe treats a 500 as indeterminate and asks you to reconcile before retrying with a new key. If your provider runs tools server-side, or earlier steps already wrote somewhere, restart only the failed model call, keep earlier tool results and reconcile anything whose outcome you do not know.

Two Continuum settings change what a client receives on the relay: the continuation-or-restart switch above, and same_model_first, which tries another backend serving the same model before advancing the chain. That narrows the model-mismatch problem. The docs state one limit plainly: "Mid-stream recoveries cannot carry X-Fallback-* headers because the SSE response is already committed." Headers will not tell your client a recovery happened. The router's attempt records, linked under one request id, are the operator's evidence.

Signaling inside the body is possible: Anthropic's streaming docs describe server-side fallback responses where a content_block_start and content_block_stop pair with no deltas marks each model boundary. Check in a recorded transcript whether your gateway's continuation leaves any such mark. If you need to know which model wrote which sentence and cannot see it, keep continuation off.

Make "incomplete" a state your code can see

One client failure mode: a connection closes, the accumulated text looks finished, and the code stores it as a complete answer. Treat completeness as an explicit signal. A sketch for Anthropic Messages events, already parsed from SSE:

from dataclasses import dataclass, field

@dataclass
class Turn:
    text: list[str] = field(default_factory=list)
    tool_json: dict[int, list[str]] = field(default_factory=dict)
    stop_reason: str | None = None
    complete: bool = False
    error: dict | None = None

def consume(events, turn: Turn) -> Turn:
    for ev in events:
        t = ev["type"]
        if t == "content_block_delta":
            d = ev["delta"]
            if d["type"] == "text_delta":
                turn.text.append(d["text"])
            elif d["type"] == "input_json_delta":
                turn.tool_json.setdefault(ev["index"], []).append(d["partial_json"])
        elif t == "message_delta":
            turn.stop_reason = ev["delta"].get("stop_reason")
        elif t == "message_stop":
            turn.complete = True
        elif t == "error":
            turn.error = ev["error"]
            break
        # ping and unknown event types are skipped on purpose:
        # Anthropic says new event types may be added
    return turn

The rules around it matter more than the code:

• The success signal depends on the protocol. Anthropic Messages ends with message_stop. Chat Completions streams end with data: [DONE]. The OpenAI Responses API uses typed events: its streaming guide names response.completed for a finished response and error for a failure. A Responses stream that ends without response.completed is incomplete.
• A closed connection without the success signal is incomplete, however finished the text looks.
• A success signal says the transport finished, not that the answer is usable. A length stop, a content_filter finish or an Anthropic refusal stop (all named in DeltaLLM's docs) arrive with a normal terminal event. Check the stop reason.
• Tool calls run only from complete turns whose stop reason you accept, and only after the arguments parse as JSON.
• An incomplete turn is never cached, never stored as final, and never fed back as an assistant message without a marker that it was cut off.

Idempotency survives a restart only if the operation id does

The tool-call id is the obvious idempotency key and a risky one. A regenerated turn is a new response, and nothing guarantees its tool call keeps the old id. If the first attempt created a ticket and the regenerated call carries a fresh id, a key built on that id lets both through.

Hashing the tool name and arguments does not fix this. The Amazon Builders' Library explains why: two identical requests can be two intended actions, such as launching two identical instances. A regenerated call with slightly different arguments also hashes differently and slips past.

Have your executor create and persist a logical operation id before dispatch, one per intended action: for example run id, step index and the call's ordinal within that step. Store it with the arguments, the original dispatch time and a retry deadline taken from the downstream API's idempotency window. Reuse that id, with the same parameters, on retries inside that window. Two intentional identical actions get two ids.

The deadline matters because keys expire. Stripe's API v1 guidance says it keeps keys for at least 24 hours and can remove them after that, and that reusing a removed key can execute a new request. It limits retries of an unresolved operation to the first 24 hours; after that, it says to retry with neither the old key nor a new one, and to reconcile the outcome instead. Other APIs set their own windows, so read each contract. Past the deadline, stop automatic retries and reconcile before anything repeats. If the parameters must change, Stripe says to first establish that the original operation did not succeed and is not still running.

Then checkpoint at turn boundaries. When the model call after a tool fails, resend the stored tool result in a new call; do not restart the agent loop from the user message. If you must regenerate a turn that already acted, match the new calls to logged operations by step and ordinal, and reconcile any mismatch with the downstream system before running anything.

This works only if the downstream system honors the key. For a tool whose API takes none, record intent before the call and check it before any retry. A window remains between the call succeeding and your record of it; write that into the runbook.

Fallback must not route around policy or capability

A gateway authorization decision is not an outage. A model allowlist refusal, a budget or quota stop, or a pending approval should end the request, not send it down the chain. A gateway that treats those like a 5xx can route around the restriction: the fallback attempt is not bound by the decision that refused the first one, unless the policy and approval checks stay authoritative for every hop. Never use fallback to get past a refusal. Your own rate limit and a provider's 429 can share a status code and need opposite handling, so test which one your gateway retries.

Provider content-policy refusals are a separate case and a deliberate choice. DeltaLLM documents content_policy_fallbacks for "when a provider rejects the content for policy reasons". Continuum's response-quality fallback is a different mechanism from streaming fallback: it applies only to non-streaming /v1/chat/completions, and its docs exclude streaming responses and the Messages and Responses ingress paths. In enforce mode it can resend a refused request to the next model, which its docs call an operational decision with safety implications; the default shadow mode only judges and logs. If you enable either, the target must still be a model the caller may use, under your mandatory guardrails.

Continuum re-checks the key's allowed_models for default-chain and quality hops, but explicit chains "are not re-gated per hop", and its docs advise mirroring mandatory guardrail route rules across chained models.

Capability mismatches are quieter. Continuum fits output-token budgets to the target and returns a 503 incompatibility error when no target fits. DeltaLLM treats unusable 200 answers, such as non-JSON tool arguments or a tool name the request did not offer, as failures that start retries and fallbacks. Those checks catch malformed output, not a fallback model that makes valid tool calls with worse judgment. That is your evaluation, per model in the chain.

The test matrix

Put a fault-injecting proxy or a provider stub between the gateway and the model, record the raw SSE stream at the client, and point tools at a sandbox that counts invocations. The matrix lists checks, not results: run it per endpoint you use and per provider in the chain.

Test	Inject	Client should see (your contract)	Verify about effects	Evidence to capture
Pre-stream connection failure	Primary refuses the connection or returns 503 before headers	A complete answer from the fallback, or a typed error once the chain is exhausted	Exactly one complete generation reached the client	Response model field, request or trace id, documented fallback headers
Headers, then nothing	Stub sends 200 and a metadata event, then closes	Recovered answer or an incomplete marker; record which	Nothing stored as final	Raw transcript with event types and timestamps
Mid-stream disconnect	Close after N text deltas	Incomplete marker	Not cached or persisted as final; a restart replaces the text	Transcript, success signal present or absent, repeated text after recovery
Partial tool call	Close in the middle of input_json_delta	Incomplete turn, no tool call	Sandbox counter at 0	Executor log, fragment length (not contents)
Side effect, then failure	Let the tool run, fail the next model call	Recovery from the stored tool result	Sandbox counter at 1 after recovery	Operation id hits, downstream record count
Gateway policy refusal	Request a disallowed model, exceed a budget or your own rate limit	A typed refusal, never a fallback answer	No request reached any provider	Status and error type, provider-side request log
Provider content-policy refusal	Stub returns a classified content-policy error or refusal stop	The refusal, or a fallback you configured on purpose	Any fallback target is permitted for the caller and passed your guardrails	Final model, guardrail result, fallback reason
Fallback capability mismatch	Primary needs tools, JSON output or a long context; fallback lacks one	A loud failure, or a fallback that supports it	No parameters silently dropped	Final model, the request as sent to the fallback, validation result

Evidence costs privacy

Each strategy needs different retained data, so decide them separately.

• Propagating incomplete needs no content: event counts, last event type, whether the success signal arrived, model, token counts and operation ids.
• Restarting the failed model call needs the request you already hold, not the partial output.
• Continuation needs a transient per-stream buffer of accumulated output at the router.
• Checkpoint recovery needs application state: tool results and prompts per turn, under your application's retention.

Diagnostic transcripts are optional, and a raw SSE transcript is user content. Keep full transcripts in a test environment that runs synthetic prompts, and the metadata above in production. Pattern-based redaction is not a substitute for that split, because it misses names and free text.

Where Worksi fits

The Worksi gateway makes up to two retries with backoff on connection errors, 429 and 5xx, then tries the fallback models you chose, and never retries or falls back after the first streamed byte. After that byte, a failure reaches your client. Keep incomplete-state handling, downstream-tool idempotency and checkpoint recovery in your application. Worksi's tool allowlists and approvals decide whether a tool call may run; whether an action that already ran may run again stays with your executor.

For the incomplete-state evidence above, the gateway's record and your client's record need to line up. A gateway request without a trace header starts its own trace, so send one per run and keep the same id in your client-side transcript. Gateway refusals come back as typed errors, such as 403 model_not_allowed or 402 budget_exceeded, and are recorded as blocked spans: gateway-side evidence for the policy-refusal test. Metadata-only capture keeps that record without prompt or output text. If you need mid-stream continuation, Worksi's documented behavior does not provide it. Read the gateway's retry and fallback rules.