An LLM invoice can match your gateway total while individual attempts are still missing

If you stop reconciling when an LLM gateway total matches the provider invoice, you have established an aggregate match, not an attempt-level one. After retries and fallbacks, the difference matters. A matching total shows that one equation holds. If a retry chain contains two attempts whose billing you cannot see one by one, one equation cannot tell you how the money splits between them. The gateway can be wrong about both attempts, by equal and opposite amounts, and the provider invoice still matches to the cent.

This article builds that counterexample with fictional numbers, then turns it into a worksheet and four verdicts: matched at attempt level, matched in aggregate only, pending, and unresolved. The point is not that matching totals are worthless. It is that they support an aggregate conclusion, not necessarily an attempt-level one.

Three ledgers that are easy to merge by accident#

A reconciliation after retries involves three different things, and each has its own unit:

• Logical requests. What your application or user asked for once. One chat turn, one agent step.
• Upstream attempts. What the gateway sent to a provider. A retry or a fallback adds attempts, not requests.
• Billed usage. What the provider charged, in whatever grouping its export or invoice uses.

Gateways do not all expose the middle layer the same way. ngrok's AI Gateway documentation, for example, says its usage list has one record per upstream attempt, that attempts of the same client request share a requestId, that the list returns only each request's terminal attempt by default, and that includeRetries=true shows the retries. The same page says that when a provider reports no usage, usageEstimated is true and the provider-reported token fields are zero. That is one product's design, described here because it makes the distinction concrete: a default view of terminal attempts would already hide part of a retry chain from a naive sum.

The worked example: one request, two attempts, one number#

Everything in this section is fictional: the IDs, the models, the rates and the usage. The rates are not any provider's prices. They are chosen so the arithmetic is easy to check.

Assumptions.

• One logical request, lr_7f3a, made two upstream attempts to the same provider project, proj-demo.
• att-1 went to model-a. The gateway sent the full request, then hit its read timeout 30 seconds later. No response, no provider request ID, no usage.
• att-2 was the fallback to model-b. It succeeded. The provider reported 12,000 input tokens and 600 output tokens. The gateway's record has no cached-token field for it.
• Both attempts carried the same 12,000-token prompt, by the gateway's own count.
• In this fictional provider's usage format, the input count includes cached tokens.
• The provider's export is one row per project per day, all models combined. That day, proj-demo carried no other traffic. If a real bucket includes other traffic whose charges are not independently known, those requests add more unknowns to the allocation.

Fictional rate card.

What the gateway recorded. att-1 has no usage, so the gateway recorded nothing for it. att-2 is priced as if all input were uncached:

att-1: no usage                            = $0.0000
att-2: 12,000 x $1.00/1M + 600 x $4.00/1M = $0.0120 + $0.0024 = $0.0144
gateway total for lr_7f3a                                      = $0.0144

What the provider export shows: proj-demo, that day, $0.0144.

The totals match. Now consider two allocations the provider could have billed.

World 1: the timeout was not billed, and the fallback had no cache hit.

att-1: $0.0000
att-2: 12,000 x $1.00/1M + 600 x $4.00/1M = $0.0144
total                                       $0.0144

World 2: the timeout was billed for its prompt, and the fallback read 8,000 tokens from cache.

att-1: 12,000 x $0.60/1M                         = $0.0072
att-2: (12,000 - 8,000) x $1.00/1M               = $0.0040
       8,000 x $0.10/1M                          = $0.0008
       600 x $4.00/1M                            = $0.0024
       att-2 subtotal                            = $0.0072
total                                              $0.0144

gateway error in World 2:
att-1: recorded $0.0000, billed $0.0072  -> understated
att-2: recorded $0.0144, billed $0.0072  -> overstated

Both worlds produce the same total. In World 1 the gateway ledger is right on every attempt. In World 2 it is wrong on both attempts, by equal and opposite amounts, and the invoice cannot tell you which world you are in.

The general form is simple. A grouped export gives one equation per bucket. Each attempt whose billing you cannot observe on its own is one unknown. With two unknowns and one equation, equality is consistent with many allocations.

Aggregate equality still rules some worlds out, and that is worth recording. If att-1 had been billed and att-2 had no cache hit, the fictional provider total would have been higher than the gateway total, and the mismatch would show. A per-model export would separate this particular pair too, since the two attempts used different models. It would not separate two attempts on the same model, or offsetting errors on other requests in the same bucket.

A timeout is unknown, not free, and a retry is not automatically billed#

att-1 is the crux. A timeout after the request left the gateway tells you the gateway stopped waiting. It does not tell you whether the provider processed the prompt, and it does not tell you whether the provider charged for it. Both directions are assumptions until evidence says otherwise.

Gateways make different choices here, which is one more reason not to treat a gateway's zero as a fact about the invoice. LiteLLM's v1.106.2 release notes describe a change in its own spend logging: a request that timed out after reaching the provider had been logged at zero cost, and is now priced on its estimated prompt tokens; a connect timeout that never reached the provider stays at zero; and with retries configured, each earlier attempt that timed out gets its own spend row. That is LiteLLM documenting its own accounting, version by version. It is evidence that implementations differ, not evidence of what any provider bills for a timeout.

Separate a timeout from a pre-provider rejection. A request the gateway refused before dispatch, by a policy or an invalid key, has a different evidence shape: the gateway's own record shows the refusal happened before any upstream connection. ngrok's page states that a request rejected before it calls a provider has no attempts. For such a request, a zero cost is supported by evidence. For att-1, it is not.

Cached input: read the fields, then subtract once#

World 2 also shows a second error to guard against: counting cached input twice, or not at all. The arithmetic depends on whether the provider's input count includes cached tokens, and providers document this differently.

OpenAI's prompt caching guide illustrates reported cached tokens as part of a request's input tokens, and says input tokens are billed at the uncached-input, cached-input or cache-write rate, so cache writing is not an additive fee. Anthropic's prompt caching documentation lists cache writes, cache reads and regular input tokens for uncached content as separate charges, and reports cache activity in cache_creation_input_tokens and cache_read_input_tokens. LiteLLM's cost discrepancy guide gives the same warning in its own words: cache read tokens are typically inside the input count on OpenAI and often reported separately on Anthropic.

Treat those as pointers, not rules. Read the usage fields for the exact API and model you call, then normalize every attempt to disjoint quantities before pricing:

if the input count includes cached tokens:
    uncached_input = input - cached_read - cached_write
if the input count excludes them:
    uncached_input = input
cost = uncached_input x uncached_rate
     + cached_read    x cache_read_rate
     + cached_write   x cache_write_rate
     + output         x output_rate

In the fictional example, suppose the gateway had seen the 8,000 cached tokens but still priced all 12,000 input tokens as uncached and then added the cached tokens on top. The result double counts:

double counted: 12,000 x $1.00/1M + 8,000 x $0.10/1M + 600 x $4.00/1M = $0.0152
normalized:      4,000 x $1.00/1M + 8,000 x $0.10/1M + 600 x $4.00/1M = $0.0072

Normalize first, then compare.

The evidence matrix#

Build one row of evidence per attempt, not per request. Include pre-provider rejections as their own column so they are not mistaken for free attempts. The rejection below, rej-1, belongs to a different fictional logical request. All values are synthetic.

Two cells carry most of the weight. The provider request ID for att-2 is a candidate correlation key, nothing more. It becomes a billing match only when the provider's usage data contains a per-request record keyed by that ID, with the same model, a consistent time window and the same token categories. A grouped export has no such record, so the ID cannot be joined to anything. For att-1, no provider request ID was received, and the supplied grouped export provides no attempt-level record to join. Further attribution would require additional provider evidence or another documented correlation key.

Gateway cost is not the invoice#

Even with perfect attempt data, the gateway's recorded amount and the final invoice differ for reasons that have nothing to do with retries. Close these before you call anything a discrepancy:

• Timing. Day and month boundaries differ when the two systems use different time zones, and a call near midnight lands in different buckets. LiteLLM's guide makes the same-window point and suggests a longer comparison window; its roughly 10% threshold is that vendor's heuristic, not a standard.
• Other traffic. Anything else on the same key or project, including scripts that call the provider directly, is in the invoice and not in the gateway.
• Rates. The gateway prices from a rate table you maintain. The provider bills at its rate on the day, including tiers or modes your table may not model.
• Credits and adjustments. Promotional credits, refunds and corrections appear on the invoice, and can post later than the usage they relate to.
• Non-token charges. Whatever your provider bills outside token usage is absent from a token-priced ledger.

Four verdicts, and what moves a case between them#

Write the verdict into the record in those words. "Matched in aggregate only" is an honest finding that a later auditor can act on. "Matched" is a stronger claim, and after lr_7f3a you can see why it may be false.

Keep the bundle free of secrets and content: IDs, timestamps with their time zone, token counts, rates and amounts are enough. Prompts and keys add risk and no evidence.

Where Worksi fits, and where it does not change the logic#

Worksi's gateway makes up to two retries, then tries your fallback models, and never after the first streamed byte. It prices every call in USD from your model registry, and its ROI view includes provider invoice reconciliation. That is a comparison with computed spend. If your provider side is a grouped export, the verdict your evidence supports is aggregate, whatever tool does the comparison.

The boundary after the first streamed byte is a separate problem, covered in LLM streaming fallback after the first token. Attributing usage to your own end users, rather than to attempts, is covered in per-user AI token quotas.

The worksheet itself needs no product. One row per attempt, normalized token categories, the provider's documented semantics, and a verdict that says what the evidence covers. If the provider gives you only a total, the verdict is aggregate, and saying so is the accurate result.