Test an AI gateway spend limit with a burst that no response can finish

A mock upstream holds every response until the burst has reached the gate, so you can see whether your gateway's spend limit checks recorded spend or reserves, and where each design leaks: low holds, missing usage, failed settlement and unreadable counters.

Written by Worksi's AI marketing agents and checked by a separate AI fact-checking agent.

How this blog is made
Seen from above, a crowd of orange coins has already passed through a grey coin slot together while the meter beside it still rests at the start.
Illustration made with AI
In this article
  1. Five calls pass the same check because none can see the others.
  2. Two designs answer the same burst differently.
  3. A bug report shows why the scope matters.
  4. Hold every response until the burst has reached the gate.
  5. Seven runs, and what each one separates.
  6. What Worksi's spend limits do in this test.

With a recorded-spend-only check, an AI gateway spend limit is compared with usage already recorded when a request arrives, and the request's cost is added only after it finishes. Between those two moments, concurrent requests can all pass the same check, and the overshoot is settled afterwards. A reservation design narrows the gap, but on its own it is not evidence of a hard ceiling. If a spend limit has to bound a burst, test what your gateway does in that gap: sequential traffic, a burst, a call that costs more than its hold, missing usage, and a counter that cannot be read.

Five calls pass the same check because none can see the others.#

A synthetic example, in fictional credits. Limit: 100. Recorded spend at the start: 70. Every call costs 12 credits once priced. The gateway refuses when recorded spend has reached the limit.

Time Event Recorded spend Check
0 to 40 ms Calls A to E arrive 70 70 < 100: all five admitted
1.2 s A settles 82
1.3 s B settles 94
1.4 s C settles 106 crosses the limit
1.5 s D and E settle 130
2.0 s F arrives 130 130 ≥ 100: first refusal

C is the crossing call: its settlement took the counter over. D and E are not crossing calls. They were already in flight, admitted against a counter that read 70, and they land on top. The 30-credit overshoot was decided at 40 ms, not at 1.4 s. F's refusal shows that the check now sees an exhausted budget, not that the earlier spending stayed within it.

Which call crosses depends on settlement order, not arrival order. Sent one at a time, with each call settled before the next check, the same traffic stops at 106: the third call is admitted at 94, settles at 106, and the fourth is refused. The overshoot is one call. In a burst, under this design, it is whatever was admitted before the counter moved, minus the headroom. That depends on your concurrency and per-call cost, so there is no universal bound to quote.

Two designs answer the same burst differently.#

Recorded-spend check. Admission compares usage already recorded with the limit; cost is added after the response is priced. Cheap and simple; for sequential traffic whose costs are recorded before the next check, it overshoots by at most the crossing call. In a burst it admits every call that arrives before the first settlement.

Reserve, then reconcile. Admission estimates each call's maximum cost, atomically adds it as a hold, and refuses the call if spend plus holds plus the new estimate would exceed the limit. When the response is priced, the hold is replaced by the actual cost. LiteLLM documents this design as enabled by default: it estimates from max_tokens or max_completion_tokens when present, otherwise from the model's configured limits, and for "routes without token pricing, such as some image and audio routes", it "enforces the budget using recorded spend".

Rerun the timeline with a 15-credit hold per call. A reserves (70 + 15 = 85). B reserves (100, equal to the limit, admitted). C would take the total to 115: refused, and so are D and E. A and B settle at 12 each: final spend 94. The burst is bounded, under these conditions:

  • The estimate is not low. With 8-credit holds, A, B and C reserve to 78, 86 and 94; D would reach 102 and is refused. Three calls settle at 12: final 106. LiteLLM's docs say the same of their TPM limits: when a model emits more than the estimate, "concurrent requests are all admitted against an under-reservation". An input pre-count does not close this: Anthropic's token counting docs call the count "an estimate" that "might differ by a small amount" from the input actually used.
  • The hold is in the scope you budgeted. A reservation on a key budget does nothing for a session budget that only reads recorded spend.
  • The state is shared and atomic. A check and an increment in two round trips, or a counter per worker, admits the burst again across replicas.
  • Settlement handles failure. A timeout or a lost response does not prove the provider charged nothing. Releasing the hold at zero undercounts; keeping it until an expiry refuses traffic the budget could serve.
  • The counter is readable. If it is not, the gateway admits (fail open) or refuses (fail closed). LiteLLM documents fail_closed_budget_enforcement, which returns a 503 when spend can be verified against neither Redis nor the database, and says that in the default mode a stale Redis counter "can let a key keep spending past its max_budget until the counter is corrected".
  • Retries are calls. Whether each retry attempt is reserved and priced is something to test.

Reservation also costs traffic. In the bounded run, spend stops at 94 and the next call is refused although 6 credits remain, because its hold is 15.

A bug report shows why the scope matters.#

LiteLLM issue #34732, opened on 26 July 2026 by emerzon, reports that session-budget admission "checks already-recorded spend but does not hold capacity for in-flight calls". The posted reproduction calls the hooks directly, and its log shows Redis connection errors and "recorded spend 0.0", so that log does not show the claimed overspend. On 4 November a commenter reported a separate run against a mock upstream on v1.108.0 and v1.106.6: with a max_budget_per_session that fit 9 calls, 30 concurrent calls all reached the upstream, and five later calls were refused with 429. In the same run, they wrote, virtual-key budgets admitted 11 of 30.

Those are their results under their configuration, not ours, and say nothing about other versions. They do show that one gateway can reserve for one budget scope and not another, so a test has to name its scope.

Hold every response until the burst has reached the gate.#

Put a mock upstream behind the gateway that holds every response until you release it. Fire a burst, wait until each request has either reached the mock or come back from the gateway, then release with known usage and read the counter.

The mock decides usage; the gateway's model registry prices it. If your gateway lets you set a custom price for a registered model, make each call a round figure: input at $0, output at $300 per million tokens, and 1,000 output tokens per response is $0.30 a call, and a hold of $0.30 if the gateway estimates from max_tokens. Otherwise use a price the registry already has and compute the expected totals from it. The figures below assume $0.30 a call and a $1.00 limit. The mock must be reachable from the gateway, and the hold must stay under the gateway's upstream timeout, or its retries appear as duplicate arrivals.

Python
import asyncio, json, time
import httpx
from aiohttp import web

GATEWAY_URL = "https://your-gateway.example/v1/chat/completions"
API_KEY = "a-test-key-used-by-nothing-else"
MODEL = "mock-priced-model"  # routed to this mock, priced in the gateway's registry
USAGE = {"prompt_tokens": 10, "completion_tokens": 1000, "total_tokens": 1010}

release = asyncio.Event()
arrivals: list[dict] = []  # duplicates kept: they show retries

async def upstream(request: web.Request) -> web.Response:
    body = await request.json()
    last = (body.get("messages") or [{}])[-1]
    arrivals.append({"t": time.time(), "tag": str(last.get("content"))})
    await release.wait()
    return web.json_response({
        "id": "chatcmpl-mock", "object": "chat.completion",
        "created": int(time.time()), "model": body.get("model", MODEL),
        "choices": [{"index": 0, "finish_reason": "stop",
                     "message": {"role": "assistant", "content": "ok"}}],
        "usage": USAGE,
    })

async def send(client: httpx.AsyncClient, i: int) -> dict:
    t0 = time.time()
    try:
        r = await client.post(GATEWAY_URL,
            headers={"Authorization": f"Bearer {API_KEY}"},
            json={"model": MODEL, "max_tokens": 1000,
                  "messages": [{"role": "user", "content": f"req-{i}"}]})
        return {"i": i, "status": r.status_code, "body": r.text[:500],
                "t0": t0, "t1": time.time()}
    except httpx.HTTPError as exc:
        return {"i": i, "status": None, "error": repr(exc),
                "t0": t0, "t1": time.time()}

async def burst(client: httpx.AsyncClient, n: int, gate_wait_s: float = 5.0) -> dict:
    tasks = [asyncio.create_task(send(client, i)) for i in range(n)]
    deadline = time.monotonic() + gate_wait_s
    while time.monotonic() < deadline:
        arrived = {a["tag"] for a in arrivals}
        returned = {f"req-{i}" for i, t in enumerate(tasks) if t.done()}
        if len(arrived | returned) >= n:
            break
        await asyncio.sleep(0.05)
    at_gate = {"arrived": sorted({a["tag"] for a in arrivals}),
               "returned_before_release": [i for i, t in enumerate(tasks) if t.done()]}
    release.set()
    results = await asyncio.gather(*tasks)
    return {"at_gate": at_gate, "arrivals": list(arrivals), "results": results}

async def main() -> None:
    app = web.Application()
    app.router.add_post("/v1/chat/completions", upstream)
    runner = web.AppRunner(app)
    await runner.setup()
    await web.TCPSite(runner, "0.0.0.0", 8099).start()
    try:
        async with httpx.AsyncClient(timeout=60) as client:
            print(json.dumps(await burst(client, 10), indent=1))
    finally:
        await runner.cleanup()

asyncio.run(main())

The wait is bounded because refused requests do not reach the mock: waiting for ten arrivals would hang. The loop ends when every request has arrived or returned, or after five seconds, and at_gate records which. If the tags do not come through, match arrivals by time and write that down.

Give each independent run its own budget scope, or a verified new budget period, and read its counter as zero before you start. A new key resets only a budget scoped to that key: where the policy is scoped to a team, workspace or agent, a second key in the same scope inherits the spend. Run 3 is the exception: it keeps the scope and settled state of run 1 or 2.

Record per run: gateway version, the policy as configured (scope, period, limit, enforcement), the model price, every status, body and timestamp, every arrival, and the counter before and after, read from the gateway's own API or console once it has settled.

Seven runs, and what each one separates.#

  1. Sequential baseline. Ten calls, one at a time. After each success, poll the budget counter until it shows that call's $0.30 and no open hold, with a bound such as 30 seconds. Accounting can be asynchronous, so a call that does not settle in time makes the baseline inconclusive. With settled counters, recorded spend admits four (at 0, 0.30, 0.60 and 0.90), final $1.20, fifth refused; reservation with a $0.30 hold admits three, final $0.90.
  2. Burst near the threshold. burst(client, 10) from zero. Recorded spend: ten arrivals, final $3.00. Reservation: three arrivals, seven refusals that never reach the mock, final $0.90.
  3. Later request after settlement. In the same scope, once run 1 or 2 has settled, send one more call. Both designs should refuse. A reservation design that admits it is leaking holds or settlements.
  4. Cost beyond the hold. A controlled injection, for a gateway where run 2 showed reservation at this scope. Keep max_tokens: 1000, so the hold is a known $0.30 if the gateway estimates from it (record the hold where the gateway shows it), and have the mock report 2,000 output tokens, $0.60 a call. That usage is synthetic and deliberately exceeds the declared cap; a gateway may cap or reject it, so record what it does. If it prices the usage, three calls fit in holds and settle at $1.80. Attribute the overshoot to the hold only when runs 1 and 2 showed one arrival per request, settled counters and no more admissions than fit; otherwise write the cause as unknown.
  5. Missing usage. Return no usage object. Does the counter move by an estimate, or not at all?
  6. Lost response. Hold one call past the gateway's upstream timeout, or drop the connection. Is its hold released, kept or priced, and when? The mock knows it served the request; a real provider may bill for one.
  7. Counter unavailable. Possible only where you run the counter store yourself: stop it, send one call, and record whether it was admitted or refused, with which status, and what the counter reads after it returns.
Case Recorded-spend check Reserve, then reconcile Evidence
Sequential Overshoot up to the crossing call Stops at or below the limit if estimates hold Admitted count, final counter (run 1)
Concurrent burst Admits all before the first settlement Admits what fits in holds Arrivals at the gate (run 2)
Cost beyond the hold No estimate at admission Can over-admit and overshoot if actual cost exceeds holds Admissions, final counter, known hold (run 4)
Missing usage Unknown until observed Unknown until observed Counter after run 5
Settlement after failure Cost recorded or not Hold released, kept or expired Counter, later admissions (run 6)
Counter unavailable or stale Fail open or fail closed Fail open or fail closed Status, arrivals (run 7)

Write any cell you did not observe as unknown.

One more design to evaluate on your side: a semaphore capping concurrent model calls per budget, with max_tokens on every call. Our opinion, not a measured result: under a recorded-spend gateway, this limits how much one process can have in flight when the counter catches up. It covers only that process: not other replicas, not retries, not the invoice. For attempt-level accounting, read reconciling gateway spend with the invoice; for identity and quota dimensions, read the per-user quota guide.

What Worksi's spend limits do in this test.#

Expect the recorded-spend column. A Worksi spend_limit policy is checked before a call against usage already recorded, so the call that crosses the limit and calls already in flight complete. Once recorded spend reaches the limit, the gateway refuses with 402 budget_exceeded. If the counter cannot be read, the spend check lets the call through. That is what Worksi documents, not something this test observes: against hosted Worksi, run 7 is not possible. A spend policy is scoped to an organization, workspace, environment, team, workflow or agent, not to a key, so isolate runs by scope. Where a provider reports no usage, the gateway estimates tokens at about four characters a token and marks the span aiw.usage_estimated. Its up to two retries send the same request again, recorded on one span with the retry count. Calls that skip the gateway are not checked at all. The hosted gateway reaches public addresses only, so expose the mock over HTTPS and register it as an OpenAI-compatible provider.

Spend policies are available from Startup. The first organization of each account gets 3 days of Scale features, no card, long enough to try the runs if you can expose the mock over HTTPS. Compare the plans.

Sources

  1. LiteLLM docs: Budgets, Rate Limits (budget reservation, fail-closed enforcement)
  2. LiteLLM issue #34732: Concurrent requests can bypass max_budget_per_session before spend is recorded
  3. Anthropic docs: Token counting

How this blog is made

Worksi's AI marketing agents write the articles here, and a separate AI fact-checking agent checks each one before it is published. The sources an article cites are listed with it. Its illustration was made with AI. If you find a mistake, write to info@worksi.ai.