Low reported token usage does not rule out an OpenAI TPM limit

An OpenAI 429 at low token usage can still be a TPM limit: the limiter's documented count is not your response usage. Read the error body, keep a header ledger per attempt, and run one comparison that changes only the output cap.

Written by Worksi's AI marketing agents and checked by a separate AI fact-checking agent.

How this blog is made
A grey funnel jammed by large hollow orange frames, each carrying only a small grey cube that would easily pass through on its own.
Illustration made with AI
In this article
  1. The sentence that matters, and what it does not say
  2. Read the 429 body before you think about tokens
  3. Keep a ledger, not a log line
  4. One controlled comparison
  5. Why a usage-based mock passes and production does not
  6. Where a gateway fits

A 429 at low reported token usage can still be an OpenAI tokens-per-minute (TPM) limit. Your calls return 429, yet the usage block on each successful response shows a few hundred output tokens, and your token totals for the minute sit far below your TPM limit. Reported usage measures what a request consumed. OpenAI's rate-limit guide describes a different number for the limiter, and on the requests it covers, that number can be much larger than the tokens you were billed for.

This article gives a way to test that hypothesis with evidence: read the error body first, keep a header ledger per attempt, and run one controlled comparison that changes only the output cap. It does not promise a verdict. Often the honest answer is "unknown", and that is still worth knowing before you raise a cap, request a higher limit or add another retry loop.

The sentence that matters, and what it does not say#

OpenAI's rate-limit guide says, under the heading "Reduce the max_tokens to match the size of your completions":

Your rate limit is calculated as the maximum of max_tokens and the estimated number of tokens based on the character count of your request. Try to set the max_tokens value as close to your expected response size as possible.

Read literally, a request with a short prompt and max_tokens set to 4,000 counts as 4,000 tokens, even if the model returns 180. Microsoft's Dev Proxy guide to LLM rate limits draws the same consequence: with max_tokens at 4,000 and 200 tokens returned, OpenAI and Azure OpenAI "still count the 4,000 against your limit", which is "how you can get 429s while your usage metrics look well below your quota." This is not a new observation, and this article does not claim it is.

Do not stretch the sentence further than it goes. It names max_tokens. It does not say that every endpoint, every model or every newer output-cap parameter is counted the same way. It does not say what stands in for the cap when you send none. It does not publish the character-to-token estimator. And it does not say whether, or when, the unused part of a cap is credited back to your window. Each of those is something you would have to observe, not assume. Azure OpenAI is a separate service with its own rules: the Dev Proxy page describes it separately, and none of its Azure detail applies to direct OpenAI traffic.

Read the 429 body before you think about tokens#

A 429 is a status, not a diagnosis. The guide says other errors use the same statuses, so "inspect the error body before choosing a recovery action". OpenAI's error codes page lists several 429s that have nothing to do with token rate:

  • credit_balance_exhausted: the organization has no prepaid credits left.
  • organization_spend_limit_exceeded and project_spend_limit_exceeded: an enforced spend limit was reached.
  • organization_usage_limit_exceeded: the OpenAI-assigned usage limit was reached.
  • slow_down, type rate_limit_error: the request rate increased too quickly.

The page is plain about the first three: "Retrying billing, spend, or quota errors won't restore API access." No output-cap experiment touches them, and a 429 is not always temporary. slow_down is a different trap. The guide says it "can occur even when your traffic is within its requests-per-minute and tokens-per-minute limits." A ramp-rate refusal at low usage looks exactly like your symptom and has a different response: follow Retry-After when present, reduce the rate and ramp up gradually.

Then check scope. The guide says limits are defined at the organization level and at the project level, not per user, and that they vary by model. Some model families share one limit: every call to any model on a shared-limit list counts toward the same TPM. Your service's reported usage covers only your service. Another job in the same project, or a sibling model in the same shared family, spends from the same window. Requests can also run out before tokens do: the guide's own example fills a 20 RPM limit with 20 small requests.

So the token-estimate hypothesis is worth testing only when the body names a rate limit, the code is not slow_down or a billing, spend or quota code, and the response's request-rate-limit headers, when present, do not show the request limit exhausted.

Keep a ledger, not a log line#

The guide lists response headers that "can" appear: x-ratelimit-limit-tokens, x-ratelimit-remaining-tokens and x-ratelimit-reset-tokens, the matching three for requests, project-scoped token headers that "may be present when a project-scoped token limit applies", and Retry-After on temporary rate-limit errors. "Can" matters. Record an absent header as absent, not as zero: a zero would read as an exhausted window.

Write one row per attempt, not per logical request:

Field Why it is there
Time, UTC with milliseconds Order, and refill between rows
Attempt number and which layer retried: app, SDK or gateway Hidden retries spend from the same window
Endpoint, model, project The scope of the limit
Output cap sent The documented input to the estimate
Prompt character count The other documented input
Returned input and output usage What was consumed
HTTP status, error.type, error.code, error.message verbatim Which limit refused it
Token and request remaining and reset headers, project headers The limiter's view, when present
Completion state: complete or truncated Whether a lower cap broke the output

The most useful derived column is the drop in x-ratelimit-remaining-tokens between consecutive responses. If it tracks the cap more closely than the returned usage, that is consistent with the guide's sentence. It is not proof: refill happens between rows, other traffic shares the window, and the guide does not define exactly when the header value is computed.

The attempt column matters more than it looks. The guide says "unsuccessful requests contribute to your per-minute limit" and that each official SDK retries eligible 429 responses automatically, subject to its retry settings. The Dev Proxy page, at the time of reading, says the OpenAI Python SDK retries 429s two times by default; check the current max_retries setting of the SDK version you run rather than relying on that figure. Put your own retry loop on top and a retrying proxy in between, and one user action becomes many attempts, each of which may be counted at the full cap. The guide's advice is to limit attempts and total retry time, and to disable SDK retries or count them in your own limits. Our article on reconciling an LLM invoice with gateway attempts covers attempt accounting in more depth.

One controlled comparison#

Hold everything constant except the output cap: same endpoint, model, project, prompt, request pace and window length. Change only the cap parameter your endpoint documents. Run arm A, wait until the token reset header says the window has fully reset (or leave a long quiet period if headers are absent), then run arm B.

The table below is synthetic. It shows the shape of a result that would support the hypothesis, assuming an illustrative 30,000 TPM limit, a prompt of about 4,800 characters and eight requests within one minute.

Arm Cap Request Returned in / out Status Remaining tokens after Completion
A 4,000 1 1,200 / 180 200 26,000 complete
A 4,000 7 1,200 / 175 200 2,000 complete
A 4,000 8 none 429, token rate limit absent none
B 600 1 1,200 / 182 200 28,800 complete
B 600 8 1,200 / 178 200 20,400 complete

By returned usage, arm A's first seven requests consumed under 10,000 tokens, about a third of the limit, yet the eighth was refused. If the guide's sentence applies literally to this endpoint, arm A was charged 4,000 per request: seven of them leave 2,000, too little for an eighth. Arm B was charged, in this illustration, an assumed character estimate of about 1,200 per request, and eight fit. The remaining-token column matches that reading.

Even a result this clean does not settle attribution:

  • Another service, or a model in the same shared family, may have spent from the window during arm A and not during arm B.
  • Refill between requests changes every drop, and a slower pace hides the effect entirely.
  • Output length varies between runs, so a cap close to typical output truncates some responses and not others.
  • Headers may be missing, which removes the strongest evidence.
  • Limits change: a tier upgrade or a project limit edit between the arms invalidates the comparison.

Use explicit verdicts. Call the result "consistent with the estimate" only when the error body names a token rate limit, headers are present in both arms, nothing else spent from the same scope, and arm B completes without truncation. Call it "inconclusive" when any of those fails. Call it "unknown" when you have no error body or no headers at all.

Do not force a 429 against production limits to run this. Arm A costs real tokens and throttles real users. Run it at low pace in a quiet project, remembering that a separate project still shares the organization-level limits, and keep the ledger passively on production traffic around the real 429s.

A lower cap is not automatically a fix. If it truncates output, the 429 has become a quality bug. Set the cap from the distribution of real output lengths with headroom, then check the endpoint's truncation signal on every response after the change.

Why a usage-based mock passes and production does not#

The Dev Proxy page is explicit that its LanguageModelRateLimitingPlugin "counts the prompt and completion tokens that each response reports" and "doesn't reproduce the max_tokens estimate." A test double configured the same way, charging each request by the usage it reported, can admit traffic that a cap-based estimate would refuse. Nothing forces a mock to work like that: it sees the prompt and the cap before it answers. The gap comes from which number it is told to count. A limiter that works the way the guide's sentence describes charges before the response, with a different number from the reported usage.

Here is the gap as two tests. The second models one documented sentence, not OpenAI's limiter, and the characters-per-token constant is a placeholder because OpenAI does not publish its estimator.

Python
# A model of one sentence in OpenAI's rate-limit guide, not OpenAI's limiter.
# CHARS_PER_TOKEN is a placeholder: OpenAI does not publish its estimator.
CHARS_PER_TOKEN = 4
TPM_LIMIT = 30_000  # illustrative

PROMPT = "x" * 4_800
CAP = 4_000
REPORTED_USAGE = {"input_tokens": 1_200, "output_tokens": 180}


class Window:
    def __init__(self, limit):
        self.limit = limit
        self.spent = 0

    def admit(self, charge):
        if self.spent + charge > self.limit:
            return False
        self.spent += charge
        return True


def charge_from_usage(usage):
    # Known only after the response: what a usage-based mock counts.
    return usage["input_tokens"] + usage["output_tokens"]


def charge_before_request(prompt, cap):
    # The guide's sentence: the maximum of the cap and a character-based estimate.
    return max(cap, len(prompt) // CHARS_PER_TOKEN)


def test_usage_based_mock_admits_all_eight():
    window = Window(TPM_LIMIT)
    results = [window.admit(charge_from_usage(REPORTED_USAGE)) for _ in range(8)]
    assert results == [True] * 8


def test_pre_request_estimate_refuses_the_eighth():
    window = Window(TPM_LIMIT)
    results = [window.admit(charge_before_request(PROMPT, CAP)) for _ in range(8)]
    assert results == [True] * 7 + [False]

By the arithmetic, the first test passes: eight requests at 1,380 reported tokens each total 11,040, well inside 30,000. The second refuses the eighth request at the same pace, because seven charges of 4,000 leave only 2,000. Set CAP to 600 and the second test's eight requests all fit again. If your pacer, your test double or your capacity plan uses the first model, it predicts headroom the provider may not give you. The fix in the test suite is not to imitate OpenAI. It is to make the pre-request charge explicit and configurable, so a test can show what your code does when the cap, not the usage, dominates.

Where a gateway fits#

If your traffic goes through a gateway, it is one more layer in the attempt column. Worksi's gateway traces and prices each model call through it, and retries a failed call up to two times before trying your fallback models, never after the first streamed byte. Count those retries with your SDK's when you cap attempts. The gateway's own refusals are typed and separate from the provider's: 429 end_user_quota_exceeded and 429 plan_quota_exceeded come from Worksi's checks, not from OpenAI, and tell you nothing about your TPM. Limits per end user are a different problem, covered in per-user AI token quotas. The comparison above is still yours to run, with the provider's own error body and headers as the evidence.

Sources

  1. OpenAI API: Rate limits
  2. OpenAI API: Error codes
  3. Microsoft Learn: LLM rate limits (Dev Proxy)

How this blog is made

Worksi's AI marketing agents write the articles here, and a separate AI fact-checking agent checks each one before it is published. The sources an article cites are listed with it. Its illustration was made with AI. If you find a mistake, write to info@worksi.ai.