Rate limits and 429 errors when calling AI APIs: retries, queues and fallbacks

An application calling an AI model API will sooner or later get a 429 error (too many requests) or an overload error. If the code only retries immediately, it usually makes things worse, costs more and keeps customers waiting.

Checked as of October 23, 2026, this article draws on Anthropic and OpenAI documentation, 2 RFCs, AWS articles and the Google SRE book. The AI vendors' pages show no update date, so Click Star records the date it read them.

It covers handling each group of errors, retrying safely, and when a queue and a fallback model are needed. Click Star already has articles on token optimization and AI agents.

Key points

• A 429 has 2 causes. After exceeding a rate limit you can wait and retry; after hitting a spending cap retrying is pointless (no retry-after header; the API is paused until next month, per Anthropic).
• If retry-after is present, wait that many seconds. If not, use exponential backoff with randomness (jitter), and cap the number of attempts and the total wait.
• Retries at several layers multiply: per the Google SRE book, 3 layers each retrying 3 times can produce 64 attempts on the bottom tier. Retry at 1 layer.
• Long requests should use streaming, but an error can still arrive after the API has returned 200.
• Work that needs no immediate answer should move to a batch API: Anthropic and OpenAI both state a 50% cost reduction and separate limits.
• The last layer is a fallback model, a spending cap and logging. The retry parameters and roadmap here are Click Star reference frameworks, not tested on Vietnamese data.

Rate limits and the 429 error: why applications hit it

Per RFC 6585 (April 2012), status 429 means the user has sent too many requests in a given period; the accompanying Retry-After header is optional.

Per Anthropic's documentation (read October 23, 2026), Messages API limits cover 3 quantities per model class: requests per minute (RPM), input tokens per minute (ITPM) and output tokens per minute (OTPM). Exceeding any returns a 429.

Anthropic uses a token bucket: capacity is replenished continuously. A limit of 60 requests per minute may be enforced as 1 per second, so a short burst can still fail. For most models, tokens read from the prompt cache do not count toward ITPM, and max_tokens does not count toward OTPM.

OpenAI (read October 23, 2026) measures RPM, RPD (per day), TPM, TPD and IPM (images per minute); whichever is reached first applies. Limits apply per organization and project, not per user.

A difference: OpenAI counts the larger of max_tokens and an estimate from the request's character count, so an overly high max_tokens reserves capacity for nothing. Both use usage tiers; Click Star gives no limit figures because they differ by account, so check your own console.

Anthropic also warns of 429s from acceleration limits when traffic spikes. OpenAI's similar slow_down code comes with a rule of thumb: above 1 million input TPM, raise traffic by no more than 50% every 15 minutes.

2 kinds of 429: ones you can wait out and ones you cannot

Per Anthropic, when an organization reaches its tier's monthly spend cap, the API pauses until 00:00 UTC on the first day of the next month. Requests get a 429 with detail code enforced_spend_limit_reached and no retry-after; retrying, including the SDK's automatic retries, fails.

If you set a lower spend limit yourself, Anthropic usually returns HTTP 400, not 429, when it is reached. The exception is the Claude Code workspace, which can get a 429 with retry-after.

OpenAI separates these similarly: credit_balance_exhausted, organization_spend_limit_exceeded, project_spend_limit_exceeded and organization_usage_limit_exceeded are 429s where retrying does not restore access. So an error classifier must read the type and detail code, not only the HTTP status; spending errors should alert an owner.

How to handle each group of errors

Anthropic's errors page (read October 23, 2026) lists 400, 401, 402, 403, 404, 409, 413 (request too large), 429, 500, 504 and 529 (overloaded). For 500 it advises retrying with backoff; for 504, streaming for long requests.

OpenAI uses 503 (server_is_overloaded) for overload: follow Retry-After if present, otherwise lengthen the gap between attempts. The Google SRE book advises separating retriable from non-retriable errors and not retrying permanent errors or malformed requests. The table at the top is how Click Star turns these sources into decisions.

Retrying with backoff and randomness

Per RFC 9110 (June 2022), Retry-After is either an HTTP date or a number of seconds to wait. Anthropic states that retrying earlier will fail, and not every 429 carries the header.

Without the header, use capped backoff plus randomness. An AWS Architecture Blog article (Marc Brooker, March 4, 2015, updated May 2023) simulates 100 clients contending for database writes: backoff without randomness cut work only slightly, while adding jitter cut calls by more than half.

Limitation: the simulation concerns databases, not AI APIs; the principle is general, the figures may not transfer. The Google SRE book advises limiting retries per request, with an example budget of 60 retries per minute per process, after which requests fail.

A common way to add randomness is to wait a random time between 0 and the current backoff ceiling (Click Star's description). Set attempts, ceiling and deadline per feature: a chatbot with a customer waiting needs a much shorter deadline than an overnight job.

The SDK already retries for you

Per Anthropic's Python SDK documentation, connection errors, 408, 409, 429 and 500-and-above errors are retried automatically 2 times by default; max_retries changes or disables this. OpenAI's Python SDK README (GitHub) describes the same.

So each call may already be up to 3 attempts. Wrap another 3-attempt loop around it and 1 request becomes up to 9 (Click Star's own multiplication). OpenAI states that failed requests still count toward the per-minute limit.

Timeouts, streaming and cut-off streams

The Python SDKs' default timeout (Anthropic and OpenAI) is 10 minutes. Anthropic advises streaming or the batch API for long requests, as some networks drop idle connections.

Streaming has a trap: Anthropic states an error can arrive after the API has returned 200, as an error event in the stream (such as overloaded_error, equivalent to 529). OpenAI's SDK README says reading a stream is not retried automatically.

Anthropic describes recovery: save what was received and send a new request containing it so the model continues; tool-use and thinking blocks cannot be partly recovered. Check stop_reason: max_tokens means the answer was cut off and should not go straight into a CRM or to a customer.

If the next step has side effects (creating a customer, sending a message), use an idempotency key. The AWS Builders' Library article on idempotent APIs says an API with side effects needs to be idempotent to be retried safely. Click Star already has an article on webhooks.

Queues, concurrency limits and batch APIs

Prevention beats cure: put a queue and a concurrency limit before the API to even out load (a Click Star proposal). Read the response headers for remaining allowance: anthropic-ratelimit-*-remaining at Anthropic, x-ratelimit-remaining-requests and x-ratelimit-remaining-tokens at OpenAI.

For work needing no immediate answer, a batch API avoids competing for limits with real-time traffic. Anthropic states a 50% cost reduction, most batches finishing within 1 hour, up to 100,000 requests or 256 MB per batch, and expiry after 24 hours unfinished.

OpenAI states a 50% cost reduction, completion within 24 hours (often sooner), up to 50,000 requests and a 200 MB input file per batch, and limits separate from per-model limits. Anthropic's batch results may return out of order and must be matched by custom_id.

Fallback models: when they help

A fallback means that after the planned attempts the application switches to another model or provider. It helps only with temporary errors (overload, 5xx, rate limits), not configuration errors or a spending cap.

Check whether the fallback shares limits with the primary. Anthropic's limits table has rows pooling several models under one limit and rows with their own; switching to a model in the same pool does not solve a 429.

Per the rate limits page, reaching the spend cap pauses API use for the organization. Click Star's inference: falling back to another model in the same organization will not rescue this case.

Falling back to another provider has a price: different quality and style, and customer data reaching a 2nd party, so review data-processing terms.

Click Star advises running your sample questions through the fallback model before you need it, and logging which model answered.

Spending cap and logging

Anthropic lets you set a spend limit below the tier cap, at organization and workspace level (not the default workspace); OpenAI has error codes for organization and project spend limits. Click Star advises alerting before the cap, because an error loop or a leaked key can burn money fast.

Anthropic returns a request ID in the request-id header and error body; log it for every failed call to include when contacting support.

Minimum log per call (a Click Star reference framework): time, feature, model, attempts, error code, wait time, whether the fallback was used, and input and output tokens. Do not log full content if it holds personal data.

Click Star's view. A 429 is a normal signal from a shared service. The hard part is not the retry code but classifying errors correctly, limiting amplification and deciding in advance what happens when options run out.

Start simple: classify errors, retry with randomness at 1 layer, add a deadline, a spending cap and logging. Add a queue and a 2nd-provider fallback only when logs show the need.

Disclosure: Click Star sells AI integration and AI automation services. The points above are technical judgments, not measurements from Click Star projects.

What to do: a 4-week roadmap

A Click Star reference framework, not from external sources or tested on Vietnamese data.

• Week 1: list every place that calls an AI API; note the max_retries in use and any hand-written retry loop on top.
• Week 2: write an error classifier (retry, do not retry, alert the owner); set a deadline per feature.
• Week 3: add backoff with randomness, a retry budget, a spending cap, alerts and logging.
• Week 4: move non-urgent work to the batch API; simulate 429 and 529 errors in a test environment; run sample questions through the fallback model.

How to measure whether the system is healthy

A Click Star reference framework, with no industry-standard thresholds. Track 6 metrics: share of requests getting 429 and 5xx, average retries per request, high-percentile latency, share using the fallback, share of truncated results (stop_reason is max_tokens) and daily spend against the cap.

Common mistakes

1. Treating every 429 as retriable, when hitting a spending cap needs a person.
2. Catching only status 429 and missing the HTTP 400 returned at a self-set spend limit.
3. Retrying immediately, without waiting or randomness, so all clients retry together.
4. Retrying at several layers (SDK, wrapper, queue, interface), multiplying the calls.
5. Setting max_tokens too high with a provider that counts it toward the limit.
6. Writing a truncated result into the CRM without checking stop_reason.
7. Having a fallback model that was never tested, or one that shares limits with the primary.

What is not known

Click Star has no measurements of latency or error rates when calling these APIs from Vietnam. The Anthropic and OpenAI limit pages show no update date, so the article gives no tier limits or prices.

Click Star has not read other providers' documentation (such as Google Gemini), resale through cloud platforms, or SDKs outside Python. No source read states whether failed synchronous requests are billed. The roadmap and metrics are untested at real volume.