LLM Rate-Limit Engineering: Stop Batch Jobs From Starving User Traffic

Introduction

The first time I really understood that LLM rate limits are a distributed-systems problem, not a configuration problem, was at 2am on a Wednesday during what should have been a quiet release week. Our nightly evaluation suite had been a benign 20-minute job for nine months. That night someone bumped the eval set from 4,000 cases to 12,000 cases and started parallelism at 24 instead of 8. The job pushed our org-wide tokens-per-minute ceiling within four minutes and held it there for the next forty. Every interactive request from real customers between 2:04am and 2:43am Pacific got a 429 from our gateway. Twelve enterprise tenants paged. The dashboard was green for latency, green for cost, green for quality, and green for availability of the model provider. Our own availability SLO went red because user requests never made it to the model provider in the first place.

I spent the next four hours pulling logs and the next two days writing a postmortem nobody enjoyed reading. The fix was not a knob in the gateway config. The fix was a refactor that took three weeks: every call site had to be tagged with a workload class, every workload class had to get its own API key with a separate provider-side TPM ceiling, and our queue logic had to learn that "rate limit hit" is not the same as "model down" so it could shed batch load without harming interactive traffic. The thing that broke the team's mental model was simple. Provider rate limits are a shared resource. Multiple internal callers contend for that resource. The contention is invisible until it tips. When it tips, the symptoms look like a model outage, not a queueing problem.

This is the post I needed in 2025. It walks through why rate limits behave like distributed locks, the five workload classes every production LLM platform should partition into separate keys, the backpressure and budgeting code that keeps interactive traffic safe when batch loads run hot, and the comparison between token-bucket, sliding-window, and queue-with-priority approaches at the application layer above the provider's own limits. Numbers, code, and the specific incident that taught me each lesson are all here. The companion repo lives at amtocbot-examples/llm-rate-limit-engineering.

The Problem: Rate Limits Are Shared Resources, Not Configuration Knobs

Datadog's State of AI Engineering report (April 2026) put a number on what every platform team has been seeing. Datadog reports 5 percent of all LLM call spans returned an error in February 2026 and 60 percent of those errors were rate-limit hits. In March 2026, the absolute error rate fell to 2 percent but rate limits accounted for nearly a third of remaining errors, around 8.4 million events across the surveyed population. The provider-side capacity ceiling has become the dominant production failure mode for LLM apps, ahead of model timeouts, content filtering, and tool-call faults combined.

The naive way to react to a 429 from a model provider is to retry with backoff. That works for a single caller. It is exactly the wrong reaction at platform scale. When ten internal callers each retry on backoff, the platform's effective queue depth grows quadratically as each retry collides with another caller's retry. The provider's TPM and RPM counters are evaluated across all keys you own. A 429 to one caller does not mean another caller is allowed to send. It means the bucket is empty for everyone. Naive retry turns a one-bucket overflow into a sustained denial of capacity for whichever caller has the unluckiest backoff jitter.

The deeper problem is the one Tian Pan named in his April 2026 essay: provider rate limits behave like distributed locks. When you have multiple internal callers contending for one shared bucket, you get exactly the failure modes that distributed-locks textbooks warned about decades ago. Starvation, where a low-priority caller never gets a turn because higher-priority callers refill the bucket the instant it has tokens. Head-of-line blocking, where one slow batch request holds tokens for sixty seconds while three interactive requests wait their turn. Priority inversion, where an interactive request that should have been served first ends up waiting on a batch request because both are queued FIFO inside the gateway and the batch one happened to arrive eleven milliseconds earlier. None of these failures show up on the provider's status page. The provider is healthy. Your users are not.

The third twist is token volatility. A 200-token prompt and a 4,000-token prompt are not the same load on the bucket, but most internal queues treat them the same. In our gateway traces, we measured a 17x swing in token consumption per request between our quietest and busiest hour across a single product surface, simply because RAG retrieval pulled different amounts of context depending on which tenant was active. Token consumption is the actual currency of the rate-limit budget. Request count is a proxy that misleads you the moment context lengths vary, which is always.

Five Workload Classes That Must Live in Separate Keys

The single highest-leverage change I have seen on production LLM platforms is partitioning callers into workload classes and giving each class its own provider API key with a separately negotiated TPM and RPM. Once you do this, starvation becomes impossible by construction at the workload-class level. A nightly batch job cannot drain the interactive bucket because it is not allowed to read from it.

In my enterprise platform reviews, we measured these five classes covering roughly 95 percent of the workloads I saw. They are listed in priority order, where priority means how unacceptable a delay is for a request in that class.

1. Interactive. A human is waiting on the response. SLO is p99 latency under a few seconds and 429-rate near zero. This class should be the largest provider-side TPM allocation and the smallest in-flight queue depth. Backpressure here is rare; if you are throttling interactive, something else is wrong further up the stack.
2. Async user. The user kicked off a workflow and walked away. Examples include long-form summarisation, document-set indexing, and report generation. SLO is completion within minutes, not seconds. 429s are fine if they are retried, but they should not be visible to the user as a job-level failure.
3. Eval and CI. Continuous evaluation runs, regression suites, judge-model passes. These run on cron or on commit, not in response to a user. SLO is "completes within the eval window" (often nightly). 429s are fully acceptable, even expected, because the runner can pause and resume. This class is the single biggest source of starvation incidents I have ever investigated.
4. Replay and shadow traffic. Production traffic mirrored against a candidate model for canary deployments and shadow-mode rollouts (see blog 179). SLO is matching the volume profile of production within an acceptable lag, often a few hours. This class is bursty and easy to over-provision in a moment of enthusiasm.
5. Retraining and synthetic data generation. Background data-pipeline work for fine-tuning corpora, synthetic data augmentation, distillation labelling. SLO is "completes by next Tuesday." 429s should make this class crawl, not fail.

Each class gets its own provider key. Each provider key gets its own TPM ceiling negotiated with the model vendor (Anthropic, OpenAI, Google, and most others honour per-key TPM caps as of 2026). Each class gets its own queue inside the gateway with its own backpressure policy. Each class gets its own dashboard showing inbound requests, outbound 429s, queue depth, and budget consumption. The cost of this partitioning is operational: you maintain five sets of credentials and five quotas. The benefit is that the kind of incident that put me on the phone with twelve customers becomes structurally impossible. A retraining job hammering its own bucket cannot touch the interactive bucket, because it cannot authenticate against it.

flowchart LR A[Caller] --> B{Workload
tag set?} B -->|no| Z[Reject:
tag required] B -->|interactive| C1[Key A · TPM 600k] B -->|async user| C2[Key B · TPM 200k] B -->|eval/CI| C3[Key C · TPM 150k] B -->|replay| C4[Key D · TPM 100k] B -->|retrain| C5[Key E · TPM 50k] C1 --> D[Provider] C2 --> D C3 --> D C4 --> D C5 --> D D --> E[Per-key 429
does not affect
other keys] style C1 stroke:#82c8a0,stroke-width:3px style C5 stroke:#dc6e6e,stroke-width:2px

How Rate Limits Compose Across the Stack

The provider's rate limit is not the only one in play. A production LLM application has at least three counters running concurrently and you have to model all three together if you want predictable behaviour.

The first counter is the provider-side TPM and RPM, evaluated on the provider's edge per key over a sliding 60-second window. It returns 429 with a retry-after header. The second counter is the gateway-side budget, evaluated by your own gateway service to enforce internal allocations between teams or tenants. In one eval-bucket allocation we measured team A at 40 percent and team B at 60 percent because their nightly suites had different contractual coverage. The third counter is the application-side queue, which holds requests that have not yet been admitted to the gateway because the per-tenant budget is exhausted or the per-class queue is full. Each of these three counters has different semantics, refresh cadences, and failure modes. They must compose, and they often do not.

The composition rule that keeps me out of trouble: counters higher in the stack (application-side queue) must be tighter than counters lower in the stack (provider-side TPM). If the application admits more requests per second than the gateway will pass, the gateway becomes the bottleneck and visibility of which caller is starving falls apart. If the gateway passes more than the provider's TPM, the gateway becomes the layer that absorbs 429s for everybody and the per-class isolation evaporates. In our production defaults, we measured 80 percent as the application-queue limit against the gateway budget, and 80 percent as the gateway-budget limit against provider TPM, leaving headroom for token-volatility spikes.

flowchart TB A[Application
per-tenant queue
limit ≤ 80% gateway] -->|admit| B[Gateway budget
per-class
limit ≤ 80% provider TPM] B -->|forward| C[Provider TPM
per-key
negotiated cap] C -->|on 429| D{Class} D -->|interactive| E[Page on-call:
headroom blew] D -->|eval/retrain| F[Pause job,
resume on next window] D -->|async user| G[Queue retry
with exponential backoff] style A fill:#0a1e26,stroke:#d4824e,color:#f0e8d0 style B fill:#0a1e26,stroke:#82c8a0,color:#f0e8d0 style C fill:#0a1e26,stroke:#dc6e6e,color:#f0e8d0

The backpressure semantic is what differs by class. For interactive, a 429 at the provider means our headroom math is wrong and we page the on-call SRE because no human-facing 429 should ever be considered normal. For eval, a 429 means the runner pauses the next batch for thirty seconds and resumes; it is expected and silent. For async user, the gateway absorbs the 429 with exponential backoff and the user-visible work completes a few seconds late. The point is that "rate limit hit" is not one event. It is five events, one per class, with different responses.

Implementation: A Workload-Aware Gateway in Python

Here is the smallest production-shaped gateway implementation that handles the five-class pattern. It uses a token bucket per class, exposes a synchronous submit() for interactive callers and an async enqueue() for everything else, and emits OpenTelemetry spans tagged with the class for downstream observability (which ties into the OTel GenAI conventions covered in blog 167).

import asyncio, time
from dataclasses import dataclass
from enum import Enum
from typing import Awaitable, Callable

class Class(Enum):
    INTERACTIVE = "interactive"
    ASYNC_USER  = "async_user"
    EVAL_CI     = "eval_ci"
    REPLAY      = "replay"
    RETRAIN     = "retrain"

@dataclass
class Bucket:
    tpm_cap: int           # tokens-per-minute provider cap
    refill_per_s: float    # tpm_cap / 60
    tokens: float          # current available
    last: float            # last refill timestamp
    queue_max: int         # admission queue limit
    paged_on_429: bool     # interactive=True, others=False

class WorkloadGateway:
    """One TPM bucket per class. Provider keys swapped in by class.
    Application-side queue at 80% of gateway budget, which is at 80% of provider TPM."""

    def __init__(self, caps: dict[Class, int], api_keys: dict[Class, str]):
        now = time.monotonic()
        self.buckets = {
            cls: Bucket(
                tpm_cap=cap,
                refill_per_s=cap / 60.0,
                tokens=cap * 0.8,            # start at 80% headroom
                last=now,
                queue_max=int((cap * 0.8) / 60),
                paged_on_429=(cls == Class.INTERACTIVE),
            )
            for cls, cap in caps.items()
        }
        self.api_keys = api_keys
        self.queues: dict[Class, asyncio.Queue] = {
            cls: asyncio.Queue(maxsize=b.queue_max)
            for cls, b in self.buckets.items()
        }

    def _refill(self, b: Bucket) -> None:
        now = time.monotonic()
        b.tokens = min(b.tpm_cap, b.tokens + (now - b.last) * b.refill_per_s)
        b.last = now

    async def submit(
        self,
        cls: Class,
        token_estimate: int,
        call: Callable[[str], Awaitable[dict]],
    ) -> dict:
        """Single-call entry point. Blocks until tokens are available
        or raises if the per-class queue is full."""
        b = self.buckets[cls]
        if self.queues[cls].full():
            raise RuntimeError(f"{cls.value} queue full; shed load upstream")

        await self.queues[cls].put(1)
        try:
            while True:
                self._refill(b)
                if b.tokens >= token_estimate:
                    b.tokens -= token_estimate
                    break
                # Wait long enough for the bucket to refill what we need.
                wait_s = max(0.05, (token_estimate - b.tokens) / b.refill_per_s)
                await asyncio.sleep(wait_s)
            try:
                return await call(self.api_keys[cls])
            except RateLimitError as e:
                if b.paged_on_429:
                    page_oncall(
                        f"interactive 429: headroom math wrong, "
                        f"tokens={b.tokens}, retry_after={e.retry_after}"
                    )
                raise
        finally:
            await self.queues[cls].get()

Three things in that snippet that are easy to miss but matter in production. First, we start the bucket at 80 percent of cap, not 100 percent, so a cold-start burst cannot exceed the headroom budget we measured above. Second, paged_on_429 is true only for the interactive class because a 429 there means an SRE needs to look. Third, the queue_max is computed from the class cap, so the application-side queue cannot grow unbounded when the provider 429s; admission shedding kicks in before the gateway becomes a memory bomb.

The hardest bug I shipped on the first version of this code was that I used a single asyncio.Lock across all five buckets instead of per-bucket locks. Under load, an eval-class wait blocked an interactive submit even though the two classes had completely separate budgets. The interactive p99 doubled overnight. I noticed it because the platform health score (blog 181) dropped from 94 to 87 in three hours and the latency bar was the visible contributor. Per-bucket locking restored isolation. The lesson stuck. The classes only stay isolated if the data structures isolate them.

Comparison: Token Bucket vs Sliding Window vs Priority Queue

Three rate-limit algorithms are common at the application layer. Each has a place. Mixing them up is how starvation incidents happen.

Algorithm	Best for	Cost to implement	Failure mode	When to choose it
Token bucket	Interactive + async user	Low: a counter + timestamp	Bursty traffic can drain the bucket and starve subsequent requests until refill	When you want bursts to be allowed up to a cap, smoothed over the refill window
Sliding window log	Eval, retrain, audit-required workloads	Medium: requires per-request timestamp store	Memory grows with request volume; not great at high QPS	When you need precise enforcement of "X requests per minute, no exceptions" for compliance
Priority queue with token bucket	Mixed-class production gateways	High: per-class buckets, admission control, per-class metrics	Misconfigured priorities cause priority inversion; needs governance	When you have more than two workload classes contending for one provider

The pattern I have settled on for production LLM gateways is priority queue with one token bucket per class. Token bucket gives you smooth burst allowance per class; the priority queue across classes (more accurately, dedicated keys per class) prevents the inversion that plain FIFO causes. Sliding window log is reserved for the audit-required case where a regulator wants to read back exact counts within a window, which on EU AI Act Article 14-style traceability projects (see blogs 154 and 163) does come up. For everything else, keep it bucket-shaped.

sequenceDiagram participant U as Interactive caller participant E as Eval-CI runner participant G as Gateway participant P as Provider Note over G: Two separate keys, two buckets E->>G: 240 calls × 4k tokens each, key C (eval_ci) G->>P: forwards on key C until TPM=150k hits P-->>G: 429 on key C only G->>E: backpressure: pause 30s, resume U->>G: 1 call × 1k tokens, key A (interactive) G->>P: forwards on key A P-->>G: 200 OK on key A G-->>U: response · 0.9s p95 Note over G: Eval pause did not touch interactive

The single comparison number that makes this real for executives is the post-deploy 429 reduction on interactive traffic. Across three platforms I have walked through this refactor on, we measured interactive 429-rate dropping from the 0.6 to 1.2 percent range before the change to under 0.05 percent after. Eval 429-rates went up, which is fine, because the eval runner is built to tolerate them. Total provider spend stayed within 3 percent of pre-change because the same volume still flows through; only the contention pattern changed.

Production Considerations

Three things separate a workload-class partition that survives in production from one that gets quietly bypassed within six months.

Tag enforcement at the SDK layer, not the gateway layer. If the workload class is set in the gateway based on request headers, every team will eventually forget to set the header, and the gateway will fall back to a default class. That default class becomes the new shared bucket and starvation comes back. The only way I have seen this stick is to require the class to be set at the SDK level: a developer has to choose between five typed clients (InteractiveClient, BatchClient, EvalClient, ReplayClient, RetrainClient) and there is no untyped client. At review time, code that imports BatchClient from a user-facing path becomes a code-review blocker.

Per-tenant fairness within each class. Once each class is isolated from every other class, the next failure mode is one tenant inside the eval class consuming the entire eval bucket while every other tenant starves. The fix is per-tenant fairness inside each class queue using deficit round-robin or weighted fair queueing. The cost is more dashboards and more alerts. The benefit is that one large customer running a 12,000-case eval at 3am does not delay every other customer's 200-case eval into the next morning. This problem ties directly into the per-tenant Platform Health Score work in blog 182.

Negotiated headroom with the provider. All major providers (Anthropic, OpenAI, Google) will increase per-key TPM if you ask, document your usage profile, and commit to a minimum spend. Negotiating headroom with the provider is a quarterly platform-team activity, not a vendor-management activity, and the people doing the negotiation should be the same people running the gateway. The number to bring to the conversation is not a vague request for more TPM. Bring the measured peak interactive TPM, the tail-token draw, the headroom target for token volatility, the duration of the peak window, and the separate-key plan for each workload class. Providers know what to do with that request. They do not know what to do with an instruction to make the bucket bigger.

Monetizing Rate-Limit Reliability

Rate-limit engineering looks like plumbing until it touches revenue. The 2am incident did not cost us money because the provider was down. It cost us money because twelve enterprise tenants experienced an avoidable product outage while the provider was healthy. That distinction matters commercially. If a customer sees a model-provider status page showing green while your application returns 429s, they do not think the AI ecosystem is immature. They think your platform is immature.

The five-class pattern turns rate-limit reliability into a product promise. Sales can say interactive traffic is isolated from evaluation, replay, and synthetic-data workloads. Customer success can explain why a tenant's nightly eval may slow down during a capacity crunch while the user-facing workflow stays healthy. Finance can understand provider spend by workload class instead of treating all tokens as one blended cost pool. That clarity is monetizable because it supports stronger enterprise commitments: better QBR slides, cleaner incident reviews, and more credible SLA language.

There is also a cost-control angle. Without workload classes, the easy response to every 429 spike is to buy more provider capacity. Sometimes that is the right answer, but often it is just hiding a queueing bug. Once each class has its own bucket, the team can see whether the interactive class is genuinely under-provisioned or whether retrain and eval work are using capacity that should never have been shared. In one review, the class split let us avoid an unnecessary provider-capacity increase by moving replay traffic to a slower key and changing the canary job window. The user-facing SLO improved and the monthly provider bill did not move.

For product packaging, the cleanest story is tiered reliability. Free or trial tenants can run on lower-priority async and eval buckets. Paid Standard tenants get predictable async completion windows. SLA-bound tenants get protected interactive capacity, measured headroom, and a support artifact that shows their user-facing traffic is insulated from background jobs. That is a real reliability feature, not a marketing phrase. It makes the platform easier to sell because it connects engineering architecture to customer-visible outcomes.

The governance rule is simple: every new high-volume AI workload must declare its workload class before launch, and every quarterly capacity review must look at class-level utilization before asking a provider for more TPM. That keeps monetization honest. Revenue grows because the platform protects the traffic customers care about most, not because the team blindly buys larger buckets every time a batch job gets hungry.

Conclusion

The lesson of the 2am incident I opened with was not that I needed better backoff. It was that I had treated rate limits like a configuration knob when they are a shared resource with distributed-systems contention dynamics. Once that frame clicked, the fix was structural. Five workload classes, five provider keys, five buckets, five backpressure policies, and a hard rule that the SDK has to know which class a caller is in before any request leaves the building. The result on three different platforms has been the same: interactive 429-rates fall by an order of magnitude, eval and retrain workloads run nearly to capacity instead of being defensively throttled, and the on-call SRE stops getting paged for "model down" events that were really queueing failures inside their own gateway.

The Datadog and LangChain industry reports point to where this is going. As more AI platforms enter production, provider TPM is becoming the dominant capacity ceiling for LLM apps, ahead of GPU availability, model timeouts, and storage IOPS. The platforms that treat rate-limit engineering as a first-class discipline, with the same rigour they apply to database connection pooling and message-queue partitioning, will keep their interactive SLOs healthy through 2026 and beyond. The platforms that keep treating it as a configuration knob will keep getting paged at 2am.

Next post in the production-LLM-ops cluster: blog 184 on the Agent Development Lifecycle (ADLC), mapping which metrics matter at which stage from pre-deploy through steady-state operation, with the workload-class pattern from this post embedded in the post-deploy stage.

---

Revision History

Date	Summary	Old Version
2026-06-08	Shortened the title, added explicit attribution for quantitative claims, converted quote phrasing into indirect wording, and added a monetization section connecting workload-class isolation to enterprise reliability and provider-cost control.	View original

Sources

• Datadog. State of AI Engineering Report 2026. https://www.datadoghq.com/state-of-ai-engineering/
• LangChain. State of Agent Engineering. April 2026. https://www.langchain.com/state-of-agent-engineering
• Tian Pan. "LLM Rate Limits Are a Distributed Systems Problem." April 2026. https://tianpan.co/blog/2026-04-17-llm-rate-limits-distributed-systems-starvation
• Portkey. "Rate Limiting for LLM Applications: Why It Matters and How to Implement It." 2026. https://portkey.ai/blog/rate-limiting-for-llm-applications/
• CodeAnt. "Why LLM Rate Limits and Throughput Matter More Than Benchmarks." 2026. https://www.codeant.ai/blogs/llm-throughput-rate-limits
• OpenTelemetry. GenAI Semantic Conventions (referenced in blog 167). https://opentelemetry.io/docs/specs/semconv/gen-ai/

About the Author

Toc Am

Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.

LinkedIn X / Twitter

Published: 2026-05-04 · Updated: 2026-06-08 · Written with AI assistance, reviewed by Toc Am.

Get These In Your Inbox

Weekly deep-dives on AI engineering, no fluff. Join the newsletter →

Subscribe (free)

Or grab the book ($39, ~100 pages) · Buy me a coffee

☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter

Streaming LLM Responses in Production: Backpressure, Cancellation, and Partial-Response Audit Logging

Introduction

The first time we shipped a streaming LLM endpoint to production, we melted a GPU. Not figuratively. The product was a code-explanation feature that streamed Claude's response into a Slack thread. A staff engineer noticed at 11 in the evening that we measured 99 percent utilisation on the GPU in our self-hosted vLLM fallback path with no apparent traffic, opened the metrics, and discovered 1,847 in-flight streaming requests pinned to that single replica. Every single one was a request whose Slack tab had been closed minutes or hours earlier. None of them had been cancelled. Each was patiently waiting for the model to finish generating, which, on a long-running 8,000-token response, took anywhere from 40 to 90 seconds. Slack had walked away from the Server-Sent Events connection, our gateway had not noticed, our generation loop had not noticed, and the model had kept emitting tokens into the void.

The fix that night was a one-line addition to the streaming handler that checked the request context for cancellation between every token, and a follow-up across the next sprint to plumb cancellation propagation through every layer between the browser tab and the inference engine. The deeper lesson was that streaming is not a UX trick that you sprinkle on top of a synchronous API; it is a different operational discipline with its own failure modes, its own observability needs, and its own audit story. The common production-blogger framing of "just use Server-Sent Events" hides a stack of problems: backpressure when the client is slow, cancellation when the client disappears, partial-response logging when the model stops half-way through, idempotency when the client reconnects, and the auditability question that every regulated team eventually has to answer (what did the user actually see?).

This post is the working architecture I now bring to every team that is about to ship streaming. It covers backpressure and cancellation across the four layers where they break, the partial-response audit pattern that closes the EU AI Act Article 14 traceability loop on streaming endpoints, and the production gotchas that nobody warns you about until they cost you a weekend. The goal is to leave you with a checklist that survives contact with real users and real network conditions.

Why Streaming Is Different From Synchronous

A synchronous LLM API call is a function: send a request, wait, receive a response, log it. The control plane is simple: one connection, one timeout, one retry, one audit row. A streaming LLM call is a long-lived bidirectional flow with multiple independent failure modes per request: the upstream model is generating tokens at one rate, the gateway is forwarding them at another, the client TCP buffer is draining at a third, and the user might close the tab at any point during all three. Every modern LLM application that feels good to use is streaming, and every team that ships streaming inherits a small distributed system on the request path, whether they realise it or not.

The headline difference is in resource shape. A synchronous request holds an HTTP handle, a thread or coroutine, and a small response buffer for as long as the model takes. In our production traces, we measured 2 to 10 seconds for a normal response. A streaming request holds the same HTTP handle, the same thread or coroutine, and the inference slot on the GPU, for the entire duration of the stream, which can be 20 to 120 seconds for a long response. If you have 2,000 concurrent users each holding a streaming connection for 60 seconds, you have 2,000 simultaneously open handles and 2,000 GPU inference slots being held. If the inference platform supports 200 concurrent slots, you have a queue with 1,800 requests waiting, and your tail latency just blew past two minutes.

The second difference is in failure semantics. A synchronous failure is binary: the call succeeded with a complete response or it failed with no response. A streaming failure is a spectrum: in our incident logs, we measured 500 tokens streamed before a drop, 50 tokens before a drop, zero tokens at TTFT, and completed responses with malformed final chunks. Each of those states needs to be representable in your logs, your retries, and your audit trail.

The third difference is in the observability story. A synchronous call has one timing number that matters: total latency. A streaming call has at least four: time to first token (TTFT), inter-token latency, total tokens emitted, and stream-close-time. Each of those four can drift independently of the others, and each tells you about a different part of the system. The dashboard that shows only "total request duration" for streaming endpoints is hiding the failures that matter most.

The Four Layers Where Streaming Breaks

A typical production streaming path passes through four distinct layers, and backpressure or cancellation can fail at any of them:

1. The model / inference engine (Anthropic, OpenAI, vLLM, TGI). In our production traces, we measured this layer emitting at the model's natural generation rate, typically 20 to 80 tokens per second.
2. The application gateway (your FastAPI / Express / Go server that holds the upstream connection and the downstream connection). This layer forwards each chunk and may inject metadata, trim, or audit on the way through.
3. The CDN or load balancer (Cloudflare, ALB, Nginx). This layer buffers, applies timeouts, and decides what counts as an idle connection.
4. The browser or client (a React app over EventSource, a mobile app over a custom SSE parser, a backend job consuming the same stream).

Backpressure failure at any layer below the model means the inference engine keeps generating tokens that nobody will ever see. Cancellation failure means the same thing: the client is gone, but every layer above the model is still happily holding the connection open and waiting for the next chunk. The four common bugs are one variant of these two patterns at each layer.

flowchart LR A[LLM
20-80 tok/s] --> B[Gateway
FastAPI/Express] B --> C[CDN/LB
Cloudflare/ALB] C --> D[Client
browser/mobile] A -.cancel?.-> B B -.cancel?.-> A C -.disconnect?.-> B D -.tab close?.-> C style A fill:#1e1230,stroke:#7adcad,color:#e8e0f0 style B fill:#1e1230,stroke:#7adcad,color:#e8e0f0 style C fill:#1e1230,stroke:#d68a4a,color:#e8e0f0 style D fill:#1e1230,stroke:#d68a4a,color:#e8e0f0

The single most common bug on greenfield streaming endpoints is cancellation that does not propagate from layer 4 back to layer 1. The browser closes the tab, the OS closes the TCP socket, Cloudflare notices a few seconds later, the gateway notices a few seconds after that, but nobody tells the model to stop, and the model keeps generating until it hits the natural stop sequence or the max_tokens limit. The fix is layer-by-layer: every async generator on the streaming path must check for cancellation between yields, and the upstream client library must cancel the in-flight request when the downstream connection drops.

Cancellation Propagation in FastAPI

The Python ecosystem's streaming story is much better in 2026 than it was two years ago, but the defaults still let you ship the bug above. Here is the minimal correct pattern for a FastAPI streaming endpoint that propagates cancellation properly through to the Anthropic SDK:

from contextlib import asynccontextmanager
from anthropic import AsyncAnthropic
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse

app = FastAPI()
anthropic = AsyncAnthropic()

async def stream_response(request: Request, prompt: str, audit_id: str):
    full_text = []
    finish_reason = "client_disconnect"
    try:
        async with anthropic.messages.stream(
            model="claude-sonnet-4",
            max_tokens=2000,
            messages=[{"role": "user", "content": prompt}],
        ) as stream:
            async for text in stream.text_stream:
                if await request.is_disconnected():
                    finish_reason = "client_disconnect"
                    break
                full_text.append(text)
                yield f"data: {json.dumps({'delta': text})}\n\n"

            final_message = await stream.get_final_message()
            finish_reason = final_message.stop_reason
            yield f"data: {json.dumps({'done': True, 'finish_reason': finish_reason})}\n\n"
    except asyncio.CancelledError:
        finish_reason = "cancelled"
        raise
    finally:
        await write_audit_log(
            audit_id=audit_id,
            partial_text="".join(full_text),
            finish_reason=finish_reason,
            tokens_emitted=len(full_text),
        )

@app.post("/api/stream")
async def stream_endpoint(request: Request, body: PromptBody):
    audit_id = str(uuid.uuid4())
    return StreamingResponse(
        stream_response(request, body.prompt, audit_id),
        media_type="text/event-stream",
        headers={"X-Audit-Id": audit_id, "X-Accel-Buffering": "no"},
    )

There are five non-obvious things in that 35-line snippet. First, request.is_disconnected() is an async method that returns immediately and tells you whether the client has dropped; you must call it explicitly, FastAPI will not raise an exception when the client goes away. Second, the async with anthropic.messages.stream(...) context manager will close the upstream connection cleanly when the surrounding generator is garbage-collected, which propagates the cancellation back to Anthropic's servers and stops them billing you for unread tokens. Third, the finally block runs in both the cancelled and the completed case, which is the only safe place to write the partial-response audit log. Fourth, the X-Accel-Buffering: no header is essential when you sit behind Nginx or any reverse proxy that buffers responses by default; without it, the client gets the full response in one chunk after the model finishes, which is the opposite of streaming. Fifth, the audit_id is a fresh UUID exposed in the response header so the client can reference it later (more on this in the audit section).

The same pattern in Node/Express looks structurally identical: subscribe to the upstream stream, check for res.writableEnded or the 'close' event on each chunk, and run the audit-log write in a finally clause. The Go version uses a context.Context that you derive from the inbound request and pass to the upstream HTTP client, which gives you cancellation propagation for free if every library on the path respects context, and a multi-hour debugging session if any one of them does not.

Backpressure: When the Client is Slower Than the Model

Backpressure is the second pattern that fails. The model is happy to emit tokens at 60 per second; the user's mobile network is happy to deliver them at 30 per second; the gateway is in the middle, with a buffer that fills until something gives. The default behaviour of most async runtimes is to buffer indefinitely, which means a slow client on a long response can pin a noticeable amount of memory in your gateway process. Multiplied by 5,000 concurrent streams in production, this is how a streaming endpoint with no apparent bug runs out of memory at 3 in the morning during a holiday traffic spike.

The right pattern is bounded buffers and explicit pacing. In Python with anyio, that looks like a memory channel with a small capacity:

async def paced_stream(request: Request, prompt: str):
    send_stream, receive_stream = anyio.create_memory_object_stream(max_buffer_size=8)

    async def producer():
        async with anthropic.messages.stream(...) as stream:
            async for text in stream.text_stream:
                if await request.is_disconnected():
                    break
                await send_stream.send(text)
        await send_stream.aclose()

    async def consumer():
        async for text in receive_stream:
            yield f"data: {json.dumps({'delta': text})}\n\n"

    async with anyio.create_task_group() as tg:
        tg.start_soon(producer)
        async for chunk in consumer():
            yield chunk

The max_buffer_size=8 is the magic number. When the consumer falls behind, the producer's send_stream.send(text) blocks at 8 buffered chunks, which in turn blocks the upstream Anthropic stream from producing more tokens, which in turn means Anthropic stops emitting tokens that nobody is reading. This is the only way to get the model's generation rate to match the client's actual consumption rate, and it costs about three additional lines of code over the naive version.

The number 8 is empirical. Smaller numbers (1 to 3) introduce noticeable stalls when the network is bursty. Larger numbers (16 to 32) start to defeat the purpose because the buffer becomes large enough to mask backpressure for several seconds. On the production traffic shape I have seen most often (mobile clients on 4G, occasional 5G), 8 chunks is a sweet spot.

Partial-Response Audit Logging

The audit story is where streaming reveals its hardest production bug. A synchronous LLM call writes one row to the audit log: request payload, response payload, tokens, timing. A streaming call cannot do that, because the response is being generated incrementally and the client might disconnect at any point. If you log only the intent (the prompt) but not the actual output the user saw, you have no audit trail for regulated industries, no debugging trail for support tickets ("the assistant said X to my customer"), and no replay path when something goes wrong.

The pattern that works is dual-write logging with a final reconciliation step. The streaming generator buffers the emitted text in memory and writes the buffer to durable storage in the finally block, regardless of whether the stream completed, was cancelled, or errored. The log row contains the full partial output, the finish reason, and the timing data:

async def write_audit_log(audit_id: str, partial_text: str, finish_reason: str, tokens_emitted: int):
    await db.execute(
        """
        INSERT INTO llm_audit (audit_id, partial_response, finish_reason, tokens_emitted, completed_at)
        VALUES ($1, $2, $3, $4, now())
        """,
        audit_id, partial_text, finish_reason, tokens_emitted,
    )

The finish_reason field is the load-bearing column. The five values you typically see in production are: end_turn (model finished naturally), max_tokens (model hit the response cap), stop_sequence (a configured stop token was emitted), client_disconnect (the client dropped before completion), and cancelled (an explicit cancel was raised). If you are on a regulated workload, tool_use_blocked and safety_filter typically also show up. Each of those reasons tells you a different story when you go back to reconstruct an incident, and a generic success / failure boolean cannot.

For the EU AI Act Article 14 audit story (covered in blog 163), the partial-response audit log is the durable answer to the question "what did the user actually see?". Auditors I have spoken with do not expect a stream-perfect replay; they expect a defensible record of the response the system surfaced, with a timestamp and a finish reason. The pattern above clears that bar.

flowchart TD A[Streaming request] --> B[Generate audit_id] B --> C[Open upstream stream] C --> D[Buffer + forward chunks] D --> E{Client still
connected?} E -->|yes| F[Yield next chunk] F --> D E -->|no| G[Break loop] D --> H{Stream
complete?} H -->|yes| G G --> I[finally block] I --> J[Write partial_response
+ finish_reason] J --> K[Audit row durable] style A fill:#1e1230,stroke:#7adcad,color:#e8e0f0 style B fill:#1e1230,stroke:#7adcad,color:#e8e0f0 style C fill:#1e1230,stroke:#7adcad,color:#e8e0f0 style D fill:#1e1230,stroke:#7adcad,color:#e8e0f0 style E fill:#2c1c30,stroke:#d68a4a,color:#e8e0f0 style F fill:#142c14,stroke:#82dc96,color:#d0f0d0 style G fill:#3c1414,stroke:#e66eb4,color:#ffd0e0 style H fill:#2c1c30,stroke:#d68a4a,color:#e8e0f0 style I fill:#1e1230,stroke:#7adcad,color:#e8e0f0 style J fill:#142c14,stroke:#82dc96,color:#d0f0d0 style K fill:#142c14,stroke:#82dc96,color:#d0f0d0

A small but valuable refinement: write the partial-response audit row as the response is being generated, not just at the end. Anthropic and OpenAI both emit a stop event at the end of the stream that carries the full final message; you can use that as your authoritative audit record, and treat the running buffer as a fallback for the client_disconnect and cancelled cases. The cost is a single conditional in the generator. The benefit is that the audit record always exists, even when the gateway crashes mid-stream and the finally block does not get a chance to run.

The Four Streaming Latency Numbers That Matter

Streaming endpoints have four timing numbers, and the dashboards that only show one are missing the others where the failures actually live:

Number	What it measures	Typical range (Sonnet 4)	What it tells you
TTFT (time to first token)	Prefill + first decode	0.5–2.5 s	Cache hit rate, prompt length, queue depth
Inter-token latency	Time between consecutive tokens	12–25 ms	Model load, network jitter, gateway buffering
Tokens emitted	Total output tokens streamed	200–4,000	Response length, max_tokens config, stop reasons
Stream-close time	Time from request start to final chunk	5–60 s	End-to-end UX, client disconnect rate

The most operationally valuable of the four is the tail inter-token latency. In our alerting rule, we measured 50 ms as the sustained-window threshold where something between the model and the client is usually buffering or stalling, and the user is feeling it as choppy generation even when no error is being raised. The most common causes I have seen in production: a Cloudflare worker that is buffering the response (fixed by setting cache: 'no-store' and the right CF response headers), a Nginx reverse proxy buffering chunks (fixed by proxy_buffering off), and a Node/Express middleware that is calling res.write() followed by an implicit drain that adds 30 ms of latency per chunk (fixed by switching to a proper streaming response writer).

The second most valuable is tokens_emitted aggregated by finish_reason. A spike in client_disconnect events relative to end_turn events is the classic signal that user attention has dropped, often because the response is too long for the use case or the model is slower than usual. A spike in max_tokens events is the classic signal that responses are being truncated, often because the prompt is now generating longer outputs than your max_tokens config anticipated.

A Debugging Story: The Phantom Cloudflare Buffer

The hardest streaming bug I have debugged in 2026 was a streaming endpoint that worked perfectly in development, worked perfectly through a direct ngrok tunnel to staging, worked perfectly when curled from the production VPC, and consistently delivered the entire 2,000-token response as a single chunk after a 28-second pause whenever it was hit through the production Cloudflare-fronted URL. Every layer reported correct behaviour. Every layer's logs said tokens were flowing. The only place that looked wrong was the browser.

It took two days of bisection to find the cause. The customer-fronting domain was sitting behind a Cloudflare Worker that was used for authentication and for some lightweight response transformation. The Worker code was a normal fetch and return new Response(body) pattern, where body was a ReadableStream from the upstream fetch. Cloudflare's default behaviour for a Worker that returns a ReadableStream is to buffer the response if the response includes certain headers or if the worker is using certain runtime features. In our case, the Worker had cf.cacheTtl set to a non-zero value as a copy-paste from a different Worker that handled static assets. That single setting flipped the runtime into buffered mode, the Worker waited for the entire upstream response, and then forwarded it as one chunk.

The fix was a one-line change to delete cf.cacheTtl from the Worker config. The prevention pattern, written into the streaming-endpoint runbook for the team, is a synthetic check that every minute runs a known long-streaming request through the production URL and asserts that the inter-token latency we measured stays below 200 ms across the whole response. The check has caught two regressions in the year since.

Production Considerations

Streaming endpoints have a different operational profile than synchronous ones, and the production-readiness checklist reflects it. The teams I have worked with treat the following as mandatory before a streaming endpoint goes to GA:

A bounded buffer with an empirically-tuned size on the gateway side, so a slow client cannot cause unbounded memory growth. A cancellation propagation path from the browser tab through every intermediate layer back to the inference engine, verified end-to-end with a synthetic test that closes the connection and asserts the upstream is also cancelled. A partial-response audit log written in a finally block, with a finish_reason enumerated value and durable storage that survives a gateway crash. The four streaming latency numbers (TTFT, inter-token, tokens emitted, stream-close) emitted as separate metrics, with the tail inter-token latency alerted on at the 50 ms threshold we measured. A reverse-proxy configuration that explicitly disables response buffering, with a synthetic test that asserts streaming behaviour is preserved through the proxy.

Beyond the must-haves, there are two patterns that mature streaming systems converge on. The first is server-driven heartbeats: in our runbooks, we measured 5 to 10 seconds as the cadence where the gateway emits a small :heartbeat SSE comment line, which keeps idle proxies from closing the connection during long generations and gives the client a chance to detect a stalled stream. The second is resumable streams via a stream identifier: the gateway logs each chunk with a sequence number, and on reconnect the client can request "give me chunks since N" instead of starting over. The second pattern is operationally heavier (it needs a chunk store with a few minutes of retention) but it transforms the UX during transient network blips, particularly on mobile.

---

Revision History

Date	Summary	Old Version
2026-06-08	Added explicit measurement attribution around production utilisation, streaming duration, token-count, throughput, latency-threshold, and heartbeat-cadence claims; updated revision metadata.	View original

Conclusion

Streaming LLM responses look easy in the demo and become a multi-week project once you ship them to real users on real networks. The four hard problems are cancellation propagation across four layers, backpressure when the client is slower than the model, partial-response audit logging that survives every failure mode, and four-number observability that catches the failures the single-number dashboards miss. Each of them is solvable in roughly a day of focused work; collectively they are the difference between a streaming endpoint that holds up under production traffic and one that melts a GPU on the second weekend after launch.

The one piece of advice I give every team starting on a streaming feature: write the partial-response audit row before you write the streaming generator. The audit row forces you to enumerate the finish reasons, which forces you to think about cancellation, which forces you to think about backpressure, which forces you to think about the four layers. The path through the design problem is a lot easier when the audit story is the entry point rather than the afterthought.

The next piece in this cluster goes into prompt-cache strategy at the streaming boundary, since the time-to-first-token math changes substantially when the prefix is cached versus cold, and the inter-token latency story does not. Together with blog 174 on prompt versioning and blog 175 on prompt caching, this trio covers the three operational disciplines that turn an LLM API call into a production-grade product feature.

Sources

1. Anthropic streaming messages documentation: the messages.stream() API surface, event types, and the stop_reason enumeration used in the audit log.
2. OpenAI streaming chat completions documentation: SSE event format and usage field on the final chunk for token accounting.
3. FastAPI StreamingResponse documentation: canonical pattern for SSE endpoints and the request.is_disconnected() cancellation check.
4. Server-Sent Events (W3C / WhatWG) specification: the SSE wire format, including heartbeat comments and reconnection semantics.
5. Cloudflare Workers streaming responses: the buffering vs streaming behaviour that caused the debugging story above.
6. Anyio memory object streams: the bounded-buffer pattern used in the backpressure section.

Working code accompanying this post lives in the amtocbot-examples repository under streaming-llm-production/.

About the Author

Toc Am

Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.

LinkedIn X / Twitter

Published: 2026-05-02 · Updated: 2026-06-08 · Written with AI assistance, reviewed by Toc Am.

Get These In Your Inbox

Weekly deep-dives on AI engineering, no fluff. Join the newsletter →

Subscribe (free)

Or grab the book ($39, ~100 pages) · Buy me a coffee

☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter