We launched a streaming chat interface on top of Claude. The first version worked fine in staging with five concurrent testers. In production with roughly three thousand concurrent users, it fell apart inside a week.
The failure mode was not what we expected. The LLM side was fine. The streaming protocol was fine. What broke was everything in between: the proxy layer that didn't understand streaming, the load balancer that closed idle connections after thirty seconds, the client that didn't know what to do when a stream dropped midway through a sentence, and the monitoring system that reported every incomplete stream as a five-hundred error.
This post covers what we rebuilt and why.
Why Streaming Matters (And Why It's Harder Than It Looks)
A non-streaming LLM call waits until the model finishes generating before returning anything. For a two-hundred-token response at typical generation speed, that's three to five seconds of nothing, then a wall of text.
Streaming returns tokens as they're generated. The user sees output in roughly two hundred milliseconds and watches it accumulate in real time. Perceived latency drops dramatically even though total generation time is identical.
The implementation complexity is the catch. Non-streaming is a request/response cycle. Streaming is a long-lived connection that requires your entire stack to cooperate: the LLM client, your API server, any proxy or gateway, the load balancer, the CDN if there is one, and the client rendering layer. Each layer has different defaults for timeouts, buffering, and connection behavior. Getting all of them right takes deliberate configuration.
SSE vs WebSocket: The Actual Tradeoff
Most teams reach for WebSockets for streaming LLM output. We did too, initially. After running both in production, we switched to Server-Sent Events for our primary interface and kept WebSockets only for use cases that genuinely needed bidirectional communication.
Why SSE won for us:
SSE is HTTP. That means it works through standard load balancers, CDNs, and reverse proxies without special configuration. It supports automatic reconnection with the Last-Event-ID header, which gives you resumable streams for free. Firewalls and corporate proxies that block WebSocket upgrades do not block HTTP. Browser support is universal and the API is simple.
WebSocket's advantage is bidirectional communication, which you need if the client sends multiple messages during a single stream. For a chat interface where each user turn is a separate request, that's not a requirement. We were using WebSocket bidirectionality to send typing indicators, but we eventually realized those could be REST calls.
The practical difference in implementation:
# SSE implementation with FastAPI
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import anthropic
import asyncio
app = FastAPI()
client = anthropic.AsyncAnthropic()
async def generate_stream(prompt: str):
"""Generate SSE events from LLM stream."""
try:
async with client.messages.stream(
model="claude-opus-4-5",
max_tokens=1024,
messages=[{"role": "user", "content": prompt}]
) as stream:
async for text in stream.text_stream:
# SSE format: data: <payload>\n\n
yield f"data: {json.dumps({'token': text})}\n\n"
# Send done signal
yield f"data: {json.dumps({'done': True})}\n\n"
except anthropic.APIError as e:
yield f"data: {json.dumps({'error': str(e)})}\n\n"
@app.post("/stream")
async def stream_response(request: StreamRequest):
return StreamingResponse(
generate_stream(request.prompt),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache",
"X-Accel-Buffering": "no", # Disable nginx buffering
"Connection": "keep-alive",
}
)
The X-Accel-Buffering: no header is critical if you're behind nginx. Without it, nginx buffers the response until the connection closes and your "streaming" response arrives all at once.
The Backpressure Problem
When the LLM generates tokens faster than the client can consume them, tokens queue in memory on the server. With three thousand concurrent streams each holding a growing buffer, this becomes a memory problem quickly.
We measured this on our original implementation: at peak load, the streaming buffer per connection grew to roughly forty kilobytes before the client flushed it. Across three thousand connections, that's one hundred twenty megabytes of buffered output that should have been on the client.
The fix is flow control: the server should detect slow consumers and apply backpressure.
import asyncio
from asyncio import Queue
class BackpressureStream:
def __init__(self, max_queue_size: int = 50):
self.queue: Queue = Queue(maxsize=max_queue_size)
self.done = False
async def producer(self, prompt: str):
"""Feed tokens into queue from LLM."""
try:
async with client.messages.stream(
model="claude-opus-4-5",
max_tokens=1024,
messages=[{"role": "user", "content": prompt}]
) as stream:
async for text in stream.text_stream:
# put() blocks when queue is full → backpressure
await self.queue.put({"token": text})
await self.queue.put({"done": True})
except Exception as e:
await self.queue.put({"error": str(e)})
finally:
self.done = True
async def consumer(self):
"""Yield SSE events, applying backpressure automatically."""
while True:
item = await self.queue.get()
yield f"data: {json.dumps(item)}\n\n"
if item.get("done") or item.get("error"):
break
@app.post("/stream")
async def stream_response(request: StreamRequest):
stream = BackpressureStream(max_queue_size=50)
# Start producer in background
asyncio.create_task(stream.producer(request.prompt))
return StreamingResponse(
stream.consumer(),
media_type="text/event-stream",
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}
)
The Queue(maxsize=50) creates the backpressure mechanism. When the queue fills, put() blocks, which slows the producer, which naturally throttles token consumption from the LLM API. The client controls pacing implicitly through how fast it reads.
Timeout Configuration Across the Stack
The second failure mode was timeouts. An LLM generating a long response takes time. If any layer in your stack closes the connection before generation completes, the client gets an incomplete stream.
Things that will kill your stream if not configured:
Load balancer idle timeout. Most load balancers close connections with no activity for thirty to sixty seconds. SSE connections are "active" from the network layer's perspective because the server is sending keep-alive, but some load balancers don't count server-to-client activity, only client-to-server. Check your specific load balancer documentation.
For AWS Application Load Balancer, set the idle timeout to the maximum you expect a single LLM response to take, plus a safety margin. We use three hundred seconds.
Nginx proxy timeout. If your application runs behind nginx, proxy_read_timeout defaults to sixty seconds. Set it to match or exceed your load balancer timeout.
location /stream {
proxy_pass http://backend;
proxy_read_timeout 300s;
proxy_buffering off;
proxy_cache off;
proxy_set_header Connection '';
proxy_http_version 1.1;
chunked_transfer_encoding on;
}
LLM client timeout. The Anthropic SDK default timeout is ten minutes for streaming. That's usually fine, but set it explicitly so you know what you're working with:
client = anthropic.AsyncAnthropic(
timeout=anthropic.Timeout(
connect=5.0, # Connection establishment
read=300.0, # Time to receive each chunk
write=10.0, # Time to send the request
pool=5.0, # Time to acquire connection from pool
)
)
Keep-alive ping. For long responses, send a keep-alive comment every fifteen seconds to prevent intermediate network equipment from closing the connection:
async def generate_stream_with_keepalive(prompt: str):
last_ping = asyncio.get_event_loop().time()
async with client.messages.stream(...) as stream:
async for text in stream.text_stream:
current_time = asyncio.get_event_loop().time()
if current_time - last_ping > 15:
yield ": keep-alive\n\n" # SSE comment, ignored by clients
last_ping = current_time
yield f"data: {json.dumps({'token': text})}\n\n"
Handling Partial Output
When a stream drops midway through generation, you have partial output. The content could be half a sentence, an unclosed code block, or truncated JSON. The right handling depends on what you're building.
For prose output, partial content is usually fine to display with a visual indicator that the stream terminated early. The user can see where it cut off.
For structured output (JSON, code), partial content is often unparseable. We added a partial output validator that runs when a stream terminates abnormally:
import json
from enum import Enum
class StreamTermination(Enum):
COMPLETE = "complete"
TRUNCATED = "truncated"
ERROR = "error"
class StreamResult:
def __init__(self, content: str, termination: StreamTermination,
stop_reason: str | None = None):
self.content = content
self.termination = termination
self.stop_reason = stop_reason
self.is_valid_json = self._check_json()
self.unclosed_code_blocks = self._count_unclosed_code_blocks()
def _check_json(self) -> bool:
try:
json.loads(self.content)
return True
except (json.JSONDecodeError, ValueError):
return False
def _count_unclosed_code_blocks(self) -> int:
blocks = self.content.count("```")
return blocks % 2 # Odd count means unclosed block
async def stream_with_validation(prompt: str) -> AsyncGenerator[dict, None]:
accumulated = []
termination = StreamTermination.ERROR
stop_reason = None
try:
async with client.messages.stream(
model="claude-opus-4-5",
max_tokens=1024,
messages=[{"role": "user", "content": prompt}]
) as stream:
async for text in stream.text_stream:
accumulated.append(text)
yield {"token": text}
final_message = await stream.get_final_message()
stop_reason = final_message.stop_reason
termination = (StreamTermination.COMPLETE
if stop_reason == "end_turn"
else StreamTermination.TRUNCATED)
except anthropic.APIStatusError:
termination = StreamTermination.ERROR
finally:
result = StreamResult(
content="".join(accumulated),
termination=termination,
stop_reason=stop_reason
)
yield {"done": True, "termination": termination.value,
"stop_reason": stop_reason,
"has_unclosed_code_blocks": bool(result.unclosed_code_blocks)}
The client uses the termination metadata to decide whether to show a "response was cut off" indicator and whether to offer a "continue" option.
Client-Side Reconnection
SSE supports automatic reconnection via the browser's EventSource API, but the default behavior retries the full request from the beginning. For LLM streaming, you want to resume from where you left off.
This requires server-side support for resumption:
// Client-side streaming with resumption
class ResumableStream {
private eventSource: EventSource | null = null;
private accumulated: string = '';
private lastEventId: string = '';
async connect(requestId: string, onToken: (token: string) => void) {
const url = `/stream?request_id=${requestId}&resume_from=${this.lastEventId}`;
this.eventSource = new EventSource(url);
this.eventSource.onmessage = (event) => {
this.lastEventId = event.lastEventId || '';
const data = JSON.parse(event.data);
if (data.token) {
this.accumulated += data.token;
onToken(data.token);
}
if (data.done || data.error) {
this.eventSource?.close();
}
};
this.eventSource.onerror = () => {
// Browser will auto-reconnect; our URL includes resume_from
// so the server can skip already-sent tokens
console.log('Stream disconnected, reconnecting...');
};
}
}
Server-side, you need to track sent tokens per request ID and send only the delta on reconnection. We use Redis for this with a short TTL:
import redis.asyncio as redis
async def generate_stream_resumable(request_id: str, prompt: str,
resume_from: int = 0):
r = redis.Redis()
token_count = 0
async with client.messages.stream(
model="claude-opus-4-5",
max_tokens=1024,
messages=[{"role": "user", "content": prompt}]
) as stream:
async for text in stream.text_stream:
token_count += 1
# Cache every token with request_id prefix
await r.rpush(f"stream:{request_id}", text)
await r.expire(f"stream:{request_id}", 300) # 5-minute TTL
# Skip tokens already sent on reconnection
if token_count <= resume_from:
continue
yield f"id: {token_count}\ndata: {json.dumps({'token': text})}\n\n"
This adds complexity. We only implemented resumption for our highest-traffic endpoint. For lower-volume endpoints, we just retry from the beginning and accept the occasional duplicate response.
Monitoring Streaming Endpoints
Standard HTTP monitoring doesn't work well for streaming. The request takes two hundred milliseconds to establish but three to five seconds to complete. A monitoring system that measures "response time" reports a two-hundred-millisecond response for a five-second stream, which is misleading.
Metrics that actually matter for streaming:
Time to first token (TTFT): How long from request to first token received. This is the perceived latency from the user's perspective. Track this as a percentile distribution.
Token generation rate: Tokens per second. Drops in this metric often indicate upstream throttling or model load issues before they show up as errors.
Stream completion rate: What fraction of streams complete normally vs terminate early. Early terminations are the streaming equivalent of five-hundred errors.
Stream duration: Total time from first to last token. Useful for capacity planning.
import time
from dataclasses import dataclass, field
from prometheus_client import Histogram, Counter, Gauge
TTFT = Histogram('llm_time_to_first_token_seconds',
'Time to first token', buckets=[0.1, 0.2, 0.5, 1.0, 2.0])
TOKEN_RATE = Histogram('llm_tokens_per_second',
'Token generation rate', buckets=[5, 10, 20, 50, 100])
COMPLETION_RATE = Counter('llm_stream_completions_total',
'Stream completions', ['status'])
ACTIVE_STREAMS = Gauge('llm_active_streams', 'Currently active streams')
@dataclass
class StreamMetrics:
start_time: float = field(default_factory=time.time)
first_token_time: float | None = None
token_count: int = 0
def record_first_token(self):
if self.first_token_time is None:
self.first_token_time = time.time()
TTFT.observe(self.first_token_time - self.start_time)
def record_token(self):
self.token_count += 1
def finalize(self, status: str):
duration = time.time() - self.start_time
if duration > 0 and self.token_count > 0:
TOKEN_RATE.observe(self.token_count / duration)
COMPLETION_RATE.labels(status=status).inc()
ACTIVE_STREAMS.dec()
async def monitored_stream(prompt: str):
metrics = StreamMetrics()
ACTIVE_STREAMS.inc()
try:
async with client.messages.stream(
model="claude-opus-4-5",
max_tokens=1024,
messages=[{"role": "user", "content": prompt}]
) as stream:
async for text in stream.text_stream:
metrics.record_first_token()
metrics.record_token()
yield f"data: {json.dumps({'token': text})}\n\n"
metrics.finalize("complete")
yield f"data: {json.dumps({'done': True})}\n\n"
except Exception as e:
metrics.finalize("error")
yield f"data: {json.dumps({'error': str(e)})}\n\n"
What We'd Do Differently
The biggest mistake was treating streaming as a simple wrapper around the LLM API. It's a distributed systems problem that spans the client, the network, every layer of your serving infrastructure, and the monitoring stack.
The changes that had the most impact, in order:
- Disabled nginx response buffering. Fixed most of our "delayed streaming" complaints immediately.
- Increased load balancer idle timeout. Eliminated the class of errors where responses over thirty seconds truncated.
- Added time-to-first-token as a primary metric. Made it obvious when upstream latency spiked.
- Implemented the backpressure queue. Dropped per-process memory usage by roughly sixty percent under load.
- Added client-side incomplete stream detection. Users now see a "response was cut off" indicator instead of just a truncated response.
Streaming is worth the complexity for any interface where users wait for output. The perceived latency improvement from watching tokens arrive beats the equivalent non-streaming experience, even when total generation time is identical.
Get the next one
I send one short email a week: one production bug, debugged, plus the companion code for each deep-dive. No spam, unsubscribe anytime.
Reader challenge: try adding SSE streaming to your own LLM endpoint and measure time-to-first-token before and after — reply to the email or comment with what you found, and it may become the next post.
Sources
- Anthropic Python SDK — Streaming — official documentation for streaming API usage and async patterns
- Server-Sent Events specification — WHATWG living standard defining SSE protocol, reconnection semantics, and event ID behavior
- nginx proxy_buffering documentation — official nginx reference for disabling response buffering on streaming endpoints
About the Author
Toc Am
Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.
Published: 2026-07-27 · Written with AI assistance, reviewed by Toc Am.
Get These In Your Inbox
Weekly deep-dives on AI engineering, no fluff. Join the newsletter →
Or grab the book ($39, ~100 pages) · Buy me a coffee
☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter
No comments:
Post a Comment