Ollama Is for Your Laptop, Not Your Users: Picking a Real Inference Server for Self-Hosted LLMs

Introduction

I stood up an Ollama-served 7B model for what was supposed to be a two-person internal tool, a coworker and me querying it occasionally during a sprint. It worked great for a week. Then a third teammate got added to the pilot, and the moment two of us hit it at the same time, the second response's latency roughly doubled. Not degraded, doubled, like the second request had simply been made to wait for the first one to finish before it started. Because that's exactly what had happened.

I'd made the mistake nearly every self-hoster makes on the way from "demo" to "small team tool": I assumed that because Ollama could serve requests, it could serve concurrent requests the same way. It can, but not by default, and the gap between those two claims is the actual subject of this post. Ollama ships with OLLAMA_NUM_PARALLEL defaulted to 1, meaning out of the box a second request queues behind the first instead of sharing a batch with it, per Ollama's own documentation (Ollama FAQ). That's not a bug. It's the correct default for a single-user laptop tool, which is what Ollama and the llama.cpp engine underneath it were built for.

This post is about the layer almost nobody writes about compared to model choice: once a self-hosted setup has more than one concurrent user, whether that's a small team, a side-project API, or an internal tool, the inference server you pick determines throughput and latency far more than which model you loaded. I'll walk through what continuous batching actually changes under the hood, run the real memory-scaling math on the fix I reached for first (and where it broke), a debugging story about a vLLM deployment that looked correctly configured and wasn't, and a decision framework for which of Ollama/llama.cpp, vLLM, SGLang, or TensorRT-LLM fits which tier of actual load.

The Problem: Every Tutorial Defaults to Ollama, and That's the Wrong Long-Term Default

Ollama is the easiest on-ramp to self-hosting an LLM, and for good reason: one command pulls a quantized model, one command serves it, and it runs comfortably on a laptop with no GPU cluster required. Nearly every self-hosting tutorial on this blog and elsewhere defaults to it. But Ollama is a wrapper around llama.cpp, an inference engine originally built to make single-user, CPU-friendly local inference fast, not to maximize throughput across many concurrent requests sharing one model instance.

That distinction doesn't matter until it does. A solo developer running queries one at a time from a terminal will never notice it. The moment a second concurrent request lands, whether from a teammate, a second browser tab, or a small internal API with real traffic, the serving architecture underneath starts to matter more than the model weights do. Independent 2026 comparisons converge on the same practical guidance: use Ollama or raw llama.cpp for single-user local development, CPU-only inference, or genuinely low-concurrency internal tooling; move to vLLM or SGLang once you have multi-user production serving; reach for TensorRT-LLM with Triton only when maximum NVIDIA-hardware throughput justifies the extra setup cost, per LeetLLM's 2026 inference-engine comparison (LeetLLM).

The recurring failure mode isn't picking the wrong model. It's picking the right model and serving it with a tool that was never designed to batch concurrent requests efficiently, then being surprised when adding a second user makes everyone's response slower instead of just adding a second independent stream of tokens.

How It Works: Continuous Batching, KV-Cache Sharing, and Why llama.cpp's Defaults Fight Concurrency

The mechanism that actually separates these engines is how they handle multiple in-flight requests against one loaded model. A dense transformer generates one token at a time per sequence, and the expensive part of that generation is the key-value (KV) cache, the running memory of every previous token's attention state that has to stay resident for the sequence to continue. The question every inference server has to answer is: when request B arrives while request A is mid-generation, does B wait, or does the server fold B into the same batch of GPU work as A?

vLLM's answer is PagedAttention. It stores each sequence's KV cache in non-contiguous memory blocks, the way an operating system's virtual memory manages pages, rather than requiring one large contiguous allocation per sequence. Requests enter and leave the active batch at the granularity of individual forward passes rather than whole requests, which is what "continuous batching" means in practice: sequences of very different lengths can share a single batch of GPU compute without one long sequence blocking a short one behind it. Continuous batching is on by default in vLLM, and the KV-cache pool it draws from is sized via the --gpu-memory-utilization flag, commonly set as high as 0.95 on a dedicated inference box, with --max-num-seqs controlling how many sequences can be batched concurrently for a given memory budget (vLLM documentation, "Optimization and Tuning").

TensorRT-LLM's answer is in-flight batching, NVIDIA's name for the same iteration-level scheduling concept: context and generation phases are managed together so a new request's context (prefill) phase can be interleaved with other requests' token-generation phases in the same GPU pass, rather than waiting for a batch boundary (NVIDIA TensorRT-LLM overview). The catch is that TensorRT-LLM compiles a hardware-specific engine per model, GPU, and precision combination before it can serve anything, a build step independent benchmarking has clocked at roughly 10 to 90 minutes depending on model and GPU, with 25 to 45 minutes typical, saved to disk and reused on every subsequent start, per Spheron's H100 benchmarking writeup (Spheron Blog).

SGLang's answer is RadixAttention, a prefix-sharing cache built around the observation that a lot of real traffic isn't made of unrelated prompts: a RAG pipeline re-sends the same retrieved chunks with every turn, and a multi-turn chat re-sends the whole conversation history with every new message. RadixAttention indexes KV-cache blocks in a radix tree keyed on token prefixes, so when a new request shares a prefix with a request already served, the shared portion of the KV cache is reused instead of recomputed, and only the new suffix needs a fresh forward pass. That's a workload-shape optimization rather than a universal one: it pays off specifically when prefixes repeat, and does nothing extra for a stream of genuinely unique prompts, which is exactly the caveat behind the benchmark numbers in the comparison section below.

llama.cpp's answer is parallel slots over one shared, fixed-size KV cache. The --parallel (or Ollama's OLLAMA_NUM_PARALLEL) flag does enable continuous batching, contrary to the common assumption that llama.cpp can only serve one request at a time. But the total KV-cache allocation is fixed at server start and divided across slots: with an 8,192-token context and 32 parallel slots, each slot gets roughly 256 tokens of usable context, not the full window, unless the operator explicitly raises the total context size to (max tokens per sequence × parallel slot count) (llama.cpp GitHub discussion #4130). Ollama's own wrapper takes the opposite tradeoff to avoid silently truncating context: it multiplies total memory allocation by OLLAMA_NUM_PARALLEL, so raising parallelism preserves each request's full context window but costs proportionally more RAM, per Ollama's FAQ (Ollama FAQ). Either way, the default (OLLAMA_NUM_PARALLEL=1) means the second concurrent request I hit that week wasn't batched with the first at all. It queued.

flowchart LR A[Request A arrives] --> S1[Server accepts, starts generating] B[Request B arrives mid-generation] --> Q{Batching-aware server?} Q -->|llama.cpp, parallel=1 default| Wait[B queues behind A, doubles wall-clock latency] Q -->|vLLM PagedAttention / TensorRT-LLM in-flight batching| Batch[B folds into the same forward pass as A] Batch --> Fast[Both finish close to single-request latency] Wait --> Slow[B finishes after A completes, plus its own generation time]

Implementation Guide: The Memory Cost of the Fix I Reached for First

My first instinct after diagnosing the queuing was the obvious one: raise OLLAMA_NUM_PARALLEL so concurrent requests stop serializing. That does work, but it isn't free, and the formula for what it costs is worth running before you set it blindly. Ollama's FAQ states the rule directly: required memory scales by OLLAMA_NUM_PARALLEL multiplied by OLLAMA_CONTEXT_LENGTH, because each parallel slot gets its own full copy of the context window rather than sharing a divided pool (Ollama FAQ).

$ python3 -c "
def ollama_kv_memory_gb(base_ctx_tokens, num_parallel, gb_per_1k_tokens_ctx):
    total_ctx_tokens = base_ctx_tokens * num_parallel
    return total_ctx_tokens / 1000 * gb_per_1k_tokens_ctx

gb_per_1k = 0.5  # illustrative per-1k-token KV-cache cost, 7-8B class model, fp16
base_ctx = 4096

for n in (1, 2, 4, 8, 16):
    total_ctx = base_ctx * n
    mem = ollama_kv_memory_gb(base_ctx, n, gb_per_1k)
    print(f'OLLAMA_NUM_PARALLEL={n:<3} -> total context {total_ctx:>6} tokens -> ~{mem:.1f} GB KV-cache memory')
"
OLLAMA_NUM_PARALLEL=1   -> total context   4096 tokens -> ~2.0 GB KV-cache memory
OLLAMA_NUM_PARALLEL=2   -> total context   8192 tokens -> ~4.1 GB KV-cache memory
OLLAMA_NUM_PARALLEL=4   -> total context  16384 tokens -> ~8.2 GB KV-cache memory
OLLAMA_NUM_PARALLEL=8   -> total context  32768 tokens -> ~16.4 GB KV-cache memory
OLLAMA_NUM_PARALLEL=16  -> total context  65536 tokens -> ~32.8 GB KV-cache memory

That's linear growth in memory for linear growth in parallelism, on top of the memory the model weights already occupy. Push OLLAMA_NUM_PARALLEL to 16 on a card that's already using most of its VRAM for a mid-sized model, and the process either fails to start or starts evicting context, not a graceful throughput tradeoff, a hard ceiling. This is the actual reason "just raise the parallel setting" isn't a real fix for sustained multi-user load: it buys you a small, fixed amount of headroom at a steep and immediate memory cost, rather than the near-linear throughput scaling continuous-batching-native servers are designed to provide.

Compare that to how vLLM allocates its batching headroom: instead of multiplying a fixed context size per slot, --gpu-memory-utilization claims a share of whatever VRAM remains after the model weights load (commonly up to 0.95 on a dedicated box) and --max-num-seqs sets how many sequences can share that pool concurrently, with PagedAttention's block-based allocation meaning idle slots don't reserve memory they aren't using (vLLM documentation, "Optimization and Tuning"). The design difference is the point: llama.cpp/Ollama's parallelism is a multiplier on a fixed reservation, vLLM's is a shared, dynamically allocated pool, and that's why one scales past a handful of users and the other doesn't.

What that actually looks like on the command line, once the memory math above says a given box can afford it, is a single serve command with the two flags this section covered set explicitly rather than left at whatever the last deployment happened to use:

$ vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --gpu-memory-utilization 0.90 \
    --max-num-seqs 256 \
    --port 8000

TensorRT-LLM's equivalent isn't a single command, it's the two-stage build-then-serve rhythm the compile step earlier in this section referred to: a trtllm-build pass that compiles the hardware-specific engine once, the same 10-to-90-minute build step Spheron's benchmarking clocked earlier in this post, followed by a Triton or trtllm-serve launch that reuses the compiled engine on every subsequent start:

$ trtllm-build --checkpoint_dir ./llama-3.1-8b-checkpoint \
    --output_dir ./engines/llama-3.1-8b-h100-fp8 \
    --gemm_plugin auto
$ trtllm-serve ./engines/llama-3.1-8b-h100-fp8 --port 8000

Neither command is exotic, but the flags that matter (--gpu-memory-utilization, --max-num-seqs, the engine directory TensorRT-LLM expects) are exactly the ones the debugging story below shows getting left at a stale default.

Debugging Story: The vLLM Deployment That Looked Correct and Wasn't

A few weeks after the Ollama queuing incident, once the pilot had grown enough to justify moving to vLLM, I hit a second, quieter failure. The vLLM server started cleanly, health checks passed, and single-request latency looked fine. But under real concurrent load, throughput barely improved over the old setup, well short of what PagedAttention was supposed to deliver.

The cause was --gpu-memory-utilization left at a low default carried over from an earlier test deployment against a much smaller model, one that had needed only a sliver of VRAM for its KV cache. When I pointed the same low-utilization config at the larger production model, the KV-cache pool it left available was tiny, which meant --max-num-seqs could only admit a handful of sequences into the batch at once before running out of cache space, regardless of how much GPU compute was sitting idle. The server wasn't broken. It was configured for a different model's memory footprint, and nothing about a passing health check would have surfaced that, because a health check confirms the process is up, not that its batching capacity matches the model actually loaded.

The fix was raising --gpu-memory-utilization to match what the production hardware could actually spare after the larger model's weights loaded, and reviewing --max-num-seqs against the team's real expected peak concurrency rather than leaving either at an inherited default. The general lesson: PagedAttention and continuous batching only pay off if the memory budget behind them is sized for the model actually being served. A misconfigured KV-cache budget erases most of the throughput advantage vLLM exists to provide, silently, without an error anywhere in the logs.

Comparison and Tradeoffs: Ollama/llama.cpp vs. vLLM vs. SGLang vs. TensorRT-LLM

Independent 2026 benchmarking gives a consistent, if scattered, picture across hardware and model sizes. On llama.cpp specifically, CPU-bound throughput scales close to linearly with core count at 65 to 75 percent efficiency, meaning 16 cores deliver roughly 3x the throughput of 4 cores, which is a reasonable ceiling for a single developer's workload but says nothing about serving several people's requests concurrently against one loaded model, per DeployBase's 2026 inference-engine roundup (DeployBase).

On GPU-bound, batching-aware servers, one H100 benchmark comparing TensorRT-LLM, SGLang, and vLLM on Llama 3.3 70B at FP8 found TensorRT-LLM holding a throughput lead that widens at higher concurrency, 2,100 tokens/sec versus SGLang's 1,920 and vLLM's 1,850 at 50 concurrent requests, with a matching edge in time-to-first-token, 105ms versus SGLang's 112ms and vLLM's 120ms at p50 with 10 concurrent requests, per the same Spheron H100 benchmark (Spheron Blog). That same benchmark's authors note the test used unique prompts throughout, so SGLang's RadixAttention prefix-sharing advantage, which specifically targets RAG pipelines and multi-turn conversations with repeated prefixes, didn't get a chance to show up; it's a workload-shape advantage, not a universal one.

Server	Concurrency model	Best fit	Real cost
Ollama / llama.cpp	Parallel slots, fixed-size shared cache, OLLAMA_NUM_PARALLEL/--parallel defaults low	Solo developer, laptop, low-concurrency internal tool	Memory multiplies with parallelism; doesn't batch efficiently at real multi-user load
vLLM	Continuous batching, PagedAttention (block-based dynamic memory)	Small team to mid-scale production API, general workloads	Requires correctly sized --gpu-memory-utilization/--max-num-seqs for the actual model in use
SGLang	Continuous batching, RadixAttention (shared-prefix caching)	RAG pipelines, multi-turn chat, repeated-prefix workloads	Advantage is workload-shape dependent, not universal
TensorRT-LLM	In-flight batching, hardware-compiled engine	Maximum throughput on NVIDIA hardware, sustained high load	10-90 minute (commonly 25-45 minute) per-model compile step, NVIDIA-only

flowchart TD Start[How many concurrent users/requests?] --> T1{Just me, one at a time} T1 -->|Yes| Ollama[Ollama / llama.cpp is fine] T1 -->|No, more than one| T2{Workload has repeated prefixes? RAG, multi-turn chat} T2 -->|Yes, and NVIDIA GPU| SGLang[SGLang for the prefix-sharing advantage] T2 -->|No, general purpose| T3{Is compile time and NVIDIA-only tooling acceptable for max throughput?} T3 -->|Yes, sustained high load justifies it| TRT[TensorRT-LLM + Triton] T3 -->|No, want flexibility, no compile step| vLLM[vLLM]

Hardware portability is the variable the table above doesn't fully capture. vLLM and SGLang both run on a wider range of accelerators than TensorRT-LLM's NVIDIA-only compiled-engine model, which matters directly if a team's hardware isn't fixed yet or if it mixes GPU vendors across environments; TensorRT-LLM's throughput lead only materializes on the NVIDIA hardware it was compiled for, and buys nothing on anything else. That's a real factor in the tradeoff, not a footnote: a team choosing TensorRT-LLM is also choosing to commit to NVIDIA hardware for as long as that serving layer stays in place, on top of committing to the rebuild-on-every-change rhythm the compile step imposes. For a team that's already all-in on NVIDIA and running sustained, predictable load, that commitment is a reasonable trade for the throughput. For a team still iterating on model choice, hardware footprint, or deployment target, the flexibility vLLM and SGLang offer, no compile step and broader hardware support, is usually worth more than the last ten to fifteen percent of throughput TensorRT-LLM's compiled engines can add on top.

Production Considerations: What Actually Changes When You Move Off Ollama

Moving from Ollama to vLLM, SGLang, or TensorRT-LLM changes more than the serving library. Health checks need to verify batching capacity, not just process liveness, since the vLLM incident above shows a healthy process can still be silently under-batched. Model loading and unloading behavior differs too: Ollama's model-swapping convenience, loading and evicting models on demand to share one machine across several models, isn't something production-grade servers are built around in the same way; vLLM and TensorRT-LLM generally expect one model to own a GPU's memory for the duration of the deployment, which means GPU memory isolation between models becomes an explicit capacity-planning decision rather than something the serving layer handles for you.

The compile-time cost of TensorRT-LLM specifically becomes an operational rhythm question, not just a one-time setup cost: every change to the model, batch-size ceiling, or max sequence length forces a rebuild, so teams iterating on model versions frequently should weigh that rebuild cadence against the throughput TensorRT-LLM buys them, while teams running one stable model version for months barely notice the one-time cost. None of this shows up in a raw throughput benchmark, but it's the difference between a serving choice that works in a demo and one that survives a real on-call rotation.

Monitoring has to change shape too, not just add a new dashboard panel. A single-user llama.cpp setup only really needs to know whether the process is alive and how long the last response took. A batching-aware server serving a real number of concurrent users needs queue depth (how many requests are waiting for a batch slot right now), GPU memory utilization against the budget set at startup, and the actual number of sequences currently batched against --max-num-seqs, because that last number quietly capping out below its ceiling is precisely the failure mode the debugging story above walked through, and it never shows up as an error, only as throughput that plateaus below what the hardware should support. Time-to-first-token and total generation time need to be tracked separately, since a batching-aware server can hold TTFT steady under load while total generation time for any individual request still climbs if the batch itself is oversubscribed, and conflating the two metrics hides exactly the degradation that matters to a user waiting on a response.

Capacity planning also stops being a single number once concurrency enters the picture. Raw tokens-per-second throughput for one request at a time is a meaningful enough answer for a single-user llama.cpp setup. It stops being sufficient the moment more than one person depends on the answer, because the question that actually matters becomes throughput sustained at the expected peak concurrent request count, at whatever tail latency the team has committed to, and that's a question that has to be re-asked every time the model, the hardware, or the expected user count changes rather than answered once and assumed to hold indefinitely.

Conclusion

The instinct to reach for Ollama on every self-hosting project is a good one for exactly as long as there's one user. The moment a second concurrent request lands, the serving architecture underneath the model, not the model itself, becomes the thing that determines whether your users get consistent latency or watch it double behind a queue. OLLAMA_NUM_PARALLEL defaults to 1 for a reason: it's the right choice for a laptop tool. Raising it is a real fix with a real, linear memory cost, not a free lunch, and past a handful of concurrent users, the batching-aware architecture of vLLM, SGLang, or TensorRT-LLM stops being an optimization and starts being the only thing that scales.

Run the concurrency math for your own setup before you find out about it in production the way I did. Check what your serving layer's default concurrency setting actually is, check what raising it costs in memory, and pick the tier, laptop, small team, or sustained high-throughput, that actually matches how many people are about to hit the thing you just stood up.

Companion repo. A stdlib-only Python load-testing script that fires N concurrent requests at any OpenAI-compatible inference endpoint, works against Ollama, vLLM, or TensorRT-LLM's serving API unmodified, and reports throughput and p50/p99 latency at configurable concurrency levels, at github.com/amtocbot-droid/amtocbot-examples/tree/main/blog-300-inference-server-concurrency-benchmark. Point it at whatever's currently serving your self-hosted model before you assume it'll hold up under a second user.

---

Get the next one

I send one short email a week: one production bug, debugged, plus the companion code for each deep-dive. No spam, unsubscribe anytime.

👉 Subscribe (free)

Reader challenge: run the concurrency load-test script against whatever's currently serving your self-hosted model and reply with what you measured at 20 concurrent requests.

Sources

• Ollama FAQ: OLLAMA_NUM_PARALLEL, OLLAMA_MAX_QUEUE, and context/memory scaling
• vLLM documentation: Optimization and Tuning (gpu-memory-utilization, max-num-seqs, continuous batching)
• llama.cpp GitHub discussion #4130: Parallelization/Batching Explanation
• NVIDIA TensorRT-LLM overview: in-flight batching and paged KV cache
• Spheron Blog, vLLM vs TensorRT-LLM vs SGLang: H100 Benchmarks (2026)
• DeployBase, Best LLM Inference Engines 2026: vLLM vs SGLang vs TGI vs llama.cpp
• LeetLLM, vLLM vs SGLang vs TensorRT-LLM vs Ollama: Choosing an Inference Engine in 2026

About the Author

Toc Am

Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.

LinkedIn X / Twitter

Published: 2026-07-19 · Written with AI assistance, reviewed by Toc Am.

Get These In Your Inbox

Weekly deep-dives on AI engineering, no fluff. Join the newsletter →

Subscribe (free)

Or grab the book ($39, ~100 pages) · Buy me a coffee

☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter

LLM Serving in Production: vLLM, Triton, and the Token Throughput Wars

Introduction

There's a large gap between running a language model and serving a language model. Running it means calling an inference API and getting a response. Serving it means operating a system that handles thousands of concurrent requests, manages GPU memory efficiently, routes traffic intelligently, meets latency SLAs, and delivers economically viable cost per token — at scale, reliably, continuously.

Most teams start in the first category. As usage grows, they eventually encounter the economics and operational complexity of the second. The transition is more substantial than it appears.

This post is for engineers who've moved beyond prototype AI applications and need to understand the infrastructure layer: how LLM serving frameworks work, what the key metrics are, how the major options compare, and how to design a serving architecture that doesn't become the bottleneck or the budget line item that kills the project.

The Serving Problem

Language model inference has unusual resource characteristics compared to traditional application serving.

Memory dominates compute for large models. A 70B parameter model in fp16 requires approximately 140GB of GPU memory just to store the weights — before any batching or KV cache. Even a 7B model requires ~14GB. GPU memory is expensive and limited: an H100 80GB card costs $30,000–$40,000. Memory efficiency is directly correlated with cost efficiency.

The KV cache is the performance bottleneck. During inference, the model computes key-value attention tensors for each token in the context. These need to be retained across generation steps (so you don't recompute from scratch for every new token). The KV cache grows linearly with sequence length and batch size. On a long context with a large batch, the KV cache can consume more memory than the model weights themselves. Managing KV cache efficiently is the central challenge of production LLM serving.

Throughput and latency are in tension. Batching multiple requests together dramatically improves GPU utilization and throughput. But it adds latency — a request that arrives when others are already being processed waits for the batch to be compiled. The optimal batching strategy depends on your workload and SLAs.

Generation is autoregressive — one token at a time. Standard inference generates one token per forward pass. A 500-token response requires 500 forward passes. This is fundamentally different from classification or embedding models that produce output in a single pass. It means time-to-first-token (TTFT) and inter-token latency (ITL) are the metrics users experience, not just total response time.

The Key Metrics

Before comparing serving frameworks, understand the metrics that matter:

Time to First Token (TTFT): how long after the request arrives before the first token is generated. Determined primarily by the prefill phase (processing the input). Critical for interactive use cases where users are watching text stream in.

Inter-Token Latency (ITL): time between each successive token during generation. Determines the "typing speed" experience. Should be consistent — variable ITL feels choppy to users.

Throughput (tokens/second): total tokens generated per second across all concurrent requests. The production efficiency metric — higher throughput means lower cost per token.

Request latency (P95/P99): end-to-end time for complete requests at the 95th and 99th percentile. The SLA metric. P95 matters more than mean for real user experience.

GPU memory utilization: what fraction of available GPU memory is actively used. Low utilization means you're paying for capacity you're not using. High utilization reduces room for traffic spikes.

vLLM: The Efficient Memory Management Baseline

vLLM, released by the Berkeley Sky Computing Lab in 2023, became the de facto standard for open-source LLM serving by solving the KV cache management problem elegantly.

PagedAttention: vLLM's core innovation, borrowed from operating system paging. Instead of pre-allocating a contiguous block of GPU memory for each request's KV cache (which leads to massive internal fragmentation — you reserve space for the maximum sequence length even when requests are short), PagedAttention allocates memory in small, fixed-size pages and maps non-contiguous physical memory to logical attention positions.

The result: near-zero internal fragmentation. A server that previously wasted 60-80% of KV cache memory on fragmentation now uses that memory productively. The practical effect is 2–4× more requests served concurrently on the same hardware.

Continuous batching: traditional batching adds all new requests to the same batch and waits for all to complete before accepting new requests. Continuous batching inserts new requests into the batch dynamically — when a request in the current batch finishes, a new request immediately takes its slot. This dramatically improves throughput under variable request lengths.

# Running vLLM with OpenAI-compatible API
# pip install vllm

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3-8B-Instruct",
    tensor_parallel_size=1,      # Number of GPUs for tensor parallelism
    gpu_memory_utilization=0.90, # Reserve 10% for overhead
    max_model_len=8192,          # Maximum context length
    dtype="bfloat16",            # Compute dtype (bf16 is usually fastest on H100/A100)
)

sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.95,
    max_tokens=512,
)

prompts = [
    "Explain the difference between a mutex and a semaphore.",
    "What are the SOLID principles in object-oriented design?",
    "How does a transformer's attention mechanism work?",
]

outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    print(f"Prompt: {output.prompt[:50]}...")
    print(f"Generated: {output.outputs[0].text}")

Production deployment: vLLM ships an OpenAI-compatible REST API server:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-8B-Instruct \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --port 8000

This makes it a drop-in replacement for the OpenAI API in applications that use the OpenAI SDK.

Speculative Decoding: The 2026 Throughput Multiplier

Speculative decoding is the most impactful throughput improvement in production serving over the past two years. The technique exploits the asymmetry between generation speed (slow — sequential, one token at a time) and verification speed (fast — can verify many tokens in a single forward pass).

A small draft model (typically 1B-3B parameters) quickly generates a speculative sequence of several tokens. The large target model then verifies the entire speculative sequence in a single parallel forward pass. Tokens that match are accepted. The first rejected token and all subsequent tokens are discarded, and the target model generates the correct continuation from the rejection point.

In practice, speculative decoding achieves 2–4× throughput improvement for structured outputs (code, JSON, formulaic text) where the draft model's guesses are frequently correct. For highly creative or open-ended generation, the improvement is smaller (the draft model's guesses are rejected more often).

# vLLM with speculative decoding
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3-70B-Instruct",     # Target model
    speculative_model="meta-llama/Llama-3-8B-Instruct",  # Draft model
    num_speculative_tokens=5,    # How many tokens to speculate ahead
    tensor_parallel_size=4,
)

sequenceDiagram participant C as Client participant D as Draft Model (fast) participant T as Target Model (slow) C->>T: Request: "Generate code..." loop Speculative decoding T->>D: Current context D-->>T: Speculative tokens [a, b, c, d, e] T->>T: Verify all 5 tokens in ONE forward pass alt All accepted T-->>C: Stream tokens a, b, c, d, e else Partial accept (a, b accepted, c rejected) T-->>C: Stream tokens a, b T->>T: Generate correct token at position c T-->>C: Stream corrected token end end

Triton Inference Server: The Enterprise Option

NVIDIA's Triton Inference Server takes a different approach from vLLM. Where vLLM is focused specifically on LLM serving efficiency, Triton is a general-purpose model serving platform that supports PyTorch, TensorFlow, ONNX, TensorRT, and custom backends — including LLMs through its TensorRT-LLM backend.

TensorRT-LLM: NVIDIA's LLM inference library, tightly optimized for NVIDIA hardware. Uses quantization (FP8, INT8, INT4), kernel fusion, and hardware-specific optimizations that extract maximum performance from H100 and A100 GPUs. In benchmarks, TensorRT-LLM with TensorRT optimization typically achieves 20-40% higher throughput than vLLM on NVIDIA hardware, at the cost of significantly higher operational complexity.

When to choose Triton over vLLM:
- You need to serve multiple model types (not just LLMs) from a single serving platform
- You're on NVIDIA hardware and need to extract maximum performance
- You have the operational capacity to handle more complex deployment pipelines
- You're doing serious quantization (INT4, INT8, FP8) and need the hardware-level optimizations

When vLLM is the right choice:
- Team is optimizing for deployment speed and operational simplicity
- Hardware-agnostic deployment is required (vLLM supports AMD ROCm, some Intel XPU)
- The memory efficiency improvements (PagedAttention) are more valuable than compute efficiency

Serving Architecture Patterns

Single-Server Deployment

For moderate traffic, a single high-memory server with multiple GPUs handles most use cases. An H100 SXM5 node with 8 GPUs (640GB total memory) can serve 70B models with excellent throughput. Tensor parallelism distributes the model across GPUs within the server; no network communication between servers required.

Multi-Server Deployment with Load Balancer

For high-traffic applications, deploy multiple inference servers behind a load balancer. Each server runs the full model (or uses tensor parallelism across its local GPUs). The load balancer distributes requests using least-connections or round-robin routing.

Key consideration: sticky sessions. Requests with the same prefix (system prompt + conversation history) benefit from prefix caching if routed to the same server. A load balancer that's aware of prefixes can route related requests to the same server, improving cache hit rates.

# Example: prefix-aware load balancing
import hashlib
from typing import Optional

def get_server_for_request(
    conversation_id: Optional[str],
    system_prompt_hash: str,
    servers: list[str]
) -> str:
    """
    Route requests with the same conversation to the same server
    for better prefix cache utilization.
    """
    routing_key = conversation_id or system_prompt_hash
    server_index = int(hashlib.md5(routing_key.encode()).hexdigest(), 16) % len(servers)
    return servers[server_index]

Disaggregated Prefill and Decode

An emerging architecture for optimizing both TTFT and throughput simultaneously: separate the prefill phase (compute-intensive, processes the entire input context in parallel) from the decode phase (memory-bandwidth intensive, generates tokens sequentially).

Prefill servers use compute-optimized instances (H100 SXM). Decode servers use memory-optimized instances. Requests hit a prefill server to process the input, then transfer their KV cache to a decode server for generation. This allows each server type to be sized independently and allows more aggressive batching of the prefill phase.

This architecture is operationally complex but can achieve 30-50% cost reduction at high scale. Companies like Groq are exploring even more extreme disaggregation with dedicated hardware.

Quantization: Trading Precision for Efficiency

Quantization reduces model weight precision from 16-bit floating point (fp16/bf16) to lower precision (INT8, INT4, FP8). The model weights consume less memory, more of the model fits in GPU memory, and memory bandwidth requirements drop — resulting in higher throughput at the cost of small quality degradation.

Format	Memory (70B model)	Quality	Throughput
fp16	~140GB	Baseline	Baseline
INT8	~70GB	≈98% of fp16	~1.5×
INT4 (GPTQ/AWQ)	~35GB	≈95-97% of fp16	~2.5×
INT4 + fp16 KV cache	~35GB weights + variable KV	≈95-97%	~2.5×
FP8 (H100 native)	~70GB	≈99% of fp16	~1.8× on H100

For production workloads, INT8 quantization is typically the right default — minimal quality degradation with meaningful efficiency gains. INT4 is appropriate where cost matters more than maximum quality (summarization, classification, code completion where re-runs are cheap).

Cost Analysis: Build vs. Buy

The classic infrastructure decision. For LLM serving, the break-even calculation is relatively clear:

API providers (Anthropic, OpenAI, Google): no infrastructure management, no GPU capital expenditure, pay per token. Makes sense for low-to-moderate volume, for accessing frontier models you can't run yourself, and when your team doesn't have ML infrastructure expertise.

Self-hosted open-source models: high upfront investment (GPU hardware or cloud GPU reservations), ongoing operational cost (engineering time, monitoring). Economically viable above approximately 1M tokens per day per model, where the cost savings over API providers justify the operational complexity.

Break-even calculation (rough):
API cost for 1M tokens/day (claude-haiku-4-5): ~$0.25-$0.80/day
Self-hosted H100 instance cost: ~$30-35/hr → ~$720-840/day
Break-even: ~900M-3.4B tokens/day per GPU

Conclusion: self-hosting is economically compelling only at very high volume,
or for use cases requiring data privacy or very low latency.

Production Considerations

Health checks and graceful degradation: model loading takes minutes. Implement liveness probes that distinguish "model loading" from "model unhealthy." Under load, implement circuit breakers that reject new requests rather than queuing indefinitely when latency exceeds thresholds.

Prompt caching: for applications with shared prefixes (system prompts, few-shot examples, document contexts that don't change), enable prefix caching. vLLM supports automatic prefix caching. The KV cache for repeated prefixes is computed once and reused, dramatically reducing TTFT and compute cost for the decode phase.

Monitoring and alerting: beyond standard application metrics, monitor model-specific signals: token generation rate, KV cache utilization, queue depth and wait time, output quality scores from automated evaluation. Set alerts on KV cache utilization above 90% (performance degrades sharply as cache fills).

Conclusion

LLM serving in production is infrastructure that requires deliberate design. PagedAttention, continuous batching, speculative decoding, and quantization are not optimizations to add later — they're the foundation of economically viable serving at scale.

For most teams: start with the OpenAI-compatible vLLM API server with PagedAttention and continuous batching enabled. Add prefix caching immediately if your workload has repeated prefixes. Evaluate speculative decoding once your traffic patterns are established. Consider quantization when memory pressure or cost become constraints.

The teams that get this right are those that treat model serving as infrastructure — not as an API call that will scale automatically.

---

Sources & References

1. vLLM Team — "Efficient Memory Management for LLM Serving with PagedAttention"
2. NVIDIA — "TensorRT-LLM"
3. DeepMind — "Speculative Decoding"
4. Anyscale — "How continuous batching enables 23× throughput"
5. vLLM Documentation

About the Author

Toc Am

Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.

LinkedIn X / Twitter

Published: 2026-05-01 · Written with AI assistance, reviewed by Toc Am.

Get These In Your Inbox

Weekly deep-dives on AI engineering, no fluff. Join the newsletter →

Subscribe (free)

Or grab the book ($39, ~100 pages) · Buy me a coffee

☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter