Model Routing And Failover Patterns

Last March, our content pipeline ground to a halt for 47 minutes. The primary LLM provider we depended on for automated blog drafts hit a regional outage, and every request returned a 503. We had no fallback. Forty-seven minutes doesn't sound like much until you realize our queue was backing up at 300 jobs per minute, and the retry storm that followed made recovery even slower. That day, we rebuilt our inference layer around model routing and failover — and we haven't had a single pipeline-wide outage since.

The Problem with Single-Model Dependencies

Most teams start with one model. You pick a provider, wire it into your application, and ship. It works — until it doesn't. Providers experience outages, throttle your requests, deprecate models, or raise prices overnight. When your entire pipeline funnels through a single endpoint, you've built a system where one HTTP 503 can take down your whole product.

The fix isn't just "add a second API key." You need deliberate patterns for routing requests across models and failing over gracefully when something goes wrong. These are two related but distinct problems: routing decides which model handles a given request, and failover decides what happens when that model can't.

Routing: Choosing the Right Model

Think of routing like a hospital triage desk. Not every patient needs the trauma surgeon — a sprained wrist can be handled by urgent care, and routing it to the ER wastes expensive resources. Similarly, not every LLM call needs a frontier model. A simple text classification or format conversion can run on a smaller, cheaper, faster model. Complex reasoning, code generation, or long-form synthesis may need the heavyweights.

A practical routing strategy considers three dimensions:

• **Cost**: Frontier models cost 10–30× more per token than compact models. If 70% of your traffic is simple tasks, routing them to cheaper models can cut your bill dramatically.
• **Latency**: Smaller models respond in 200–500ms; frontier models can take 2–5 seconds. For real-time interfaces, this matters.
• **Capability**: Some models excel at code, others at multilingual content. Routing by task type improves quality.

The simplest effective approach is rule-based routing: classify the request by task type or token length, then map each category to a model. More sophisticated setups use a lightweight classifier model to predict which backend should handle the request, but rule-based routing covers 80% of cases with far less complexity.

Failover: Surviving When Models Fail

Failover is your safety net. When a model endpoint returns errors or times out, failover ensures the request still gets served — either by retrying, falling back to another model, or degrading gracefully.

The key patterns are:

1. Retry with exponential backoff — transient errors (429, 503) often resolve in seconds. Retry up to 3 times with increasing delays.

2. Circuit breaker — if a provider fails repeatedly, stop sending traffic temporarily. This prevents retry storms and lets the provider recover.

3. Model fallback chain — define an ordered list of models. If the primary fails after retries, try the next one.

4. Health checks — periodically ping endpoints and route around unhealthy ones proactively, not just reactively.

Putting It Together: A Minimal Router with Failover

Here's a self-contained router using only Python's standard library. It supports rule-based routing, exponential backoff retries, a simple circuit breaker, and a fallback chain:

import json
import time
import urllib.request
import urllib.error
from dataclasses import dataclass
from typing import Optional

@dataclass
class ModelEndpoint:
    name: str
    url: str
    api_key: str
    max_tokens: int
    cost_per_1k: float  # USD per 1K output tokens
    failure_count: int = 0
    circuit_open_until: float = 0.0

@dataclass
class Router:
    endpoints: dict  # task_type -> list[ModelEndpoint] (ordered fallback chain)
    max_retries: int = 3
    base_backoff: float = 0.5
    circuit_threshold: int = 5
    circuit_cooldown: float = 60.0

    def _is_healthy(self, ep: ModelEndpoint) -> bool:
        """Check if the circuit breaker allows traffic to this endpoint."""
        return ep.circuit_open_until <= time.time()

    def _record_failure(self, ep: ModelEndpoint):
        ep.failure_count += 1
        if ep.failure_count >= self.circuit_threshold:
            ep.circuit_open_until = time.time() + self.circuit_cooldown
            print(f"[CIRCUIT] Open for {ep.name} — cooling down {self.circuit_cooldown}s")

    def _record_success(self, ep: ModelEndpoint):
        ep.failure_count = 0
        ep.circuit_open_until = 0.0

    def _call_endpoint(self, ep: ModelEndpoint, prompt: str) -> Optional[str]:
        """Make a single HTTP call to a model endpoint. Returns text or None."""
        payload = json.dumps({"prompt": prompt, "max_tokens": ep.max_tokens}).encode()
        req = urllib.request.Request(
            ep.url, data=payload,
            headers={"Content-Type": "application/json",
                     "Authorization": f"Bearer {ep.api_key}"},
            method="POST"
        )
        try:
            with urllib.request.urlopen(req, timeout=30) as resp:
                return json.loads(resp.read().decode()).get("text", "")
        except (urllib.error.URLError, urllib.error.HTTPError, TimeoutError) as e:
            print(f"[ERROR] {ep.name}: {e}")
            return None

    def route(self, task_type: str, prompt: str) -> Optional[str]:
        """Route a request through the fallback chain for its task type."""
        chain = self.endpoints.get(task_type, [])
        if not chain:
            print(f"[ROUTE] No endpoints for task: {task_type}")
            return None

        for ep in chain:
            if not self._is_healthy(ep):
                print(f"[SKIP] {ep.name} — circuit open")
                continue

            for attempt in range(self.max_retries):
                result = self._call_endpoint(ep, prompt)
                if result is not None:
                    self._record_success(ep)
                    print(f"[OK] Served by {ep.name} (attempt {attempt + 1})")
                    return result
                self._record_failure(ep)
                if not self._is_healthy(ep):
                    break  # circuit just opened — move to next endpoint
                backoff = self.base_backoff * (2 ** attempt)
                print(f"[RETRY] Backing off {backoff:.1f}s")
                time.sleep(backoff)

            print(f"[FAILOVER] {ep.name} exhausted — trying next in chain")

        print(f"[EXHAUSTED] All endpoints failed for: {task_type}")
        return None

Usage looks like this:

router = Router(endpoints={
    "simple": [
        ModelEndpoint("compact-a", "https://api.provider-a.com/v1/generate",
                      "key-a", max_tokens=256, cost_per_1k=0.15),
        ModelEndpoint("compact-b", "https://api.provider-b.com/v1/generate",
                      "key-b", max_tokens=256, cost_per_1k=0.20),
    ],
    "complex": [
        ModelEndpoint("frontier-a", "https://api.provider-a.com/v1/generate",
                      "key-a", max_tokens=4096, cost_per_1k=5.00),
        ModelEndpoint("frontier-b", "https://api.provider-b.com/v1/generate",
                      "key-b", max_tokens=4096, cost_per_1k=4.50),
    ],
})

# Route by task complexity — simple tasks hit cheaper models first
result = router.route("simple", "Summarize this paragraph: ...")

The router tries the first endpoint, retries transient failures with backoff, opens a circuit breaker after repeated failures, and falls through to the next model in the chain. In production, you'd add observability — logging which model served each request, tracking p99 latency per endpoint, and alerting when circuits open frequently.

Key Takeaways

• **Never depend on a single model endpoint.** A fallback chain with at least two providers per task type is the minimum viable resilience.
• **Route by task complexity.** Sending simple tasks to frontier models wastes money and adds latency. Rule-based routing captures most of the benefit with little complexity.
• **Retry transient errors, but cap it.** Three retries with exponential backoff handles most transient failures. More than that and you're contributing to the problem.
• **Use circuit breakers to protect providers and yourself.** When an endpoint is struggling, stop hammering it. Give it time to recover.
• **Measure everything.** Track cost, latency, and success rate per model. You can't optimize what you don't measure.
• **Test your failover before you need it.** Simulate outages in staging by pointing endpoints at unreachable hosts. If your fallback doesn't work in staging, it won't work in production.

Wrapping Up

Model routing and failover aren't optional architecture for production LLM systems — they're the difference between a pipeline that degrades gracefully and one that falls off a cliff. The patterns above are deliberately simple: you can implement them in an afternoon, and they'll pay for themselves the first time a provider has a bad day.

For more on building resilient AI pipelines, check out our companion code and our earlier post on building content automation pipelines with LLMs. If you're evaluating AI content automation for your team, AmtocSoft's platform handles routing, failover, and observability out of the box — so you can focus on content quality, not infrastructure.

Written with AI assistance — reviewed by Toc Am