I was running a RAG pipeline over a corpus that mixed technical runbooks, policy documents, and a real-time status feed. A user asked "is the payments API down right now?" The dense retriever returned a runbook from over a year ago about a different outage. The answer existed — in the status feed index — but the query never reached it.
The problem was not retrieval quality. It was that the query went to the wrong index entirely.
Query routing is the step before retrieval: classify the incoming query and dispatch it to the retriever or index best suited to answer it. A lookup query (e.g. what is the timeout for the payments gateway?) belongs in the static documentation index. A status query (e.g. is X down right now?) belongs in the live-data index. A synthesis query (e.g. explain how the payment flow works end to end) belongs in the dense semantic index with a larger K.
This post covers how to implement query routing in a self-hosted pipeline, what classifiers work at production latency, and how to measure whether routing is improving outcomes.
Why a Single Retriever Path Is Not Enough
The retrieval strategies in this series (chunking, metadata filtering, hybrid search, reranking) all assume the query is going to the right place. They improve recall and precision within a retriever, but none of them reroute a query that is structurally mismatched to the index it lands in.
Three mismatches that routing solves:
Index mismatch: Your pipeline has multiple indexes (docs, tickets, status feed). A semantic query over the wrong index returns plausible-sounding but wrong results. The user asked about current state; you returned historical documentation.
Retrieval strategy mismatch: A lookup query (exact identifier) should use sparse/BM25 retrieval. A conceptual query should use dense retrieval. Sending a lookup query to a dense retriever consistently underperforms even when the right index is used.
Complexity mismatch: A simple factual query needs K=3 chunks. A synthesis query ("explain the entire onboarding flow") may need K=20 across multiple sub-queries. Treating all queries identically wastes tokens on simple queries and under-retrieves on complex ones.
The Routing Architecture
Query routing sits between query intake and retrieval:
user query
│
▼
[query classifier]
│
├──► static docs index (dense, K=10)
├──► status/live index (direct lookup)
├──► ticket/incident index (hybrid, K=5)
└──► fallback: all indexes (merge results)
The classifier produces a route label (and optionally a confidence score). The router dispatches to the corresponding retrieval path. Results are returned through a common interface.
Implementing the Classifier
For latency-sensitive pipelines, a lightweight classifier is better than a large LLM call. Three options in increasing complexity:
Option 1: Rule-Based Routing
Fast, deterministic, no model required. Works well when query types are syntactically distinguishable.
import re
def classify_query_rules(query: str) -> str:
query_lower = query.lower()
# Status/real-time queries
status_patterns = [
r"\b(is|are)\b.*(down|up|running|available|broken)",
r"\bcurrent(ly)?\b",
r"\bright now\b",
r"\bstatus\b",
r"\boutage\b",
]
for pattern in status_patterns:
if re.search(pattern, query_lower):
return "status"
# Exact lookup queries
lookup_patterns = [
r"\bwhat is the\b",
r"\bwhere is\b",
r"\bwhat (are|were) the\b",
r"\bERR[_-]?\d+\b", # error codes
r"\b[A-Z]{2,}_[A-Z_]{2,}\b", # ALL_CAPS_IDENTIFIERS
]
for pattern in lookup_patterns:
if re.search(pattern, query_lower):
return "lookup"
# Synthesis/explanation queries
synthesis_patterns = [
r"\bhow does\b",
r"\bexplain\b",
r"\bwalk me through\b",
r"\bend.to.end\b",
r"\boverview\b",
]
for pattern in synthesis_patterns:
if re.search(pattern, query_lower):
return "synthesis"
return "semantic" # default
Option 2: Embedding-Based Routing
Embed the query and compare cosine similarity to prototype embeddings for each route. Requires a few representative examples per route but no LLM call at inference time.
import ollama
import numpy as np
ROUTE_PROTOTYPES = {
"status": [
"is the payments API down right now",
"what is the current status of the data pipeline",
"are any services experiencing outages",
],
"lookup": [
"what is the timeout value for the gateway",
"where is the rate limit configuration",
"what does error code ERR_4023 mean",
],
"synthesis": [
"explain how the payment flow works end to end",
"walk me through the onboarding process",
"how does the authentication system work",
],
}
def embed(text: str) -> np.ndarray:
vec = ollama.embeddings(model="nomic-embed-text", prompt=text)["embedding"]
return np.array(vec)
def build_prototype_embeddings() -> dict[str, np.ndarray]:
prototypes = {}
for route, examples in ROUTE_PROTOTYPES.items():
vecs = [embed(ex) for ex in examples]
prototypes[route] = np.mean(vecs, axis=0)
return prototypes
PROTOTYPES = build_prototype_embeddings()
def classify_query_embedding(query: str) -> tuple[str, float]:
query_vec = embed(query)
scores = {}
for route, proto_vec in PROTOTYPES.items():
cosine = np.dot(query_vec, proto_vec) / (
np.linalg.norm(query_vec) * np.linalg.norm(proto_vec)
)
scores[route] = float(cosine)
best_route = max(scores, key=scores.get)
return best_route, scores[best_route]
Option 3: LLM-Based Routing
Highest accuracy, highest latency. Use only if the classification decision materially affects answer quality and you can afford the extra call.
import ollama
ROUTING_PROMPT = """Classify this query into one of these categories:
- status: asks about current state, availability, or live system health
- lookup: asks for a specific fact, value, or error code definition
- synthesis: asks for explanation, overview, or multi-step process
- semantic: general question best answered by semantic search
Query: {query}
Respond with exactly one word: status, lookup, synthesis, or semantic."""
def classify_query_llm(query: str) -> str:
response = ollama.generate(
model="llama3.2:3b",
prompt=ROUTING_PROMPT.format(query=query),
options={"temperature": 0}
)
label = response["response"].strip().lower()
if label not in {"status", "lookup", "synthesis", "semantic"}:
return "semantic"
return label
Using a small local model like llama3.2:3b keeps classification latency acceptable for production use.
The Router
from typing import Any
def route_and_retrieve(
query: str,
classifier: str = "rules", # "rules", "embedding", or "llm"
k: int = 10
) -> list[dict]:
# Classify
if classifier == "rules":
route = classify_query_rules(query)
confidence = 1.0
elif classifier == "embedding":
route, confidence = classify_query_embedding(query)
else:
route = classify_query_llm(query)
confidence = 1.0
# Low-confidence fallback
if confidence < 0.6:
route = "semantic"
# Dispatch
if route == "status":
return retrieve_from_status_index(query, k=k)
elif route == "lookup":
return retrieve_hybrid(query, k=k, dense_weight=0.3, sparse_weight=0.7)
elif route == "synthesis":
return retrieve_semantic(query, k=min(k * 2, 20))
else:
return retrieve_semantic(query, k=k)
Measuring Routing Quality
Route the query to the wrong index and even a perfect retriever returns garbage. Measurement should be at two levels:
Classification accuracy: Label a sample of real queries by correct route. Measure classifier accuracy on that sample. Target: above 85% on the query distribution you actually receive.
End-to-end recall by route: Run your existing eval set with and without routing. Measure Recall@10 for each query type separately. Routing should improve recall on mismatched query types without degrading the default case.
def eval_routing(eval_set: list[dict]) -> dict:
results = {"with_routing": {}, "without_routing": {}}
for item in eval_set:
query = item["query"]
relevant_ids = set(item["relevant_doc_ids"])
query_type = item["type"] # ground-truth label
# With routing
routed = route_and_retrieve(query)
retrieved_ids = {r["id"] for r in routed[:10]}
hit = len(relevant_ids & retrieved_ids) > 0
results["with_routing"].setdefault(query_type, []).append(hit)
# Without routing (always semantic)
baseline = retrieve_semantic(query, k=10)
baseline_ids = {r["id"] for r in baseline}
baseline_hit = len(relevant_ids & baseline_ids) > 0
results["without_routing"].setdefault(query_type, []).append(baseline_hit)
return {
route: {
qtype: sum(hits) / len(hits)
for qtype, hits in by_type.items()
}
for route, by_type in results.items()
}
When Routing Is Worth the Complexity
Routing adds a classification step, more code paths, and more indexes to maintain. It is worth it when:
- You have structurally different query types that perform differently across retrieval strategies
- You have multiple indexes covering different data sources (docs, live data, tickets)
- You have already tuned chunking, metadata filtering, and hybrid search and want the next increment
If your corpus is homogeneous and your queries are mostly semantic, routing adds overhead without meaningful gain. Measure first: compare query types in your logs before building a router.
Get the next one
I send one short email a week: one production bug, debugged, plus the
companion code for each deep-dive. No spam, unsubscribe anytime.
Reader challenge: are you routing queries in your RAG pipeline? What classifier approach worked for your query distribution? Reply with what you tried.
Sources
- Qdrant collection routing patterns: https://qdrant.tech/documentation/guides/multiple-partitions/
- nomic-embed-text on Ollama: https://ollama.com/library/nomic-embed-text
- llama3.2 on Ollama: https://ollama.com/library/llama3.2
About the Author
Toc Am
Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.
Published: 2026-07-27 · Written with AI assistance, reviewed by Toc Am.
Get These In Your Inbox
Weekly deep-dives on AI engineering, no fluff. Join the newsletter →
Or grab the book ($39, ~100 pages) · Buy me a coffee
☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter
