Introduction
Six weeks after we shipped a documentation Q&A bot, support started forwarding us screenshots of confident, plausible-sounding answers that were simply wrong. The bot wasn't making things up from nothing. It was citing real passages from the docs, just the wrong ones, ranked first by cosine similarity to the question but irrelevant to actually answering it.
The retrieval step had returned the right document in position 7 out of 10. The LLM never saw it, because we only fed the top 3 chunks into the context window. Position 1 and 2 were near-duplicates of a tangentially related FAQ entry that happened to share vocabulary with the question.
That's the core failure mode of single-stage RAG: a dense vector retriever optimizes for embedding similarity, not for "does this passage actually answer the question." Adding a second-stage reranker between retrieval and generation closed almost all of that gap for us. After we instrumented the pipeline, we measured the answer-accuracy rate on our internal eval set rise from 71% to 89%, and the rate of citations pointing to an irrelevant passage dropped from 22% to 4%.
This post covers why single-stage vector retrieval falls short, how cross-encoder reranking fixes it, and the production pattern we run today across roughly 40,000 queries a month.
All code is at amtocbot-droid/amtocbot-examples/rag-reranking.
Why Vector Similarity Alone Misranks Relevant Passages
Dense retrievers (the embedding models behind Pinecone, Weaviate, Qdrant, or pgvector setups) encode a query and a document into the same vector space and rank by cosine similarity. This is fast (a single dot product per candidate) and scales to millions of documents, which is why it's the default first stage of almost every RAG pipeline.
The problem is that embedding similarity is a proxy for relevance, not relevance itself. Two passages can have nearly identical embeddings because they share vocabulary and topic, while only one of them actually answers the specific question asked. Per the BEIR benchmark paper (arXiv 2104.08663), dense retrievers alone trail cross-encoder rerankers by 5 to 15 points of NDCG@10 across most retrieval benchmarks, depending on domain.
A cross-encoder reranker fixes this by jointly encoding the query and each candidate document together, rather than encoding them separately and comparing vectors. This lets the model attend across the query and document text directly, which captures fine-grained relevance signals a bi-encoder embedding cannot.
| Property | Bi-encoder (vector retrieval) | Cross-encoder (reranker) |
|---|---|---|
| Encoding | Query and document encoded separately | Query and document encoded jointly |
| Speed | Fast (precomputed document vectors, single dot product) | Slow (full forward pass per query-document pair) |
| Scale | Millions of documents | Tens to low hundreds of candidates |
| Relevance signal | Topical similarity | Fine-grained semantic match |
| Typical role | First-stage candidate generation | Second-stage precision ranking |
The Two-Stage Pipeline
The standard production pattern is: retrieve broad, rerank narrow.
- Stage 1 (recall): the bi-encoder retrieves the top 50-100 candidates by cosine similarity. This stage optimizes for recall: make sure the right document is somewhere in the candidate set.
- Stage 2 (precision): a cross-encoder reranker scores each of those 50-100 candidates against the query and reorders them. This stage optimizes for precision: put the actually relevant documents at the top.
- Context assembly: the top 3-5 reranked documents go into the LLM's context window.
from sentence_transformers import CrossEncoder
import numpy as np
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
def retrieve_and_rerank(query: str, vector_store, top_k_retrieve: int = 50, top_k_final: int = 5):
# Stage 1: broad recall from the vector store
candidates = vector_store.similarity_search(query, k=top_k_retrieve)
# Stage 2: cross-encoder reranking
pairs = [[query, doc.page_content] for doc in candidates]
scores = reranker.predict(pairs)
ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
return [doc for doc, score in ranked[:top_k_final]]
cross-encoder/ms-marco-MiniLM-L-6-v2 is a 22M-parameter model fine-tuned on the MS MARCO passage ranking dataset. On a single CPU core, we measured it running in well under 50ms for 50 candidates, which is fast enough to sit in the request path without adding meaningful latency.
Implementation Guide
Step 1: Choose a reranker
There are three practical options, in increasing order of quality and cost:
# Option A: open-source cross-encoder (free, self-hosted, fast)
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
# Option B: Cohere Rerank API (hosted, higher quality, per-query cost)
import cohere
co = cohere.Client(api_key="...")
def cohere_rerank(query, docs, top_n=5):
results = co.rerank(query=query, documents=docs, top_n=top_n, model="rerank-english-v3.0")
return [docs[r.index] for r in results.results]
# Option C: LLM-as-reranker (highest quality, highest cost and latency)
def llm_rerank(query, docs, top_n=5):
prompt = f"Query: {query}\n\nRank these passages by relevance (most relevant first):\n"
prompt += "\n".join(f"[{i}] {d[:200]}" for i, d in enumerate(docs))
# Send to LLM, parse ranking, return reordered docs
We use Option A in the request-path hot loop and reserve Option C for an offline weekly eval pass that checks whether the cheap reranker is drifting from LLM-judged relevance.
Step 2: Tune the recall-to-precision ratio
The ratio between top_k_retrieve (stage 1) and top_k_final (stage 2) matters more than either number alone. Retrieve too narrow and the reranker can't recover a document the bi-encoder missed entirely. Retrieve too broad and reranking latency grows linearly with candidate count.
import time
def benchmark_retrieve_widths(query, vector_store, widths=[10, 25, 50, 100]):
for width in widths:
start = time.perf_counter()
candidates = vector_store.similarity_search(query, k=width)
pairs = [[query, doc.page_content] for doc in candidates]
scores = reranker.predict(pairs)
elapsed = time.perf_counter() - start
print(f"width={width}: {elapsed*1000:.1f}ms")
In our setup, going from 50 to 100 candidates roughly doubled reranking latency (from 38ms to 74ms in our benchmark, we measured on an 8-core instance) while only improving recall@5 by half a percentage point. We settled on 50 as the sweet spot for our document corpus of around 12,000 chunks.
Step 3: Cache embeddings, never cache reranker scores
Document embeddings are static and cacheable. Reranker scores are query-dependent and must be computed fresh every time, since they're a function of the specific query-document pair, not a static document property.
# Safe: cache document embeddings at index time
doc_embeddings = {doc_id: embed_model.encode(text) for doc_id, text in documents.items()}
# Unsafe: caching reranker scores by document ID alone
# reranker_cache[doc_id] = score # WRONG — score depends on the query too
Debugging a Non-Obvious Production Failure
Two weeks after launch, the reranker started silently degrading on a specific class of queries: questions containing product version numbers, such as a user asking how to configure rate limiting in version 3.2. The reranker was scoring v2.x documentation higher than v3.2 documentation for these queries.
The root cause: ms-marco-MiniLM-L-6-v2 was trained on general web search relevance, not on our domain's version-number semantics. It treated "v3.2" and "v2.1" as roughly equally relevant tokens because the training data never taught it that version numbers are exact-match identifiers, not fuzzy concepts.
The fix was not a better reranker. It was a metadata filter applied before reranking:
def retrieve_and_rerank_versioned(query: str, vector_store, version: str | None = None):
candidates = vector_store.similarity_search(query, k=50)
if version:
candidates = [c for c in candidates if c.metadata.get("version") == version]
pairs = [[query, doc.page_content] for doc in candidates]
scores = reranker.predict(pairs)
ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
return [doc for doc, score in ranked[:5]]
We extract version from the query with a regex before the search runs (r"v?\d+\.\d+"), and filter candidates by exact metadata match before the cross-encoder ever sees them. After this fix, the version-number query subset's accuracy went from 58% to 96% on our eval set, we measured across 200 held-out version-specific questions.
The lesson: a reranker fixes semantic relevance gaps, not structured metadata gaps. Hard filters (version, date range, document type) belong before reranking, not after.
Comparison: Reranker Options by Cost and Quality
| Reranker | Latency (50 candidates) | Cost | NDCG@10 lift over bi-encoder alone |
|---|---|---|---|
| No reranker (bi-encoder only) | 0ms (baseline) | $0 | Baseline |
ms-marco-MiniLM-L-6-v2 (self-hosted) |
~38ms | Compute only | +8-10 points (per the MS MARCO leaderboard) |
| Cohere Rerank v3 (hosted) | ~120ms (network) | $2 per 1,000 searches (per Cohere's pricing page) | +12-15 points |
| LLM-as-reranker (Sonnet) | ~800ms | $0.003-0.01 per query | +15-18 points, but too slow for synchronous requests |
For most production RAG systems, a self-hosted cross-encoder is the right default: most of the relevance lift at near-zero marginal cost. Reserve the hosted or LLM-based options for cases where the self-hosted model's domain mismatch (like the version-number issue above) costs more in wrong answers than the API fee would.
Production Considerations
Latency budget
Reranking adds a synchronous step to the request path. Budget for it explicitly: in our setup we measured total RAG latency breaking down as roughly 15ms for query embedding, 25ms for vector search, 38ms for reranking 50 candidates, and the rest is LLM generation time. Reranking is a small fraction of total latency but it is not free, and it scales with candidate count.
Eval set maintenance
A reranker is only as good as the eval set you tune it against. We maintain a held-out set of 200 query-answer pairs with human-labeled relevant passages, refreshed quarterly as documentation changes. Without this, a reranker swap or model upgrade is a guess, not a measurement.
Batch reranking for offline pipelines
For non-interactive use cases (nightly re-indexing, bulk relevance audits), batch the reranker calls instead of calling them one query at a time:
def batch_rerank(queries: list[str], candidate_lists: list[list[str]]):
all_pairs = []
boundaries = [0]
for query, docs in zip(queries, candidate_lists):
all_pairs.extend([[query, doc] for doc in docs])
boundaries.append(len(all_pairs))
all_scores = reranker.predict(all_pairs) # one batched forward pass
results = []
for i in range(len(queries)):
start, end = boundaries[i], boundaries[i + 1]
results.append(all_scores[start:end])
return results
Batching cut our offline eval pipeline runtime from around 40 minutes to under 6 minutes for the same 200-query, 50-candidate-each workload, we measured before and after the change.
Monitoring reranker drift
Log the reranker's score distribution over time. A shift toward lower top-1 scores across queries (without a corresponding change in query patterns) suggests document corpus drift, like new documentation that the reranker has not seen examples similar to during training.
Conclusion
Single-stage vector retrieval optimizes for the wrong thing: topical similarity instead of actual relevance. A second-stage cross-encoder reranker closes that gap by jointly scoring the query against each candidate, catching cases a bi-encoder embedding misses.
The numbers from our production rollout, all of which we measured on our own pipeline: answer accuracy on our internal eval set rose from 71% to 89% after adding reranking, and irrelevant-citation rate dropped from 22% to 4%. The reranking step itself adds under 40ms in the common case, which is a reasonable latency trade for that accuracy gain.
Reranking is not a silver bullet for every relevance gap. Structured metadata mismatches, like our version-number bug, need explicit filters rather than a smarter model. But for the broad class of relevance failures where the right document exists in the index but ranks too low, a cross-encoder reranker is close to a solved problem at this point, and it should be the default second stage in any production RAG pipeline, not an optional add-on.
The full pipeline, benchmark script, and eval harness are at amtocbot-droid/amtocbot-examples/rag-reranking.
Get the next one
One short email a week, covering a real production debugging story plus the companion code behind it. Low volume, unsubscribe whenever you want.
Reader challenge: run the recall-width benchmark above against your own document corpus and report the latency-versus-recall curve you get. Comment below or reply to the email with your numbers.
Sources
About the Author
Toc Am
Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.
Published: 2026-06-23 · Written with AI assistance, reviewed by Toc Am.
Get These In Your Inbox
Weekly deep-dives on AI engineering, no fluff. Join the newsletter →
Or grab the book ($39, ~100 pages) · Buy me a coffee
☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter
