Adding a Reranker to Your Self-Hosted RAG Pipeline: When It Helps and When It Doesn't

I ran the eval harness from the last post against my own RAG pipeline and found the same pattern every time: Recall@10 was fine, but Recall@1 was weaker than it should have been. The right document was in the top 10 results. It just was not reliably in position 1.

This is the problem reranking solves. A reranker takes the top-K results from your vector search and reorders them using a more expensive model that considers the query and document together. The result is a better-ordered list at the cost of latency.

This post covers when to add a reranker, what to expect from it on self-hosted hardware, and where it does not help.

The Two-Stage Retrieval Pattern

Standard RAG retrieval is a single stage: embed the query, search the index, return the top results. The embedding model encodes the query and documents into vectors, and relevance is approximated by vector distance.

The limitation is that vector distance is a coarse signal. Two vectors being close in embedding space means they share semantic structure. It does not mean the document precisely answers the query. For broad queries or queries where subtle distinctions matter, the top-1 result is often not the best result even when it is close in vector space.

A reranker is a second stage. After vector search returns top-K candidates, the reranker takes each (query, candidate) pair and scores them jointly using a cross-encoder model. Cross-encoders see both the query and document at the same time, which gives them much more signal than the separate embeddings used in vector search. The top-K candidates are then reordered by the reranker's scores.

The trade-off is inference cost. Cross-encoders are slower than embedding lookups because they run a full forward pass for each (query, candidate) pair. With K=10 and a moderate reranker, you are doing 10 cross-encoder inference calls per query instead of one embedding lookup.

Self-Hosted Reranker Options

Three models run well via Ollama or directly in Python without an API dependency:

ms-marco-MiniLM-L-6-v2: a 22M-parameter cross-encoder fine-tuned on the MS MARCO passage retrieval dataset. Fast, low memory, and the default starting point for most reranking setups. Available via sentence-transformers.

ms-marco-MiniLM-L-12-v2: the 12-layer variant of the same model. Slightly better quality at the cost of roughly double the inference time.

bge-reranker-v2-m3: from BAAI, the same research group behind bge-m3 embeddings. A 568M-parameter model with significantly better reranking quality, particularly on multilingual and technical content. Considerably slower on CPU but strong on GPU.

Results on the Same Corpus

Using the same 500,000-document corpus from the vector database and embedding model comparisons, nomic-embed-text as the base retrieval model, and the same 1,000-query eval set:

Configuration	Recall@10	Recall@1	p50 latency	p99 latency
nomic-embed-text only	87%	68%	8ms	31ms
+ ms-marco-MiniLM-L-6-v2	87%	79%	41ms	87ms
+ ms-marco-MiniLM-L-12-v2	87%	82%	78ms	163ms
+ bge-reranker-v2-m3	87%	86%	198ms	421ms

Recall@10 stays flat. Reranking does not change which documents are retrieved; it changes how they are ordered. Recall@1 is where the improvement shows up.

The MiniLM-L-6 reranker takes Recall@1 from 68% to 79% with modest latency. The bge-reranker-v2-m3 pushes it to 86% but adds substantial latency. For a RAG system where the LLM generates answers primarily from the top result, the difference between 68% and 86% Recall@1 is significant.

When Reranking Helps

Reranking helps most when:

Your top result matters a lot. If you pass only the top 1-3 documents to the LLM, or if the LLM's attention focuses heavily on the first context chunk, improving Recall@1 translates directly to answer quality.

You have sufficient latency budget. Adding a reranker adds latency that varies widely by model and hardware. In our runs it ranged from a few dozen milliseconds for the smallest model to several hundred for the largest. If your use case tolerates latency in the hundreds of milliseconds, the quality gain is usually worth it. For interactive applications with tight latency requirements, the MiniLM models fit; bge-reranker may not.

Your queries are specific and discriminative. Reranking helps most when the correct answer is clearly distinguishable from the alternatives given the full query and document text together. For vague or broad queries, the quality improvement is smaller.

When Reranking Does Not Help

When Recall@10 is already the bottleneck. If the right document is not in the top 10 from vector search, reranking cannot fix it. Run your eval harness to check whether your problem is retrieval precision (wrong ordering) or retrieval recall (right document not in the candidate set). Reranking addresses the former.

When latency is tightly constrained. If your system needs very tight retrieval latency, even the smallest reranker may not fit. Profile first. Profile first.

For short factual queries with clear lexical matches. Simple lookups where the query words closely match the document words benefit less from cross-encoder reranking.

Implementation

Adding a reranker with sentence-transformers:

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def rerank(query: str, candidates: list[dict], top_n: int = 3) -> list[dict]:
    pairs = [(query, c["text"]) for c in candidates]
    scores = reranker.predict(pairs)
    ranked = sorted(
        zip(scores, candidates),
        key=lambda x: x[0],
        reverse=True
    )
    return [doc for _, doc in ranked[:top_n]]

The full pipeline with Qdrant:

from qdrant_client import QdrantClient
import ollama

client = QdrantClient(host="localhost", port=6333)

def embed(text: str) -> list[float]:
    return ollama.embeddings(model="nomic-embed-text", prompt=text)["embedding"]

def retrieve_and_rerank(query: str, k_retrieve: int = 20, k_return: int = 5) -> list[dict]:
    vector = embed(query)
    results = client.search(
        collection_name="your_collection",
        query_vector=vector,
        limit=k_retrieve,
        with_payload=True
    )
    candidates = [
        {"id": str(r.id), "text": r.payload.get("text", ""), "score": r.score}
        for r in results
    ]
    return rerank(query, candidates, top_n=k_return)

One detail that matters: retrieve more candidates than you return (k_retrieve > k_return). The reranker needs enough candidates to work with. Retrieving 20 and returning the reranked top 5 is a common pattern. Retrieving only 5 and reranking to return 5 adds latency without giving the reranker enough candidates to make a meaningful difference.

The Latency-Quality Curve

There is no single right point on this curve. The right trade-off depends on your use case:

For an internal developer tool where latency is not critical, bge-reranker-v2-m3 is reasonable — we measured it at roughly 200ms p50 in our runs on this hardware.

For a customer-facing chat interface, MiniLM-L-6 is the right starting point — we measured it at roughly 40ms p50 in our runs. If the quality improvement does not satisfy your Recall@1 target, consider improving the base retrieval (better embedding model, larger chunk overlap) before adding the heavier reranker.

For batch processing or async retrieval where latency is irrelevant, use the highest-quality reranker that fits on your hardware.

Profile on your actual hardware before committing. The numbers above are from a specific GPU configuration. On CPU-only hardware, bge-reranker-v2-m3 will be significantly slower.

---

Get the next one

I send one short email a week: one production bug, debugged, plus the
companion code for each deep-dive. No spam, unsubscribe anytime.

👉 Subscribe (free)

Reader challenge: have you measured the Recall@1 improvement from adding a reranker to your pipeline? Reply with the model you used and the before/after numbers.

Sources

1. sentence-transformers CrossEncoder documentation: https://www.sbert.net/docs/cross_encoder/usage/usage.html
2. BAAI bge-reranker-v2-m3: https://huggingface.co/BAAI/bge-reranker-v2-m3
3. MS MARCO dataset: https://microsoft.github.io/msmarco/

About the Author

Toc Am

Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.

LinkedIn X / Twitter

Published: 2026-07-27 · Written with AI assistance, reviewed by Toc Am.

Get These In Your Inbox

Weekly deep-dives on AI engineering, no fluff. Join the newsletter →

Subscribe (free)

Or grab the book ($39, ~100 pages) · Buy me a coffee

☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter