Every time I share benchmark numbers in this series, I add the same caveat: run your own evaluation on your own data. This post is how to actually do that.
The benchmark numbers for embedding models, vector databases, and chunking strategies are useful for narrowing the field. They tell you which options are worth testing. They do not tell you which option works for your documents, your queries, and your latency requirements. That requires a retrieval evaluation harness built around your actual workload.
This post walks through building a minimal but honest eval harness for a self-hosted RAG pipeline.
What a Retrieval Eval Harness Does
A retrieval eval harness answers one question: given this query, did the retrieval system return the right documents?
It does this by running a set of queries against your index, comparing the returned documents against a ground-truth set of relevant documents, and computing recall metrics. The output is a number you can track over time and compare across configuration changes.
The three things you need to build one:
- A set of queries representative of real user questions
- Ground-truth labels: for each query, which documents in your corpus are relevant?
- A scoring function: recall, precision, or NDCG depending on your use case
The hard part is step two. Getting ground-truth labels is the main reason most teams skip building an eval harness, and the main reason they cannot tell whether their retrieval is actually working.
Building the Query Set
The best queries come from real users. If your system is in production, sample actual queries and filter to ones where you know the answer. If you are building before production, generate synthetic queries from your corpus.
Synthetic query generation using a local LLM works well enough for an initial harness. For each document chunk you want to test, generate a question that the chunk would answer:
import ollama
def generate_query_for_chunk(chunk_text: str, model: str = "llama3.1:8b") -> str:
response = ollama.chat(
model=model,
messages=[{
"role": "user",
"content": (
"Generate one short question that this text answers directly. "
"Return only the question, nothing else.\n\n"
f"Text: {chunk_text}"
)
}]
)
return response["message"]["content"].strip()
Run this over a sample of your corpus, not every chunk. A representative sample works well here. For a corpus of hundreds of thousands of chunks, a sample in the low thousands gives stable metrics.
The query-chunk pairs from this process become your ground-truth dataset: for each generated query, the source chunk is the relevant document.
Labeling Ground Truth
For each query, you need to know which documents are relevant. There are three ways to do this:
Synthetic labels from generation: the chunk you generated the query from is the relevant document. Fast, zero human effort, and good enough to catch major regressions. The weakness is that it only measures whether retrieval can find the specific chunk, not whether other chunks might also be relevant.
LLM-as-judge: for each (query, candidate document) pair, ask a local LLM to rate relevance. This is slower and adds noise but can catch cases where multiple documents are relevant to the same query.
Human labels: the most accurate, required for production evals where you care about fine-grained recall at the top of the list. For an initial harness, human labels for a few hundred queries is a good investment.
For a self-hosted pipeline getting started, synthetic labels with a few hundred query-chunk pairs get you 80% of the value at near-zero cost.
The Scoring Function
Recall@K measures whether a relevant document appears in the top K results. It is the right metric for RAG because the LLM can only use what the retrieval system returns.
def recall_at_k(retrieved_ids: list[str], relevant_ids: set[str], k: int) -> float:
top_k = retrieved_ids[:k]
hits = sum(1 for doc_id in top_k if doc_id in relevant_ids)
return hits / len(relevant_ids) if relevant_ids else 0.0
Recall@1 is particularly useful for RAG: it tells you whether the single most relevant document is at the top of the list. When your LLM only reads the top result (or when the top result heavily influences the generated answer), Recall@1 matters more than Recall@10.
Mean Reciprocal Rank (MRR) is useful when you care about ranking quality rather than just presence in the top K:
def mrr(retrieved_ids: list[str], relevant_ids: set[str]) -> float:
for rank, doc_id in enumerate(retrieved_ids, start=1):
if doc_id in relevant_ids:
return 1.0 / rank
return 0.0
Putting It Together
A minimal eval loop that runs your query set against your index and computes recall:
from qdrant_client import QdrantClient
import ollama
client = QdrantClient(host="localhost", port=6333)
COLLECTION = "your_collection"
EMBED_MODEL = "nomic-embed-text"
def embed(text: str) -> list[float]:
return ollama.embeddings(model=EMBED_MODEL, prompt=text)["embedding"]
def retrieve(query: str, k: int = 10) -> list[str]:
vector = embed(query)
results = client.search(
collection_name=COLLECTION,
query_vector=vector,
limit=k
)
return [str(r.id) for r in results]
def run_eval(query_pairs: list[dict], k: int = 10) -> dict:
recall_scores = []
recall1_scores = []
for pair in query_pairs:
query = pair["query"]
relevant = {pair["relevant_chunk_id"]}
retrieved = retrieve(query, k=k)
recall_scores.append(recall_at_k(retrieved, relevant, k))
recall1_scores.append(recall_at_k(retrieved, relevant, 1))
return {
f"recall@{k}": sum(recall_scores) / len(recall_scores),
"recall@1": sum(recall1_scores) / len(recall1_scores),
"n_queries": len(query_pairs)
}
Run this before and after any configuration change. Embedding model swap, chunk size adjustment, index parameter tuning: all of these should go through the eval before you deploy.
What to Track Over Time
The numbers are only useful if you track them. Store eval results with enough context to understand what changed:
import json
from datetime import datetime
def save_eval_result(metrics: dict, config: dict, path: str = "eval_results.jsonl"):
record = {
"timestamp": datetime.utcnow().isoformat(),
"config": config,
"metrics": metrics
}
with open(path, "a") as f:
f.write(json.dumps(record) + "\n")
# Example config dict
config = {
"embed_model": "nomic-embed-text",
"collection": "docs_v3",
"chunk_size": "medium",
"index_params": {"m": 16, "ef_construct": 100}
}
A few hundred eval results stored in a JSONL file is enough to track trends and spot regressions. You do not need a dedicated metrics platform for a self-hosted setup.
Common Issues and What They Indicate
Recall@10 is good but Recall@1 is poor: the relevant document is in the top 10 but rarely at position 1. Consider adding a reranker after retrieval to push the most relevant result to the top.
Recall drops on filtered queries: your index does not handle metadata filters efficiently. Check whether your vector database applies filters during search or post-search. Qdrant filters during HNSW traversal; others filter after retrieval, which cuts effective recall with selective filters.
High recall on synthetic queries but poor quality answers from the LLM: retrieval is finding the right chunks but they may not contain enough context. Try larger chunks or a sliding window approach that overlaps adjacent chunks.
Recall is stable but answer quality regressed: the issue is downstream of retrieval. Check your prompt template, context assembly, and LLM.
A Note on Eval Harness Maintenance
The query set goes stale as your corpus evolves. Regenerate synthetic queries when you add a major new document category. Add real user queries to the set whenever you can — even a slow trickle of labeled real queries improves the harness over time.
The eval harness is not a one-time project. It is infrastructure. Treat it like your index backup: set it up once, run it on a schedule, and update it when the underlying system changes.
Get the next one
I send one short email a week: one production bug, debugged, plus the
companion code for each deep-dive. No spam, unsubscribe anytime.
Reader challenge: what query set are you using to evaluate your RAG pipeline? Synthetic, real, or none? Reply with what you are doing.
Sources
- MTEB Leaderboard, retrieval benchmarks: https://huggingface.co/spaces/mteb/leaderboard
- Qdrant filtering documentation: https://qdrant.tech/documentation/concepts/filtering/
- Ollama embeddings API: https://github.com/ollama/ollama/blob/main/docs/api.md#generate-embeddings
About the Author
Toc Am
Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.
Published: 2026-07-27 · Written with AI assistance, reviewed by Toc Am.
Get These In Your Inbox
Weekly deep-dives on AI engineering, no fluff. Join the newsletter →
Or grab the book ($39, ~100 pages) · Buy me a coffee
☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter
No comments:
Post a Comment