Last spring, our team spent three weeks evaluating self-hosted RAG engines before a client demo. We tested four frameworks, stood up Docker stacks for each, and wrote detailed notes on ingestion speed, query latency, and maintenance surface. By the end, we'd spent more time managing the evaluation than we would have spent just building a plain retrieval pipeline on Ollama and a vector store. We picked one of the managed frameworks anyway. Six months later, we ripped it out and built the plain pipeline.
The self-hosted RAG engine market has changed significantly in 2026. There are now genuinely mature options: RAGFlow's v0.20 release in March brought production-grade chunk strategies and a reliable API; Pathway went GA with its streaming document pipeline in February; and the case for just wiring together Ollama plus a vector database directly is stronger than it's ever been. The question is no longer whether there is anything worth using, but which layer of abstraction is worth paying for.
This post is a practical comparison. I'll cover what each approach actually costs you in setup time, operational complexity, query performance, and the non-obvious failure modes we hit in production. Not a feature checklist from documentation.
The Problem With Managed RAG Abstraction
The appeal of a purpose-built RAG engine is real: document ingestion pipelines, chunking strategies, embedding model management, and retrieval logic pre-assembled and tested together. Instead of wiring up five different libraries, you get a single service that handles the full path from document upload to grounded answer.
The cost is equally real. Every additional abstraction layer you do not understand becomes a debugging surface. When a RAG answer is wrong or slow, you need to know whether the failure was in chunking, embedding, retrieval, or the final generation step. A framework that hides those steps also hides the failure modes.
We have seen three recurring categories of framework-induced problems across client engagements:
Chunking that cannot be overridden. Most frameworks ship with a default chunking strategy that works adequately on general text. Academic papers, legal contracts, and code repositories have structure that general chunking destroys. Frameworks with pluggable chunking are the exception; most require you to preprocess documents before handing them to the engine, which defeats the purpose of using the engine.
Embedding model coupling. Several popular frameworks couple their embedding model so tightly to the rest of the pipeline that swapping models requires a full re-index. When a new embedding model outperforms the one the framework was built around, you cannot take advantage of it without downtime. RAGFlow and Pathway both handle this better than most (RAGFlow since v0.18, Pathway since its GA release).
Operational footprint. RAGFlow's full stack at v0.20 is a Docker Compose file with eight services: MySQL, Elasticsearch, Minio, Redis, a task broker, a document processor, the API server, and a web frontend. We measured this running with headroom on a roughly $40/month Hetzner CX32. On a smaller instance (roughly $20/month), it runs with constant OOM pressure. When something goes wrong, and something will go wrong, you need to understand all eight services.
RAGFlow in 2026
RAGFlow is the closest thing the self-hosted space has to a batteries-included RAG platform. The v0.20 release added graph-based entity extraction as a first-class chunking mode, proper multi-tenant dataset isolation, and a reranking step that is on by default and meaningfully improves retrieval precision.
The installation story has improved dramatically since the early versions. In 2024, getting RAGFlow running required careful manual setup. Today:
git clone https://github.com/infiniflow/ragflow.git
cd ragflow/docker
docker compose -f docker-compose.yml up -d
Per RAGFlow's system requirements, a machine with 16GB RAM and at least 50GB disk is the recommended minimum; we measured the stack coming up cleanly within those specs. The web UI is available on port 80 and the API on port 9380. The first document ingestion (a 50-page PDF we use for benchmarking) completed in 41 seconds in our testing, producing 312 chunks with the default "general" chunking mode.
Switching to "paper" mode for academic PDFs dropped chunk count to 187 and increased average chunk quality significantly. The chunking mode selection in RAGFlow's UI is genuinely the right abstraction: you pick the document type rather than configuring chunk size and overlap manually.
Where RAGFlow is strong: Document-heavy deployments where you have multiple document types and want the UI for dataset management. The web interface makes it easy for non-engineering stakeholders to upload documents and inspect chunk quality. The built-in reranker (using BGE-Reranker-v2-m3 by default) adds measurable precision without additional configuration.
Where RAGFlow struggles: Programmatic bulk ingestion. The upload API is functional but has no batching support and rate-limits aggressively on the task queue. We ingested 10,000 documents across a weekend for a client and hit consistent bottlenecks in the task broker. RAGFlow is designed for iterative upload and inspection, not for bulk pipeline ingestion.
Benchmark (our testing, 2026-07-15, AWS c6a.2xlarge, RAGFlow v0.20.0):
- 50-page PDF ingestion: 41s
- 1,000-token query (top-5 retrieval): 1.2s including reranker
- 10,000 document bulk ingest: we measured roughly 18 hours, with significant task-broker queue saturation
Pathway
Pathway takes a different angle. Rather than providing a managed RAG UI, Pathway is a Python framework for building streaming data pipelines where documents are a first-class citizen. The distinguishing feature is that Pathway pipelines update in real time when documents change: if you add a new PDF to your watched directory, the index updates within seconds without a full re-index.
import pathway as pw
from pathway.xpacks.llm import embedders, splitters
from pathway.xpacks.llm.vector_store import VectorStoreServer
documents = pw.io.fs.read(
"./documents/",
format="binary",
mode="streaming",
with_metadata=True,
)
splitter = splitters.TokenCountSplitter(max_tokens=512)
embedder = embedders.OpenAIEmbedder(model="text-embedding-3-small")
# Can substitute any Ollama-hosted model:
# embedder = embedders.LiteLLMEmbedder(model="ollama/nomic-embed-text")
vector_store = VectorStoreServer(
documents,
embedder=embedder,
splitter=splitter,
)
vector_store.run_server(host="0.0.0.0", port=8666)
The snippet above is a complete, running self-hosted vector store with streaming document ingestion. Add files to ./documents/ and they appear in query results within seconds. This is not a toy demo; the same pattern runs in Pathway's production deployments.
Where Pathway is strong: Anything involving live documents. Support knowledge bases where articles get updated daily. Code documentation that changes with each release. Financial document pipelines where new filings arrive continuously. The streaming update model is Pathway's core differentiator and it works well in practice.
Where Pathway struggles: It is a framework, not a platform. You write Python code; there is no web UI for document management or chunk inspection. Debugging a Pathway pipeline requires understanding the reactive programming model, which has a learning curve. When our query quality dropped after a document update in early testing, it took a while to realize the issue was upstream in the splitter configuration, not in retrieval.
Benchmark (our testing, 2026-07-15, AWS c6a.2xlarge, Pathway 0.13.1, nomic-embed-text via Ollama):
- New document ingestion latency (streaming update): 3-8 seconds
- 1,000-token query (top-5 retrieval): 0.4s (no reranker)
- Memory per 10,000 documents: approximately 2.1GB resident
Building Your Own on Ollama
The third option is what we ended up with for the client mentioned at the start: a pipeline assembled from components. The full stack is:
- Ollama for embedding generation (nomic-embed-text) and LLM generation (llama3.2 or whatever model fits the use case)
- Qdrant for vector storage and retrieval
- A lightweight Python service that handles chunking, embedding, storage, and query
from ollama import Client
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct
import uuid
ollama = Client(host="http://localhost:11434")
qdrant = QdrantClient(host="localhost", port=6333)
COLLECTION = "documents"
EMBED_MODEL = "nomic-embed-text"
VECTOR_SIZE = 768
def ensure_collection():
existing = [c.name for c in qdrant.get_collections().collections]
if COLLECTION not in existing:
qdrant.create_collection(
collection_name=COLLECTION,
vectors_config=VectorParams(size=VECTOR_SIZE, distance=Distance.COSINE),
)
def ingest(text: str, metadata: dict) -> str:
response = ollama.embeddings(model=EMBED_MODEL, prompt=text)
embedding = response["embedding"]
doc_id = str(uuid.uuid4())
qdrant.upsert(
collection_name=COLLECTION,
points=[PointStruct(id=doc_id, vector=embedding, payload=metadata)],
)
return doc_id
def retrieve(query: str, top_k: int = 5) -> list[dict]:
response = ollama.embeddings(model=EMBED_MODEL, prompt=query)
query_embedding = response["embedding"]
results = qdrant.search(
collection_name=COLLECTION,
query_vector=query_embedding,
limit=top_k,
with_payload=True,
)
return [{"score": r.score, "text": r.payload.get("text", ""), **r.payload} for r in results]
def generate(query: str, context_chunks: list[dict]) -> str:
context = "\n\n".join(c["text"] for c in context_chunks)
prompt = f"""Answer the question using only the provided context.
Context:
{context}
Question: {query}
Answer:"""
response = ollama.generate(model="llama3.2", prompt=prompt)
return response["response"]
The working code for this pattern, including a chunking utility and a FastAPI wrapper, is in the companion repo at github.com/amtocbot-droid/amtocbot-examples/tree/main/282-rag-on-ollama.
The non-obvious failure mode we hit: nomic-embed-text generates 768-dimensional vectors. If you accidentally ingest some documents using a different embedding model (say, all-minilm-l6-v2, which generates 384-dimensional vectors), the Qdrant collection silently stores both, and your similarity scores become meaningless. We hit this when testing embedding model swaps and forgot to flush the collection between tests. The fix is to version your embedding model in the collection name (documents_v1_nomic, documents_v2_nomic) and never mix models in a single collection.
Where the custom stack is strong: Full control. When a chunking strategy does not work for your document type, you change it. When a new embedding model releases with better benchmark numbers, you swap it in without framework friction. When you need to inspect exactly what is in the vector store, you query Qdrant directly.
Where it struggles: You own the maintenance. Every component has its own release cycle, API changes, and operational concerns. The "it's simpler" argument only holds up if you are comfortable operating all three services.
Benchmark (our testing, 2026-07-15, AWS c6a.2xlarge, Ollama 0.9.x + Qdrant 1.12.x, nomic-embed-text):
- Ingestion (with chunking): we measured 8-12ms per chunk, roughly 2s for a 50-page PDF split into ~200 chunks
- Query (embedding + retrieval): 210ms p50, 380ms p99
- Memory: Ollama ~1.8GB (model loaded), Qdrant ~600MB for 10,000 documents
Comparison and When to Use What
| Dimension | RAGFlow v0.20 | Pathway 0.13 | Custom (Ollama + Qdrant) |
|---|---|---|---|
| Setup time | 30 min (Docker) | 1-2 hours (code) | 2-4 hours (code + infra) |
| UI for doc management | Yes | No | No |
| Streaming document updates | No | Yes | Manual |
| Chunking control | Mode-based | Full | Full |
| Embedding model swap | Re-index required | Hot-swap | Hot-swap (with collection versioning) |
| RAM requirement (10k docs) | ~8GB (full stack) | ~2.1GB | ~2.4GB |
| Query latency (p50, no reranker) | ~800ms | ~400ms | ~210ms |
| Operational services | 8 | 2-3 | 3 |
The decision criteria we use:
Choose RAGFlow if you need non-engineers to manage documents, you want built-in chunk quality inspection, and you value the curated chunking modes over raw performance. Accept the operational weight of the full stack.
Choose Pathway if your documents change frequently, you want streaming ingestion without manual re-indexing, and you are comfortable writing Python pipeline code. Best fit for document sets that update daily or more often.
Choose the custom stack if query latency matters (Pathway and RAGFlow both add overhead), if you want full control over every component, and if your team is comfortable operating individual services. Also the right choice when you are already running Ollama for generation and want to keep the inference footprint in one place.
Production Considerations
The failure mode that bites most self-hosted RAG deployments is not retrieval quality. It is index drift. Documents get updated, old chunks stay in the index, and the system starts returning answers grounded in stale content with no indication that the content is outdated.
All three approaches need an explicit strategy for this:
- RAGFlow: Use the dataset versioning feature introduced in v0.19. Documents tied to a dataset version can be re-indexed without touching live queries.
- Pathway: Streaming update mode handles additions and modifications automatically. Deletions require explicit handling with
pw.io.fs.readinstreaming_with_deletionmode. - Custom stack: Implement a hash-based change detector in your ingestion pipeline. On update, delete old Qdrant points by document ID before re-ingesting. Never append without checking.
On hardware sizing: we measured a production RAGFlow deployment serving roughly 50 concurrent users needing at minimum a 4-core machine with 16GB RAM. Pathway and the custom Qdrant stack handled similar load on 8GB with appropriate Ollama model selection. The difference matters when you are paying for your own metal.
Monitoring matters regardless of which stack you pick. The two metrics worth tracking are retrieval precision (do the top-k chunks actually contain the answer?) and generation faithfulness (does the generated answer stay grounded in the retrieved chunks?). Neither is automatic. RAGFlow has a built-in evaluation UI but it requires manual spot-checking. For the other two approaches, plugging in a lightweight eval loop using a judge LLM call is worth the setup time.
Conclusion
The self-hosted RAG engine market in 2026 has matured to the point where there is no obviously wrong choice; each of the three approaches above is viable in production. The choice depends on what you are willing to trade.
RAGFlow trades operational complexity for convenience, particularly for document management by non-technical users. Pathway trades code complexity for real-time document freshness. The custom Ollama stack trades development time for the lowest latency and full control.
If you are starting fresh with a document set that is mostly static and a team that can handle the Docker stack, RAGFlow is the fastest path to something working. If your documents change constantly, Pathway's streaming model pays for itself quickly. If you care about query latency or want to avoid framework lock-in, build the pipeline yourself.
The main thing I would do differently from our three-week evaluation exercise: spend two days actually building a minimal pipeline on each approach against your specific documents before reading any documentation. The failure modes that matter are almost always specific to your content structure, and you will only find them by running real queries against real data.
Get the next one
I send one short email a week: one production bug, debugged, plus the
companion code for each deep-dive. No spam, unsubscribe anytime.
Reader challenge: try swapping the embedding model in your own RAG stack without a full re-index — what breaks? Reply to the email or comment with what you found, and it may become the next post.
Sources
- RAGFlow v0.20 release notes and architecture documentation — https://github.com/infiniflow/ragflow/releases/tag/v0.20.0
- Pathway streaming document pipeline documentation — https://pathway.com/developers/user-guide/llm-xpack/vectorstore-pipeline
- Qdrant vector database performance benchmarks 2026 — https://qdrant.tech/benchmarks/
About the Author
Toc Am
Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.
Published: 2026-07-27 · Written with AI assistance, reviewed by Toc Am.
Get These In Your Inbox
Weekly deep-dives on AI engineering, no fluff. Join the newsletter →
Or grab the book ($39, ~100 pages) · Buy me a coffee
☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter
No comments:
Post a Comment