Context Compression in Self-Hosted RAG: Fitting More Signal Into the Context Window

I was retrieving K=10 chunks for every query and watching the model's answer quality plateau. Increasing K to 20 didn't help, and it made latency and token cost worse. The problem wasn't that I needed more retrieved content. The problem was that most of what I retrieved wasn't relevant to the specific question.

Context compression is the step between retrieval and generation: take the retrieved chunks, extract only the parts that directly address the query, and pass a shorter, denser context to the model. The same context window fits more signal and less noise.

This post covers how to implement context compression in a self-hosted pipeline, what compressors work at production latency, and how to measure whether compression is improving answer quality.

Why Retrieved Chunks Are Noisy

Retrieval operates at the chunk level. Each chunk was split at a fixed boundary: by token count, by paragraph, or by heading. The boundary has no knowledge of future queries. When a user asks a narrow question, the relevant sentences may be scattered across several chunks, each of which also contains unrelated content.

Three types of noise in retrieved chunks:

Structural noise: headers, footers, navigation text, repeated boilerplate. These score well on embeddings because they appear near relevant content, but add nothing to the answer.

Topical noise: a chunk about topic A that also mentions topic B. The query is about topic B, so the chunk retrieves, but half its tokens are about topic A.

Redundancy: multiple chunks that say the same thing in slightly different words. Reranking helps, but rarely eliminates all redundancy.

Passing all of this to the model wastes tokens and dilutes the signal-to-noise ratio in the prompt.

Compression Approaches

Approach 1: Extractive Compression

Extract only the sentences from each chunk that are relevant to the query. No summarization: the compressed output is a verbatim subset of the original.

import ollama

EXTRACTIVE_PROMPT = """Given the following retrieved passage and a user query, extract only the sentences from the passage that directly help answer the query. Return only the extracted sentences, preserving their original wording. If no sentences are relevant, return an empty string.

Query: {query}

Passage:
{passage}

Extracted sentences:"""

def extractive_compress(query: str, passage: str, model: str = "llama3.2:3b") -> str:
    response = ollama.generate(
        model=model,
        prompt=EXTRACTIVE_PROMPT.format(query=query, passage=passage),
        options={"temperature": 0}
    )
    return response["response"].strip()

Extractive compression is fast with a small model (llama3.2:3b adds a few hundred milliseconds per chunk in our tests) and preserves exact wording, which matters for factual queries where paraphrasing introduces error.

Approach 2: Abstractive Compression

Summarize each chunk in the context of the query. The output is shorter than extractive but may rephrase content.

ABSTRACTIVE_PROMPT = """Summarize the following passage to include only information relevant to answering the query. Be concise. If the passage contains nothing relevant, return an empty string.

Query: {query}

Passage:
{passage}

Summary:"""

def abstractive_compress(query: str, passage: str, model: str = "llama3.2:3b") -> str:
    response = ollama.generate(
        model=model,
        prompt=ABSTRACTIVE_PROMPT.format(query=query, passage=passage),
        options={"temperature": 0}
    )
    return response["response"].strip()

Abstractive compression produces more compact output but introduces a small risk of hallucination in the compression step itself. Use it for conceptual questions where paraphrase is acceptable, and extractive for factual lookups.

Approach 3: Sentence-Level Filtering (No LLM)

Score each sentence in a chunk by cosine similarity to the query embedding. Keep only sentences above a threshold. Fast, no LLM call, but misses cross-sentence context.

import ollama
import numpy as np

def embed(text: str) -> np.ndarray:
    vec = ollama.embeddings(model="nomic-embed-text", prompt=text)["embedding"]
    return np.array(vec)

def sentence_filter_compress(
    query: str,
    passage: str,
    threshold: float = 0.6
) -> str:
    query_vec = embed(query)
    sentences = [s.strip() for s in passage.split('.') if s.strip()]
    kept = []
    for sentence in sentences:
        if len(sentence) < 10:
            continue
        sent_vec = embed(sentence)
        score = float(np.dot(query_vec, sent_vec) / (
            np.linalg.norm(query_vec) * np.linalg.norm(sent_vec)
        ))
        if score >= threshold:
            kept.append(sentence)
    return '. '.join(kept) + ('.' if kept else '')

Sentence filtering adds an embedding call per sentence (fast) instead of an LLM generation call. At K=10 chunks with an average of 8 sentences each, that's 80 embedding calls, which completes in well under a second with nomic-embed-text on a CPU in our tests.

Full Pipeline with Compression

def rag_with_compression(
    query: str,
    k: int = 10,
    compression: str = "extractive",  # "extractive", "abstractive", "sentence"
    min_compressed_length: int = 20
) -> dict:
    # Retrieve
    chunks = retrieve(query, k=k)

    # Compress each chunk
    compressed = []
    for chunk in chunks:
        if compression == "extractive":
            result = extractive_compress(query, chunk["text"])
        elif compression == "abstractive":
            result = abstractive_compress(query, chunk["text"])
        else:
            result = sentence_filter_compress(query, chunk["text"])

        # Drop empty or near-empty results
        if len(result.strip()) >= min_compressed_length:
            compressed.append(result)

    if not compressed:
        # Fall back to top-3 uncompressed if compression removed everything
        compressed = [chunk["text"] for chunk in chunks[:3]]

    context = "\n\n".join(compressed)
    response = generate(query, context)
    return {"response": response, "compressed_chunks": len(compressed), "original_chunks": len(chunks)}

When to Apply Compression

Compression adds latency. Apply it selectively:

Good candidates for compression:
- Long chunks (500+ tokens) where queries are narrow
- FAQ and support RAG where the query asks for a single fact buried in a larger document
- Synthesis queries where multiple chunks overlap significantly

Poor candidates for compression:
- Very short chunks (a sentence or two) where compression overhead exceeds benefit
- Chunks that are already tightly scoped to one topic
- Queries that need full procedural context (step-by-step instructions where any omitted step breaks the answer)

def should_compress(chunks: list[dict], query: str, token_threshold: int = 200) -> bool:
    avg_chunk_tokens = sum(len(c["text"].split()) for c in chunks) / len(chunks)
    return avg_chunk_tokens > token_threshold

Measuring Compression Effectiveness

Track three metrics:

Compression ratio: tokens in compressed context / tokens in original context. A ratio below 0.5 is aggressive; 0.6-0.8 is typical for extractive compression.

Answer quality: measure faithfulness (does the answer contradict the source?) and relevance (does the answer address the query?) against a labeled eval set. Use an LLM judge at each threshold.

Latency delta: compression adds time. Track whether the token savings at generation offset the compression overhead in wall-clock time.

class CompressionMetrics:
    def __init__(self):
        self.total = 0
        self.original_tokens = []
        self.compressed_tokens = []
        self.latency_compression_ms = []
        self.latency_generation_ms = []

    def record(
        self,
        original_tokens: int,
        compressed_tokens: int,
        compression_ms: float,
        generation_ms: float
    ) -> None:
        self.total += 1
        self.original_tokens.append(original_tokens)
        self.compressed_tokens.append(compressed_tokens)
        self.latency_compression_ms.append(compression_ms)
        self.latency_generation_ms.append(generation_ms)

    def report(self) -> dict:
        avg = lambda lst: sum(lst) / len(lst) if lst else 0
        return {
            "avg_compression_ratio": avg(self.compressed_tokens) / avg(self.original_tokens) if avg(self.original_tokens) else 0,
            "avg_compression_latency_ms": avg(self.latency_compression_ms),
            "avg_generation_latency_ms": avg(self.latency_generation_ms),
            "total_queries": self.total,
        }

Combining Compression with Reranking

Compression and reranking are complementary. Reranking selects the best chunks; compression extracts the best content from those chunks. A typical order:

1. Retrieve K=20 (broad recall)
2. Rerank to top 5 (precision)
3. Compress each of the top 5 (reduce noise)
4. Generate from the compressed context

This gives the model a context that is both high-precision (from reranking) and high-density (from compression).

When Context Compression Is Worth the Complexity

Compression adds a model call or embedding calls per retrieved chunk, plus code to handle empty outputs and fallbacks. It is worth it when:

• Your chunks are large relative to your queries
• You are hitting context window limits at K values that give acceptable recall
• Token cost at generation is a meaningful constraint (large hosted models)

If your chunks are small and well-scoped, or your context window is large relative to K * chunk_size, compression adds complexity without meaningful benefit. Profile token usage per query before adding compression.

---

Get the next one

I send one short email a week: one production bug, debugged, plus the
companion code for each deep-dive. No spam, unsubscribe anytime.

👉 Subscribe (free)

Reader challenge: are you compressing retrieved context before generation? What compressor approach works for your chunk sizes and query distribution? Reply with what you found.

Sources

1. LangChain ContextualCompressionRetriever: https://python.langchain.com/docs/how_to/contextual_compression/
2. nomic-embed-text on Ollama: https://ollama.com/library/nomic-embed-text
3. llama3.2 on Ollama: https://ollama.com/library/llama3.2

About the Author

Toc Am

Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.

LinkedIn X / Twitter

Published: 2026-07-27 · Written with AI assistance, reviewed by Toc Am.

Get These In Your Inbox

Weekly deep-dives on AI engineering, no fluff. Join the newsletter →

Subscribe (free)

Or grab the book ($39, ~100 pages) · Buy me a coffee

☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter