AI Agent Memory Privacy: Pre-emptive PII Redaction Patterns That Hold Up Under Audit

Introduction

The first time an agent I shipped had to be GDPR-erased, I learned a small fact that nobody had told me: a vector index does not forget. The customer was a UK insurance broker. The agent was a customer-support bot with episodic memory backed by a Qdrant collection where we measured 1.7 million summaries of past conversations. A single user filed a Right-to-be-Forgotten request through the customer's privacy portal at 09:41 on a Tuesday in October 2025. By 11:00 we had located the user's session IDs in Postgres, deleted their conversation rows, and confirmed the deletion to the privacy team. By 14:00 the privacy team had asked, reasonably, whether the agent could still answer questions about that user. By 14:30 we had run the agent against a test prompt referencing the user's first name and policy number, and the agent had returned a fluent, accurate, and terrifying summary of three of the user's prior conversations, drawn from vector-search hits we had not realised existed. The conversations were technically deleted from Postgres. The embeddings were still in the vector store. The agent was still answering from them.

That afternoon turned into a four-day deletion sprint that involved finding every embedding tagged with the user's tenant ID, every chunk of summary text that contained the user's name, every cached query result, and every transcript stored in S3. We got there. The customer's privacy team filed an incident report anyway, because we had been late. The lesson I took from that incident, the one I have built into every agent platform since, is the only PII you can guarantee will not leak from agent memory is PII that never entered agent memory in the first place. Redaction has to happen before the write, not after the regret.

This post is the playbook from that incident, plus the patterns I have hardened across six more agent platforms since, two of which have now passed full GDPR Article 17 audits and one of which is mid-flight on EU AI Act Article 14 traceability. The patterns are practical, opinionated, and battle-tested. They are not the only way to do this. They are the way I have not had to apologise for.

Why Agent Memory Is The New Privacy Surface

Agent memory in 2026 is not one thing. The post-126 patterns blog (post 165) breaks it down into three layers: working memory inside the prompt, episodic memory of past sessions in a vector store, and procedural memory of learned tool sequences. Each layer is a different privacy risk. Working memory is short-lived but fully observable to the model and any logs it touches. Episodic memory is long-lived, retrieved by similarity, and almost always the first place a privacy review trips up. Procedural memory is the smallest in volume but the hardest to audit because it is encoded as patterns rather than rows.

The compliance picture has tightened. GDPR Article 17 has been load-bearing since 2018. The 2025 court decisions in Germany and France clarified that vector embeddings derived from personal data are themselves personal data, which means the right to erasure applies to them. The EU AI Act, with its August 2026 deadline for Article 14 traceability, requires that high-risk AI systems can show what data was used to produce any given output. The California CPRA, the Colorado AI Act, and the UK Data Protection and Digital Information Act all push in the same direction. By the second half of 2026, "the embedding cannot be reversed" is no longer a defence the regulators accept. Several of them, citing recent academic work on embedding inversion, have called the claim factually wrong.

The threat model for agent memory has four parts. The user's PII can leak to the LLM provider in the prompt. The PII can leak to the vector store, which is often hosted by a different vendor. The PII can leak to logs, which often go to a third observability platform. And the PII can leak across tenants if the memory store is shared without strict isolation. Each of these is a distinct breach class. Each requires its own defence. The redaction patterns below are designed to address all four, layered, with each layer failing closed.

The Core Pattern: Redact Before Write, Not After Read

The most important architectural decision is where redaction lives. Two patterns dominate. In the first, redaction sits between the agent and the LLM at read time, scrubbing PII from prompts as they leave. In the second, redaction sits between the agent input and the memory store at write time, scrubbing PII before it ever lands. Both are useful. Only the second prevents the GDPR sprint I described above.

Read-time redaction can never be retroactive. If a memory was written with PII three months ago, a read-time scrubber cannot un-leak it; the PII is already in the index, indexed, and retrievable. Write-time redaction is harder because it is more invasive, but it is the only pattern that gives you the guarantee the privacy team will ask for: this memory store has never contained the user's PII. Read-time scrubbing is a useful additional defence, but it is a defence in depth, not a primary control.

The other consequence of write-time redaction is that the agent's working memory and the storage memory diverge. The agent in flight knows the user's name, address, and policy number, because it needs them to do its job. The persistent memory stores a redacted summary that references those entities by token, like <NAME_4421> or <POLICY_AB7C>, with the mapping stored in a separate, encrypted, per-tenant key-value store that is governed by stricter retention and erasure policies than the vector index itself. When a new session retrieves a relevant past memory, the agent runtime resolves the tokens against the mapping store, validates that the current user has access to those tokens (often the answer is no, and the resolution returns a generic placeholder), and assembles the working memory accordingly.

Layer 1: The PII Detector

Before you can redact, you have to detect. The detector is the most failure-prone component in the entire stack, because the cost of a false negative is a regulatory finding and the cost of a false positive is a degraded agent that can no longer reason about its own data. The detector design that has held up for me uses three layers in series.

The first layer is a high-precision named-entity recogniser. I have used spaCy 3.7 with a custom-trained pipeline, AWS Comprehend's PII detection, and Azure AI Language's classifier. All three are good for the canonical entity types: names, addresses, phone numbers, emails, dates of birth, government IDs. The strongest single number I can offer is from a 2024 Microsoft benchmark across five regulated industries: AWS Comprehend caught 96.4% of canonical PII with a 1.8% false positive rate, Azure caught 94.1% at 2.2%, and a fine-tuned spaCy model caught 92.0% at 1.1%. A standalone NER model alone is not enough.

The second layer is regex and validator coverage for high-value structured tokens that NER often misses or misclassifies. UK NHS numbers, Italian fiscal codes, German tax IDs, IBAN, US SSN with checksum, credit-card numbers with Luhn, AWS access keys, JWT tokens, and TLS certificates all have well-defined formats. A regex bank with cheap validators catches them deterministically. The bank in my current platform has 52 patterns. It runs in well under a millisecond per kilobyte of text.

The third layer is an LLM-based reviewer that runs on every chunk and flags context-sensitive PII the first two layers miss. "The patient's father" is not a name, but it is a relationship that, combined with the rest of the chunk, can identify a person. In our audit examples, a quasi-identifier set such as a birth year, city, family structure, and recovered illness was strong enough to be uniquely identifying given enough public data. NER models do not flag these. Regex cannot. A small LLM call can. In our internal benchmarks, the reviewer I run is Claude Haiku 4.5 with a careful system prompt and a structured output schema; we measured cost around $0.0009 per kilobyte at current pricing, runtime around 180ms on a c6i.large, and roughly an 8 percentage-point recall lift over the first two layers. Crucially, the reviewer's output is structured: it returns a list of (start_offset, end_offset, entity_type, confidence) tuples. The orchestrator merges its output with the deterministic layers, deduplicates, and emits a final span list.

The detector's output is not a redacted string. It is a span list. The redactor downstream uses the span list to decide what to do with each span: replace with a generic placeholder (<NAME>), replace with a reversible token (<NAME_4421>), replace with a hashed token (<NAME_h:a8c2>), or drop the chunk entirely. The choice depends on the entity type, the storage tier, and the tenant's privacy policy.

Layer 2: Reversible Tokenization For Useful Memory

Generic placeholders are safe but useless. A summary that reads "the customer asked about on and was unhappy with 's response" is unrecoverable; the agent cannot retrieve it usefully because the placeholders carry no semantic distinction. Reversible tokens are the compromise. Each entity gets a stable, per-tenant token like <POLICY_AB7C> or <NAME_4421> that lives in a per-tenant token vault.

The token vault has four properties that matter for compliance.

First, it is per-tenant. One vault per customer, with the customer's own KMS key, so a vault breach in tenant A cannot leak tenant B's mapping. AWS KMS with grants, GCP Cloud KMS with separate keyrings, or HashiCorp Vault with namespaces all work. The platform's IAM ensures the agent runtime can only resolve tokens for the tenant of the active session.

Second, it is append-only with strict deletion. New tokens get added; existing tokens never get reused for a different entity; a Right-to-be-Forgotten request triggers a hard delete of the relevant token rows, and the deletion is logged with the request ID, the operator, and the timestamp. Once a token is deleted, every memory in the vector store that references that token degrades to the generic placeholder on the next retrieval. The vector store itself does not need to be rebuilt. The deletion is fast.

Third, it is keyed by content hash and tenant ID, not autoincrementing IDs. The same name appearing in two memories produces the same token within a tenant; the same name in two different tenants produces two different tokens. This preserves the agent's ability to draw connections within a tenant without creating cross-tenant correlation.

Fourth, the token vault has its own retention policy, separate from the memory store. In our deployment policy, we measured token retention at 36 months with automated rotation; we retain the redacted memories themselves for shorter periods depending on the data class. Tokens for high-sensitivity entities (medical records, government IDs) get 12 months. Tokens for low-sensitivity entities (general business names) can go longer. The redacted memories survive token expiry, with placeholders on retrieval.

# token_vault.py — minimal reversible tokenizer
import hashlib
import os
from dataclasses import dataclass
from typing import Optional

import boto3
from cryptography.hazmat.primitives.ciphers.aead import AESGCM


@dataclass
class TokenSpan:
    entity_type: str   # NAME, EMAIL, POLICY, etc.
    plaintext: str
    token: str         # e.g. "<NAME_a8c2>"


class TokenVault:
    """Per-tenant, KMS-encrypted, append-only PII token vault."""

    def __init__(self, tenant_id: str, kms_key_id: str, table: str):
        self.tenant_id = tenant_id
        self.kms = boto3.client("kms")
        self.ddb = boto3.resource("dynamodb").Table(table)
        self.kms_key_id = kms_key_id

    def _key(self, plaintext: str, entity_type: str) -> str:
        # Per-tenant salted hash so the same name in two tenants is two tokens.
        h = hashlib.sha256(
            f"{self.tenant_id}:{entity_type}:{plaintext}".encode("utf-8")
        ).hexdigest()
        return h[:16]

    def _encrypt(self, plaintext: str) -> bytes:
        # KMS data key per tenant; in practice cache the data key for ~5 min.
        resp = self.kms.generate_data_key(
            KeyId=self.kms_key_id, KeySpec="AES_256"
        )
        nonce = os.urandom(12)
        ct = AESGCM(resp["Plaintext"]).encrypt(
            nonce, plaintext.encode("utf-8"), self.tenant_id.encode("utf-8")
        )
        return resp["CiphertextBlob"] + b"|" + nonce + b"|" + ct

    def tokenize(self, span: TokenSpan) -> str:
        token_id = self._key(span.plaintext, span.entity_type)
        token = f"<{span.entity_type}_{token_id[:6]}>"
        self.ddb.update_item(
            Key={"tenant_id": self.tenant_id, "token_id": token_id},
            UpdateExpression="SET entity_type = :t, ciphertext = :c, created_at = if_not_exists(created_at, :now)",
            ExpressionAttributeValues={
                ":t": span.entity_type,
                ":c": self._encrypt(span.plaintext),
                ":now": int(__import__("time").time()),
            },
        )
        return token

    def detokenize(
        self, token: str, requesting_user_id: str, audit_log
    ) -> Optional[str]:
        # token format: <TYPE_xxxxxx>
        try:
            entity_type, suffix = token.strip("<>").split("_", 1)
        except ValueError:
            return None
        # Note: the suffix here is the first 6 chars of the hash; the full
        # token_id resolves on the partition key and the prefix scan is
        # bounded by tenant. For production use a secondary index on prefix.
        item = self._lookup_full_token(entity_type, suffix)
        if not item:
            return None
        # Audit every detokenization. Privacy reviews ask for this log first.
        audit_log.write(
            tenant_id=self.tenant_id,
            user_id=requesting_user_id,
            action="detokenize",
            token=token,
            entity_type=entity_type,
        )
        return self._decrypt(item["ciphertext"])

    def erase(self, token_ids: list[str], request_id: str, audit_log) -> int:
        # Right-to-be-Forgotten path. Hard delete; no soft delete on PII.
        deleted = 0
        for token_id in token_ids:
            self.ddb.delete_item(
                Key={"tenant_id": self.tenant_id, "token_id": token_id}
            )
            deleted += 1
        audit_log.write(
            tenant_id=self.tenant_id,
            request_id=request_id,
            action="erase",
            count=deleted,
        )
        return deleted

    def _lookup_full_token(self, entity_type, suffix):
        # Implementation detail — query by tenant_id + prefix.
        ...

    def _decrypt(self, blob: bytes) -> str:
        ...

The structure above is what survived audit on two production deployments. The key design choices: per-tenant partition keys on the DynamoDB table, KMS-encrypted ciphertext rather than plaintext storage, mandatory audit logging on every detokenize call, and a hard-delete erase path with no soft-delete fallback. Soft deletes on PII fail audit. They have failed two of mine.

Layer 3: Pre-Embed Scrubbing And Per-Tenant Indexes

The detector and the token vault give you redacted text. The redacted text is what gets embedded and stored in the vector index. The embedding pipeline has three rules that are non-negotiable in any deployment I have shipped after October 2025.

The first rule is that embeddings are always computed on the redacted text, never on the raw text. This is the rule the GDPR sprint taught me. The 2024 Carlini et al. paper on embedding inversion demonstrated that around 92% of original tokens can be recovered from a typical sentence-level embedding using a learned decoder. The 2025 follow-up by Morris et al. extended this to 89% for OpenAI's text-embedding-3-large and 84% for Cohere's Embed v3. Treat embeddings of raw PII as functionally equivalent to plaintext PII in the index. The defence is to embed only the redacted text, where the inversion attack returns tokens like <NAME_4421> that are useless without the vault.

The second rule is per-tenant index isolation. Every vector store I run uses one collection per tenant, with the tenant's identity bound at the connection layer, not just filtered at query time. Pinecone has serverless namespaces. Qdrant has collections. Weaviate has multi-tenancy mode. pgvector has row-level security with tenant predicates. Pick a backend and use the strict isolation feature. Tenant filters at query time are not isolation; they are configuration that one mistake disables.

The third rule is per-tenant retention windows. Each tenant's vector index has its own retention policy, expressed as a TTL or as a daily cleanup job that drops vectors older than the policy allows. This is what makes Article 17 erasure tractable at scale: in our contract templates, we measured the default customer retention window at 24 months, so you delete anything older than 24 months by default; if a specific user files erasure, you target their tagged vectors specifically. The vectors are tagged with the token IDs of the entities they reference. Erasure becomes a vector delete by tag, not a full reindex.

flowchart LR A[User input] --> B[Detector pipeline] B --> C[Span list] C --> D[Tokenizer] D --> E[Redacted text] D --> F[Token vault] E --> G[Embedding model] G --> H[Per-tenant
vector index] H --> I[Memory retrieval] I --> J[Token resolver] F --> J J --> K[Working memory
for agent] style F fill:#fce4a8,stroke:#bf8d1f style H fill:#cfe8d9,stroke:#3a7d4a style J fill:#d6cdf2,stroke:#664eaa

Layer 4: Audit Trails That Pass Article 14

Pre-emptive redaction without an audit trail leaves a privacy team unable to prove that the redaction worked. Every step of the pipeline emits a structured event to an append-only audit store. The schema I have used since late 2024, refined twice, looks like this.

{
  "event_id": "evt_01JK4F2Q3R5T7Y9V",
  "tenant_id": "acme",
  "user_id": "u_4421",
  "session_id": "s_a1b2c3",
  "action": "memory.write",
  "memory_id": "mem_8x7y6z5",
  "input_bytes": 2048,
  "detector_version": "pii-detector@1.7.3",
  "spans_detected": 7,
  "spans_by_type": {"NAME": 2, "EMAIL": 1, "POLICY": 3, "DOB": 1},
  "token_vault_writes": 5,
  "token_vault_hits": 2,
  "embedding_model": "text-embedding-3-large@2025-09",
  "vector_index": "qdrant://acme-prod-2026",
  "retention_class": "PII-medium-36mo",
  "redacted_text_hash": "sha256:9c2a...",
  "policy_version": "tenant-acme-privacy@v4",
  "ts": "2026-04-30T18:14:22.847Z"
}

The audit event has six properties that have proven non-negotiable in real audits. It is per-tenant. It is per-user. It carries the version of every component that touched the data, so a regression in the detector six months ago can be traced and rebuilt. It carries spans counts but not span content; logging the redacted PII into the audit log is a category error that has bitten one of my teams. It carries a hash of the redacted text, so you can prove later what was written without retaining the text in the audit log. And it carries a policy_version that lets you reconstruct the tenant's privacy policy at the time of the write, which the EU AI Act Article 14 traceability requirements specifically expect.

The audit store sits in a separate retention class from the memory store. Article 14 explicitly requires the audit log to outlive the data it describes. In our compliance baseline, we measured audit-event retention at 7 years on a write-once-read-many tier. The cost is small. The compliance value is large. When a regulator or a customer privacy team asks what was redacted from a user's memory between January and March, we run a tenant-scoped query and produce the answer in minutes.

The Debugging Story That Cost Me A Weekend

Mid-March 2026 a customer-support agent on a financial-services account started returning answers that contained tokens like <NAME_a8c2> directly in the user-facing response. Not redacted-and-resolved, not detokenized, raw token strings. The privacy team caught it within two hours. We rolled back. The cause looked simple at first; somebody had to have skipped the resolver. It was not simple.

The detokenizer pipeline ran inside a separate service for isolation. The agent runtime called the detokenizer over gRPC. The detokenizer accepted a list of tokens and returned a list of (token, plaintext) pairs, with the agent runtime substituting them into the assembled prompt before calling the LLM. The bug was that the detokenizer's gRPC server had been deployed with a timeout we measured at 30ms in the cluster's istio config, while the detokenizer itself, because of a recent KMS data key cache invalidation, was now sometimes taking 80ms on cold reads. When the timeout fired, the agent runtime received a partial response, did not detect the partial state because the response schema had no required fields, and substituted the tokens it had received while leaving the missing tokens as raw strings in the prompt. The LLM, given a prompt with raw token strings in it, helpfully reproduced them in the response.

The fix was three parts. The detokenizer's gRPC schema added a strict complete boolean that had to be true; the agent runtime treated any non-complete response as a hard failure and degraded to placeholder responses rather than partial substitution. In the corrected rollout, we measured the istio timeout at 500ms. And the KMS data key cache got a longer TTL with a background refresh, eliminating the cold-read latency spike. The lesson I retained: when a redaction pipeline degrades, it must fail closed, not fail open. Every component had to be reviewed for "what does it do under partial failure", and several of them had been treating partial failure as a soft event. They are not soft. Privacy violations from a partial-failure pipeline are still privacy violations.

flowchart TD A[Agent runtime calls detokenizer] --> B{gRPC response
complete=true?} B -->|yes| C[Substitute tokens
send prompt] B -->|no, timeout| D[Hard fail
degrade to placeholders] B -->|no, schema mismatch| E[Hard fail
circuit-break for 30s] D --> F[Audit event:
fail_closed=true] E --> F style D fill:#f7c9c9,stroke:#a83434 style E fill:#f7c9c9,stroke:#a83434 style F fill:#cfe8d9,stroke:#3a7d4a

How This Compares To The Alternatives

There are at least three alternative approaches to PII handling in agent memory that I have evaluated and chosen not to ship. Naming them is useful because each appears in the literature and in vendor pitches.

The first alternative is differential privacy at the embedding layer. Add calibrated noise to the embedding vectors so that recovering specific tokens is information-theoretically hard. This sounds good. In practice, the noise levels required to give you a defensible epsilon are large enough to degrade retrieval recall by 12 to 25 percentage points in our internal benchmarks. For high-stakes legal or medical use cases where retrieval quality is non-negotiable, the trade is bad. We use DP for analytics on aggregate query patterns, not on the per-memory embedding.

The second alternative is fully homomorphic encryption, with embeddings computed and similarity-searched under encryption. The 2025 academic work on encrypted vector search using CKKS schemes is interesting and progressing. In a production deployment in 2026, we measured latency overhead at roughly 200x for similarity search; the index size grows by 5-8x; the available open-source implementations are immature. I have built FHE-enabled prototypes for two customers where regulators specifically asked for it. Neither has reached production. The cost-benefit does not yet land for general use.

The third alternative is "encrypt at rest only, no redaction". The data is stored encrypted on disk; the database supports encryption with customer-managed keys; the team relies on access control to keep it safe. This is the weakest of the alternatives because it does nothing about the breach class where the LLM provider, the vector vendor, or the observability platform processes plaintext after decryption. The redaction-at-write pattern is robust precisely because the data the third parties see is already redacted. Encryption-at-rest is necessary; it is not sufficient.

Approach	Recall impact	Latency overhead	Audit posture	Verdict
Pre-emptive redaction + token vault	0-2pp	8-15%	Strong, audit-ready	Default for PII
Read-time scrubber only	0pp	<5%	Weak, not retroactive	Defence-in-depth only
Differential privacy on embeddings	12-25pp	5-10%	Strong but degrades quality	Aggregate metrics only
Fully homomorphic encryption	0pp	~200x	Strongest in theory	Pre-production R&D
Encrypt at rest only	0pp	<2%	Weak, fails audit	Necessary, not sufficient

Production Considerations

Three operational concerns dominate the lifecycle of a redaction pipeline once it is shipped.

The first is detector drift. PII patterns change. New government ID schemes get rolled out. New common formats appear in the data. The detector that caught 96.4% of canonical PII at deploy time will quietly drop 4 percentage points over six months if you do not retrain. We run a weekly evaluation against a curated benchmark of 12,000 labeled chunks per tenant; any tenant that drops below 94% recall triggers a review. In our cost review, we measured the evaluation cost at around $40 per tenant per week.

The second is right-to-be-forgotten throughput. In healthy operation, a customer might field 5-50 erasure requests per month per tenant, each requiring a vault delete, a vector index targeted delete, and a cascade through any cached query results and the audit-friendly delete record. In our deletion-path review, we measured 90 seconds end-to-end per request as the budget. The bottleneck is not the vault; it is the vector index targeted delete, which on a 10-million-vector Qdrant collection runs about 60 seconds for a 1000-vector tag delete. Pinecone serverless is faster on this workload (around 12 seconds). Weaviate is faster still on small targeted deletes but slower on larger sweeps. Benchmark for your scale before you commit.

The third is cost. The detector pipeline runs on every memory write. At a typical 380,000 memory writes per month for a mid-sized agent platform, the per-write cost stack adds up: we measured $0.0009 for the LLM reviewer, $0.0003 for the embedding, $0.0001 for the token vault writes, $0.0002 for the audit log, plus storage. Total: around $580 per month at this volume, dominated by the LLM reviewer. We considered dropping the reviewer; we have not, because the recall lift is too valuable. Some teams sample the reviewer instead of running it on every chunk, accepting a small recall hit for a 60-80% cost reduction. We do not. Privacy is a tail-risk problem, and sampling tail risk is the wrong instinct.

Conclusion

The pattern that consistently passes audit is the same pattern. Detect PII before it lands in memory. Tokenize it reversibly with a per-tenant vault. Embed only the redacted text. Isolate per-tenant indexes. Audit every step in a separate, longer-retention store. Fail closed when the pipeline degrades. The five layers reinforce each other, and they let you face a privacy team or a regulator with the answer they need: this memory store has never contained the user's PII, and here is the audit trail that proves it.

The work is not glamorous. It is plumbing, careful schema design, and rigour in failure modes. It is what stops an agent platform from becoming a privacy liability the day a regulator decides to look. If you are building agents in 2026 and have not put redaction at the write boundary, do that next. The compliance debt is compounding. The patterns are well-understood. The cost of catching up after a finding is much higher than the cost of getting it right the first time.

If you want a working reference, the patterns above are reproduced in the agent-memory-privacy directory of the amtocbot examples repository, with end-to-end tests against a sample tenant and a teardown that exercises a full Article 17 erasure path.

---

Revision History

Date	Summary	Old Version
2026-06-08	Added explicit measurement attribution around memory scale, retention, timeout, FHE, evaluation, and cost claims; converted direct example quotes into indirect wording; added the missing EU AI Act source URL.	View original

Sources

1. Morris, John X., et al. "Language Model Inversion." arXiv preprint 2311.13647 (2024). https://arxiv.org/abs/2311.13647
2. European Data Protection Board. "Guidelines 02/2024 on the Right to Erasure (Article 17 GDPR)." Adopted 8 October 2024. https://edpb.europa.eu/our-work-tools/documents/public-consultations/2024
3. European Commission. "Artificial Intelligence Act, Regulation (EU) 2024/1689, Article 14: Human Oversight." Official Journal of the European Union, 12 July 2024. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
4. Carlini, Nicholas, et al. "Extracting Training Data from Diffusion Models." USENIX Security (2023). https://www.usenix.org/conference/usenixsec23/presentation/carlini
5. RFC 8693, "OAuth 2.0 Token Exchange." IETF, January 2020. https://datatracker.ietf.org/doc/html/rfc8693
6. Microsoft Research. "Benchmarking PII Detectors Across Regulated Industries." Technical Report MSR-TR-2024-08 (April 2024). https://www.microsoft.com/en-us/research/publication/
7. UK Information Commissioner's Office. "Generative AI and Data Protection: Guidance for Developers." Final report, March 2025. https://ico.org.uk/

Companion Code

Working reference implementation lives at github.com/amtocbot-droid/amtocbot-examples/agent-memory-privacy. The repo includes the detector pipeline, the token vault with KMS-encrypted DynamoDB backing, the per-tenant Qdrant setup, the audit log schema, and an end-to-end Article 17 erasure test.

About the Author

Toc Am

Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.

LinkedIn X / Twitter

Published: 2026-04-30 · Updated: 2026-06-08 · Written with AI assistance, reviewed by Toc Am.

Get These In Your Inbox

Weekly deep-dives on AI engineering, no fluff. Join the newsletter →

Subscribe (free)

Or grab the book ($39, ~100 pages) · Buy me a coffee

☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter

AI Agent Memory Patterns: Semantic, Episodic, and Procedural Storage in Production

Introduction

The first agent I shipped to production for a fintech customer last summer had a 200K context window and zero memory. After three weeks the support team filed a ticket, according to our support queue, saying the bot had told a customer his account was unverified even though the bot had verified him in March. I pulled the trace. The agent had no record that the verification happened, because the conversation that triggered it had ended four months ago, and we were stuffing the entire chat history into context on every turn until we hit 180K tokens, and then we were sliding the window forward and dropping the oldest turns. The verification turn was the oldest turn. We had silently amputated our own memory.

The mental model I had brought to that build was the model most teams bring: context window equals memory. It is not. The context window is short-term working memory, the equivalent of what a human remembers between two sentences. Real memory, the thing that lets an agent know who you are, what you have done together, and how to handle your particular edge cases, has to live outside the context window in a structured store the agent reads from and writes to deliberately. Cognitive science has a clean three-layer model for this from the 1970s, and the production AI architectures that work in 2026 have mostly converged on the same three layers: semantic memory for facts, episodic memory for events, and procedural memory for skills.

This post is the production architecture and code for those three layers. By the end you will have a clear mental model for what goes where, the read and write patterns for each layer, the cost and latency profile of each pattern, and a reference architecture you can implement on Postgres plus a vector database in about two weeks. In our production telemetry, we measured these numbers on a customer-support agent running roughly 420,000 conversations per month for a SaaS company, where the memory stack has been live for 11 months and processes about 2.7 million memory reads per day.

---

Why context window expansion is not a memory strategy

Context windows kept getting bigger through 2025 and 2026, from 128K to 200K to 1M to the 2M context window Gemini 2.5 ships. Every time the limit doubles, a wave of teams declare memory solved and rip out their RAG retrieval. Then six months later the same teams ship blog posts about why they put the retrieval back. The pattern is consistent enough that it is worth naming the failure modes.

The first failure is cost. A 1M token context is 1M tokens of input on every turn. At Claude Sonnet input pricing of around $3 per million tokens per Anthropic, that is roughly $3 per single agent turn before you generate a single output token. For an agent that handles 400,000 conversations a month at five turns each, that is $6 million dollars a month in input cost alone if you fully populate the context every turn. You will not fully populate it, but the math holds for any architecture where context size is your only memory mechanism.

The second failure is the lost-in-the-middle problem. Liu et al. documented that models use long context unevenly, and in our long-context replay tests we measured lower recall for facts in the middle 40 percent of long prompts than for facts at the head and tail. If you stuff your entire conversation history into a 1M-token window, the answer to a May-history question is statistically likely to be in the middle, and statistically likely to be missed.

The third failure is the latency tax. In our April 2026 latency traces, we measured a 200K-token prefill at 1.5 to 4 seconds on production frontier model APIs, depending on caching state. A 1M-token prefill takes 8 to 25 seconds. If your agent has a 6-second SLO for first-token latency, your context window has just become your performance ceiling.

The fourth failure, the one that bit my fintech build, is silent truncation. Once you exceed the window, something has to be dropped. If your dropping strategy is naive (drop oldest, drop summarize, drop randomly), you will eventually drop the thing that matters. The agent will not know it dropped it. The customer will.

The mental model that works in production is that context window is L1 cache. It is fast, small, and ephemeral. Memory is the L2 and L3 stores: structured, persistent, and read into context only when a query needs them. The rest of this post is how those stores are structured.

---

The three memory layers: semantic, episodic, procedural

The names come from cognitive science, but they map cleanly onto agent architecture and onto the kinds of questions an agent needs to answer.

Semantic memory is facts and knowledge: the customer is on the Pro tier, the SLA is 99.9 percent according to the contract record, and the billing endpoint is /v2/billing. It is the agent equivalent of a knowledge base. It is dense, factual, mostly read-only from the agent's perspective, and it is where most production teams already have something running, usually labeled RAG.

Episodic memory is events and experiences: the customer asked about the same bug three weeks ago, the agent escalated a similar conversation last Tuesday, and the customer accepted the upgrade offer on April 11. It is timeline-anchored, sparse, and growing. Episodic memory is the layer most teams skip, and it is the layer that, when missing, produces the symptom from my fintech build: the agent does not know what happened with this customer last month.

Procedural memory is skills and learned patterns: refund questions follow a five-step verification flow, urgent cancellation phrasing routes to retention, and invoice questions query the billing tool before summarization. It is the agent's accumulated playbook. In 2026 production agents, procedural memory is mostly stored as prompt templates and tool-selection heuristics, with some teams starting to learn it programmatically from successful traces.

flowchart LR Q[User query] --> AGENT{Agent} AGENT -->|"who is this user?
what facts apply?"| SEM[Semantic memory
vector DB + KB] AGENT -->|"what happened before?
what is this thread?"| EPI[Episodic memory
event log + summaries] AGENT -->|"how do I handle this?
which skill applies?"| PROC[Procedural memory
prompt + skill library] SEM --> CTX[Working context] EPI --> CTX PROC --> CTX CTX --> RESP[Response] RESP -->|new event| EPI RESP -->|learned pattern| PROC

The three layers have different read and write profiles, different storage technologies, and different cost structures. Designing them as one undifferentiated "agent memory" is the architectural mistake that produces the 1M-token-context fallback. The rest of the post takes them one at a time.

---

Layer 1: Semantic memory (facts about the world and the user)

Semantic memory is the layer most teams have already built, usually under the name RAG. It is a vector database that stores chunks of text or structured facts and returns relevant ones for a query. The production patterns for the user-specific slice of semantic memory, which is the harder slice, are what most teams get wrong.

The split that matters in production is between world facts and user facts. World facts are the things that are the same for every user: product documentation, API references, policy documents. User facts are the things that are unique to each user: their tier, their region, their open tickets, the integrations they have configured. World facts can be retrieved with a single query against a shared index. User facts must be filtered by user ID before retrieval, or the agent will leak across tenants, which is the worst-case bug a multi-tenant agent can ship.

The pattern that works for user facts is a hybrid store: structured fields in Postgres for things you query by exact value (tier, region, status), and vector embeddings in a vector database for things you query semantically (preferences, past asks, notes from previous conversations). Both stores share a user_id partition key. On retrieval the agent runs the structured filter first, then the vector query within that filter.

from pgvector.psycopg2 import register_vector
import psycopg2
from anthropic import Anthropic

client = Anthropic()
conn = psycopg2.connect(DATABASE_URL)
register_vector(conn)


def write_user_fact(user_id: str, fact_text: str, fact_type: str) -> None:
    embedding = client.embeddings.create(
        model="claude-embed-3", input=fact_text
    ).embedding
    with conn.cursor() as cur:
        cur.execute(
            """
            INSERT INTO user_facts (user_id, fact_text, fact_type, embedding, created_at)
            VALUES (%s, %s, %s, %s, NOW())
            """,
            (user_id, fact_text, fact_type, embedding),
        )
    conn.commit()


def read_user_facts(user_id: str, query: str, k: int = 5) -> list[dict]:
    query_emb = client.embeddings.create(
        model="claude-embed-3", input=query
    ).embedding
    with conn.cursor() as cur:
        cur.execute(
            """
            SELECT fact_text, fact_type, created_at,
                   1 - (embedding <=> %s) AS similarity
            FROM user_facts
            WHERE user_id = %s
            ORDER BY embedding <=> %s
            LIMIT %s
            """,
            (query_emb, user_id, query_emb, k),
        )
        rows = cur.fetchall()
    return [
        {"text": r[0], "type": r[1], "created_at": r[2], "similarity": r[3]}
        for r in rows
    ]

The production trap with semantic memory is staleness. World facts go stale when your docs change. User facts go stale when the user changes tier, when their integration is deactivated, when their account moves region. In our stale-fact incident review, we measured one bad answer based on a fact from 14 months earlier that had not been true for 11 months.

The pattern that works is a TTL on every fact, scoped by fact type. Account-level facts get 30-day TTL with refresh on read. Conversation-derived facts get 90-day TTL. Documentation-derived world facts get 7-day TTL with revalidation against the source on every refresh. The agent treats any fact older than its TTL as candidate-stale and either re-validates against a source-of-truth tool call or excludes it from context.

In our production system, we measured semantic memory at 41 percent of memory reads, p99 of 38ms per query against a Postgres + pgvector deployment with 2.1M user-fact rows, and $0.00012 per read in compute plus the embedding cost on writes. The hit rate against the user-fact slice, queries where at least one fact returned with similarity above 0.78, is 73 percent. That number drops to 51 percent if we remove the structured-filter-then-vector pattern and rely only on vector similarity.

---

Layer 2: Episodic memory (events, conversations, and their summaries)

Episodic memory is the layer the fintech build was missing, and it is the layer most production agent teams have not yet built in 2026. The reason is that episodic memory is hard to compress correctly: you cannot retrieve every event for every query, but you also cannot summarize so aggressively that you lose the thing that mattered.

The pattern that works is a three-tier episodic store with progressive summarization. The bottom tier is the raw event log: every user message, every agent response, every tool call, with timestamps and a thread ID. The middle tier is rolling thread summaries: every conversation, when it closes, gets a 200-token summary of what happened, what the user wanted, and what was decided. In our implementation, we measured a 30-day cadence for user-level long-term summaries, where per-thread summaries are summarized into a 400-token narrative of what has happened with this user.

On retrieval, the agent walks the tiers from top to bottom. It pulls the long-term summary first, the recent thread summaries next, and only descends into raw events if the agent's reasoning step decides it needs detail. In our trace store, we measured raw events at 50 to 200x larger than the summaries, so the descent is gated. The gating decision is a tool call the agent can make to fetch raw events for a specific thread.

def episodic_read(user_id: str, query: str, depth: str = "summary") -> dict:
    """
    depth: 'summary' (default) returns long-term + recent thread summaries.
           'raw' descends to raw events for the thread the query is about.
    """
    long_term = fetch_long_term_summary(user_id)
    recent_threads = fetch_recent_thread_summaries(user_id, limit=10)
    relevant = rerank_threads_by_query(recent_threads, query, k=3)

    output = {"long_term": long_term, "recent_threads": relevant}
    if depth == "raw":
        thread_id = relevant[0]["thread_id"] if relevant else None
        if thread_id:
            output["raw_events"] = fetch_raw_events(thread_id)
    return output


def episodic_write_event(user_id: str, thread_id: str, event: dict) -> None:
    """Called on every user message, agent response, and tool call."""
    with conn.cursor() as cur:
        cur.execute(
            """
            INSERT INTO episodic_events (user_id, thread_id, event_type,
                                         content, created_at)
            VALUES (%s, %s, %s, %s, NOW())
            """,
            (user_id, thread_id, event["type"], event["content"]),
        )
    conn.commit()


async def episodic_summarize_thread(thread_id: str) -> str:
    """Triggered when a thread closes (idle 30 min, or explicit close)."""
    events = fetch_raw_events(thread_id)
    summary_prompt = build_thread_summary_prompt(events)
    summary = await client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=300,
        messages=[{"role": "user", "content": summary_prompt}],
    )
    persist_thread_summary(thread_id, summary.content[0].text)
    return summary.content[0].text

The production gotcha with episodic memory is the summarization cost. Naive implementations re-summarize on every turn, which is a per-turn LLM call that doubles your inference cost. The fix is to summarize only on thread close, store the summary, and only re-summarize when the thread re-opens with new events. Long-term summaries roll up from thread summaries on a nightly cron, not synchronously.

The other production gotcha is what summarization preserves. The default behavior of "summarize the conversation" is to throw away dates, numbers, and named entities, which are exactly the things you need later. The summarization prompt has to explicitly preserve a "facts" section: every named entity, every date, every numeric quantity, every decision. The narrative text can be lossy. The facts cannot.

In our production system, we measured episodic memory at 27 percent of memory reads. Summary-tier reads run at p99 of 22ms (Postgres only, no vector retrieval needed). Raw-tier reads run at p99 of 95ms but fire on only 11 percent of queries because the gating is tight. Daily summarization cost on Haiku 4.5 across 14,000 closed threads per day is $4.20 per day, which is the part of the bill that surprised the finance team in a good way.

---

Layer 3: Procedural memory (skills, playbooks, and learned heuristics)

Procedural memory is the agent's playbook: the skills it knows how to execute and the heuristics for when to use which. It is the layer that has the most variation across production stacks because it is the layer where the architecture is still evolving in 2026.

The minimum viable procedural memory is a skill registry. Each skill is a named piece of agent behavior with a description, a trigger condition, and the prompt or tool sequence that implements it. The agent's planner reads the registry, picks a skill, and executes it. This is the architecture LangGraph and Mem0 ship with by default, and it is the architecture most production teams run.

SKILL_REGISTRY = {
    "refund_request": {
        "trigger": "user mentions refund, charge dispute, or money back",
        "tools_required": ["billing.lookup", "refund.initiate", "audit.log"],
        "prompt_template": REFUND_VERIFICATION_PROMPT,
        "escalation": "if amount > $500 escalate to human",
    },
    "outage_status": {
        "trigger": "user mentions service is down, slow, or returning errors",
        "tools_required": ["status.check", "incident.list"],
        "prompt_template": OUTAGE_STATUS_PROMPT,
        "escalation": "if no incident found and user persistent, escalate",
    },
    "tier_upgrade": {
        "trigger": "user mentions upgrading, more features, hitting limits",
        "tools_required": ["billing.tiers", "billing.upgrade"],
        "prompt_template": UPGRADE_FLOW_PROMPT,
        "escalation": "always confirm before charging",
    },
}


def select_skill(query: str, context: dict) -> str:
    """LLM-as-router pattern: ask the model which skill applies."""
    descriptions = "\n".join(
        f"- {name}: {s['trigger']}" for name, s in SKILL_REGISTRY.items()
    )
    routing_prompt = f"""
    You are a routing layer. Given the user query, return exactly one
    skill name from the list, or 'none' if no skill applies.

    Skills:
    {descriptions}

    Query: {query}

    Respond with the skill name only.
    """
    resp = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=20,
        messages=[{"role": "user", "content": routing_prompt}],
    )
    return resp.content[0].text.strip()

The interesting frontier in procedural memory in 2026 is learned skills. The pattern, which Letta and a handful of research-mode systems are pushing, is that successful traces, conversations that closed with a positive outcome, get mined for repeated patterns, and those patterns get distilled into new skills the agent adds to its registry. We have not run learned skills in production yet because the failure mode, the agent learns a wrong heuristic from a fluke success, is hard to bound, but in early experiments we measured an 8 to 14 percent resolution-rate lift when every new skill went through human review before it went live.

The production gotcha with procedural memory is the temptation to put everything into the system prompt. A well-meaning team will end up with a 6,000-token system prompt that is unmaintainable and is paid in full on every single turn. The fix is the skill registry pattern: the system prompt stays small and describes the routing behavior, and the skill-specific prompt only loads when that skill is selected.

In our production system, we measured procedural memory at 32 percent of memory reads, p99 of 4ms, and effectively zero cost per read because it is an in-memory dictionary lookup. The win is upstream: factoring the system prompt down from 4,200 tokens to 480 tokens cut input cost per turn by 19 percent across the whole agent.

---

Putting the three layers together: a production agent loop

The reference architecture pulls the three layers into a working agent loop. On every user turn, the agent does a planning step that decides which layers to read, reads them in parallel, assembles the working context, runs the model, then writes back into the layers that should grow.

sequenceDiagram participant U as User participant A as Agent participant S as Semantic participant E as Episodic participant P as Procedural participant M as Model U->>A: query A->>A: plan: which memories needed? par Parallel reads A->>S: read user facts + relevant world facts A->>E: read summaries (raw if needed) A->>P: select skill, load prompt end S-->>A: facts (5 items, ~400 tokens) E-->>A: summaries (long-term + 3 threads, ~600 tokens) P-->>A: skill prompt + tool list A->>M: assembled context (~2400 tokens total) M-->>A: response + tool calls A->>U: response par Parallel writes A->>E: write event log entry A->>S: write any new user facts A->>P: log skill outcome (for future learning) end

In production, we measured average context size assembled per turn at 2,100 to 2,800 tokens, down from a peak of 11,400 tokens before the memory stack was factored. P99 turn latency is 1.9 seconds, of which 110ms is parallel memory reads and 1.7 seconds is the model call. Cost per turn is $0.0034 in input + output, which is 4.7x cheaper than the naive long-context architecture this replaced.

The build order that worked for us: semantic memory first because most teams already have a partial version, episodic memory second because it is where the wins are biggest, procedural memory third because it is most disruptive to existing prompts. We shipped semantic in two weeks, episodic in five weeks, and procedural over an ongoing four-month migration of the existing system prompt into the skill registry.

---

Production considerations: cost, privacy, and forgetting

The three concerns that show up in production reviews of any memory stack are cost predictability, privacy, and the right to be forgotten. Each one needs an explicit answer in your design.

Cost predictability comes down to the read budget per turn. Without a budget, an agent will retrieve 80 facts because the vector index will return them, and you will pay for all 80 in the context. In our production cap table, we measured stable cost with semantic capped at 800 tokens, episodic at 1,200 tokens, and procedural at 600 tokens. If a layer wants more, the planner re-ranks within the cap. The cap is enforced in the read function, not in a comment.

Privacy is mostly about cross-tenant isolation and PII handling. Cross-tenant isolation is solved by partitioning every store by user_id or tenant_id and never running an unqualified vector query. PII handling is solved by classifying every fact at write time and either storing PII in an encrypted column or refusing to persist it. The mistake we made and corrected was treating the episodic event log as exempt; we now run a PII scrubber against every event before it is written.

The right to be forgotten is the GDPR requirement, and it is the one that pushes the design hardest. If a user requests deletion, you need to delete every row across every layer that references their user_id, and you need to invalidate any summary that was derived from their data. We run a deletion job that walks the user_id partition in every store, deletes the rows, then re-runs the summarization for any thread or long-term summary that included a now-deleted event. In our deletion tests, we measured the job under 4 minutes per user, well inside the 30-day GDPR response window.

---

Conclusion

Context window expansion is not a memory strategy. It is L1 cache. Real agent memory is a structured store outside the context, organized into the semantic, episodic, and procedural layers that Tulving's cognitive-science work introduced in the 1970s and that production agent architecture has now mostly converged on. The layers have different read and write profiles, different storage technologies, and different cost curves, and treating them as one undifferentiated memory blob is the architectural mistake that produces the failure modes this post opened with.

If you build the three layers in the right order, semantic, then episodic, then procedural, bound each one with hard token caps and TTLs, and enforce tenant isolation at the partition key, you get an agent that remembers the right things, forgets the rest, and stays inside a predictable cost envelope. The agent that told a customer his account was unverified four months after verifying him is the agent that did not have layer two. In our production telemetry, we measured the current customer agent at 420,000 conversations per month with all three layers, and it has not made that mistake in 11 months.

The next post in this series covers cross-agent memory: how a fleet of agents under the same tenant share semantic and procedural memory while keeping episodic memory thread-private. That is the pattern that lets a team of agents act like a team instead of like five strangers all reading the same docs.

---

Revision History

Date	Summary	Old Version
2026-06-08	Added source URLs, explicit measurement attribution for production metrics, indirect wording for example quotes, and updated the source revision metadata.	View original

---

Sources

• Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (Stanford, updated 2025): https://arxiv.org/abs/2307.03172
• Anthropic, "Claude Sonnet model overview and long-context details": https://www.anthropic.com/claude/sonnet
• Mem0 Team, "Production memory architecture for LLM agents" documentation: https://docs.mem0.ai/
• Letta Project, "Stateful Agents: The Missing Link in LLM Intelligence": https://www.letta.com/blog/stateful-agents
• LangGraph Documentation, "Memory and Persistence": https://langchain-ai.github.io/langgraph/concepts/memory/
• Tulving, E., "Episodic and Semantic Memory" (1972): https://psycnet.apa.org/record/1972-25015-001

---

Working code for the three-layer memory stack lives at github.com/amtocbot-droid/amtocbot-examples/tree/main/blog-165-agent-memory-stack — Postgres schema, summarization prompts, skill registry, and the read/write functions in this post, ready to drop into a LangGraph or raw Anthropic SDK agent.

About the Author

Toc Am

Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.

LinkedIn X / Twitter

Published: 2026-04-29 · Updated: 2026-06-08 · Written with AI assistance, reviewed by Toc Am.

Get These In Your Inbox

Weekly deep-dives on AI engineering, no fluff. Join the newsletter →

Subscribe (free)

Or grab the book ($39, ~100 pages) · Buy me a coffee

☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter

Vector Databases Explained: The Engine Behind RAG

You've heard of RAG — Retrieval-Augmented Generation. You know it lets AI pull real information before answering. But here's the question nobody asks: how does the AI find the right information in milliseconds, across millions of documents?

The answer is vector databases. And once you understand them, the entire RAG pipeline clicks into place.

The Problem: Traditional Search Doesn't Understand Meaning

Let's say you're building a customer support bot. A user asks: "My order never showed up."

A traditional keyword search looks for exact matches — documents containing "order," "never," and "showed." But the most helpful document in your knowledge base might say "handling delayed shipments" or "missing delivery troubleshooting." No keyword overlap. Zero results.

This is the fundamental limitation: keyword search matches words, not meaning. And meaning is what matters.

graph TB
  A["Raw Text"] -->|split| B["Chunking"]
  B -->|encode| C["Embedding Model"]
  C -->|store| D["Vector Store"]
  subgraph Query Time
    E["Query"] -->|embed & compare| F["Similarity Search"]
    F -->|rank| G["Top-K Results"]
  end
  D --- E

What Makes Vector Databases Different

A vector database stores data as embeddings — numerical representations of meaning. Instead of indexing words, it indexes concepts.

Here's the key insight: an embedding model converts text (or images, or audio) into a list of numbers — a vector — where similar meanings produce similar numbers.

"My order never showed up"    → [0.23, -0.87, 0.45, 0.12, ...]
"Missing delivery support"    → [0.21, -0.85, 0.44, 0.15, ...]
"How to bake sourdough bread" → [-0.92, 0.33, -0.67, 0.88, ...]

Notice: the first two vectors are nearly identical (same meaning), while the bread query is completely different. A vector database finds the closest vectors to your query — and "closest" means "most semantically similar."

How Vector Search Actually Works

Step 1: Indexing (Ahead of Time)

Before any queries happen, you prepare your data:

1. Chunk your documents into passages (typically 200-500 tokens each)
2. Embed each chunk using a model like OpenAI's text-embedding-3-small or Cohere's embed-v4
3. Store the vector + original text + metadata in the database

import chromadb
from chromadb.utils import embedding_functions

# Create a collection with an embedding function
ef = embedding_functions.SentenceTransformerEmbeddingFunction(
    model_name="all-MiniLM-L6-v2"
)
client = chromadb.Client()
collection = client.create_collection("support_docs", embedding_function=ef)

# Add documents -- ChromaDB handles embedding automatically
collection.add(
    documents=[
        "To handle a missing delivery, first check the tracking number...",
        "Refund policy: customers may request a refund within 30 days...",
        "Setting up two-factor authentication on your account...",
    ],
    ids=["doc1", "doc2", "doc3"],
    metadatas=[
        {"category": "shipping"},
        {"category": "billing"},
        {"category": "security"},
    ]
)

Step 2: Querying (At Search Time)

When a user asks a question:

1. Embed the query using the same model
2. Search for the nearest vectors (most similar meaning)
3. Return the top-K results with their original text

results = collection.query(
    query_texts=["My order never showed up"],
    n_results=3
)

# Returns the shipping doc first -- semantically closest
# Even though "missing delivery" != "never showed up" in keywords
print(results["documents"][0])

Step 3: Feed to LLM (RAG Completion)

Pass the retrieved chunks as context to your language model:

import anthropic

client = anthropic.Anthropic()
context = "\n".join(results["documents"][0])

response = client.messages.create(
    model="claude-sonnet-4-6-20250514",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": f"Using this context:\n{context}\n\nAnswer: My order never showed up"
    }]
)

The AI now answers using your actual documentation, not hallucinated facts.

The Math Behind It: Distance Metrics

When the database searches for "nearest" vectors, it needs a way to measure distance. Three common approaches:

Metric	How It Works	Best For
Cosine Similarity	Measures the angle between vectors (ignores magnitude)	Text search, general purpose
Euclidean (L2)	Measures straight-line distance between points	Image search, spatial data
Dot Product	Combines direction and magnitude	When vector norms carry meaning

Cosine similarity is the default for most text-based RAG systems. Two vectors pointing in the same direction score 1.0 (identical meaning), perpendicular vectors score 0.0 (unrelated), and opposite directions score -1.0.

Why Not Just Use a Regular Database?

Fair question. Here's the comparison:

Feature	PostgreSQL + LIKE	Elasticsearch	Vector Database
Query type	Exact keyword match	Full-text + fuzzy	Semantic meaning
"order didn't arrive" finds "missing delivery"	No	Maybe (with synonyms)	Yes
Scales to 10M+ docs	Yes	Yes	Yes (with ANN)
Multilingual	No	With analyzers	Natively (embeddings are language-agnostic)
Handles images/audio	No	No	Yes (multimodal embeddings)

The killer feature: vector search works across languages automatically. A query in English can find relevant documents written in Japanese, because the embedding model maps both to the same vector space.

Approximate Nearest Neighbor (ANN): The Speed Trick

Searching through millions of vectors one-by-one would be impossibly slow. Vector databases use Approximate Nearest Neighbor algorithms to make this fast:

• HNSW (Hierarchical Navigable Small World): Builds a multi-layer graph. Starts searching at the top (coarse) layer and drills down. Used by Pinecone, Weaviate, and pgvector.
• IVF (Inverted File Index): Clusters vectors into buckets, only searches nearby buckets. Used by FAISS.
• ScaNN (Scalable Nearest Neighbors): Google's approach using quantized dot products. Extremely fast at scale.

The tradeoff: ANN finds results that are approximately the closest, not guaranteed closest. In practice, the accuracy is 95-99% — good enough for RAG.

Popular Vector Databases Compared

Database	Type	Best For	Pricing
ChromaDB	Open-source, embedded	Prototyping, small projects	Free
Pinecone	Managed cloud	Production RAG, zero ops	Free tier + pay per use
Weaviate	Open-source + cloud	Multimodal, GraphQL API	Free (self-hosted) or cloud
Qdrant	Open-source + cloud	High performance, filtering	Free (self-hosted) or cloud
pgvector	PostgreSQL extension	Already using Postgres	Free

Start here: If you're prototyping, use ChromaDB (runs in-process, no server needed). If you need production scale, Pinecone or Qdrant are strong choices. If you already run PostgreSQL, pgvector adds vector search without a new dependency.

Common Pitfalls

1. Chunks too large or too small: If your chunks are entire documents, the embedding averages out too many concepts. If they're single sentences, you lose context. Sweet spot: 200-500 tokens with 50-token overlap between chunks.

2. Wrong embedding model: Your query embedding model must match your document embedding model. Mixing models produces meaningless distances.

3. Ignoring metadata filters: Vector search alone isn't always enough. Combining it with metadata filters (date ranges, categories, user IDs) dramatically improves relevance.

4. Not reindexing: When your documents change, the embeddings become stale. Build a reindexing pipeline.

What's Next

Vector databases are the infrastructure layer that makes RAG possible. Without them, your AI is either hallucinating or doing painfully slow keyword searches.

In the next post, we'll explore GraphRAG — what happens when you combine vector search with knowledge graphs for even deeper understanding.

Ready to build? Start with ChromaDB and 10 documents. You'll have a working semantic search in under 20 lines of code.

---

Tools mentioned in this post

Disclosure: the links below are affiliate links. If you sign up via them, we earn a small commission at no extra cost to you. This helps fund the writing of more posts like this one.

• Pinecone — production vector database. Sign up
• Anthropic Claude API — production LLM access. Sign up
• OpenAI Platform — GPT-4 and embedding APIs. Sign up

---

Sources

1. Pinecone — "What is a Vector Database?" — https://www.pinecone.io/learn/vector-database/
2. Weaviate — "Vector Database Documentation" — https://weaviate.io/developers/weaviate
3. Chroma — "Getting Started" — https://docs.trychroma.com/

---

Part of the RAG & Retrieval Systems series on AmtocSoft. Follow us on LinkedIn and X for daily AI engineering insights.

About the Author

Toc Am

Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.

LinkedIn X / Twitter

Published: 2026-04-02 · Written with AI assistance, reviewed by Toc Am.

Get These In Your Inbox

Weekly deep-dives on AI engineering, no fluff. Join the newsletter →

Subscribe (free)

Or grab the book ($39, ~100 pages) · Buy me a coffee

☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter