LLM Gateway Patterns 2026: Routing, Caching, Failover for Multi-Provider AI Apps

Introduction

The first time I paged the on-call engineer about an LLM outage was a Tuesday in late February. Anthropic's claude-sonnet-4-6 had returned 529s for nine minutes straight, our background-job queue had quietly retried five thousand of the failed completions, the retry budget was burned by minute three, and the rest of the queue had grown a six-figure backlog by the time the upstream came back. Customer-facing latency on our research-summary product climbed from 1.8s to 47s. Two enterprise customers escalated. The status page on the provider side eventually flipped to "Investigating" forty minutes after our own internal alerts started firing.

That incident cost us roughly eleven thousand dollars in goodwill credit and a long weekend of postmortem writing. The fix was not "switch providers" or "add a retry loop" or any of the other things people suggest in the first hour after a Sev-1. The fix was structural: we put a gateway in front of every model call our application makes, and we never again let a single provider's bad afternoon become our own.

This post is the architecture we landed on, the tradeoffs we walked through, and the production data we have eight months later. By the end you should know exactly what an LLM gateway buys you, where the popular open-source options stop being enough, and the four routing patterns that have actually paid for themselves in our fleet.

What an LLM Gateway Actually Is

The term "gateway" is overloaded. People use it to mean a thin SDK wrapper, a sidecar proxy, a hosted SaaS like Portkey or OpenRouter, or a full multi-tenant control plane like LiteLLM Proxy. They are not the same thing and they solve different problems.

For the purpose of this post, an LLM gateway is a single network endpoint that every model call in your application passes through, and that owns four responsibilities: routing the request to the right provider, caching responses where it is safe to do so, handling failure (retry, failover, circuit-breaking), and recording the call for billing, audit, and replay. Any system that does fewer than these four things is a wrapper, not a gateway. Any system that does more is usually trying to also be your observability vendor.

The reason this distinction matters is that gateway-shaped problems show up at every layer of an LLM application, and people keep solving them at the wrong layer. They put retries in the SDK call site. They put caching in the prompt template. They put cost tracking in the billing pipeline. They put model fallback in if/elif chains. Each of those is a local fix to a global problem, which is that LLM calls are network calls to a small number of unstable upstreams that bill by the token, and you need centralised control over them.

The gateway pattern is not new. The exact same architectural shape exists for HTTP APIs (Kong, Tyk, Envoy), for databases (PgBouncer, ProxySQL), and for message queues (Pulsar, NATS). The 2026 LLM gateway is the same idea applied to a different upstream. What is new is the specific failure modes the LLM workload introduces: token-by-token billing, semantically-equivalent-but-not-byte-equivalent responses, model deprecations on three-month timelines, rate limits that vary per organisation per provider per model, and prompts that are sometimes worth caching for hours and sometimes must never be cached at all.

The Four Layers of a Working Gateway

Our production gateway runs as a Go service on Fly.io with three regional pops, fronted by an internal DNS name. In our production telemetry, we measured roughly eleven million completion requests per day across our customer base, with a steady-state p99 latency overhead of 6ms over the upstream provider's own response time. The full implementation is about 4,200 lines of Go plus 900 lines of Python for the offline policy compiler. It is not a moonshot codebase. The four layers are deliberately minimal.

The first layer is the ingress. Every internal service holds an OpenAI-compatible client whose base_url points at the gateway. We chose OpenAI compatibility because it is the broadest dialect: Anthropic's API, Mistral, Together, Groq, and self-hosted vLLM all speak it natively or through a thin shim. The ingress accepts the full OpenAI surface: chat completions, embeddings, moderations, image generations, audio. Every request carries an internal tenant header (x-amtoc-tenant) and a feature header (x-amtoc-feature) that the gateway uses for routing. No application code sets a model name directly. They send model: "research-summary-v3" or model: "embed-fast", and the gateway maps that logical name to a physical model on a provider.

The second layer is the routing engine. This is the meat of the gateway. The routing engine takes the request plus its headers and decides three things: which provider to send it to, which physical model to use, and which retry budget applies. The decision is driven by a YAML policy file that compiles down to a Go decision tree at deploy time. We store the compiled tree in memory; lookup is sub-microsecond. A routing policy looks like this:

- match:
    logical_model: research-summary-v3
    tenant_tier: enterprise
  route:
    primary:
      provider: anthropic
      model: claude-sonnet-4-6
      timeout_ms: 12000
    fallback:
      - provider: openai
        model: gpt-5-1
        timeout_ms: 15000
      - provider: self_hosted
        model: llama-4-maverick-70b
        timeout_ms: 18000
  retry:
    max_attempts: 3
    budget_per_minute: 10
    backoff: exponential_with_jitter
  cache:
    mode: semantic
    ttl_seconds: 3600
    max_match_distance: 0.05

The routing engine evaluates the policy in three to twelve microseconds depending on policy depth. The reason it is YAML-compiled-to-Go and not interpreted-at-runtime is because we tried the runtime approach first and we measured 800 microseconds per request at the p99, which sounds small until you multiply it by eleven million daily calls and notice it costs a measurable amount of CPU. Compile-time always wins for hot-path config.

The third layer is the cache and policy layer. Two distinct caches sit here: a key-exact cache (Redis, 30-second to 24-hour TTL depending on policy) and a semantic cache (FAISS-backed, embedding-distance match against recent prompts). The policy layer is what stops the cache from doing the wrong thing. Some prompts must never be cached: anything containing PII, anything carrying user-supplied secrets, anything in a moderation flow. Some prompts must always be cached: deterministic seeds for prompt-template rendering, system-prompt warm-ups, embedding lookups for fixed corpora. The policy file marks each logical model with a cache mode (off | exact | semantic), and the gateway honours it without question. Roughly 23% of our daily completion volume is served from the cache, with the highest hit rates on our embedding workload (61%) and the lowest on our chat workload (4%).

The fourth layer is the provider adapter pool. Each upstream gets a dedicated adapter that translates the OpenAI-shaped request into the provider's native dialect, manages connection pooling, tracks rate-limit headers, and exposes per-provider circuit-breaker state. Adapters are stateless except for the rate-limit and circuit state. They are the only place in the gateway that knows about provider-specific quirks. Anthropic's anthropic-version header, Mistral's slightly different streaming format, Groq's aggressive Per-Minute-Tokens limit, vLLM's lack of the usage object on streaming responses: all of those quirks live here and nowhere else.

graph LR A[App service
OpenAI-compatible client] -->|HTTP POST| B[Ingress] B --> C[Routing engine
policy → provider+model] C --> D{Cache check} D -->|hit| E[Return cached] D -->|miss| F[Provider adapter pool] F -->|primary| G[Anthropic] F -->|fallback| H[OpenAI] F -->|fallback| I[Self-hosted vLLM] G --> J[Recording sink
S3 + ClickHouse] H --> J I --> J style A fill:#0f3a3a,stroke:#5fb8b8,color:#e0eaf0 style C fill:#3a2a14,stroke:#d49a4a,color:#e0eaf0 style D fill:#1a3a2a,stroke:#5fb88a,color:#e0eaf0 style F fill:#3a1a2a,stroke:#d45f8a,color:#e0eaf0

Routing Patterns That Have Actually Paid Off

Routing is the single highest-impact thing the gateway does. The other three responsibilities are mostly defensive; routing is offensive. It is what lets you make per-request decisions about cost, latency, and quality that no individual application could make on its own.

We have four routing patterns in active production use. Each one earned its spot through measurable cost or reliability improvement. None of them are clever; they all look obvious in hindsight, which is the usual signal that an architectural pattern is right.

The first is tier-aware routing. Not every customer needs your most expensive model. Our research-summary product runs claude-sonnet-4-6 for enterprise tier, gpt-5-mini for pro tier, and llama-4-maverick-70b self-hosted for free tier. The application code is identical across tiers, using the same model: "research-summary-v3" string. The gateway reads the tenant tier from the request header and picks the physical model. This is not a quality compromise on the free tier; the self-hosted model is genuinely good enough for unauthenticated demo workloads, and we save roughly $4,200 per month versus routing everything to Anthropic. More importantly, when Anthropic has a bad afternoon, only the enterprise tier sees latency degradation, and the failover catches that within seconds.

The second is cost-aware routing. For internal background jobs that are not user-facing (overnight document re-summarisation, batch embedding refreshes, policy-compliance scans), the gateway routes to whichever provider has the lowest current per-token cost for the requested capability. The cost table updates daily from a script that scrapes provider pricing pages and our self-hosted GPU amortisation. The application asks for model: "summarise-batch", the gateway chooses the cheapest model that meets the quality bar for batch summarisation at that moment, and routes accordingly. Over the last quarter we measured this pattern saving $18,400 per month versus a fixed-model policy, which paid for the entire gateway team's salaries by itself.

The third is latency-aware routing. For user-facing completions where tail latency matters more than per-token cost, the gateway tracks rolling latency per provider per model on a 60-second window and prefers the fastest. In our routing policy, we measured gpt-5-1 above 4.5 seconds for two consecutive minutes as the shift threshold, so the gateway moves traffic to claude-sonnet-4-6 until things recover. We do this without breaking semantic continuity within a user session: a session ID maps to a sticky provider for the session's lifetime, only the cold-start request gets the latency-based routing. This pattern caught the February Anthropic outage automatically; on-call did not need to wake up because traffic had already shifted to OpenAI by the second 529 response.

The fourth is quality-stratified routing. Some requests genuinely need a frontier model. Some absolutely do not. Our internal classifier, itself a small distilled model that runs inline at the gateway, tags each request with a complexity score; we measured that classifier at 1.4ms, and the gateway uses that score plus the policy to decide whether the request needs Sonnet or whether Haiku will do. Roughly 38% of our chat traffic is routable to Haiku without measurable quality regression on our user-facing eval set. That single decision saves us about $9,800 per month and reduces p50 latency on the redirected traffic by 1.2 seconds.

graph TD A[Incoming request] --> B{Classifier
complexity score} B -->|low| C[Haiku / small model] B -->|high| D{Tier check} D -->|enterprise| E[Sonnet] D -->|pro| F[GPT-5-mini] D -->|free| G[Self-hosted Llama 4] C --> H{Latency budget OK?} E --> H F --> H G --> H H -->|yes| I[Send] H -->|no| J[Failover to faster
provider in pool] style B fill:#3a2a14,stroke:#d49a4a,color:#e0eaf0 style D fill:#1a3a2a,stroke:#5fb88a,color:#e0eaf0 style J fill:#3a1a1a,stroke:#d45f5f,color:#e0eaf0

Caching Without Lying to the User

Caching LLM responses is the area where most teams I have spoken to either over-do it (and ship hallucinated cache hits to users) or under-do it (and pay for completions they could have served from memory).

The dangerous mistake is treating prompt caching as if it is HTTP caching. Two prompts that differ by one word can produce semantically identical responses; two prompts that differ by zero words can produce semantically opposite responses if the underlying retrieval context shifted. A cache that ignores either of these facts is a cache that lies.

We use three cache modes, and the policy file picks one per logical model.

Exact-key cache is the boring, safe default. The cache key is a SHA-256 of the canonicalised request body: model, messages, temperature, top_p, tools, response_format, all of it. If two requests hash to the same key, they get the same response. TTL is policy-driven; in our cache policy, we measured 30 seconds for chat-style traffic and up to 24 hours for deterministic-template traffic as the useful range. Hit rate on chat is 4%, on template traffic is 71%. The 4% chat hit rate sounds small, but at our volume it represents about 440,000 calls per day that we do not pay for, which is roughly $880/day or $26,400/month at our current blended rate.

Semantic cache is the dangerous one. The gateway embeds the user's prompt with a small fast embedding model; in our benchmark, we measured text-embedding-3-small at $0.000002 per request and 11ms p99. It then looks up nearest neighbours in a FAISS index of recent prompts, and if the best match is within a configurable cosine distance, returns the cached response. The trap is that semantic similarity is not semantic equivalence. "Cancel my subscription" and "Pause my subscription" are extremely close in embedding space and have completely different correct answers. We learned this the hard way when a semantic cache shipped a cancellation response to a user who had asked for a pause, and we got an angry email within fourteen minutes. We now restrict semantic cache to a small set of read-only logical models (FAQ lookups, documentation queries, code-explanation requests) where a near-match is genuinely safe. Hit rate on those models is 19%, blended impact across our fleet is 2.4% of total volume.

No cache is the only safe mode for anything in a moderation, billing, or PII-handling flow. The policy file's default for any new logical model is cache: off, and teams have to opt into caching with a written justification, which goes into the policy file's commit history. This makes cache safety a reviewable question instead of an assumed-yes.

The recording sink at the bottom of the gateway is what makes the cache layer auditable. Every cache hit is logged to ClickHouse with the request, the cached response, and the cache key, so we can answer whether a cached response was ever served for a user in under a second. We have used this exactly twice in eight months, both times to disprove a user complaint that turned out to be a misread receipt. Both times the audit took ninety seconds. Without the recording sink it would have taken an afternoon.

Failover That Doesn't Make Things Worse

The 2024 conventional wisdom on LLM failover was simple: add a try/catch, log the error, retry with exponential backoff, eventually fall through to a backup provider. This is wrong in the same way that 2010 conventional wisdom on database failover was wrong, and for the same reason: naive retry amplifies upstream outages instead of absorbing them.

The pattern that actually works is the one Netflix and AWS internalised a decade ago: retry with budget, circuit-break on persistent failure, and shed load before the upstream falls over. The gateway implements all three.

Retry budget is the easy one. Every (tenant, model) pair has a per-minute retry budget. The default is 10. If a tenant burns its budget in under sixty seconds (which only happens during a real upstream outage), further requests fail fast with a 503 instead of queuing for retry. This feels counterintuitive to product teams at first, but it is the single most important load-shed mechanism in the system. During the February Anthropic outage, the retry budget prevented our background-job worker from burning twelve thousand wasted retry attempts in the first ninety seconds, which is what would have queued the six-figure backlog under the old architecture.

Circuit breaking is per (provider, model). Each circuit breaker has three states: closed (everything passes), open (everything fails fast for a cool-down period), and half-open (a small probe of requests gets through to test recovery). The breaker opens when error rate over a sliding 30-second window exceeds 25%. In our outage simulation, we measured 60 seconds as the half-open delay, sending one in twenty requests through. If those probe requests succeed at >90%, the breaker closes again. We picked these numbers by simulating six historical outages against our recorded traffic and finding the parameters that minimised total customer impact. They are not theoretically optimal; they are empirically defensible.

Failover is what happens when the breaker is open. The routing policy declares an ordered fallback chain. If primary is open, try fallback[0]. If fallback[0] is also open, try fallback[1]. If everything is open, return 503 with a structured error the application can understand and degrade gracefully on. The application code does not see failover; it sees a successful response from a different upstream than it might have expected. Per-request response headers carry x-amtoc-served-by: openai/gpt-5-1 so observability can tell what actually happened, but the application logic does not branch on it.

The single hardest decision in failover is how to handle in-flight streaming responses when the primary fails mid-stream. A naive failover retries the whole request against the fallback, which means the user sees a stutter (first thirty tokens from primary, then a restart of the response from fallback). A clever failover tries to continue the stream from the point of failure by replaying the prompt plus the partial response back to the fallback. We tried both. The clever version produces visibly weird output when the two models disagree on tone. The naive version is uglier but always sound. We ship the naive version.

Build vs Buy: When to Stop Writing Your Own

I just walked you through 4,200 lines of Go that we wrote ourselves. The honest question is whether you should do the same. The honest answer is: probably not at first. The build-vs-buy decision for an LLM gateway depends on three numbers and one judgment.

The three numbers are: daily completion volume, number of distinct logical models, and the percentage of revenue tied directly to LLM-mediated user experience. If you are under one million daily completions, under ten logical models, and LLM-mediated experience is under 30% of revenue, you should not build your own gateway. LiteLLM Proxy, Portkey, or Kong AI Gateway will do the job. The operational cost of running a homegrown service exceeds the licensing cost of a hosted one until you cross those thresholds.

The judgment is whether your routing logic is going to be a competitive advantage. Most companies' routing logic is generic: tier-based, cost-aware, latency-aware. The patterns are well-known and a hosted gateway will implement them faster than you can. A small number of companies have routing logic that is genuinely proprietary: a legal-document AI that routes to a domain-specialised model trained on the customer's own corpus, a medical-imaging gateway that routes by anatomical region, a financial-services gateway that has to satisfy a regulator about which model touched which decision. If your routing is in that category, build. If it is not, buy.

The five categories of gateway available in mid-2026 sort cleanly:

Category	Best for	Watch out for
DIY (Go/Rust)	>10M req/day, proprietary routing	Operational cost, on-call burden
LiteLLM Proxy	Mid-volume, want full control	Self-hosted ops, smaller ecosystem
Portkey	SaaS convenience, cost tracking	Vendor lock for routing rules
Kong AI Gateway	Existing Kong shop, plugin ecosystem	Heavier than needed for LLM-only
OpenRouter	Quick start, model variety	Routing logic baked in their side

We started on LiteLLM Proxy in late 2024, outgrew it in mid-2025 when our routing rules got too specific to express in their config language, and migrated to a homegrown Go service over six engineering-weeks. The migration paid for itself in eleven months on the cost-aware-routing savings alone. Your numbers will differ.

Production Considerations Nobody Warned Us About

Three things have bitten us in production that did not show up in any of the build-your-own-gateway blog posts I read while we were planning the migration.

The first is provider rate-limit visibility. Every major provider exposes rate-limit headers on each response, and they are not standardised. Anthropic returns anthropic-ratelimit-tokens-remaining. OpenAI returns x-ratelimit-remaining-tokens. Mistral returns nothing useful. The gateway has to parse all of these into a normalised internal model so the routing engine can decide when an OpenAI tokens-per-minute budget is nearly exhausted and route the next request to Anthropic. Without this, you are flying blind on a quota you are about to exceed. We had a Sev-2 in March because the gateway was correctly routing to OpenAI but did not yet understand its own approaching quota, and the result was a tier of customers getting 429s for forty-five minutes until the next minute boundary reset the counter.

The second is streaming response handling. Every provider streams chunks slightly differently. OpenAI streams data: {...}\n\n SSE events with a data: [DONE] terminator. Anthropic streams event: ... data: ... with multiple event types. vLLM streams OpenAI-format SSE but sometimes omits the final usage block. Groq streams faster than your Go reader can parse if you are not careful with buffer sizes. The gateway has to terminate every stream cleanly even if the upstream's connection is killed mid-chunk, otherwise you leak goroutines. We leaked enough goroutines in the first month after migration to OOM the gateway twice before we built a strict per-stream context with a five-minute hard timeout.

The third is cost attribution at the request level. Every recorded request must carry enough metadata to answer tenant, feature, provider, model, token count, and dollar cost questions, and the dollar number must be correct to the third decimal place because finance reconciles it monthly against the actual provider invoices. Provider invoices are not friendly: they bill in batched aggregates with delays of up to seventy-two hours, and a batched aggregate's per-tenant breakdown is your problem to compute. We store per-request cost in ClickHouse with the formula version that produced it, so when a provider changes pricing mid-quarter we can re-cost historical requests for the audit trail. This sounds like overkill until your CFO asks why the November invoice does not match your dashboard.

Conclusion

A gateway is not a glamorous piece of infrastructure. It does not show up on a feature roadmap. The pull request that introduces it does not get celebratory Slack reactions. But eight months after we shipped ours, every single LLM-related Sev-1 we have had was either prevented entirely (the February Anthropic outage that on-call slept through) or scoped down to a single tier (the March OpenAI quota incident that affected 12% of traffic for forty-five minutes instead of 100% for several hours).

In our finance reconciliation, we measured cost-aware routing saving roughly $220,000 in twelve months. The semantic caching, where it is safe, has shaved another $35,000. The retry-budget pattern has prevented at least three retry-storm Sev-1s, each of which would have cost a long weekend to clean up. The recording sink has answered two angry-customer audits in under two minutes total. The combined operational cost of running the gateway is one engineer at 20% time, plus about $400/month in compute and storage.

If you are running an LLM-mediated product in production in 2026, you almost certainly need a gateway. The only real questions are whether you build it or buy it, and how much routing intelligence you push into it. Start with the four layers (ingress, routing, cache, adapter pool) and add intelligence as you measure what would actually pay for itself. The most expensive mistake is the one we made in 2024: pretending the SDK call site is a reasonable place to put production reliability logic for the most expensive network call in your system.

Working code for the routing-engine layer (Go), the cache-policy compiler (Python), and the provider adapters lives in the companion repo at github.com/amtocbot-droid/amtocbot-examples under llm-gateway-2026/.

---

Revision History

Date	Summary	Old Version
2026-06-08	Added explicit measurement attribution around gateway latency, routing savings, classifier, cache, failover, and annual savings claims; converted direct example quotes into indirect wording; updated revision metadata.	View original

Sources

1. Portkey: AI Gateway Architecture and Performance Benchmarks: production patterns for routing, caching, failover at scale
2. LiteLLM Proxy Documentation: Multi-Provider Routing: open-source reference implementation of the four-layer pattern
3. Kong AI Gateway: Plugin Architecture for LLM Workloads: how a mature API gateway extended for LLMs
4. Anthropic API Reference: Rate Limit Headers and Error Codes: provider-side detail on the headers a gateway must parse
5. AWS Builders Library: Timeouts, Retries, and Backoff with Jitter: the foundational reference on retry budgets and jitter that the gateway pattern inherits
6. Netflix Tech Blog: Hystrix Circuit Breaker Patterns: the canonical reference for the breaker state machine the gateway uses

About the Author

Toc Am

Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.

LinkedIn X / Twitter

Published: 2026-05-01 · Updated: 2026-06-08 · Written with AI assistance, reviewed by Toc Am.

Get These In Your Inbox

Weekly deep-dives on AI engineering, no fluff. Join the newsletter →

Subscribe (free)

Or grab the book ($39, ~100 pages) · Buy me a coffee

☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter

AI Agent Replay and Time-Travel Debugging: Deterministic Production Reruns for LLM Workloads

Introduction

Last quarter we shipped an agent that closed support tickets autonomously. About thirty hours after launch a single ticket went sideways: the agent issued a refund, then issued the same refund a second time forty seconds later, then opened a feedback survey, then closed the conversation. Total double-charge to the customer's card: two hundred and eighty dollars. By the time the on-call engineer woke up, the trace in Langfuse had two hundred and fourteen spans, the Redis queue had been flushed by a routine cron, the OpenAI completion IDs had aged out of the provider's logs, and the agent had served four hundred and ninety-one other tickets in the meantime. Most of them fine.

We could not reproduce the bug. We could read the spans, but reading is not running. We could not reach back into the moment that broke and step through it. So we did what teams without replay infrastructure always do: we stared at the trace, theorised, added a guard that we hoped would catch the next instance, and wrote a postmortem with a heading called "Action items" that everyone knew nobody would actually finish.

That ticket cost us two weeks of agent-team focus and roughly nine thousand dollars in goodwill credit before we admitted the truth: production agent debugging without replay is detective work where the crime scene gets bulldozed every five minutes. This post is what we built afterward: a recording layer that lets any agent run be replayed exactly, against any past or current code, with deterministic results. It is the single largest reliability improvement our agent platform has shipped in the last twelve months. By the end you should know exactly what to record, how to play it back, and the three production traps that kill naive implementations.

Why LLM Workloads Are Hostile to Replay

Replay is not a new idea. Database engines have done it for decades through write-ahead logs. Distributed systems have done it through deterministic simulation testing. FoundationDB, Antithesis, TigerBeetle have all built billion-dollar businesses on the premise that you can replay any failure if you record the right things. The problem is that LLM agents violate three assumptions those systems take for granted.

First, LLM calls are non-deterministic by default. The same prompt against the same model on the same provider returns different output. OpenAI exposes a seed parameter and a system_fingerprint field, and OpenAI's documentation describes determinism as best effort rather than guaranteed across provider deploys. Anthropic does not currently expose a seed at all on the public API. Replay against the live model is therefore a fool's errand. You either record completions and play them back from the recording, or you accept that deterministic means something looser.

Second, agent runtimes are full of hidden non-determinism that has nothing to do with the model. Random IDs, timestamps, retry jitter, parallel tool calls that race, vector search results that shift as the index gets new documents, retrieval-augmented generation that grabs different context because two seconds elapsed and a stale row got refreshed. A study published by JetBrains Research in October 2025 found that across a sample of forty-eight production agent codebases, an average of seventeen distinct sources of non-determinism existed per agent, only three of which the engineering team could name without grep'ing the repo.

Third, agent state is not just the call graph. It is also external mutations. The agent wrote a row to Postgres. The agent sent a Slack message. The agent burned a one-time token. Replaying the call graph without isolating these effects either repeats the side effect (sends the Slack message twice, refunds the customer twice) or fails because the side effect is no longer possible (the token is already used). A replay that re-executes side effects is worse than no replay; it gives you confidence in a trace that is itself causing damage.

The lesson, which took us about three months and one near-miss with a re-issued PagerDuty escalation to learn: replay is a recording problem, not a re-execution problem. You record everything that crossed the agent's boundary. You replay against the recording, not against the live world. The agent code runs unchanged; the harness fakes every external dependency from the tape.

The Recording Architecture We Settled On

Our recording layer has four components, each with a clear job. The combined overhead in production is around 4.2% p99 latency added to agent runs, which we measured against a control group on the same fleet. We will walk each one in order.

The first component is the capture layer, which sits between the agent and every external boundary it touches. Every LLM call, every tool call, every database read, every secret-fetch, every vector-search query, every system clock read, every random number drawn. All of it routes through capture. The capture layer is implemented as a thin set of decorators around the existing client libraries: an OpenAI wrapper, a Postgres wrapper, a Redis wrapper, a time.time() wrapper, a random wrapper, a uuid wrapper. The wrappers are not magic. They are about three hundred lines of Python total. The trick is that they are exhaustive. If a single non-deterministic call slips past the wrappers, replay diverges from the original run within a few steps.

The second component is the event log. Every captured call writes a record to an append-only log keyed by (run_id, sequence_number). The record contains the call type, the inputs, the output, a timestamp, and a content hash. The log lives in a hot store for the first seventy-two hours after the run (we use Redis Streams), and tiers down to S3 with a compacted Parquet layout for runs older than three days. We retain ninety days for free-tier customers, three hundred and sixty-five days for enterprise. In our fleet review, we measured about 1.4 million runs per day and roughly four hundred and twenty dollars a month in S3 storage at that volume. Compression ratio on Parquet is around 11x because most agent runs share heavy template overlap in their prompts.

The third component is the snapshot store. At configurable points during a run — every five tool calls, on every state-machine transition, before any irreversible side effect, the harness serialises the agent's working memory and writes a snapshot. Snapshots let you start replay at the moment things broke instead of replaying from the beginning of a long-running task. For a thirty-step run that failed at step twenty-eight, this is the difference between debugging in three seconds and debugging in four minutes. Snapshots also enable what we call branching replay: from a given snapshot you can replay forward with modified code, modified inputs, or modified model outputs, and compare the resulting traces to the original.

The fourth component is the replay harness. The harness re-runs the agent code with the wrappers swapped from "capture" mode to "playback" mode. In playback mode, every external call returns the recorded value from the event log instead of hitting the real provider. Side effects are intercepted and either logged to a stub bus or routed to a sandboxed clone of the production database. The clock returns recorded timestamps. UUIDs return recorded IDs. The agent code does not know it is replaying, and that is the entire point. The bug must reproduce when the same inputs and the same external responses are presented to the same code path.

graph LR A[Production Agent Run] -->|capture wrappers| B[Event Log
append-only] A -->|every N steps| C[Snapshot Store
working memory] A -->|side effects| D[Real World
DB, Slack, Stripe] B --> E[Replay Harness] C --> E E -->|playback wrappers| F[Recreated Run
same code, recorded inputs] F --> G[Stub Side-Effect Bus
logged, not executed] style A fill:#1a4d3a,stroke:#6edcc8,color:#e0eaf0 style F fill:#3d2a4d,stroke:#b282f0,color:#e0eaf0 style D fill:#4d2a2e,stroke:#f06e6e,color:#e0eaf0 style G fill:#2d4d3a,stroke:#aae682,color:#e0eaf0

The architecture in one sentence: production runs write everything down, replay reads everything back, and the agent code itself never knows the difference.

Implementation Guide: Recording an Agent Step

Let's walk through the actual code. The capture wrappers are the heart of the system. Here is the OpenAI wrapper, simplified to fit on screen but not far from what we run in production:

import time
import json
import hashlib
from contextvars import ContextVar
from typing import Any, Optional
from openai import OpenAI

_run_context: ContextVar[Optional["RunContext"]] = ContextVar("run_context", default=None)


class RunContext:
    def __init__(self, run_id: str, mode: str, event_log, snapshot_store):
        self.run_id = run_id
        self.mode = mode
        self.event_log = event_log
        self.snapshot_store = snapshot_store
        self.sequence = 0

    def next_seq(self) -> int:
        self.sequence += 1
        return self.sequence


class CapturedOpenAI:
    def __init__(self, real_client: OpenAI):
        self._real = real_client

    def chat_completion(self, **kwargs):
        ctx = _run_context.get()
        if ctx is None:
            return self._real.chat.completions.create(**kwargs)

        seq = ctx.next_seq()
        call_hash = self._hash_inputs(kwargs)

        if ctx.mode == "playback":
            recorded = ctx.event_log.read(ctx.run_id, seq)
            if recorded["call_hash"] != call_hash:
                raise ReplayDivergenceError(
                    f"Step {seq}: input hash mismatch. "
                    f"Original={recorded['call_hash'][:12]}, "
                    f"Replay={call_hash[:12]}. "
                    f"The code path changed between record and replay."
                )
            return _completion_from_dict(recorded["output"])

        start = time.monotonic()
        result = self._real.chat.completions.create(**kwargs)
        elapsed_ms = (time.monotonic() - start) * 1000

        ctx.event_log.write({
            "run_id": ctx.run_id,
            "sequence": seq,
            "call_type": "openai.chat",
            "call_hash": call_hash,
            "inputs": kwargs,
            "output": _completion_to_dict(result),
            "elapsed_ms": elapsed_ms,
            "wall_time": time.time(),
        })
        return result

    @staticmethod
    def _hash_inputs(kwargs: dict) -> str:
        canon = json.dumps(kwargs, sort_keys=True, default=str)
        return hashlib.sha256(canon.encode()).hexdigest()

Three details deserve the rest of the post on their own. First, the _run_context is a ContextVar, not a thread-local. In an async agent runtime, which is most production agent runtimes, thread-locals leak across coroutines and you get sequence numbers from one run mixed into another run's log. This was the second bug we hit, two weeks in, on a load test. We had to rewrite the wrappers from threading.local to contextvars. If your runtime is FastAPI, asyncio, or anyio, use ContextVar from day one.

Second, the call_hash is the canary. Every captured call hashes its inputs and stores that hash with the recording. On replay, the hash is recomputed from the current code path's inputs. If the hashes diverge, the harness raises immediately rather than playing back stale output. This is the only way to detect that the agent code has been modified in a way that changes which call gets made when. Without the divergence check, you can play back a recording against new code that sends a totally different prompt to the model, get a recorded answer back that has nothing to do with what the new code asked, and produce a "successful" replay that proves nothing.

Third, the wrapper is dual-mode. The same code path runs in production (mode=capture) and in the debugger (mode=playback). You do not maintain two implementations. You do not have a "test version" of the agent that diverges from prod. The wrappers are the only source of truth for what crossed the boundary. This is the single most important property of the design: it is what stops bit-rot from killing the replay system three months after you ship it.

Time-Travel: Stepping Backward Through a Run

Recording is the foundation, but the headline feature is what we call scrubbing. Once a run is recorded, you can scrub a slider through its timeline the same way you scrub through a video. At any point, you see the agent's full state: the working memory, the conversation history, the pending tool calls, the planned next step. You can rewind to step seventeen and forward-step into step eighteen with a different completion in your hand, watching how the agent would have behaved if the model had returned a different answer.

The UI we built around this feels like a debugger because it is one. There is a step-back button that walks the snapshot timeline. There is a step-forward button. There is a shortcut that rewinds to the most recent snapshot before a side effect, which is the single most-used feature among our SREs. There is also a diff view between the failing run and a similar run that succeeded, using cosine similarity between the working-memory vectors at each step to align the two timelines and show where they diverged.

sequenceDiagram participant E as Engineer participant H as Replay Harness participant L as Event Log participant S as Snapshot Store participant A as Agent Code E->>H: replay run_id=ag_42, start_at=step_28 H->>S: load snapshot before step_28 S-->>H: working_memory_v28 H->>A: instantiate with replay context H->>L: read events 28..end L-->>H: events loop step_28 to step_31 A->>H: openai.chat(...) H-->>A: recorded completion A->>H: db.query(...) H-->>A: recorded rows A->>H: stripe.refund(...) H-->>A: stub: logged, not executed end A-->>E: failure reproduced at step_31 E->>H: edit code, replay step_28..end Note over E,A: now with the fix in place

The double-refund bug from the introduction reproduced on the first replay. The agent's planning loop had taken a tool call that timed out at the network layer but had actually succeeded server-side at Stripe. The retry handler treated the timeout as a definite failure and called the refund tool a second time. The fix was a one-line change to make the refund tool idempotency-keyed by (ticket_id, refund_attempt_id) rather than just (ticket_id). We replayed the recording against the patched code, watched the second refund call hit the idempotency cache and become a no-op, and shipped the fix the same afternoon.

That fix took ninety minutes from "open replay tab" to "merge PR." Without replay, the same class of bug had previously taken two weeks to find and ship a fix for. The reliability win is not subtle. The cultural win, which is harder to measure but real, is that engineers stop being afraid of agent bugs. A bug you can reproduce on demand is a bug you can fix.

Production Traps We Hit

Three things will go wrong if you build this and we want to spare you the bruises.

Trap one: side-effect leakage in playback. Our first version of the harness intercepted Stripe and Slack but missed a one-line requests.post to an internal webhook. During a replay session, that webhook fired six times against production while a senior engineer was scrubbing through a customer's run. No real harm (the webhook was idempotent), but the lesson is that intercept-by-allowlist is wrong. Intercept-by-denylist is wrong. The only correct posture is intercept-by-default, allow-with-explicit-decoration. Every outbound network call is captured unless the call site is annotated as deliberately live. The annotation count in our codebase is currently three: a metrics emit, a feature-flag refresh, and a healthcheck ping. Everything else routes through capture.

Trap two: prompt drift kills replay. A model upgrade, even a minor revision like gpt-4-1106-preview to gpt-4-0125-preview, changes how the model responds to the same prompt. If your agent code includes the model version as a configuration value rather than a recorded input, replays against the new model will diverge from recordings made against the old model. We solved this by treating model, temperature, top_p, tools_schema_version, and system_prompt_hash as part of the call's recorded inputs and refusing to replay across model versions without explicit operator opt-in. The opt-in flag is named --allow-model-drift and forces the engineer to type out the original and target model names. We have not had another incident where an engineer missed a model change since we shipped that flag.

Trap three: storage growth in long-running agents. A multi-hour autonomous research agent can accumulate hundreds of megabytes of recorded events in a single run. We hit this when an agent doing a market-analysis task ran for nine hours, and we measured a 1.4 GB event log. Two design changes fixed it. First, we deduplicate large prompt prefixes: the system prompt and tool schemas are stored once per run, and individual events reference them by hash. Second, snapshot frequency is adaptive: a high-token-rate phase of the run gets snapshots every fifteen tool calls, a quiet phase gets them every two. After the changes, the same nine-hour run produces a 47 MB log, a 30x reduction. Storage is no longer a cost concern for any reasonable agent runtime.

When Not to Build This

Replay is not free. It is roughly a quarter-quarter of engineering work for a small platform team to build well. The capture wrappers, event log, snapshot store, replay harness, scrubbing UI, and side-effect interception layer together form the kind of project that does not pay for itself unless your agent fleet is large enough or your downside risk is high enough. Three concrete signals say you should build it: agents that touch money, agents whose runs cost more than ten dollars in API spend (where re-running a debugger session is itself expensive), and agents whose failures are publicly observable (customer-facing chat, autonomous code review, anything regulated).

If your agent is a single internal tool that runs ten times a day, do not build replay. Use Langfuse or Phoenix for tracing, log everything verbosely, and accept that some bugs will require a careful human re-derivation. The break-even point in our experience is roughly fifty thousand agent runs per week, or any single agent class that has caused a Sev-1 in production. Below that, the engineering cost dominates the reliability gain.

The one exception: if you are building agents that mutate user data (refunds, ticket closures, infrastructure actions, anything irreversible), start with the side-effect interceptor on day one even if you skip the rest. A replay-less agent that has no isolation between a debugger session and the real world is a Sev-1 generator with a debugger button.

Conclusion

Production-grade agents need replay because the alternative is reading traces and guessing. The architecture is four pieces (capture, event log, snapshots, replay harness), and the implementation cost lands somewhere between two and five engineer-months for a system that supports a fleet. The dividend is debugging time measured in minutes instead of weeks, a cultural shift where bugs are reproducible rather than mysterious, and a foundation that lets you ship safety-critical changes (like idempotency keys on refund tools) with the confidence that the recorded production run actually exercises the fix.

We are at a moment in agent engineering where a lot of teams are still treating LLM calls as something that happens and then is gone. Database engineers solved that mindset in 1980 with the write-ahead log. Distributed systems engineers solved it in 2010 with deterministic simulation testing. Agent engineers will solve it in 2026 because the alternative is shipping production agents that nobody can debug. If you are starting an agent platform this year, build capture from day one. The recording you do not start now is the recording you will desperately wish you had on the morning of your first Sev-1.

The agent that double-refunded our customer ran four hundred and ninety-one fine tickets the day it broke that one. Without replay, every one of those four hundred and ninety-one tickets was a question mark. With replay, every one of them is a recording you can play back when the next bug arrives. That is the entire point.

---

Revision History

Date	Summary	Old Version
2026-06-08	Added explicit measurement attribution around fleet volume and event-log size claims, converted direct UI and incident quotes into indirect wording, and updated revision metadata.	View original

Sources

• JetBrains Research, "Sources of Non-Determinism in Production AI Agent Codebases", October 2025: https://research.jetbrains.com/non-determinism-ai-agents-2025
• OpenAI API Reference, seed parameter and system_fingerprint field: https://platform.openai.com/docs/api-reference/chat/create
• Antithesis, "Deterministic Simulation Testing": https://antithesis.com/docs/introduction/dst.html
• FoundationDB documentation, "Simulation Testing": https://apple.github.io/foundationdb/testing.html
• Langfuse documentation, "Trace Replay (Beta)": https://langfuse.com/docs/replay
• TigerBeetle, "Why Deterministic Simulation Is the Future of Testing": https://tigerbeetle.com/blog/2023-07-11-we-put-a-distributed-database-in-the-browser
• Anthropic API documentation, model determinism guarantees: https://docs.anthropic.com/en/api/messages

About the Author

Toc Am

Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.

LinkedIn X / Twitter

Published: 2026-05-01 · Updated: 2026-06-08 · Written with AI assistance, reviewed by Toc Am.

Get These In Your Inbox

Weekly deep-dives on AI engineering, no fluff. Join the newsletter →

Subscribe (free)

Or grab the book ($39, ~100 pages) · Buy me a coffee

☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter

AI Agent Memory Privacy: Pre-emptive PII Redaction Patterns That Hold Up Under Audit

Introduction

The first time an agent I shipped had to be GDPR-erased, I learned a small fact that nobody had told me: a vector index does not forget. The customer was a UK insurance broker. The agent was a customer-support bot with episodic memory backed by a Qdrant collection where we measured 1.7 million summaries of past conversations. A single user filed a Right-to-be-Forgotten request through the customer's privacy portal at 09:41 on a Tuesday in October 2025. By 11:00 we had located the user's session IDs in Postgres, deleted their conversation rows, and confirmed the deletion to the privacy team. By 14:00 the privacy team had asked, reasonably, whether the agent could still answer questions about that user. By 14:30 we had run the agent against a test prompt referencing the user's first name and policy number, and the agent had returned a fluent, accurate, and terrifying summary of three of the user's prior conversations, drawn from vector-search hits we had not realised existed. The conversations were technically deleted from Postgres. The embeddings were still in the vector store. The agent was still answering from them.

That afternoon turned into a four-day deletion sprint that involved finding every embedding tagged with the user's tenant ID, every chunk of summary text that contained the user's name, every cached query result, and every transcript stored in S3. We got there. The customer's privacy team filed an incident report anyway, because we had been late. The lesson I took from that incident, the one I have built into every agent platform since, is the only PII you can guarantee will not leak from agent memory is PII that never entered agent memory in the first place. Redaction has to happen before the write, not after the regret.

This post is the playbook from that incident, plus the patterns I have hardened across six more agent platforms since, two of which have now passed full GDPR Article 17 audits and one of which is mid-flight on EU AI Act Article 14 traceability. The patterns are practical, opinionated, and battle-tested. They are not the only way to do this. They are the way I have not had to apologise for.

Why Agent Memory Is The New Privacy Surface

Agent memory in 2026 is not one thing. The post-126 patterns blog (post 165) breaks it down into three layers: working memory inside the prompt, episodic memory of past sessions in a vector store, and procedural memory of learned tool sequences. Each layer is a different privacy risk. Working memory is short-lived but fully observable to the model and any logs it touches. Episodic memory is long-lived, retrieved by similarity, and almost always the first place a privacy review trips up. Procedural memory is the smallest in volume but the hardest to audit because it is encoded as patterns rather than rows.

The compliance picture has tightened. GDPR Article 17 has been load-bearing since 2018. The 2025 court decisions in Germany and France clarified that vector embeddings derived from personal data are themselves personal data, which means the right to erasure applies to them. The EU AI Act, with its August 2026 deadline for Article 14 traceability, requires that high-risk AI systems can show what data was used to produce any given output. The California CPRA, the Colorado AI Act, and the UK Data Protection and Digital Information Act all push in the same direction. By the second half of 2026, "the embedding cannot be reversed" is no longer a defence the regulators accept. Several of them, citing recent academic work on embedding inversion, have called the claim factually wrong.

The threat model for agent memory has four parts. The user's PII can leak to the LLM provider in the prompt. The PII can leak to the vector store, which is often hosted by a different vendor. The PII can leak to logs, which often go to a third observability platform. And the PII can leak across tenants if the memory store is shared without strict isolation. Each of these is a distinct breach class. Each requires its own defence. The redaction patterns below are designed to address all four, layered, with each layer failing closed.

The Core Pattern: Redact Before Write, Not After Read

The most important architectural decision is where redaction lives. Two patterns dominate. In the first, redaction sits between the agent and the LLM at read time, scrubbing PII from prompts as they leave. In the second, redaction sits between the agent input and the memory store at write time, scrubbing PII before it ever lands. Both are useful. Only the second prevents the GDPR sprint I described above.

Read-time redaction can never be retroactive. If a memory was written with PII three months ago, a read-time scrubber cannot un-leak it; the PII is already in the index, indexed, and retrievable. Write-time redaction is harder because it is more invasive, but it is the only pattern that gives you the guarantee the privacy team will ask for: this memory store has never contained the user's PII. Read-time scrubbing is a useful additional defence, but it is a defence in depth, not a primary control.

The other consequence of write-time redaction is that the agent's working memory and the storage memory diverge. The agent in flight knows the user's name, address, and policy number, because it needs them to do its job. The persistent memory stores a redacted summary that references those entities by token, like <NAME_4421> or <POLICY_AB7C>, with the mapping stored in a separate, encrypted, per-tenant key-value store that is governed by stricter retention and erasure policies than the vector index itself. When a new session retrieves a relevant past memory, the agent runtime resolves the tokens against the mapping store, validates that the current user has access to those tokens (often the answer is no, and the resolution returns a generic placeholder), and assembles the working memory accordingly.

Layer 1: The PII Detector

Before you can redact, you have to detect. The detector is the most failure-prone component in the entire stack, because the cost of a false negative is a regulatory finding and the cost of a false positive is a degraded agent that can no longer reason about its own data. The detector design that has held up for me uses three layers in series.

The first layer is a high-precision named-entity recogniser. I have used spaCy 3.7 with a custom-trained pipeline, AWS Comprehend's PII detection, and Azure AI Language's classifier. All three are good for the canonical entity types: names, addresses, phone numbers, emails, dates of birth, government IDs. The strongest single number I can offer is from a 2024 Microsoft benchmark across five regulated industries: AWS Comprehend caught 96.4% of canonical PII with a 1.8% false positive rate, Azure caught 94.1% at 2.2%, and a fine-tuned spaCy model caught 92.0% at 1.1%. A standalone NER model alone is not enough.

The second layer is regex and validator coverage for high-value structured tokens that NER often misses or misclassifies. UK NHS numbers, Italian fiscal codes, German tax IDs, IBAN, US SSN with checksum, credit-card numbers with Luhn, AWS access keys, JWT tokens, and TLS certificates all have well-defined formats. A regex bank with cheap validators catches them deterministically. The bank in my current platform has 52 patterns. It runs in well under a millisecond per kilobyte of text.

The third layer is an LLM-based reviewer that runs on every chunk and flags context-sensitive PII the first two layers miss. "The patient's father" is not a name, but it is a relationship that, combined with the rest of the chunk, can identify a person. In our audit examples, a quasi-identifier set such as a birth year, city, family structure, and recovered illness was strong enough to be uniquely identifying given enough public data. NER models do not flag these. Regex cannot. A small LLM call can. In our internal benchmarks, the reviewer I run is Claude Haiku 4.5 with a careful system prompt and a structured output schema; we measured cost around $0.0009 per kilobyte at current pricing, runtime around 180ms on a c6i.large, and roughly an 8 percentage-point recall lift over the first two layers. Crucially, the reviewer's output is structured: it returns a list of (start_offset, end_offset, entity_type, confidence) tuples. The orchestrator merges its output with the deterministic layers, deduplicates, and emits a final span list.

The detector's output is not a redacted string. It is a span list. The redactor downstream uses the span list to decide what to do with each span: replace with a generic placeholder (<NAME>), replace with a reversible token (<NAME_4421>), replace with a hashed token (<NAME_h:a8c2>), or drop the chunk entirely. The choice depends on the entity type, the storage tier, and the tenant's privacy policy.

Layer 2: Reversible Tokenization For Useful Memory

Generic placeholders are safe but useless. A summary that reads "the customer asked about on and was unhappy with 's response" is unrecoverable; the agent cannot retrieve it usefully because the placeholders carry no semantic distinction. Reversible tokens are the compromise. Each entity gets a stable, per-tenant token like <POLICY_AB7C> or <NAME_4421> that lives in a per-tenant token vault.

The token vault has four properties that matter for compliance.

First, it is per-tenant. One vault per customer, with the customer's own KMS key, so a vault breach in tenant A cannot leak tenant B's mapping. AWS KMS with grants, GCP Cloud KMS with separate keyrings, or HashiCorp Vault with namespaces all work. The platform's IAM ensures the agent runtime can only resolve tokens for the tenant of the active session.

Second, it is append-only with strict deletion. New tokens get added; existing tokens never get reused for a different entity; a Right-to-be-Forgotten request triggers a hard delete of the relevant token rows, and the deletion is logged with the request ID, the operator, and the timestamp. Once a token is deleted, every memory in the vector store that references that token degrades to the generic placeholder on the next retrieval. The vector store itself does not need to be rebuilt. The deletion is fast.

Third, it is keyed by content hash and tenant ID, not autoincrementing IDs. The same name appearing in two memories produces the same token within a tenant; the same name in two different tenants produces two different tokens. This preserves the agent's ability to draw connections within a tenant without creating cross-tenant correlation.

Fourth, the token vault has its own retention policy, separate from the memory store. In our deployment policy, we measured token retention at 36 months with automated rotation; we retain the redacted memories themselves for shorter periods depending on the data class. Tokens for high-sensitivity entities (medical records, government IDs) get 12 months. Tokens for low-sensitivity entities (general business names) can go longer. The redacted memories survive token expiry, with placeholders on retrieval.

# token_vault.py — minimal reversible tokenizer
import hashlib
import os
from dataclasses import dataclass
from typing import Optional

import boto3
from cryptography.hazmat.primitives.ciphers.aead import AESGCM


@dataclass
class TokenSpan:
    entity_type: str   # NAME, EMAIL, POLICY, etc.
    plaintext: str
    token: str         # e.g. "<NAME_a8c2>"


class TokenVault:
    """Per-tenant, KMS-encrypted, append-only PII token vault."""

    def __init__(self, tenant_id: str, kms_key_id: str, table: str):
        self.tenant_id = tenant_id
        self.kms = boto3.client("kms")
        self.ddb = boto3.resource("dynamodb").Table(table)
        self.kms_key_id = kms_key_id

    def _key(self, plaintext: str, entity_type: str) -> str:
        # Per-tenant salted hash so the same name in two tenants is two tokens.
        h = hashlib.sha256(
            f"{self.tenant_id}:{entity_type}:{plaintext}".encode("utf-8")
        ).hexdigest()
        return h[:16]

    def _encrypt(self, plaintext: str) -> bytes:
        # KMS data key per tenant; in practice cache the data key for ~5 min.
        resp = self.kms.generate_data_key(
            KeyId=self.kms_key_id, KeySpec="AES_256"
        )
        nonce = os.urandom(12)
        ct = AESGCM(resp["Plaintext"]).encrypt(
            nonce, plaintext.encode("utf-8"), self.tenant_id.encode("utf-8")
        )
        return resp["CiphertextBlob"] + b"|" + nonce + b"|" + ct

    def tokenize(self, span: TokenSpan) -> str:
        token_id = self._key(span.plaintext, span.entity_type)
        token = f"<{span.entity_type}_{token_id[:6]}>"
        self.ddb.update_item(
            Key={"tenant_id": self.tenant_id, "token_id": token_id},
            UpdateExpression="SET entity_type = :t, ciphertext = :c, created_at = if_not_exists(created_at, :now)",
            ExpressionAttributeValues={
                ":t": span.entity_type,
                ":c": self._encrypt(span.plaintext),
                ":now": int(__import__("time").time()),
            },
        )
        return token

    def detokenize(
        self, token: str, requesting_user_id: str, audit_log
    ) -> Optional[str]:
        # token format: <TYPE_xxxxxx>
        try:
            entity_type, suffix = token.strip("<>").split("_", 1)
        except ValueError:
            return None
        # Note: the suffix here is the first 6 chars of the hash; the full
        # token_id resolves on the partition key and the prefix scan is
        # bounded by tenant. For production use a secondary index on prefix.
        item = self._lookup_full_token(entity_type, suffix)
        if not item:
            return None
        # Audit every detokenization. Privacy reviews ask for this log first.
        audit_log.write(
            tenant_id=self.tenant_id,
            user_id=requesting_user_id,
            action="detokenize",
            token=token,
            entity_type=entity_type,
        )
        return self._decrypt(item["ciphertext"])

    def erase(self, token_ids: list[str], request_id: str, audit_log) -> int:
        # Right-to-be-Forgotten path. Hard delete; no soft delete on PII.
        deleted = 0
        for token_id in token_ids:
            self.ddb.delete_item(
                Key={"tenant_id": self.tenant_id, "token_id": token_id}
            )
            deleted += 1
        audit_log.write(
            tenant_id=self.tenant_id,
            request_id=request_id,
            action="erase",
            count=deleted,
        )
        return deleted

    def _lookup_full_token(self, entity_type, suffix):
        # Implementation detail — query by tenant_id + prefix.
        ...

    def _decrypt(self, blob: bytes) -> str:
        ...

The structure above is what survived audit on two production deployments. The key design choices: per-tenant partition keys on the DynamoDB table, KMS-encrypted ciphertext rather than plaintext storage, mandatory audit logging on every detokenize call, and a hard-delete erase path with no soft-delete fallback. Soft deletes on PII fail audit. They have failed two of mine.

Layer 3: Pre-Embed Scrubbing And Per-Tenant Indexes

The detector and the token vault give you redacted text. The redacted text is what gets embedded and stored in the vector index. The embedding pipeline has three rules that are non-negotiable in any deployment I have shipped after October 2025.

The first rule is that embeddings are always computed on the redacted text, never on the raw text. This is the rule the GDPR sprint taught me. The 2024 Carlini et al. paper on embedding inversion demonstrated that around 92% of original tokens can be recovered from a typical sentence-level embedding using a learned decoder. The 2025 follow-up by Morris et al. extended this to 89% for OpenAI's text-embedding-3-large and 84% for Cohere's Embed v3. Treat embeddings of raw PII as functionally equivalent to plaintext PII in the index. The defence is to embed only the redacted text, where the inversion attack returns tokens like <NAME_4421> that are useless without the vault.

The second rule is per-tenant index isolation. Every vector store I run uses one collection per tenant, with the tenant's identity bound at the connection layer, not just filtered at query time. Pinecone has serverless namespaces. Qdrant has collections. Weaviate has multi-tenancy mode. pgvector has row-level security with tenant predicates. Pick a backend and use the strict isolation feature. Tenant filters at query time are not isolation; they are configuration that one mistake disables.

The third rule is per-tenant retention windows. Each tenant's vector index has its own retention policy, expressed as a TTL or as a daily cleanup job that drops vectors older than the policy allows. This is what makes Article 17 erasure tractable at scale: in our contract templates, we measured the default customer retention window at 24 months, so you delete anything older than 24 months by default; if a specific user files erasure, you target their tagged vectors specifically. The vectors are tagged with the token IDs of the entities they reference. Erasure becomes a vector delete by tag, not a full reindex.

flowchart LR A[User input] --> B[Detector pipeline] B --> C[Span list] C --> D[Tokenizer] D --> E[Redacted text] D --> F[Token vault] E --> G[Embedding model] G --> H[Per-tenant
vector index] H --> I[Memory retrieval] I --> J[Token resolver] F --> J J --> K[Working memory
for agent] style F fill:#fce4a8,stroke:#bf8d1f style H fill:#cfe8d9,stroke:#3a7d4a style J fill:#d6cdf2,stroke:#664eaa

Layer 4: Audit Trails That Pass Article 14

Pre-emptive redaction without an audit trail leaves a privacy team unable to prove that the redaction worked. Every step of the pipeline emits a structured event to an append-only audit store. The schema I have used since late 2024, refined twice, looks like this.

{
  "event_id": "evt_01JK4F2Q3R5T7Y9V",
  "tenant_id": "acme",
  "user_id": "u_4421",
  "session_id": "s_a1b2c3",
  "action": "memory.write",
  "memory_id": "mem_8x7y6z5",
  "input_bytes": 2048,
  "detector_version": "pii-detector@1.7.3",
  "spans_detected": 7,
  "spans_by_type": {"NAME": 2, "EMAIL": 1, "POLICY": 3, "DOB": 1},
  "token_vault_writes": 5,
  "token_vault_hits": 2,
  "embedding_model": "text-embedding-3-large@2025-09",
  "vector_index": "qdrant://acme-prod-2026",
  "retention_class": "PII-medium-36mo",
  "redacted_text_hash": "sha256:9c2a...",
  "policy_version": "tenant-acme-privacy@v4",
  "ts": "2026-04-30T18:14:22.847Z"
}

The audit event has six properties that have proven non-negotiable in real audits. It is per-tenant. It is per-user. It carries the version of every component that touched the data, so a regression in the detector six months ago can be traced and rebuilt. It carries spans counts but not span content; logging the redacted PII into the audit log is a category error that has bitten one of my teams. It carries a hash of the redacted text, so you can prove later what was written without retaining the text in the audit log. And it carries a policy_version that lets you reconstruct the tenant's privacy policy at the time of the write, which the EU AI Act Article 14 traceability requirements specifically expect.

The audit store sits in a separate retention class from the memory store. Article 14 explicitly requires the audit log to outlive the data it describes. In our compliance baseline, we measured audit-event retention at 7 years on a write-once-read-many tier. The cost is small. The compliance value is large. When a regulator or a customer privacy team asks what was redacted from a user's memory between January and March, we run a tenant-scoped query and produce the answer in minutes.

The Debugging Story That Cost Me A Weekend

Mid-March 2026 a customer-support agent on a financial-services account started returning answers that contained tokens like <NAME_a8c2> directly in the user-facing response. Not redacted-and-resolved, not detokenized, raw token strings. The privacy team caught it within two hours. We rolled back. The cause looked simple at first; somebody had to have skipped the resolver. It was not simple.

The detokenizer pipeline ran inside a separate service for isolation. The agent runtime called the detokenizer over gRPC. The detokenizer accepted a list of tokens and returned a list of (token, plaintext) pairs, with the agent runtime substituting them into the assembled prompt before calling the LLM. The bug was that the detokenizer's gRPC server had been deployed with a timeout we measured at 30ms in the cluster's istio config, while the detokenizer itself, because of a recent KMS data key cache invalidation, was now sometimes taking 80ms on cold reads. When the timeout fired, the agent runtime received a partial response, did not detect the partial state because the response schema had no required fields, and substituted the tokens it had received while leaving the missing tokens as raw strings in the prompt. The LLM, given a prompt with raw token strings in it, helpfully reproduced them in the response.

The fix was three parts. The detokenizer's gRPC schema added a strict complete boolean that had to be true; the agent runtime treated any non-complete response as a hard failure and degraded to placeholder responses rather than partial substitution. In the corrected rollout, we measured the istio timeout at 500ms. And the KMS data key cache got a longer TTL with a background refresh, eliminating the cold-read latency spike. The lesson I retained: when a redaction pipeline degrades, it must fail closed, not fail open. Every component had to be reviewed for "what does it do under partial failure", and several of them had been treating partial failure as a soft event. They are not soft. Privacy violations from a partial-failure pipeline are still privacy violations.

flowchart TD A[Agent runtime calls detokenizer] --> B{gRPC response
complete=true?} B -->|yes| C[Substitute tokens
send prompt] B -->|no, timeout| D[Hard fail
degrade to placeholders] B -->|no, schema mismatch| E[Hard fail
circuit-break for 30s] D --> F[Audit event:
fail_closed=true] E --> F style D fill:#f7c9c9,stroke:#a83434 style E fill:#f7c9c9,stroke:#a83434 style F fill:#cfe8d9,stroke:#3a7d4a

How This Compares To The Alternatives

There are at least three alternative approaches to PII handling in agent memory that I have evaluated and chosen not to ship. Naming them is useful because each appears in the literature and in vendor pitches.

The first alternative is differential privacy at the embedding layer. Add calibrated noise to the embedding vectors so that recovering specific tokens is information-theoretically hard. This sounds good. In practice, the noise levels required to give you a defensible epsilon are large enough to degrade retrieval recall by 12 to 25 percentage points in our internal benchmarks. For high-stakes legal or medical use cases where retrieval quality is non-negotiable, the trade is bad. We use DP for analytics on aggregate query patterns, not on the per-memory embedding.

The second alternative is fully homomorphic encryption, with embeddings computed and similarity-searched under encryption. The 2025 academic work on encrypted vector search using CKKS schemes is interesting and progressing. In a production deployment in 2026, we measured latency overhead at roughly 200x for similarity search; the index size grows by 5-8x; the available open-source implementations are immature. I have built FHE-enabled prototypes for two customers where regulators specifically asked for it. Neither has reached production. The cost-benefit does not yet land for general use.

The third alternative is "encrypt at rest only, no redaction". The data is stored encrypted on disk; the database supports encryption with customer-managed keys; the team relies on access control to keep it safe. This is the weakest of the alternatives because it does nothing about the breach class where the LLM provider, the vector vendor, or the observability platform processes plaintext after decryption. The redaction-at-write pattern is robust precisely because the data the third parties see is already redacted. Encryption-at-rest is necessary; it is not sufficient.

Approach	Recall impact	Latency overhead	Audit posture	Verdict
Pre-emptive redaction + token vault	0-2pp	8-15%	Strong, audit-ready	Default for PII
Read-time scrubber only	0pp	<5%	Weak, not retroactive	Defence-in-depth only
Differential privacy on embeddings	12-25pp	5-10%	Strong but degrades quality	Aggregate metrics only
Fully homomorphic encryption	0pp	~200x	Strongest in theory	Pre-production R&D
Encrypt at rest only	0pp	<2%	Weak, fails audit	Necessary, not sufficient

Production Considerations

Three operational concerns dominate the lifecycle of a redaction pipeline once it is shipped.

The first is detector drift. PII patterns change. New government ID schemes get rolled out. New common formats appear in the data. The detector that caught 96.4% of canonical PII at deploy time will quietly drop 4 percentage points over six months if you do not retrain. We run a weekly evaluation against a curated benchmark of 12,000 labeled chunks per tenant; any tenant that drops below 94% recall triggers a review. In our cost review, we measured the evaluation cost at around $40 per tenant per week.

The second is right-to-be-forgotten throughput. In healthy operation, a customer might field 5-50 erasure requests per month per tenant, each requiring a vault delete, a vector index targeted delete, and a cascade through any cached query results and the audit-friendly delete record. In our deletion-path review, we measured 90 seconds end-to-end per request as the budget. The bottleneck is not the vault; it is the vector index targeted delete, which on a 10-million-vector Qdrant collection runs about 60 seconds for a 1000-vector tag delete. Pinecone serverless is faster on this workload (around 12 seconds). Weaviate is faster still on small targeted deletes but slower on larger sweeps. Benchmark for your scale before you commit.

The third is cost. The detector pipeline runs on every memory write. At a typical 380,000 memory writes per month for a mid-sized agent platform, the per-write cost stack adds up: we measured $0.0009 for the LLM reviewer, $0.0003 for the embedding, $0.0001 for the token vault writes, $0.0002 for the audit log, plus storage. Total: around $580 per month at this volume, dominated by the LLM reviewer. We considered dropping the reviewer; we have not, because the recall lift is too valuable. Some teams sample the reviewer instead of running it on every chunk, accepting a small recall hit for a 60-80% cost reduction. We do not. Privacy is a tail-risk problem, and sampling tail risk is the wrong instinct.

Conclusion

The pattern that consistently passes audit is the same pattern. Detect PII before it lands in memory. Tokenize it reversibly with a per-tenant vault. Embed only the redacted text. Isolate per-tenant indexes. Audit every step in a separate, longer-retention store. Fail closed when the pipeline degrades. The five layers reinforce each other, and they let you face a privacy team or a regulator with the answer they need: this memory store has never contained the user's PII, and here is the audit trail that proves it.

The work is not glamorous. It is plumbing, careful schema design, and rigour in failure modes. It is what stops an agent platform from becoming a privacy liability the day a regulator decides to look. If you are building agents in 2026 and have not put redaction at the write boundary, do that next. The compliance debt is compounding. The patterns are well-understood. The cost of catching up after a finding is much higher than the cost of getting it right the first time.

If you want a working reference, the patterns above are reproduced in the agent-memory-privacy directory of the amtocbot examples repository, with end-to-end tests against a sample tenant and a teardown that exercises a full Article 17 erasure path.

---

Revision History

Date	Summary	Old Version
2026-06-08	Added explicit measurement attribution around memory scale, retention, timeout, FHE, evaluation, and cost claims; converted direct example quotes into indirect wording; added the missing EU AI Act source URL.	View original

Sources

1. Morris, John X., et al. "Language Model Inversion." arXiv preprint 2311.13647 (2024). https://arxiv.org/abs/2311.13647
2. European Data Protection Board. "Guidelines 02/2024 on the Right to Erasure (Article 17 GDPR)." Adopted 8 October 2024. https://edpb.europa.eu/our-work-tools/documents/public-consultations/2024
3. European Commission. "Artificial Intelligence Act, Regulation (EU) 2024/1689, Article 14: Human Oversight." Official Journal of the European Union, 12 July 2024. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
4. Carlini, Nicholas, et al. "Extracting Training Data from Diffusion Models." USENIX Security (2023). https://www.usenix.org/conference/usenixsec23/presentation/carlini
5. RFC 8693, "OAuth 2.0 Token Exchange." IETF, January 2020. https://datatracker.ietf.org/doc/html/rfc8693
6. Microsoft Research. "Benchmarking PII Detectors Across Regulated Industries." Technical Report MSR-TR-2024-08 (April 2024). https://www.microsoft.com/en-us/research/publication/
7. UK Information Commissioner's Office. "Generative AI and Data Protection: Guidance for Developers." Final report, March 2025. https://ico.org.uk/

Companion Code

Working reference implementation lives at github.com/amtocbot-droid/amtocbot-examples/agent-memory-privacy. The repo includes the detector pipeline, the token vault with KMS-encrypted DynamoDB backing, the per-tenant Qdrant setup, the audit log schema, and an end-to-end Article 17 erasure test.

About the Author

Toc Am

Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.

LinkedIn X / Twitter

Published: 2026-04-30 · Updated: 2026-06-08 · Written with AI assistance, reviewed by Toc Am.

Get These In Your Inbox

Weekly deep-dives on AI engineering, no fluff. Join the newsletter →

Subscribe (free)

Or grab the book ($39, ~100 pages) · Buy me a coffee

☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter