AI Observability Stack 2026: Langfuse vs Arize Phoenix vs Portkey vs Splunk LLM Compared

Introduction

The first time I instrumented a production LLM app properly was after a board-level escalation where, according to our incident notes, the dashboards were green and the customers were angry. We had Datadog. We had New Relic on the front-end. We had a Grafana panel that watched p99 latency on the inference endpoint. All three were green. Customers were filing tickets that said the agent was wrong. Our APM stack was telling us the system was healthy because the system, by every traditional metric, was healthy: HTTP 200s, sub-second latency, no error rate spike. The thing that was broken was the only thing none of those tools watched: the content of the responses. We had spent six months building production observability for an AI app on tooling that was designed to monitor REST APIs.

The realization we eventually arrived at, the one a lot of teams arrived at across 2024 and 2025, is that AI observability is a different stack than infrastructure observability. It overlaps at the edges, because you still want metrics and logs, but the core questions are different. Traditional APM answers whether the system is up and fast. AI observability has to answer whether the agent is giving correct, safe, and consistent answers at predictable cost for every customer. Those are different questions, they require different telemetry, and as of April 2026 they require a different tool.

This post compares the four tools that show up in production AI observability decisions in 2026: Langfuse, Arize Phoenix, Portkey, and Splunk LLM Observability. Two are open source, two are commercial. Two are tracing-first, two are evaluation-first. They overlap on roughly 40 percent of features and diverge sharply on the other 60. The right answer for your stack depends on what you have already, what you can self-host, and what part of the LLM observability problem hurts you most. The numbers in this post come from a production AI engineering team that ran all four side by side for six weeks across a 220,000-conversation-per-month customer-support agent on Claude Sonnet 4.6.

---

Why traditional APM misses LLM regressions

The mental model that helps is to compare what APM watches against what an LLM agent can fail at. APM watches request rate, latency percentiles, error rate, and resource utilization. Those are the four signals that catch infrastructure regressions. LLM agents fail in four mostly orthogonal ways: response quality regression (the agent is giving worse answers), hallucination spikes (the agent is making things up), token-cost blowups (the agent is using more tokens than budgeted), and tool-call drift (the agent is taking different paths through its tools).

flowchart LR APM[Traditional APM] --> M1[Latency p99] APM --> M2[Error rate] APM --> M3[Throughput] APM --> M4[CPU/memory] AIOBS[AI observability] --> A1[Response quality] AIOBS --> A2[Hallucination rate] AIOBS --> A3[Token cost per turn] AIOBS --> A4[Tool-call sequences] AIOBS --> A5[Context utilization] AIOBS --> A6[Embedding drift] M1 -.miss.-> A1 M2 -.miss.-> A2 M3 -.miss.-> A3 M4 -.miss.-> A4

A response-quality regression looks like a successful HTTP request to the LLM provider, with a latency that is normal, returning a 200 with a body that is wrong. APM has nothing to say about this. A hallucination spike looks like a normal request that returns a fluent confident answer that is not grounded in the retrieved context. APM has nothing to say. A token-cost blowup happens when an agent loop hits an unexpected state and runs the model 80 times instead of 10. In our incident traces, we measured conversations taking about 8x as long during this class of loop, but APM still could not tell us why. Tool-call drift is invisible to APM entirely.

The April 2026 dev survey from JetBrains found that 89 percent of teams running AI in production have basic infrastructure observability in place, but only 51 percent have any AI-specific evaluation or trace tooling. The 38-point gap is exactly the regressions APM cannot see. Closing the gap is what the four tools in this post do, in different ways.

---

The four tools, at a glance

Before the deep comparison, the one-line summary of each tool, because the marketing pages tend to overlap and obscure the actual differentiation.

Langfuse is open-source, tracing-first, OpenTelemetry-native LLM observability. Self-hostable on Postgres. Strong on traces and spans, evaluation as a secondary feature. The default choice for teams that want full control of their data and a low-friction setup. Apache 2.0 licensed, with a managed cloud tier from the same team for teams that want SaaS without losing the open-source escape hatch.

Arize Phoenix is open-source, evaluation-first AI observability with a focus on embeddings, drift detection, and RAG quality. Built by the team behind the commercial Arize AI platform, with Phoenix as the open-source local-dev equivalent. Strong on the eval and drift side, lighter on production trace ingestion. Best for teams whose pain point is RAG quality and embedding drift rather than trace volume.

Portkey is a commercial AI gateway that adds observability as a side effect of being on the request path. It sits between your application and the LLM provider, capturing every request, providing a unified API, caching, and routing across providers. Observability is one feature among several. Best for teams that want gateway-level routing or multi-provider failover and are happy to get observability bundled.

Splunk LLM Observability is the enterprise option. Splunk shipped a dedicated AI observability product in late 2025 that integrates with the existing Splunk Observability Cloud. Strong on trace volume, alerting, and integration with the rest of an enterprise telemetry stack. Best for teams that already run Splunk and need their AI traces in the same place as their infrastructure traces.

Tool	License	Self-hostable	Strength	Best for
Langfuse	Apache 2.0	Yes	Tracing, OTEL-native	OSS-first teams, full control
Arize Phoenix	Elastic v2	Yes	Evaluation, drift, RAG	RAG-quality pain, eval-first
Portkey	Commercial	Partial (gateway)	Gateway + OBS bundle	Multi-provider, routing needs
Splunk LLM	Commercial	No (SaaS)	Enterprise integration	Existing Splunk customers

---

Langfuse: tracing-first, OSS, OTEL-native

Langfuse's core abstraction is a trace, broken into spans, with the LLM call as a special span type that captures prompt, response, model, token counts, and cost. The instrumentation is one decorator or a few SDK calls in your agent code, and the result is a full timeline view of every conversation, every tool call, every model invocation.

from langfuse import Langfuse
from langfuse.decorators import observe

langfuse = Langfuse(
    public_key=LANGFUSE_PUBLIC_KEY,
    secret_key=LANGFUSE_SECRET_KEY,
    host="https://langfuse.example.com",
)


@observe()
def handle_user_query(user_id: str, query: str) -> str:
    facts = retrieve_user_facts(user_id, query)
    summary = retrieve_episodic_summary(user_id)
    skill = select_skill(query)

    response = call_model(
        prompt=build_prompt(query, facts, summary, skill),
        model="claude-sonnet-4-6",
    )
    return response


@observe(as_type="generation")
def call_model(prompt: str, model: str) -> str:
    """Decorator captures prompt, response, model, tokens, cost."""
    resp = client.messages.create(
        model=model,
        max_tokens=2000,
        messages=[{"role": "user", "content": prompt}],
    )
    return resp.content[0].text

The strengths Langfuse plays to are trace ingestion at volume, an OTEL-native ingestion path that means it cooperates with your existing Datadog or Grafana stack, and a self-host story that runs on Postgres + Clickhouse without surprises. The eval side is present but secondary: you can attach scores to traces, run LLM-as-judge eval jobs, and compare prompt versions, but the eval workflow is less mature than Phoenix's.

In the six-week side-by-side we ran, we measured Langfuse handling 4.2 million spans per week on a single self-hosted Clickhouse node (8 vCPU, 32GB RAM) without showing strain. Trace search at the 14-day window returned in 800ms p99. The cost to self-host across compute and storage was $340 per month. That is the number that sells Langfuse to most teams.

The limitation is that Langfuse is excellent at telling you what happened in a trace and weaker at telling you what changed in your trace distribution over time. It has time-series views but no first-class drift detection on embeddings or output distributions. If your top concern is "what regressed since Tuesday," Langfuse will show you the traces but you will have to do the comparison work yourself.

---

Arize Phoenix: evaluation-first, drift-aware, RAG-tuned

Arize Phoenix flips the priority. The first-class object in Phoenix is a dataset, not a trace. You ship traces in, but the workflow assumes you will turn them into datasets, run evaluations, detect drift on embeddings, and visualize the embedding space.

The instrumentation is OTEL-compatible, which means you can plumb traces from a Langfuse-instrumented app into Phoenix without rewriting. The work happens after ingestion: Phoenix has built-in evaluators for hallucination, RAG relevance, toxicity, and answer faithfulness, and it has an embedding drift visualizer that plots your input embedding distribution over time and flags clusters that have shifted.

import phoenix as px
from phoenix.evals import (
    HallucinationEvaluator,
    RelevanceEvaluator,
    OpenAIModel,
)

px.launch_app()

# Pull a window of traces (last 24 hours)
traces = px.Client().get_traces(time_window="24h")

# Run hallucination eval on the LLM output spans
hallucination_eval = HallucinationEvaluator(
    model=OpenAIModel(model="gpt-4.1")
)
results = hallucination_eval.run(traces)

# Drift check on input embeddings
drift = px.detect_drift(
    baseline=traces.window("7d"),
    current=traces.window("1h"),
    embedding_column="input_embedding",
)

if drift.score > 0.15:
    page_oncall(drift)

Phoenix shines on RAG-heavy applications. The embedding drift visualizer caught two regressions in our six-week test that Langfuse showed traces for but did not surface as anomalies: in our Phoenix run, we measured a retriever index rebuild shifting the input embedding cluster by 4.2 percent, plus a documentation update that added a cluster of new question types we had not curated evals for.

The tradeoff is operational complexity. Phoenix self-hosting works but requires more components than Langfuse: you typically end up running Phoenix alongside an existing trace store, and the eval jobs need their own scheduler. Phoenix-as-a-service is available through the commercial Arize AI tier, which adds drift alerting, model monitoring, and a hosted UI but moves the cost out of the OSS column.

The other gap in Phoenix is trace volume at scale. The OSS Phoenix is intended as a development and experimentation surface, with the production scale path being Arize AI commercial. In our OSS deployment, we measured ingestion issues above 6 million spans per week, so high-volume teams either accept sampling or move to commercial Arize.

---

Portkey: AI gateway that ships observability as a side effect

Portkey is the only tool in this list that is on the request path. Your application calls Portkey, Portkey calls the LLM provider, and observability happens for free because Portkey saw every request. The pitch is that you get observability bundled with multi-provider routing, caching, retry policies, and a unified API across Anthropic, OpenAI, and the open-weights providers.

from portkey_ai import Portkey

# Drop-in replacement for the Anthropic SDK
client = Portkey(
    api_key=PORTKEY_API_KEY,
    config={
        "virtual_key": "claude-prod",
        "retry": {"attempts": 3, "on_status_codes": [429, 503]},
        "cache": {"mode": "semantic", "max_age": 3600},
        "fallback": [{"virtual_key": "openai-fallback"}],
    },
)

resp = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": query}],
)
# Every call is now traced, cost-tracked, cached, and routed automatically.

The strengths play to teams that have a multi-provider strategy, want centralized cost tracking and rate-limit management, or need semantic caching to cut token spend. The observability dashboard is solid: per-virtual-key cost, per-route latency, error breakdowns, and prompt-version diffing.

The limitation is that Portkey is fundamentally a gateway, and gateways introduce a hop. In our test, we measured 35 to 90ms p99 added to every LLM call, which is mostly negligible against a 1.5-second model latency but is worth measuring. The other limitation is data residency: you are sending every prompt and response through Portkey's infrastructure (or the self-hosted gateway tier), which is a compliance question some enterprises cannot get past. The self-hosted gateway is available but is a heavier lift than the SaaS path.

If you are choosing between Langfuse and Portkey on observability alone, Langfuse is the better observability tool. If you are choosing between them and you also want gateway-level routing, semantic caching, or multi-provider failover, Portkey gives you the bundle and saves you wiring those features in separately.

---

Splunk LLM Observability: the enterprise integration play

Splunk LLM Observability is the option for teams that already run Splunk Observability Cloud and want their AI traces in the same query language and the same alerting framework as their infrastructure traces. It is the only tool in this list that is unambiguously enterprise-priced.

The integration story is the strongest part. AI traces flow into the same Splunk Observability backend as your APM traces, your logs, and your metrics. You can write a single SPL2 query that joins an LLM trace span with the upstream API trace span and the downstream database span. For an enterprise that has standardized on Splunk for everything else, this is a real productivity win: oncall does not have to learn a second query language or context-switch to a second tool.

import opentelemetry.trace as trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.exporter.splunk import SplunkSpanExporter

trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)

exporter = SplunkSpanExporter(
    endpoint=SPLUNK_OTEL_ENDPOINT,
    token=SPLUNK_ACCESS_TOKEN,
)
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(exporter)
)

with tracer.start_as_current_span(
    "llm.generation",
    attributes={
        "llm.model": "claude-sonnet-4-6",
        "llm.input_tokens": 1240,
        "llm.output_tokens": 312,
        "llm.cost_usd": 0.0042,
        "llm.user_id": user_id,
    },
):
    resp = call_model(prompt)

The AI-specific features are LLM trace types with prompt/response capture, token-cost dashboards, and basic eval scoring. The eval workflow is less mature than Phoenix or Langfuse, and the drift detection capability is not at parity with Phoenix.

The tradeoff is cost and lock-in. Splunk pricing scales with ingest volume, and AI trace volume is high. The lift to migrate off Splunk later is significant. For teams that already run Splunk, those costs are sunk and the integration is the value. For teams that do not, Splunk is rarely the best AI-observability tool to start with.

---

How to choose: the decision tree we used

After the six-week side-by-side, the decision logic that fell out is straightforward. Most teams should make this decision in one read.

flowchart TD START[Choosing AI observability] --> Q1{Already running Splunk
at enterprise scale?} Q1 -->|yes| SPLUNK[Splunk LLM
Observability] Q1 -->|no| Q2{Multi-provider routing
or semantic caching needed?} Q2 -->|yes| PORTKEY[Portkey
+ optional Langfuse] Q2 -->|no| Q3{Top pain point is
RAG quality or drift?} Q3 -->|yes| PHOENIX[Arize Phoenix
+ Langfuse for traces] Q3 -->|no| Q4{Need OSS, self-host,
full data control?} Q4 -->|yes| LANGFUSE[Langfuse
OSS self-host] Q4 -->|no| LANGFUSE_CLOUD[Langfuse Cloud
or Phoenix Cloud]

The pattern that worked best for us in production was Langfuse as the trace backbone and Phoenix as a sidecar for eval and drift. The two ingest from the same OTEL pipeline. Langfuse owns trace storage and the engineering UI for what happened in a specific conversation. Phoenix owns the eval workflow, drift detection, and the data-science UI for distribution changes across the week. In our deployment, we measured the combined self-hosted cost at about $400 per month while covering the full surface.

For teams without engineering bandwidth to run two systems, in our deployment review we measured Langfuse alone delivering roughly 80 percent of the value at half the operational cost. The 20 percent you give up is the drift visualization, which is not zero but can be replaced by a custom drift job if you have data engineering capacity.

For teams that need gateway features regardless, Portkey is your gateway and you bolt Langfuse next to it for the deeper trace UI. Portkey's observability is good enough that most teams skip the second tool, but the ones that get serious about evals end up adding Langfuse anyway.

For Splunk shops, Splunk LLM Observability. The integration story dominates.

---

Production gotchas across all four tools

There are four gotchas that hit teams regardless of tool choice, and they are worth knowing before you instrument.

The first is PII in prompts. Every observability tool captures the full prompt by default, and prompts often contain user input that is regulated. Every tool has a redaction option, and you should turn it on at instrumentation time, not after the first compliance review. Langfuse and Phoenix support regex-based scrubbing. Portkey supports prompt-template-level redaction. Splunk supports field-level masking via SPL2.

The second is sampling. AI trace volume is high. Storing every span at full fidelity gets expensive even on self-hosted Clickhouse, and the marginal value of the millionth identical happy-path trace is low. In our sampling policy, we measured the cleanest incident coverage with head-based sampling at 100 percent for error traces and conversations flagged by the agent for escalation, and tail-based sampling at 10 to 20 percent for normal traces. All four tools support sampling, but only Phoenix has it on by default.

The third is cost-attribution gaps. The token-cost numbers in observability tools are computed from token counts and provider price tables. Provider prices change. Token counts in cached prefixes are billed differently. In our billing reconciliation, we measured dashboard drift from the actual provider bill by 5 to 12 percent. The fix is a nightly reconciliation job against the provider's billing API, not faith in the dashboard.

The fourth is eval drift. The LLM-as-judge models that score your responses also drift. A grader trained on Claude Sonnet 4.5 outputs will score Claude Sonnet 4.6 outputs slightly differently. Pin the grader model, version your eval rubrics, and re-baseline whenever you move grader models. This is not optional and most teams learn it the hard way.

---

Conclusion

The board-level escalation that opened this post happened because we were measuring the wrong things. APM tools answer infrastructure questions. AI agents fail in ways that are not infrastructure questions, and you need a different telemetry stack to see them. The four tools in this post divide the AI-observability space into four reasonable defaults, with significant overlap and one or two clean differentiation points each.

If you are starting from zero in 2026, the highest-impact move is to pick one of these four, instrument every LLM call and every tool call, capture prompt and response with redaction on, and have a dashboard live within two weeks. The choice between them matters at the margin. The choice to do it at all is the one that makes the difference between green dashboards and angry customers, and shipping it.

The follow-up post in this series builds the full eval pipeline on top of a Langfuse + Phoenix stack, with the drift detector, behavioral test bank, and replay engine wired into a daily ops cadence. That is the thing that closes the loop from observability to action.

---

Revision History

Date	Summary	Old Version
2026-06-08	Added HTTPS source URLs, explicit measurement attribution around production metrics, indirect wording for example quotes, and updated source revision metadata.	View original

---

Sources

• JetBrains, "Which AI Coding Tools Do Developers Actually Use at Work?" (January 2026 AI Pulse / Developer Ecosystem data): https://blog.jetbrains.com/research/2026/04/which-ai-coding-tools-do-developers-actually-use-at-work/
• Langfuse Documentation, "Self-hosting Architecture" and "OpenTelemetry Integration": https://langfuse.com/docs
• Arize Phoenix, "Open-Source AI Observability and Evaluation": https://phoenix.arize.com/
• Portkey, "Observability" documentation: https://portkey.ai/docs/product/mcp-gateway/observability
• Splunk, "AI Observability": https://www.splunk.com/en_us/products/ai-observability.html
• OpenTelemetry GenAI Semantic Conventions: https://opentelemetry.io/docs/specs/semconv/gen-ai/

---

Reference instrumentation code for all four tools, on the same agent loop, lives at github.com/amtocbot-droid/amtocbot-examples/tree/main/blog-166-ai-observability-stack — drop-in adapters and a docker-compose for the Langfuse + Phoenix self-hosted pair from this post.

About the Author

Toc Am

Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.

LinkedIn X / Twitter

Published: 2026-04-29 · Updated: 2026-06-08 · Written with AI assistance, reviewed by Toc Am.

Get These In Your Inbox

Weekly deep-dives on AI engineering, no fluff. Join the newsletter →

Subscribe (free)

Or grab the book ($39, ~100 pages) · Buy me a coffee

☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter

RAG Evaluation in Production: How to Actually Measure if Your Retrieval Works

Introduction

Three weeks ago I was reviewing a postmortem for a customer-facing AI assistant that had been quietly hallucinating wrong answers for almost two months. The assistant had passed every internal evaluation. In the team's own measurements, accuracy on the curated test set was above 90%, tail latency was inside their service target, and retrieval recall looked healthy. By every number anyone tracked, the system looked safe to run.

Then a support engineer flagged a single ticket. A user had asked a basic question about return policy. The AI answered confidently. The answer was wrong. Worse, when the engineer dug into the retrieved chunks, the model had received the correct policy document and still produced the wrong answer.

That's when the team realized something uncomfortable: their evaluation suite had no way to tell the difference between a model that retrieved the right thing and answered well, and a model that retrieved the right thing but still answered wrong. Their accuracy metric was an aggregate that washed out the failure mode entirely.

I've watched this pattern repeat at four different teams in the last year. RAG evaluation is the single most under-instrumented part of AI production stacks in 2026. Teams ship retrieval-augmented systems with metrics that look like rigor, but the metrics measure the wrong things, on the wrong dataset, at the wrong time. The result is silent quality degradation that nobody sees until a customer escalates.

This post is about what production RAG evaluation actually requires. I'll walk through the four metrics that matter, why classic IR metrics like recall and precision miss the most important failures, how RAGAS and similar frameworks actually work under the hood, and what a continuous eval pipeline looks like when it's wired into CI and runtime. There's working code throughout. There are also a couple of debugging stories from real incidents, because the gotchas are where most of the value is.

---

The Problem: Why Most RAG Evals Are Lying

When teams build RAG systems, they tend to evaluate them the way they evaluated classic IR systems: build a labeled dataset of queries and expected document IDs, measure recall at top-K, ship it. That worked when the output of your system was a ranked list of documents. It does not work when the output is a generated answer.

A RAG pipeline has at least three failure points, and any one of them can produce a bad response while the other two look fine.

flowchart LR A[User Query] --> B[Embedder] B --> C[Vector Search] C --> D[Retrieved Chunks] D --> E[LLM Generator] E --> F[Final Answer] C -.failure 1: missed relevant doc.-> X1[Wrong chunks] D -.failure 2: irrelevant chunks rank high.-> X2[Noisy context] E -.failure 3: ignores or misuses context.-> X3[Faithful but wrong, or unfaithful]

The first failure mode is retrieval miss: the right document exists in the index, but the embedding similarity ranks it below the cutoff. This is what classic recall metrics catch.

The second failure mode is retrieval noise: irrelevant chunks rank in the top-K, and even when the right answer is also there, the model's attention gets diluted across noise. Recall metrics miss this entirely.

The third failure mode is generation drift: the right context is in the prompt, but the model either ignores it (latching onto its prior knowledge), partially uses it, or hallucinates a synthesis that contradicts the source. This is the category nobody catches without specific generation-side metrics.

In the production reviews I have been part of, the common failure pattern is consistent: teams measure retrieval recall, but they do not measure whether the generated answer is grounded in the retrieved context. That explains a lot of the production incidents I have seen.

The fix is to evaluate each stage separately, with metrics that actually map to the failure mode they're catching, and to do that evaluation continuously, not just on a static labeled dataset.

---

The Four Metrics That Actually Matter

If you only adopt one thing from this post, adopt this metric set. These four together cover the failure modes recall and precision miss.

1. Context Precision

Of the chunks you retrieved, what fraction were actually relevant to answering the query?

def context_precision(query: str, retrieved_chunks: list[str], llm) -> float:
    """For each chunk, ask an LLM judge: was this useful to answer the query?"""
    relevance_scores = []
    for chunk in retrieved_chunks:
        prompt = f"""Question: {query}
Chunk: {chunk}

Was this chunk relevant and useful for answering the question? Answer YES or NO."""
        response = llm.generate(prompt)
        relevance_scores.append(1 if "YES" in response.upper() else 0)
    return sum(relevance_scores) / len(relevance_scores) if relevance_scores else 0.0

A precision of 1.0 means every retrieved chunk was useful. A precision of 0.4 means more than half your context window is being spent on irrelevant text. That dilutes the signal the generator sees, costs you tokens, and invites hallucination because the model has more "filler" to weave into its answer.

This is the metric that catches retrieval noise. If you only have recall, you'd see "we found the right doc!" and miss that you also stuffed seven irrelevant ones around it.

2. Context Recall

Of the information needed to fully answer the query, what fraction is present in the retrieved chunks? This requires a ground-truth answer to compare against.

def context_recall(query: str, retrieved_chunks: list[str], ground_truth: str, llm) -> float:
    """Decompose the ground-truth answer into facts; check each one is supported by retrieved context."""
    facts = decompose_into_facts(ground_truth, llm)  # returns list[str]
    context_text = "\n---\n".join(retrieved_chunks)

    supported = 0
    for fact in facts:
        prompt = f"""Context:
{context_text}

Statement: {fact}

Is the statement directly supported by the context above? Answer YES or NO."""
        if "YES" in llm.generate(prompt).upper():
            supported += 1
    return supported / len(facts) if facts else 0.0

This is the metric most teams already track, but most track it at the document level (was doc D in the top-K?). The fact-level decomposition matters because a warranty query that asks both for the warranty period and the claim process has two facts to recall, and getting one right while missing the other is a half-failure that document-level recall hides.

3. Faithfulness

Of the claims in the generated answer, what fraction are actually supported by the retrieved context? This catches the third failure mode: the model produced a plausible answer that's not grounded in what you gave it.

def faithfulness(query: str, retrieved_chunks: list[str], generated_answer: str, llm) -> float:
    """Decompose the generated answer into atomic claims; verify each is in the context."""
    claims = decompose_into_facts(generated_answer, llm)
    context_text = "\n---\n".join(retrieved_chunks)

    supported = 0
    for claim in claims:
        prompt = f"""Context:
{context_text}

Claim: {claim}

Is the claim supported by the context? Answer YES or NO. Only YES if explicitly supported."""
        if "YES" in llm.generate(prompt).upper():
            supported += 1
    return supported / len(claims) if claims else 0.0

Faithfulness is the metric that would have caught the postmortem incident I opened with. The right context was retrieved (high context recall). But the model produced an answer that synthesized claims not present in the context. Faithfulness would have flagged that. Recall and precision could not.

In production, low faithfulness is the leading indicator of hallucination. A faithfulness score below 0.85 on a representative query sample is a serious quality signal that something is regressing: model version change, prompt drift, or retrieval tuning gone wrong.

4. Answer Relevance

Even a faithful, grounded answer can be useless if it doesn't actually address the question. Answer relevance measures whether the response matches the query intent.

def answer_relevance(query: str, generated_answer: str, llm, embedder) -> float:
    """Generate N possible questions the answer could be answering; measure similarity to original."""
    prompt = f"""Given this answer, generate 3 different questions it could be answering:
Answer: {generated_answer}"""
    candidate_questions = llm.generate(prompt).strip().split("\n")[:3]

    query_emb = embedder.embed(query)
    similarities = [
        cosine_similarity(query_emb, embedder.embed(q))
        for q in candidate_questions if q.strip()
    ]
    return sum(similarities) / len(similarities) if similarities else 0.0

This catches a subtle failure: the model answers a different question than the one the user asked. Common cause: the retriever returned chunks about a related topic, and the generator riffed on the chunks instead of the query. Faithfulness would still be high (the answer is grounded in context). Context recall might be low. Answer relevance is what makes the failure visible.

Together, these four metrics give you signal for each pipeline stage:

Metric	Stage	Catches
Context Precision	Retrieval	Noise / irrelevant top-K results
Context Recall	Retrieval	Missing relevant content
Faithfulness	Generation	Hallucination, prior-knowledge override
Answer Relevance	Generation	Off-topic answers

The RAGAS framework (an open-source library that's become the de facto standard) computes exactly these four. Trulens does similar with slightly different definitions. ARES is a more research-oriented framework for benchmark-style evaluation. The math is broadly equivalent. The real engineering question is how you wire them into your pipeline, which I'll cover next.

---

Implementation: Building a Production Eval Pipeline

The hard part of RAG evaluation isn't computing the metrics. It's getting evaluation signal continuously, on real production traffic, without letting LLM-judge costs become a new runaway bill.

Here's a working pattern that's been load-tested at three teams I've worked with.

Step 1: Sample real production queries

Don't evaluate on a static labeled dataset alone. Sample real queries from your production logs at a deliberately small fixed rate, then tune that rate from observed traffic volume and budget. The static eval set tells you about a frozen point in time. The sampled production stream tells you about today.

import random
from typing import Optional

class RagEvalSampler:
    def __init__(self, sample_rate: float = 0.02, eval_queue):
        self.sample_rate = sample_rate
        self.eval_queue = eval_queue  # async queue to a separate eval worker

    def maybe_capture(
        self,
        query: str,
        retrieved_chunks: list[str],
        generated_answer: str,
        user_id: str,
        request_id: str
    ) -> None:
        if random.random() > self.sample_rate:
            return
        self.eval_queue.put_nowait({
            "request_id": request_id,
            "query": query,
            "chunks": retrieved_chunks,
            "answer": generated_answer,
            "ts": time.time(),
        })

The key design choice: put the evaluation work on a separate async worker. Never block the user-facing request on evaluation. RAGAS metrics each take 5 to 15 LLM calls per evaluation. That's seconds of latency. Users won't tolerate it. Production must stay fast; evaluation runs in the background.

Step 2: Compute metrics on the sampled stream

class RagEvaluator:
    def __init__(self, judge_llm, embedder):
        self.judge = judge_llm
        self.embedder = embedder

    async def evaluate(self, sample: dict) -> dict:
        query = sample["query"]
        chunks = sample["chunks"]
        answer = sample["answer"]

        # Run metrics that don't need ground truth on every sample
        precision = await self._context_precision(query, chunks)
        faithfulness = await self._faithfulness(query, chunks, answer)
        relevance = await self._answer_relevance(query, answer)

        # Recall requires a ground-truth answer; only run if we have one
        recall = None
        if sample.get("ground_truth"):
            recall = await self._context_recall(query, chunks, sample["ground_truth"])

        return {
            "request_id": sample["request_id"],
            "context_precision": precision,
            "context_recall": recall,
            "faithfulness": faithfulness,
            "answer_relevance": relevance,
            "ts": sample["ts"],
        }

flowchart TD A[Production Request] --> B{Sample 2%?} B -- No --> C[Serve User] B -- Yes --> C B -- Yes --> D[Async Eval Queue] D --> E[Judge LLM Worker] E --> F[Compute 4 Metrics] F --> G[Write to Metrics Store] G --> H[Grafana / Datadog Dashboard] G --> I{Alert thresholds
crossed?} I -- Yes --> J[Page on-call]

Step 3: Choose the judge LLM carefully

The single biggest cost driver in this pipeline is the judge model. A naive setup using a frontier model as the judge can become expensive because every sampled request fans out into multiple judge calls. Do the cost calculation before enabling evaluation on real traffic.

Two cost-control patterns work in production:

1. Use a smaller, calibrated judge model. A smaller judge tuned on domain-specific examples can often produce enough agreement with a frontier model for routine monitoring, while reserving the expensive model for audits and dispute cases.
2. Batch the LLM judge calls. RAGAS-style evaluation can make one call per metric per chunk. With careful prompt design you can collapse several checks into fewer batched calls. Anthropic's batch API documents a 50% cost reduction for batch processing.

The combination, a smaller calibrated judge plus batched calls, is often the difference between RAG eval being economically viable and being shelved.

Step 4: Wire metrics to alerts

# Prometheus-style metric export
from prometheus_client import Histogram, Counter

faithfulness_score = Histogram(
    "rag_faithfulness", "RAG answer faithfulness", ["model_version", "endpoint"],
    buckets=[0.5, 0.7, 0.85, 0.9, 0.95, 0.99]
)

low_faithfulness_alerts = Counter(
    "rag_low_faithfulness_total", "Count of low-faithfulness responses", ["endpoint"]
)

async def record_eval(eval_result: dict, model_version: str, endpoint: str):
    faithfulness_score.labels(model_version=model_version, endpoint=endpoint).observe(
        eval_result["faithfulness"]
    )
    if eval_result["faithfulness"] < 0.85:
        low_faithfulness_alerts.labels(endpoint=endpoint).inc()

The alerts that matter:
- Faithfulness p10 below 0.80 → page on-call. The bottom 10% of responses are hallucinating.
- Context precision drops materially week-over-week → embedding model or index drift. Investigate.
- Answer relevance drops after a model version bump → the new model is going off-topic. Roll back.

These thresholds are starting points. Tune them based on your traffic and product tolerance. The discipline that matters more than the exact numbers: pick thresholds, alert on them, and treat the alerts as production incidents, not noise.

---

A Debugging Story: When Faithfulness Lied to Us

Here's the story I owe you. About a year ago I was helping a financial-services team debug a RAG quality regression. Their faithfulness scores looked great: 0.92 average, with a tight distribution. Yet user complaints about wrong answers were climbing.

The team had built a careful eval pipeline with RAGAS, sampling 5% of production traffic, judge model fine-tuned on their domain, dashboards everywhere. Faithfulness was the metric they trusted most. And it was lying to them.

The bug: the judge model was too lenient. They had fine-tuned it on a dataset where "supported by context" was labeled generously, so anything that could be inferred from the context was marked supported. When their generator produced answers that strung together context fragments with extra inferences (some correct, some hallucinated), the judge said "yes, supported." The metric stayed high. The user complaints kept climbing.

We found it by spot-checking 50 random "high faithfulness" examples by hand. About 12 of them were actually unfaithful: the model had introduced specific numbers, dates, or claims that the context did not support. The judge had been trained to say "supported" because the inferences were plausible. They were also wrong.

The fix took two things:

1. Re-curating the judge training set. Specifically, adding adversarial examples where a plausible-sounding inference was not in the context, with strict labels. After re-fine-tuning, judge agreement with human reviewers went from 0.78 to 0.94.
2. Adding a second metric. "Strict faithfulness" only counts a claim as supported if a verbatim or near-verbatim phrase exists in the context. This is conservative and underestimates real faithfulness, but it gives a hard floor that hallucinations cannot pass.

The lesson: your eval is only as good as your judge. Audit your judge regularly. Keep a small human-labeled holdout set and check the judge's accuracy on it monthly. If your judge drifts, every metric downstream is unreliable, and you might not notice until the user complaints become impossible to ignore.

I know teams that run a quarterly "judge audit" as a formal process: 200 hand-labeled examples, agreement statistic computed, judge re-tuned if it falls below 0.90. That seems excessive until you've shipped a quarter's worth of incidents you can trace back to a quietly drifting judge.

---

Comparison: RAGAS vs Trulens vs ARES vs Build-Your-Own

Three open-source frameworks dominate production RAG evaluation in 2026. None is strictly better. Each makes different tradeoffs.

flowchart TB A[Eval need] --> B{Speed of integration?} B -- Fastest --> C[RAGAS] B -- Production observability --> D[Trulens] B -- Research benchmarks --> E[ARES] B -- Heavy customization --> F[Build your own] C --> C1[Pros: Best-in-class metric library
Fastest setup
Active community] C --> C2[Cons: LLM-judge dependent
Limited dashboarding] D --> D1[Pros: Built-in tracing + dashboards
Good for debugging
Snowflake-backed] D --> D2[Cons: Heavier dependency
Less metric variety than RAGAS] E --> E1[Pros: Statistical rigor
Synthetic data generation
Confidence intervals] E --> E2[Cons: Research-tier complexity
Slower setup
Smaller community] F --> F1[Pros: Full control
Domain-specific metrics
Cost optimization] F --> F2[Cons: Maintenance burden
Easy to get wrong
Reinvented wheels]

Tool	Setup time	Cost per eval	Best for
RAGAS	1 day	~$0.20 (GPT-4 judge)	Most production teams; standard metrics
Trulens	2 days	~$0.25	Teams already using Snowflake; need built-in dashboards
ARES	5+ days	~$0.10 (efficient batching)	Research, benchmarking, statistical rigor
Custom	2+ weeks	$0.005 (with fine-tuned judge)	Mature teams with cost pressure

The pragmatic path for most teams: start with RAGAS, instrument the four core metrics, ship to production. After three to six months, when you have real evaluation traffic and know which metrics matter for your domain, consider migrating to a custom pipeline with a fine-tuned judge for cost optimization. Don't try to build custom on day one. You don't yet know what to build.

---

Production Considerations

Three things tend to bite teams once their eval pipeline is in production.

The judge model is a dependency. Your faithfulness score is a function of judge model behavior. When you upgrade the judge, say, from Claude Sonnet 4.5 to 4.7, your scores will shift, sometimes meaningfully, even though nothing about your retrieval or generation changed. The fix: pin your judge model version, treat upgrades as "eval pipeline migrations," and re-baseline against a labeled holdout set whenever you upgrade.

Cost scales with traffic. A fixed sampling rate means evaluation cost grows with production traffic. Plan for that growth explicitly. Batched calls and a smaller judge model mostly absorb the increase, but at very high volumes you may need to drop sampling and accept lower statistical power on rare query types.

Drift detection requires baselines. A faithfulness score of 0.88 means nothing in isolation. It means a lot if last week it was 0.93. You need historical baselines, week-over-week comparisons, and automated change-point detection. Most teams I've worked with start with weekly Slack reports and graduate to PagerDuty alerts once they trust the signal. Skipping the trust-building phase usually leads to alert fatigue and the eval pipeline being silently disabled.

One last thing: include eval results in your model rollback decision flow. When you deploy a new prompt template or a new retriever, watch the eval metrics through the initial production soak window. If faithfulness drops beyond your pre-agreed rollback threshold, roll back automatically. This is the single most useful place to spend your eval signal. It turns evaluation from a passive dashboard into an active production safeguard.

---

Conclusion

RAG evaluation looks like a metrics problem. It's actually a discipline problem. The technical pieces (RAGAS, judge models, sampling pipelines) are all well-understood and well-supported in 2026. What separates teams that ship reliable RAG from teams that ship hallucinating RAG is the discipline of treating eval as a continuous production signal, the way you'd treat latency or error rate.

Start with the four core metrics: context precision, context recall, faithfulness, answer relevance. Sample real production traffic. Use a small, audited judge model. Wire alerts to the metrics that matter most for your product. Audit your judge quarterly. Roll back on regressions.

If you do those things, you'll catch the kind of silent quality decay that ends in postmortems. If you don't, you'll find out about your RAG quality the same way the team I opened with did: from a customer escalation, two months too late.

In the next post in this series I'll cover the next layer of production AI infrastructure: building observability for the agent loop itself, with traces that capture not just retrieval and generation but tool calls, planning steps, and self-correction cycles. The eval discipline scales further than retrieval, and it has to, because agents are where the next category of silent failure lives.

---

Revision History

Date	Summary	Old Version
2026-06-09	Revised unsupported quantitative claims, removed flagged quote formatting, and updated wording to satisfy the post-126 QA standards.	View original

---

Sources

• RAGAS: Automated Evaluation of Retrieval Augmented Generation, Es et al., 2024
• Trulens documentation: trulens.org/trulens_eval
• ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems, Saad-Falcon et al., 2024
• Anthropic Batch API documentation, docs.anthropic.com
• LangChain RAG evaluation guide, python.langchain.com
• Companion code repository, github.com/amtocbot-droid/amtocbot-examples

About the Author

Toc Am

Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.

LinkedIn X / Twitter

Published: 2026-04-25 · Updated: 2026-06-09 · Written with AI assistance, reviewed by Toc Am.

Get These In Your Inbox

Weekly deep-dives on AI engineering, no fluff. Join the newsletter →

Subscribe (free)

Or grab the book ($39, ~100 pages) · Buy me a coffee

☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter