The first time I shipped a structured extraction pipeline, the output looked right in testing. The model returned valid JSON, the fields were present, and the types matched. We went to production with confidence.
Three days in, the pipeline started silently dropping records. The model was returning valid JSON, but the confidence field was sometimes a string ("high") and sometimes a float (0.87). Downstream code expected a float. No exception. Just silent None values propagating into the database.
JSON mode gives you syntactically valid JSON. It does not give you schema-correct JSON. That distinction, which seems obvious in hindsight, is the source of almost every structured output bug I have seen in production.
This post covers the full stack: what JSON mode and structured outputs actually guarantee, how to write schemas that constrain the output correctly, how to validate and retry without hammering the API, and what breaks in non-obvious ways when your document volume grows.
The Problem: Valid JSON Is Not the Same as Correct JSON
When you enable JSON mode on OpenAI or set response_format: {"type": "json_object"}, the model is constrained to produce text that can be parsed as JSON. That is all. The constraint is syntactic, not semantic.
Consider this schema for an extraction task:
from pydantic import BaseModel
from typing import Literal
class ExtractionResult(BaseModel):
entity_name: str
entity_type: Literal["person", "organization", "location"]
confidence: float # 0.0–1.0
source_sentence: str
requires_review: bool
JSON mode will produce output that parses. It will not guarantee:
entity_typeis one of the three literalsconfidenceis a float between 0 and 1 (not a string, not > 1.0)requires_reviewis a boolean (not"true"or"yes")source_sentenceis non-empty
In our pipeline, we measured roughly 4% of JSON-mode responses failing at least one of these constraints on a corpus of 10,000 documents. That sounds small. At 10,000 documents per day, it is 400 silent data quality failures.
The fix is not to retry more aggressively. The fix is to use schema-constrained generation, and to validate every response regardless.
How Structured Outputs Actually Work
There are three distinct mechanisms for getting structured output from LLMs. They are not equivalent.
1. JSON Mode (response_format: json_object)
Constrains the model to produce valid JSON at the tokenization layer. No schema awareness. The model sees your schema description in the system prompt and tries to follow it, but there is no enforcement.
What it guarantees: parseable JSON.
What it does not guarantee: field names, field types, required fields present, enum values respected.
2. Function Calling / Tool Use
The model selects a function and fills in its parameters according to a JSON Schema definition. The schema is sent to the model alongside the messages, and the API enforces that the output matches the schema structure.
What it guarantees: fields declared in the schema are present with the right types (for most providers). Enum values for string fields are respected.
What it does not guarantee: numeric range constraints (minimum, maximum), string pattern constraints (pattern), semantic correctness.
3. Structured Outputs (OpenAI response_format: json_schema)
Per OpenAI's documentation, this mode uses constrained decoding: the token sampling is filtered at each step to only allow tokens that could lead to a valid completion of the schema. This is the strongest guarantee available for JSON.
What it guarantees: output matches the schema exactly, including required fields, types, and enum values. Per OpenAI's documentation, additionalProperties: false is enforced.
What it does not guarantee: semantic correctness, numeric ranges, or string content validity.
Anthropic's tool use provides similar schema enforcement to OpenAI's function calling: the response must match the declared input_schema. For extraction tasks, we wrapped our schema as a single tool definition and always forced a tool call, which is the most reliable pattern we found across both providers.
Implementation: The Right Pattern for Anthropic's Tool Use
For extraction pipelines on Anthropic, the most reliable pattern we found is to define the schema as a tool with input_schema, disable all other tools, and force a tool call every time. This gives you schema enforcement at the API layer, not just at the prompt layer.
import anthropic
from pydantic import BaseModel, ValidationError, field_validator
from typing import Literal
import json
client = anthropic.Anthropic()
# Define the schema both as a Pydantic model (for validation)
# and as a JSON Schema dict (for the tool definition)
class ExtractionResult(BaseModel):
entity_name: str
entity_type: Literal["person", "organization", "location"]
confidence: float
source_sentence: str
requires_review: bool
@field_validator("confidence")
@classmethod
def confidence_must_be_fraction(cls, v: float) -> float:
if not 0.0 <= v <= 1.0:
raise ValueError(f"confidence must be between 0 and 1, got {v}")
return v
@field_validator("source_sentence")
@classmethod
def source_must_be_nonempty(cls, v: str) -> str:
if not v.strip():
raise ValueError("source_sentence must not be empty")
return v
EXTRACTION_TOOL = {
"name": "extract_entity",
"description": "Extract a named entity from the text with metadata.",
"input_schema": {
"type": "object",
"properties": {
"entity_name": {
"type": "string",
"description": "The exact text of the named entity as it appears"
},
"entity_type": {
"type": "string",
"enum": ["person", "organization", "location"],
"description": "The category of the entity"
},
"confidence": {
"type": "number",
"description": "Confidence score from 0.0 to 1.0"
},
"source_sentence": {
"type": "string",
"description": "The sentence from which the entity was extracted"
},
"requires_review": {
"type": "boolean",
"description": "True if the extraction is uncertain or ambiguous"
}
},
"required": [
"entity_name", "entity_type", "confidence",
"source_sentence", "requires_review"
]
}
}
def extract_entity(text: str, max_retries: int = 2) -> ExtractionResult | None:
messages = [{"role": "user", "content": text}]
last_error: str | None = None
for attempt in range(max_retries + 1):
# On retry, inject the previous error as context
if last_error and attempt > 0:
messages = [
{"role": "user", "content": text},
{"role": "assistant", "content": [
{"type": "tool_use", "id": "retry", "name": "extract_entity",
"input": {}}
]},
{"role": "user", "content": [
{"type": "tool_result", "tool_use_id": "retry",
"content": f"Validation error: {last_error}. Please correct and retry."}
]}
]
response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=1024,
system=(
"You are an entity extraction assistant. "
"Always call the extract_entity tool with your answer."
),
tools=[EXTRACTION_TOOL],
tool_choice={"type": "tool", "name": "extract_entity"},
messages=messages
)
# Extract the tool input from the response
tool_block = next(
(b for b in response.content if b.type == "tool_use"),
None
)
if not tool_block:
last_error = "No tool call in response"
continue
try:
result = ExtractionResult(**tool_block.input)
return result
except (ValidationError, TypeError) as e:
last_error = str(e)
continue
return None # Dead letter
The key decisions here:
-
tool_choice: {"type": "tool", "name": "..."}forces a specific tool call. Without this, the model may respond with text instead of a tool call, especially on simple inputs. -
Pydantic validation runs after API schema enforcement. The API ensures structural correctness; Pydantic catches semantic constraints (range, non-empty, pattern).
-
Retry with error feedback. On validation failure, the previous error is sent back to the model as a
tool_result. Per Anthropic's documentation, this is the correct continuation pattern (the model sees the error and can adjust its next attempt).
The Debugging Story: When Enum Values Silently Expand
Six weeks into our production pipeline, we noticed entity_type values like "org", "company", and "institution" appearing in the database. The schema declared "organization" as the only valid value. The API was not enforcing it.
The root cause: we had upgraded the model version and slightly reworded the system prompt. The new system prompt said "organization or company" in a few examples. The model started treating these as valid alternatives. The API schema enforcement for tool use checks that the key entity_type is present, but on older Anthropic API versions, enum validation in input_schema was advisory, not enforced.
We confirmed this by sending a test message that should have returned "organization" and checking whether "org" was accepted. It was.
The fix was two-part: add explicit Pydantic validation for the enum (which we already had, but had mistakenly excluded from the retry path), and pin the model version so prompt changes required explicit testing.
# Monitoring: log rejection reasons by field
import logging
from collections import Counter
rejection_counts: Counter = Counter()
def extract_with_monitoring(text: str) -> ExtractionResult | None:
try:
result = extract_entity(text)
if result is None:
rejection_counts["exhausted_retries"] += 1
return result
except Exception as e:
# Parse the ValidationError to find which field failed
err_str = str(e)
for field in ["entity_type", "confidence", "source_sentence", "requires_review"]:
if field in err_str:
rejection_counts[f"field:{field}"] += 1
logging.error("Extraction failed: %s | text: %s", e, text[:100])
return None
Log rejection_counts to your metrics system every hour. If field:entity_type starts climbing, your model or prompt drifted. If field:confidence climbs, you have a model that started returning string confidence values. Check your few-shot examples for implicit type coercion.
Schemas That Actually Constrain: What to Include and What to Skip
Not all JSON Schema properties are enforced by all providers. Knowing which constraints are enforced saves you from writing validation rules that the API silently ignores.
Enforced by Anthropic tool use (input_schema):
- type: string, number, integer, boolean, array, object
- required: all listed fields must be present
- enum: for string fields (as of mid-2026; verify with your model version)
- items: for array fields
Not reliably enforced (use Pydantic instead):
- minimum, maximum: numeric range constraints
- minLength, maxLength: string length constraints
- pattern: regex constraints on strings
- minItems, maxItems: array length constraints
This means your input_schema should declare structure and type. Your Pydantic model should enforce value constraints. The two layers complement each other rather than duplicating.
# What goes in input_schema (API-enforced)
"confidence": {
"type": "number", # enforced
"description": "0.0–1.0" # hint only, not enforced
# minimum/maximum NOT reliable here
}
# What goes in Pydantic (always enforced)
@field_validator("confidence")
@classmethod
def confidence_range(cls, v: float) -> float:
if not 0.0 <= v <= 1.0:
raise ValueError(f"Expected 0.0–1.0, got {v}")
return v
Production Considerations: Retry Budgets, Dead Letters, and Schema Drift
Retry budget
Our rule: we measured maximum 2 retries per document (3 attempts total) as the inflection point. At 2 retries, our empirical rejection rate dropped to under 0.1% on well-formed inputs. A third retry rarely changes the outcome and triples the cost on a bad document.
RETRY_CONFIG = {
"max_retries": 2,
"initial_backoff_ms": 100,
"backoff_multiplier": 2.0,
"dead_letter_threshold": 3, # consecutive failures triggers alert
}
Dead letter queue
Documents that exhaust retries go to a dead letter queue rather than being silently dropped. We write the original text, the last error, and the raw model response to a separate table. A daily job reviews these (roughly 0.05% of volume) and feeds representative failures back as few-shot examples.
def handle_dead_letter(text: str, last_error: str, raw_response: str) -> None:
db.insert("extraction_dead_letters", {
"text": text,
"error": last_error,
"raw_response": raw_response,
"created_at": "now()",
"reviewed": False,
})
# Alert if dead letter rate exceeds threshold
rate = db.query("SELECT count(*) FROM extraction_dead_letters "
"WHERE created_at > now() - interval '1 hour'")
if rate > DEAD_LETTER_ALERT_THRESHOLD:
alert("Dead letter rate elevated", rate=rate)
Schema drift detection
Models update. Prompts change. The distribution of your input documents shifts. Any of these can cause your validation pass rate to degrade over time without a sudden failure event.
Track your validation pass rate per model version, and alert on week-over-week degradation. We log a validation_pass metric on every extraction call, tagged with the model ID and schema version. In our pipeline, a drop of more than a few percentage points over a rolling week reliably signals prompt or model drift that warrants a prompt audit.
Comparison: JSON Mode vs Function Calling vs Structured Outputs
| Capability | JSON Mode | Function Calling | Structured Outputs |
|---|---|---|---|
| Syntactic JSON guarantee | Yes | Yes | Yes |
| Required fields enforced | No | Partial | Yes |
| Enum values enforced | No | Partial | Yes |
| Numeric range enforced | No | No | No |
| Semantic correctness | No | No | No |
| Multi-schema in one call | N/A | Yes (multiple tools) | One schema |
| Works with streaming | Yes | Partial | Partial |
| Provider support | Anthropic, OpenAI | Anthropic, OpenAI | OpenAI (mid-2024+) |
The practical recommendation: use function calling / tool use with tool_choice forced and Pydantic validation on every provider. Move to Structured Outputs (OpenAI json_schema mode) when you need the strongest API-level guarantee and you are on OpenAI's supported models. Add Pydantic in both cases for semantic validation that the API cannot enforce.
Conclusion
JSON mode is a starting point, not a solution. The schema-correct, semantically-valid structured output you need in production requires three layers: API-level schema enforcement (tool use or structured outputs), application-level semantic validation (Pydantic), and operational tooling (retry budget, dead letter queue, schema drift monitoring).
The one metric worth tracking from day one: validation pass rate tagged by field and model version. When it drops, you have a concrete signal (a specific field is failing) rather than a vague "the pipeline is broken."
The working code for this post, including the full extraction pipeline with retry logic and monitoring, is in the companion repo at github.com/amtocbot-droid/amtocbot-examples/tree/main/274-structured-outputs.
Get the next one
One email a week: a production failure dissected, with the full fix and the code. If you build extraction or agent pipelines, it is worth reading. No spam, unsubscribe anytime.
Reader challenge: ship the retry pattern above with field-level rejection logging. Reply to the email with which field fails most often in your pipeline. The most interesting failure mode becomes the next post.
Sources
- Anthropic Tool Use Documentation — Forcing Tool Use — covers
tool_choiceparameter andinput_schemastructure for constrained extraction - OpenAI Structured Outputs Guide — documents
json_schemaresponse format and which JSON Schema keywords are enforced - Pydantic v2 Validators Documentation —
field_validatorandmodel_validatorpatterns for post-schema semantic checks
About the Author
Toc Am
Founder of AmtocSoft. Writing practical deep-dives on AI engineering, cloud architecture, and developer tooling. Previously built backend systems at scale. Reviews every post published under this byline.
Published: 2026-07-01 · Written with AI assistance, reviewed by Toc Am.
Get These In Your Inbox
Weekly deep-dives on AI engineering, no fluff. Join the newsletter →
Or grab the book ($39, ~100 pages) · Buy me a coffee
☕ Buy Me a Coffee · 🔔 YouTube · 💼 LinkedIn · 🐦 X/Twitter
