Structured outputs: getting reliable JSON out of LLMs in production

Getting consistent JSON out of an LLM sounds trivial until you hit production at scale. Ask for structured JSON and you get inconsistent formats, malformed syntax, and unpredictable field names — and the problem compounds in production, where a prompt that works perfectly in testing starts failing after a model update, your parser breaks on unexpected field types, and your application crashes because the model renamed "status" to "current_state" without warning. The ecosystem has converged on a clear solution stack: native structured outputs at the model level, type-safe schema libraries, and a disciplined retry + escalation pattern. Here’s how each layer works and where it fits.

The two things “structured output” actually means

OpenAI JSON Mode guarantees only syntactical validity, whereas Structured Outputs goes further by enforcing strict adherence to a provided JSON Schema. That distinction matters in production. JSON Mode (type: json_object) only guarantees valid JSON syntax and is now considered “legacy.” Strict mode guarantees full schema adherence, and the current paradigm shift is schema-first development: define schemas in Zod (TypeScript) or Pydantic (Python) first, then build prompts around them.

There’s also a second form: tool/function calling with strict: true. OpenAI introduced Structured Outputs in two forms in the API — the first being function calling, where Structured Outputs via tools is available by setting strict: true within your function definition.

Function or tool calling is the other major structured-output shape, where the model chooses a name and passes arguments that should match a JSON Schema you control — and OpenAI recommends strict: true on tool definitions to keep arguments aligned with that schema.

Where each major provider stands today

OpenAI was first to the party and now treats JSON Mode as legacy. Structured Outputs solves the schema compliance problem by constraining OpenAI models to match developer-supplied schemas. On complex JSON schema following evals, gpt-4o-2024-08-06 with Structured Outputs scored a perfect 100%, compared to less than 40% for gpt-4-0613.

The feature adds a new failure mode operators need to handle: the model may return a refusal object instead of JSON. Key 2026 patterns include using Strict Mode exclusively and handling refusals as first-class errors.

Anthropic shipped native Structured Outputs later but caught up meaningfully. On November 14, 2025, Anthropic announced Structured Outputs in public beta for Claude Sonnet 4.5 and Opus 4.1.

The mechanism is constrained decoding: unlike prompting the model to “please return valid JSON,” structured outputs compile your JSON schema into a grammar and actively restrict token generation during inference. As of early 2026, supported models include Claude Opus 4.6, Claude Sonnet 4.6, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Haiku 4.5, and the feature is generally available on the Anthropic API and Amazon Bedrock.

Anthropic’s API exposes two modes: JSON outputs mode uses the output_format parameter for data extraction tasks like parsing emails or structuring unstructured data, with the response landing as guaranteed-valid JSON. Strict tool use mode adds strict: true to tool definitions, ensuring that when Claude calls functions, the parameters exactly match your input schema.

Schema limitations are consistent across providers. Schema limitations are similar between OpenAI and Anthropic: no recursive schemas, no complex enum types, and additionalProperties must be false on all objects. Design your schemas with these constraints in mind from day one.

The cross-provider layer: Instructor and Pydantic AI

For teams that don’t want to wire up each provider’s native API separately, two libraries have become the de facto standard.

Instructor is the most popular Python library for extracting structured data from LLMs, with over 3 million monthly downloads, 11k stars, and 100+ contributors.

It’s a Python library (with multi-language bindings and a Rust port) built on Pydantic for schema definitions, and it abstracts away prompting, parsing, retrying, and more.

Whether you’re using OpenAI’s GPT models, Anthropic’s Claude, Google’s Gemini, open-source models via Ollama, or DeepSeek, Instructor supports 15+ providers with a consistent API.

Pydantic AI takes a slightly different angle: the agent declares an output type (class Plan(BaseModel): ...), and the framework handles prompt formatting, validation, and the retry-with-error-feedback loop.

The current operator guidance from the Instructor maintainers themselves is to use Instructor for fast, schema-first extraction and reach for Pydantic AI when your project needs quality gates, shareable runs, or built-in observability.

If you’re managing multi-model workflows at a higher abstraction level, Vellum provides a graph-canvas orchestration layer with first-class versioning and observability on top of whichever structured-output provider you call underneath.

Building a production-grade validation + retry pipeline

Native structured outputs reduce failures dramatically, but they don’t eliminate them — refusals, context-length truncations, and cross-field semantic failures still occur. In production, structured output validation is a pipeline, not a toggle.

The practical pattern that operators converge on has three tiers:

Primary attempt with native structured output. Let constrained decoding do the heavy lifting. Most calls succeed here.
Retry with targeted error feedback. If validation fails, feed the error back to the LLM for one regeneration.

Pydantic provides detailed error messages indicating which fields are invalid and why, and these errors can inform retry logic, helping you construct better prompts that address specific validation failures. Instructor automates exactly this loop.

Graceful degradation or human escalation. If it fails again, return default values or move the task to a human-in-the-loop queue.

The right posture when validation fails: reject the payload, log why, retry with targeted feedback when the task is worth another attempt, and fail closed instead of coercing bad data into a queue.

A response can pass guardrails and fail validation (unparseable JSON), or pass validation and fail guardrails (clean JSON containing PII). Both layers are necessary. Don’t conflate schema compliance with data quality.

For teams building more complex orchestration — multi-step agent flows where one tool call’s JSON output feeds the next node’s input — LangGraph Cloud and CrewAI both have native support for passing structured tool outputs between nodes. n8n is worth considering for lower-code pipelines that need structured JSON as a pass-through between external APIs.

Schema design patterns that hold up under load

A few hard-won patterns that surface consistently across operator teams:

Keep schemas flat where possible. Deeply nested schemas fail more often at constrained decoding boundaries. If you need hierarchy, validate each level independently before assembling.

Make optional fields explicit. With strict: true, all properties must be listed in required. To make a field optional, use a union type with null — {"type": ["string", "null"]} — and in Pydantic, use Optional[str] with a default of None.

Snapshot real outputs as regression fixtures. Validate golden fixtures with jsonschema so every field in the contract is exercised, validate semantics with Pydantic and add adversarial cases like illegal enum strings and cross-field contradictions, and if you snapshot real model outputs, scrub PII and treat them as regression fixtures.

Wire schema checks into CI. If your team runs on the OpenAI stack, the Evals API includes structured-output evaluation recipes for testing tasks that depend on machine-readable formats. If you keep raw schema files in the repo, wire check-jsonschema into CI or pre-commit.

Operational risks operators underestimate

Prompt refinements might yield expected results 80–90% of the time, but in an enterprise system, the remaining 1–10% of output failure isn’t just a bug — it’s a business risk.

Downstream systems receiving unexpected data formats cause runtime errors and crashes. Users see null values or broken JSON strings in the UI. And engineering resources get drained by monitoring error logs and performing manual retries.

Model updates are the silent killer. A prompt that works perfectly in testing starts failing after a model update. Pin model versions in production, run your schema regression suite against every new model version before promoting, and treat constrained decoding as a floor, not a guarantee of semantic correctness.

Bottom line: Native structured outputs from OpenAI and Anthropic have genuinely solved the syntactic reliability problem — constrained decoding means the JSON either conforms to schema or the call fails cleanly, which is a better failure mode than silently malformed data. The remaining work lives at the semantic layer: designing schemas that capture actual business constraints, building three-tier retry pipelines that fail closed, and wiring schema compliance into CI the same way you’d wire type checks. Reach for Instructor or Pydantic AI to abstract provider differences; reach for Vellum if you need versioned orchestration and observability on top.