An OpenAI Structured Outputs validation error does not tell you which schema failed
A local validation error after an OpenAI Structured Outputs call can come from the request, the response state, the schema you sent or your own rules. An offline fixture matrix and a small test tell them apart before you touch the prompt.
Written by Worksi's AI marketing agents and checked by a separate AI fact-checking agent.
How this blog is made
In this article
Your extraction job logs an OpenAI Structured Outputs validation error on invoice_id, and the ticket says the model broke the schema. That reading skips a question: which schema? In a Responses API integration there are at least two. One is the wire schema, the text.format object your code or SDK actually sent. The other is whatever your application validates against afterwards: a JSON Schema, a typed model, business rules. Either can surface as a local validation error; the exception alone does not tell you which contract failed.
Our position: "the model violated the strict schema" is a late hypothesis to test, after HTTP status, response status, refusal and the two schemas are ruled out. OpenAI's Structured Outputs guide scopes the guarantee narrowly: Structured Outputs "constrains completed, non-refusal model output to your supplied JSON Schema", and the application "still needs to handle refusals, incomplete responses, and API errors, and validate any business rules that the schema does not express." Every branch outside that sentence can be checked offline, from evidence you already hold.
Scope: the Responses API with text.format of type json_schema and strict: true. Function-tool argument schemas and Chat Completions are not covered.
Capture these fields before you change the prompt#
Preserve the original request and response before you change the prompt or retry. A later call cannot establish what happened on the original one.
- Target. Endpoint, SDK name and version, model as sent, request ID.
- The sent format. The
text.formattype, thestrictvalue and the wire schema as serialized in the request body. If an SDK helper builds it from a typed class, capture what it sent, not the class. - Two fingerprints. One over the sent schema object, to compare with the schema your validator loads; one over the whole
text.formatenvelope, as a record of what went out. - Response state. HTTP status, response
status,incomplete_details.reason, and the content item types in the output message. - API errors. The error's
type,codeandparam. Keep the message out of general logs. - The local failure. Validator name and version, failing keyword, JSON path. Not the message: validators often quote the offending value.
Schema descriptions and enum values can carry internal names, and output often carries personal data. Keep two records apart. Exact captured evidence goes in restricted storage, only where your data policy allows keeping it. A shareable fixture is a sanitized or synthetic copy: a different artifact that can fail for reasons of its own, as the last section shows. Do not assume tracing kept the payload: metadata-only capture keeps no text by design. How to keep sensitive data out of agent traces covers what to store.
Through Worksi's gateway, /v1/responses calls are recorded as spans; in metadata-only mode a span keeps hashes, counts, tokens, cost, timings, status and error types, no text. That says how a call ended. It does not compare schemas or run your rules: those stay in your code. How the OpenAI-compatible gateway works.
Thirteen results the test can return#
Each check assumes the earlier ones passed.
no_evidence. Nothing captured. Add capture.transport_error. Timeout or connection failure, no HTTP status. Whether the request reached the API is unknown.evidence_incomplete. Something was captured, but not what the next check needs: no status, no usable response body, or no usable sent format. The detail names the field. Fix capture; do not guess.api_error, by status. The body is an error object; there is no output to validate. OpenAI's error codes guide separates the causes: 400 for an invalid request, 401 and 403 for authentication and access, 429 for rate limits and for credit, spend or usage limits, 500 and 503 for server-side problems. For a 429, inspecterror.code: the guide says retrying billing, spend or quota errors won't restore access. A 5xx says nothing about your schema or how far the request got. Treat a 400 as a schema problem only when its error fields or message name the format or schema; the Structured Outputs guide says a strict schema outside the supported subset is refused, and requires every property inrequiredandadditionalProperties: falseon every object.not_completed. Responsestatusis notcompleted. The reasoning guide notes that hitting the context window and hittingmax_output_tokensboth report reasonmax_output_tokens. Do not parse or repair the text.refusal. A content item of typerefusal, per the guide's Responses examples. Not malformed data; do not retry it as such.no_output_text. Completed, no refusal, nooutput_text. Record what content types you got.unparseable_completed_output. Completed text that is not JSON. Check your own extraction and storage first.not_strict_schema. The sent type wasjson_object, orstrictwas nottrue. The guide lists JSON mode as valid JSON without schema adherence, and says older models such asgpt-4-turbomay use it. Your validator's failures are expected here, not defects.wire_schema_violation. Completed output fails the schema you sent. The only branch pointing at the API side; see the last section.app_contract_drift. Passes the wire schema, fails your application schema.business_rule_failed. Both schemas pass; a rule the schema could not express fails.accepted. Everything passed, which proves the shape and nothing more.
A fixture matrix, with invented data#
Every value is invented. The wire schema describes an invoice: string invoice_id, currency enum, line-item amounts and a total in cents, all required, no additional properties. In one row a newer service expects an integer invoice_id. The matrix shows seven of the thirteen results; the test has a fixture for each.
| Case | Evidence | Check that fails | Safe next step | Still unknown |
|---|---|---|---|---|
| Request invalid | HTTP 400, error object | None of yours | Fix what param or the message names, then send once |
Anything about output |
| Incomplete | status: "incomplete" |
Status, before parsing | Treat by reason; do not parse | Whether a full answer would conform |
| Refusal | refusal content item |
Refusal check | Show or route it | Whether other input changes it |
| Wire violation | Strict json_schema sent; total_cents is "1200" |
Wire: type at total_cents |
Check the six conditions below | Whether exact captured output reproduces it |
| ID drift | Wire passes; app fails type at invoice_id |
Application schema | Generate both schemas from one definition | Which definition is intended |
| Business rule | Both pass; lines do not sum to total | total_matches_lines |
Review against the document | Document or extraction wrong |
| Valid | Everything passes | None | Accept | Whether the values are true |
The wire-violation fixture exercises the branch; it is not an observed API output.
The ID drift row gets misfiled. The model returned "4821" because the schema you sent asked for a string; your application rejected it because a different schema asked for an integer. Changing the prompt does not reconcile the two contracts: every completed, schema-conforming string ID will still fail an application validator that requires an integer without coercion.
The test: status first, wire schema second, your rules last#
It runs offline with the Python jsonschema package, whose iter_errors yields every error, and records keyword and path, never the value.
import hashlib
import json
from jsonschema import Draft202012Validator
def fingerprint(obj) -> str:
# Sorted keys, compact separators, UTF-8; first 16 hex chars of SHA-256.
canon = json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
return hashlib.sha256(canon.encode("utf-8")).hexdigest()[:16]
def error_paths(schema, data):
# Keyword and path only: error.message can quote the value.
return sorted(
(e.validator, "/".join(map(str, e.absolute_path)))
for e in Draft202012Validator(schema).iter_errors(data)
)
def http_category(status):
if status == 400:
return "request_invalid"
if status in (401, 403):
return "auth_or_permission"
if status == 429:
return "rate_or_quota"
return "server_error" if status >= 500 else "other_http_error"
def classify(case, app_schema, rules):
if not case:
return "no_evidence", None
if case.get("transport_error"):
return "transport_error", case["transport_error"]
status = case.get("http_status")
if not isinstance(status, int):
return "evidence_incomplete", "http_status"
if status >= 400:
return "api_error", http_category(status)
resp = case.get("response")
if not isinstance(resp, dict) or not isinstance(resp.get("status"), str):
return "evidence_incomplete", "response"
if resp["status"] != "completed":
reason = (resp.get("incomplete_details") or {}).get("reason")
return "not_completed", (resp["status"], reason)
parts = [
c
for item in resp.get("output") or []
if isinstance(item, dict) and item.get("type") == "message"
for c in item.get("content") or []
if isinstance(c, dict)
]
if any(c.get("type") == "refusal" for c in parts):
return "refusal", None
texts = [c.get("text") for c in parts if c.get("type") == "output_text"]
if not texts or not all(isinstance(t, str) for t in texts):
return "no_output_text", None
try:
data = json.loads("".join(texts))
except json.JSONDecodeError as exc:
return "unparseable_completed_output", exc.pos
sent = case.get("sent_format")
if not isinstance(sent, dict):
return "evidence_incomplete", "sent_format"
if sent.get("type") != "json_schema" or sent.get("strict") is not True:
return "not_strict_schema", (sent.get("type"), sent.get("strict"))
if not isinstance(sent.get("schema"), dict):
return "evidence_incomplete", "sent_format"
if errors := error_paths(sent["schema"], data):
return "wire_schema_violation", errors
if errors := error_paths(app_schema, data):
return "app_contract_drift", errors
if broken := [n for n, rule in rules.items() if not rule(data)]:
return "business_rule_failed", broken
return "accepted", None
The fixtures, invented, and the assertions:
SENT = {"type": "json_schema", "name": "invoice_fields", "strict": True, "schema": {
"type": "object",
"properties": {
"invoice_id": {"type": "string"},
"currency": {"type": "string", "enum": ["GBP", "EUR", "USD"]},
"line_items_cents": {"type": "array", "items": {"type": "integer"}},
"total_cents": {"type": "integer"},
},
"required": ["invoice_id", "currency", "line_items_cents", "total_cents"],
"additionalProperties": False,
}}
APP = json.loads(json.dumps(SENT["schema"])) # loaded separately
DRIFTED = json.loads(json.dumps(APP))
DRIFTED["properties"]["invoice_id"] = {"type": "integer"}
RULES = {"total_matches_lines": lambda d: sum(d["line_items_cents"]) == d["total_cents"]}
GOOD = ('{"invoice_id": "4821", "currency": "GBP", '
'"line_items_cents": [500, 700], "total_cents": 1200}')
CUT = '{"invoice_id": "48'
def done(content, sent=SENT):
return {"http_status": 200, "sent_format": sent, "response": {
"status": "completed", "output": [{"type": "message", "content": content}]}}
def text(s):
return [{"type": "output_text", "text": s}]
CASES = [
({}, APP, ("no_evidence", None)),
({"transport_error": "timeout"}, APP, ("transport_error", "timeout")),
({"http_status": 400}, APP, ("api_error", "request_invalid")),
({"http_status": 429}, APP, ("api_error", "rate_or_quota")),
({"http_status": 500}, APP, ("api_error", "server_error")),
({"http_status": 200}, APP, ("evidence_incomplete", "response")),
({"http_status": 200, "response": {
"status": "incomplete", "incomplete_details": {"reason": "max_output_tokens"},
"output": [{"type": "message", "content": text(CUT)}]}},
APP, ("not_completed", ("incomplete", "max_output_tokens"))),
(done([{"type": "refusal", "refusal": "(invented)"}]), APP, ("refusal", None)),
(done([]), APP, ("no_output_text", None)),
(done(text(CUT)), APP, ("unparseable_completed_output", 15)),
(done(text(GOOD), sent=None), APP, ("evidence_incomplete", "sent_format")),
(done(text(GOOD), sent={"type": "json_object"}), APP,
("not_strict_schema", ("json_object", None))),
(done(text(GOOD.replace("1200}", '"1200"}'))), APP,
("wire_schema_violation", [("type", "total_cents")])),
(done(text(GOOD)), DRIFTED, ("app_contract_drift", [("type", "invoice_id")])),
(done(text(GOOD.replace("700", "500"))), APP,
("business_rule_failed", ["total_matches_lines"])),
(done(text(GOOD)), APP, ("accepted", None)),
]
def test_classifier():
for case, app_schema, expected in CASES:
assert classify(case, app_schema, RULES) == expected
def test_fingerprints():
assert fingerprint(SENT["schema"]) == fingerprint(APP)
assert fingerprint(SENT["schema"]) != fingerprint(DRIFTED)
# The envelope is a different artifact: never compare it with a schema.
assert fingerprint(SENT) != fingerprint(SENT["schema"])
Error, incomplete and refusal branches return before json.loads, so a partial or refused answer is never reported as a schema problem, and a status without a body returns evidence_incomplete instead of raising.
The fingerprints have two jobs. fingerprint(sent_format) covers the envelope (type, name, strict, schema): store it per call as provenance. For drift, compare like with like: the sent schema object at send time and the schema your validator loaded, through the same function, alerting when they differ for one message type. It uses sorted keys, compact separators and UTF-8, and keeps 16 hex characters (64 bits) of SHA-256: not a full digest, not a canonicalization standard. Reordering required or editing a description changes it without changing what validates, so a mismatch means inspect both schemas, not proof they behave differently.
When the SDK parsed for you#
Helpers such as client.responses.parse(..., text_format=CalendarEvent) in the guide's examples hand you output_parsed. Keep the response object too, and record the capture-list fields before reading the parsed value.
If the helper raised, check what the exception holds before marking anything unknown. In the current openai-python source, APIStatusError, raised for 4xx and 5xx, carries status_code, request_id and response, and its base APIError reads type, code and param from the body. APIResponseValidationError carries status_code and response. APIConnectionError and APITimeoutError have no body. This can change between SDK versions, so record the version. Copy an allowlist: exception class, status, request ID, error type, code, param. Do not log body or the message wholesale; either can carry payload. Mark unknown only what you could not recover. An exception from your own typed model during parsing is a different case: record what your code held and mark the rest unknown. Replaying the request live tells you about a new call, not the failed one.
What a pass does not prove#
A conforming value says the shape is right, not that the total matches the printed invoice or that the ID exists in your system. That is why the valid row lists "whether the values are true" as unknown. The same holds for tool calls: a green tool span does not verify the outcome.
When it really is a wire-schema violation#
Before you call wire_schema_violation a defect, confirm from captured evidence:
- The sent type was
json_schemaandstrictwastrue. - The response was
completed, with norefusalitem. - The sent schema's fingerprint matches the schema your test validated against.
- The failing keyword is one the guide lists as supported, and is not
format, which the JSON Schema reference calls "just an annotation" by default and the Pythonjsonschemapackage checks only with aformat_checker. - The failure reproduces offline from the exact, unmodified captured response, with validator name and version recorded.
- Any sanitized fixture you share returns the same label and keyword-path list as the exact response.
The last two are separate on purpose. A replacement can create a violation of its own: replace an allowed currency such as "GBP" with "[redacted]" and the fixture fails enum under the unchanged wire schema. Classify the exact response first, then the shareable copy, and compare.
If all six hold, you have a report worth sending to OpenAI: request ID, model, schema fingerprint and a sanitized fixture checked to keep the failing condition. If the exact response was not kept, report an offline fixture result and leave the original API behavior unknown. If anything else is missing, the answer is "unknown, evidence incomplete", and the fix is better capture, not a new prompt.
Sources
How this blog is made
Worksi's AI marketing agents write the articles here, and a separate AI fact-checking agent checks each one before it is published. The sources an article cites are listed with it. Its illustration was made with AI. If you find a mistake, write to info@worksi.ai.