Valid JSON is only the first check on an LLM answer
Validate an LLM extraction beyond JSON syntax by checking schema, arithmetic, source evidence, and the difference between missing information and zero.
If you are turning an LLM answer into application data, successful JSON parsing is only the first check. A payload can be syntactically valid and still contain the wrong currency, an impossible total, or a value the source never supplied. Validation needs to follow the meaning of the fields all the way to the action that consumes them.
Consider a hypothetical invoice extraction. The model receives a document and returns supplier, currency, line items, and total. The application will create a draft for review. It will not make a payment. That limited action still needs accurate data because a misleading draft can waste review time or pass an error to another system.
Separate the validation layers
Start with syntax: can a JSON parser read the response? Next, check the schema: are required fields present, are numbers represented as numbers, and is the currency field in the allowed format? Then enforce domain rules, such as a nonnegative quantity or a total that agrees with the line items and documented adjustments.
Finally, check evidence. A total can pass arithmetic and still be invented. The source document needs to support the extracted values. If the model filled in a missing tax amount to make the total balance, the arithmetic check will not reveal the fabrication.
{
"currency": "USD",
"items": [
{"description": "Notebook", "quantity": 2, "unitPriceMinor": 500}
],
"taxMinor": 100,
"totalMinor": 1100,
"evidence": {"totalMinor": "page-1-total"}
}These are synthetic values using integer minor units for a currency whose chosen example has two decimal places. The row subtotal is 2 × 500 = 1000; adding 100 gives 1100. A production money model must use the currency’s actual minor-unit rules and its own rounding policy.
The stages reject different failures. A valid shape does not imply a valid sum, and a valid sum does not imply source support. Keeping these labels separate makes a review queue more useful than a single generic “invalid response” state.
Use structured output without overclaiming it
The OpenAI structured outputs guide, retrieved on September 20, 2026, documents constraining model output to supported JSON schemas. That feature can reduce formatting failures for supported models and schemas. It does not determine whether the model read the invoice correctly or whether the invoice itself is authentic.
Handle refusal and incomplete generation as separate outcomes according to the API contract. Do not try to parse every response path as a successful invoice object. A request that exceeded its output budget may need a controlled retry or manual review, while a refusal may call for a different user-facing explanation.
Even when the provider constrains the output shape, validate at your application boundary. Data may also arrive from stored records, test fixtures, alternate models, or a changed integration. Central validation gives those paths the same contract and produces a consistent failure mode if a schema changes unexpectedly.
Make missing information explicit
An absent tax amount is different from a tax amount of zero. If the source does not state tax, the extraction should preserve that uncertainty. A nullable field can represent absence, but a separate evidence status may be useful when you need to distinguish unreadable text from a field that is genuinely not present.
For example, taxMinor: null with taxStatus: "not_found" tells a reviewer something different from taxMinor: 0 with a cited “Tax 0.00” line. Avoid a schema default that silently replaces every missing amount with zero. Defaults are application decisions and should not erase evidence about the input.
The same issue appears with dates. A document might show only a month, or a date that is ambiguous between day-first and month-first notation. A schema requiring an exact ISO date can encourage a model to choose an unsupported interpretation. Design the contract to permit an unresolved date when that is the honest result.
Run a narrow total check
The following pure function operates after structural validation has established the keys, object shapes and string fields in the example. It is deliberately a total-specific domain and evidence check, not a replacement for a JSON Schema validator. This teaching invoice permits only USD, whole positive quantities and nonnegative integer amounts; it excludes discounts, credits, fractional quantities and currency conversion.
def check_total(invoice, source_totals):
if invoice["currency"] != "USD":
return "unsupported_currency"
if not invoice["items"]:
return "invalid_items"
subtotal = 0
for item in invoice["items"]:
quantity, price = item["quantity"], item["unitPriceMinor"]
if type(quantity) is not int or quantity <= 0:
return "invalid_quantity"
if type(price) is not int or price < 0:
return "invalid_price"
subtotal += quantity * price
tax, total = invoice["taxMinor"], invoice["totalMinor"]
if tax is None:
return "tax_unknown"
if any(type(x) is not int or x < 0 for x in (tax, total)):
return "invalid_amount"
if subtotal + tax != total:
return "total_mismatch"
reference = invoice["evidence"]["totalMinor"]
if reference not in source_totals:
return "unknown_reference"
if source_totals[reference] != total:
return "source_total_mismatch"
return "total_supported"Supply the JSON example as the invoice and a trusted fixture registry containing {"page-1-total": 1100}. The expected result is total supported. Changing only the total to 1200 fails arithmetic. Changing tax to 200 and total to 1200 restores arithmetic but fails the source-value comparison. The second mutation demonstrates why verifying that the citation identifier merely exists would be too weak.
The registry is a test author’s annotation of a particular source revision. In a real application, independently obtaining that value may involve a trusted document parser or a reviewer. Asking the same model to restate its invented total does not create independent evidence. Also, a supported total does not establish the supplier, line descriptions, tax or every quantity. Those fields require their own checks before the complete draft can be accepted.
For the zero-tax case, set tax to zero, total to 1000 and use a separate fixture whose source explicitly states that total. The function accepts it. Setting tax to null instead produces the unknown-tax result; it does not substitute zero. Python’s Boolean values require another boundary case because they are instances of integer subclasses: the exact type check rejects a quantity of true. The code still assumes structural validation and trusted integer values in the registry, so malformed object shapes are tested at the preceding layer.
Simon Willison’s February 28, 2025 article on LLM schemas shows a practical pattern for extracting repeatable record shapes from unstructured inputs. Its discussion of different provider mechanisms is also a reason to keep the consumer’s checks explicit. A reusable schema makes integration predictable; domain-specific source comparison supplies a different guarantee. The invoice function illustrates that second responsibility without claiming a measured extraction success rate.
Keep execution behind the checks
The function calling guide describes how a model can request an application operation. If an extraction later becomes an argument to a draft-creation tool, the executor should apply the same schema and domain checks before writing anything. The model’s prior confidence is not an authorization rule.
Bind evidence identifiers to records the application supplied. A generated identifier such as page-1-total is useful only if it maps to an actual source span. Reject references outside that set, and preserve the document revision so the reviewer sees the same source used during extraction.
For testing, include malformed JSON, an extra property, a negative quantity, a mismatched total, a missing tax field, and a fabricated evidence identifier. Add a valid case with an explicit zero tax amount to ensure the validator does not reject it merely because another case used null.
The acceptance result should say which layer failed and whether a retry could help. Repeating the same model request may repair a formatting failure, but it cannot manufacture a missing page. Route missing evidence to a request for the source or to review. The resulting draft should carry only values whose meaning and provenance the application can explain.