midPracticecurated
Your JSON parsing works in development but fails intermittently in production even at temperature 0. Diagnose it and describe the durable fix.
30-second answer
- Temperature zero is greedy, not deterministic: batching, floating-point reduction order, mixture-of-experts routing and provider model updates all change the tokens, so occasional invalid JSON is expected.
- Reproduce with the raw response and the finish reason first, because a response cut off at max_tokens produces a validation error that looks like a schema bug.
- Fix it structurally with schema-constrained decoding so invalid tokens are masked, then validate with a schema library at the boundary and retry once with the validation error included.
- Make genuinely absent fields nullable, put reasoning before the label, and keep the schema flat, because strict schemas and constrained decoding can reduce quality or add latency on complex outputs.
- Parsing alone does not make the output true: a valid object can still carry the wrong total, so cross-field checks and field-level evals remain necessary.
Likely follow-ups
Open a question to rehearse the answer.
What does stop_reason equal max_tokens tell you, and why is repair the wrong response?
- It means generation hit the output cap, so the object is truncated and no parser can recover the missing fields.
- Repair is the wrong response because the cause is the budget, not the format; raise the cap or shorten the prompt and re-run.
- Track truncation rate separately from validation failure rate, since they point at different fixes.
How does field order inside a schema change accuracy?
- The decoder masks every token that would make the partial object invalid, so once the label field is open only label values are legal and the model cannot look ahead to evidence it has not written.
- Put reasoning or evidence fields before the conclusion, and keep enums and short labels late, so the model conditions the answer on what it just wrote.
- Test the same content with two field orders and compare field-level accuracy, because the effect is empirical rather than universal.
Red flags
- Wraps json.loads in try/except and calls it fixed
- Asserts a random seed guarantees identical output
Concepts this tests
Tags
More in Prompting & context engineering
- You stuff 60 retrieved documents into one prompt and the model uses facts from the beginning and end but ignores a crucial fact in the middle. Diagnose it and fix it.
- An agent reads retrieved documents and tool outputs and can take actions. How do you defend against indirect prompt injection, and why can't a better system prompt solve it?
- Your chain-of-thought prompt is not improving accuracy on a reasoning task. What do you check and what do you change?