Skip to content
AI Engineer Guide.
midPracticecurated

Your JSON parsing works in development but fails intermittently in production even at temperature 0. Diagnose it and describe the durable fix.

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

What does stop_reason equal max_tokens tell you, and why is repair the wrong response?
  • It means generation hit the output cap, so the object is truncated and no parser can recover the missing fields.
  • Repair is the wrong response because the cause is the budget, not the format; raise the cap or shorten the prompt and re-run.
  • Track truncation rate separately from validation failure rate, since they point at different fixes.
How does field order inside a schema change accuracy?
  • The decoder masks every token that would make the partial object invalid, so once the label field is open only label values are legal and the model cannot look ahead to evidence it has not written.
  • Put reasoning or evidence fields before the conclusion, and keep enums and short labels late, so the model conditions the answer on what it just wrote.
  • Test the same content with two field orders and compare field-level accuracy, because the effect is empirical rather than universal.

Red flags

  • Wraps json.loads in try/except and calls it fixed
  • Asserts a random seed guarantees identical output

Concepts this tests

Tags

More in Prompting & context engineering