Structured Outputs: Getting Reliable JSON and Function Calls from LLMs

Asking Nicely Does Not Scale So far your triage service gets JSON by asking for it in the system prompt and forcing the shape with a { prefill. That combination is good. At volume it is still not good enough. Instruction-based formatting fails in a long tail of ways: a markdown fence wrapped around the object, a trailing explanatory sentence, a trailing comm...