Guide · Prompting & Output Craft

Why does the output format drift, and how do you stop it?

Published 1 August 2026

The prompt worked for a week. Then one response arrived wrapped in a code fence, another added a friendly sentence before the JSON, and a third renamed a field. Nothing changed on your side. Output format drift is not randomness — it has a small number of identifiable causes, and all of them are fixable without begging the model to behave.

Why format drifts

  • The format was described, not constrained. "Return JSON with a title and tags" is a request. A schema is a contract. Requests are honoured most of the time, which is the worst possible reliability profile.
  • The input changed shape. Long inputs, unusual inputs and inputs containing their own formatting push the model toward imitating the input rather than your instruction. Drift often correlates with input length, not with time.
  • Instructions compete. A system prompt asking for a friendly tone and a user message asking for bare JSON will produce friendly JSON. The preamble is not disobedience; it is the tone rule winning.
  • Your examples disagree with your rule. One few-shot example with a trailing comment teaches that comments are acceptable. Examples outrank prose every time.
  • Distance in the window. A format rule stated once, forty turns ago, competes with everything since. Format rules belong in standing context, not in a transcript that scrolls.
  • The model changed. A version upgrade shifts default verbosity. If drift starts abruptly and correlates with nothing in your code, check the model identifier you are actually sending.

Fix it structurally first

  1. Use the mechanism, not the sentence. If your provider offers a structured-output or tool-call schema, use it. That moves the constraint from persuasion to enforcement and eliminates the whole class of preamble-and-fence bugs.
  2. Validate and reject, never coerce. Parse against the schema; on failure, retry once with the validation error appended. Do not strip fences and hope — a repair layer that always succeeds hides a rising error rate.
  3. Make the tolerant parse a logged event. If you must strip fences for legacy reasons, count it. The metric "responses needing repair" is your drift detector.
  4. Separate prose from data. If you need both an explanation and machine-readable output, give them separate fields. Asking for JSON "with a short note before it" guarantees an unparseable response eventually.

Then fix the prompt

  • State the negative explicitly, once. "Output only the JSON object. No code fences, no preamble, no trailing commentary." Vague accuracy pleas do nothing; specific prohibitions of the exact observed failure do measurably help.
  • Put the format rule last. Whatever sits closest to the generation boundary carries the most weight.
  • Make every example exact. Two examples that are byte-identical in shape are worth more than five that vary. If your examples include an optional field sometimes, the model treats the whole schema as optional.
  • Show the empty and error cases. Most drift appears on edge inputs, because the format was only ever demonstrated on happy ones. Include an example of the empty result.
  • Remove the conflicting instruction. Cheerfulness and bare JSON cannot both win. Decide which one this endpoint needs.

A worked example

An extraction endpoint returning { items: [...] } started emitting fenced blocks on roughly one call in thirty. The pattern, once logged: every failure had an input longer than about 4,000 tokens, and the long inputs were pasted Markdown documents. The model was mirroring the fenced code blocks in the input.

What fixed it: moving to a tool-call schema so the format could not be expressed in prose at all, plus a rejecting parser with one retry, plus a counter on repairs. The prompt wording was never changed. The repair counter went from about 3% to 0.1%, and the residual failures turned out to be genuinely truncated responses — a different bug, now visible because the first one stopped masking it.

Watch for the near-miss failures

Total format breakage is easy; the expensive drift is subtler and passes a lax parser:

  • A field renamed to a synonym — tags becomes labels.
  • A number sent as a string, or a boolean as "true".
  • A single object where the schema says array, when there is exactly one result.
  • null versus omitted versus empty string, used interchangeably.
  • An enum value that is plausible but not in your set.

Strict validation catches all five. A permissive parser turns each one into a downstream bug you will debug three layers away from its cause.

Review checklist

  • Is the format enforced by a schema, or only described in text?
  • Does validation reject unknown fields and out-of-set enums?
  • Do you count repaired and rejected responses over time?
  • Do your examples agree with each other and with the rule?
  • Is there any instruction in the prompt that competes with the format?
  • Does the drift correlate with input length or with a model version change?

Where this fits

Format stability sits in the Prompting & Output Craft part of the track, next to structured output and tool use. The judgement being tested is whether a constraint is enforced or merely requested — and whether you would notice the day it starts failing.

Keep going