Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 16 of 28

Stop sequences

0 of 28 complete

Why it matters

A model will continue past the end of the JSON object, the answer or the turn, because continuation is its default. The application needs a deterministic boundary so it can parse, route or hand off. Stop sequences provide that boundary without retraining, but they are a lexical match against generated text, so they fail in the same way any pattern match fails: on legitimate content that contains the pattern.

Mental model

A stop sequence is a period at the end of a form field: it tells the writer where to stop, not what to write. The comparison breaks because the model may produce the period as part of the content, and then the form ends mid-sentence.

How it works

Mechanism, step by stepInteractive
  1. 1Decide the boundary the parser needs: end of a JSON object, end of a list, end of a turn marker, or a sentinel you control.
  2. 2Prefer a sentinel that cannot occur in normal content, and instruct the model to emit it only at the end.
  3. 3Configure the stop sequence in the request and verify the marker is excluded from the returned text.
  4. 4Cap output tokens as a backstop, since a stop string may never appear.
  5. 5Handle the case where generation ends by length rather than by stop, and treat that as an incomplete result.
  6. 6Test with inputs likely to contain the marker, such as documents that quote your own prompt format.

Step 1 of 6

Decide the boundary the parser needs: end of a JSON object, end of a list, end of a turn marker, or a sentinel you control.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Decide the boundary the parser needs: end of a JSON object, end of a list, end of a turn … 2. Prefer a sentinel that cannot occur in normal content, and instruct the model to emit it … 3. Configure the stop sequence in the request and verify the marker is excluded from the ret… 4. Cap output tokens as a backstop, since a stop string may never appear. 5. Handle the case where generation ends by length rather than by stop, and treat that as an… 6. Test with inputs likely to contain the marker, such as documents that quote your own prom…

  1. Decide the boundary the parser needs: end of a JSON object, end of a list, end of a turn marker, or a sentinel you control.
  2. Prefer a sentinel that cannot occur in normal content, and instruct the model to emit it only at the end.
  3. Configure the stop sequence in the request and verify the marker is excluded from the returned text.
  4. Cap output tokens as a backstop, since a stop string may never appear.
  5. Handle the case where generation ends by length rather than by stop, and treat that as an incomplete result.
  6. Test with inputs likely to contain the marker, such as documents that quote your own prompt format.

Trade-offs

ChoiceGood forCostFails when
Sentinel stringUnambiguous boundary, easy to parseModel may forget to emit itGeneration is long and the instruction is buried
Structural stop such as a closing braceNo extra instruction neededNested structures end earlyThe output contains the same delimiter inside a string
Token-level stopExact and cheap to detectModel-specific ids, brittle across tokenisersYou need portability across providers
Max token cap onlyAlways terminatesCuts mid-sentence with no signalThe consumer parses the output
Trade-off mapInteractive

Good for

Unambiguous boundary, easy to parse

Cost

Model may forget to emit it

Fails when

Generation is long and the instruction is buried

  1. Generation is long and the instruction is buried
  2. The output contains the same delimiter inside a string
  3. You need portability across providers
  4. The consumer parses the output

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Sentinel string is good for Unambiguous boundary, easy to parse, costs Model may forget to emit it, and fails when Generation is long and the instruction is buried. Structural stop such as a closing brace is good for No extra instruction needed, costs Nested structures end early, and fails when The output contains the same delimiter inside a string. Token-level stop is good for Exact and cheap to detect, costs Model-specific ids, brittle across tokenisers, and fails when You need portability across providers. Max token cap only is good for Always terminates, costs Cuts mid-sentence with no signal, and fails when The consumer parses the output

Failure modes

  • Stop inside content. A quoted document contains the sentinel and the answer is truncated. Debug: search truncated outputs for the stop string and check whether it appeared in the input.
  • Silent length stop. Generation hits the token cap and the client treats a partial answer as complete. Debug: log the finish reason and assert it is the stop condition you expected.
  • Marker retained. The provider includes the stop string in the completion, so parsing sees trailing text. Debug: assert the completion ends exactly at the boundary.

In production

Alert on the ratio of stops by sentinel versus by length; a rising length-stop share means the model is ignoring the boundary. Keep the cap tight enough that a missed stop costs little.

Common misconceptions

People think: A stop sequence guarantees well-formed output.

Actually: It only cuts the string; the content before it can still be invalid or incomplete.

People think: Stop strings work the same on every provider.

Actually: Matching may happen on text or token ids, and inclusion of the marker in the response differs.

Interview angle

Frame stops as a boundary in the output contract, then discuss the matching semantics and the failure when the marker appears in content. Strong answers mention the length cap as a backstop and how they detect a length stop. Weak answers treat the stop string as a guarantee of valid output.

Sources