Why it matters
Some tasks are hard to specify in prose and easy to demonstrate. A support ticket triage policy has a dozen boundary cases. An extraction schema has fields that are obvious once you see one filled example. Few-shot prompting is the cheapest way to communicate those cases without training a model.
It is also one of the few techniques that works across providers without special features. The mechanism is in-context learning: the model conditions on the demonstrations and continues the pattern. No gradients, no checkpoint, no deployment. That makes it attractive for fast iteration and dangerous as a permanent substitute for evaluation, because an example set is a dataset with all the usual dataset problems.
Mental model
Showing a new hire three finished tickets before asking them to handle the fourth. The examples convey the format, the level of detail and where the boundary sits.
The analogy breaks because the new hire builds a model of the task, while the language model predominantly matches patterns in the prompt. It can copy surface features you did not intend — a date format, a phrase, a label distribution — and it does not know which parts of the example are essential.
How it works
The rungs ask whether examples are the right fix, then how to select them; each edge carries the answer.
Step 1 of 5
Try the instruction first
If the format and the boundary are describable, a zero-shot baseline is the cheapest test and the control every example set must beat.
Condition · yes
Zero-shot baseline
Standard
Cheapest and often enough; measure per class before adding anything.
Keep it in the evaluation suite so every added example has to earn its tokens.
Path walked
- A task that is easier to show than to describe
- Can you state the format and boundary in prose?
- yesZero-shot baseline
Ruled out: Do you have examples of the format and one boundary case?, Collect examples first
Figure. Examples earn their tokens only when the instruction cannot carry the format or the boundary. The rungs separate a describable task from a missing example pool, a token bill that favours a fine-tune, and inputs too varied for a fixed set.
What to notice. The rungs follow the page's argument, not a measured ordering. The three-to-five count is the documentation's starting point, not a rule, and the token break-even depends on your prices and volume. The figure assumes the examples are representative; majority-label bias, format leakage and eval contamination are failures the diagram cannot show.
Text version
A decision ladder for choosing a prompting approach. A task that is easier to show than to describe starts rung 1: can you state the format and boundary in prose? Yes leads to a zero-shot baseline, measured per class. No continues to rung 2: do you have examples of the format and one boundary case? No leads to collecting examples first; yes continues. Rung 3 asks whether the token bill dominates at stable, high volume; yes leads to considering a fine-tune instead, because few-shot wins when the task changes often or volume is low. No continues to rung 4: do inputs vary widely in style or domain? Yes leads to retrieval-selected examples with one deliberately dissimilar boundary case; no leads to a fixed set of three to five, chosen by the held-out evaluation rather than by the count alone.
The evidence on what matters is useful. Min et al. found that replacing gold labels with random labels barely hurt classification accuracy in their experiments, while the label space, the input distribution and the format of the demonstrations mattered a great deal. That result is often misread as "labels don't matter". The correct reading is narrower: for classification-style tasks, the model is learning the format and the candidate set from the demonstrations, so a wrong label is less damaging than a wrong label space or an inconsistent format.
Liu et al. showed that example selection and ordering change accuracy, and that retrieving examples similar to the query outperforms a fixed set. That is the bridge to retrieval-selected examples: instead of a static k, select demonstrations per input from a pool.
Anthropic's documentation recommends around three to five examples for multishot prompting. Treat that as a starting point, not a rule; the right count is the one your eval set prefers.
Quick facts
- Mechanism
- In-context conditioning
- Weights
- Unchanged
- Typical count
- A handful
- Main cost
- Input tokens per call
- Main risk
- Label and selection bias
- Best evidence
- Held-out eval
The cost is linear and permanent
With demonstrations of average length tokens, every request pays roughly extra input tokens:
That cost recurs on every call, unlike a fine-tune which pays once. For a high-volume endpoint, the break-even point between few-shot and fine-tuning is a real calculation: few-shot wins when the task changes often or volume is low; fine-tuning wins when the format is stable and the token bill dominates.
Selecting and rendering examples
def render_examples(examples: list[dict]) -> str:
blocks = []
for ex in examples:
blocks.append(
f"<example>\n<input>{ex['input']}</input>\n"
f"<output>{ex['output']}</output>\n</example>"
)
return "\n".join(blocks)
def build_prompt(instruction: str, examples: list[dict], query: str) -> str:
# Keep examples and query in separate sections so the model can see the boundary.
return (
f"{instruction}\n\n"
f"<examples>\n{render_examples(examples)}\n</examples>\n\n"
f"<input>{query}</input>\n"
)For retrieval-selected examples, rank a pool by embedding similarity to the query, take the top few, and deduplicate near-identical cases. Re-ranking by similarity alone can produce a set that is too homogeneous; add one deliberately dissimilar boundary case to keep the decision boundary visible.
Trade-offs
| Approach | Good for | Cost | Fails when |
|---|---|---|---|
| Zero-shot | Simple, well-described tasks | Lowest token cost | The format or boundary is hard to describe |
| Fixed few-shot set | Stable format, consistent categories | Input tokens on every call | Inputs vary widely in style or domain |
| Retrieval-selected examples | Heterogeneous corpora | Selection latency and infrastructure | The example pool is small or unlabelled |
| Fine-tuning | High volume, stable task | Training data and pipeline | Requirements change monthly |
Good for
Simple, well-described tasks
Cost
Lowest token cost
Fails when
The format or boundary is hard to describe
- The format or boundary is hard to describe
- Inputs vary widely in style or domain
- The example pool is small or unlabelled
- Requirements change monthly
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Zero-shot is good for Simple, well-described tasks, costs Lowest token cost, and fails when The format or boundary is hard to describe. Fixed few-shot set is good for Stable format, consistent categories, costs Input tokens on every call, and fails when Inputs vary widely in style or domain. Retrieval-selected examples is good for Heterogeneous corpora, costs Selection latency and infrastructure, and fails when The example pool is small or unlabelled. Fine-tuning is good for High volume, stable task, costs Training data and pipeline, and fails when Requirements change monthly
Failure modes
Majority-label bias. The model predicts the label that dominates the examples. Debug by measuring per-class recall, not just accuracy, and balance the demonstration set.
Format leakage. The model copies an artefact from the examples — a sample name, a date, a citation marker. Debug by diffing outputs against the example text and by replacing real entities in examples with clearly synthetic ones.
Unrepresentative examples. All demonstrations come from one customer segment, and the model fails on the rest. Debug by stratifying the eval set by segment and sampling examples from the same strata.
Eval contamination. The examples include the exact inputs used for evaluation, so scores are inflated. Debug by deduplicating the demonstration pool against the eval set and keeping a held-out slice that never enters a prompt.
Symptom
The model predicts the label that dominates the examples.
Debug step
Measuring per-class recall, not just accuracy, and balance the demonstration set.
Figure. Each failure with its symptom and the check that would confirm it.
What to notice. A failure without a debug step is a rumour. Pick the symptom you are seeing; the check is the next action.
Text version
Failure modes: Majority-label bias: The model predicts the label that dominates the examples. Debug by measuring per-class recall, not just accuracy, and balance the demonstration set. | Format leakage: The model copies an artefact from the examples — a sample name, a date, a citation marker. Debug by diffing outputs against the example text and by replacing real entities in examples with clearly synthetic ones. | Unrepresentative examples: All demonstrations come from one customer segment, and the model fails on the rest. Debug by stratifying the eval set by segment and sampling examples from the same strata. | Eval contamination: The examples include the exact inputs used for evaluation, so scores are inflated. Debug by deduplicating the demonstration pool against the eval set and keeping a held-out slice that never enters a prompt.
In production
Treat the example set as a versioned asset with an owner, a selection policy and a token budget. Log which examples were used for each request, because "the model changed" often means "the selector returned different examples". Cache the rendered prefix where prefix caching is supported. Review the pool quarterly for stale formats and leaked customer data. If the pool grows past a handful per request, you have a retrieval problem, not a prompting problem.
Common misconceptions
People think: More examples always improve accuracy.
Actually: Accuracy often plateaus or declines as the prompt grows. Extra examples add cost and can introduce conflicting patterns or recency bias.
People think: Few-shot teaches the model new facts.
Actually: It conditions the existing model on a pattern. Facts come from pretraining, retrieval or tools; demonstrations mainly convey format and task framing.
Interview angle
Expect "the classifier works for two labels but never predicts the third". The strong answer checks label balance in the examples, per-class recall, example order and similarity to the failing input, then proposes retrieval-selected examples or a fine-tune if volume justifies it. The weak answer adds more examples of the missing class without measuring. See the prompting bank.
Sources
- Brown et al., Language Models are Few-Shot Learners — in-context learning at scale.
- Min et al., Rethinking the Role of Demonstrations — what demonstrations actually convey.
- Liu et al., What Makes Good In-Context Examples for GPT-3? — selection and ordering effects.
- Anthropic, Use examples (multishot prompting) — practical guidance on example count and ordering.
Related
- Zero-shot — the baseline to beat.
- Example selection — choosing demonstrations per input.
- Prompt engineering — the wider discipline.
- Fine-tuning — when examples become training data.
- Evaluation datasets — avoiding contamination.