Why it matters
A few-shot prompt is a tiny training set assembled at request time, and like any training set it carries bias, coverage and ordering effects. Teams often fix a handful of examples once and never revisit them, so the prompt encodes whatever cases were top of mind. Selection treats examples as a retrieval problem: choose for coverage of the decision boundary, balance of labels, and similarity to the current input.
Mental model
Example selection is casting for a play: the same script reads differently depending on who is on stage. The comparison breaks because the audience also changes, since the examples are chosen per request rather than once for a run, and every performance is judged separately.
How it works
- 1Build a pool of labelled examples with metadata: label, difficulty, length and the feature that makes each one distinctive.
- 2Decide the selection goal: coverage of labels and formats, similarity to the query, or both.
- 3For a fixed prompt, choose a balanced, diverse set and record the order, since order changes results.
- 4For variable inputs, retrieve the nearest examples by embedding and cap how many come from one cluster.
- 5Hold out examples from the evaluation set, so the prompt is not tested on its own demonstrations.
- 6Re-test the example set whenever the model or the input distribution changes.
Step 1 of 6
Build a pool of labelled examples with metadata: label, difficulty, length and the feature that makes each one distinctive.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Build a pool of labelled examples with metadata: label, difficulty, length and the featur… 2. Decide the selection goal: coverage of labels and formats, similarity to the query, or bo… 3. For a fixed prompt, choose a balanced, diverse set and record the order, since order chan… 4. For variable inputs, retrieve the nearest examples by embedding and cap how many come fro… 5. Hold out examples from the evaluation set, so the prompt is not tested on its own demonst… 6. Re-test the example set whenever the model or the input distribution changes.
- Build a pool of labelled examples with metadata: label, difficulty, length and the feature that makes each one distinctive.
- Decide the selection goal: coverage of labels and formats, similarity to the query, or both.
- For a fixed prompt, choose a balanced, diverse set and record the order, since order changes results.
- For variable inputs, retrieve the nearest examples by embedding and cap how many come from one cluster.
- Hold out examples from the evaluation set, so the prompt is not tested on its own demonstrations.
- Re-test the example set whenever the model or the input distribution changes.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Fixed example set | Predictable, cacheable | Misses rare input types | Traffic is heterogeneous |
| Retrieved per query | Relevant demonstrations | Retrieval latency and infrastructure | The example pool is small |
| Label-balanced examples | Reduces majority bias | May look unnatural for skewed tasks | The real distribution is heavily skewed |
| Many examples | Strong format adherence | Token cost and longer prompts | Context budget is tight |
Good for
Predictable, cacheable
Cost
Misses rare input types
Fails when
Traffic is heterogeneous
- Traffic is heterogeneous
- The example pool is small
- The real distribution is heavily skewed
- Context budget is tight
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Fixed example set is good for Predictable, cacheable, costs Misses rare input types, and fails when Traffic is heterogeneous. Retrieved per query is good for Relevant demonstrations, costs Retrieval latency and infrastructure, and fails when The example pool is small. Label-balanced examples is good for Reduces majority bias, costs May look unnatural for skewed tasks, and fails when The real distribution is heavily skewed. Many examples is good for Strong format adherence, costs Token cost and longer prompts, and fails when Context budget is tight
Failure modes
- Label skew. The model over-predicts the label that appears most in the examples. Debug: count labels in the prompt and compare predicted distribution with the true one.
- Leakage. An example is nearly identical to an evaluation item, inflating the score. Debug: check nearest-neighbour similarity between prompt examples and test items.
- Order sensitivity. Shuffling examples changes accuracy by more than the change being tested. Debug: run the same prompt with three orderings and report the spread.
In production
Freeze the example set with the prompt version and log which examples were used per request when selection is dynamic. Monitor per-label accuracy, since example bias shows up as a class-specific drop.
Common misconceptions
People think: More examples are always better.
Actually: Gains saturate and can reverse; unrepresentative examples add bias as well as tokens.
People think: Example order is cosmetic.
Actually: Order changes results measurably, so it belongs in the versioned prompt, not in a random shuffle.
Interview angle
Treat examples as a small dataset and discuss coverage, label balance, similarity and order as separate variables. Strong answers mention leakage between examples and tests. Weak answers say to add more examples and never check whether the examples represent the traffic.
Sources
- Liu et al., What Makes Good In-Context Examples for GPT-3? — similarity-based example selection and its gains.
- Lu et al., Fantastically Ordered Prompts and Where to Find Them — how example ordering changes accuracy.
- Brown et al., Language Models are Few-Shot Learners — in-context learning from demonstrations.