Why it matters
Retrieval returns a ranked list; a prompt needs a structured document. Between them sit decisions that no retriever makes: whether to deduplicate overlapping chunks, whether to keep a passage that contradicts another, where to put the strongest evidence, and what to do when the budget runs out. Left implicit, these are made by string concatenation and a truncation call, and the result is a prompt whose best evidence was cut off the end.
Mental model
Assembly is laying out a briefing folder for a meeting: order the pages, label them, and leave out the ones that do not fit. The metaphor breaks because a folder can be flipped through, while a model's attention over a long prompt is uneven and positional.
How it works
Event · step 1 of 7
0.28 s · twelve candidates returned
At 0.28 s
Hybrid retrieval returns twelve candidates with ids and scores. Nothing has been admitted yet; the budget is still untouched and the prompt is empty.
On screen at this event
- search
Next at 0.46 s: 0.46 s · rerank keeps eight
- Candidates
- 12
- Admitted
- 0
- Tokens
- 0
- Dropped
- 0
- TTFT
- —
Figure. A synthetic 1.8 s turn: twelve candidates, eight after rerank, six after dedupe, a 6,000-token budget with 1,200 reserved, four passages admitted at 1,200 tokens each, and two dropped, including the reranker's last promotion. Counts and times are illustrative.
What to notice. The pre-model path is 0.66 s of the 1.18 s time to first token: retrieval 0.28 s, rerank 0.18 s, and assembly plus prompt build 0.20 s. The assembly lane itself is 0.12 s. The budget is exhausted before every candidate is considered, and the fill follows retrieval order, so it removes H, which reranking promoted last: the strongest tail evidence is cut by position rather than by relevance. Log admitted and dropped ids with token counts; a drop rate that rises usually means chunk size or k drifted.
Text version
Six lanes over 1.8 seconds: retrieval, rerank, assembly, prompt, model and dropped. Search runs 0 to 0.28, rerank 0.28 to 0.46, dedupe 0.46 to 0.5, a fill span 0.5 to 0.56 and a red drop span 0.56 to 0.58 on three rows. The prompt assembles 0.56 to 0.64 and sends 0.64 to 0.66. Model prefill runs 0.66 to 1.18 and decode 1.18 to 1.8. Two dashed ghost spans sit in the dropped lane from 0.58: H, status top tail dropped, and F, status tail cut, both running from 0.58 to the end of the turn. Counters: candidates 12, then 8, then 6; admitted 4 from 0.56; tokens 6,000 of 6,000; dropped 0 until 0.58 then 2; TTFT 1.18 s at the last step.
- Deduplicate candidates by document id and overlapping spans before counting tokens.
- Attach a stable label to each passage, such as source title, section and chunk id, so citations resolve to something.
- Reserve tokens for the system prompt, the question and the expected answer before admitting passages.
- Order by a deliberate policy: strongest first, or strongest at both ends when the context is long.
- Wrap each passage in delimiters and state in the instructions that passage text is evidence, not instruction.
- When the budget is exceeded, drop by a documented rule and record which passages were dropped.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Strongest-first ordering | Simple and effective for short contexts | Middle passages are weakest | The context is long and the answer is in the tail |
| Strongest at both ends | Exploits positional attention | Harder to explain and to evaluate | The prompt is short enough that position hardly matters |
| Include conflicting passages | Lets the model report disagreement | More tokens and a harder synthesis task | One passage is outdated and the model picks it |
| Drop by lowest score | Deterministic budget control | May drop a passage that resolves an ambiguity | Scores are uncalibrated across retrievers |
Good for
Simple and effective for short contexts
Cost
Middle passages are weakest
Fails when
The context is long and the answer is in the tail
- The context is long and the answer is in the tail
- The prompt is short enough that position hardly matters
- One passage is outdated and the model picks it
- Scores are uncalibrated across retrievers
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Strongest-first ordering is good for Simple and effective for short contexts, costs Middle passages are weakest, and fails when The context is long and the answer is in the tail. Strongest at both ends is good for Exploits positional attention, costs Harder to explain and to evaluate, and fails when The prompt is short enough that position hardly matters. Include conflicting passages is good for Lets the model report disagreement, costs More tokens and a harder synthesis task, and fails when One passage is outdated and the model picks it. Drop by lowest score is good for Deterministic budget control, costs May drop a passage that resolves an ambiguity, and fails when Scores are uncalibrated across retrievers
Failure modes
- Tail truncation. The prompt builder cuts the last passages, which are often the ones a reranker promoted late. Debug: log the admitted and dropped passage ids for a failing query and check where the gold chunk fell.
- Unlabelled passages. The answer cites a source that cannot be resolved because the passage carried no id. Debug: assert every passage in the prompt has a unique label present in the citation map.
- Duplicate evidence. Overlapping chunks appear twice and the model treats repetition as confirmation. Debug: compute span overlap between admitted passages and count near-duplicates.
In production
Log admitted and dropped passages per request with token counts. Alert when the drop rate rises, which usually means chunk size or k drifted rather than the query changing.
Common misconceptions
People think: Assembly is just joining the retrieved chunks.
Actually: Ordering, labelling, deduplication and budgeting each change the answer independently of retrieval quality.
People think: Truncation is a safe fallback.
Actually: It removes evidence silently and usually removes the lowest-ranked tail, which is where late reranker promotions land.
Interview angle
Treat assembly as its own stage with a budget, a deduplication rule, labels and an ordering policy. Explain positional attention and the drop policy. Concatenating chunks into a prompt, with nothing said about the token limit, is the answer that hides the failure.
Sources
- Liu et al., Lost in the Middle — positional effects on how models use evidence in a long context.
- Lewis et al., Retrieval-Augmented Generation — retrieved passages as the generator's conditioning context.
- Es et al., RAGAS — context precision and recall as measures of what assembly admitted.