seniorReportedcurated
Defending your take-home: if you had to simplify the system, which component would you remove, and what would you lose?
30-second answer
- I would remove the reranker first, because it bought about four points of grounded correctness on ambiguous long-tail queries but doubled p95 latency and added a model to operate.
- I would replace unconditional reranking with routing only low-margin queries to it, so the precision gain stays where it pays and the median request keeps its latency.
- I would keep citations and the abstain path, because those are the product guarantee rather than an optimisation, and cutting them to hit a latency number changes what the system promises.
- The cache is the least justified decision, but removing it changes the load target rather than the task path; the reranker is the first component cut because its quality gain is the one I can trade back for latency.
- Dropping the reranker widens the gap on paraphrased questions, so the eval slice covering those has to stay in the report rather than disappear with the component.
Likely follow-ups
Open a question to rehearse the answer.
Which decision would you redo, and why did it look right at the time?
- I would redo the caching layer, because it answered a load target I could measure and the task benefit was assumed rather than tested.
- It looked right because the benchmark was a throughput test with repeated queries, which is the case a cache flatters most.
- I would keep the reranker but make it conditional from the start, so latency and precision were a routing decision rather than a global one.
What would break first if usage tripled?
- Ingestion and re-embedding cost would break first, because the corpus grows with usage and a model change means re-embedding all of it.
- Retrieval p95 latency comes next, since the reranker and two indexes sit on the critical path and the tail compounds under concurrency.
- Cache effectiveness, because a larger and more varied query mix lowers the hit rate, so the load target that justified the cache moves.
Red flags
- Cannot name anything they would cut, implying no prioritisation
- Cuts the safety-critical component and keeps the cosmetic one
Concepts this tests
Tags
Source: landthelab.com