midReportedcurated
Walk me through an AI feature you shipped end to end and your exact role. What decisions were yours?
30-second answer
- I owned retrieval and evaluation rather than the whole product, and the decision was to fix evidence before model size, because the failures were missing passages, not weak reasoning.
- I chose hybrid BM25 plus dense retrieval with citations over a bigger model, since a bigger model cannot answer from a passage that was never retrieved.
- The cost was explicit: hybrid retrieval added roughly 150 ms at p95 and one more index to operate.
- On a frozen 350-question set, faithfulness moved from 0.71 to 0.89 and escalations dropped 32% over six weeks.
- I did not own the frontend or the prompt copy, and I said so; my contribution was the retrieval path, citation validation and the eval harness that measured it.
- Hybrid would have been the wrong first move if recall were already healthy, because the same measurement would then point at generation rather than retrieval.
Likely follow-ups
Open a question to rehearse the answer.
What exactly did you implement versus what did the team do?
- I owned ingestion and the retrieval path, including fusion, citation validation and the evaluation harness; teammates owned the frontend, prompt copy and rollout.
- I set the interface between those parts, which is why I can describe the whole system without claiming their work.
- The boundary matters in review, because a candidate who says we everywhere cannot be assessed on the decisions they actually made.
How did you know the improvement was not just the new eval set being easier?
- The set was frozen before the change and the old system was run against the same questions, so the baseline was measured on the same items.
- I kept the old system runnable as a permanent baseline and compared item-level results, not just the aggregate.
- Questions used to tune retrieval were held out of the reported set, so the number was not measured on the examples that shaped the fix.
Red flags
- Uses we throughout with no clear personal contribution
- Reports a demo impression rather than a measured outcome
Concepts this tests
Tags
Source: codesnatch.io
More in Behavioural & project deep-dives
- Tell me about a time a model surprised you in production. How did you find out, and what did you change?
- Describe a time you disagreed with a product or engineering stakeholder about using an agent, fine-tuning, or RAG.
- Tell me about a model-quality regression you introduced or discovered after deployment. What did you do?