midReportedcurated
Take-home: build a RAG assistant over a document set with citations. What do you build first, and how do you scope it?
30-second answer
- I write a one-page scope note before code: what I will build, what I am cutting, and why, so the reviewer can see the trade rather than infer it.
- I build the eval set first, roughly 20 to 30 questions split into answerable, partially answerable and unanswerable, because a submission that never measures anything cannot be assessed.
- Then I build the thinnest end-to-end path: ingest, chunk, retrieve, answer with citations, and abstain when evidence is missing.
- I report retrieval and generation separately, recall at the k passed to the model and then answer correctness, with citations resolving to retrieved passage IDs.
- I spend the time on citations and abstention and leave authentication and UI polish out, documenting that choice in a README with the numbers and a short walkthrough.
- At 100x documents the work becomes ingestion idempotency, index freshness and retrieval latency, because a pipeline that only grows the index returns stale or duplicated passages.
Likely follow-ups
Open a question to rehearse the answer.
What did you deliberately leave out, and what would break first at 100x documents?
- I left out authentication, per-tenant filtering and UI polish, because none of them change whether the answers are grounded.
- At 100x documents, ingestion idempotency and deletion propagation break first, since a re-sync that only adds leaves stale versions retrievable.
- Retrieval latency and re-embedding cost come next, which is why the scope note names them as the next engineering steps rather than hiding them.
How did you use AI tools, and where did you reject their output?
- I used them for scaffolding, boilerplate and a first draft of the README, which is work that does not carry a design decision.
- I rejected generated eval questions that the retriever found easy, because a set written by the same system that answers it measures the wrong thing.
- I also rejected any citation the code could not resolve to retrieved text, since an unverifiable citation is worse than no citation.
Red flags
- Builds a polished UI with no evaluation harness
- Cannot explain why the chunk size or top-k was chosen
Concepts this tests
Tags
Source: github.com