Skip to content
AI Engineer Guide.
juniorReportedcurated

The PM says: fine-tune the model on our docs so it knows our product. How do you respond?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

What eval would you build before deciding?
  • Build a golden set from real support questions with the correct answer and supporting document, and score the prompted baseline first.
  • Separate metrics for answer correctness, citation coverage and abstention, because a model can sound right while citing nothing.
  • Slice by question type so a retrieval fix is not credited for gains on questions that never needed retrieval.
When is the honest answer both RAG and fine-tuning?
  • Use retrieval for facts and fine-tuning for the behaviour around them, such as citation format, tool-call syntax or routing to the right index.
  • Evaluate each independently: fix retrieval first, then tune only the residual behaviour the prompt cannot hold.
  • Keep the fine-tune free of volatile facts, or every policy update becomes a retraining job.

Red flags

  • Agrees to fine-tune on documents to inject facts
  • Has no baseline metric, so improvement cannot be demonstrated

Concepts this tests

Tags

Source: github.com

More in Fine-tuning & adaptation