seniorReportedcurated
You want an LLM judge to gate deploys on faithfulness. How do you validate it before you trust it?
30-second answer
- Treat the judge as a model that needs its own eval: 50-100 labelled examples are the smoke-test floor, while a deploy gate needs a few hundred items stratified by category.
- Target kappa around 0.6 before the judge gates anything, state the class balance, and report a confidence interval rather than a single number.
- Decompose faithfulness into per-claim binary checks instead of a 1-10 score, average both orderings to cancel position bias, and use a different model family from the generator.
- Re-validate whenever the judge prompt or model version changes, since a silent judge upgrade is a silent metric change.
- Human labels are slow and costly, but an uncalibrated judge drifts silently and blocks good releases.
Likely follow-ups
Open a question to rehearse the answer.
Kappa falls to 0.5 after a judge model upgrade, do you fix the judge or re-label, and how do you decide?
- Re-run the frozen calibration set on both judge versions first, because the drop may be the model change rather than the labels.
- If humans still agree with each other, fix the judge by restoring the pinned version or rewriting the rubric; if they do not, adjudicate the labels.
- Do not adopt the new judge on its average score, since what changed is the measurement rather than the system under test.
Would you gate on a binary per-claim rubric or a 1-10 score, and what does each do to calibration?
- Binary per-claim verdicts calibrate and audit more easily, because each failure names the claim that broke and the span that failed to support it.
- A 1-10 score compresses several properties into one number, hiding offsetting changes and resisting a defensible threshold.
- If you need a scalar, aggregate the binary verdicts rather than asking the judge for a grade.
How do you stop the judge from drifting when the generator model changes underneath it?
- Pin the judge model and prompt version, and store both next to every score so a comparison across versions is visible.
- Keep a frozen human-labelled calibration set and re-run it on every generator or judge change.
- Judge with a different model family, so a shared blind spot cannot silently confirm the generator's output.
Red flags
- percent agreement only
- uncalibrated judge gating
Concepts this tests
Tags
Source: algorithmine.com
More in RAG & retrieval pipelines
- Explain parent-child (small-to-big) retrieval. What problem does it solve and what does it cost you?
- Explain HNSW. What do M, ef_construction and ef_search each control, and which can you change in production?
- How would you tune HNSW for 10 million vectors when the product needs p95 retrieval under 100 ms?