Skip to content
AI Engineer Guide.
seniorReportedcurated

You want an LLM judge to gate deploys on faithfulness. How do you validate it before you trust it?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Kappa falls to 0.5 after a judge model upgrade, do you fix the judge or re-label, and how do you decide?
  • Re-run the frozen calibration set on both judge versions first, because the drop may be the model change rather than the labels.
  • If humans still agree with each other, fix the judge by restoring the pinned version or rewriting the rubric; if they do not, adjudicate the labels.
  • Do not adopt the new judge on its average score, since what changed is the measurement rather than the system under test.
Would you gate on a binary per-claim rubric or a 1-10 score, and what does each do to calibration?
  • Binary per-claim verdicts calibrate and audit more easily, because each failure names the claim that broke and the span that failed to support it.
  • A 1-10 score compresses several properties into one number, hiding offsetting changes and resisting a defensible threshold.
  • If you need a scalar, aggregate the binary verdicts rather than asking the judge for a grade.
How do you stop the judge from drifting when the generator model changes underneath it?
  • Pin the judge model and prompt version, and store both next to every score so a comparison across versions is visible.
  • Keep a frozen human-labelled calibration set and re-run it on every generator or judge change.
  • Judge with a different model family, so a shared blind spot cannot silently confirm the generator's output.

Red flags

  • percent agreement only
  • uncalibrated judge gating

Concepts this tests

Tags

Source: algorithmine.com

More in RAG & retrieval pipelines