Score answers with binary gates plus 1-5 scales; watch for position and verbosity bias in LLM judges.
1. Binary gates first
Citation present + supports claim? Refusal correct on OOD? Any unsupported factual sentence? A single gate failure fails the item — scales only rank items that pass gates.
2. 1-5 scales
Faithfulness (claims <-> context), completeness (covered the gold points?), concision (no padding). Calibrate with 3 anchored examples per level before scoring the set.
3. Judge-bias guards
Position bias: shuffle A/B order across runs. Verbosity bias: cap lengths or score blind to length. Log judge model + temperature with every score.
4. Cost-latency alongside quality
Every eval row carries tokens, USD, and latency — a +2 faithfulness gain that 10x's cost ships only with a written sign-off (capstone dashboard rule).