LLM-as-Judge

Using a language model to grade outputs: pointwise vs pairwise vs reference-guided modes, known biases, calibration against humans, and when not to use it.

Contents

LLM-as-judge means using a language model as the grader in an eval: it reads an output (and optionally the input, a reference answer, retrieved context or a second candidate) and returns a verdict against a rubric. It exists because open-ended qualities such as helpfulness, tone, faithfulness or task completion cannot be checked with string matching, and human grading does not scale. Strong judges can match human agreement levels (GPT-4 agreed with humans over 80% of the time on MT-Bench, about the human-human rate), but they carry systematic biases and must be validated against human labels before their scores are trusted (Zheng et al., OpenAI).

Judging modes

ModeJudge seesOutputStrengthsWeaknesses
Pointwise (single-answer)one output + rubricscore (1-5) or pass/failabsolute; scales linearly; works for monitoring a single systemscale drift; scores bunch on one value; Likert points are ill-defined
Pairwisetwo outputsA / B / tiemore reliable for “which is better”; matches A/B decisionsposition bias; O(n²) for many systems; no absolute level
Reference-guidedoutput + gold answercorrect / incorrect or scoreanchors the judge on facts and maths it would otherwise missneeds references
Decomposed (claims)output split into claims + contextfraction supportedfine-grained faithfulness for RAGmany calls; extraction errors

Sources: Zheng et al., OpenAI, Ragas. OpenAI recommends “pairwise comparison or pass/fail for more reliability” (OpenAI). Practitioner guidance goes further: Likert scales are hard to act on (nobody can define the gap between 3 and 4), so use binary pass/fail plus a written critique (Husain). Anthropic’s docs accept either a correct/incorrect label or a 1-5 scale, as long as output is constrained rather than free-form (Claude docs). Pairwise human judging at scale is what Chatbot Arena does (Chiang et al.).

How a good judge prompt works

  • Explicit rubric with concrete pass conditions, possibly several rubrics per criterion (Claude docs).
  • Reason, then score. Let the judge reason (or use thinking) before the verdict; this helps on complex judgments (Claude docs, OpenAI). G-Eval automates this: the model first writes its own evaluation steps from the criterion, then fills in a score form (G-Eval).
  • Structured output, e.g. JSON {score, reason} so scores parse and reasons can be audited (notebook).
  • An “Unknown” escape so the judge is not forced to guess when information is missing (Anthropic).
  • A different model from the generator, to reduce self-preference (Claude docs); the most capable judge available (OpenAI).
  • Probability-weighted scores. Integer scores bunch on one value and tie; G-Eval computes Σ p(s)·s over score tokens (estimated from 20 samples when log-probs are unavailable) for a continuous score, reaching Spearman 0.514 with humans on SummEval versus 0.474 for UniEval, the best prior evaluator (G-Eval).
  • Few-shot examples from expert critiques. Few-shot raised GPT-4’s position consistency from 65.0% to 77.5% at ~4x prompt cost (Zheng et al.); expert critiques double as examples (Husain).

Known biases

BiasEvidenceMitigation
PositionGPT-4 gave the same verdict after swapping order only 65% of the time; Claude-v1 favoured the first slot 75% (Zheng). Order alone let Vicuna-13B “beat” ChatGPT on 66/80 queries (Wang). Worse with 3-4 candidates (CALM).Judge both orders; disagreement = tie (Zheng); balanced position calibration (Wang)
Verbosity / lengthPadding with rephrased list items fooled GPT-3.5 and Claude-v1 in 91% of 23 cases, GPT-4 in 9% (Zheng)Rubric penalising padding; regression length control raised AlpacaEval’s Arena correlation from 0.94 to 0.98 (Dubois)
Self-preferenceJudges rate their own outputs higher; strength tracks how well they recognise their own text (Panickssery); G-Eval rated GPT-3.5 summaries above human ones humans preferred (G-Eval)Use a different model family as judge; multi-judge panels
Sentiment, authority, bandwagon, distractionNegative emotional tone cut verdict robustness to ~0.24-0.38 for strong 2024 judges; fake citations and “majority opinion” also sway some judges (CALM)Strip irrelevant framing; test the judge with perturbations
Being misled by the answerJudges accept wrong maths they could solve; a reference answer cut GPT-4’s failure rate from 70% to 15% (Zheng)Reference-guided grading; have judge solve first

CALM lists twelve biases in all, including compassion-fade (model names shown), refinement-aware and identity-marker sensitivity (CALM).

Calibration: validating the judge

A judge is itself a classifier and needs its own eval. The recipe (Husain, Shankar et al.):

flowchart TD
  A["Domain expert labels outputs<br/>pass/fail + critique"] --> B["Split: 10-20% few-shot / 40-45% dev / 40-45% test"]
  B --> C["Write judge prompt<br/>rubric + critiques as examples"]
  C --> D["Run judge on dev set"]
  D --> E{"TPR and TNR<br/>high enough?"}
  E -->|"no: read disagreements"| C
  E -->|"yes"| F["Measure once on held-out test"]
  F --> G["Use at scale; re-check<br/>on new labels periodically"]
  • Measure true positive and true negative rates separately, not raw agreement: if 5% of outputs fail, an always-pass judge has 95% agreement and catches nothing (Husain).
  • Sizing: start around 30 labelled examples, aim for ~100 per failure mode; under ~60 the confidence intervals are too wide (Husain). The Honeycomb case passed 90% agreement in three prompt iterations, but its data were roughly balanced (about half failures), which is the case where raw agreement is meaningful.
  • Criteria drift: labelling changes the expert’s own criteria, so expect to revise the rubric and re-validate (Shankar et al.).
  • Correlation is not validation: in GroUSE’s 144 unit tests over seven grounded-QA failure modes, automated RAG evaluation frameworks missed important failures even with GPT-4 as judge, and open-weight judges that correlated well with GPT-4 still failed the tests, so add per-failure-mode unit tests rather than relying on agreement with a reference judge (Muller et al.). A 2025 survey finds the same validation gap for every generation of automatic metric, LLM judges included, and recommends combining complementary evaluations (Oliva et al.).
  • Human-in-the-loop calibration: route examples where position-swapped verdicts disagree to humans (Wang).

Synthesis: once TPR/TNR are known, a judge’s raw pass rate can be corrected for its error rates, and a judge prompt change should be treated like any other change: re-run the judge eval before trusting new numbers.

Limitations and failure modes

  • Non-determinism and cost: each grade is a model call, scores vary between runs, and pairwise or multi-sample schemes multiply cost (Anthropic).
  • Shared blind spots: a judge from the same family can share the generator’s concepts of quality; using judges as reward signals risks self-reinforcement (G-Eval, Panickssery).
  • Hard questions: on items beyond the judge’s ability, the judge is easily misled (Zheng); this is why contamination-limited benchmarks such as LiveBench avoid judges entirely (see benchmarks).
  • Gaming: verbosity and formatting tricks inflate scores when the judge is the target (Zheng, Dubois).

When to use, when not

  • Use for open-ended qualities (tone, helpfulness, faithfulness, rubric adherence), for sampled online scoring of production traces (LLM observability), and to triage transcripts for human review (Anthropic, notebook).
  • Do not use when code can decide: schema validity, exact labels, unit tests, database state, tool-call arguments. Code graders are faster, cheaper and deterministic (Claude docs, Husain).
  • Pairwise when choosing between two prompts or models; pointwise pass/fail for regression gates and monitoring. Synthesis: pairwise results do not tell you whether either version is good enough to ship, so pair them with a pointwise gate.

Sources

Suggest changes →