LLM-as-a-Judge from Scratch — How AI Grades AI, and Where It Breaks

A ground-up guide to using one model to grade another. Covers reading a verdict as a probability distribution, the three recurring biases (position, verbosity, self-enhancement), why pairwise comparison cost grows quadratically, and how to validate the judge itself against human labels.

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Putting a machine in the grader's chair

Picture a writing contest. With 100 entries, a panel of judges can read everything and rank it. With 100,000 entries it falls apart — and you want to re-read all of them every week, each time someone tweaks a prompt. That is roughly the position anyone building on language models is in.

If output quality reduced to "does it match the answer key," none of this would be hard. Checking whether a math answer is 42 is a string comparison. But the properties we actually care about — is this summary faithful to the source, is this reply rude, is this explanation clear — have no single correct answer. Measuring them with human raters costs thousands of dollars and several days per round.

LLM-as-a-Judge is the obvious response: have another language model do the grading. Hand it a rubric and the output under test, get back a score or a verdict. That is the whole mechanism. Because it drops evaluation cost by orders of magnitude, it now sits inside the development loop of most model and product teams.

It is also a measuring instrument with unusually strong quirks. This article walks through the mechanism, the source of the quirks, and the procedure for using it anyway. For how to read benchmarks in general, see LLM Evaluation from Scratch; here the subject is the grader itself.

There are only three ways to ask

Implementations vary endlessly, but the question format comes in three shapes.

1. Pointwise. Show one output, ask for a score from 1 to 5. Cheap, easy to run at volume. The weakness is drift: the same answer scores differently depending on the wording of the prompt that day or the examples preceding it. Never trust the absolute value — use it only for relative comparison within one fixed setup.

2. Pairwise. Show outputs A and B side by side, ask which is better. Just as humans are steadier at relative than absolute judgment, agreement goes up. In exchange you inherit position bias, and the number of comparisons balloons.

3. Reference-based. Supply a gold answer and have the judge check against it. The most stable of the three, but only available for tasks where you can produce a gold answer.

Which one you pick follows from the decision you are trying to make. Choosing between model A and B for deployment calls for pairwise; tracking a daily quality number as a regression test calls for reference-based.

The mechanism: a verdict is a classification, and the answer is a distribution

When a judge emits 4, what happened internally was a probability computation over the tokens "1", "2", "3", "4", "5". The 4 you receive is just the single highest-probability point pulled out of that distribution.

Here $K$ is the number of score levels (5 for a 5-point scale), $p(i)$ is the probability the model assigns to answering "$i$", and $s$ is the final score. What equation (1) says in words is just this: average the possible scores, weighting each by how likely the judge was to say it. If the judge puts 0.6 on a 4, 0.3 on a 3, and 0.1 on a 5, the score is $3 \times 0.3 + 4 \times 0.6 + 5 \times 0.1 = 3.6$ — a 4, but a 4 leaning toward 3. The hesitation survives into the number.

Why bother? Because if you accept integers only, the judge returns "3" or "4" for almost everything, the distribution collapses, and a hundred outputs end up sharing a handful of scores. Weighting by probability preserves the gap between 3.7 and 3.2, which is where fine-grained ranking lives. This technique was introduced in G-Eval as a way to improve correlation with human judgment, and it has become common practice.

The fastest way to feel what "read it as a distribution" means is to move the slider. Notice that a confident judge (a peaked distribution) and a torn one (a flat distribution) can share the same mode of 4 while meaning completely different things.

Taking a single integer from a flat distribution is like cornering an undecided judge and demanding a number. What comes back is less an answer than a shrug rounded to the nearest integer.

The three biases

Three failure modes show up again and again when judges diverge from humans. The MT-Bench paper named and measured them early.

Position bias. Shown A and B, judges favor whichever came first. Swapping the order alone can flip the winner. This is not a matter of the judge being weak; it means an irrelevant signal — ordering — is leaking into the verdict. The fix is blunt: run both AB and BA order. If the same side wins in both, accept it; if they disagree, record a tie. Comparison cost doubles, and the position effect largely disappears.

Verbosity bias. Judges score longer answers higher. Adding bullet points, a preamble, and a summary can raise the score with no change in substance. This one is dangerous because if you use the judge as a reward signal for training, the model optimizes straight toward padding. Length-Controlled AlpacaEval addressed it by regressing out the length effect to produce a length-debiased win rate. For your own setup, the first diagnostic is simpler: correlate output length against judge score. If the correlation is strong, length is what you are measuring.

Self-enhancement bias. Judges rate their own outputs — and those of models in the same family — more highly. Grading a GPT-family model with a GPT-family judge invites favoritism. Which makes "we improved our model, as scored by our model" a conflict of interest in structure, not just in appearance. The mitigations are to use a judge from a different family than the system under test, or to vote across several judges.

The common thread is structural: the judge is tracking something correlated with quality rather than quality itself. Order, length, stylistic familiarity — human raters have the same tendencies, but a machine applies them consistently across tens of thousands of items, so they survive aggregation instead of averaging out.

Comparison count grows quadratically

Pairwise judging has a volume problem independent of bias. Comparing $n$ models round-robin needs $n(n-1)/2$ pairs. Each pair gets multiple questions, and each of those doubles for the order swap:

$n$ is the number of models, $m$ the questions per pair, and the trailing 2 covers the order swap. What equation (2) says in words: multiply the number of pairs by the questions per pair by the two orderings, and $C$ is how many times you actually call the judge. Ten models over 80 questions is $45 \times 80 \times 2 = 7{,}200$ calls. Five models is 10 pairs; twenty models is 190. Quadrupling the model count multiplies comparisons by roughly 19.

So production setups give up on round-robin. Either fix one baseline model and compare everyone against it ($n$ comparisons), or sample matchups randomly and estimate a ranking with a rating system. Both are fine as long as you remember you did not actually compare everything.

Judging the judge

This is the step most often skipped and the one that matters most. Once you adopt a judge, measure how well it agrees with humans on your own task. Someone else's reported agreement rate on someone else's benchmark guarantees nothing about your data.

The procedure: pull 100–300 items from your task, label them by hand, run the judge on the same items, and compare. Three numbers are worth looking at.

Raw agreement. What percentage matched. This number flatters itself — on a binary choice, random guessing already scores 50%. So correct for chance:

$p_o$ is the observed agreement, $p_e$ the agreement you would expect from chance alone, and $\kappa$ (kappa) the corrected value. What equation (3) says in words: subtract the luck from the raw rate, then divide by the headroom that was left to win. On a binary choice with $p_e = 0.5$, an observed 75% gives $\kappa = (0.75 - 0.5) / 0.5 = 0.5$ — what looked like "right three times out of four" is half that once you take the coin flips out. Zero means no better than chance; one means perfect agreement.

Comparison against human–human agreement. This is the actual pass mark. Two human raters disagree too. The MT-Bench paper showed a strong judge reaching agreement with humans at a level comparable to agreement between humans. Read the other way: on a subjective task where two people agree only 60% of the time, demanding 90% from a judge is incoherent. The ceiling is on the human side.

Order-swap consistency. Judge the same pair in AB and BA order and count how often the verdict holds. This needs no human labels, which makes it the cheapest sanity check you have. A judge that scores low here is unstable regardless of what it is measuring.

There are also public datasets built specifically to evaluate judges and reward models — RewardBench and JudgeBench collect comparison pairs with known-correct answers — and they are useful for shortlisting candidate judges. The final call still belongs to your own data.

The minimal judge in code

Including the order swap, a usable implementation is about this size.

def judge_pair(model, question, a, b):
    """Ask in both orders; call it a tie when they disagree."""
    v1 = ask(model, question, first=a, second=b)   # -> "first" | "second"
    v2 = ask(model, question, first=b, second=a)

    if v1 == "first" and v2 == "second":
        return "A"          # A won regardless of position
    if v1 == "second" and v2 == "first":
        return "B"
    return "tie"            # position flipped the verdict = undecidable

The point is to count the ties rather than discard them. The tie rate is a direct measurement of how strong position bias is in your setup, which lets you see whether swapping judges or rewriting the rubric actually helped.

How this gets used in practice

Who touches it, and when. Mostly application developers and ML engineers detecting regressions before and after swapping a prompt or a model. Next, data and evaluation owners re-measuring agreement with human labels on a schedule. And beyond that, teams wiring the judge in as a reward signal for RLHF or on-policy distillation — that last use raises the bar considerably, because any skew in the instrument gets baked into the model's habits.

Parameters you will actually set. On the judge side: temperature (pin it near 0 so verdicts don't wander between runs), logprobs / top_logprobs if you want probability-weighted scoring, and a rubric split by dimension. Scoring "factual accuracy," "instruction following," and "safety" separately, instead of asking "is this good overall," improves reproducibility and tells you why a run dropped. Common tooling: OpenAI Evals, LangSmith, Ragas, DeepEval — most of them ship order-swapping and per-dimension rubrics as features.

Failure modes that turn into incidents.

• Not pinning the judge version. When a hosted model updates quietly, you get a chart showing quality improving while nothing on your side changed. Version-control the judge's model name, version, and prompt alongside your code, and re-measure history whenever you change them.
• Judge and subject from the same family. Self-enhancement creeps in. Avoid it at minimum whenever you plan to claim your model's superiority publicly.
• Not watching length. Always report the correlation between output length and score. Skip this while using the judge as a reward and outputs will inflate by hundreds of characters per round.
• Believing absolute scores. "Average 4.2" means nothing on its own. Only differences under a fixed judge, prompt, and question set are readable.
• Ignoring significance. With 200 items, a 52%-vs-48% win rate is almost certainly noise. Sizing that error bar is the same exercise as in The Statistics of A/B Testing.
• Measuring human agreement exactly once. Task distributions shift. Build a few-hundred-item re-measurement into the quarterly routine.

The question you will be asked. "How did you verify that the judge is right?" Agreement plus kappa, the human–human agreement rate, and order-swap consistency answer it. If you need to grade the intermediate steps rather than only the final artifact — as with agents — the design changes depending on whether you reward outcomes or process; that is covered in Evaluating Agents — Benchmarks and Harnesses.

Summary

• LLM-as-a-Judge measures open-ended quality cheaply and at volume, in three shapes: pointwise, pairwise, and reference-based
• A verdict is really a probability distribution over score tokens; rounding to one integer discards the information that the judge was undecided
• Position, verbosity, and self-enhancement bias all come from the same structure — the judge tracks something correlated with quality instead of quality. Swap the order, monitor and debias length, and use a judge from a different model family
• Once adopted, validate against human labels on your own task. The pass mark is not 100% — it is the agreement rate between humans