llm-as-a-judge

Definition

Using a strong LLM to score or compare other models' outputs. It is far cheaper and faster than human annotation and correlates reasonably with human preference on many tasks, which is why it underpins much current evaluation. It also carries documented biases: position bias toward whichever answer is shown first, a preference for longer and more verbose responses, and self-preference for outputs resembling the judge's own style. Serious use pairs it with position swapping and a published rubric.

Explainers using this term

• LLM-as-a-Judge from Scratch — How AI Grades AI, and Where It BreaksJudging LLM-as-a-Judge with MT-Bench and Chatbot Arena