judging

Definition

Scoring or ranking outputs. For open-ended tasks where no single reference answer exists, string-overlap metrics break down, so evaluation falls back on human raters or on a strong model acting as grader. Because the verdict depends entirely on what the judge was told to value, the credibility of any judging-based result rests on the rubric, the instructions given, and whether ties and refusals were handled consistently.

Explainers using this term

• Paper Walkthrough: S³Gym — Does Self-Testing and Self-Judging Add Up to Self-Improvement?S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?
• LLM-as-a-Judge from Scratch — How AI Grades AI, and Where It BreaksJudging LLM-as-a-Judge with MT-Bench and Chatbot Arena