#llm-as-a-judge
3 articles
01
·Evaluation & Judging·FREE·PAPER·10 min read
LLM-as-a-Judge from Scratch — How AI Grades AI, and Where It Breaks
A ground-up guide to using one model to grade another. Covers reading a verdict as a probability distribution, the three recurring biases (position, verbosity, self-enhancement), why pairwise comparison cost grows quadratically, and how to validate the judge itself against human labels.
02
·RAG & Retrieval·★ MEMBER·PAPER·13 min read
Evaluating RAG in Practice — Turning “Seems Better” Into a Number
RAG improvement work gets lost because quality is judged as one vague feeling. This article builds up the metrics that separate retrieval from generation, the definitions of faithfulness and relevance, how to build an eval set with synthetic QA and where that goes wrong, and how many questions you need before the number means anything.
03
·Large Language Models·★ MEMBER·PAPER·11 min read
LLM Evaluation from Scratch — Reading Benchmarks and the Contamination Problem
A guide to reading the bar charts in model release posts with the right kind of suspicion. Covers how the scoring method alone moves MMLU numbers, the error bar that comes from question count, why public benchmarks get contaminated structurally rather than accidentally, the Bradley-Terry model behind Chatbot Arena and where it breaks, and the three biases in LLM-as-a-judge.