arena

Definition

A live evaluation format where models are pitted against each other anonymously and human voters pick a winner, with results aggregated into an Elo-style rating — Chatbot Arena being the canonical instance. Its advantage over fixed benchmarks is resistance to contamination, since prompts come from users and keep changing. Its weakness is the flip side of the same property: rankings reflect whoever shows up to vote and whatever they happen to ask.

Explainers using this term

• LLM-as-a-Judge from Scratch — How AI Grades AI, and Where It BreaksJudging LLM-as-a-Judge with MT-Bench and Chatbot Arena