alpacaeval

Definition

An automatic benchmark for instruction following, from the Stanford group behind Alpaca. It feeds a fixed prompt set to the model under test and to a reference model, asks an LLM judge which reply is better, and reports a win rate; because that tracks human preference rankings closely, it spread quickly. Its length bias is well documented, so the length-controlled variant is now the number people quote.