#hutter-prize

1 articles

01 ·Information Theory·★ MEMBER·PAPER·11 min read Compression Is Prediction Is Intelligence — LLMs Through Information Theory Training a model to guess the next token is training it to shrink a file. Route the prediction through arithmetic coding and "probability model" and "code" turn out to be the same object — which makes cross-entropy loss literally the size of the compressed output. Built up from zero, ending at why the Hutter Prize treats compression ratio as an intelligence test.