Students Professional
JA EN

#arithmetic-coding

2 articles

01 ·Information Theory·★ MEMBER·PAPER·11 min read Compression Is Prediction Is Intelligence — LLMs Through Information Theory Training a model to guess the next token is training it to shrink a file. Route the prediction through arithmetic coding and "probability model" and "code" turn out to be the same object — which makes cross-entropy loss literally the size of the compressed output. Built up from zero, ending at why the Hutter Prize treats compression ratio as an intelligence test. 02 ·Coding Theory·★ MEMBER·9 min read Entropy Coding from Scratch — From Huffman to Arithmetic Coding "Information content becomes code length" — that one line is all of compression. From Shannon's bound to building a Huffman code by hand, the integer-bit ceiling, the arithmetic coders and ANS that break it, and what is actually running inside JPEG, PNG, H.264 and zstd. Ending with how to diagnose a compression ratio that has stopped improving.