Audio Compression from Scratch — The Art of Discarding What You Cannot Hear

How does MP3 sound 'the same' at less than a tenth of the size? We build up audio codecs from zero — auditory masking, the MDCT, and bit allocation — and finish with when to reach for MP3, AAC, or Opus.

Raw audio is heavier than you think

Sound is air pressure wobbling very fast. A microphone turns that wobble into a voltage, and an AD converter measures the voltage at fixed intervals, producing a stream of numbers. At CD quality that is 44,100 measurements per second, 16 bits each, times two stereo channels: 44,100 × 16 × 2 ≈ 1.4 million bits per second, or roughly 10 MB per minute. A five-minute song is over 50 MB — the world where your phone streams music all day would never have arrived at that rate.

MP3's classic 128 kbps setting is about 1/11 of that. Lossless compression — the ZIP kind, where you get the original back exactly — barely helps here, because an audio waveform looks nearly random as a number sequence, with almost no repeating patterns to exploit. To drop an order of magnitude you must give up perfect reconstruction and throw something away. The audio codec's answer is precise: throw away what the human ear cannot hear.

An analogy: a whisper on a moving train

A whisper you would hear in a quiet room vanishes inside the roar of a train. The sound still physically exists — it just isn't perceived. This is auditory masking. Inside the ear, a loud sound occupies the sensitivity of the basilar membrane around its own frequency, and information about nearby quiet sounds never reaches the brain.

Codecs exploit two versions of this:

• Frequency masking: a loud sound drowns out quieter sounds at nearby frequencies. While a loud 1 kHz tone plays, a faint tone just next to it in frequency is inaudible. How far the effect reaches is set by the ear's resolution units, called critical bands.
• Temporal masking: quiet sounds just after (and, slightly, just before) a loud sound are also inaudible. For a few tens of milliseconds after a cymbal crash, the ear is still "recovering" and misses fine detail.

The human ear is a superb instrument, then, but one full of blind spots. An audio codec computes a map of those blind spots for each moment of the signal — the psychoacoustic model — and pushes all of its damage into them.

The intuition: don't delete sound — hide noise

Strictly speaking, a codec doesn't pick sounds and "delete" them. What it does is round numbers coarsely (quantization). Rounding saves bits, but the difference from the original becomes quantization noise. The essence of audio compression is hiding that noise under the umbrella of masking.

For each frequency band, the psychoacoustic model computes a masking threshold: "noise below this level will not be heard here." The ratio between the signal's energy and that threshold (the SMR, signal-to-mask ratio) decides how many bits the band needs — generous quantization where the ear is sharp, brutal quantization where it is blind. If the noise stays below the threshold in every band, a file one-tenth the size is indistinguishable to the ear. That is what "discarding inaudible sound" really means.

This three-step shape — transform to frequencies, round coarsely, tuck the damage into perceptual blind spots — is exactly what JPEG does to images. The effect of coarser rounding is easier to feel with a picture, because you can see it. Try it here:

The mechanism: MDCT — slicing into blocks without seams

To vary the rounding per frequency band, we first need to decompose the waveform into frequencies. Since sound changes from moment to moment, we cut the waveform into short blocks (say 2048 samples ≈ 43 ms) and transform each block separately.