Audio Compression from Scratch — The Art of Discarding What You Cannot Hear

How does MP3 sound 'the same' at less than a tenth of the size? We build up audio codecs from zero — auditory masking, the MDCT, and bit allocation — and finish with when to reach for MP3, AAC, or Opus.

Raw audio is heavier than you think

Sound is air pressure wobbling very fast. A microphone turns that wobble into a voltage, and an AD converter measures the voltage at fixed intervals, producing a stream of numbers. At CD quality that is 44,100 measurements per second, 16 bits each, times two stereo channels: 44,100 × 16 × 2 ≈ 1.4 million bits per second, or roughly 10 MB per minute. A five-minute song is over 50 MB — the world where your phone streams music all day would never have arrived at that rate.

MP3's classic 128 kbps setting is about 1/11 of that. Lossless compression — the ZIP kind, where you get the original back exactly — barely helps here, because an audio waveform looks nearly random as a number sequence, with almost no repeating patterns to exploit. To drop an order of magnitude you must give up perfect reconstruction and throw something away. The audio codec's answer is precise: throw away what the human ear cannot hear.

An analogy: a whisper on a moving train

A whisper you would hear in a quiet room vanishes inside the roar of a train. The sound still physically exists — it just isn't perceived. This is auditory masking. Inside the ear, a loud sound occupies the sensitivity of the basilar membrane around its own frequency, and information about nearby quiet sounds never reaches the brain.

Codecs exploit two versions of this:

• Frequency masking: a loud sound drowns out quieter sounds at nearby frequencies. While a loud 1 kHz tone plays, a faint tone just next to it in frequency is inaudible. How far the effect reaches is set by the ear's resolution units, called critical bands.
• Temporal masking: quiet sounds just after (and, slightly, just before) a loud sound are also inaudible. For a few tens of milliseconds after a cymbal crash, the ear is still "recovering" and misses fine detail.

The human ear is a superb instrument, then, but one full of blind spots. An audio codec computes a map of those blind spots for each moment of the signal — the psychoacoustic model — and pushes all of its damage into them.

The intuition: don't delete sound — hide noise

Strictly speaking, a codec doesn't pick sounds and "delete" them. What it does is round numbers coarsely (quantization). Rounding saves bits, but the difference from the original becomes quantization noise. The essence of audio compression is hiding that noise under the umbrella of masking.

For each frequency band, the psychoacoustic model computes a masking threshold: "noise below this level will not be heard here." The ratio between the signal's energy and that threshold (the SMR, signal-to-mask ratio) decides how many bits the band needs — generous quantization where the ear is sharp, brutal quantization where it is blind. If the noise stays below the threshold in every band, a file one-tenth the size is indistinguishable to the ear. That is what "discarding inaudible sound" really means.

This three-step shape — transform to frequencies, round coarsely, tuck the damage into perceptual blind spots — is exactly what JPEG does to images. The effect of coarser rounding is easier to feel with a picture, because you can see it. Try it here:

The mechanism: MDCT — slicing into blocks without seams

To vary the rounding per frequency band, we first need to decompose the waveform into frequencies. Since sound changes from moment to moment, we cut the waveform into short blocks (say 2048 samples ≈ 43 ms) and transform each block separately.

But if you chop naively, then after quantization the reconstructed waveform disagrees at every block boundary, producing a periodic sputtering noise. The fix is the MDCT (modified discrete cosine transform), the beating heart of virtually every modern audio codec.

It rests on two ideas:

1. Cut blocks with 50% overlap. Each block shares half its samples with its neighbor, and a window function tapers the edges smoothly. Boundary information always lands in two blocks, so the seams disappear.
2. Overlapping would normally double the data, but the MDCT emits only N coefficients from 2N input samples. When reconstruction adds the overlapping halves of adjacent blocks back together, the distortion created by that halving (aliasing) is engineered to cancel exactly, sign against sign. This property is called TDAC — time-domain aliasing cancellation.

The definition:

Reading the symbols one at a time: $x_n$ is one of the $2N$ audio samples inside the window, $w_n$ is the window function that fades the edges to zero, the $\cos[\cdots]$ term is the "stencil" for the $k$-th frequency, $X_k$ measures how much the waveform resembles that stencil, and the $\sum$ is the instruction "multiply sample by stencil at every position, then add the whole lot into one number." Or, put in words with no symbols at all: take a short slice of the waveform, taper its edges so it fades in and out, hold it up against pure tones of many different pitches one by one, and write down how strongly each pitch is present. What comes back is a list of strengths, one per frequency — the picture of this block that the codec will go on to round coarsely. The key is that only half as many numbers come out as went in, so 50% overlap adds no data overall (critical sampling).

The MDCT has one famous weakness. If a sharp attack — a castanet click — lands in the second half of a block, the quantization noise smears across the whole block, so a faint hiss leaks in before the attack. This artifact is called pre-echo. The countermeasure: when the encoder detects a transient, it temporarily switches to shorter blocks (in AAC, 2048 samples down to 256). Long blocks buy frequency resolution, short blocks buy time resolution, and choosing well is where encoders earn their reputation.

Allocate the bits, then pack them

The MDCT coefficients are quantized with a step size the psychoacoustic model picks per band. Bands with headroom under the threshold get rounded aggressively, and most of their coefficients become zero. A stream full of zeros then shrinks further under entropy coding such as Huffman codes. Transform, quantize, entropy-code — this three-stage skeleton is shared by every lossy compressor.

The MDCT in code

In numpy the core is a few lines:

import numpy as np

def mdct(x, N):                       # x: 2N samples (overlap halfway with the previous frame)
    n = np.arange(2 * N)
    w = np.sin(np.pi / (2 * N) * (n + 0.5))       # sine window (edges taper to 0)
    k = np.arange(N)[:, None]
    basis = np.cos(np.pi / N * (n + 0.5 + N / 2) * (k + 0.5))
    return basis @ (w * x)            # N coefficients

Apply the window, take inner products with the cosine stencils — a literal translation of equation (1). Real codecs compute this with fast FFT-based algorithms, but the result is identical.

MP3, AAC, Opus: three generations, three jobs

Everything above — psychoacoustic model, MDCT, quantization, entropy coding — is common to all three. What differs is the generation and the specialty.

• MP3 (MPEG-1 Audio Layer III, standardized in the early 1990s): a hybrid design combining a 32-band filterbank with an MDCT. It loses to its successors on efficiency, but its asset is unmatched: almost no device on Earth fails to play it, and the related patents have expired.
• AAC (MPEG-2/4): MP3's direct successor. A pure-MDCT design with long/short block switching and more flexible coding tables improved quality at the same bitrate. It is the default for Apple devices, YouTube, broadcast, and HLS streaming — the standard for "produce and distribute."
• Opus (IETF RFC 6716, 2012): merges SILK, a speech coder built on linear prediction modeling the vocal tract, with CELT, an MDCT coder for music, in one codec. Frames run 2.5–60 ms for low latency, it operates internally at 48 kHz, it is royalty-free, and it is the standard audio codec of WebRTC.

The working rule today: AAC for storage and distribution, Opus for two-way real-time audio — calls, meetings, in-game voice chat — and MP3 only where legacy compatibility demands it.

How this is used in practice

Who touches this, and when: a media-pipeline engineer at a streaming service designing the audio quality tiers; an app developer wiring up WebRTC calls and choosing codec parameters; an editor deciding export settings for a podcast or video — all of them are turning these exact knobs.

What you actually type: the front door of practice is ffmpeg.

ffmpeg -i in.wav -c:a libopus -b:a 96k -application audio out.opus   # streaming / music
ffmpeg -i in.wav -c:a aac -b:a 192k out.m4a                          # AAC delivery
ffmpeg -i in.wav -c:a libmp3lame -q:a 2 out.mp3                      # LAME quality-based VBR

-b:a sets the bitrate, LAME's -q:a selects quality-based VBR (lower numbers mean higher quality), and libopus's -application switches the psychoacoustic tuning between voip and audio.

Pitfalls that become incidents:

1. Generation loss. Lossy-to-lossy re-encoding stacks fresh quantization noise on every pass. Keep masters lossless (WAV/FLAC) and go lossy exactly once, right before delivery. "Received an MP3, converted it to AAC, shipped it" is a quiet incident.
2. Bluetooth adds another pass. The moment a listener uses wireless earbuds, the device re-encodes with SBC, AAC, or similar. What reaches the ear is always double-compressed — set your quality tiers with that assumption.
3. The sample-rate trap. Opus runs internally at 48 kHz, so 44.1 kHz material gets resampled. If the project touches video (where 48 kHz is standard), recording at 48 kHz from the start avoids grief.
4. Encoder delay and gapless playback. The MDCT overlap and encoder padding insert a sliver of silence at the start of the output. If looping material or a gapless album clicks between tracks, suspect missing gapless metadata first.

If an interview asks "why can MP3 shrink audio to a tenth?", the chain of reasoning is: ① lossless cannot get you there → ② the ear has masking blind spots → ③ transform with the MDCT, hide the quantization noise below the masking threshold, and entropy-code the zero-heavy coefficients.

Wrapping up

• The essence of audio compression is not "removing sounds" but hiding quantization noise below the auditory masking threshold
• The MDCT uses 50% overlap plus aliasing cancellation to erase block seams without increasing the data
• Block-length switching against pre-echo is the shared pressure point of all transform codecs
• The working split: AAC for distribution and storage, Opus for real-time, MP3 for compatibility

Next, see what happens when this skeleton gains prediction along the time axis — that is video: Video Compression from Scratch.