MP3 and Psychoacoustics — The Science of Sounds You Cannot Hear

MP3 does not throw away sounds that aren't there — it throws away sounds that are there and inaudible. This piece works through the three holes in human hearing (absolute threshold, simultaneous masking, temporal masking) down to the equations, then follows SMR into bit allocation, the encoder's two nested loops, and the bit reservoir.

Stars Are Out in the Daytime, Too

You can't see stars at noon, but they haven't gone anywhere. The sun is simply too bright for that light to reach your perception. The same thing happens in your ears, and it is called masking. What MP3 discards is not sound that is absent — it is sound that is present and inaudible.

The previous article, Audio Compression from Scratch, followed the skeleton: transform (MDCT) → quantize → entropy-code. This one pulls out the middle of that chain — the psychoacoustic model, which decides every frame how coarsely each frequency band may be rounded. Once you see it, the single line of config that says "128 kbps" turns into what it really is: the average of an enormous number of per-band decisions.

The Ear Is a Measuring Instrument Full of Holes

Human hearing is remarkably sensitive, but as an instrument it is riddled with defects. The useful part is that those defects are reproducible enough to measure. There are three holes.

1. Absolute threshold of hearing — sounds below a certain level are simply not registered, and that level differs by orders of magnitude across frequency
2. Simultaneous masking — while a loud sound is playing, quiet sounds near it in frequency disappear
3. Temporal masking — quiet sounds just after (and slightly before) a loud one disappear

Start with the first. In a silent room, raise the volume from zero and at some point the tone becomes audible. Plot that boundary against frequency and you get a U-shaped curve, lowest — that is, most sensitive — around 3–4 kHz, rising steeply on both sides. The dip sits there because the ear canal, roughly 2.5 cm long and closed at one end, resonates in that region. It is no accident that screams live in that band.

Terhardt's approximation is the standard way to write the curve.

Here $f$ is frequency in Hz and $T_q(f)$ is the sound pressure level at which that frequency becomes audible. Read in words: it is a sum of three parts — the first term is the wall on the bass side, the second carves the dip near 3.3 kHz, the third is the wall on the treble side. Anything below this curve can be deleted outright and nobody notices.

Critical Bands — The Ear's Own Frequency Ruler

Inside the cochlea, the basilar membrane is stiff and narrow at the entrance and floppier and wider deeper in. High tones shake the entrance, low tones the far end: position maps to frequency, so the membrane is a frequency analyzer.

Its resolution, though, is finite. A single point on the membrane responds over a range of frequencies, which makes the ear a bank of overlapping bandpass filters. The width of one filter is a critical band: roughly 100 Hz below 500 Hz, and about 20% of the center frequency above that. Twenty-four to twenty-five of them cover 20 Hz to 20 kHz. So "I can hear up to 20 kHz" is true, and also means you have about 25 distinguishable slots to work with.

Take one of those slots as a unit and you get the Bark scale, usually written with the Zwicker–Terhardt approximation.

$z(f)$ is the Bark value of frequency $f$, and a step of 1 in $z$ is exactly one critical band. Put in words, the formula swaps a physical ruler (hertz) for the ear's ruler (critical bands): the bass end gets stretched, the treble end compressed. On this axis, masking spreads out along near-straight lines, which is what makes the next section tractable.

When a codec says it "varies the coarseness of rounding per band," those bands follow critical bands. In MP3 they are called scalefactor bands — 21 of them for long blocks.

Images work the same way. A JPEG quantization table is essentially a psychoacoustic model for vision, baked into a fixed table instead of computed per frame. Sound doesn't play on a page, so see what coarser rounding does with your eyes first.

Masks Spill Sideways: The Spreading Function

A loud sound — a masker — puts an umbrella over its neighborhood: below this level, nothing is heard. The umbrella spills into adjacent frequencies, and its shape is strongly asymmetric. On the Bark axis the lower side falls off steeply at roughly 27 dB per Bark, while the upper side is a gentle 6–15 dB per Bark, and it flattens further as the masker gets louder, reaching further up. So a loud sound broadly covers everything above it and protects almost nothing below it. That's why bass-heavy material eats bits.

There is a second asymmetry, in the character of the masker. Noise masks tones well: a tone about 5 dB below the noise is already gone. Tones mask noise poorly, needing more than 20 dB of margin. Since the thing we are hiding — quantization noise — is noise, the amount we may spend swings by nearly 20 dB depending on the masker, and getting that call wrong wrecks the audio. MPEG's psychoacoustic model 2 estimates it with an unpredictability measure: extrapolate the current frame's magnitude and phase from the previous two, and call it tone-like when the prediction lands, noise-like when it misses. Sustained tones land; percussion misses.

The final threshold is the envelope of every masker's umbrella plus the absolute threshold, summed in the power domain. Not in dB — convert back with $10^{L/10}$, add, then take the logarithm again.

From SMR to a Number of Bits

For each band, take the distance between the signal level and the threshold. That is the SMR (signal-to-mask ratio). A large SMR means a loud sound in a quiet neighborhood with no room to hide noise; a small SMR means the band is already buried under the umbrella and can be treated carelessly.

The bridge to bits is that each extra bit of quantization buys about 6 dB less noise. Perception's ruler (dB) and bit count are locked to the same step size, so the number of bits falls out of a division.

$b_i$ is the number of bits given to band $i$. Put in words: "you have 30 dB you cannot hide, so pay five bits to crush it." A band whose signal sits entirely below the threshold gets zero bits — the whole band goes. That accumulation of decisions is why low-bitrate MP3s lose their top end wholesale.

Sum $b_i$ over all bands and you get what Johnston named perceptual entropy: an estimate of the bits required to code this instant indistinguishably from the original. It spikes on sharp transients and dense chords. That is the rationale behind variable bitrate.

The Two Nested Loops Inside an MP3 Encoder

So much for theory; the implementation has a problem. You do not know the actual bit count until you have quantized and Huffman-coded. So the encoder doesn't solve for the answer, it searches for it with two nested loops. Quantization itself is nonlinear as well: coefficients pass through a power law, $|x|^{3/4}$, before rounding, which compresses large values and evens out the signal-to-noise ratio across magnitudes.

• Inner loop (rate loop): raise global_gain, the global coarseness, quantize and Huffman-code, and repeat until the bits fit the frame budget
• Outer loop (noise control loop): measure the noise actually produced in each band; wherever it exceeds the masking threshold, raise that band's scalefactor — quantizing that band alone more finely — and go back to the inner loop

The pair terminates when every band is under threshold, when scalefactors hit their ceiling, or when nothing more can be squeezed out. Bad-sounding CBR is usually one of the latter two.

MP3 also has a bit reservoir. Frames are fixed length, but leftover bytes can be borrowed by later frames: a 9-bit pointer called main_data_begin reaches back up to 511 bytes. Savings from a quiet bar get poured into the moment a cymbal hits. That mechanism is why "constant 128 kbps" still has a moment-to-moment variable allocation.

The Hole in Time, and Where Pre-Echo Comes From

The third hole, briefly. After a loud sound stops, the ear misses quiet sounds for something like 50–200 ms (post-masking). Masking also works before a sound starts, but only for about 2–5 ms. That asymmetry produces the worst failure mode of transform codecs. One MP3 granule is 576 samples — about 13 ms at 44.1 kHz. When a sharp attack lands in the back half of a block, quantization noise is smeared evenly across the whole block, including the silence more than 10 ms before the attack. Pre-masking is far too short to cover it, so you hear a "swish": pre-echo. The psychoacoustic model therefore doubles as an attack detector — a spike in perceptual entropy switches the encoder to short blocks, shrinking the window over which noise can smear.

The Model in Code

The skeleton is a few dozen lines of numpy.

import numpy as np

def bark(f):                               # Hz → Bark (the ear's ruler)
    return 13*np.arctan(0.00076*f) + 3.5*np.arctan((f/7500)**2)

def ath(f):                                # absolute threshold of hearing [dB SPL]
    k = f/1000
    return 3.64*k**-0.8 - 6.5*np.exp(-0.6*(k-3.3)**2) + 1e-3*k**4

def spread(dz, level):                     # the umbrella [dB]; dz = target - masker [Bark]
    upper = -(6 + 0.4*level)               # gentle upward, reaching further when loud
    lower = -27.0                          # steep downward
    return np.where(dz >= 0, upper*dz, lower*(-dz))

def threshold(f, spl):                     # every umbrella plus ATH, summed as power
    z = bark(f)
    umbrella = spl[:, None] + spread(z[None, :] - z[:, None], spl[:, None])
    power = np.sum(10**(umbrella/10), axis=0) + 10**(ath(f)/10)
    return 10*np.log10(power)

def bit_allocation(f, spl):                # SMR → bits per band
    return np.maximum(0, np.round((spl - threshold(f, spl))/6.02))

A real encoder layers tonality correction, smoothing, and the two loops above on top — but the core is this: stack umbrellas into an envelope, then convert the gap to the signal into bits.

How This Shows Up on the Job

Who and when: a streaming platform's encoding owner choosing quality tiers; a game developer fitting music and effects into a hard size budget; and an ML engineer designing preprocessing for speech recognition or acoustic anomaly detection — that last one is the easiest to overlook and the most damaging.

What you actually touch: you rarely edit the model's constants directly. You influence it through these.

ffmpeg -i in.wav -c:a libmp3lame -q:a 2 out.mp3        # VBR quality (0 is best; 2 is a sane default)
lame --preset standard -q 0 in.wav out.mp3             # -q is search effort (0 = most thorough, slowest)
lame --lowpass 16 in.wav out.mp3                       # cut the top end explicitly instead of trusting the model

LAME's -q is not a quality knob but a statement of how stubbornly to run those two loops. -q 0 finds a better solution at the same bitrate and takes longer. --lowpass is manual intervention for when you dislike how the model spends bits up high; it earns its keep at low bitrates.

Traps that turn into incidents:

1. Running MP3 through your ML preprocessing. The model's reference is a human ear, not your classifier. In anomaly detection or fault diagnosis, the faint high-frequency content that carries the signal gets deleted for being "inaudible." Keep analysis data in WAV/FLAC.
2. Silence that isn't silent. Thresholds are computed from the signal, so when the signal stops the threshold drops to the absolute threshold and previously hidden quantization noise is exposed. That's the swirling you hear between tracks at low bitrates.
3. Ultrasonic tones and watermarks vanish. Control signals or beacons parked above 18 kHz are the first thing discarded. They cannot survive a path through MP3 or AAC.
4. The model approximates an average ear. Thresholds vary between people, and high-frequency sensitivity declines with age. "I can't hear it, so it's fine" is not a test — use something statistical like an ABX trial.

If it comes up in an interview: "How does MP3 decide what to throw away?" has a four-step answer — drop everything under the absolute threshold; stack maskers per critical band into a threshold envelope; divide the SMR by 6 dB to get bits; then encode and let the loops absorb the overshoot. Add that it hides quantization noise under the threshold rather than deleting sound, and the answer is precise.

Takeaways

• The psychoacoustic model measures the holes in hearing, not the sound: absolute threshold, simultaneous masking, temporal masking
• Critical bands (the Bark scale) are the ear's frequency resolution, and codec band splits follow them
• Masking umbrellas spill much further upward than downward, and noise-like maskers hide far more than tonal ones
• SMR divided by 6.02 dB gives bits; MP3 fits that to the budget with two nested loops and a bit reservoir

How the rounded coefficients get packed at the end is Entropy Coding from Scratch; the machinery that splits a waveform into frequencies in the first place is Understanding the FFT from Scratch.