Why Opus Won — The Design of a Modern Audio Codec

Audio codecs used to split cleanly into speech codecs and music codecs. Opus ended the split by stacking two engines, SILK and CELT, inside a single bitstream. This piece works up from zero: how linear prediction and the MDCT divide the spectrum between them, why CELT refuses to let band energy collapse, what actually sets a codec's latency, and the technical and institutional reasons Opus became WebRTC's mandatory codec.

Definition of the Opus Audio Codec

For a long time, no instrument played everything

Until roughly 2010, tools for compressing sound fell into two clean camps.

One was the telephone lineage: built for the human voice alone, running at very low bitrates and very short latency. G.729, AMR, Speex. Conversation works fine. Play music through one and it sounds like you are listening from underwater.

The other was the music lineage: MP3, AAC, Vorbis. Full-band music comes through beautifully, but latency is high, and squeezing speech down to a few tens of kbps makes them fall apart.

So videoconferencing apps carried both, guessed whether music was currently playing, and switched between codecs mid-call. Every switch was audible as a hiccup.

Opus (RFC 6716, 2012) is a codec designed to erase that split rather than manage it. From 6 kb/s of thin speech to 510 kb/s of stereo music, from 8 kHz to 48 kHz sampling, from 2.5 ms to 60 ms frames — one codec, one bitstream, covering the whole range continuously. And it switches per packet, with no out-of-band signalling.

When two browsers talk over WebRTC, when Discord carries a voice channel, when YouTube ships audio inside WebM, this is usually what is playing. The question this article answers is how one codec could cover all of that, and the answer lives inside the two engines.

For background: the transform-quantize-pack skeleton is covered in Audio compression from scratch, and the psychoacoustic model that measures the holes in your hearing is in MP3 and psychoacoustics. What follows is the next axis — the one where latency and the network, not the ear, drive the design.

The metaphor: a model of the mouth vs. a recipe of paint

The two lineages disagree about what sound is.

The telephone lineage builds a model of the mouth. A human voice is a buzz from the vocal folds, coloured by resonance in the tube running from throat to lips. So don't send the sound at all — send this frame's tube shape and this frame's buzz, and let the receiver rebuild the sound. Because it knows how speech is made, it is astonishingly cheap. But the model has never heard of a piano or a cymbal, so music does not fit it.

The music lineage writes a recipe of paint. Break the sound into frequency components, tabulate how much of each pitch is present, then round the table's numbers coarsely wherever the ear won't notice. It handles any sound at all — but building the table means looking at a stretch of audio all at once, and that costs latency.

Opus refused to pick one. It stacks both — SILK for the mouth model (descended from the speech codec Skype developed) and CELT (Constrained Energy Lapped Transform) for the recipe — and divides the spectrum between them. That gives three modes: SILK-only, CELT-only, and Hybrid.

The intuition: two engines braided into one packet

In Hybrid mode, SILK takes everything below 8 kHz and CELT takes everything above it. Most of the information in a voice sits low, so that region gets the cheap, accurate mouth model; the highs, where harmonics and consonant sizzle live, get the broad, shallow recipe.

Which of the three modes runs is decided by the encoder from the bitrate and the character of the signal. Told to carry speech at a very low bitrate, it picks SILK-only; given headroom and music, CELT-only; in between, Hybrid. All the caller specifies is a bitrate and an intended use — which engine is running stays hidden.

The decisive part is that both layers share one range coder's state and write into a single interleaved bit string. They are not two files concatenated, so there is no byte-alignment padding wasted at the seam between layers. (Range coding itself is covered in Entropy coding.)

And the first byte of every packet — the TOC byte — announces that packet's own mode, bandwidth, frame size, and channel count. So when the network narrows, the sender can drop to speech mode, narrowband, low bitrate on the very next packet, then climb back to full-band music when it recovers, without renegotiating SDP. The "switching makes it stutter" problem is gone at the design level.