What Is Audio Tokenization? How Neural Codecs Turn Sound into Tokens
EnCodec, SoundStream and RVQ — why modern TTS runs like an LLM
Audio Tokenization is a core technology changing the paradigm of speech/audio AI. Previously, TTS used a continuous pipeline of text→mel spectrogram→waveform, but Neural Audio Codec converts audio into "tokens." Meta's EnCodec and Google's SoundStream are representative examples. Audio is compressed with an encoder and then converted to multi-layer discrete codebook indices via Residual Vector Quantization (RVQ). The first codebook handles semantic/prosody information, while subsequent codebooks handle acoustic details. By generating these token sequences with a Transformer, high-quality audio can be produced in the same way as text generation. VALL-E, Bark, GPT-SoVITS, etc. utilize this paradigm.
Key Concepts
Compress audio waveform into latent representation using CNN encoder
Residual Vector Quantization (RVQ): quantize latent representation into multi-layer codebooks
Layer 1 codebook: semantic/prosody information (semantic tokens) — most important information
Layer 2-8 codebooks: acoustic details (acoustic tokens) — progressively adding detail
CNN decoder reconstructs audio waveform from quantized codes
Result: audio represented as discrete token sequences like text → can be input to Transformers
Pros
- ✓ Audio can be processed by Transformers like text
- ✓ LLM advances directly applicable to speech/audio
- ✓ Very high compression rate (24kHz → 75 tokens per second)
- ✓ Foundation for multimodal unified models (text+audio in same token space)
Cons
- ✗ Audio quality loss during quantization
- ✗ Codebook size/layer count tuning required
- ✗ Difficulty maintaining consistency in long audio
- ✗ Large training data required (codec training itself)