Concept · Audio AI
Audio Source Separation Explained
Every song is a mixture. Source separation is the art of reversing the mix — splitting the single audio file back into its component stems with machine learning. Here's how it works, how good it is, and where it breaks.
Background
The core idea
A mix is the sum of its parts: vocals over drums over bass over guitars, each shaped by pan, EQ and effects. Separation models learn the inverse mapping — from the mixture spectrogram back to the individual instrument spectrograms — by training on millions of real (mixture, stems) pairs.
The model families that matter
| Family | Representative | Strength | Weakness |
|---|---|---|---|
| Spectrogram U-Net | Open-Unmix, Spleeter | Fast, small, good vocals | Less instrument detail |
| Waveform-spectrogram hybrid | Demucs / HTDemucs | State of the art quality | Heavier compute |
| Transformers on spectrograms | BS-RoFormer, recent SOTA | Top quality, music-aware | Larger models, slower |
| Speech-specific | Trained on dialogue/music | Best for podcasts & VO | Not for songs |
How quality is measured
The standard metric is SDR (Signal-to-Distortion Ratio, in dB): how much of the output is the true stem vs. artifacts and bleed. Human listening matters more — SDR correlates with but does not equal perceived quality. Modern models land ~8–12 dB SDR for vocals on clean mixes, which sounds like 'very clean but not identical to the original track'.
What separation is good at — and bad at
- Good: vocals vs instrumental (karaoke), drums vs rest, bass isolation, speech from music
- Decent: guitar vs keys when timbres differ clearly
- Bad: two similar-timbre instruments panned center (e.g. doubled guitars)
- Worse: heavy effects (reverb/delay) smear the stem boundaries
Speech vs music modes
Speech separation (voiceover over music, podcasts, meetings) is a different problem from music separation — the target is one voice, not a set of instruments. Dedicated speech modes use models trained for exactly that, with denoising built in. Use the right mode for the right input.
FAQ
Common questions
Is AI vocal separation perfect?
No. Expect clean but imperfect stems: some bleed and artifacts remain, especially at low bitrates and with heavy effects. On good sources it is transparent enough for karaoke, remixing and content work.
What is a stem?
A stem is an isolated audio component of a mix — e.g. just the vocals, or just the drums. Separating a song into stems is the standard precursor to remixing, upmixing and DJing tools.
Does separation work on low-quality audio?
It degrades gracefully but noticeably. 128 kbps MP3 sources lose high-frequency detail the models need. Use the best available source.
Keep exploring
Related guides
Vocal Separator API
Isolate vocals or instrumentals from speech and music mixes.
Learn moreSource Separation API
Separate any mix into clean stems — including environmental sound.
Learn moreSpleeter vs Demucs
Spleeter vs Demucs (HTDemucs) — the two open-source separation standards compared on quality, speed, GPU/CPU needs,...
Read guide