Concept · Computer vision
How Hardcoded Subtitle Extraction Works
A burned-in subtitle is pixels, not text. Getting the text out means finding the pixels that are words, recognizing them with OCR, grouping them into lines and scenes, and timing them — a pipeline that looks easy and is full of failure modes.
Under the hood
The five-stage pipeline
The output is a timed subtitle file (SRT/VTT) that matches the burned-in text frame-for-frame.
- 1. Text detection — find regions that look like text (scene text detection, not OCR yet)
- 2. Text recognition — OCR each region into characters and words
- 3. Tracking — follow each subtitle box across frames so it is counted once
- 4. Grouping & filtering — drop watermark text, logos, in-scene signage; keep dialogue lines
- 5. Timing — derive in/out timestamps from when a line appears and disappears
Why OCR on video is harder than OCR on documents
Documents are clean, straight, high-contrast. Video subtitles are anti-aliased, moving, over live picture, sometimes stylized, and often cropped mid-line between scenes. A document OCR engine alone fails on 20–30% of frames; the video-aware pipeline above is what closes that gap.
Where extraction fails — honestly
| Failure case | Why | Mitigation |
|---|---|---|
| Stylized/colored text | Low OCR confidence | Multi-model OCR, voting |
| Text over busy backgrounds | Detection merges text & texture | Higher-res detection, patch-based OCR |
| Scrolling credits | Text never static | Dedicated credit-scroll mode |
| Watermarks & logos | False positives | Filtering rules, post-edit pass |
Timing accuracy matters
A subtitle file whose text is perfect but whose timing is off by half a second is useless for localization or accessibility. The pipeline derives timing from detection events across frames, not frame-grabs — which is why the tracking stage exists.
API shortcut
FAQ
Common questions
Can I extract subtitles from any video?
If the text is legible, usually yes. Extremely stylized or very low-resolution text is the hard tail.
Is the result as good as the original subtitle file?
Text accuracy is very high on clean sources; timing matches the pixels. It will not recover styling metadata that the original file may have had (e.g. colors, positions).
Does it also capture on-screen text like signs?
Detection sees it, but the pipeline filters to subtitle-like regions by default — you can toggle capturing all scene text.
Keep exploring
Related guides
Subtitle Extractor API
Extract on-screen text and spoken dialogue with frame-accurate timestamps.
Learn moreVideo OCR vs Speech To Text
Speech-to-text listens to audio; video OCR reads text burned into the picture. Learn which one extracts the subtitl...
Read guideSRT vs VTT Subtitle Formats
SRT vs WebVTT for subtitles and captions: file structure, styling, positioning, cue settings, browser & platform su...
Read guide