ElevenLabs Launches Dubbing v2 for Enhanced AI Voice Translation

ElevenLabs has unveiled its latest innovation, Dubbing v2, marking a major advancement in AI-powered dubbing technology. This new model takes a fundamentally different approach by listening to the original audio performance rather than relying solely on text transcripts, resulting in translated audio that closely resembles the original speaker's delivery.

How Dubbing v2 Works

Unlike traditional AI dubbing methods that transcribe audio to text, translate it, and then synthesize new speech, Dubbing v2 directly conditions its output on the source audio. This allows the model to capture vocal nuances that a text transcript cannot convey. It supports over 90 languages with regional variants and can handle up to 32 speakers per audio file, making it suitable for various formats, from solo podcasts to ensemble productions.

A notable feature of Dubbing v2 is its sync-aware translation, which automatically synchronizes speech timing with the original audio, eliminating the need for time-consuming lip-sync adjustments. Additionally, it retains the original background audio, ensuring that ambient sounds and music remain intact while only the vocal track is translated.

Availability and Pricing

ElevenLabs is distributing Dubbing v2 through two platforms: ElevenCreative, aimed at individual creators like YouTubers and podcasters, and ElevenProductions, which caters to enterprise clients with larger localization needs. To promote adoption, ElevenLabs has launched a Creator Dubbing Partner Program, offering limited free usage of up to 30 minutes during an initial seven-day promotional period. An API for developers and larger organizations was also rolled out, allowing integration of dubbing capabilities into their products.