Video auto-dubbing utilizing Amazon Translate, Amazon Bedrock, and Amazon Polly

This publish is co-written with MagellanTV and Mission Cloud. 

Video dubbing, or content material localization, is the method of changing the unique spoken language in a video with one other language whereas synchronizing audio and video. Video dubbing has emerged as a key device in breaking down linguistic obstacles, enhancing viewer engagement, and increasing market attain. Nevertheless, conventional dubbing strategies are expensive (about $20 per minute with human evaluation effort) and time consuming, making them a standard problem for corporations within the Media & Leisure (M&E) business. Video auto-dubbing that makes use of the facility of generative synthetic intelligence (generative AI) affords creators an inexpensive and environment friendly answer.

This publish exhibits you a cost-saving answer for video auto-dubbing. We use Amazon Translate for preliminary translation of video captions and use Amazon Bedrock for post-editing to additional enhance the interpretation high quality. Amazon Translate is a neural machine translation service that delivers quick, high-quality, and inexpensive language translation.

Amazon Bedrock is a totally managed service that gives a selection of high-performing basis fashions (FMs) from main AI corporations reminiscent of AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon by a single API, together with a broad set of capabilities that can assist you construct generative AI functions with safety, privateness, and accountable AI.

MagellanTV, a number one streaming platform for documentaries, needs to broaden its international presence by content material internationalization. Confronted with handbook dubbing challenges and prohibitive prices, MagellanTV sought out AWS Premier Tier Associate Mission Cloud for an revolutionary answer.

Mission Cloud’s answer distinguishes itself with idiomatic detection and computerized alternative, seamless computerized time scaling, and versatile batch processing capabilities with elevated effectivity and scalability.

Answer overview

The next diagram illustrates the answer structure. The inputs of the answer are specified by the consumer, together with the folder path containing the unique video and caption file, goal language, and toggles for idiom detector and ritual tone. You’ll be able to specify these inputs in an Excel template and add the Excel file to a chosen Amazon Easy Storage Service (Amazon S3) bucket. This may launch the entire pipeline. The ultimate outputs are a dubbed video file and a translated caption file.

We use Amazon Translate to translate the video caption, and Amazon Bedrock to reinforce the interpretation high quality and allow computerized time scaling to synchronize audio and video. We use Amazon Augmented AI for editors to evaluation the content material, which is then despatched to Amazon Polly to generate artificial voices for the video. To assign a gender expression that matches the speaker, we developed a mannequin to foretell the gender expression of the speaker.

Within the backend, AWS Step Capabilities orchestrates the previous steps as a pipeline. Every step is run on AWS Lambda or AWS Batch. By utilizing the infrastructure as code (IaC) device, AWS CloudFormation, the pipeline turns into reusable for dubbing new overseas languages.

Within the following sections, you’ll learn to use the distinctive options of Amazon Translate for setting formality tone and for customized terminology. Additionally, you will learn to use Amazon Bedrock to additional enhance the standard of video dubbing.

Why select Amazon Translate?

We selected Amazon Translate to translate video captions primarily based on three components.

• Amazon Translate helps over 75 languages. Whereas the panorama of enormous language fashions (LLMs) has repeatedly advanced up to now 12 months and continues to alter, most of the trending LLMs help a smaller set of languages.
• Our translation skilled rigorously evaluated Amazon Translate in our evaluation course of and affirmed its commendable translation accuracy. Welocalize benchmarks the efficiency of utilizing LLMs and machine translations and recommends utilizing LLMs as a post-editing device.
• Amazon Translate has numerous distinctive advantages. For instance, you may add customized terminology glossaries, whereas for LLMs, you would possibly want fine-tuning that may be labor-intensive and expensive.

Use Amazon Translate for customized terminology

Amazon Translate permits you to enter a customized terminology dictionary, guaranteeing translations replicate the group’s vocabulary or specialised terminology. We use the customized terminology dictionary to compile often used phrases inside video transcription scripts.

Right here’s an instance. In a documentary video, the caption file would sometimes show “(talking in overseas language)” on the display because the caption when the interviewee speaks in a overseas language. The sentence “(talking in overseas language)” itself doesn’t have correct English grammar: it lacks the correct noun, but it’s generally accepted as an English caption show. When translating the caption into German, the interpretation additionally lacks the correct noun, which will be complicated to German audiences as proven within the code block that follows.

## Translate - with out customized terminology (default)
import boto3
# Initialize a session of Amazon Translate
translate=boto3.consumer(service_name="translate", region_name="us-east-1", use_ssl=True)
def translate_text(textual content, source_lang, target_lang):
    consequence=translate.translate_text(
        Textual content=textual content, 
        SourceLanguageCode=source_lang, 
        TargetLanguageCode=target_lang)
    return consequence.get('TranslatedText')
textual content="(talking in a overseas language)"
output=translate_text(textual content, "en", "de")
print(output)
# Output: (in einer Fremdsprache sprechen)

As a result of this phrase “(talking in overseas language)” is usually seen in video transcripts, we added this time period to the customized terminology CSV file translation_custom_terminology_de.csv with the vetted translation and offered it within the Amazon Translate job. The interpretation output is as supposed as proven within the following code.

## Translate - with customized terminology
import boto3
import json
# Initialize a session of Amazon Translate
translate=boto3.consumer('translate')
with open('translation_custom_terminology_de.csv', 'rb') as ct_file:
    translate.import_terminology(
        Identify="CustomTerminology_boto3",
        MergeStrategy='OVERWRITE',
        Description='Terminology for Demo by boto3',
        TerminologyData={
            'File':ct_file.learn(),
            'Format':'CSV',
            'Directionality':'MULTI'
        }
    )
textual content="(talking in overseas language)"
consequence=translate.translate_text(
    Textual content=textual content,
    TerminologyNames=['CustomTerminology_boto3_2024'], 
    SourceLanguageCode="en",
    TargetLanguageCode="de"
)
print(consequence['TranslatedText'])
# Output: (Particular person spricht in einer Fremdsprache)

Set formality tone in Amazon Translate

Some documentary genres are typically extra formal than others. Amazon Translate permits you to outline the specified degree of formality for translations to supported goal languages. By utilizing the default setting (Casual) of Amazon Translate, the interpretation output in German for the phrase, “[Speaker 1] Let me present you one thing,” is casual, in accordance with knowledgeable translator.

## Translate - with casual tone (default) 
import boto3
# Initialize a session of Amazon Translate
translate=boto3.consumer(service_name="translate", region_name="us-east-1", use_ssl=True)
def translate_text(textual content, source_lang,target_lang):
    consequence=translate.translate_text(
        Textual content=textual content, 
        SourceLanguageCode=source_lang, 
        TargetLanguageCode=target_lang)
    return consequence.get('TranslatedText')
textual content="[Speaker 1] Let me present you one thing."
output=translate_text(textual content, "en", "de")
print(output)
# Output: [Sprecher 1] Lass mich dir etwas zeigen.

By including the Formal setting, the output translation has a proper tone, which inserts the documentary’s style as supposed.

## Translate - with formal tone 
import boto3
# Initialize a session of Amazon Translate
translate=boto3.consumer(service_name="translate", region_name="us-east-1", use_ssl=True)
def translate_text(textual content, source_lang, target_lang):
    consequence=translate.translate_text(
        Textual content=textual content, 
        SourceLanguageCode=source_lang, 
        TargetLanguageCode=target_lang,
        Settings={'Formality':'FORMAL'})
    return consequence.get('TranslatedText')
textual content="[Speaker 1] Let me present you one thing."
output=translate_text(textual content, "en", "de")
print(output)
# Output: [Sprecher 1] Lassen Sie mich Ihnen etwas zeigen.

Use Amazon Bedrock for post-editing

On this part, we use Amazon Bedrock to enhance the standard of video captions after we receive the preliminary translation from Amazon Translate.

Idiom detection and alternative

Idiom detection and alternative is important in dubbing English movies to precisely convey cultural nuances. Adapting idioms prevents misunderstandings, enhances engagement, preserves humor and emotion, and in the end improves the worldwide viewing expertise. Therefore, we developed an idiom detection perform utilizing Amazon Bedrock to resolve this problem.

You’ll be able to flip the idiom detector on or off by specifying the inputs to the pipeline. For instance, for science genres which have fewer idioms, you may flip the idiom detector off. Whereas, for genres which have extra informal conversations, you may flip the idiom detector on. For a 25-minute video, the whole processing time is about 1.5 hours, of which about 1 hour is spent on video preprocessing and video composing. Turning the idiom detector on solely provides about 5 minutes to the whole processing time.

We’ve developed a perform bedrock_api_idiom to detect and change idioms utilizing Amazon Bedrock. The perform first makes use of Amazon Bedrock LLMs to detect idioms within the textual content after which change them. Within the instance that follows, Amazon Bedrock efficiently detects and replaces the enter textual content “properly, I hustle” to “I work laborious,” which will be translated appropriately into Spanish by utilizing Amazon Translate.

## A uncommon idiom is well-detected and rephrased by Amazon Bedrock 
text_rephrased=bedrock_api_idiom(textual content)
print(text_rephrased)
# Output: I work laborious
response=translate_text(text_rephrased, "en", "es-MX")
print(response)
# Output: yo trabajo duro
response=translate_text(response, "es-MX", "en")
print(response)
# Output: I work laborious

Sentence shortening

Third-party video dubbing instruments can be utilized for time-scaling throughout video dubbing, which will be expensive if performed manually. In our pipeline, we used Amazon Bedrock to develop a sentence shortening algorithm for computerized time scaling.

For instance, a typical caption file consists of a piece quantity, timestamp, and the sentence. The next is an instance of an English sentence earlier than shortening.

Authentic sentence:

A big portion of the photo voltaic vitality that reaches our planet is mirrored again into area or absorbed by mud and clouds.

Right here’s the shortened sentence utilizing the sentence shortening algorithm. Utilizing Amazon Bedrock, we are able to considerably enhance the video-dubbing efficiency and cut back the human evaluation effort, leading to price saving.

Shortened sentence:

A big a part of photo voltaic vitality is mirrored into area or absorbed by mud and clouds.

Conclusion

This new and continually growing pipeline has been a revolutionary step for MagellanTV as a result of it effectively resolved some challenges they have been dealing with which can be frequent inside Media & Leisure corporations usually. The distinctive localization pipeline developed by Mission Cloud creates a brand new frontier of alternatives to distribute content material internationally whereas saving on prices. Utilizing generative AI in tandem with good options for idiom detection and determination, sentence size shortening, and customized terminology and tone leads to a very particular pipeline bespoke to MagellanTV’s rising wants and ambitions.

If you wish to be taught extra about this use case or have a consultative session with the Mission group to evaluation your particular generative AI use case, be at liberty to request one by AWS Market.

In regards to the Authors

Na Yu is a Lead GenAI Options Architect at Mission Cloud, specializing in growing ML, MLOps, and GenAI options in AWS Cloud and dealing intently with prospects. She obtained her Ph.D. in Mechanical Engineering from the College of Notre Dame.

Max Goff is a knowledge scientist/information engineer with over 30 years of software program growth expertise. A printed creator, blogger, and music producer he generally goals in A.I.

Marco Mercado is a Sr. Cloud Engineer specializing in growing cloud native options and automation. He holds a number of AWS Certifications and has in depth expertise working with high-tier AWS companions. Marco excels at leveraging cloud applied sciences to drive innovation and effectivity in numerous tasks.

Yaoqi Zhang is a Senior Massive Information Engineer at Mission Cloud. She makes a speciality of leveraging AI and ML to drive innovation and develop options on AWS. Earlier than Mission Cloud, she labored as an ML and software program engineer at Amazon for six years, specializing in recommender programs for Amazon style buying and NLP for Alexa. She obtained her Grasp of Science Diploma in Electrical Engineering from Boston College.

Adrian Martin is a Massive Information/Machine Studying Lead Engineer at Mission Cloud. He has in depth expertise in English/Spanish interpretation and translation.

Ryan Ries holds over 15 years of management expertise in information and engineering, over 20 years of expertise working with AI and 5+ years serving to prospects construct their AWS information infrastructure and AI fashions. After incomes his Ph.D. in Biophysical Chemistry at UCLA and Caltech, Dr. Ries has helped develop cutting-edge information options for the U.S. Division of Protection and a myriad of Fortune 500 corporations.

Andrew Federowicz is the IT and Product Lead Director for Magellan VoiceWorks at MagellanTV. With a decade of expertise working in cloud programs and IT along with a level in mechanical engineering, Andrew designs builds, deploys, and scales ingenious options to distinctive issues. Earlier than Magellan VoiceWorks, Andrew architected and constructed the AWS infrastructure for MagellanTV’s 24/7 globally obtainable streaming app. In his free time, Andrew enjoys sim racing and horology.

Qiong Zhang, PhD, is a Sr. Associate Options Architect at AWS, specializing in AI/ML. Her present areas of curiosity embody federated studying, distributed coaching, and generative AI. She holds 30+ patents and has co-authored 100+ journal/convention papers. She can be the recipient of the Greatest Paper Award at IEEE NetSoft 2016, IEEE ICC 2011, ONDM 2010, and IEEE GLOBECOM 2005.

Cristian Torres is a Sr. Associate Options Architect at AWS. He has 10 years of expertise working in expertise performing a number of roles reminiscent of: Assist Engineer, Presales Engineer, Gross sales Specialist and Options Architect. He works as a generalist with AWS companies specializing in Migrations to assist strategic AWS Companions develop efficiently from a technical and enterprise perspective.