Live Streaming with Automated Multi-Language Subtitling

Live Streaming with Automated Multi-Language Subtitling is a GitHub project that automatically generates multi-language subtitles for live events.

This project uses Amazon Machine Learning (ML) services for transcription and translation. This implementation provides subtitles not to be confused with captions that are used for broadcast television.

On this Page

• Live Streaming with Automated Multi-Language Subtitling
  • On this Page
  • Architecture Overview
  • Launching the stack
  • Deployment
  • Considerations:
    • Amazon Transcribe streaming limits
    • Encoding profile
    • Amazon VPC limits
    • Changing input language
    • Changing translated languages
    • Regional deployment
    • Additional considerations
    • Troubleshooting
  • Creating a custom build
    • Prerequisites:
    • 1. Clone the repo
    • 2. Go to the deployment directory
    • 3. Create an Amazon S3 Bucket
    • 4. Create the deployment packages
    • 5. Launch the CloudFormation template.
  • License

Architecture Overview

The included AWS CloudFormation template deploys the following AWS services:

• AWS Elemental MediaLive
• AWS Elemental MediaPackage
• Amazon CloudFront
• Amazon CloudWatch Events
• Amazon Simple Notification Service (Amazon SNS)
• Amazon Simple Storage Service (Amazon S3)
• Amazon Transcribe
• Amazon Elastic Container Service (Amazon ECS)
• Amazon Translate
• AWS Lambda

In order to get WebVTT subtitles into AWS Elemental MediaPackage, we use the [email protected] function in Amazon CloudFront to insert subtitles into the HLS stream, which is sent from AWS Elemental MediaLive to AWS Elemental MediaPackage. Amazon MediaLive outputs HLS to an Amazon CloudFront endpoint. This endpoint passes through the video files, manifests, and only invokes the [email protected] function for the WebVTT subtitle files passing from AWS Elemental MediaLive to AWS Elemental MediaPackage. Subtitles are inserted into these WebVTT files before they are passed on to Amazon MediaPackage.

AWS Elemental MediaLive outputs an audio-only User Datagram Protocol (UDP) stream to an Amazon ECS container. This container transmits the audio to Amazon Transcribe Streaming, which receives the text contained in the stream as asynchronous responses and writes each text response to an Amazon Dynamo DB table. This Amazon ECS container also sends Amazon SNS notifications to an Amazon Translate Lambda function, which creates translated subtitles that are written to the same Amazon Dynamo DB table.

Each WebVTT file invokes the [email protected] function, which inserts the subtitles and then transmits them onto the MediaPackage. MediaLive provides the authentication headers. MediaPackage ingests the files and packages them into formats that are delivered to four MediaPackage custom endpoints.

An Amazon CloudFront distribution is configured to use the MediaPackage custom endpoints as its origin. This CloudFront URL is what is provided to the viewers of the live steam.

Launching the stack

This automated AWS CloudFormation template deploys Live Streaming with Automated Multi-Language Subtitling on the AWS Cloud.

Note You are responsible for the cost of the AWS services used while running this project. See the Cost section for more details. For full details, see the pricing webpage for each AWS service you will be using in this project.

1. Log in to the AWS Management Console and click the button below to launch the live-streaming-with-automated-multi-language-subtitling AWS CloudFormation template.

You can also download the template as a starting point for your own implementation.

2. The template is launched in the US East (N. Virginia) Region by default. To launch this project in a different AWS Region, use the region selector in the console navigation bar.

Note This project uses the Amazon Translate, Amazon Transcribe, AWS Elemental MediaLive, MediaPackage, and MediaConnect services, which are currently available in specific AWS Regions only. Therefore, you must launch this project in an AWS Region where these services are available. For the most current service availability by region, see the AWS service offerings by region.

3. On the Select Template page, verify that you selected the correct template and choose Next.

4. On the Specify Details page, assign a name to your stack.

5. Under Parameters, review the parameters for the template, and modify them as necessary.

The following default values are used.

Deployment

This is a link to the CloudFormation template to deploy this project.

Considerations:

Amazon Transcribe streaming limits

Amazon Transcribe Streaming is used within the Amazon ECS container. The Amazon Transcribe Streaming quota is five concurrent streams and we recommend requesting a service limit increase for the number of Amazon Transcribe Streams. For more information on limits, refer to Amazon Transcribe Limits. To request a limits increase, use the Amazon Transcribe service limits increase form.

Encoding profile

This project leverages the AWS Elemental MediaLive encoding profile from Live Streaming on AWS. The encoding profile is listed below. • 1080p profile: [email protected], [email protected], [email protected], [email protected]

Amazon VPC limits

This project deploys a new Amazon Virtual Private Cloud (VPC) for the Amazon Transcribe ECS instance. If you plan to deploy more than once instance of this project in one AWS Region, you may need to increase the Amazon VPC quota for your target Region. The default Amazon VPC limit is five per Region.

Changing input language

This project uses AWS Transcribe Streaming to transcribe the source language stored in AWS Elemental MediaLive. The default source language for transcriptions is English (en-US). If your source content is in a different language, change the TranslateLanguage input when launching your CloudFormation template. In addition, you must edit the AWS Elemental MediaLive channel Output 5: english. Modify the Name Modifier language code from _en to your selected language. For example, if your selected language is Spanish (es-US), update the Name Modifier with _es when deploying the CloudFormation stack. You can also change the language code and language description as well.

This project currently only supports the 16 kHz Amazon Transcribe Streaming languages.

Changing translated languages

To change the output languages, you must 1. update the caption output and 2. update the Name Modifier. If you add additional caption outputs to the MediaLive channel, you must add the language code to the SNSTriggerAWSTranslateLambda function as well. To change the translated language:

1. Log in to the AWS MediaLive console.

2. Locate the appropriate channel and under Channel Actions, select Edit Channel. If the channel is already running, choose Stop Channel first.

3. Under Output groups, choose Live (HLS) and choose Add output to add additional translated output languages. Update the Name Modifier with the language code from the Supported language codes in the Amazon Translate Developer Guide. For example: The Name Modifier for Spanish is _es.

4. Choose Update Channel.

5. Navigate to the AWS Lambda console.

6. Locate the SNSTriggerAWSTranslateLambda Lambda function, and update the CAPTION_LANGUAGES variable using the appropriate language code from the Supported language codes in the Amazon Translate Developer Guide. Use a comma to separate multiple languages. For example: CAPTION_LANGUAGES:en,es,fr,de.

Regional deployment

This project uses Amazon Translate, AWS Elemental MediaLive, AWS Elemental MediaPackage, Amazon Transcribe Streaming, and AWS Elemental MediaConnect which are currently available in specific AWS Regions only. Therefore, you must launch this project in an AWS Region where these services are available. For the most current service availability by region, refer to AWS Service offerings by Region.

Additional considerations

This project allows for a single input language and up to five translated caption languages. Similar to a stenographer, the subtitles are slightly time-delayed from the audio. This project is optimized for two second HTTP Live Streaming (HLS) segments on AWS Elemental MediaLive, results are unknown with different segment sizes and may have a poor user experience. This implementation may not be suitable as a replacement for a human stenographer, especially for broadcast applications where users are familiar with human generated subtitles.

Troubleshooting

If you are not getting subtitles when watching your live stream, follow these troubleshooting instructions:

1. Log in to the AWS Management console.

2. Navigate to the Amazon ECS console and locate the ECS container that contains the Stack Name you used to create your ECS service.

3. On the Tasks tab in the ECS service, choose the task name.

4. Choose the log tab. The logs listed can show exceptions, such as LimitExceeded. In this case you have to increase your Amazon Transcribe Streaming limit on your AWS account for the region you are running in.

If you are still not seeing subtitles, restart the ECS task by choosing stop. When you stop an ECS task a new task should start in a few minutes.

Creating a custom build

Prerequisites:

• AWS Command Line Interface
• Python 3.x or later

Follow these steps to generate a CloudFormation template and deploy custom resources to an S3 bucket for launch. To launch the project the Lambda source code has to be deployed to an Amazon S3 bucket in the region you intend to deploy the project.

1. Clone the repo

Download or clone the repo and make the required changes to the source code.

2. Go to the deployment directory

Go to the deployment directory:

cd ./deployment

3. Create an Amazon S3 Bucket

The CloudFormation template is configured to pull the Lambda deployment packages from Amazon S3 bucket in the region the template is being launched in. Create a bucket in the desired region with the region name appended to the name of the bucket. eg: for us-east-1 create a bucket named: my-bucket-us-east-1

aws s3 mb s3://my-bucket-us-east-1

4. Create the deployment packages

Build the distributable:

chmod +x ./build-s3-dist.sh
./build-s3-dist.sh <bucketnsme> live-streaming-with-automated-multi-language-subtitling <version>

Notes: The build-s3-dist script expects the bucket name as one of its parameters, and this value should not include the region suffix

Deploy the distributable to the Amazon S3 bucket in your account:

aws s3 sync ./regional-s3-assets/ s3://my-bucket-us-east-1/live-streaming-on-aws-with-mediastore/<version>/ --acl public-read
aws s3 sync ./global-s3-assets/ s3://my-bucket-us-east-1/live-streaming-on-aws-with-mediastore/<version>/ --acl public-read

5. Launch the CloudFormation template.

• Get the link of the live-streaming-with-automated-multi-language-subtitling.template uploaded to your Amazon S3 bucket.
• Deploy the project in the Amazon CloudFormation console.

License

• This project is licensed under the terms of the Apache 2.0 license. See LICENSE.