Amazon IVS Auto-captions Web demo

A demo web application for demonstrating how you can use Amazon IVS in conjunction with Amazon Transcribe to deliver real-time captions for live streams. This demo also shows how Amazon Translate can be used to deliver auto-translated captions to viewers (optional during deployment).

This project is intended for education purposes only and not for production usage.

This is a serverless web application, leveraging Amazon IVS, Amazon Transcribe, Amazon ECS, Amazon API Gateway, AWS Lambda, Amazon DynamoDB, Amazon S3 and Amazon CloudFront. The web user interface is a single page application built using React.js and the Amazon IVS Player. The demo showcases how you can add real-time live captioning to an Amazon IVS stream using Amazon Transcribe. It also showcases how to configure image overlays to appear on top of the video player based on specific keywords, using TimedMetadata. This demo uses Amazon API Gateway WebSockets to deliver the captions to the connected clients, which are then used as a WebVTT track.

Getting Started

⚠️ IMPORTANT NOTE: Deploying this demo application in your AWS account will create and consume AWS resources, which will cost money.

To get the demo running in your own AWS account, follow these instructions.

1. If you do not have an AWS account, please see How do I create and activate a new Amazon Web Services account?
2. Log into the AWS console if you are not already. Note: If you are logged in as an IAM user, ensure your account has permissions to create and manage the necessary resources and components for this application.
3. Follow the instructions for deploying to AWS.

Deploying to AWS

• To deploy this demo, follow the deployment instructions.
• Once deployed, to configure (optional), follow the configuration instructions.

Architecture

Known issues and limitations ⚠️

• The solution was built for demonstration purposes only and not for production use.
• The solution requires streaming to an ECS container instead of directly to Amazon IVS, which may add points of failure and additional latency.
• The solution is currently limited to a maximum connected viewers of ~200 (this limitation comes from the captions delivery mechanism, not Amazon IVS). Starting from 200 connected users and up, the execution time of the process to deliver the captions to connected clients increases and causes a timeout in the Lambda function (which is set at 3 seconds), resulting in captions not being delivered at all. A possible alternative approach to overcome this limitation would require replacing the WebSocket infrastructure (built on top of API Gateway, Lambda and DynamoDB) with a custom WebSocket Server implementation running in Amazon ECS and AWS Fargate. Read more here.
• The solution's client-side caption syncing mechanism currently relies on an undocumented Player API. This API may be changed or deprecated in the future without notice.
• In Firefox, captions may appear very close to the bottom border of the video when there are 4 or more rows of captions.
• The solution was only tested in us-west-2 (Oregon) and us-east-1 (N. Virginia) regions. Additional regions may be supported depending on service availability.

Estimated costs

Deploying this solution in your AWS account will create and consume AWS resources, which will cost money.

Below is a table with estimated costs for scenarios with 1, 10, and 100 viewers, each receiving video in 1080p resolution during 1 hour with four translations enabled.

Note: These costs are estimates. Cost may vary depending on multiple factors such as (but not limited to) region, amount of viewers, duration, number of captions in the video, enabling the Translate feature, the number of translations that are activated, etc. Note that the estimated prices are in dollars and do not include taxes.

About Amazon IVS

• Amazon Interactive Video Service (Amazon IVS) is a managed live streaming solution that is quick and easy to set up, and ideal for creating interactive video experiences. Learn more.
• Amazon IVS docs
• User Guide
• API Reference
• Learn more about Amazon IVS on IVS.rocks
• View more demos like this