Amazon Rekognition Shot Detection Demo using Segment API

This demo solution demostrates how to use Amazon Rekognition Video Segment Detection to detect segments whenever a camera shot has changed and technical cues such as Opening Credits, Content, End Credits, Black Frames, Slate, Studio Logo, and Color Bar.

The animated GIFs illustrate how this demo solution works.

Check out our blog post, Streamline content preparation and quality control for VOD platforms using Amazon Rekognition Video to learn more about the new Segment Version 2 API.

__

Architecture overview

The solution is designed with serverless architecture. It consists of three components, a backend AWS Step Functions state machine, a frontend web user interface, and an Amazon API Gateway RESTful endpoint. The backend state machine uses AWS Elemental MediaConvert to create proxy video and JPEG thumbnails (for streaming and analysis purposes), and runs Amazon Rekognition Video Segment detection to extract the shot segments and technical cues metadata. The frontend web user interface allows user to upload video(s), monitors the detection process, and view the detection results. The frontend and backend communicate through an RESTful API endpoint created by Amazon API Gateway.

The architectural diagram below illustrates an overview of the solution.

User first signs in to the web portal using Amazon Cognito service. The web application is hosted on an Amazon Simple Storage Service (S3), a web bucket indicated in the diagram. The web bucket is protected by Amazon CloudFront distribution with Origin Access Identity (OAID) which restricts direct access to the bucket.

Upon sign-in, the user is authenticated and is given a temporary security credential to access limited AWS resources such as permission to call a specific Amazon API Gateway endpoint and permission to upload video(s) to a specific Amazon Simple Storage Service (S3) bucket, the source bucket. The source S3 bucket is configured to take advantage of Amazon S3 Transfer Acceleration.

The analysis workflow can only be accessed through Amazon API Gateway endpoint where the incoming requests are authenticated with AWS_IAM, the temporary security credential. The API endpoint invokes a lambda function to handle the incoming requests.

An Amazon S3 logs bucket is created to store access logs from Amazon S3 buckets and Amazon CloudFront distribution.

Learn more about the implementation of:

• RESTful API Component
• Shot Detection State Machine
• Webapp Component
• Custom Resources Component used by AWS CloudFormation stack

__

Launching solution with Pre-built AWS CloudFormation Template

The solution is deployed using an AWS CloudFormation template with AWS Lambda backed custom resources. To deploy the solution, use one of the following CloudFormation templates and follows the instructions.

__

• Under Create stack page, click Next to continue

• In Specify stack details page, provide Email and Price Class parameters as follows. Click Next to continue

• In Review stack page, scroll to the bottom and make sure I acknowledge that AWS CloudFormation might create IAM resources. under Capabilities is checked. Then, click on Create stack

The stack creation takes roughly 15 minutes to complete the stack as Amazon CloudFront distribution takes about 15 minutes to propagate to the edge locations.

__

After the stack is created (~15 minutes), you should receive an invitation email from [email protected]. The email contains an Amazon CloudFront URL link to access the demo portal, your login username, and a temporary password.

__

Building and customizing the solution

Build Environment

The sample code is written in NodeJS v14.x. Before you start, please make sure NodeJS has been installed. You would also need to create an Amazon Simple Storage Service (Amazon S3) bucket to store the build artifacts.

NodeJS

Make sure you install NodeJS 14.x onto your system.

For MAC user, download and install from nodejs.org. Alternatively, you can also use Homebrew.

For Linux or Amazon EC2 user, follow Tutorial: Setting Up Node.js on an Amazon EC2 Instance

For Windows 10 user, make sure to install Windows Subsystem for Linux before installing NodeJS. The build and deploy scripts are written in Bash script.

AWS CLI

The deploy-s3-dist.sh shell script uploads the build artifacts to your S3 bucket; thus, it requires AWS CLI to be installed and configured.

# after AWS CLI is installed, run the command and follow the instruction to configure your profile

aws configure

Amazon S3 Bucket

Create a S3 bucket to store the build artifacts: AWS CloudFormation (CFN) templates and Amazon Lambda packages.

Note: make sure to choose the region you intend to run the workflow; for example, us-east-1 region.

__

To build the sample code,

• Clone the repository

git clone [email protected]:awslabs/amazon-rekognition-shot-detection-demo-using-segment-api.git

• Change directory to deployment and run build-s3-dist.sh

cd amazon-rekognition-shot-detection-demo-using-segment-api/deployment
bash build-s3-dist.sh --bucket your-bucket

• Deploy the package to your S3 bucket

bash deploy-s3-dist.sh --bucket your-bucket

# optionally you could specify different AWS CLI Profile,
# AWS CLI profile (default to 'default')
bash deploy-s3-dist.sh --bucket your-bucket \
--profile DevProfile

Now you should have all the code packages and CFN templates uploaded to your S3 bucket.

Log in to AWS S3 Console and navigate to the bucket you created.

Make sure you see the following files under /shot-detection-demo/2.0.0/

__

Launching your customized solution

This section covers two different methods to deploy your customized solution: 1) using AWS CloudFormation Console and 2) using AWS CLI command.

Method 1: Using AWS CloudFormation Console

• Log on to AWS CloudFormation Console
• Click on Create stack with new resources (standard)
• Follow the instruction described earlier. Make sure to specify the S3 location of your customized CloudFormation template under the Amazon S3 URL field.

Method 2: Using AWS CLI Command

• Create a JSON input file as follows and save it locally; ie., cfn-input.json

[
  {
    "ParameterKey": "Email",
    "ParameterValue": "[email protected]"
  },
  {
    "ParameterKey": "PriceClass",
    "ParameterValue": "Use Only U.S., Canada and Europe [PriceClass_100]"
  }
]

• Run the following AWS CLI command to create the stack

aws cloudformation create-stack \
--stack-name shot-detection \
--template-url https://your-bucket.s3.amazonaws.com/shot-detection-demo/latest/main.template  \
--parameters file://cfn-input.json \
--capabilities "CAPABILITY_IAM"

__

Deleting the demo solution

To delete the demo solution, simply delete the CloudFormation stack that was deployed earlier.

Important note: deleteing the CloudFormation stack does NOT remove the following resources. Therefore, make sure you manually clean up these resources to avoid potential recurring cost.

• Amazon S3 bucket (web)
• Amazon S3 bucket (source)
• Amazon S3 bucket (logs)

__

Security

When you build systems on AWS infrastructure, security responsibilities are shared between you and AWS. This shared model can reduce your operational burden as AWS operates, manages, and controls the components from the host operating system and virtualization layer down to the physical security of the facilities in which the services operate. For more information about security on AWS, visit the AWS Security Center.

Server-Side Encryption

AWS highly recommends that customers encrypt sensitive data in transit and at rest. This demo solution automatically encrypts image files and metadata at rest with Amazon Simple Storage Service (Amazon S3) Server-Side Encryption (SSE).

Amazon CloudFront

This demo solution deploys a static website hosted in an Amazon S3 bucket. To help reduce latency and improve security, this solution includes an Amazon CloudFront distribution with an origin access identity, which is a special CloudFront user that helps restrict access to the solution’s website bucket contents. For more information, see Restricting Access to Amazon S3 Content by Using an Origin Access Identity.

__

Cost Estimation

The solution uses various AWS services. Please make sure to check the pricing for each of the services.

• Amazon Rekognition Segment Detection
• AWS Elemental MediaConvert
• Amazon S3
  • Intelligent - Tiering
  • Transfer acceleration
• Amazon CloudFront - on-demand
• Amazon Lambda - request and duration
  • 128MB Memory for status-updater lambda
  • 128MB Memory for shot-detection-large state lambda (mediainfo)
  • 256MB Memory for api lambda to handle OPTIONS, GET, POST requests
• Amazon DynamoDB - on-demand
• Amazon Simple Notification Service
• Amazon Cognito
• Amazon API Gateway - REST API
• AWS CloudFormation
• Amazon CloudWatch Event

Pricing example

Let's assume that we have a 60-minute video with H.264, 1920x1080@24fps@8Mpbs. The total file size is 3.6GB. Let's also assume that we are using US East (N. Virginia) region.

Cost breakdown

The cost of Amazon Cognito and AWS CloudFormation are not included in the estimation.

A few notes

• Total cost to try out the demo solution for an one-hour long HD video is $7.67.
• The Upload (Amazon S3 Storage) cost ($0.0828) is a monthly cost.
• The demo uses both Shot Segment and Technical Cues features. Therefore, the detection cost is multiplied by two.
• The demo uses AWS Elemental MediaConvert Professional Tier features such as:
  • Quality Defined Variable Bitrat (QVBR) to optimize quality and file size and
  • QuickTime output format to carry SMPTE Timecode from original content

__

Supported Regions

Amazon Rekognition Video Segment API is supported in the following regions:

• US East (N. Virginia) (us-east-1)
• US East (Ohio) (us-east-2)
• US West (N. California) (us-west-1)
• US West (Oregon) (us-west-2)
• Europe (Frankfurt) (eu-central-1)
• Europe (Ireland) (eu-west-1)
• Europe (London) (eu-west-2)
• Asia Pacific (Mumbai) (ap-south-1)
• Asia Pacific (Seoul) (ap-northeast-2)
• Asia Pacific (Singapore) (ap-southeast-1)
• Asia Pacific (Sydney) (ap-southeast-2)
• Asia Pacific (Tokyo) (ap-northeast-1)

Make sure to check AWS Region Table for any updated region support for the service.

__

Resources

AWS services

The solution uses the following AWS resources:

• Amazon S3 Buckets (x3)
  • a web bucket to host the web application
  • a source bucket to store images for training data, images for analysis, and dataset manifests
  • a log bucket to store access logs from other AWS resources used in the demo solution
• Amazon Lambda (x3)
  • an API backend lambda function to handle POST request with 1024MB Memory
  • a smaller instance with 256MB Memory of the same lambda function to handle GET and OPTIONS requests
  • a Custom Resource lambda funciton with 256MB Memory to provision resources during the stack creation
• Amazon Cognito
  • an User Pool and Identity Pool to provide sign-in, sign-up, and authentication
• Amazon API Gateway
  • RESTful API endpoints to interact with web appplication
• Amazon CloudFront
  • a CloudFront distribution to host web application
• Amazon Rekognition
  • Video Segment detection APIs to extract shot segments and technical cues (black frame and end credits)
• AWS Elemental MediaConvert
  • Create a MP4 proxy of the original video for analysis and streaming purposes
• AWS Step Functions
  • A state machine that implements the core logic
• Amazon DynamoDB
  • a dynamoDB table to temporarily store Step Functions execution service token, used to communicate back to the state machine
• Amazon Simple Notification Service (SNS)
  • a SNS topic, used by Amazon Rekognition to send notification to our process when a detection job is completed
• Amazon Identity and Access Management
  • IAM Roles for the custom resource and the Lambda function
• AWS CloudFormation

__

Attributions

Images and videos used in this README are courtesy of the Blender Foundation, shared under Creative Commons Attribution 3.0 license.

Tears of Steel: (CC) Blender Foundation | mango.blender.org

__

License

MIT-0

Next to RESTful API Component