Status: Work-in-progress. Please create issues or pull requests if you have ideas for improvement.

Scalable Big Blue Button Video Conference Example

Example deployment of a scalable Video Conference System

Summary

This project demonstrates the deployment of a scalable Video Conference Setup jointly using open-source software components and AWS Services.

High-Level architecture

EC2 based deployment - architecture

Serverless based deployment - architecture

Disclaimer

This project is an example of an deployment and meant to be used for testing and learning purposes only. Do not use in production.

Please note that running this code will cause software that is licensed under AGPL-3.0 to be deployed in the user’s account.

Be aware that the deployment is not covered by the AWS free tier. Please use the AWS pricing calculator to an estimation beforehand

Table of Contents

1. Getting started
2. Prerequisites
3. Parameters
4. Templates
5. Code updates
6. Versioning
7. Troubleshooting common errors
8. Resources
9. Security
10. License

Getting started

Just a few steps are needed to get started with the example deployment. the deployment process is separated in a prerequisites deployment containing the creation of the source file Amazon S3 Bucket and another containing the actual deployment of the infrastructure and application layer.

You may use the included setup script to simplify and automatic deployment or alternatively you can run the deployment step-by-step.

Prerequisites

To run the automated, setup script based deployment you need to have some software installed and configured on your device:

• bash (zsh, csh, sh should also work, not tested though)
• an installed and configured aws-cli
• a named profile at the aws-cli configuration reflecting the account you are planning to use for the deployment
• jq

To run the step-by-step setup:

• an installed and configured aws-cli

either way you choose the following have to be in place:

• a valid Route53 Hosted Zone (and it's ZoneID). Either a registered Domain or an external registered Domain (at the target account)
• one or more valid Email Addresses
• Amazon Simple Email Service needs to either being out of the Sandbox mode OR you need to validate destination email adresses piece by piece for the Invitation Emails

Be sure to check the Troubleshooting common errors when deploying.

Parameters

Dynamic parameters

These parameters you have to pass to the setup script

Deployment parameters:

The deployment parameters are placed into the bbb-on-aws-param.json or to be set via cli/console ( if you choose the step-by-step setup. )

Deployment

Automatic

For the automatic deployment just run the included setup script

Example:

./setup.sh -e [email protected] -p bbb_example -h 1XXX02XXXXBMXXXXXZXXXX -s bbbexample -d bbbexample.example.com

The automatic deployment works as follows:

• The setup script will validate the device prerequisites are met and all needed parameters are set.
• It will then validate the syntax of the Amazon Cloudformation templates prior to execute any deployment.
• It's going to deploy the Amazon S3 Bucket needed by the main deployment and read out the Bucket name as well as the name of the Stack deployed.
• It will copy the needed scripts, config files for application and services as well as nested templates to the the deployed Bucket.
• The main deployment will be executed. The script will read the content of the bbb-on-aws-param.json file and pass it through the stack deployment

Step-by-step

If you want to attempt the deployment step-by-step via Console or aws-cli please use the following steps:

• deploy the Source Amazon S3 Bucket for scripts, config files and nested templates

aws cloudformation deploy --stack-name bbbexample-sources --profile=bbb_example --template ./templates/bbb-on-aws-buildbuckets.template.yaml

• copy the content of the templates abd scripts into the Source Bucket

    aws s3 sync --profile=bbb_example ./templates s3://NAMEOFCREATEDBUCKET
    aws s3 sync --profile=bbb_example ./scripts s3://NAMEOFCREATEDBUCKET

• start the deployment using the stackname of the stack deployed beforehand as one of the parameters:

using aws-cli:

aws cloudformation deploy --profile=bbb_example --stack-name bbbexample \
    --capabilities CAPABILITY_NAMED_IAM \
    --parameter-overrides [email protected] BBBStackBucketStack=bbbexample-sources BBBDomainName=bbbexample.example.com BBBHostedZone=1XXX02XXXXBMXXXXXZXXXX \
    --template ./bbb-on-aws-master.template.yaml

The deployment will take approx 30-45 minutes.

Logging into the Big Blue Button:

When deployment went through you find the Administrator login within Secretsmanager. The password is located at the secret starting with BBBAdministratorlogin- While your login has been set to your selected Operator Email Address the password has been generated for you.

log into the Frontend using conference.example.com

Template structure and deployment workflow

The Deployment consists of 2 main templates and 13 nested templates.

Main templates

• The deployment of prerequisites via bbb-on-aws-buildbuckets.template.yaml

  The template deploys the Amazon S3 Bucket containing the scripts used at the turn, application and scalelite deployment as well as the nested templates source files.

• The Master Template for the main deployment bbb-on-aws-master.template.yaml

  The template initiates the overall deployment of the scaling Big Blue Button example deployment.

Nested templates

• Providing the Certificate Automation via: bbb-on-aws-acmcert.template.yaml

  This template deploys a custom resource into your AWS Account which provides full automation of requesting and validating Amazon Certificate manager based TLS Certificates. The validation will be done via Route53 DNS records.

• Dynamically provide the latest AMI to be used with: bbb-on-aws-amifinder.template.yaml

  The template deploys a custom resource into your AWS Account which will access the AWS API to find the ami of the desired non-Amazon Linux Operating System.

• Setup Email sending via Amazon SES: bbb-on-aws-ses.template.yaml

  This template deploys a custom resource which fully automates the Amazon SES configuration for sending mails and validating the domains in Amazon SES via Route53 records.

• Building the network infrastructure using: bbb-on-aws-network.template.yaml

  The template deploys a full Amazon VPC with public and private subnets, route tables, internet gateways, nat gateways.

• Create needed Security Groups: bbb-on-aws-securitygroups.template.yaml

  This template creates all needed security groups for service exposure as well as application backend communication

• Provisioning of the Shared Storage: bbb-on-aws-storage.template.yaml

  This template deploys an Amazon Elastic File System into the setup to provide shared storage for the video conference recordings

• Deploy Amazon Aurora (Postgres): bbb-on-aws-database.template.yaml

  The deployment of Amazon Aurora is needed to provide a database for Greenlight and Scalelite where the video conference schedules, user data and recording information are persistent

• Deploy Amazon Elasticache (Redis): bbb-on-aws-cachedb.template.yaml

  This template deploys an Amazon Elasticache (redis) cluster where security token and conference IDs are located for the call handling via Scalelite

• Fire up the ECS Cluster: bbb-on-aws-ecs.template.yaml

  The template deploys the ECS cluster and EC2 Autoscaling Group with the Launch Configuration for the Amazon EC2 worker nodes. If the parameter BBBECSInstanceType is set to "fargate" the ECS Cluster will be utilizing AWS Fargate for the tasks and EC2 worker instances as well as Autoscaling Groups will not be created

• Add a turnserver to the stack: bbb-on-aws-bbbturn.template.yaml

  We need a turn server to handle the video call if the attendees are located behind a restrictive firewall

• Initiate the frontend and inner-application load balancer to the ECS Cluster: bbb-on-aws-frontendapps.template.yaml

  Greenlight is providing the interface for the Users and the video conference landing page. Scalelite is the inner-application load balancer which provides conference load balancing over multiple big blue button application servers

• Setting up the Big Blue Button Application Instance(s): bbb-on-aws-bbbappscalable.template.yaml (for single server deployments bbb-on-aws-bbbappsingle.template.yaml)

  Finally we will deploy the Big Blue Button application instances. Each instance is basically a separate video conference system.

Custom scripts

During the deployment the EC2 instances will be bootstrapped using UserData. To orchestrate the turn and Big Blue Button servers we will deploy some custom scripts helping to glue the components of the system

• route53-handler systemd service and script deployed to the application and turn server instances creates dynamic records in your hosted zone (Route53) on boot/bootstrap of an instance and removes the record from the hosted zone.
  We do not use ElasticIPs or fixed hostnames at the setup to prevent e.g. service quotas. Also we use dynamic hostnames for each bootstrapped instance to prevent e.g. Let's encrypt penalties

• scalelite-handler systemd service and script *deployed to each application instance adds the instance to scalelite on boot/bootstrap as active video conference instance and evaluates/removes the instance on shutdown"

• turn-handler systemd service/timer and script the turn-handler the currently active turnserver hostname can be found by the application instances. As we dynamically set hostnames the turn servername might change on scaling or termination events. the turn-handler is started via systemd timer every X seconds

• scalelite post script and config the script and config files are added to the Big Blue Button application instances to enable the recordings import into scalelite and Amazon EFS

Configuration adjustments (diffs from the defaults)

• Cloudwatch Agent for application and turn instances the agent is automatically setup via UserData on bootstrap. To sent valid data to the Amazon Cloudwatch Service a custom agent config is used for application and turn servers

Customizing your Big Blue Button deployment

There are several ways how you can further customize your deployment. Apart from the infrastructure components you can customize using the parameters mentioned earlier at the documentation you can also adjust the bootstrap of the Big Blue Button or Greenlight deployment according to your needs. A good starting point is to take a look at the UserData Section of the nested stack for the application instances like: bbb-on-aws-bbbappscalable.template.yaml (for single server deployments bbb-on-aws-bbbappsingle.template.yaml)

Our recommendation is to hook into the bootstrap and alter/extend the Scripts and/or code there. this makes sure your customization will be persistent for all of your deployments and also if you decide to scale-out the application servers. Basically the customization section of the Big Blue Button documentation does content all steps you need.

When it comes to Greenlight there is also a part at the official documentation covering this. As we do use the containerized version of the Greenlight deployment at the scalable option the best way to approach it is to customize and extend the related Greenlight container, push it into your private container registy. Amazon ECR or any docker compatible registry of your choise. And continue with your customized container image setting the related parameter.

Code updates

to update an already deployed stack just pull the current version of the IaC code repository. Afterwards you can start the upgrade process the same way as you would do the initial setup.

Versioning

We're using the Semantic Versioning for this repo. Each major release will be tagged and can be pulled seperately. Be sure NOT to use the main branch if you want to be sure not pulling potential huge changes to the infrastrucutre unintentionally. Use the branches regarding to the major Version you want to stick to.

Troubleshooting common errors

• Failed to create: [BBBTurnAutoScaling] Issue:

  One of the most common errors is, that the Hosted Zone metioned at the Prerequisites is not setup properly or you're at the state of DNS delay. Use the following commands on your command line to evaluate if your DNS Setup is working:

  nslookup thedomain.setupashosted.zone

  This should reply with some basic domain information like assigned DNS Servers. If you get a domain not found error wait a bit if you're sure you followed the docomentation above ( registered Domain or an external registered Domain (at the target account) ) and try like 10min later. If still no results, check your hosted zone and DNS setup. If you use an external Registar (and did not buy a domain using Route53) make sure you registered the Route53 DNS Servers with the Domain as mentioned at the related documentation.

  You do need an own domain to deploy in any case. you cannot mock it using bbb.example.com or any invalid setup here.

• Failed to create: [BBBECSCapacityProvider] Issue:

  There are some cases where the needed Service Role is non-existent at the target account. Solution: Create the missing Service Role manually using the following aws-cli command:

  aws iam create-service-linked-role --aws-service-name ecs.amazonaws.com

Resources

• AWS Services

  • Amazon Cloudformation
  • Amazon EC2
  • Amazon ECS
  • AWS Fargate
  • Amazon Aurora
  • Amazon Elasicache
  • AWS Systems Manager
  • AWS Secrets Manager
  • AWS Certificate Manager
  • Amazon Cloudwatch
  • Amazon Route53
  • Elastic Load Balancing
  • AWS Auto Scaling
  • Amazon EC2 Auto Scaling
  • Amazon Virtual Private Cloud
  • Amazon Elastic File Service
  • Amazon Simple Email Service
  • Amazon S3

• Open Source Projects

  • Big Blue Button
  • Greenlight
  • Scalelite
  • Coturn
  • cfn-ses-provider
  • cfn-certificate-provider
  • cli53

Security

See CONTRIBUTING for more information.

License

This Example is licensed under the MIT-0 License. See the LICENSE file.

3rd party Licensing

Please be aware of the deviating licenses of the deployed open-source software components.

• Big Blue Button: GNU Lesser General Public License v3.0
• Greenlight: GNU Lesser General Public License v3.0
• Scalelite: GNU Affero General Public License v3.0
• cli53: MIT License
• Coturn: New BSD License
• cfn-ses-provider - Apache-2.0 License
• cfn-certificate-provider - Apache-2.0 License

Please note that running this code will cause software that is licensed under AGPL-3.0 to be deployed in the user’s account.