smithclay / Lambdium

lambdium

headless chrome + selenium webdriver in AWS Lambda

Lambdium uses Selenium Webdriver with Headless Chromium to run Webdriver scripts written in JavaScript on AWS Lambda. Since this project was created, AWS now offers this as a completely managed service called Device Farm Desktop Browser Testing.

You can use this AWS Lambda function by itself, bundled with your own application as a standalone AWS Lambda layer, or with other AWS services to:

• Run many concurrent selenium scripts at the same time without worrying about the infrastructure
• Run execute a selenium script via an HTTP call using API Gateway (example app)
• Configure Cloudwatch events to run a script on a schedule (example app)
• Integrate selenium tests running in Chrome into different event-driven workflows (like CodeDeploy checks, webhooks, or uploads to an S3 bucket)

Since this Lambda function is written using node.js, you can run almost any script written for selenium-webdriver. Example scripts can be found in the examples directory.

This uses a special version of Chrome (headless chromium) from the serverless-chrome project.

This is highly experimental and not all chromedriver functions will work. Check issues for known issues.

Requirements and setup for local development

This project is on the AWS Serverless Application Repository, allowing you to install it in your AWS account with one click. Install in your AWS account here. Quickstart instructions are in the README-SAR.md file.

• An AWS Account
• The AWS SAM Local running functions locally with the Serverless Application Model (see: template.yaml, install: npm install -g aws-sam-local)
• node.js + npm
• modclean npm modules for reducing function size
• Bash

Local development setup

1. Fetching large binary dependencies

The headless chromium binary is too large for Github, you need to fetch it using a script bundled in this repository. Marco Lüthy has an excellent post on Medium about how he built chromium for for AWS Lambda here.

    $ ./layer/fetch-binaries.sh

2. Building

This is now handled by the sam build command. In the root of this project, run:

    $ sam build

3. Running locally with SAM Local

SAM Local can run this function on your computer inside a Docker container that acts like AWS Lambda. To run the function with an example event trigger that uses selenium to use headless chromium to visit google.com, run this:

    $ sam local invoke Lambdium -e event.json

Deploying the function to AWS

To deploy the function to your AWS account, you'll need to have followed the instructions above to fetch dependencies. Running it locally with SAM local and the test event (in event.json) is a good idea to verify everything works correctly before running it in the cloud.

1. Creating a S3 bucket for the function deployment

This will create a file called packaged.yaml you can use with Cloudformation to deploy the function.

You need to create a S3 bucket configured on your AWS account to upload the packed function files. For example:

    $ export LAMBDA_BUCKET_NAME=lambdium-upload-bucket

2. Packaging the function for Cloudformation using SAM

    $ sam package --s3-bucket $LAMBDA_BUCKET_NAME > packaged.yaml

3. Deploying the package using SAM

This will create the function using Cloudformation after packaging it is complete.

    $ sam deploy --template-file ./packaged.yaml --stack-name <<your-cloudformation-stack-name>> --capabilities CAPABILITY_IAM

If set, the optional DEBUG_ENV environment variable will log additional information to Cloudwatch.

Running the function

Post-deploy, you can have AWS Lambda run a selenium script. There's an example of a selenium-webdriver simple script in the examples/ directory that the Lambda function can now run.

Expected JSON input for the function event trigger is: {"Base64Script": "<Base64 Encoding of Selenium Script>"} (this can also be provided as an environment variable named BASE64_SCRIPT).

To run the example Selenium script, you can use the example with the AWS CLI in the scripts directory. It takes care of base64 encoding the file and assumes the function name is lambdium running in us-west-2:

    $ scripts/invoke.sh

To use your own selenium-webdriver script:

    $ scripts/invoke.sh ~/Desktop/my-script.js

Related projects

• serverless-chrome
• How to Get Headless Chrome on Lambda by Marco Lüthy
• Getting Started with Headless Chrome
• Selenium Webdriver 3.0 Docs