AWS Lambda Benchmark

A project that contains AWS Lambda function implementations for several runtimes e.g. Nodejs, Haskell, Python, Go, Rust, Java, etc.

• Examples and Performance
  • Hello World
  • CRUD
• CloudWatch Dashboard Screenshots
• Manually Deploying Lambda function
• Triggering your function through API Gateway
• Using Artillery for testing
• Creating a Dashboard using AWS CloudWatch
  • Custom CloudWatch Dashboard Widgets

Examples

Hello World

Hello World HOW-TO

Notes: 1024 MB of memory

CloudWatch Dashboard Screenshots

25-06-2019 Screenshot With Cold Start and Screenshot Without Cold Start

• Created new CloudWatch Dashboard to reflect the duration through Line Graphs

19-06-2019 Screenshot

• Improvements in Lambda Haskell Runtime
• Added Ruby 2.5, C# and F# .NET 2.1 to Benchmarks

18-06-2019 Screenshot

• Improvements in Lambda Haskell Runtime

17-06-2019 Screenshot

• Baseline

CRUD

CRUD HOW-TO

Notes:

• API Gateway Latency is the actual duration of the HTTP Request

17-07-2019 Added an example with C# .NET 2.1 runtime as a layer Screenshot 25-06-2019 Screenshot

• Baseline

Manually deploying Lambda functions

We have avaliable a set of examples that will give you all the stuff required to get your function deployed in AWS.

For example, this page gives you a step by step guide on how to deploy a Rust Lambda function. The Hello-World code we used for benchmarking is here

Triggering your function through API Gateway

First of all, we will need to create a few resources before we can trigger our Lambda Function. Go to API Gateway in the AWS Console.

• Create a new API, select Rest and from New API and choose a name for your API.
• Then create a stage named e.g. dev
• Create a resource named e.g. nodejs-hello and enable CORS
• Within that resource, create a method GET, enable Use Lambda Proxy Integration and type the name of your function under Lambda Function
• Finally click save and click deploy under the dropdown menu of Actions
• Your endpoint URL will be: /, e.g. GET https://0c9lfg7004.execute-api.us-east-1.amazonaws.com/dev/nodejs-hello

If you experience 403 errors when triggering your endpoint, go to Actions and click Deploy API

Note: This API Gateway could be reused for many different Lambda functions by creating a resource for each of them

Using Artillery for testing

Install serverless, artillery and serverless-artillery if you don't have them yet

yarn global add serverless
yarn global add artillery
yarn global add serverless-artillery

or

npm install --global serverless
npm install --global artillery
npm install --global serverless-artillery

Manual Testing Approach

Run a quick test that will perform 10 rps per second during 10 seconds coming from 10 different sources each second

artillery quick --duration 10 --rate 10 -n 1 https://0c9lfg7004.execute-api.us-east-1.amazonaws.com/dev/nodejs-hello

optional -o <your-file.json> could be added if you want the report to be output to a json file.

Automated Testing Approach

Go to the example artillery directory, e.g. cd examples/hello-world/artillery

and then run the following command:

slsart invoke -p artillery-test.yml

This will trigger a set requests for each of the languages we are currently benchmarking in the hello-world example.

If you want the output to be dump to a file, unfortunately, there is no optional parameter -o in Serverless Artillery, we will need to dump the console output to a file as follow: slsart invoke -p artillery-test.yml > results.json

Creating a Dashboard using AWS CloudWatch

AWS CloudWatch is the service where you could find Analytics about your Lambda function. Information about execution time, # invocations, # errors or # throttles. Go to AWS CloudWatch in the AWS Console.

• Click on Dashboards and Create Dashboard
• Create the first Widget of type Number
• Select Lambda as the source for your metric
• Select By Function Name
• And click on all the metrics that you want to track for you would like to track e.g. Duration, Errors
• Click Create Widget and you will see the metrics being displayed

In this example, we selected the following metrics:

Custom CloudWatch Dashboard Widgets

Custom Widgets can be created out of information extracted from Lambda Logs. These metrics are extracted through Queries, for example:

filter @message like /(?i)(Init Duration)/
| parse "REPORT RequestId: *Duration: * ms\tBilled Duration: * ms \tMemory Size: * MB\tMax Memory Used: * MB" as RequestId, Duration,BilledDuration,MemorySize,MaxMemoryUsed
| parse Duration "Duration: *" as actualDuration
| stats max(actualDuration) as ColdStart, max(MaxMemoryUsed) as MaximumMemoryUsed, max(MemorySize) - max(MaxMemoryUsed) as OverProvisionedMemory

The above example will perform the following actions:

• Filter logs that contain "Init Duration"
• Parse the logs and extract variables
• Parse the variable Duration to extract actual duration
• Display stats from variables