All Projects → antonpaquin → Tensorflow Lambda Layer

antonpaquin / Tensorflow Lambda Layer

Licence: mit
Lets you import Tensorflow + Keras from an AWS lambda

Programming Languages

python
139335 projects - #7 most used programming language
shell
77523 projects
python3
1442 projects

Projects that are alternatives of or similar to Tensorflow Lambda Layer

Node Lambda Log
Basic logging mechanism for Node 6.10+ Lambda Functions
Stars: ✭ 115 (+45.57%)
Mutual labels:  aws, serverless, cloud, aws-lambda
Serverlessbydesign
A visual approach to serverless development. Think. Build. Repeat.
Stars: ✭ 254 (+221.52%)
Mutual labels:  aws, serverless, cloud, aws-lambda
Awesome Layers
λ A curated list of awesome AWS Lambda Layers. Sponsored by https://cloudash.dev
Stars: ✭ 1,655 (+1994.94%)
Mutual labels:  aws, serverless, cloud, aws-lambda
Aws Auto Cleanup
Open-source application to programmatically clean your AWS resources based on a whitelist and time to live (TTL) settings
Stars: ✭ 276 (+249.37%)
Mutual labels:  aws, serverless, cloud, aws-lambda
Apex
Old apex/apex
Stars: ✭ 20 (-74.68%)
Mutual labels:  aws, serverless, aws-lambda, faas
Chalice
Python Serverless Microframework for AWS
Stars: ✭ 8,513 (+10675.95%)
Mutual labels:  aws, serverless, cloud, aws-lambda
Components
The Serverless Framework's new infrastructure provisioning technology — Build, compose, & deploy serverless apps in seconds...
Stars: ✭ 2,259 (+2759.49%)
Mutual labels:  aws, serverless, aws-lambda, faas
Cloud Custodian
Rules engine for cloud security, cost optimization, and governance, DSL in yaml for policies to query, filter, and take actions on resources
Stars: ✭ 3,926 (+4869.62%)
Mutual labels:  aws, serverless, cloud, cloud-computing
Webiny Js
Enterprise open-source serverless CMS. Includes a headless CMS, page builder, form builder and file manager. Easy to customize and expand. Deploys to AWS.
Stars: ✭ 4,869 (+6063.29%)
Mutual labels:  aws, serverless, cloud, aws-lambda
Lambdalogs
A CLI tool to trace AWS Lambda calls over multiple CloudWatch log groups.
Stars: ✭ 18 (-77.22%)
Mutual labels:  aws, serverless, aws-lambda, faas
Pulumi
Pulumi - Developer-First Infrastructure as Code. Your Cloud, Your Language, Your Way 🚀
Stars: ✭ 10,887 (+13681.01%)
Mutual labels:  aws, serverless, cloud, cloud-computing
Serverless Es Logs
A Serverless plugin to transport logs to ElasticSearch
Stars: ✭ 51 (-35.44%)
Mutual labels:  aws, serverless, aws-lambda
Serverless Plugin Webpack
Serverless Plugin Webpack
Stars: ✭ 72 (-8.86%)
Mutual labels:  aws, serverless, aws-lambda
Aws Github Actions
Deploy 🚀 to AWS ☁️ with GitHub Actions!
Stars: ✭ 70 (-11.39%)
Mutual labels:  aws, serverless, aws-lambda
Aws Power Tuner Ui
AWS Lambda Power Tuner UI is an open source project creating a deployable easy to use website built on a layered technology stack allowing you to optimize your Lambda functions for cost and/or performance in a data-driven way via an easy to use UI.
Stars: ✭ 52 (-34.18%)
Mutual labels:  aws, serverless, aws-lambda
Aws Serverless Java Container
A Java wrapper to run Spring, Jersey, Spark, and other apps inside AWS Lambda.
Stars: ✭ 1,054 (+1234.18%)
Mutual labels:  aws, serverless, aws-lambda
Serverless Export Env
Serverless plugin to export environment variables into a .env file
Stars: ✭ 51 (-35.44%)
Mutual labels:  aws, serverless, aws-lambda
Serverless Cljs Plugin
Serverless plugin for Clojurescript deployment w/ cljs-lambda
Stars: ✭ 72 (-8.86%)
Mutual labels:  aws, serverless, aws-lambda
Alagarr
🦍 Alagarr is a request-response helper library that removes the boilerplate from your Node.js (AWS Lambda) serverless functions and helps make your code portable.
Stars: ✭ 58 (-26.58%)
Mutual labels:  serverless, aws-lambda, faas
Serverless Node Simple Image Resize
Simple image resize AWS lambda function
Stars: ✭ 74 (-6.33%)
Mutual labels:  aws, serverless, aws-lambda

Tensorflow-Lambda-Layer

Lets you import Tensorflow + Keras from an AWS lambda

What is this?

It's a lambda layer that includes Tensorflow, Keras, and Numpy. You can use it to deploy serverless machine learning models.

Serverless is especially nice for when you want to serve a model that will be accessed infrequently, without paying for an always-on ec2 instance.

If you're a single developer or small org and all you want to do is show off your binary classifier, it's usually possible to stay within the free tier limits if you set things up right.

And even if you're larger, serverless brings a lot of benefits, like transparent scaling and the ability to mostly ignore the hardware.

The problem is, some packages (like Tensorflow) end up hard to use. This repo is an attempt to alleviate that problem.

How do I use it?

Pick an ARN from the tables for the region and Tensorflow version you want (for example, arn:aws:lambda:us-west-2:347034527139:layer:tf_1_11_keras:1)

Tables:

In the AWS lambda management console, create a new function you want to use Tensorflow in, or pick an existing function.

Click

  • layers
    • add layer
      • provide a layer version ARN

Paste the ARN in, add the layer, and you should be able to use the libraries as normal.

Build it yourself

The code involved in generating these layers is all included in src and build_targets. The code is a collection of shell scripts that constructs, uploads, and publishes the lambda zipfiles. It reads AWS credentials from $HOME/.aws, and spawns an instance to actually run the build process (note: does not shut it off automatically).

If you have a set of dependencies you'd like built into a layer, you should add a new directory, following this structure:

  • requirements.txt: The pip packages to install
  • description.txt: The description that will be attached to the published layer
  • hook.sh: Extra commands that run as the last phase of the build step
  • test.py: A python file that should trigger an access of every file used by the library in the course of its execution

If you send this in a pull request and it seems like something people will use, I'll run the build the next chance I get and add it here.

Or you can run it yourself with aws-build-lambda.sh. Make sure you know what this script is doing before you run it!

I think you should build a layer with <packages x,y,z>

Let me know! It's fairly low cost to add a new layer. It doesn't even need to involve tensorflow.

Caveats

This repo will minimize a deployment package by:

  • Only copying source files and files accessed when test.py is run
  • Stripping symbols from shared objects

These steps usually produce good results, but they may end up leaving out something essential. If you see an error that you're not expecting, file an issue including the error and the code that generates it, and I'll see what I can fix.

Tensorflow 1.12 + Keras clocks in at 282M, which is too big to fit into a lambda layer. Unless I can find a way to further reduce the size, I can't support this combination.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].