GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects β†’ ryfeus β†’ Stepfunctions2processing

ryfeus / Stepfunctions2processing

Licence: mit
Configuration with AWS step functions and lambdas which initiates processing from activity state

Programming Languages

python
139335 projects - #7 most used programming language
processing
702 projects

Labels

awsserverlessworkflowlambdabatch

Projects that are alternatives of or similar to Stepfunctions2processing

Serverless Stack
πŸ’₯ Serverless Stack (SST) is a framework that makes it easy to build serverless apps.
Stars: ✭ 1,252 (+1291.11%)
Mutual labels:  aws, serverless, lambda
Serverless Slackbot
A boilerplate Serverless Slackbot framework with a custom scripts folder (like Hubot)
Stars: ✭ 72 (-20%)
Mutual labels:  aws, serverless, lambda
Serverless Api Example
Example of a Golang, Serverless API
Stars: ✭ 62 (-31.11%)
Mutual labels:  aws, serverless, lambda
Lambda Audio
Run Sound eXchange (SoX), the Swiss Army knife of audio manipulation, with Lame on AWS Lambda
Stars: ✭ 53 (-41.11%)
Mutual labels:  aws, serverless, lambda
Aws Serverless Airline Booking
Airline Booking is a sample web application that provides Flight Search, Flight Payment, Flight Booking and Loyalty points including end-to-end testing, GraphQL and CI/CD. This web application was the theme of Build on Serverless Season 2 on AWS Twitch running from April 24th until end of August in 2019.
Stars: ✭ 1,290 (+1333.33%)
Mutual labels:  aws, serverless, lambda
Apex
Old apex/apex
Stars: ✭ 20 (-77.78%)
Mutual labels:  aws, serverless, lambda
Ask Around Me
The Ask Around Me example serverless web application. See the Compute Blog series and video series for more information. Contact @jbesw for info.
Stars: ✭ 67 (-25.56%)
Mutual labels:  aws, serverless, lambda
Terraform Nextjs Plugin
A plugin to generate terraform configuration for Nextjs 8 and 9
Stars: ✭ 41 (-54.44%)
Mutual labels:  aws, serverless, lambda
Lambda Refarch Webapp
The Web Application reference architecture is a general-purpose, event-driven, web application back-end that uses AWS Lambda, Amazon API Gateway for its business logic. It also uses Amazon DynamoDB as its database and Amazon Cognito for user management. All static content is hosted using AWS Amplify Console.
Stars: ✭ 1,208 (+1242.22%)
Mutual labels:  aws, serverless, lambda
Serverless Node Simple Messaging
Simple email AWS lambda function
Stars: ✭ 75 (-16.67%)
Mutual labels:  aws, serverless, lambda
Serverless Es Logs
A Serverless plugin to transport logs to ElasticSearch
Stars: ✭ 51 (-43.33%)
Mutual labels:  aws, serverless, lambda
Historical
A serverless, event-driven AWS configuration collection service with configuration versioning.
Stars: ✭ 85 (-5.56%)
Mutual labels:  aws, serverless, lambda
Serverless Application Model
AWS Serverless Application Model (SAM) is an open-source framework for building serverless applications
Stars: ✭ 8,305 (+9127.78%)
Mutual labels:  aws, serverless, lambda
Up
Up focuses on deploying "vanilla" HTTP servers so there's nothing new to learn, just develop with your favorite existing frameworks such as Express, Koa, Django, Golang net/http or others.
Stars: ✭ 8,439 (+9276.67%)
Mutual labels:  aws, serverless, lambda
Chalice
Python Serverless Microframework for AWS
Stars: ✭ 8,513 (+9358.89%)
Mutual labels:  aws, serverless, lambda
Binaryalert
BinaryAlert: Serverless, Real-time & Retroactive Malware Detection.
Stars: ✭ 1,125 (+1150%)
Mutual labels:  aws, serverless, lambda
Workshop Donkeytracker
Workshop to build a serverless tracking application for your mobile device with an AWS backend
Stars: ✭ 27 (-70%)
Mutual labels:  aws, serverless, lambda
Serverless Plugin Stackstorm
Plugin for serverless framework to run ready to use actions from StackStorm Exchange as AWS Lambda.
Stars: ✭ 28 (-68.89%)
Mutual labels:  aws, serverless, lambda
Serverless Node Simple Image Resize
Simple image resize AWS lambda function
Stars: ✭ 74 (-17.78%)
Mutual labels:  aws, serverless, lambda
Seoul Bike
μ„œμšΈμ‹œ μžμ „κ±° 따릉이λ₯Ό μœ„ν•œ λŒ€μ—¬μ†Œ μ°ΎκΈ° μ„œλΉ„μŠ€ μž…λ‹ˆλ‹€.
Stars: ✭ 78 (-13.33%)
Mutual labels:  aws, serverless, lambda
View All Similar Projects βž”

stepfunctions2processing

Configuration with AWS step functions and lambdas which initiates processing from activity state

Motivation

Currently AWS Step Function service provides the best way to unite together serverless processing and cluster processing. Serverless processing can be extremely useful for the cases when you need to have a very good scalability or cheap processing. But for the long or CPU heavy processing you may still need to use cluster. The current version of step function expects cluster (except for ECS and Batch) to make requests to the Step Function activity. For clusters with extremely expensive machines it may not be the best way for handling so this project allows to transform activity state into active processing trigger. Examples here also include AWS Batch and AWS Fargate

TL;DR: Project allows to initiate processing from AWS Step function instead of making state requests towards it.

Examples

AWS Lambda example

Scheme

Image

  1. Step function process starts and input data is provided to lambda
  2. Lambda processes data and produces json output which is then given to the activity.
  3. Checker lambda makes regular requests to the activity in case activity is in "ActivityScheduled" state.
  4. Activty returns token and input json.
  5. Checker lambda initiates processing lambda which takes token and input json as input.
  6. Processing lambda conducts processing part and return token and output json to activity.
  7. Step functions provides output json from the activity as output

AWS Fargate example

How to deploy

  1. Push docker image into ECR (https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html)
  2. Update serverless.yml with image name and instance parameters
  3. Install and configure serverless (https://serverless.com/framework/docs/providers/aws/guide/installation/)
  4. Install plugins serverless-step-functions and serverless-pseudo-parameters
  5. Run serverless deploy

Scheme

Image

AWS Batch example

How to deploy

  1. Push docker image into ECR (https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html)
  2. Update serverless.yml with image name and instance parameters
  3. Install and configure serverless (https://serverless.com/framework/docs/providers/aws/guide/installation/)
  4. Install plugins serverless-step-functions and serverless-pseudo-parameters
  5. Run serverless deploy

Scheme

Image

Tools

Project utilises serverless framework with plugin for step functions for orchestration and deployment of the application.

References

Next steps

  • [x] Add AWS Batch processing example
  • [x] Add AWS Fargate processing example
  • Create example for deep learning/machine learning training processes
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].