All Projects → fnproject → Flow

fnproject / Flow

Licence: apache-2.0
Fn Flow Server

Programming Languages

go
31211 projects - #10 most used programming language

Labels

Projects that are alternatives of or similar to Flow

Serverless Faas Workbench
FunctionBench
Stars: ✭ 32 (-73.77%)
Mutual labels:  faas
Hook.io
Open-Source Microservice Hosting Platform
Stars: ✭ 1,201 (+884.43%)
Mutual labels:  faas
Openfaas Operator
Official OpenFaaS Operator and CRD for Kubernetes
Stars: ✭ 105 (-13.93%)
Mutual labels:  faas
Mongodb Function
OpenFaaS Function that makes use of a connection pool to access MongoDB
Stars: ✭ 44 (-63.93%)
Mutual labels:  faas
Community Cluster
OpenFaaS Cloud Cluster for Community
Stars: ✭ 59 (-51.64%)
Mutual labels:  faas
Faas
Make any Go function into a API (FaaS)
Stars: ✭ 89 (-27.05%)
Mutual labels:  faas
Saasify
The easiest way to monetize your API. 🚀
Stars: ✭ 912 (+647.54%)
Mutual labels:  faas
Antikythera
Elixir framework to build your own in-house PaaS
Stars: ✭ 111 (-9.02%)
Mutual labels:  faas
Openwhisk Cli
Apache OpenWhisk Command Line Interface (CLI)
Stars: ✭ 73 (-40.16%)
Mutual labels:  faas
Riff Tutorial
How-to guide for testing the riff FaaS platform and Istio on Google Kubernetes Engine.
Stars: ✭ 99 (-18.85%)
Mutual labels:  faas
Openwhisk Apigateway
Apache OpenWhisk API Gateway service for exposing actions as REST interfaces.
Stars: ✭ 56 (-54.1%)
Mutual labels:  faas
Apex
Old apex/apex
Stars: ✭ 20 (-83.61%)
Mutual labels:  faas
Faast.js
Serverless batch computing made simple.
Stars: ✭ 1,323 (+984.43%)
Mutual labels:  faas
Openwhisk Runtime Nodejs
Apache OpenWhisk Runtime NodeJS supports Apache OpenWhisk functions written in JavaScript for NodeJS
Stars: ✭ 43 (-64.75%)
Mutual labels:  faas
Functions Samples
Collection of sample apps showcasing popular use cases using Cloud Functions for Firebase
Stars: ✭ 10,576 (+8568.85%)
Mutual labels:  faas
Je
A distributed job execution engine for the execution of batch jobs, workflows, remediations and more.
Stars: ✭ 30 (-75.41%)
Mutual labels:  faas
Tensorflow Lambda Layer
Lets you import Tensorflow + Keras from an AWS lambda
Stars: ✭ 79 (-35.25%)
Mutual labels:  faas
Baetyl
Extend cloud computing, data and service seamlessly to edge devices.
Stars: ✭ 1,655 (+1256.56%)
Mutual labels:  faas
Functionscript
An API gateway and framework for turning functions into web services
Stars: ✭ 1,518 (+1144.26%)
Mutual labels:  faas
Faasd
A lightweight & portable faas engine
Stars: ✭ 1,330 (+990.16%)
Mutual labels:  faas

WARNING Fn Flow is an experimental feature please get in touch on Fn Slack if you are interested in contributing to/using it.

Fn Flow

Fn Project

CircleCI GoDoc Go Report Card

Fn Flow lets you build long-running, reliable and scalable functions using Fn that only consume compute resources when they have work to do and are written purely in code. Flow supports building complex parallel processes that are readable, testable (including via unit testing) using standard programming tools. Flow empowers you to build workflows as distributed programs that are as complex as you need them to be and supports a rich set of concurrency primitives including fork-join, chaining, delays and error handling.

How Fn Flow is different from other workflow tools

Most workflow systems strongly separate the coordination of a process from its execution, typically using declarative workflow programming languages like state machines to execute the workflow in one system and the units of work (e.g. steps, stages) in another.

This creates several new problems that Fn Flow aims to overcome:

  • Two languages, two sets of tools : As a software engineer you are already likely to be skilled at solving problems using existing programming languages and tools; requiring you to spontaneously move to a different paradigm and language just to distribute your process is hard work and slows you down. We think this context-switching is harmful and Flow is designed to let you stick to the languages and tools that you already know to quickly add features.
  • Restricted Semantics : While workflow languages are programming languages like any other, they typically do not have the same semantics as the languages you would use for other tasks, forcing you to reduce your process to something that fits in the semantic model of the workflow system. For state machine type systems this is often problematic as processes grow in complexity. The number of links between states can result in more programming errors and programs that are hard to read and reason about. Flow allows you to code in existing languages (like Java) and use existing code to implement your process.
  • Error handling : In the cloud, processes often need to handle and recover from failures in other systems, this recovery is frequently more complex than just trying again a few more times. Flow exposes these errors in a similar way to exceptions allowing you to handle, recover and take action as you see fit - including running more code to resolve the issue.
  • Glue code and context : Workflow systems are frequently used to orchestrate independent, heterogeneous tasks together. On paper this sounds simple, just drag and drop the tasks into a graph and run the code, right? In practice one task is unlikely to to take exactly the same input as another tasks output (if they did they would not be be very independent!). Tasks may also need context data from other tasks that executed before the previous step. In current workflow languages this often results in you having to add new tasks to translate data from one task to another and introduce complex data handling to pass context around. Flow allows you to use the language features of an existing programming language - glue code can be expressed using (e.g.) Java methods and expressions - without sacrificing type-safety.

For example...

The following Java Fn Flow function coordinates five other functions to:

  1. Scrape results for an image search (using the scraper function)
  2. In parallel, detect number plates in each image in the results (using detect-plates),
  3. For each detected plate, draw the location of the plate on the image (draw) then send an alert containing the rendered image (alert)
  4. Wait for the process to finish and then send a completion message (complete)
public void handleRequest(ScrapeReq input) throws Exception {
   FlowFuture<ScrapeResp> scrapes = currentFlow().invokeFunction("./scraper", input, ScrapeResp.class);
   scrapes.thenCompose(resp -> {
      List<FlowFuture<?>> pendingTasks = resp.result
         .stream()
         .map(scrapeResult -> {
            String id = scrapeResult.id;
            return currentFlow()
                  .invokeFunction("./detect-plates", 
                              new DetectPlateReq(scrapeResult.image_url, "us"), DetectPlateResp.class)
                  .thenCompose((plateResp) -> {
                     if (!plateResp.got_plate) { 
                         return currentFlow().completedValue(null);
                     }
                     return currentFlow()
                           .invokeFunction("./draw", 
                                       new DrawReq(id, scrapeResult.image_url, plateResp.rectangles,"300x300"), DrawResp.class)
                           .thenCompose((drawResp) -> currentFlow()
                                     .invokeFunction("./alert", new AlertReq(plateResp.plate, drawResp.image_url)));
                  });
         }).collect(Collectors.toList());
         
      return currentFlow()
            .allOf(pendingTasks.toArray(new FlowFuture[pendingTasks.size()]))
            .whenComplete((v, throwable) -> {
                  if (throwable != null) {
                     log.info("Scraping completed with at least one error", throwable);
                  } else {
                     currentFlow()
                        .invokeFunction("./complete",CompleteResult("Scraped " + pendingTasks.size() + " Images")));
                  }
               });;
   });
}

While the above program can be written and reasoned about as a single method, (and tested using a JUnit rule) it is in fact executed by braking each stage of the computation down into Fn Functions, each of which runs as an independent Fn call - for instance the results of 'detect-plates' may be processed on one or more different containers. Functions like 'detect-plates' may take a while to run and when they are running none of the surrounding code blocks are consuming resources in the platform.

Writing Flow functions

Flow Functions are currently supported in Java but the platform is not language specific and new language bindings can be build by implementing the function side of the Flow API. We are working on adding support for JavaScript, Python and Go.

To find out how to use Fn Flow in Java read the user guide.

Debugging flows

The Flow Service retains a trace of each Flow's execution and can publish the real time state of an ongoing flow. This can be used to diagnose errors in flows (in a similar way to using a stack trace in a debugger) - We have developed an experimental UI that shows this data in real time and lets you backtrack to the source of problems.

Flow UI

Running the Flow Service

Make sure the functions server is running

$ fn start                                                                                                                                                 ....
time="2017-09-16T22:04:49Z" level=info msg="available memory" ram=1590210560

      ______
     / ____/___
    / /_  / __ \
   / __/ / / / /
  /_/   /_/ /_/

time="2017-09-16T22:04:49Z" level=info msg="Serving Functions API on address `:8080`"

Set FNSERVER_IP to the IP address of the Fn Server:

FNSERVER_IP=$(docker inspect --type container -f '{{.NetworkSettings.IPAddress}}' fnserver)

Then run the Flow Service:

docker run --rm -d \
      -p 8081:8081 \
      -e API_URL="http://$FNSERVER_IP:8080/invoke" \
      -e no_proxy=$FNSERVER_IP \
      --name flowserver \
      fnproject/flow:latest

Configure via the environment

Env Default Usage
API_URL http://localhost:8080 sets the FN API endpoint for outbound invocations
DB_URL sqlite3://./data/flow.db DB url, you may also use "inmem:/" for in memory storage
LISTEN :8081 listen host/port (overrides PORT)

Also see our Flow UI

Get help

Contributing

Please see CONTRIBUTING.md.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].