Easily build data processing pipelines in F#

Provides an extensible solution for creating Data Processing Pipelines in F#.

This solution offers the basic infrastructure to build data processing pipelines in F# (or C# due to it's interoperability with F#).

The implementation is based on the Pipes and Filters Pattern. The goal is to provide the basic Infrastructure as well as some External Interfaces implementations to allow an easy and fast setup of a typical Data Processing Pipeline, following Pipes and Filters pattern.

The typical scenario for establishing communication between Pipes and Filters is by providing integration with Queues, so if you have a scenario like this:

You can implement an External Integration with a typical Messaging Queue System and you get a Pub/Sub system (Event-oriented):

The project FSharp.DataProcessingPipelines.Core provides the basic definition of the Core Entities like Messages, Pipes, Filters and Runners.

Currently supported integrations

For more details about how to use / extend this see the Wiki. If you find any issues, please report in Issues.

This is a open source project from and for the community so feel free to contribute (don't forget to send your contact) - just pick a task in Trello Project!

Note: The solution structure is based on the prototypical .NET solution (file system layout and tooling), recommended by the F# Foundation..