All Projects → apache → Incubator Streampipes

apache / Incubator Streampipes

Licence: apache-2.0
Apache StreamPipes - A self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams.

Programming Languages

java
68154 projects - #9 most used programming language

Projects that are alternatives of or similar to Incubator Streampipes

Samsara
Samsara is a real-time analytics platform
Stars: ✭ 132 (-36.84%)
Mutual labels:  analytics, stream-processing, iot
Bistro
A general-purpose data analysis engine radically changing the way batch and stream data is processed
Stars: ✭ 333 (+59.33%)
Mutual labels:  analytics, stream-processing, iot
Mainflux
Industrial IoT Messaging and Device Management Platform
Stars: ✭ 1,341 (+541.63%)
Mutual labels:  iot, edge
Baetyl
Extend cloud computing, data and service seamlessly to edge devices.
Stars: ✭ 1,655 (+691.87%)
Mutual labels:  iot, edge
Keylime
A CNCF Project to Bootstrap & Maintain Trust on the Edge / Cloud and IoT
Stars: ✭ 128 (-38.76%)
Mutual labels:  iot, edge
Iotplatform
An open-source IoT platform that enables rapid development, management and scaling of IoT projects. With this IoT platform, you are able to: 1) Provision and control devices, 2) Collect and visualize data from devices, 3) Analyze device data and trigger alarms, 4) Deliver device data to other systems, 5) Enable use-case specific features using customizable rules and plugins.
Stars: ✭ 82 (-60.77%)
Mutual labels:  analytics, iot
Neardb
Simple document db made for infinitely scalable globally distributed reads.
Stars: ✭ 92 (-55.98%)
Mutual labels:  iot, edge
Timescaledb
An open-source time-series SQL database optimized for fast ingest and complex queries. Packaged as a PostgreSQL extension.
Stars: ✭ 12,211 (+5742.58%)
Mutual labels:  analytics, iot
Macchina.io
macchina.io IoT Edge Device SDK is a powerful C++ and JavaScript SDK for edge devices, IoT gateways and connected embedded systems.
Stars: ✭ 437 (+109.09%)
Mutual labels:  iot, edge
Awesome Edge Machine Learning
A curated list of awesome edge machine learning resources, including research papers, inference engines, challenges, books, meetups and others.
Stars: ✭ 139 (-33.49%)
Mutual labels:  iot, edge
Flogo
Project Flogo is an open source ecosystem of opinionated event-driven capabilities to simplify building efficient & modern serverless functions, microservices & edge apps.
Stars: ✭ 1,891 (+804.78%)
Mutual labels:  iot, edge
Redisgears
Dynamic execution framework for your Redis data
Stars: ✭ 152 (-27.27%)
Mutual labels:  analytics, stream-processing
Esp8266workshop
IoT workshop based on ESP8266, a DHT11/22 and neopixel RGB LED
Stars: ✭ 71 (-66.03%)
Mutual labels:  analytics, iot
Stream
Elegant stream processing pipeline written entirely in Golang
Stars: ✭ 45 (-78.47%)
Mutual labels:  stream-processing, edge
Logisland
Scalable stream processing platform for advanced realtime analytics on top of Kafka and Spark. LogIsland also supports MQTT and Kafka Streams (Flink being in the roadmap). The platform does complex event processing and is suitable for time series analysis. A large set of valuable ready to use processors, data sources and sinks are available.
Stars: ✭ 97 (-53.59%)
Mutual labels:  analytics, stream-processing
Iot Edge V1
Azure IoT Edge
Stars: ✭ 522 (+149.76%)
Mutual labels:  iot, edge
Walkoff Apps
WALKOFF-enabled applications. #nsacyber
Stars: ✭ 125 (-40.19%)
Mutual labels:  analytics, iot
Kuiper
A lightweight IoT edge analytics software
Stars: ✭ 327 (+56.46%)
Mutual labels:  iot, edge
Vectorsql
VectorSQL is a free analytics DBMS for IoT & Big Data, compatible with ClickHouse.
Stars: ✭ 171 (-18.18%)
Mutual labels:  stream-processing, iot
Salt
Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
Stars: ✭ 12,086 (+5682.78%)
Mutual labels:  iot, edge

Github Actions Docker pulls Maven central License Last commit Twitter


StreamPipes Logo

Self-Service Data Analytics for the (Industrial) IoT

StreamPipes is a self-service (Industrial) IoT toolbox to enable non-technical users to connect , analyze and explore IoT data streams.

StreamPipes Pipeline
     Editor


Table of contents


About Apache StreamPipes

Apache StreamPipes (incubating) enables flexible modeling of stream processing pipelines by providing a graphical modeling editor on top of existing stream processing frameworks.

It empowers non-technical users to quickly define and execute processing pipelines based on an easily extensible toolbox of data sources, data processors and data sinks. StreamPipes has an exchangeable runtime execution layer and executes pipelines using one of the provided wrappers, e.g., standalone or distributed in Apache Flink.

Pipeline elements in StreamPipes can be installed at runtime - the built-in SDK allows to easily implement new pipeline elements according to your needs. Pipeline elements are standalone microservices that can run anywhere - centrally on your server, in a large-scale cluster or close at the edge.

Use Cases

StreamPipes allows you to connect IoT data sources using the SDK or the built-in graphical tool StreamPipes Connect.

The extensible toolbox of data processors and sinks supports use cases such as

  • Continuously store IoT data streams to third party systems (e.g., databases)
  • Filter measurements on streams (e.g., based on thresholds or value ranges)
  • Harmonize data by using data processors for transformations (e.g., by converting measurement units and data types or by aggregating measurements)
  • Detect situations that should be avoided (e.g., patterns based on time windows)
  • Wrap Machine Learning models into data processors to perform classifications or predictions on sensor and image data
  • Visualize real-time data from sensors and machines using the built-in Live Dashboard

Installation

The quickest way to run StreamPipes including the latest extensions (adapters, pipeline elements) is by using our Docker-based installation & operation options, namely:

NOTE: StreamPipes CLI & k8s are highly recommended for developers or operators. Standard users should stick to StreamPipes Compose.

Please follow the instructions provided in the corresponding README.md to get started.

For a more in-depth manual, read the installation guide.

Building StreamPipes

To properly build the StreamPipes core, the following tools should be installed:

Prerequisites

  • Java 8 JDK (minimum)
  • Maven (tested with 3.6)
  • NodeJS + NPM (tested with v12+/ v6+)
  • Docker + Docker-Compose

Building

To build the core project, do the following:

    mvn clean package

To build the ui, switch to the ui folder and perform the following steps:

    npm install
    npm run build

Starting

To start StreamPipes, run docker-compose up -d from the root directory.

You can also use the installer or CLI as described in the Installation section.

Pipeline Elements

StreamPipes includes a repository of extensions for

  • Connect adapters for a variety of IoT data sources as well as
  • Data Processors and Data Sinks as ready-to-use pipeline elements.

A description of the standard elements can be found in the Github repository streampipes-extensions.

Extending StreamPipes

You can easily add your own data streams, processors or sinks. A Java-based SDK and several run-time wrappers for popular streaming frameworks such as Apache Flink, Apache Spark and Apache Kafka Streams (and also plain Java programs) can be used to integrate your existing processing logic into StreamPipes. Pipeline elements are packaged as Docker images and can be installed at runtime, whenever your requirements change.

Check our developer guide at https://streampipes.apache.org/docs/docs/dev-guide-introduction.

Bugs and Feature Requests

If you've found a bug or have a feature that you'd love to see in StreamPipes, feel free to create an issue in our Jira: https://issues.apache.org/jira/projects/STREAMPIPES

Get help

If you have any problems during the installation or questions around StreamPipes, you'll get help through one of our community channels:

And don't forget to follow us on Twitter!

Contribute

We welcome contributions to StreamPipes. If you are interested in contributing to StreamPipes, let us know! You'll get to know an open-minded and motivated team working together to build the next IIoT analytics toolbox.

Here are some first steps in case you want to contribute:

  • Subscribe to our dev mailing list [email protected]
  • Send an email, tell us about your interests and which parts of Streampipes you'd like to contribute (e.g., core or UI)!
  • Ask for a mentor who helps you understanding the code base and guides you through the first setup steps
  • Find an issue in our Jira which is tagged with a newbie tag
  • Have a look at our developer wiki at https://cwiki.apache.org/confluence/display/STREAMPIPES to learn more about StreamPipes development.

Have fun!

Feedback

We'd love to hear your feedback! Subscribe to [email protected]

License

Apache License 2.0

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].