All Categories → Data Processing → streaming-data

Top 52 streaming-data open source projects

Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

Data Accelerator for Apache Spark simplifies onboarding to Streaming of Big Data. It offers a rich, easy to use experience to help with creation, editing and management of Spark jobs on Azure HDInsights or Databricks while enabling the full power of the Spark engine.

✭ 247

react nodejs docker iot kafka azure spark streaming big-data apache-spark kafka-streams spark-streaming streaming-data

Shioaji

Shioaji all new cross platform api for trading ( 跨平台證券交易API )

✭ 203

python docker dockerfile cross-platform trading trade trading-platform streaming-data market-data trading-api

Smart open

Utils for streaming large files (S3, HDFS, gzip, bz2...)

✭ 2,306

python shell streaming s3 file hdfs streaming-data hacktoberfest webhdfs boto gzip-stream bz2

Kafka Streams In Action

Source code for the Kafka Streams in Action Book

✭ 167

java kafka streaming stream-processing streaming-data

Rangeless

c++ LINQ -like library of higher-order functions for data manipulation

✭ 148

cpp cpp11 functional-programming pipeline functional parallel linq range streaming-data

Real Time Sentiment Tracking On Twitter For Brand Improvement And Trend Recognition

A real-time interactive web app based on data pipelines using streaming Twitter data, automated sentiment analysis, and MySQL&PostgreSQL database (Deployed on Heroku)

✭ 127

jupyter-notebook dashboard twitter data-analysis dash stream-processing plotly tweets streaming-data

River

🌊 Online machine learning in Python

✭ 2,980

python cython machine-learning data-science streaming streaming-data online-learning concept-drift incremental-learning online-machine-learning online-statistics

Awesome Bigdata

A curated list of awesome big data frameworks, ressources and other awesomeness.

✭ 10,478

awesome awesome-list database data-science data-visualization data bigdata stream-processing distributed-database data-analytics streaming-data data-stream series-database data-warehouse visualize-data

Gsf

Grid Solutions Framework

✭ 106

time-series stream-processing libraries streaming-data

Toolbox

A Java Toolbox for Scalable Probabilistic Machine Learning

✭ 105

java machine-learning data-science streaming-data bayesian-methods

Pravega

Pravega is 100% open source and community-driven. All components are available under Apache 2 License on GitHub.

✭ 1,653

java streaming streaming-data distributed-storage real-time-data data-ingestion

Tractor

structured concurrent, Python parallelism

✭ 88

python distributed-systems rpc async-await actor-model multiprocessing streaming-data

Pysad

Streaming Anomaly Detection Framework in Python (Outlier Detection for Streaming Data)

✭ 87

python machine-learning real-time unsupervised-learning anomaly-detection intrusion-detection streaming-data

Optbinning

Optimal binning: monotonic binning with constraints. Support batch & stream optimal binning

✭ 79

python optimization stream streaming-data batch-processing

Axway Amplify Streams Js

AMPLIFY Streams Javascript package containing SDK, documentation and sample applications

✭ 79

javascript js nodejs angular vuejs sse streaming-data server-sent-events

Machine

Machine is a workflow/pipeline library for processing data

✭ 78

go golang workflow pipeline stream-processing mit-license workflow-engine streaming-data

Trill

Trill is a single-node query processor for temporal or streaming data.

✭ 1,120

streaming-data

Nsdb

Natural Series Database

✭ 49

scala database metrics analytics time-series nosql akka lucene data-analytics streaming-data

Flexible Clustering

Clustering for arbitrary data and dissimilarity function

✭ 37

python clustering streaming-data

Saber

Window-Based Hybrid CPU/GPU Stream Processing Engine

✭ 35

java gpu streaming stream stream-processing hybrid streaming-data multicore

Go Mesh

Realtime data exchange platform for Smart Cities

✭ 20

go data realtime streaming-data

Streamz

Real-time stream processing for python

✭ 924

python async real-time streaming-data

Go Streams

A lightweight stream processing library for Go

✭ 615

go redis websocket kafka pipeline etl stream-processing streams pipelines kafka-streams streaming-data

Sparta

Real Time Analytics and Data Pipelines based on Spark Streaming

✭ 513

scala kafka spark workflow analytics streaming real-time lambda hdfs spark-streaming streaming-data olap

Onlinestats.jl

Single-pass algorithms for statistics

✭ 507

julia statistics big-data streaming-data

Scikit Multiflow

A machine learning package for streaming data in Python. The other ancestor of River.

✭ 485

python machine-learning stream scikit-learn streaming-data

Awesome Kafka

A list about Apache Kafka

✭ 397

kafka infrastructure apache-spark stream-processing apache-kafka kafka-streams data-processing streaming-data

Swim

Distributed software platform for building stateful, massively real-time streaming applications.

✭ 368

java serverless websockets distributed-systems real-time serverless-framework microservices-architecture actor-model streaming-data decentralized-applications asynchronous-programming streaming-api

Rrcf

🌲 Implementation of the Robust Random Cut Forest algorithm for anomaly detection on streams

✭ 289

python machine-learning tree anomaly-detection random-forest streaming-data

Benthos

Fancy stream processing made operationally mundane

✭ 3,705

go javascript CSS HTML shell Makefile kafka rabbitmq cqrs event-sourcing etl stream-processing amqp message-queue logs data-engineering nats message-bus streaming-data stream-processor data-ops

Cloudflow

Cloudflow enables users to quickly develop, orchestrate, and operate distributed streaming applications on Kubernetes.

✭ 278

scala kubernetes spark akka flink streaming-data

meros

🪢 A fast utility that makes reading multipart responses simple

✭ 109

typescript javascript fetch graphql relay stream multipart streaming-data defer multipart-mixed

spark-streaming-visualize

Simple demonstration of how to build a complex real time machine learning visualization tool.

✭ 16

python scala shell HTML flask apache-spark zeromq streaming-data

pravega-samples

Sample Applications for Pravega.

✭ 43

java python scala shell sample-app streaming-data pravega data-streaming

icicle

Icicle Streaming Query Language

✭ 16

haskell c machine-learning event-sourcing feature-engineering streaming-data

transit

Massively real-time city transit streaming application

nestjs-file-streaming

NestJS File Streaming With MongoDB

✭ 28

typescript javascript Dockerfile nodejs docker npm mongoose gridfs file-upload efficient streaming-data nestjs gridfs-bucket

richflow

A Node.js and JavaScript synchronous data pipeline processing, data sharing and stream processing library. Actionable & Transformable Pipeline data processing.

✭ 17

javascript nodejs data-stream data-transformation pipeline-framework data-flow synchronous data-pipeline streaming-data data-processor pipe-data

OnlineStatsBase.jl

Base types for OnlineStats.

✭ 26

julia statistics big-data streaming-data onlinestats online-algorithm

Twitter-Stream-API-Dataset

Twitter Dynamic Dataset Api. Create any dataset YOU want.

✭ 20

python flask mongodb twitter-api rest-api dataset streaming-data tweepy-api twitter-dataset twitter-dynamic-dataset

godsend

A simple and eloquent workflow for streaming messages to micro-services.