666 Open source projects that are alternatives of or similar to Airflow Pipeline

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.

A Scala kernel for Jupyter

Spring-Shiro-Spark是Spring-Boot Hibernate Spark Spark-SQL Shiro iView VueJs... ...的集成尝试

Scalable stream processing platform for advanced realtime analytics on top of Kafka and Spark. LogIsland also supports MQTT and Kafka Streams (Flink being in the roadmap). The platform does complex event processing and is suitable for time series analysis. A large set of valuable ready to use processors, data sources and sinks are available.

Schema registry for CSV, TSV, JSON, AVRO and Parquet schema. Supports schema inference and GraphQL API.

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

Go package to read and write parquet files. parquet is a file format to store nested data structures in a flat columnar data format. It can be used in the Hadoop ecosystem and with tools such as Presto and AWS Athena.

Relation Extraction using Deep learning(CNN)

🌟 ✨ Analyze and visualize Twitter Sentiment on a world map using Spark MLlib

Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks

Boiler plate framework to use Spark and ZIO together.

基于wifi抓取信息的大数据查询分析系统

spark summit 2017 SanFrancisco

🔧 Use dplyr to analyze Big Data 🐘

Data science and Big Data with Python

Python client for Hadoop® YARN API

Spark on Kubernetes infrastructure Helm charts repo

pyspark🍒🥭 is delicious，just eat it!😋😋

An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.

Bitnami Docker Image for Apache Airflow

Distributed Deep learning with Keras & Spark

Mirror of Apache Hadoop MapReduce

Run spark calculations from Ammonite

Griffon Data Science Virtual Machine

Read and write Parquet in Scala. Use Scala classes as schema. No need to start a cluster.

Apache Spark and Apache Kafka integration example

Applying Lambda Architecture with Spark, Kafka, and Cassandra.

Models and Pipelines for the Spark NLP library

Spark 2.0 Python Machine Learning examples

Fast iterative local development and testing of Apache Airflow workflows

Example end to end data engineering project.

Google Authenticator support for Laravel Spark

Spark Streaming监控平台，支持任务部署与告警、自启动

Example MapReduce jobs in Java, Hive, Pig, and Hadoop Streaming that work on Avro data.

A framework for writing Spark 2.x applications in a pretty way

Webex Bot SDK for Node.js (deprecated in favor of https://github.com/webex/webex-bot-node-framework)

🐘 Source code for assignments of Udacity course "Introduction to Hadoop and MapReduce"

Examples for Deep Learning/Feature Store/Spark/Flink/Hive/Kafka jobs and Jupyter notebooks on Hops

BigQuery data source for Apache Spark: Read data from BigQuery into DataFrames, write DataFrames into BigQuery tables.

Kinesis Connector for Structured Streaming

Airflow training for the crunch conf

Custom state store providers for Apache Spark

java 进阶之路：面试高频算法、akka、多线程、NIO、Netty、SpringBoot、Spark&&Flink 等

Multiple node cluster on Docker for self development.

Spark job for dependency links

Elassandra = Elasticsearch + Apache Cassandra

Two-day workshop that covers how to use R to interact databases and Spark

Mirror of Linkedin's Camus

MLeap: Deploy ML Pipelines to Production

Spark SQL index for Parquet tables

Visualize data using relative ordering

Hybrid model of Gradient Boosting Trees and Logistic Regression (GBDT+LR) on Spark

🌿 基于springboot的快速学习示例,整合自己遇到的开源框架,如：rabbitmq(延迟队列)、Kafka、jpa、redies、oauth2、swagger、jsp、docker、spring-batch、异常处理、日志输出、多模块开发、多环境打包、缓存cache、爬虫、jwt、GraphQL、dubbo、zookeeper和Async等等📌

A robust, and flexible open source User & Entity Behavior Analytics (UEBA) framework used for Security Analytics. Developed with luv by Data Scientists & Security Analysts from the Cyber Security Industry. [PRE-ALPHA]

There are pieces of scala code that explain Scala syntax and related things - like what you can do with all this

HDFS Shell is a HDFS manipulation tool to work with functions integrated in Hadoop DFS

Apache Airflow in Docker Compose (for both versions 1.10.* and 2.*)

Road to Machine Learning

Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks

A simple Spark-powered ETL framework that just works 🍺