339 Open source projects that are alternatives of or similar to Luigi

Saisoku is a Python module that helps you build complex pipelines of batch file/directory transfer/sync jobs.

Mirror of Apache Hadoop common

An Airflow docker image preconfigured to work well with Spark and Hadoop/EMR

Staffjoy V2 - all microservices in a monorepo

A Web Audio framework for making interactive music in the browser.

Big Data Ecosystem Docker

Hadoop Crypto Ledger - Analyzing CryptoLedgers, such as Bitcoin Blockchain, on Big Data platforms, such as Hadoop/Spark/Flink/Hive

Hive JDBC "uber" or "standalone" jar based on the latest Apache Hive version

Distributed Workflow Engine for Microservices Orchestration

DataX is an open source universal ETL tool that support Cassandra, ClickHouse, DBF, Hive, InfluxDB, Kudu, MySQL, Oracle, Presto(Trino), PostgreSQL, SQL Server

Support TensorFlow on YARN

AI on Hadoop

Suite of tools for deploying and training deep learning models using the JVM. Highlights include model import for keras, tensorflow, and onnx/pytorch, a modular and tiny c++ library for running math code and a java based math library on top of the core c++ library. Also includes samediff: a pytorch/tensorflow like library for running deep learni…

Mirror of Apache Calcite - Avatica

Quarzite is a thin idiomatic Clojure layer on top the Quartz Scheduler

Griffon Data Science Virtual Machine

The official home of the Presto distributed SQL query engine for big data

schedule alarm and local notification in react-native

An end-to-end machine learning and data mining framework on Hadoop

Apache Drill is a distributed MPP query layer for self describing data

Mirror of Apache Hadoop HDFS

🎒 Simple schedule optimization library for Python

AI constraint solver in Java to optimize the vehicle routing problem, employee rostering, task assignment, maintenance scheduling, conference scheduling and other planning problems.

Apache Parquet implementation in Rust

extremely distributed machine learning

Example MapReduce jobs in Java, Hive, Pig, and Hadoop Streaming that work on Avro data.

🏭 Open Source Manufacturing Execution System for JobShop type manufacturer.

Resource scheduling and cluster management for AI

HBase客户端数据管理软件

实践源码库：https://github.com/jast90/bigdata 。 微信搜索Jast关注公众号，获取最新技术分享😯。

Extended datasource support for Spark/Hadoop on Aliyun E-MapReduce.

Spark、Flink等离线任务的调度以及实时任务的监控

A large-scale entity and relation database supporting aggregation of properties

Facebook's Hive UDFs

Ephemeral Hadoop clusters using Google Compute Platform

📚 Notebook storage and publishing workflows for the masses

Kubernetes Scheduler for Deep Learning

Apache Nutch is an extensible and scalable web crawler

Read and write Parquet in Scala. Use Scala classes as schema. No need to start a cluster.

Cylc: a workflow engine for cycling systems. Repository master branch: core meta-scheduler component of cylc-8 (in development); Repository 7.8.x branch: full cylc-7 system.

A tool for scale and performance testing of HDFS with a specific focus on the NameNode.

RDMA accelerated, high-performance, scalable and efficient ShuffleManager plugin for Apache Spark

HDFS Shell is a HDFS manipulation tool to work with functions integrated in Hadoop DFS

基于Spark的电影推荐系统，包含爬虫项目、web网站、后台管理系统以及spark推荐系统

A pandas-like deferred expression system, with first-class SQL support

Quickly and easily install, uninstall, and set up automatic updates for any of Steven Black's unified hosts files.

Web application for solving Employee Rostering using OptaPlanner

Fundamentals of Spark with Python (using PySpark), code examples

Asakusa Framework

Java开发者或者大数据开发者面试知识点整理

Go package to read and write parquet files. parquet is a file format to store nested data structures in a flat columnar data format. It can be used in the Hadoop ecosystem and with tools such as Presto and AWS Athena.

Apache Hadoop

liteflow是一个基于任务版本来实现的分布式任务流调度系统

A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL

🐘 Source code for assignments of Udacity course "Introduction to Hadoop and MapReduce"

Explore the energy-efficient dataflow scheduling for neural networks.

Libraries and tools for interoperability between Hadoop-related open-source software and Google Cloud Platform.

Apache Calcite

推荐项目（实时推荐和离线推荐）

The vip.com's distributed job scheduling platform.