276 Open source projects that are alternatives of or similar to etl_manager

Example of an ETL Pipeline using Airflow

A tool for building feature stores.

Versatile Data Kit (VDK) is an open source framework that enables anybody with basic SQL or Python knowledge to create their own data pipelines.

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

Airbyte is an open-source EL(T) platform that helps you replicate your data in your warehouses, lakes and databases.

A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.

A schema-aware Scala library for data transformation

Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes

Beneath is a serverless real-time data platform ⚡️

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Powerful RDF Knowledge Graph Generation with [R2]RML Mappings

ELT Code for your Data Warehouse

Example project implementing best practices for PySpark ETL jobs and applications.

[READ-ONLY] PHP - ETL (Extract Transform Load) data processing library

A simple Spark-powered ETL framework that just works 🍺

Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift

Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).

A client for connecting and running DDLs on hive metastore.

Fancy stream processing made operationally mundane

Blog post on ETL pipelines with Airflow

Enterprise-grade, production-hardened, serverless data lake on AWS

An Elasticsearch client tailored to data science workflows.

Python library for building Pangeo Forge recipes.

Astro allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

Addax is an open source universal ETL tool that supports most of those RDBMS and NoSQLs on the planet, helping you transfer data from any one place to another.

This repo contains commands that data engineers use in day to day work.

Enterprise Grade Single-Step Streaming Data Infrastructure Setup. (Under Development)

Virtual Machine for Desktop Search with Open Semantic Search

Generate beautiful documentation for your data pipelines in markdown format

Odoo Integration Addons

GAMECHANGER aspires to be the Department’s trusted solution for evidence-based, data-driven decision-making across the universe of DoD requirements

为DataX(https://github.com/alibaba/DataX) 提供远程多语言调用（ThriftServer，HttpServer） 分布式运行（DataX on YARN） 功能

A lightweight data processing framework for Apache Spark

Basin is a visual programming editor for building Spark and PySpark pipelines. Easily build, debug, and deploy complex ETL pipelines from your browser

The practical use-cases of how to make your Machine Learning Pipelines robust and reliable using Apache Airflow.

OpenRefine is a free, open source power tool for working with messy data and improving it. This repository contains Dockerbuild files for automated builds.

A Kafka Connect source connector that generates data for tests

Rivery CLI

Python3 lib and cli for operating a Cardano Passive Node and using the API's. (PRE-ALPHA)

Personal Data Engineering Projects

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

dbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL databases.

M-Lab ingestion pipeline

consume maxwell generated message from kafka,export it to another mysql.

Real-Time Event Streaming & Change Data Capture

Multi-core processing of 'Gameday' data from Major League Baseball Advanced Media. Additional tools to parallelize large data sets and write them to a database.

Ruby scripting for pentaho-kettle

A persistence framework for game developers

Monitoring tool to measure flow throughput of data sources and processing components that are part of Data Ingestion and ETL pipelines.

Automate building ML classification pipelines in .NET

C++ implementation of CAN ISO 15765-2 also known as CAN ISO transport protocol. CPP CAN isotp.

ETL for the SPDR ETF holdings XLS documents

an open-source data management platform for knowledge workers (https://github.com/dswarm/dswarm-documentation/wiki)

The OpenRefine Python Client from Paul Makepeace provides a library for communicating with an OpenRefine server. This fork extends the command line interface (CLI) and is distributed as a convenient one-file-executable (Windows, Linux, Mac). It is also available via Docker Hub, PyPI and Binder.

网站点击流离线日志分析

ETL for OIC Options Chains

A comprehensive list of 180+ YouTube Channels for Data Science, Data Engineering, Machine Learning, Deep learning, Computer Science, programming, software engineering, etc.

In-memory Java DataFrame library

The Taxonomy for ETL Automation Metadata (TEAM) is a metadata management tool for data warehouse automation. It is part of the ecosystem for data warehouse automation, alongside the Virtual Data Warehouse pattern manager and the generic schema for Data Warehouse Automation.

🇨🇳 MyDataHarbor是一个致力于解决任意数据源到任意数据源的分布式、高扩展性、高性能、事务级的数据同步中间件。帮助用户可靠、快速、稳定的对海量数据进行准实时增量同步或者定时全量同步，主要定位是为实时交易系统服务，亦可用于大数据的数据同步（ETL领域）。