281 Open source projects that are alternatives of or similar to arthur-redshift-etl

Airbyte is an open-source EL(T) platform that helps you replicate your data in your warehouses, lakes and databases.

Versatile Data Kit (VDK) is an open source framework that enables anybody with basic SQL or Python knowledge to create their own data pipelines.

Astro allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

Powerful RDF Knowledge Graph Generation with [R2]RML Mappings

[READ-ONLY] PHP - ETL (Extract Transform Load) data processing library

A schema-aware Scala library for data transformation

A simple Spark-powered ETL framework that just works 🍺

Beneath is a serverless real-time data platform ⚡️

Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).

Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift

A tool for building feature stores.

Rivery CLI

A client for connecting and running DDLs on hive metastore.

Blog post on ETL pipelines with Airflow

An Elasticsearch client tailored to data science workflows.

Fancy stream processing made operationally mundane

dbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL databases.

Python based Wikidata framework for easy dataframe extraction

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

Example of an ETL Pipeline using Airflow

A python package to create a database on the platform using our moj data warehousing framework

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Example project implementing best practices for PySpark ETL jobs and applications.

A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.

Enterprise-grade, production-hardened, serverless data lake on AWS

Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes

Python library for building Pangeo Forge recipes.

Ruby scripting for pentaho-kettle

Data transformation framework for LinkML data models

A persistence framework for game developers

Multi-core processing of 'Gameday' data from Major League Baseball Advanced Media. Additional tools to parallelize large data sets and write them to a database.

ElasticSearch to PostgreSQL loader

an open-source data management platform for knowledge workers (https://github.com/dswarm/dswarm-documentation/wiki)

网站点击流离线日志分析

Enterprise Grade Single-Step Streaming Data Infrastructure Setup. (Under Development)

ETL for OIC Options Chains

GAMECHANGER aspires to be the Department’s trusted solution for evidence-based, data-driven decision-making across the universe of DoD requirements

ETL for the SPDR ETF holdings XLS documents

In-memory Java DataFrame library

🇨🇳 MyDataHarbor是一个致力于解决任意数据源到任意数据源的分布式、高扩展性、高性能、事务级的数据同步中间件。帮助用户可靠、快速、稳定的对海量数据进行准实时增量同步或者定时全量同步，主要定位是为实时交易系统服务，亦可用于大数据的数据同步（ETL领域）。

Generate beautiful documentation for your data pipelines in markdown format

Code relating to scraping public police data.

Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.

The OpenRefine Python Client from Paul Makepeace provides a library for communicating with an OpenRefine server. This fork extends the command line interface (CLI) and is distributed as a convenient one-file-executable (Windows, Linux, Mac). It is also available via Docker Hub, PyPI and Binder.

This repo contains commands that data engineers use in day to day work.

Configurable data bridge for permanent ETL jobs

bqloader is a simple ETL framework to load data from Cloud Storage into BigQuery.

Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data sc…

A COBOL parser and Mainframe/EBCDIC data source for Apache Spark

CubETL - Framework and tool for data ETL (Extract, Transform and Load) in Python (PERSONAL PROJECT / SELDOM MAINTAINED)

Virtual Machine for Desktop Search with Open Semantic Search

OpenRefine is a free, open source power tool for working with messy data and improving it. This repository contains Dockerbuild files for automated builds.

A comprehensive list of 180+ YouTube Channels for Data Science, Data Engineering, Machine Learning, Deep learning, Computer Science, programming, software engineering, etc.

为DataX(https://github.com/alibaba/DataX) 提供远程多语言调用（ThriftServer，HttpServer） 分布式运行（DataX on YARN） 功能

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.

A Pachyderm deep learning tutorial for conference workshops

A lightweight data processing framework for Apache Spark

A data transformation package for deep learning with Autonomio, Keras and TensorFlow.

The Move to Islandora Kit is an extensible PHP command-line tool for converting source content and metadata into packages suitable for importing into Islandora (or other digital repository and preservations systems).

The Taxonomy for ETL Automation Metadata (TEAM) is a metadata management tool for data warehouse automation. It is part of the ecosystem for data warehouse automation, alongside the Virtual Data Warehouse pattern manager and the generic schema for Data Warehouse Automation.