1594 Open source projects that are alternatives of or similar to Butterfree

Example project implementing best practices for PySpark ETL jobs and applications.

Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Airbyte is an open-source EL(T) platform that helps you replicate your data in your warehouses, lakes and databases.

A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.

A simple Spark-powered ETL framework that just works 🍺

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.

A schema-aware Scala library for data transformation

Real-Time Event Streaming & Change Data Capture

Collection of open-source Spark tools & frameworks that have made the data engineering and data science teams at Swoop highly productive

一款基于kettle的数据处理web调度控制平台，支持文档资源库和数据库资源库，通过web平台控制kettle数据转换，可作为中间件集成到现有系统中

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for writing various different tasks, jobs on GCP and AWS.

Versatile Data Kit (VDK) is an open source framework that enables anybody with basic SQL or Python knowledge to create their own data pipelines.

CubETL - Framework and tool for data ETL (Extract, Transform and Load) in Python (PERSONAL PROJECT / SELDOM MAINTAINED)

Python library for building Pangeo Forge recipes.

ELT Code for your Data Warehouse

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

A python package to create a database on the platform using our moj data warehousing framework

Example of an ETL Pipeline using Airflow

Fancy stream processing made operationally mundane

An orchestration platform for the development, production, and observation of data assets.

Always know what to expect from your data.

Extract, Transform, Load: Any SQL Database in 4 lines of Code.

The easiest way to automate your data

python ETL framework

csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream operations, indices and joins.

ETL jobs for Firefox Telemetry

An Elasticsearch client tailored to data science workflows.

This repository is a getting started guide to Singer.

🚚 Agile Data Preparation Workflows made easy with dask, cudf, dask_cudf and pyspark

Configurable Extract, Transform, and Load

Accelerate data science

Configurable data bridge for permanent ETL jobs

Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes

Generate beautiful documentation for your data pipelines in markdown format

This repo contains commands that data engineers use in day to day work.

Beneath is a serverless real-time data platform ⚡️

A lightweight data processing framework for Apache Spark

A SQL-like language for performing ETL transformations.

Basin is a visual programming editor for building Spark and PySpark pipelines. Easily build, debug, and deploy complex ETL pipelines from your browser

📊 📋 Dashboards using YAML or JSON files

ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

The premier open source Data Quality solution

ETL Framework for .NET / c# (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

A simplified, lightweight ETL Framework based on Apache Spark

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Word2Vec models with Twitter data using Spark. Blog:

📝 A compilation of everything that I learn; Computer Science, Software Development, Engineering, Math, and Coding in General. Read the rendered results here ->

Actively curated list of awesome BI tools. PRs welcome!

(Spatial) data harmonisation with hale studio (formerly HUMBOLDT Alignment Editor)

Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks

📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.

ETL with Python - Taught at DWH course 2017 (TAU)

Powerful RDF Knowledge Graph Generation with [R2]RML Mappings

🐍 Quick reference guide to common patterns & functions in PySpark.

Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift

Stetl, Streaming ETL, is a lightweight geospatial processing and ETL framework written in Python.

Apache Superset is a Data Visualization and Data Exploration Platform