774 Open source projects that are alternatives of or similar to polygon-etl

Astro allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for writing various different tasks, jobs on GCP and AWS.

ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ

Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes

Example of an ETL Pipeline using Airflow

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

Blog post on ETL pipelines with Airflow

A Python framework for data processing on GCP.

A tool for building feature stores.

Enterprise-grade, production-hardened, serverless data lake on AWS

Powerful RDF Knowledge Graph Generation with [R2]RML Mappings

A python package to create a database on the platform using our moj data warehousing framework

Fancy stream processing made operationally mundane

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Example project implementing best practices for PySpark ETL jobs and applications.

A Data Engineering & Machine Learning Knowledge Hub

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

Viewflow is an Airflow-based framework that allows data scientists to create data models without writing Airflow code.

Hive Storage Handler for interoperability between BigQuery and Apache Hive

ETLy is an add-on dashboard service on top of Apache Airflow.

Example end to end data engineering project.

Starlake is a Spark Based On Premise and Cloud ELT/ETL Framework for Batch & Stream Processing

An upgraded and improved version of the Iris automatic GCP-labeling project

☁️ nodejs kafka connect connector for Google BigQuery

Use Airflow to move data from multiple MySQL databases to BigQuery

Beneath is a serverless real-time data platform ⚡️

ELT Code for your Data Warehouse

Dataform is a framework for managing SQL based data operations in BigQuery, Snowflake, and Redshift

Python library for building Pangeo Forge recipes.

A simple Spark-powered ETL framework that just works 🍺

Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).

Airbyte is an open-source EL(T) platform that helps you replicate your data in your warehouses, lakes and databases.

A schema-aware Scala library for data transformation

Campaign Manager 360 and Display & Video 360 Reports to BigQuery connector

Google Cloud Platform Machine Learning Samples

dbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL databases.

Personal Data Engineering Projects

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

JobAnalytics system consumes data from multiple sources and provides valuable information to both job hunters and recruiters.

A collection of online resources to help you on your Tech journey.

Any Airflow project day 1, you can spin up a local desktop Kubernetes Airflow environment AND one in Google Cloud Composer with tested data pipelines(DAGs) 🖥 >> [ 🚀, 🚢 ]

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

Versatile Data Kit (VDK) is an open source framework that enables anybody with basic SQL or Python knowledge to create their own data pipelines.

Example DAGs using hooks and operators from Airflow Plugins

Metric collection, data testing and monitoring for SQL accessible data

Airflow Deployment on AWS ECS Fargate Using Cloudformation

bqloader is a simple ETL framework to load data from Cloud Storage into BigQuery.

Open Source Feature Flagging and A/B Testing Platform

Run Airflow in AWS ECS(Elastic Container Service) using Fargate tasks

DIY commercial datasets on Google Cloud Platform

Airflow DAGs for exporting, loading, and parsing the Ethereum blockchain data. What datasets do you want to be added to Ethereum ETL? Vote here: https://blockchain-etl.convas.io.

Useful SQL queries for Blockchain ETL datasets in BigQuery.

An example mini data warehouse for python project stats, template for new projects

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.

An Elasticsearch client tailored to data science workflows.

A scalable general purpose micro-framework for defining dataflows. You can use it to create dataframes, numpy matrices, python objects, ML models, etc.

Udacity Data Engineering Nano Degree (DEND)

Loads Snowplow enriched events into Google BigQuery

A client for connecting and running DDLs on hive metastore.