GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → blockchain-etl → polygon-etl

blockchain-etl / polygon-etl

Licence: MIT license
ETL (extract, transform and load) tools for ingesting Polygon blockchain data to Google BigQuery and Pub/Sub

Programming Languages

python
139335 projects - #7 most used programming language
java
68154 projects - #9 most used programming language

Labels

bigqueryairflowetlgcppolygoncryptocurrencydata-engineeringmaticnetworkmatic-network

Projects that are alternatives of or similar to polygon-etl

blockchain-etl-streaming
Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes
Stars: ✭ 57 (+7.55%)
Mutual labels:  etl, gcp, data-engineering
Ethereum Etl
Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
Stars: ✭ 956 (+1703.77%)
Mutual labels:  bigquery, etl, gcp
astro
Astro allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
Stars: ✭ 79 (+49.06%)
Mutual labels:  bigquery, airflow, etl
Bitcoin Etl
ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ
Stars: ✭ 174 (+228.3%)
Mutual labels:  bigquery, etl, gcp
AirflowDataPipeline
Example of an ETL Pipeline using Airflow
Stars: ✭ 24 (-54.72%)
Mutual labels:  airflow, etl, data-engineering
etlflow
EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for writing various different tasks, jobs on GCP and AWS.
Stars: ✭ 38 (-28.3%)
Mutual labels:  bigquery, etl, gcp
AirflowETL
Blog post on ETL pipelines with Airflow
Stars: ✭ 20 (-62.26%)
Mutual labels:  airflow, etl, data-engineering
bigflow
A Python framework for data processing on GCP.
Stars: ✭ 96 (+81.13%)
Mutual labels:  bigquery, gcp
snowplow-bigquery-loader
Loads Snowplow enriched events into Google BigQuery
Stars: ✭ 15 (-71.7%)
Mutual labels:  bigquery, gcp
Datashare Toolkit
DIY commercial datasets on Google Cloud Platform
Stars: ✭ 41 (-22.64%)
Mutual labels:  bigquery, gcp
Mara Example Project 2
An example mini data warehouse for python project stats, template for new projects
Stars: ✭ 154 (+190.57%)
Mutual labels:  bigquery, etl
dbd
dbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL databases.
Stars: ✭ 30 (-43.4%)
Mutual labels:  bigquery, etl
growthbook
Open Source Feature Flagging and A/B Testing Platform
Stars: ✭ 2,342 (+4318.87%)
Mutual labels:  bigquery, data-engineering
airflow-tutorial
Use Airflow to move data from multiple MySQL databases to BigQuery
Stars: ✭ 96 (+81.13%)
Mutual labels:  bigquery, airflow
Ethereum Etl Airflow
Airflow DAGs for exporting, loading, and parsing the Ethereum blockchain data. What datasets do you want to be added to Ethereum ETL? Vote here: https://blockchain-etl.convas.io.
Stars: ✭ 89 (+67.92%)
Mutual labels:  bigquery, gcp
airflow-dbt-python
A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.
Stars: ✭ 111 (+109.43%)
Mutual labels:  airflow, data-engineering
gcp-ml
Google Cloud Platform Machine Learning Samples
Stars: ✭ 31 (-41.51%)
Mutual labels:  bigquery, gcp
etl
[READ-ONLY] PHP - ETL (Extract Transform Load) data processing library
Stars: ✭ 279 (+426.42%)
Mutual labels:  etl, data-engineering
hive-metastore-client
A client for connecting and running DDLs on hive metastore.
Stars: ✭ 37 (-30.19%)
Mutual labels:  etl, data-engineering
Everything-Tech
A collection of online resources to help you on your Tech journey.
Stars: ✭ 396 (+647.17%)
Mutual labels:  gcp, data-engineering
View All Similar Projects ➔

Polygon-etl

Overview

Polygon ETL allows you to setup an ETL pipeline in Google Cloud Platform for ingesting Polygon blockchain data into BigQuery and Pub/Sub. It comes with CLI tools for exporting Polygon data into convenient formats like CSVs and relational databases.

Architecture

polygon_etl_architecture.svg

  1. The nodes are run in a Kubernetes cluster.

  2. Airflow DAGs export and load Polygon data to BigQuery daily. Refer to Polygon ETL Airflow for deployment instructions.

  3. Polygon data is polled periodically from the nodes and pushed to Google Pub/Sub. Refer to Polygon ETL Streaming for deployment instructions.

  4. Polygon data is pulled from Pub/Sub, transformed and streamed to BigQuery. Refer to Polygon ETL Dataflow for deployment instructions.

Setting Up

  1. Follow the instructions in Polygon ETL Airflow to deploy a Cloud Composer cluster for exporting and loading historical Polygon data. It may take several days for the export DAG to catch up. During this time "load" and "verify_streaming" DAGs will fail.

  2. Follow the instructions in Polygon ETL Streaming to deploy the Streamer component. For the value in last_synced_block.txt specify the last block number of the previous day. You can query it in BigQuery: SELECT number FROM crypto_polygon.blocks ORDER BY number DESC LIMIT 1.

  3. Follow the instructions in Polygon ETL Dataflow to deploy the Dataflow component. Monitor "verify_streaming" DAG in Airflow console, once the Dataflow job catches up the latest block, the DAG will succeed.

Code quality

Over time, we intend to format python files in this repo using isort and black. At the moment, we are only formatting any changed or added files

We have not implemented any sort of automation (e.g. pre-commit), but a requirements_dev.txt is provided for contributors to use.

Testing

Various tests are implemented (airflow/tests, cli/tests and ./tests). As part of an effort towards consistency, they all source the same requirements_test.txt.

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].