blockchain-etl / Ethereum Etl Airflow
Licence: mit
Airflow DAGs for exporting, loading, and parsing the Ethereum blockchain data. What datasets do you want to be added to Ethereum ETL? Vote here: https://blockchain-etl.convas.io.
Stars: ✭ 89
Programming Languages
python
139335 projects - #7 most used programming language
Projects that are alternatives of or similar to Ethereum Etl Airflow
Ethereum Etl
Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
Stars: ✭ 956 (+974.16%)
Mutual labels: ethereum, erc20, gcp, bigquery
Ethereum Erc20 Wallet
ERC20 Wallet for Metamask or Mist
Stars: ✭ 32 (-64.04%)
Mutual labels: ethereum, erc20
Hdwallet
Simple Swift library for creating HD cryptocurrencies wallets and working with crypto Coins/ERC20 tokens.
Stars: ✭ 80 (-10.11%)
Mutual labels: ethereum, erc20
Maker Otc
Simple on-chain market for ERC20 tokens
Stars: ✭ 37 (-58.43%)
Mutual labels: ethereum, erc20
Upchain Wallet
以太坊钱包(支持DApp 浏览器)- A Powerful Ethereum Android Wallet & DApp Browser
Stars: ✭ 359 (+303.37%)
Mutual labels: ethereum, erc20
Eth Hodler
A simple DApp & ERC20 token written in Solidity running on the Ethereum blockchain www.hdao.org
Stars: ✭ 31 (-65.17%)
Mutual labels: ethereum, erc20
Etherscan Ml
Python Data Science and Machine Learning Library for the Ethereum and ERC-20 Blockchain
Stars: ✭ 55 (-38.2%)
Mutual labels: ethereum, erc20
Datashare Toolkit
DIY commercial datasets on Google Cloud Platform
Stars: ✭ 41 (-53.93%)
Mutual labels: gcp, bigquery
Protocol
Hydro Protocol decentralized exchange smart contracts
Stars: ✭ 83 (-6.74%)
Mutual labels: ethereum, erc20
Uniflash
A simple and decentralized protocol for flash loan. https://t.me/Uniflash
Stars: ✭ 64 (-28.09%)
Mutual labels: ethereum, erc20
Uniswap V2 Core
🎛 Core smart contracts of Uniswap V2
Stars: ✭ 889 (+898.88%)
Mutual labels: ethereum, erc20
Uniswap V2 Periphery
🎚 Peripheral smart contracts for interacting with Uniswap V2
Stars: ✭ 267 (+200%)
Mutual labels: ethereum, erc20
Tokens
Upload yours token, coin and dApp image to get displayed in the Trust Wallet
Stars: ✭ 260 (+192.13%)
Mutual labels: ethereum, erc20
gcp-ml
Google Cloud Platform Machine Learning Samples
Stars: ✭ 31 (-65.17%)
Mutual labels: bigquery, gcp
snowplow-bigquery-loader
Loads Snowplow enriched events into Google BigQuery
Stars: ✭ 15 (-83.15%)
Mutual labels: bigquery, gcp
Miner Monitor
Miner, balance, wallet and pool monitoring software
Stars: ✭ 38 (-57.3%)
Mutual labels: ethereum, erc20
Revoke.cash
❌ Revoke or change your ERC20 token allowances
Stars: ✭ 60 (-32.58%)
Mutual labels: ethereum, erc20
Ethereum ETL Airflow
Read this article: https://cloud.google.com/blog/products/data-analytics/ethereum-bigquery-how-we-built-dataset
Setting up Airflow DAGs using Google Cloud Composer
Create BigQuery Datasets
- Sign in to BigQuery https://bigquery.cloud.google.com/
- Create new datasets called
crypto_ethereum
,crypto_ethereum_raw
,crypto_ethereum_temp
Create Google Cloud Storage bucket
- Create a new Google Storage bucket to store exported files https://console.cloud.google.com/storage/browser
Create Google Cloud Composer environment
Create a new Cloud Composer environment:
export ENVIRONMENT_NAME=ethereum-etl-0
gcloud composer environments create $ENVIRONMENT_NAME --location=us-central1 --zone=us-central1-a \
--disk-size=100GB --machine-type=n1-standard-4 --node-count=3 --python-version=3 --image-version=composer-1.8.3-airflow-1.10.3 \
--network=default --subnetwork=default
gcloud composer environments update $ENVIRONMENT_NAME --location=us-central1 --update-pypi-package=ethereum-etl==1.4.1
Create variables in Airflow (Admin > Variables in the UI):
Variable | Description |
---|---|
ethereum_output_bucket | GCS bucket to store exported files |
ethereum_provider_uris | Comma separated URIs of Ethereum nodes |
ethereum_destination_dataset_project_id | Project ID of BigQuery datasets |
notification_emails | email for notifications |
Check other variables in dags/ethereumetl_airflow/variables.py
.
Upload DAGs
> ./upload_dags.sh <airflow_bucket>
Running Tests
pip install -r requirements.txt
export PYTHONPATH='dags'
pytest -vv -s
Creating Table Definition Files for Parsing Events and Function Calls
Read this article: https://medium.com/@medvedev1088/query-ens-and-0x-events-with-sql-in-google-bigquery-4d197206e644
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].