All Projects → blockchain-etl → Bitcoin Etl

blockchain-etl / Bitcoin Etl

Licence: mit
ETL scripts for Bitcoin, Litecoin, Dash, Zcash, Doge, Bitcoin Cash. Available in Google BigQuery https://goo.gl/oY5BCQ

Programming Languages

python
139335 projects - #7 most used programming language

Projects that are alternatives of or similar to Bitcoin Etl

Ethereum Etl
Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ
Stars: ✭ 956 (+449.43%)
Mutual labels:  aws, gcp, sql, etl, export, bigquery
Bitcoinlib
Bitcoin Core RPC compatible, battle-tested .NET library and RPC wrapper for Bitcoin and Altcoins
Stars: ✭ 350 (+101.15%)
Mutual labels:  bitcoin, cryptocurrency, litecoin, dash
Antminer Monitor
Cryptocurrency ASIC mining hardware monitor using a simple web interface
Stars: ✭ 176 (+1.15%)
Mutual labels:  bitcoin, cryptocurrency, litecoin, dash
Cryptocurrency Dashboard
Crypto Currency Dashboard Using Twitter 🐦 And Coinmarketcap 🚀 API
Stars: ✭ 54 (-68.97%)
Mutual labels:  bitcoin, cryptocurrency, litecoin, dash
Moneda Cli
Command line to track cryptocurrency prices
Stars: ✭ 114 (-34.48%)
Mutual labels:  bitcoin, cryptocurrency, litecoin, dash
Crypto Whale Watching App
Python Dash app that tracks whale activity in cryptocurrency markets.
Stars: ✭ 389 (+123.56%)
Mutual labels:  bitcoin, cryptocurrency, litecoin, dash
Cryptocurrency Cli
💰 Cryptocurrency Portfolio On The Command Line 💰
Stars: ✭ 99 (-43.1%)
Mutual labels:  bitcoin, cryptocurrency, litecoin, dash
Coinpusher
📈 real-time cryptocurrency chart prediction based on neuronal-networks
Stars: ✭ 141 (-18.97%)
Mutual labels:  bitcoin, cryptocurrency, litecoin
Ether sql
A python library to push ethereum blockchain data into an sql database.
Stars: ✭ 41 (-76.44%)
Mutual labels:  sql, etl, export
Straks
A new decentralised, open source, community driven digital currency, focusing on e-commerce utility
Stars: ✭ 53 (-69.54%)
Mutual labels:  bitcoin, cryptocurrency, dash
Cryptolights
Live visualisation of blockchain transactions for popular cryptocurrencies
Stars: ✭ 54 (-68.97%)
Mutual labels:  bitcoin, cryptocurrency, litecoin
Locopy
locopy: Loading/Unloading to Redshift and Snowflake using Python.
Stars: ✭ 73 (-58.05%)
Mutual labels:  aws, sql, etl
Cloudquery
cloudquery transforms your cloud infrastructure into SQL or Graph database for easy monitoring, governance and security.
Stars: ✭ 1,300 (+647.13%)
Mutual labels:  aws, gcp, sql
Cbpro Trader
Automated cryptocurrency trading on Coinbase Pro (formerly gdax-trader)
Stars: ✭ 171 (-1.72%)
Mutual labels:  bitcoin, cryptocurrency, litecoin
Shapeshift
ShapeShift API for Go Language - convert cryptocurrencies with ease using ShapeShift and golang
Stars: ✭ 35 (-79.89%)
Mutual labels:  bitcoin, cryptocurrency, dash
Cryptex
Gemini, GDAX, Bitfinex, Poloniex, Binance, Kraken, Cryptopia, Koinex, BitGrail and CoinMarketCap cryptocurrency exchange API clients in Swift / iOS SDK. Check prices and account balances using Sample iOS app.
Stars: ✭ 51 (-70.69%)
Mutual labels:  bitcoin, cryptocurrency, litecoin
Jekyll Paspagon
Sell your Jekyll blog posts in various formats for cryptocurrencies.
Stars: ✭ 8 (-95.4%)
Mutual labels:  bitcoin, cryptocurrency, aws
Bot18
Bot18 is a high-frequency cryptocurrency trading bot developed by Zenbot creator @carlos8f
Stars: ✭ 157 (-9.77%)
Mutual labels:  bitcoin, cryptocurrency, litecoin
Stocklook
crypto currency library for trading & market making bots, account management, and data analysis
Stars: ✭ 119 (-31.61%)
Mutual labels:  bitcoin, cryptocurrency, litecoin
Cryptocurrency Arbitrage
A cryptocurrency arbitrage opportunity calculator. Over 800 currencies and 50 markets.
Stars: ✭ 836 (+380.46%)
Mutual labels:  bitcoin, cryptocurrency, litecoin

Bitcoin ETL

Join the chat at https://gitter.im/ethereum-eth Build Status Join Telegram Group

Install Bitcoin ETL:

pip install bitcoin-etl

Export blocks and transactions (Schema, Reference):

> bitcoinetl export_blocks_and_transactions --start-block 0 --end-block 500000 \
--provider-uri http://user:[email protected]:8332 --chain bitcoin \
 --blocks-output blocks.json --transactions-output transactions.json

Supported chains:

  • bitcoin
  • bitcoin_cash
  • bitcoin_gold
  • dogecoin
  • litecoin
  • dash
  • zcash

Stream blockchain data continually to console (Reference):

> pip install bitcoin-etl[streaming]
> bitcoinetl stream -p http://user:[email protected]:8332 --start-block 500000

Stream blockchain data continually to Google Pub/Sub (Reference):

> export GOOGLE_APPLICATION_CREDENTIALS=/path_to_credentials_file.json
> bitcoinetl stream -p http://user:[email protected]:8332 --start-block 500000 --output projects/your-project/topics/crypto_bitcoin

For the latest version, check out the repo and call

> pip install -e .[streaming]
> python bitcoinetl.py

Table of Contents

Schema

blocks.json

Field Type
hash hex_string
size bigint
stripped_size bigint
weight bigint
number bigint
version bigint
merkle_root hex_string
timestamp bigint
nonce hex_string
bits hex_string
coinbase_param hex_string
transaction_count bigint

transactions.json

Field Type
hash hex_string
size bigint
virtual_size bigint
version bigint
lock_time bigint
block_number bigint
block_hash hex_string
block_timestamp bigint
is_coinbase boolean
index bigint
inputs []transaction_input
outputs []transaction_output
input_count bigint
output_count bigint
input_value bigint
output_value bigint
fee bigint

transaction_input

Field Type
index bigint
spent_transaction_hash hex_string
spent_output_index bigint
script_asm string
script_hex hex_string
sequence bigint
required_signatures bigint
type string
addresses []string
value bigint

transaction_output

Field Type
index bigint
script_asm string
script_hex hex_string
required_signatures bigint
type string
addresses []string
value bigint

You can find column descriptions in schemas

Notes:

  1. Output values returned by Dogecoin API had precision loss in the clients prior to version 1.14. It's caused by this issue https://github.com/dogecoin/dogecoin/issues/1558 The explorers that used older versions to export the data may show incorrect address balances and transaction amounts.

  2. For Zcash, vjoinsplit and valueBalance fields are converted to inputs and outputs with type 'shielded' https://zcash-rpc.github.io/getrawtransaction.html, https://zcash.readthedocs.io/en/latest/rtd_pages/zips/zip-0243.html

Exporting the Blockchain

  1. Install python 3.5.3+ https://www.python.org/downloads/

  2. Install Bitcoin node https://hackernoon.com/a-complete-beginners-guide-to-installing-a-bitcoin-full-node-on-linux-2018-edition-cb8e384479ea

  3. Start Bitcoin. Make sure it downloaded the blocks that you need by executing $ bitcoin-cli getblockchaininfo in the terminal. You can export blocks below blocks, there is no need to wait until the full sync

  4. Install Bitcoin ETL:

    > pip install bitcoin-etl
    
  5. Export blocks & transactions:

    > bitcoinetl export_all --start 0 --end 499999  \
    --partition-batch-size 100 \
    --provider-uri http://user:[email protected]:8332 --chain bitcoin
    

    The result will be in the output subdirectory, partitioned in Hive style:

    output/blocks/start_block=00000000/end_block=00000099/blocks_00000000_00000099.csv
    output/blocks/start_block=00000100/end_block=00000199/blocks_00000100_=00000199.csv
    ...
    output/transactions/start_block=00000000/end_block=00000099/transactions_00000000_00000099.csv
    ...
    

    In case bitcoinetl command is not available in PATH, use python -m bitcoinetl instead.

Running in Docker

  1. Install Docker https://docs.docker.com/install/

  2. Build a docker image

    > docker build -t bitcoin-etl:latest .
    > docker image ls
    
  3. Run a container out of the image

    > docker run -v $HOME/output:/bitcoin-etl/output bitcoin-etl:latest export_blocks_and_transactions --start-block 0 --end-block 500000 \
        --provider-uri http://user:[email protected]:8332 --blocks-output output/blocks.json --transactions-output output/transactions.json
    
  4. Run streaming to console or Pub/Sub

    > docker build -t bitcoin-etl:latest-streaming -f Dockerfile_with_streaming .
    > echo "Stream to console"
    > docker run bitcoin-etl:latest-streaming stream -p http://user:[email protected]:8332 --start-block 500000
    > echo "Stream to Pub/Sub"
    > docker run -v /path_to_credentials_file/:/bitcoin-etl/ --env GOOGLE_APPLICATION_CREDENTIALS=/bitcoin-etl/credentials_file.json bitcoin-etl:latest-streaming stream -p http://user:[email protected]:8332 --start-block 500000 --output projects/your-project/topics/crypto_bitcoin
    
  5. Refer to https://github.com/blockchain-etl/bitcoin-etl-streaming for deploying the streaming app to Google Kubernetes Engine.

Command Reference

All the commands accept -h parameter for help, e.g.:

> bitcoinetl export_blocks_and_transactions --help
Usage: bitcoinetl.py export_blocks_and_transactions [OPTIONS]

  Export blocks and transactions.

Options:
  -s, --start-block INTEGER   Start block
  -e, --end-block INTEGER     End block  [required]
  -b, --batch-size INTEGER    The number of blocks to export at a time.
  -p, --provider-uri TEXT     The URI of the remote Bitcoin node
  -w, --max-workers INTEGER   The maximum number of workers.
  --blocks-output TEXT        The output file for blocks. If not provided
                              blocks will not be exported. Use "-" for stdout
  --transactions-output TEXT  The output file for transactions. If not
                              provided transactions will not be exported. Use
                              "-" for stdout
  --help                      Show this message and exit.

For the --output parameters the supported type is json. The format type is inferred from the output file name.

export_blocks_and_transactions

> bitcoinetl export_blocks_and_transactions --start-block 0 --end-block 500000 \
  --provider-uri http://user:[email protected]:8332 \
  --blocks-output blocks.json --transactions-output transactions.json

Omit --blocks-output or --transactions-output options if you want to export only transactions/blocks.

You can tune --batch-size, --max-workers for performance.

Note that required_signatures, type, addresses, and value fields will be empty in transactions inputs. Use enrich_transactions to populate those fields.

enrich_transactions

You need to run bitcoin daemon with option txindex=1 for this command to work.

> bitcoinetl enrich_transactions  \
  --provider-uri http://user:[email protected]:8332 \
  --transactions-input transactions.json --transactions-output enriched_transactions.json

You can tune --batch-size, --max-workers for performance.

get_block_range_for_date

> bitcoinetl get_block_range_for_date --provider-uri http://user:[email protected]:8332 --date=2017-03-01

This command is guaranteed to return the block range that covers all blocks with block.time on the specified date. However the returned block range may also contain blocks outside the specified date, because block times are not monotonic https://twitter.com/EvgeMedvedev/status/1073844856009576448. You can filter blocks.json/transactions.json with the below command:

> bitcoinetl filter_items -i blocks.json -o blocks_filtered.json \
-p "datetime.datetime.fromtimestamp(item['timestamp']).astimezone(datetime.timezone.utc).strftime('%Y-%m-%d') == '2017-03-01'"

export_all

> bitcoinetl export_all --provider-uri http://user:[email protected]:8332 --start 2018-01-01 --end 2018-01-02

You can tune --export-batch-size, --max-workers for performance.

stream

> bitcoinetl stream --provider-uri http://user:[email protected]:8332 --start-block 500000
  • This command outputs blocks and transactions to the console by default.
  • Use --output option to specify the Google Pub/Sub topic where to publish blockchain data, e.g. projects/your-project/topics/crypto_bitcoin. Blocks and transactions will be pushed to projects/your-project/topics/crypto_bitcoin.blocks and projects/your-project/topics/crypto_bitcoin.transactions topics.
  • The command saves its state to last_synced_block.txt file where the last synced block number is saved periodically.
  • Specify either --start-block or --last-synced-block-file option. --last-synced-block-file should point to the file where the block number, from which to start streaming the blockchain data, is saved.
  • Use the --lag option to specify how many blocks to lag behind the head of the blockchain. It's the simplest way to handle chain reorganizations - they are less likely the further a block from the head.
  • Use the --chain option to specify the type of the chain, e.g. bitcoin, litecoin, dash, zcash, etc.
  • You can tune --period-seconds, --batch-size, --max-workers for performance.

Running Tests

> pip install -e .[dev]
> echo "The below variables are optional"
> export BITCOINETL_BITCOIN_PROVIDER_URI=http://user:[email protected]:8332
> export BITCOINETL_LITECOIN_PROVIDER_URI=http://user:[email protected]:8331
> export BITCOINETL_DOGECOIN_PROVIDER_URI=http://user:[email protected]:8330
> export BITCOINETL_BITCOIN_CASH_PROVIDER_URI=http://user:[email protected]:8329
> export BITCOINETL_DASH_PROVIDER_URI=http://user:[email protected]:8328
> export BITCOINETL_ZCASH_PROVIDER_URI=http://user:[email protected]:8327
> pytest -vv

Running Tox Tests

> pip install tox
> tox

Public Datasets in BigQuery

https://cloud.google.com/blog/products/data-analytics/introducing-six-new-cryptocurrencies-in-bigquery-public-datasets-and-how-to-analyze-them

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].