bitcoin-core / bitcoin-maintainer-tools

External repository for Bitcoin Core related maintenance tools.

github-merge

A small script to automate merging pull-requests securely and sign them with GPG.

For example, if the "to" repo is identical to the "from" repo:

./github-merge.py 1234

(in any git repository) will help you merge pull request #1234 for the configured repository.

Otherwise, for a differing "from" repo:

./github-merge.py --repo-from=bitcoin-core/gui 1234

will fetch the pull request from another monotree repository. Be sure to also set githubmerge.pushmirrors (see below).

What it does:

• Fetch master and the pull request.
• Locally construct a merge commit.
• Show the diff that merge results in.
• Ask you to verify the resulting source tree (so you can do a make check or whatever).
• Ask you whether to GPG sign the merge commit.
• Ask you whether to push the result upstream.

This means that there are no potential race conditions (where a pull request gets updated while you're reviewing it, but before you click merge), and when using GPG signatures, that even a compromised GitHub couldn't mess with the sources.

Setup

Configuring the github-merge tool for the bitcoin repository is done in the following way:

git config githubmerge.repository bitcoin/bitcoin
git config githubmerge.pushmirrors "[email protected]:bitcoin-core/gui.git,[email protected]:YourPrivateMirror/bitcoin-core.git"
git config githubmerge.testcmd "make -j4 check" (adapt to whatever you want to use for testing)
git config --global user.signingkey mykeyid

If you want to use HTTPS instead of SSH for accessing GitHub, you need set the host additionally:

git config githubmerge.host "https://github.com"  (default is "[email protected]", which implies SSH)

Authentication (optional)

The API request limit for unauthenticated requests is quite low, but the limit for authenticated requests is much higher. If you start running into rate limiting errors it can be useful to set an authentication token so that the script can authenticate requests.

• First, go to Personal access tokens.
• Click 'Generate new token'.
• Fill in an arbitrary token description. No further privileges are needed.
• Click the Generate token button at the bottom of the form.
• Copy the generated token (should be a hexadecimal string)

Then do:

git config --global user.ghtoken "pasted token"

Create and verify timestamps of merge commits

To create or verify timestamps on the merge commits, install the OpenTimestamps client via pip3 install opentimestamps-client. Then, download the gpg wrapper ots-git-gpg-wrapper.sh and set it as git's gpg.program. See the ots git integration documentation for further details.

update-translations

Run this script from the root of a repository to update all translations from Transifex. It will do the following automatically:

• Fetch all translations
• Post-process them into valid and committable format
• Add missing translations to the build system (TODO)

To be able to pull translation files from the Transifex website, it needs the Transifex CLI.

clang-format

A script to format cpp source code according to the .clang-format file in the bitcoin repo. This should only be applied to new files or files which are currently not actively developed on. Also, git subtrees are not subject to formatting.

Note: The script is currently untested and unmaintained, but kept for archival reasons, in case it is planned to be used some day.

build-for-compare

Build for binary comparison.

See build-for-compare.py --help for more information.

Builds from current directory, which is assumed to be a git clone of the bitcoin repository.

DO NOT RUN this with the nocopy=1 flag set on working tree if you have any local additions, it will nuke all non-repository files, multiple times over. By leaving nocopy off (default) the git tree is copied to a temporary directory and all operations are performed there.

Example:

git clone https://github.com/bitcoin/bitcoin.git bitcoin-compare
cd bitcoin-compare
../bitcoin-maintainer-tools/build-for-compare.py 4731cab 2f71490
sha256sum /tmp/compare/bitcoind.*.stripped
git diff -W --word-diff /tmp/compare/4731cab /tmp/compare/2f71490

backport

Script to backport pull requests in order of merge, to minimize number of conflicts. Pull ids are listed in to_backport.txt or given on the command line, and they must be prefixed with the repository name, e.g.:

../bitcoin-maintainer-tools/backport.py bitcoin/bitcoin#21907 bitcoin-core/gui#277 bitcoin-core/gui#365

Requires pip3 install gitpython or similar.

unittest-statistics

unittest-statistics.py can be used to print a table of the slowest 20 unit tests.

Usage:

unittest-statistics.py </path/to/test_bitcoin> [<subtest>]

For example:

unittest-statistics.py src/test/test_bitcoin wallet_tests

treehash512

This script will show the SHA512 tree has for a certain commit, or HEAD by default.

Usage:

treehash512.py [<commithash>]

This should match the Tree-SHA512 commit metadata field added by github-merge.

signoff

This is an utility to manually add a treehash to the HEAD commit and then gpg-sign it. This is useful when there is the need to manually add a commit.

Usage:

signoff.py

(no command line arguments)

When there is already a treehash on the HEAD commit, it is compared against what is computed. If this matches, it continues. If the treehash mismatches an error is thrown. If there is no treehash it adds the "Tree-SHA512:" header with the computed hash to the commit message.

After making sure the treehash is correct it verifies whether the commit is signed. If so it just displays the signature, if not, it is signed.

subtree updates

Bitcoin Core comes with several subtrees (c.f. https://github.com/bitcoin/bitcoin/tree/master/test/lint#git-subtree-checksh) To update the subtree, make sure to fetch the remote of the subtree. Then a simple call should pull in and squash the changes:

git subtree pull --prefix src/${prefix} ${remote_repo} ${ref} --squash

For setting up a subtree, refer to git help subtree.

check-dnsseeds

Sanity-check the DNS seeds used by Bitcoin Core.

Usage:

check-dnsseeds.py

Example output:

* Mainnet
OK   seed.bitcoin.sipa.be (40 results)
OK   dnsseed.bluematt.me (33 results)
FAIL dnsseed.bitcoin.dashjr.org
OK   seed.bitcoinstats.com (50 results)
OK   seed.bitcoin.jonasschnelli.ch (38 results)
OK   seed.btc.petertodd.org (23 results)
OK   seed.bitcoin.sprovoost.nl (35 results)
OK   dnsseed.emzy.de (41 results)

* Testnet
OK   testnet-seed.bitcoin.jonasschnelli.ch (36 results)
OK   seed.tbtc.petertodd.org (38 results)
OK   testnet-seed.bluematt.me (5 results)

delete non-reduced fuzz inputs

Refer to the documentation inside the script.

fastcopy-chaindata

Fast local copy of Bitcoin Core blockchain state.

fastcopy-chaindata.py ~/.bitcoin /path/to/temp/datadir

This utility hardlinks all but the last block data file (rev and blk), and hardlinks all .ldb files to the destination. The last data files as well as the other leveldb data files (such as the log) are copied.

This relies on the fact that block files (except the last) and ldb files are read-only once they are written.

Warning: Hardlinking only works within a filesystem, and may not work for all filesystems.

list-pulls

Script to parse git commit list, extract github issues to create a changelog in text and json format.

Run this in the root directory of the repository.

This requires an up-to-date checkout of https://github.com/zw/bitcoin-gh-meta.git in the parent directory, or environment variable GHMETA.

It takes a range of commits and a .json file of PRs to exclude, for example if these are already backported in a minor release. This can be the pulls.json generated from a previous release.

Example usage:

../maintainer-tools/list-pulls.py v0.18.0 0.19 relnot/pulls-exclude.json > relnot/pulls.md

The output of this script is a first draft based on rough heuristics, and likely needs to be extensively manually edited before ending up in the release notes.

make-tag

Make a new release tag, performing a few checks.

Usage: make-tag.py <tag>.

guix-verify

A script to verify guix deterministic build signatures for a release in one glance. It will print a matrix of signer versus build package ("noncodesigned" and "all"), and a list of missing keys.

To be able to verify PGP signatures, it needs the gpg modules. This can be installed from pip, for example:

pip3 install --user gpg

(or install the distribution package, in Debian/Ubuntu this is python3-gpg).

Example usage: ./guix-verify.py -r 24.0 -d ../guix.sigs -k ../bitcoin/contrib/builder-keys/keys.txt

Where

• -r 24.0 specifies the release to verify signatures for.
• -d ../gitian.sigs specifies the directory where the repository with signatures, gitian.sigs is checked out.
• ../bitcoin/contrib/builder-keys/keys.txt is the path to keys.txt file inside the main repository that specifies the valid keys and what signers they belong to.

Example output:

Signer        noncodesigned       all
0xb10c           No Key            -
achow101           OK             OK       
benthecarman     No Key            -
...

Missing keys
norisg         3A51FF4D536C5B19BE8800A0F2FC9F9465A2995A  from GPG, from keys.txt
...

See --help for the full list of options and their descriptions.

The following statuses can be shown:

• Ok Full match.
• No key Signer name/key combination not in keys.txt, or key not known to GPG (which one of these it is, or both, will be listed under "Missing keys").
• Expired Known key but it has expired.
• Bad Known key but invalid PGP signature.
• Mismatch Correct PGP signature but mismatching binaries.

ghwatch

This is a script to watch your github notifications in the terminal. It will show a table that is refreshed every 10 minutes (configurable). It can be exited by pressing ESC or Ctrl-C.

Dependencies

The github python module is a required dependency for github API access. This can be installed for your user using pip3 install --user PyGithub, or globally using your distribution's package manager e.g. apt-get install python3-github.

Configuration

To generate a default configuration file in ~/.config/ghwatch/ghwatch.conf do

./ghwatch.py --default-config

Then, edit the configuration file. Only thing that is necessary to change is ghtoken. You will need to create a github authentication token then insert it here:

    "ghtoken": "<token from github>",

Depending on your browser preference you might want to change browser, this is the command that will be invoked when clicking on an issue number. It defaults to null which indicates to use the system web browser.

If you want to see PR status (and other issue details like labels), point meta for the bitcoin/bitcoin repository to an up-to-date checkout of bitcoin-gh-meta.

    "meta": {
        "bitcoin/bitcoin": "/path/to/bitcoin-gh-meta"
    },

To keep this repository up to date you can set the interval in seconds in 'auto_update', default is 0 (i.e. no automatic update). Be aware that bitcoin-gh-meta is being refreshed every two hours (7200 seconds).

Sorting the notifications by {reason, time} can be enabled with the 'sort_notifications' boolean field (default=false).

By editing the label_prio structure it is possible to affect what labels will be shown. The first label encountered in this list for an issue in the associated repository will be shown as the label in the table.

Command-line options

Some other settings can be set through command line options. See ./ghwatch.py --help for the list of command line options and their descriptions.

Display

Most of the columns are self-explanatory, except for:

• r this notification reason from the GH API as a two-letter code. This can be:
  • as assign
  • au author
  • co comment
  • in invitation
  • ma manual
  • me mention
  • rr review requested
  • sa security alert
  • sc state change
  • su subscribed
  • tm team mention
• k the kind of notification as a letter. This can be:
  • P pull request
  • I issue
  • C commit

Controls

Left-click on a PR number to show details in a web browser.

The program can be exited by pressing ESC or Ctrl-C.