vertigo
Vertigo is a mutation testing framework designed to work specifically for smart contracts. This mutation testing framework implements a range of mutation operators that are either selected from previous works or tailored to solidity.
Quick Start Guide
To install vertigo, execute the following command:
pip3 install --user eth-vertigoYou can now run vertigo on a truffle project with the following command (assuming you have a development network configured in yourtruffle-config.js):
vertigo run --network developmentDepending on your environment it might be required to specify the location of the truffle executable:
vertigo run --network development --truffle-location <node_dir>/bin/truffle Or, if you're using Hardhat, just use dynamic networks:
vertigo run --hardhat-parallel 8There are a few additional parameters available that allow you to tweak the execution of vertigo:
$ vertigo run --help
Usage: vertigo run [OPTIONS]
Performs a core test campaign
Options:
--output TEXT Output core test results to file
--network TEXT Network names that vertigo can use
--ganache-path TEXT Path to ganache binary
--ganache-network <TEXT INTEGER>...
Dynamic networks that vertigo can use eg.
(develop, 8485)
--ganache-network-options TEXT Options to pass to dynamic ganache networks
--hardhat-parallel INTEGER Amount of networks that hardhat should be
using in parallel
--rules TEXT Universal Mutator style rules to use in
mutation testing
--truffle-location TEXT Location of truffle cli
--sample-ratio FLOAT If this option is set. Vertigo will apply
the sample filter with the given ratio
--exclude TEXT Vertigo won't mutate files in these
directories
--incremental TEXT File where incremental mutation state is
stored
--help Show this message and exit.
Known Issues
Ganache is generally used only for a single run of the entire test suite. For the general use case, it does not matter if Ganache creates a few thousand files. Unfortunately, once you start executing the entire test suite hundreds of times, you can end up with millions of files, and your machine could run out of free inode's. You can check whether this happens to you by running:
df -i
This issue (#1) is known, and we're working on a fix.
In the meanwhile. If your test suite is large enough to munch all your inodes, then there are two options:
- You can use the command line option
--sample-ratioto select a random subsample of the mutations (reducing the number of times that the test suite is run) - You can create a partition that has a sufficient amount of inodes available
Publications and Articles
Practical Mutation Testing for Smart Contracts - Joran J. Honig, Maarten H. Everts, Marieke Huisman
Introduction into Mutation Testing - Joran Honig
Mutation Testing for Smart Contracts - A step by step guide - Joran Honig
If you want to cite vertigo, please use the following:
@InProceedings{10.1007/978-3-030-31500-9_19,
author="Honig, Joran J.
and Everts, Maarten H.
and Huisman, Marieke",
title="Practical Mutation Testing for Smart Contracts",
booktitle="Data Privacy Management, Cryptocurrencies and Blockchain Technology",
year="2019",
publisher="Springer International Publishing",
pages="289--303"
}