golden
A benchmarking tool that measures and records runtime of any executable and also happens to know how to compile Nim.
cpp +/ nim-1.0arc +/ cpp +/ nim-1.3
The idea here is that we're gonna make a record of everything we run, everything we build, and be able to discover and pinpoint regressions automatically using native git-fu performed by the tool. Fire and forget!
Installation
Nimph
$ nimph clone golden
Nimble
$ nimble install golden
Usage
If you pass it a binary, it'll run it a bunch of times and report some runtime statistics periodically.
If you pass it some Nim source, it will compile it for you and report some compilation and runtime statistics periodically.
By default, it will run until you interrupt it.
$ golden --truth=0.002 bench.nim
compilations after 0s
ββββββββββ¬βββββββββββ¬βββββββββββ¬βββββββββββ¬βββββββββββ
β Builds β Min β Max β Mean β StdDev β
ββββββββββΌβββββββββββΌβββββββββββΌβββββββββββΌβββββββββββ€
β 1 β 0.396129 β 0.396129 β 0.396129 β 0.000000 β
ββββββββββ΄βββββββββββ΄βββββββββββ΄βββββββββββ΄βββββββββββ
benchmark after 1s
ββββββββββ¬βββββββββββ¬βββββββββββ¬βββββββββββ¬βββββββββββ
β Runs β Min β Max β Mean β StdDev β
ββββββββββΌβββββββββββΌβββββββββββΌβββββββββββΌβββββββββββ€
β 1 β 1.959187 β 1.959187 β 1.959187 β 0.000000 β
ββββββββββ΄βββββββββββ΄βββββββββββ΄βββββββββββ΄βββββββββββ
benchmark after 3s
ββββββββββ¬βββββββββββ¬βββββββββββ¬βββββββββββ¬βββββββββββ
β Runs β Min β Max β Mean β StdDev β
ββββββββββΌβββββββββββΌβββββββββββΌβββββββββββΌβββββββββββ€
β 2 β 1.958892 β 1.959187 β 1.959039 β 0.000147 β
ββββββββββ΄βββββββββββ΄βββββββββββ΄βββββββββββ΄βββββββββββ
completed benchmark after 5s
ββββββββββ¬βββββββββββ¬βββββββββββ¬βββββββββββ¬βββββββββββ
β Runs β Min β Max β Mean β StdDev β
ββββββββββΌβββββββββββΌβββββββββββΌβββββββββββΌβββββββββββ€
β 3 β 1.958892 β 1.961293 β 1.959791 β 0.001069 β
ββββββββββ΄βββββββββββ΄βββββββββββ΄βββββββββββ΄βββββββββββ
Benchmarking the compilation of Nim itself:
$ cd ~/git/Nim
$ golden koch -- boot -d:danger
# ...
ββββββββββ¬βββββββββββ¬βββββββββββ¬βββββββββββ¬βββββββββββ
β # β Min β Max β Mean β StdDev β
ββββββββββΌβββββββββββΌβββββββββββΌβββββββββββΌβββββββββββ€
β 12 β 8.846606 β 9.485832 β 8.945023 β 0.165638 β
ββββββββββ΄βββββββββββ΄βββββββββββ΄βββββββββββ΄βββββββββββ
Benchmarking compilation of slow-to-compile Nim:
$ golden --compilation openapi.nim
ββββββββββ¬ββββββββββββ¬ββββββββββββ¬ββββββββββββ¬βββββββββββ
β # β Min β Max β Mean β StdDev β
ββββββββββΌββββββββββββΌββββββββββββΌββββββββββββΌβββββββββββ€
β 1 β 91.946370 β 91.946370 β 91.946370 β 0.000000 β
ββββββββββ΄ββββββββββββ΄ββββββββββββ΄ββββββββββββ΄βββββββββββ
ββββββββββ¬ββββββββββββ¬ββββββββββββ¬ββββββββββββ¬ββββββββββββ
β # β Min β Max β Mean β StdDev β
ββββββββββΌββββββββββββΌββββββββββββΌββββββββββββΌββββββββββββ€
β 2 β 29.271556 β 91.946370 β 60.608963 β 31.337407 β
ββββββββββ΄ββββββββββββ΄ββββββββββββ΄ββββββββββββ΄ββββββββββββ
Command Line Options
trutha float percentage indicating how much jitter you'll acceptruntimea float of seconds after which we should kill each invocationiterationsa number of invocations after which we should stop the benchmarkstoragethe path to a database file you wish to use; must end in.golden-lmdbinteractive-forcedassume output friendly to humansjson-outputassume output friendly to machines (work in progress)color-forcedenable color output when not ininteractivemodeprune-outliersthrow out this percentage of aberrant invocations with long runtime in order to clean up the histogramdry-rundon't write any results to the databasehistogram-classesthe number of points in the histogramcompilation-onlybenchmark the Nim compiler on the given source(s)briefonly output the statistics at the completion of the benchmarknever-outputnever emit anything via stdout/stderrdump-outputalways print the stdout/stderr of the benchmarked program--the following arguments are passed to the compiler and runtime. Note that if you supply-- cppfor compilation via C++, you will need to supply your own defines such as-d:danger.
License
MIT