Catch–doctest–lest feature comparison

Tabularised feature comparison between Catch, doctest and lest C++ test frameworks.

Skip to the feature comparison.

Note: This is an initial draft, it is incomplete and likely contains errors.

Ideas, additions, corrections, signaling omissions, etc. welcome!

The frameworks in this comparison share the following properties:

• The frameworks are available as single-file header-only library.
• Test cases are described by a string, they are not identified by a function name.
• There is a single assertion macro for expressions.
  Expressions are decomposed and original expression code can be shown alongside the values they represent.
• Fixtures can be created inside a test case. They are called sections or sub cases.

Advent of Catch Expression-decomposing assertions and function-level fixtures (sections) were introduced by Kevlin Henney in his talk Rethinking Unit Testing in C++ at ACCU London in May 2010 [1]. At that time, Phil Nash was working on a testing framework called YACUTS (Yet Another C++ Unit Test System) that would bring similar capabilities, be it a little more intrusively [2]. Kevlin's talk inspired Phil to develop Catch [3].

Kevlin's emphasis on simplicity in testing and the desire to understand how expression decomposition works inspired Martin Moene to write lest [4]. The code of lest is intentionally kept small, so that it should be fairly easy to gain insight in how things work.

Accompanying test code with the code to test was one of the drivers for Viktor Kirilov to write doctest [5]. Others are to enable using doctest with many different compilers and to shine in compile-time and run-time performance. At CppCon 2017 Viktor gave a presentation that explains many techniques used in doctest and Catch [23].

References

[1] Kevlin Henney. Rethinking Unit Testing in C++ (Video). Skills Matter, May 2010.
[2] Phil Nash. The Ultimate C++ Unit Test Framework. Blog, May 2010.
[3] Phil Nash. Unit Testing in C++ and Objective-C just got easier. Blog announcement, December 2010.
[4] Martin Moene. lest errors escape testing. Blog announcement, June 2013.
[5] Viktor Kirilov. doctest – the Lightest C++ Unit Testing Framework. Announcement in Overload 137, February 2017.
[6] Phil Nash. Catch-1 on GitHub. 2010.
[7] Phil Nash. Catch-2 on GitHub. 2017.
[8] Viktor Kirilov. doctest on GitHub. 2014.
[9] Martin Moene. lest on GitHub. 2013.
[10] Joel Falcou. Design rationale for using ULP with stf.
[11] Gennadiy Rozental and Raffi Enficiaud. Checkpoints in Boost.Test.
[12] Matchers in Catch-1.
[13] Matchers in Catch-2.
[14] Matchers in lest: via hamlest.
[15] Hamcrest 1.3 Quick Reference (PDF).
[16] Decorators in doctest.
[17] Wikipedia. List of C++ IDEs.
[18] CodeBlocks IDE. Non-empty, non-comment lines, via File > Properties.
[19] StackOverflow. Questions tagged catch-unit-test.
[20] Viktor Kirilov. doctest Benchmarks. May 2017.
[21] Linux Foundation (LF). CII Best Practices.
[22] Jacob Kaplan-Moss. Writing great documentation.
[23] Viktor Kirilov. Mix Tests and Production Code With Doctest - Implementing and Using the Fastest Modern C++ Testing Framework (slides). CppCon 2017.