= QCheck :toc: macro :toclevels: 4 :source-highlighter: pygments

QuickCheck inspired property-based testing for OCaml, and combinators to generate random values to run tests on.

The documentation can be found https://c-cube.github.io/qcheck/[here]. This library spent some time in https://github.com/vincent-hugot/iTeML[qtest], but is now standalone again!

To construct advanced random generators, the following libraries might be of interest:

• https://gitlab.inria.fr/fpottier/feat/[Feat]
• @gasche's https://github.com/gasche/random-generator/[generator library]

Jan Midtgaard has http://janmidtgaard.dk/quickcheck/index.html[a lecture] about property-based testing that relies on QCheck.

toc::[]

image::https://travis-ci.org/c-cube/qcheck.svg?branch=master[alt="Build Status", link="https://travis-ci.org/c-cube/qcheck"]

== Use

See the documentation. I also wrote https://cedeela.fr/quickcheck-for-ocaml[a blog post] that explains how to use it and some design choices; however, be warned that the API changed in lots of small ways (in the right direction, I hope) so the code will not work any more. <> is an updated version of the blog post's examples.

== Build

$ make

You can use opam:

$ opam install qcheck

== License

The code is now released under the BSD license.

[[examples]] == An Introduction to the Library

First, let's see a few tests. Let's open a toplevel (e.g. utop) and type the following to load QCheck:

[source,OCaml]

#require "qcheck";;

NOTE: alternatively, it is now possible to locally do: dune utop src to load qcheck.

=== List Reverse is Involutive

We write a random test for checking that List.rev (List.rev l) = l for any list l:

[source,OCaml]

let test = QCheck.Test.make ~count:1000 ~name:"list_rev_is_involutive" QCheck.(list small_nat) (fun l -> List.rev (List.rev l) = l);;

(* we can check right now the property... *) QCheck.Test.check_exn test;;

In the above example, we applied the combinator list to the random generator small_nat (ints between 0 and 100), to create a new generator of lists of random integers. These builtin generators come with printers and shrinkers which are handy for outputting and minimizing a counterexample when a test fails.

Consider the buggy property List.rev l = l:

[source,OCaml]

let test = QCheck.Test.make ~count:1000 ~name:"my_buggy_test" QCheck.(list small_nat) (fun l -> List.rev l = l);;

When we run this test we are presented with a counterexample:

[source,OCaml]

QCheck.Test.check_exn test;;

Exception: QCheck.Test.Test_fail ("my_buggy_test", ["[0; 1] (after 23 shrink steps)"]).

In this case QCheck found the minimal counterexample [0;1] to the property List.rev l = l and it spent 23 steps shrinking it.

Now, let's run the buggy test with a decent runner that will print the results nicely (the exact output will change at each run, because of the random seed):

QCheck_runner.run_tests [test];;

--- Failure --------------------------------------------------------------------

Test my_buggy_test failed (10 shrink steps):

[0; 1]

failure (1 tests failed, 0 tests errored, ran 1 tests)

• : int = 1

For an even nicer output QCheck_runner.run_tests also accepts an optional parameter ~verbose:true.

=== Mirrors and Trees

QCheck provides many useful combinators to write generators, especially for recursive types, algebraic types, and tuples.

Let's see how to generate random trees:

[source,OCaml]

type tree = Leaf of int | Node of tree * tree

let leaf x = Leaf x let node x y = Node (x,y)

let tree_gen = QCheck.Gen.(sized @@ fix (fun self n -> match n with | 0 -> map leaf nat | n -> frequency [1, map leaf nat; 2, map2 node (self (n/2)) (self (n/2))] ));;

(* generate a few trees, just to check what they look like: *) QCheck.Gen.generate ~n:20 tree_gen;;

let arbitrary_tree = let open QCheck.Iter in let rec print_tree = function | Leaf i -> "Leaf " ^ (string_of_int i) | Node (a,b) -> "Node (" ^ (print_tree a) ^ "," ^ (print_tree b) ^ ")" in let rec shrink_tree = function | Leaf i -> QCheck.Shrink.int i >|= leaf | Node (a,b) -> of_list [a;b] <+> (shrink_tree a >|= fun a' -> node a' b) <+> (shrink_tree b >|= fun b' -> node a b') in QCheck.make tree_gen ~print:print_tree ~shrink:shrink_tree;;

Here we write a generator of random trees, tree_gen, using the fix combinator. fix is sized (it is a function from int to a random generator; in particular for size 0 it returns only leaves). The sized combinator first generates a random size, and then applies its argument to this size.

Other combinators include monadic abstraction, lifting functions, generation of lists, arrays, and a choice function.

Then, we define arbitrary_tree, a tree QCheck.arbitrary value, which contains everything needed for testing on trees:

• a random generator (mandatory), weighted with frequency to increase the chance of generating deep trees
• a printer (optional), very useful for printing counterexamples
• a shrinker (optional), very useful for trying to reduce big counterexamples to small counterexamples that are usually more easy to understand.

The above shrinker strategy is to

• reduce the integer leaves, and
• substitute an internal Node with either of its subtrees or by splicing in a recursively shrunk subtree.

A range of combinators in QCheck.Shrink and QCheck.Iter are available for building shrinking functions.

We can write a failing test using this generator to see the printer and shrinker in action:

[source,OCaml]

let rec mirror_tree (t:tree) : tree = match t with | Leaf _ -> t | Node (a,b) -> node (mirror_tree b) (mirror_tree a);;

let test_buggy = QCheck.Test.make ~name:"buggy_mirror" ~count:200 arbitrary_tree (fun t -> t = mirror_tree t);;

QCheck_runner.run_tests [test_buggy];;

This test fails with:

[source,OCaml]

--- Failure --------------------------------------------------------------------

Test mirror_buggy failed (6 shrink steps):

Node (Leaf 0,Leaf 1)

failure (1 tests failed, 0 tests errored, ran 1 tests)

• : int = 1

With the (new found) understanding that mirroring a tree changes its structure, we can formulate another property that involves sequentializing its elements in a traversal:

[source,OCaml]

let tree_infix (t:tree): int list = let rec aux acc t = match t with | Leaf i -> i :: acc | Node (a,b) -> aux (aux acc b) a in aux [] t;;

let test_mirror = QCheck.Test.make ~name:"mirror_tree" ~count:200 arbitrary_tree (fun t -> List.rev (tree_infix t) = tree_infix (mirror_tree t));;

QCheck_runner.run_tests [test_mirror];;

=== Preconditions

The functions QCheck.assume and QCheck.(==>) can be used for tests with preconditions. For instance, List.hd l :: List.tl l = l only holds for non-empty lists. Without the precondition, the property is false and will even raise an exception in some cases.

[source,OCaml]

let test_hd_tl = QCheck.(Test.make (list int) (fun l -> assume (l <> []); l = List.hd l :: List.tl l));;

QCheck_runner.run_tests [test_hd_tl];;

=== Long tests

It is often useful to have two version of a testsuite: a short one that runs reasonably fast (so that it is effectively run each time a projet is built), and a long one that might be more exhaustive (but whose running time makes it impossible to run at each build). To that end, each test has a 'long' version. In the long version of a test, the number of tests to run is multiplied by the ~long_factor argument of QCheck.Test.make.

=== Runners

The module QCheck_runner defines several functions to run tests, including compatibility with OUnit. The easiest one is probably run_tests, but if you write your tests in a separate executable you can also use run_tests_main which parses command line arguments and exits with 0 in case of success, or an error number otherwise.

=== Integration within OUnit

https://github.com/gildor478/ounit[OUnit] is a popular unit-testing framework for OCaml. QCheck provides a sub-library qcheck-ounit with some helpers, in QCheck_ounit, to convert its random tests into OUnit tests that can be part of a wider test-suite.

[source,OCaml]

let passing = QCheck.Test.make ~count:1000 ~name:"list_rev_is_involutive" QCheck.(list small_nat) (fun l -> List.rev (List.rev l) = l);;

let failing = QCheck.Test.make ~count:10 ~name:"fail_sort_id" QCheck.(list small_nat) (fun l -> l = List.sort compare l);;

let _ = let open OUnit in run_test_tt_main ("tests" >::: List.map QCheck_ounit.to_ounit_test [passing; failing])

NOTE: the package qcheck contains the module QCheck_runner which contains both custom runners and OUnit-based runners.

=== Integration within alcotest

https://github.com/mirage/alcotest/[Alcotest] is a simple and colorful test framework for OCaml. QCheck now provides a sub-library qcheck-alcotest to easily integrate into an alcotest test suite:

[source,OCaml]

let passing = QCheck.Test.make ~count:1000 ~name:"list_rev_is_involutive" QCheck.(list small_int) (fun l -> List.rev (List.rev l) = l);;

let failing = QCheck.Test.make ~count:10 ~name:"fail_sort_id" QCheck.(list small_int) (fun l -> l = List.sort compare l);;

let () = let suite = List.map QCheck_alcotest.to_alcotest [ passing; failing] in Alcotest.run "my test" [ "suite", suite ]

=== Integration within Rely https://reason-native.com/docs/rely/[Rely] is a Jest-inspire native reason testing framework. @reason-native/qcheck-rely is available via NPM and provides matchers for the easy use of qCheck within Rely.

[source, Reason]

open TestFramework; open QCheckRely;

let {describe} = extendDescribe(QCheckRely.Matchers.matchers);

describe("qcheck-rely", ({test}) => { test("passing test", ({expect}) => { let passing = QCheck.Test.make( ~count=1000, ~name="list_rev_is_involutive", QCheck.(list(small_int)), l => List.rev(List.rev(l)) == l ); expect.ext.qCheckTest(passing); (); }); test("failing test", ({expect}) => { let failing = QCheck.Test.make( ~count=10, ~name="fail_sort_id", QCheck.(list(small_int)), l => l == List.sort(compare, l) );

expect.ext.qCheckTest(failing);
();

}); });

=== Compatibility notes

Starting with 0.9, the library is split into several components:

• qcheck-core depends only on unix and bytes. It contains the module QCheck and a QCheck_base_runner module with our custom runners.
• qcheck-ounit provides an integration layer for OUnit
• qcheck provides a compatibility API with older versions of qcheck, using both qcheck-core and qcheck-ounit. It provides QCheck_runner which is similar to older versions and contains both custom and Ounit-based runners.
• qcheck-alcotest provides an integration layer with alcotest

Normally, for contributors, opam pin https://github.com/c-cube/qcheck will pin all these packages.