Fuzzing introspection testcases
Test a fuzzer what fuzzing challenges it can solve.
The goal is to have testcases that afl++ can all complete - once they are all implemented.
Currently the different solving mechanisms have to be set as defined in src/afl-fuzz-redqueen.c
.
The comparisons to honggfuzz and libfuzzer are biased, as the testcases are how afl-fuzz sees things.
The libfuzzer/
directories has libfuzzer's applicable own testcases, of which afl++ is only able to solve 64% (vs 86% honggfuzz and 92% libfuzzer).
Setup
The fuzzers and necessary compilers must be in the path: afl-clang-fast
, afl-fuzz
, hfuzz-clang
, honggfuzz
, clang
.
Running
./test.sh TARGET [SPECIFIC_TESTCASE]
TARGET
can be (currently) one of:
- afl++
- afl++-qemu
- afl++-frida
- libafl (WIP)
- honggfuzz
- libfuzzer
- kirenenko (via
test-kirenenko.sh
, use it's docker container) - symcc + qemu (via
test-symcc.sh
andtest-symqemu.sh
, use it's docker container) - manticore (via
test-manticore.sh
)
Note that manticore and symqemu success can depend on compile options.
Testcases:
test-u8
- several chained 8 bits checkstest-u16
- several chained 16 bits checkstest-u32
- several chained 32 bits checkstest-u64
- several chained 64 bits checkstest-u128
- several chained 128 bits checkstest-u32-cmp
- several chained 32 bit lesser/greater checkstext-extint
- llvm _ExtInt() teststest-float
- several chained float checkstest-double
- several chained double checkstest-longdouble
- several chained long double checkstest-memcmp
- several chained memcmp checkstest-strcmp
- several chained strncasecmp checkstest-transform
- different transforming string checkstest-crc32
- several chained crc32 checks
The testcases are WIP - some might be further refined in the future and more added.
Results
On success the time to solve is displayed.
On failure the generated corpus files are displayed (so you can see how many finds there were). There is also an output log called TESTCASE.log
.
Test as of July 2022
All from current repository state (afl++ is CMPLOG instrumented and afl-fuzz -l3AT -Z
).
Solve time: 120 seconds for afl++/honggfuzz/libfuzzer
Sym*, Kirenenko and Manticore are not fuzzers but solvers, hence no time restriction.
SymCC and SymQEMU have the same results so they are combined to save space.
testcase | afl++ | libafl | kirenenko | sym* | manticore | afl++-qemu/afl++-frida | honggfuzz-2.5 | libfuzzer-13 |
---|---|---|---|---|---|---|---|---|
test-crc32 | 0m1,735s | OK | OK | OK | OK | 0m14,609s | FAIL | 0m14,207s |
test-double | 0m26,823s | FAIL | FAIL | FAIL | FAIL | FAIL | FAIL | FAIL |
test-extint | 0m0,429s | OK | FAIL | FAIL | FAIL | FAIL | FAIL | FAIL |
test-float | 0m4,657s | FAIL | FAIL | FAIL | FAIL | FAIL | FAIL | FAIL |
test-longdouble | 0m1,031s | FAIL | FAIL | FAIL | FAIL | FAIL | FAIL | FAIL |
test-memcmp | 0m0,837s | OK | OK | OK | OK | 0m6,494s | 0m1,005s | 0m0,308s |
test-strcmp | 0m0,835s | OK | FAIL | FAIL | FAIL | 0m5,727s | 0m1,004s | 0m1,040s |
test-transform | 0m4,334s | FAIL | FAIL | FAIL | FAIL | FAIL | FAIL | FAIL |
test-u128 | 0m0,682s | FAIL | FAIL | OK | FAIL | FAIL | FAIL | FAIL |
test-u16 | 0m1,252s | OK | OK | OK | OK | 0m8,132s | 0m1,005s | 0m3,741s |
test-u32 | 0m0,844s | OK | OK | OK | OK | 0m5,185s | 0m1,004s | 0m2,887s |
test-u32-cmp | 0m1,332s | OK | OK | OK | OK | 1m42,470s | 0m6,404s | 0m0,454s |
test-u64 | 0m0,655s | OK | OK | OK | OK | 0m3,844s | 0m1,005s | 0m5,465s |
test-u8 | 0m2,263s | OK | OK | OK | OK | 0m18,186s | 0m1,004s | 0m1,370s |
afl++ has the most solves, but due to the many solve attempts overall fuzzing performance is decreased, as can be seen at https://www.fuzzbench.com/reports/experimental/2021-01-12-aflpp/. Interpretation: the slowest solver is the best in real-world fuzzing.
More testcases or fuzzers?
Just add them and send a PR.