Fuzz tests

Pyth's test suite includes a battery of fuzz tests. Currently, the fuzz testing covers the pd Pyth decimal library. We believe that generative tests, like fuzzing, are an excellent way to improve the security of a project. Thus, we developed a new fuzzing harness for Pyth's sorting algorithm implementation to augment Pyth's test suite.

The fuzzing harness is a straightforward C program which reads test cases from stdin, and calls Pyth's sorting library on it. To best reflect real-world conditions, our harness invokes the sorting library with the same parameters and instantiation as is used in Pyth. The harness tests sorting inputs of variable size and variable values. To validate correctness, the harness compares Pyth sort's results against libc's qsort as a ground truth. To validate memory safety, we compiled the harness with clang's AddressSanitizer (ASAN) and MemorySanitizer (MSAN).

We ran the harness in two regimes: (1) "dumb", black-box fuzzing and (2) coverage-guided gray-box fuzzing. For black-box fuzzing, we simply piped /dev/urandom into the harness for each execution. We ran the harness under this regime for 12 hours on a 32-core machine. For coverage-guided fuzzing, we built and ran the harness under the AFL++ fuzzer↗. Under this regime, we ran 16 instances of the fuzzer for 12 hours on a 32-core machine. The results of the fuzz testing did not yield any memory safety or correctness violations.

To facilitate the integration of this new fuzz test into the development lifecycle as part of continuous integration, we also integrated the harness with LLVM libfuzzer.

As part of this engagement, we provided the fuzzing framework to Pyth Data Association, and we recommend integrating fuzz tests with the existing test suite. Fuzzers expand branch, path, and state space coverage and decrease the likelihood of bugs. This is because fuzzers regularly catch corner cases that human programmers fail to consider when writing unit tests. Another benefit of fuzz testing is that developers can stress test business-critical invariants that are specific to the application. This benefits developer confidence, making the development lifecycle both faster and more secure.