[go: up one dir, main page]

Introduce QCheck library for Property Based Tests

Julien Debonrequested to merge
nomadic-labs/tezos:sir4ur0n/test/firstQcheckTests into master

Requires opam-repository!155 (merged) to be merged

Context

See original discussion on Slack.

This MR is a visual support and POC for considering adding QCheck to Tezos test codebase.

This MR introduces QCheck to Tezos test codebase as a replacement for Crowbar (as PBT library).

Manually testing the MR

As QCheck is not (yet) part of the Tezos Opam dependencies, to build this MR you need to run

# Add the default Opam repository (doesn't remove the Tezos one)
$ opam repository add default
# Install QCheck
$ opam install qcheck-alcotest

Then you can run the following commands:

# This file was some sort of manual PBT, so I took the opportunity to migrate it to a "real" PBT
$ dune exec src/proto_alpha/lib_protocol/test/test_gas_properties.exe
# This file was using Crowbar, I migrated it to QCheck
$ dune exec src/lib_base/test/test_time.exe

Feedback

The following feedback is based on the current state of QCheck. See below, I have opened various issues to improve QCheck, and the author is overall positive about almost all changes, though we will probably need to contribute if we expect those changes quickly.

The good

  • Generator and arbitrary combination is better than Crowbar (richer set of OOTB generators and richer API to manipulate said generators/arbitraries)
  • Shrinking ❤ Note that there's no magic involved: one needs to help QCheck figuring out how to shrink by:
    • either explaining how to map back to basic random inputs (typically when using map, by passing the rev argument)
    • or by linking the type to a simpler arbitrary that has a shrinker (typically when using map_keep_input)
    • or by writing the shrink function directly
    • or do nothing: shrinking is optional
  • Unit and property tests can now be in the same file (if using the Alcotest runner)
  • QCheck has more traction and is more active

The bad

  • Output can only be a bool, meaning the compared values (usually for equality) are not displayed OOTB when a test fails (QCheck will display the inputs it used though).
    • This problem is circumvented with the qcheck_eq function
  • Alcotest runner reporting (see section below)

Runner reporting

When a test fails and using the Alcotest runner, it displays an error, which unnecessarily clutters the logs. Below a comparison of the QCheck test runner and the Alcotest runner, when a test fails and when all succeed.

Alcotest success

image

QCheck success

image

Alcotest fail

image

QCheck fail

image

QCheck runner

  • 👍 more information
  • 👍 prettier (subjective)
  • 👎 no clean exit management (not problematic when running only QCheck tests, but it is problematic when run with other tests)

Alcotest runner

  • 👍 put PBT and unit tests in the same file seamlessly
  • 👍 unified reporting with unit tests
  • 👎 output is a bit rough, particularly when a test fails (mitigated by this issue)

I feel like using the Alcotest runner is better in the long term, but I am ok with both. This MR currently uses the Alcotest runner.

QCheck issues created

Checklist

Edited by Julien Debon

Merge request reports