An overview of property-based testing functionality
Property-based testing (PBT) frameworks come with a number of different features, but which library supports which features? For a PBT newcomer, it can be hard to tell. Strictly speaking you don't even need a PBT framework. Property-based tests can be written from scratch on a case-by-case basis using a random number generator. One can even test stateful code without a state-machine framework, e.g., as outlined here. However a framework provides reusable parts and infrastructure, thus paving the way for bigger developments, such as property-based testing automotive software against the AUTOSAR specification.
This overview is to help myself keep track. It has been compiled over a number of years teaching PBT. As features are gradually added to a framework the table's entries may unfortunately become outdated. YMMV.
I'll be happy to accept PRs for updating entries and adding new frameworks.
Framework | Language | Gen. EDSL | Shrinking | Int. shr. | State machine | Par. st. mach. | Cov. guidance |
---|---|---|---|---|---|---|---|
QuickCheck | Erlang | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |
PropEr | Erlang | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |
QuickCheck | Haskell | :heavy_check_mark: | :heavy_check_mark: | ( :heavy_check_mark: ) | ( :heavy_check_mark: ) | ||
SmallCheck | Haskell | :heavy_check_mark: | 1 | 1 | |||
Hedgehog | Haskell | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |
Scala-Hedgehog | Scala / JVM | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |
R-Hedgehog | R | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | ||
FSharp-Hedgehog | F# / .Net | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |||
Hypothesis | Python | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | ( :heavy_check_mark: )2 | ( :heavy_check_mark: ), ( :heavy_check_mark: ) |
TSTL | Python | 3 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |
ScalaCheck | Scala / JVM | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | ||
gopter | Go | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |||
Rapid | Go | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | ||
propCheck | Kotlin | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |
FsCheck | F# / .Net | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |||
CsCheck | C# / .Net | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |
fast-check | JS / TS | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | ( :heavy_check_mark: )2 | |
QCheck | OCaml | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark:4 | :heavy_check_mark: | :heavy_check_mark: | |
Crowbar | OCaml | :heavy_check_mark: | ( :heavy_check_mark: )5 | :heavy_check_mark: | |||
Monolith | OCaml | :heavy_check_mark: | ( :heavy_check_mark: )5 | :heavy_check_mark: | |||
Base_quickcheck | OCaml | :heavy_check_mark: | :heavy_check_mark: | ||||
Popper | OCaml | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |||
junit-quickcheck | Java | :heavy_check_mark: | :heavy_check_mark: | ( :heavy_check_mark: )6 | |||
QuickTheories | Java | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | ( :heavy_check_mark: ) | ( :heavy_check_mark: ) | :heavy_check_mark: |
jqwik | Java | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | ||
quickcheck | Rust | :heavy_check_mark: | :heavy_check_mark: | ||||
propcheck | Rust | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | |||
Fox | Obj.C / Swift | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | ( :heavy_check_mark: ) | |
SwiftCheck | Swift | :heavy_check_mark: | :heavy_check_mark: | ||||
RapidCheck | C++ | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | ( :heavy_check_mark: ) | ||
test.check | Clojure | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | ? | ? | |
Lua-QuickCheck | Lua | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | ||
theft | C | ( :heavy_check_mark: ) | :heavy_check_mark: | ( :heavy_check_mark: ) | ( :heavy_check_mark: ) | ||
DeepState | C / C++ | ( :heavy_check_mark: ) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | ||
Echidna | Solidity / EVM | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | ||
... |
Legend:
int
, list
, map
, ...)Footnotes
afl-tmin
which is unaware of (and hence may break) OCaml typing.The term property-based testing seems to originate from 'Property-Based Testing; A New Approach to Testing for Assurance' by Fink and Bishop (SE Notes 1997). The approach was popularized as an embedded domain-specific language in 'QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs' by Claessen and Hughes (ICFP 2000) which inspired ports to many other languages.
Integrated shrinking is explained in more detail in
State machines to test protocols and systems with state are described in
eqc_commands
moduleqcstm
for OCamlParallel state-machine tests for race conditions were later introduced in