Property-Based Testing with QuickCheck

Instead of writing examples, QuickCheck lets you state a property that should hold for all inputs. It then generates hundreds of random inputs and tries to break it. QuickCheck was invented in Haskell and has since been copied into almost every language.

Properties

import Test.QuickCheck
import Data.List (sort)

prop_reverseTwice :: [Int] -> Bool
prop_reverseTwice xs = reverse (reverse xs) == xs

prop_sortKeepsLength :: [Int] -> Bool
prop_sortKeepsLength xs = length (sort xs) == length xs

prop_sortIsOrdered :: [Int] -> Bool
prop_sortIsOrdered xs = and (zipWith (<=) s (drop 1 s))
  where
    s = sort xs

main :: IO ()
main = do
  quickCheck prop_reverseTwice
  quickCheck prop_sortKeepsLength
  quickCheck prop_sortIsOrdered
  quickCheck (withNumTests 1000 prop_reverseTwice)

+++ OK, passed 100 tests.
+++ OK, passed 100 tests.
+++ OK, passed 100 tests.
+++ OK, passed 1000 tests.

A property is just a function returning Bool (or Property). QuickCheck uses the argument types to decide what to generate.

When a property fails

-- illustrative: the counts differ from run to run
import Test.QuickCheck

prop_smallNumbers :: Int -> Bool
prop_smallNumbers n = n < 50

main :: IO ()
main = quickCheck prop_smallNumbers

*** Failed! Falsified (after 64 tests and 2 shrinks):
50

QuickCheck found a failing input, then shrank it to the simplest counterexample it could: exactly 50. Shrinking is what makes property failures easy to debug — a random list of 40 numbers becomes something like [0, 1].

Conditions and custom generators

import Test.QuickCheck

newtype Even = Even Int
  deriving (Show)

instance Arbitrary Even where
  arbitrary = Even . (* 2) <$> arbitrary
  shrink (Even n) = [Even (m * 2) | m <- shrink (n `div` 2)]

prop_evenHalves :: Even -> Bool
prop_evenHalves (Even n) = (n `div` 2) * 2 == n

prop_divMod :: Int -> NonZero Int -> Bool
prop_divMod a (NonZero b) = (a `div` b) * b + (a `mod` b) == a

prop_percent :: Property
prop_percent = forAll (choose (0, 100)) $ \p -> p >= 0 && p <= (100 :: Int)

main :: IO ()
main = do
  quickCheck prop_evenHalves
  quickCheck prop_divMod
  quickCheck prop_percent

+++ OK, passed 100 tests.
+++ OK, passed 100 tests.
+++ OK, passed 100 tests.

• Modifiers such as NonZero, Positive, and NonEmptyList restrict the generated values. The alternative, b /= 0 ==> property, generates any value and discards the ones that don't meet the precondition (QuickCheck reports how many it discarded).
• forAll generator property uses a specific generator; choose, elements, listOf, oneof, and frequency build generators.
• An Arbitrary instance teaches QuickCheck to generate (and shrink) your own types.

Good properties to look for

• Round trips — decode (encode x) == Just x, parse (render x) == x.
• Invariants — sorting keeps the length; a balanced tree stays balanced after every insert.
• Equivalence with a simple model — your fast Map-based cache gives the same answers as a slow list-based version.
• Idempotence — normalize (normalize x) == normalize x.

With Hspec, prop "reverses twice" prop_reverseTwice (from Test.Hspec.QuickCheck) runs properties alongside unit tests.