Property-Based Testing
Overview
Property-based testing (PBT) generates random inputs and verifies that properties hold for all of them. Instead of testing specific examples, you test invariants.
When PBT beats example-based tests:
- Serialization pairs (encode/decode)
- Pure functions with clear contracts
- Validators and normalizers
- Data structure operations
Property Catalog
| Property | Formula | When to Use |
|---|---|---|
| Roundtrip | decode(encode(x)) == x | Serialization, conversion pairs |
| Idempotence | f(f(x)) == f(x) | Normalization, formatting, sorting |
| Invariant | Property holds before/after | Any transformation |
| Commutativity | f(a, b) == f(b, a) | Binary/set operations |
| Associativity | f(f(a,b), c) == f(a, f(b,c)) | Combining operations |
| Identity | f(x, identity) == x | Operations with neutral element |
| Inverse | f(g(x)) == x | encrypt/decrypt, compress/decompress |
| Oracle | new_impl(x) == reference(x) | Optimization, refactoring |
| Easy to Verify | is_sorted(sort(x)) | Complex algorithms |
| No Exception | No crash on valid input | Baseline (weakest) |
Strength hierarchy (weakest to strongest):
No Exception -> Type Preservation -> Invariant -> Idempotence -> RoundtripAlways aim for the strongest property that applies.
Pattern Detection
Use PBT when you see:
| Pattern | Property | Priority |
|---|---|---|
encode/decode, serialize/deserialize | Roundtrip | HIGH |
toJSON/fromJSON, pack/unpack | Roundtrip | HIGH |
| Pure functions with clear contracts | Multiple | HIGH |
normalize, sanitize, canonicalize | Idempotence | MEDIUM |
is_valid, validate with normalizers | Valid after normalize | MEDIUM |
| Sorting, ordering, comparators | Idempotence + ordering | MEDIUM |
| Custom collections (add/remove/get) | Invariants | MEDIUM |
| Builder/factory patterns | Output invariants | LOW |
When NOT to Use
- Simple CRUD without transformation logic
- UI/presentation logic
- Integration tests requiring complex external setup
- Code with side effects that cannot be isolated
- Prototyping where requirements are fluid
- Tests where specific examples suffice and edge cases are understood
Library Quick Reference
| Language | Library | Import |
|---|---|---|
| Python | Hypothesis | from hypothesis import given, strategies as st |
| TypeScript/JS | fast-check | import fc from 'fast-check' |
| Rust | proptest | use proptest::prelude::* |
| Go | rapid | import "pgregory.net/rapid" |
| Java | jqwik | @Property annotations |
| Haskell | QuickCheck | import Test.QuickCheck |
For library-specific syntax and patterns: Use @ed3d-research-agents:internet-researcher to get current documentation.
Input Strategy Best Practices
- Constrain early: Build constraints INTO the strategy, not via
assume()# GOOD st.integers(min_value=1, max_value=100) # BAD - high rejection rate st.integers().filter(lambda x: 1 <= x <= 100) - Size limits: Prevent slow tests
st.lists(st.integers(), max_size=100) st.text(max_size=1000) - Realistic data: Match real-world constraints
st.integers(min_value=0, max_value=150) # Real ages, not arbitrary ints - Reuse strategies: Define once, use across tests
valid_users = st.builds(User,...) @given(valid_users) def test_one(user):... @given(valid_users) def test_two(user):...
Settings Guide
# Development (fast feedback)
@settings(max_examples=10)
# CI (thorough)
@settings(max_examples=200)
# Nightly/Release (exhaustive)
@settings(max_examples=1000, deadline=None)Quality Checklist
Before committing PBT tests:
- Not tautological (assertion doesn't compare same expression)
- Strong assertion (not just "no crash")
- Not vacuous (inputs not over-filtered by
assume()) - Edge cases covered with explicit examples (
@example) - No reimplementation of function logic in assertion
- Strategy constraints are realistic
- Settings appropriate for context
Red Flags
- Tautological:
assert sorted(xs) == sorted(xs)tests nothing - Only "no crash": Always look for stronger properties
- Vacuous: Multiple
assume()calls filter out most inputs - Reimplementation:
assert add(a, b) == a + bif that's how add is implemented - Missing edge cases: No
@example([]),@example([1])decorators - Overly constrained: Many
assume()calls means redesign the strategy
Common Mistakes
| Mistake | Fix |
|---|---|
| Testing mock behavior | Test real behavior |
| Reimplementing function in test | Use algebraic properties |
| Filtering with assume() | Build constraints into strategy |
| No edge case examples | Add @example decorators |
| One property only | Add multiple properties (length, ordering, etc.) |