Property-Based Testing: Research Software Guide

Property-based testing is a software testing methodology where test suites verify general specifications—invariants, algebraic properties, and conservation laws—across an automatically generated space of randomized inputs, rather than asserting individual hand-crafted input-output examples. In research software engineering and scientific computing, numerical simulations and data analysis pipelines frequently handle high-dimensional, continuous state spaces where manual edge-case enumeration fails to catch subtle bugs. By combining generative input sampling with automated counterexample minimization (shrinking), property-based testing systematically hardens scientific algorithms against floating-point anomalies, boundary errors, and regression faults.

What it is used for in research

In empirical laboratory and computational research, software algorithms often process complex data structures, sensor measurements, and continuous numerical parameters. Traditional example-based unit testing tests isolated cases—such as verifying that a custom matrix normalization function succeeds on a specific 3×3 identity matrix. However, scientific code routinely encounters ill-conditioned matrices, subnormal floating-point values, empty data frames, and division by near-zero. Hand-crafted test cases rarely anticipate these edge cases, leading to silent calculation errors or reproducibility failures during publication analysis.

Pioneered by Claessen and Hughes (2000) with QuickCheck in Haskell and adapted for scientific Python by MacIver et al. (2019) through Hypothesis, property-based testing shifts the testing paradigm from checking specific answers to verifying universal invariants. In research software, developers define mathematical invariants that must hold true for all valid inputs ∀ x ∈ S. By integrating property-based testing into research software engineering best practices, scientific teams establish rigorous verification standards that ensure computational reproducibility across disparate computing environments.

Researchers commonly structure properties using five core testing patterns:

Round-trip transformations (inversion): Operations that invert one another should return the original data within machine precision: f⁻¹(f(x)) ≈ x. This applies directly to data serialization, file format parsers, coordinate system projections, and Fourier transformations.
Idempotence: Applying a transformation multiple times must yield the identical state as applying it once: f(f(x)) = f(x). This is essential for data cleaning pipelines, missing-value imputation, deduplication algorithms, and sorting operations.
Metamorphic relations: When ground-truth outputs are unknown for complex simulations, metamorphic testing asserts how a specific transformation of the input alters the output: f(k · x) = k^p · f(x), or adding positive observations increases mean estimates.
Test oracle equivalence: High-performance, parallelized, or GPU-accelerated algorithms are verified against a simple, trusted reference implementation across thousands of generated inputs to ensure computational equivalence.
Invariant bounds and conservation laws: Mathematical and physical constraints must never be violated regardless of input magnitude: probability values must satisfy 0 ≤ P(x) ≤ 1, energy or mass conservation must balance, and variance-covariance matrices must remain positive semi-definite.

Install or access (official source only)

In the Python research ecosystem, property-based testing is implemented via the open-source Hypothesis framework. Hypothesis integrates natively with Pytest and the standard Python unittest runner.

Install the core Hypothesis package from PyPI using pip:

pip install hypothesis

For scientific computing environments that manipulate NumPy arrays or Pandas data frames, install Hypothesis with the optional scientific extras:

pip install "hypothesis[numpy,pandas]"

Install Pytest as the test runner:

pip install pytest

Verify that Hypothesis and Pytest are correctly installed in your research virtual environment by inspecting the version information:

python -m hypothesis --version
pytest --version

Hypothesis is published by the Python Software Foundation community under the Mozilla Public License 2.0 (MPL-2.0). All documentation and package releases are distributed through the official Python Package Index (PyPI) and Read the Docs.

Step-by-step task

The following task demonstrates how to implement a property-based test suite for a scientific vector normalization algorithm using Hypothesis and Pytest.

Define the research function under test:

Create a module named normalization.py containing an Euclidean (L2 norm) vector normalization function. The function rescales an n-dimensional vector so that its total Euclidean length equals 1.0:

import math

def normalize_vector(vec: list[float]) -> list[float]:
    """Rescale a real vector to have unit Euclidean (L2) norm."""
    if not vec:
        raise ValueError("Cannot normalize an empty vector.")
    norm = math.sqrt(sum(x * x for x in vec))
    if norm == 0.0:
        raise ValueError("Cannot normalize a zero vector.")
    return [x / norm for x in vec]

Formulate the mathematical invariants:

Identify the universal properties that must hold for any non-zero real vector of arbitrary length:
- Unit length invariant: The Euclidean norm of the normalized output must equal 1.0 within floating-point tolerance (ε ≤ 10⁻⁷).
- Directional invariance: Each element in the normalized vector must share the exact sign and proportional ratio of the original vector.
- Idempotence: Normalizing an already normalized vector must return the identical vector.
Configure input generation strategies:

Use hypothesis.strategies to define input generators. For this task, use st.lists() containing st.floats(), bounding numbers to avoid floating-point overflow while excluding non-finite values (NaN and infinity). Use hypothesis.assume() to discard zero-norm vectors, ensuring tests focus on valid domain inputs.

Author the property-based test file:

Create a test script named test_normalization.py. Decorate test functions with @given to supply randomized inputs and @settings to specify the test budget. You can incorporate test runner fixtures for environment configuration and explore randomized data generation for custom distributions:

import math
from hypothesis import given, settings, assume, strategies as st
from normalization import normalize_vector

# Strategy: lists of 1 to 50 finite floats between -1e6 and 1e6
vector_strategy = st.lists(
    st.floats(min_value=-1e6, max_value=1e6, allow_nan=False, allow_infinity=False),
    min_size=1,
    max_size=50
)

@given(vec=vector_strategy)
@settings(max_examples=100)
def test_unit_norm_invariant(vec):
    """Property 1: The Euclidean norm of the output must always equal 1.0."""
    raw_norm = math.sqrt(sum(x * x for x in vec))
    assume(raw_norm > 1e-9)  # Discard vectors indistinguishable from zero

    result = normalize_vector(vec)
    result_norm = math.sqrt(sum(x * x for x in result))
    assert math.isclose(result_norm, 1.0, rel_tol=1e-7, abs_tol=1e-7)

@given(vec=vector_strategy)
@settings(max_examples=100)
def test_normalization_idempotence(vec):
    """Property 2: Normalizing an already normalized vector is idempotent."""
    raw_norm = math.sqrt(sum(x * x for x in vec))
    assume(raw_norm > 1e-9)

    first_pass = normalize_vector(vec)
    second_pass = normalize_vector(first_pass)

    assert len(first_pass) == len(second_pass)
    for a, b in zip(first_pass, second_pass):
        assert math.isclose(a, b, rel_tol=1e-7, abs_tol=1e-7)

Execute the test suite with Pytest:

Run the test suite from the terminal:
```
pytest -v test_normalization.py
```
Hypothesis automatically generates 100 distinct randomized test vectors per test function, evaluating each against the specified invariants.

Output and how to read it

When executing property-based tests via Pytest, the output differs depending on whether all randomized evaluations succeed or a falsifying counterexample is uncovered.

When all properties hold across all generated examples, Pytest outputs standard passing status:

============================= test session starts ==============================
rootdir: /research/project
collected 2 items

test_normalization.py::test_unit_norm_invariant PASSED                    [ 50%]
test_normalization.py::test_normalization_idempotence PASSED              [100%]

============================== 2 passed in 0.42s ===============================

The 2 passed notification indicates that Hypothesis generated and verified 100 pseudo-random vectors for each test function (200 total executions) without discovering any invariant violation.

When an invariant fails—for example, if an algorithm fails on extreme values or negative numbers—Hypothesis reports a failure accompanied by a minimized counterexample:

=================================== FAILURES ===================================
___________________________ test_unit_norm_invariant ___________________________

    @given(vec=vector_strategy)
>   def test_unit_norm_invariant(vec):

test_normalization.py:22: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    assert math.isclose(result_norm, 1.0, rel_tol=1e-7, abs_tol=1e-7)
E   AssertionError: assert False
E    +  where False = math.isclose(0.0, 1.0, rel_tol=1e-07, abs_tol=1e-07)

---------------------------------- Hypothesis ----------------------------------
Falsifying example: test_unit_norm_invariant(
    vec=[1e-10],
)

To read this output effectively, focus on three diagnostic sections:

Assertion failure location: Identifies the exact line of code and mathematical condition that failed (e.g., math.isclose(0.0, 1.0) returned False).
Falsifying example: Hypothesis isolates the specific input argument that produced the failure. Here, vec=[1e-10] reveals that when squared, 1e-10 * 1e-10 = 1e-20 underflows to zero in intermediate calculations.
Shrunk counterexample: Rather than printing a 50-element list with random decimals, Hypothesis’s Conjecture engine systematically stripped unnecessary elements and reduced floating-point magnitudes to find the simplest possible failing input.
Hypothesis example database: Upon encountering a failure, Hypothesis saves the counterexample byte sequence in .hypothesis/examples. On subsequent test executions, Hypothesis replays this exact counterexample first before generating new pseudo-random inputs, preventing regressions from escaping notice.

Troubleshooting

When deploying property-based testing in scientific and research codebases, teams encounter specific execution warnings and diagnostic errors. Address common failure modes with the following interventions:

FailedHealthCheck: filter_too_much
Cause: The test invokes assume(condition) so frequently that Hypothesis discards over 90% of generated inputs, wasting compute time and failing to generate enough valid test cases.
Solution: Refactor the generator strategy to construct valid data by definition rather than filtering afterward. For instance, replace assume(x > 0) with st.floats(min_value=0.001) or st.integers(min_value=1). If heavy filtering is unavoidable for domain-specific constraints, suppress the health check using @settings(suppress_health_check=[HealthCheck.filter_too_much]).
DeadlineExceeded: Test took 230ms, which exceeds the deadline of 200ms
Cause: Hypothesis enforces a default execution deadline of 200 milliseconds per generated example. Numerical simulations, differential equation solvers, and large matrix operations often exceed this limit.
Solution: Increase the per-example deadline or disable it entirely for computationally intensive models:
```
@settings(deadline=1000)  # Allow up to 1,000 ms per test run
@settings(deadline=None)  # Disable deadline check completely
```
Flaky: Hypothesis found that your test failed on an example, but when re-run on the same example it passed
Cause: The code under test contains non-deterministic behavior, such as unseeded random number generators (e.g., calling numpy.random.rand() without a fixed seed), access to global mutable state, or timestamp-dependent calculations.
Solution: Pass explicit random seeds or use the st.randoms() strategy to provide deterministic pseudo-random generators to functions under test. Ensure tests run without global side effects.
Floating-point equality failures and IEEE 754 precision
Cause: Using strict equality (assert a == b) on floating-point numbers fails due to machine precision limitations, rounding errors, and non-associative arithmetic in floating-point operations: (a + b) + c ≠ a + (b + c).
Solution: Always use tolerance-based comparisons such as math.isclose(a, b, rel_tol=1e-7, abs_tol=1e-9) or numpy.isclose(). Configure float strategies with allow_nan=False, allow_infinity=False unless your function explicitly implements IEEE 754 special-value handling. For algorithms requiring absolute mathematical proofs, compare property testing with formal verification and model checking.

Free alternatives

The table below summarizes leading free and open-source property-based testing libraries across major scientific and research programming languages:

Framework	Ecosystem / Language	License	Key Capabilities	Shrinking Mechanism
Hypothesis	Python	MPL-2.0	NumPy/Pandas array strategies, stateful testing, Pytest integration	Automated byte-stream shrinking (Conjecture engine)
QuickCheck	Haskell	BSD-3-Clause	Original PBT library, monadic testing, typeclass-derived generators	Type-directed generator shrinking via `shrink`
fast-check	TypeScript / JavaScript	MIT License	Async property testing, race condition detection, model-based stateful suites	Integrated tree-based automatic shrinking
jqwik	Java / Kotlin (JVM)	EPL-2.0	JUnit 5 engine integration, exhaustive and randomized sampling, statistical reporting	Automated structural shrinking
proptest	Rust	MIT / Apache-2.0	Macro-based strategy definitions, zero-allocation harnesses, Cargo integration	Integrated test-case simplification and shrinking
Supposition.jl	Julia	MIT License	Native Julia type integration, numerical scientific computing generators	Byte-stream representation shrinking
hedgehog	R / Haskell / Scala	BSD-3-Clause / MPL	Range-based generator sizing, integrated shrinking without typeclass boilerplate	Integrated automatic shrinking

FAQ

What is the difference between unit testing and property-based testing?

Traditional unit testing (example-based testing) evaluates hand-picked inputs against expected outputs (e.g., verifying that sort([3, 1, 2]) returns [1, 2, 3]). Property-based testing evaluates general invariants across hundreds of automatically generated inputs (e.g., verifying that for any generated list, the output list has identical length, identical element frequencies, and non-decreasing order). Property-based testing explores edge cases that developers frequently overlook.

How does Hypothesis shrink failing counterexamples?

When Hypothesis discovers an input that triggers an assertion error or exception, it activates its Conjecture shrinking engine. Rather than requiring developers to write custom reduction rules, Hypothesis treats test data as a stream of raw bytes. It systematically truncates byte sequences, lowers numerical values toward zero, and removes array elements while repeatedly re-running the test. It stops when it reaches the smallest possible input that still triggers the failure, delivering a clear, minimal counterexample.

Can property-based testing replace traditional unit tests in scientific research?

Property-based testing complements rather than replaces unit testing. Unit tests provide clear regression checks for documented edge cases and specific experimental observations. Property-based tests explore wide parameter spaces and verify systemic invariants, such as numerical conservation laws and mathematical symmetries. Using both techniques provides comprehensive verification for research software.

How many test examples should be generated per property test?

By default, Hypothesis generates 100 test examples per test function (max_examples=100). In routine local development, 50 to 100 examples balance execution speed with exploratory coverage. In continuous integration (CI) environments or pre-publication validation, researchers often increase this parameter to 500 or 1,000 examples using test configuration profiles to thoroughly stress-test numerical boundaries.

How can property-based testing test functions when the exact mathematical output is unknown?

When testing complex scientific models where closed-form analytical solutions do not exist, researchers rely on metamorphic testing, invariant verification, and oracle comparisons. Even if the exact output of a differential equation solver or climate simulation cannot be computed by hand, researchers can verify physical invariants (e.g., total energy cannot be negative, mass must be conserved) or assert that doubling an input parameter scales the output proportionally.

Sources

Claessen, K., & Hughes, J. (2000). QuickCheck: A lightweight tool for random testing of Haskell programs. ACM SIGPLAN Notices, 35(9), 268–279. https://doi.org/10.1145/351240.351266 (Accessed 2026-11-09).
MacIver, D. R., Hatfield-Dodds, Z., & contributors. (2019). Hypothesis: A new approach to property-based testing. Journal of Open Source Software, 4(43), 1891. https://doi.org/10.21105/joss.01891 (Accessed 2026-11-09).
Hypothesis Works. (2026). Hypothesis Documentation (Release 6.x). Python Software Foundation Ecosystem. https://hypothesis.readthedocs.io/ (Accessed 2026-11-09).
Pytest Development Team. (2026). Pytest: Simple Powerful Testing with Python. https://docs.pytest.org/ (Accessed 2026-11-09).