{ "id": "epk18p1pdy", "slug": "property-based-testing", "title": "Property-based testing", "updated": "2026-10-15", "topic_path": [ { "id": "technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/topics/technology" }, { "id": "technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing" }, { "id": "technology.computing.software", "label": "Software and programming", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.software" }, { "id": "technology.computing.software.software_engineering", "label": "Software engineering and development process", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.software.software_engineering" }, { "id": "technology.computing.software.software_engineering.software-testing-and-quality", "label": "Software testing and quality", "api_url": "https://www.edgepedia.org/api/v1/topics/technology.computing.software.software_engineering.software-testing-and-quality" } ], "geo": [ { "id": "geo.nongeo.t2001.technology.computing.software", "label": "Non-geographic · 2001 to 2020: Software and programming", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.software", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t2001", "label": "Non-geographic · 2001 to 2020", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001" }, { "id": "geo.nongeo.t2001.technology", "label": "Technology and the built world", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology" }, { "id": "geo.nongeo.t2001.technology.computing", "label": "Computing and digital systems", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing" }, { "id": "geo.nongeo.t2001.technology.computing.software", "label": "Software and programming", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.technology.computing.software" } ] } ], "excerpt": "Property-based testing is a software testing approach in which developers state general properties a program should satisfy, and tools check them against large numbers of generated inputs.", "snippet": "Property-based testing is a software testing approach in which developers state general properties a program should satisfy, and tools check them against large numbers of generated inputs.", "node": "technology.computing.software.software_engineering.software-testing-and-quality", "markdown": "# Property-based testing\n\nProperty-based testing is a software testing approach in which the developer states general properties that a program should satisfy, and a tool checks those properties against large numbers of automatically generated inputs instead of hand-picked examples.[1](https://www.cse.chalmers.se/~rjmh/QuickCheck/) An example-based unit test fixes one input and one expected output; a property describes a whole family of inputs and a relation that must hold for all of them, and the tool reports a minimized failing input when a property is violated.[2](https://research.chalmers.se/publication/517894/file/517894_Fulltext.pdf) David MacIver, author of the [Hypothesis](https://www.edgepedia.org/hypothesis) framework, frames the field as building tests that, when fuzzed, reveal problems that direct fuzzing of the system could not reveal; the core of Hypothesis is accordingly a structured fuzzing library called [Conjecture](https://www.edgepedia.org/conjecture).[3](https://hypothesis.works/articles/what-is-property-based-testing/)\n\n| Key fact | Detail |\n|---|---|\n| Defining idea | Check stated properties over randomly generated inputs, reporting minimized counterexamples[2](https://research.chalmers.se/publication/517894/file/517894_Fulltext.pdf) |\n| Origin | QuickCheck, by Koen Claessen and John Hughes, ICFP 2000, Montreal[4](https://research.chalmers.se/en/publication/237427) |\n| Default effort | 100 randomly generated tests per property in QuickCheck and Hypothesis[5](https://www.cs.tufts.edu/~nr/cs257/archive/john-hughes/quick.pdf) • [6](https://dl.acm.org/doi/full/10.1145/3764068) |\n| Effectiveness | In a Python corpus study, each property-based test killed about 50 times as many mutants as the average unit test[6](https://dl.acm.org/doi/full/10.1145/3764068) |\n| Main limitation | The oracle problem: finding strong properties to test is the hard part[7](https://doi.org/10.1109/tse.2014.2372785) |\n| Adoption | 5% of 25,000 surveyed Python developers used Hypothesis in 2023, the 6th most used testing framework[6](https://dl.acm.org/doi/full/10.1145/3764068) |\n\n## How it works\n\nA property is a universally quantified assertion, for example that reversing a list twice returns the original list, written as an ordinary function in the host language. The framework generates inputs, runs the property, and treats any assertion failure as a counterexample.[1](https://www.cse.chalmers.se/~rjmh/QuickCheck/)\n\nGenerators carry an implicit size parameter that starts small and grows during testing, so cheap small cases run first.[5](https://www.cs.tufts.edu/~nr/cs257/archive/john-hughes/quick.pdf) In QuickCheck the `Arbitrary` class supplies default generators, and combinators such as `choose` for uniform choice from an interval and `frequency` for weighted alternatives build custom distributions.[8](https://www.cse.chalmers.se/~rjmh/QuickCheck/manual_body.html)\n\nWhen a test fails, the framework shrinks it: a systematic greedy search for a simpler failing test, automating the first stage of fault diagnosis.[2](https://research.chalmers.se/publication/517894/file/517894_Fulltext.pdf) Lists shrink by removing elements and numbers shrink toward zero, so a failing reverse property almost always yields the counterexample `[0,1]`.[2](https://research.chalmers.se/publication/517894/file/517894_Fulltext.pdf)\n\n## How it is done\n\nA practitioner writes properties as functions (by QuickCheck convention named `prop_`), quantified over their parameters, and runs them; QuickCheck executes 100 random cases by default, configurable with `withNumTests`.[8](https://www.cse.chalmers.se/~rjmh/QuickCheck/manual_body.html) • [9](https://hackage-content.haskell.org/package/QuickCheck-2.18.0.0/docs/Test-QuickCheck.html) A generator specifies a set of possible test data, a probability distribution over that set, and a shrinking strategy.[10](http://aszt.inf.elte.hu/~cefp2009/materials/papers/Hughes.pdf) On failure the tool prints the shrunk counterexample, for example `*** Failed! Falsified (after 3 tests and 3 shrinks): [0,1]`, which is then turned into a regression test.[9](https://hackage-content.haskell.org/package/QuickCheck-2.18.0.0/docs/Test-QuickCheck.html)\n\nA workflow pitfall to avoid: the conditional combinator `==>` discards inputs that fail its precondition, and can skew the test distribution badly; one documented run generated 795 cases of which 695 were discarded to obtain 100 valid ones, so custom `forAll` generators are usually preferable.[8](https://www.cse.chalmers.se/~rjmh/QuickCheck/manual_body.html) • [11](https://jesper.sikanda.be/posts/quickcheck-intro.html)\n\n## Origin\n\nQuickCheck was introduced by Koen Claessen and John Hughes in a paper presented at ICFP 2000 in Montreal, published as volume 35(9) pages 268–279 with DOI 10.1145/351240.351266.[4](https://research.chalmers.se/en/publication/237427) The original implementation was a single pure Haskell'98 module of about 300 lines, used mainly from the Hugs interpreter.[5](https://www.cs.tufts.edu/~nr/cs257/archive/john-hughes/quick.pdf) The paper credits the earlier DAISTS system with the idea of testing equational properties from a specification, though DAISTS required the user to supply test cases.[5](https://www.cs.tufts.edu/~nr/cs257/archive/john-hughes/quick.pdf) The term \"property-based testing\" was also used for specification-driven testing of security properties of C programs in the TASPEC language, a different lineage from QuickCheck's random testing.[12](https://seclab.cs.ucdavis.edu/projects/vulnerabilities/scriv/1997-testing.pdf)\n\n## Variants\n\nSmallCheck and Lazy SmallCheck, by Colin Runciman, Matthew Naylor, and Fredrik Lindblad in 2008, kept QuickCheck's type-based generators but tested properties for all values up to a progressively increasing depth limit instead of random sampling.[13](https://doi.org/10.1145/1411286.1411292) The Erlang community produced PropEr (GPLv3) and Triq (Apache v2); ports exist for C, C++, Clojure, F#, Go, Java, JavaScript, Julia, OCaml, Python, R, Ruby, Rust, Scala, and Swift.[14](https://propertesting.com/book_property_based_testing_tools.html)\n\nShrinking strategy separates the major frameworks. QuickCheck uses manual shrinking, an explicit function packaged with the generator. Hedgehog uses integrated shrinking: generators produce a tree of values whose children are shrink steps.[15](https://well-typed.com/blog/2019/05/integrated-shrinking/) Hypothesis instead reduces the sequence of random choices made during generation, cast as a shortlex optimization over choice sequences, so every reduced test is one the generator could have produced; this reducer has been its supported method since early 2016.[16](https://drops.dagstuhl.de/storage/00lipics/lipics-vol166-ecoop2020/LIPIcs.ECOOP.2020.13/LIPIcs.ECOOP.2020.13.pdf) JQF blurs the line with fuzzing by generating structured inputs via coverage-guided fuzzing to check user-written properties.[6](https://dl.acm.org/doi/full/10.1145/3764068)\n\n## Applications\n\nQuviq, founded by Thomas Arts and John Hughes in 2006, applied QuickCheck at Ericsson, Volvo Cars, Klarna, Basho, and Dropbox; its largest project developed acceptance tests for AUTOSAR C code on behalf of [Volvo Cars](https://www.edgepedia.org/volvo-cars).[17](https://publications.lib.chalmers.se/records/fulltext/232550/local_232550.pdf) At Klarna, a notorious bug could be provoked with a database of at most one record using 5–6 API calls, and fixing took less than a day once the minimal failing test was found.[17](https://publications.lib.chalmers.se/records/fulltext/232550/local_232550.pdf) A qualitative study of 30 QuickCheck users at the financial firm Jane Street found about one-third of participants reported property-based tests uncovered bugs not detected by other testing methods.[18](https://link.springer.com/article/10.1007/s10664-026-10953-w)\n\nQuantitatively, a corpus study of 426 Python programs using Hypothesis found each property-based test kills about 50 times as many mutants as the average unit test (\\( p < .0001 \\)); 55% of killed mutations fell to a single generated input and 76% within the first 20.[6](https://dl.acm.org/doi/full/10.1145/3764068) Property choice matters: for one union bug, a postcondition property failed after 50 tests on average while a logically equivalent model-based property failed after 8.4.[2](https://research.chalmers.se/publication/517894/file/517894_Fulltext.pdf) LLM-assisted property synthesis has become an active line: An agent that autonomously writes Hypothesis tests for existing Python code produced top-scoring bug reports of which 86% were valid and 81% both valid and reportable, including a `numpy.random.wald` bug traced to catastrophic cancellation.[19](https://www.anthropic.com/research/property-based-testing)\n\n## Limitations and alternatives\n\nThe central difficulty is the oracle problem, the challenge of identifying properties to write, which is common to all test-generation approaches; Barr, Harman, McMinn, Shahbaz, and Yoo survey it in IEEE Transactions on Software Engineering.[7](https://doi.org/10.1109/tse.2014.2372785) Weak properties miss bugs: validity properties missed five of eight bugs in Hughes's benchmark, and preconditions that exclude tricky cases are a key pitfall.[2](https://research.chalmers.se/publication/517894/file/517894_Fulltext.pdf) Hughes also warns that the biggest danger of QuickCheck is a false sense of security from skewed test-data distributions invisible to the user, and that errors may lie in the specification or the generator rather than the code; the original paper found errors split roughly evenly among generators, specifications, and the program.[10](http://aszt.inf.elte.hu/~cefp2009/materials/papers/Hughes.pdf) • [5](https://www.cs.tufts.edu/~nr/cs257/archive/john-hughes/quick.pdf) Shrinking is expensive, accounting for 80% of QuickCheck testing cost when enabled in one experiment, and Jane Street participants found writing shrinkers undesirable and error-prone.[20](https://publications.lib.chalmers.se/records/fulltext/232554/local_232554.pdf) • [21](https://www.cis.upenn.edu/~bcpierce/papers/icse24-pbt-in-practice)\n\nCompared with fuzzing, the techniques overlap but are tuned differently: fuzz testing is typically used in integration testing of complete systems to detect crashes and memory unsafety, while property-based testing flushes out logical errors in smaller modules.[21](https://www.cis.upenn.edu/~bcpierce/papers/icse24-pbt-in-practice) [Metamorphic testing](https://www.edgepedia.org/metamorphic-testing) can be seen as a specific kind of property-based testing, and PBT tools can automate its test generation and verification.[22](https://arxiv.org/html/2211.12003v1)\n\n## References\n\n1. [QuickCheck: An Automatic Testing Tool for Haskell (original project page)](https://www.cse.chalmers.se/~rjmh/QuickCheck/)\n2. [How to Specify It!: A Guide to Writing Properties of Pure Functions (Hughes)](https://research.chalmers.se/publication/517894/file/517894_Fulltext.pdf)\n3. [What is Property Based Testing? (Hypothesis works, David MacIver)](https://hypothesis.works/articles/what-is-property-based-testing/)\n4. [Chalmers research record for the QuickCheck ICFP'00 paper](https://research.chalmers.se/en/publication/237427)\n5. [QuickCheck: A lightweight tool for random testing of Haskell programs (ICFP '00 paper PDF)](https://www.cs.tufts.edu/~nr/cs257/archive/john-hughes/quick.pdf)\n6. [An Empirical Evaluation of Property-Based Testing in Python (PACMPL / OOPSLA 2025)](https://dl.acm.org/doi/full/10.1145/3764068)\n7. [Earl T. Barr and colleagues (2014). The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering.](https://doi.org/10.1109/tse.2014.2372785)\n8. [QuickCheck manual (original QuickCheck 1)](https://www.cse.chalmers.se/~rjmh/QuickCheck/manual_body.html)\n9. [Test.QuickCheck, QuickCheck 2.18.0.0 documentation](https://hackage-content.haskell.org/package/QuickCheck-2.18.0.0/docs/Test-QuickCheck.html)\n10. [Software Testing with QuickCheck (Hughes, CEFP 2009 course notes)](http://aszt.inf.elte.hu/~cefp2009/materials/papers/Hughes.pdf)\n11. [An introduction to property-based testing with QuickCheck (Jesper Cockx)](https://jesper.sikanda.be/posts/quickcheck-intro.html)\n12. [Property-Based Testing; A New Approach to Testing for Assurance (Tester's Assistant, UC Davis)](https://seclab.cs.ucdavis.edu/projects/vulnerabilities/scriv/1997-testing.pdf)\n13. [Smallcheck and lazy smallcheck | Proceedings of the first ACM SIGPLAN symposium on Haskell](https://doi.org/10.1145/1411286.1411292)\n14. [Property-Based Testing Tools (PropEr Testing book, ch. 2)](https://propertesting.com/book_property_based_testing_tools.html)\n15. [Integrated versus Manual Shrinking, Well-Typed](https://well-typed.com/blog/2019/05/integrated-shrinking/)\n16. [Test-Case Reduction via Test-Case Generation: Insights from the Hypothesis Reducer (ECOOP 2020)](https://drops.dagstuhl.de/storage/00lipics/lipics-vol166-ecoop2020/LIPIcs.ECOOP.2020.13/LIPIcs.ECOOP.2020.13.pdf)\n17. [Experiences with QuickCheck: testing the hard stuff and staying sane (Hughes)](https://publications.lib.chalmers.se/records/fulltext/232550/local_232550.pdf)\n18. [Property-based testing in Python: empirical insights (Empirical Software Engineering)](https://link.springer.com/article/10.1007/s10664-026-10953-w)\n19. [Finding bugs with Claude and property-based testing (Anthropic, 2025)](https://www.anthropic.com/research/property-based-testing)\n20. [Find More Bugs with QuickCheck (MoreBugs extension)](https://publications.lib.chalmers.se/records/fulltext/232554/local_232554.pdf)\n21. [Property-Based Testing in Practice (ICSE 2024)](https://www.cis.upenn.edu/~bcpierce/papers/icse24-pbt-in-practice)\n22. [Application of property-based testing tools for metamorphic testing (ENASE 2022)](https://arxiv.org/html/2211.12003v1)\n\n---\n*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Software and programming › Software engineering and development process › Software testing and quality*\n\n*Initially written Sep 29, 2026 · Edited: Oct 15, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [ "https://www.cs.tufts.edu/~nr/cs257/archive/john-hughes/quick.pdf", "https://www.cis.upenn.edu/~bcpierce/papers/icse24-pbt-in-practice" ], "url": "https://www.edgepedia.org/property-based-testing", "markdown_url": "https://www.edgepedia.org/property-based-testing.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Property-based testing\", Edgepedia (EdgeChat), https://www.edgepedia.org/property-based-testing. Edgepedia Community License 1.0.", "credit_md": "\"[Property-based testing](https://www.edgepedia.org/property-based-testing)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/property-based-testing](https://www.edgepedia.org/property-based-testing). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Property-based testing\", Edgepedia (EdgeChat), https://www.edgepedia.org/property-based-testing. Edgepedia Community License 1.0.", "speakable": "Property-based testing is a software testing approach in which developers state general properties a program should satisfy, and tools check them against large numbers of generated inputs." }