{ "id": "epm5c0jn68", "slug": "anytime-valid-inference", "title": "Anytime-valid inference", "updated": "2026-10-15", "topic_path": [ { "id": "physical", "label": "Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/topics/physical" }, { "id": "physical.math", "label": "Mathematics and statistics", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math" }, { "id": "physical.math.statistics", "label": "Statistics and probability", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math.statistics" }, { "id": "physical.math.statistics.stats_statistical_inference", "label": "Statistical inference, estimation, sampling, and testing", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math.statistics.stats_statistical_inference" }, { "id": "physical.math.statistics.stats_statistical_inference.hypothesis_testing", "label": "Hypothesis testing", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math.statistics.stats_statistical_inference.hypothesis_testing" }, { "id": "physical.math.statistics.stats_statistical_inference.sequential_multiplicity.sequential_tests_general", "label": "Sequential tests and stopping-based inference", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.math.statistics.stats_statistical_inference.sequential_multiplicity.sequential_tests_general" } ], "geo": [ { "id": "geo.nongeo.t2001.physical.math.statistics", "label": "Non-geographic · 2001 to 2020: Statistics and probability", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.physical.math.statistics", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t2001", "label": "Non-geographic · 2001 to 2020", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001" }, { "id": "geo.nongeo.t2001.physical", "label": "Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.physical" }, { "id": "geo.nongeo.t2001.physical.math", "label": "Mathematics and statistics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.physical.math" }, { "id": "geo.nongeo.t2001.physical.math.statistics", "label": "Statistics and probability", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.physical.math.statistics" } ] } ], "excerpt": "Anytime-valid inference, also called safe anytime-valid inference (SAVI), is a statistical framework using e-processes and confidence sequences whose error guarantees hold at any stopping time, useful in A/B testing.", "snippet": "Anytime-valid inference, also called safe anytime-valid inference (SAVI), is a statistical framework using e-processes and confidence sequences whose error guarantees hold at any stopping time, useful in A/B testing.", "node": "physical.math.statistics.stats_statistical_inference.sequential_multiplicity.sequential_tests_general", "markdown": "# Anytime-valid inference\n\nAnytime-valid inference is a statistical framework that provides tests and confidence sequences whose error guarantees hold at arbitrary [stopping times](https://www.edgepedia.org/stopping-time), so that accumulating data can be monitored continuously and analyzed or stopped at any point without inflating the type-I error rate. It is also called safe anytime-valid inference (SAVI). The central objects are e-processes, nonnegative evidence processes for testing, and confidence sequences, interval sequences for estimation, both constructed so that their guarantees survive optional stopping or continuation for any reason.[1](https://arxiv.org/html/2210.01948v2)\n\nClassical fixed-sample p-values and confidence intervals are unreliable when users endogenously choose sample sizes by continuously monitoring a test, a practice common in [A/B testing](https://www.edgepedia.org/a-b-testing) and, in survey data, admitted by more than 55% of psychologists as \"adding data until the results look good\".[2](https://arxiv.org/pdf/1512.04922) • [3](https://ar5iv.labs.arxiv.org/html/2106.02693) SAVI methods allow continuous monitoring, adaptive halting or continuation, and peeking without violating validity, which is useful in exploratory settings without oversight such as university labs and the tech industry.[4](https://stat.cmu.edu/~aramdas/icml25/ramdas1.pdf)\n\n| Key fact | Detail |\n|---|---|\n| Guarantee | Error bounds hold for any stopping time τ, possibly not specified or anticipated in advance[1](https://arxiv.org/html/2210.01948v2) • [4](https://stat.cmu.edu/~aramdas/icml25/ramdas1.pdf) |\n| Core object | An e-process for a null \\( H_{0} \\) is a nonnegative sequence with \\( E_{H_{0}}[e_{\\tau}] \\le 1 \\) at every stopping time τ[5](https://doi.org/10.48550/arxiv.2009.03167) |\n| p-value | If M is an e-process, \\( 1/(\\max_{s \\le t} M_{s}) \\) is an anytime-valid p-value[1](https://arxiv.org/html/2210.01948v2) |\n| Stopping rule | Reject the null when the e-process crosses the threshold \\( 1/\\alpha \\)[6](https://arxiv.org/html/2602.06379v1) |\n| Price | In a normal-location example (\\( \\mu = 0 \\) vs \\( \\mu = .3 \\), \\( N = 100 \\)), an anytime-valid test rejects with probability 79% at any time n ≤ N versus 91% for the fixed-sample z-test[7](https://academic.oup.com/jrsssb/article/88/4/1366/8493290) |\n| Width | Discrete-mixture confidence sequences stay within a factor of two of fixed-sample CLT interval widths over five orders of magnitude in time[8](https://par.nsf.gov/servlets/purl/10251927) |\n| Admissibility | All admissible anytime-valid p-processes, confidence sequences, e-processes, and sequential tests must employ nonnegative martingales, explicitly or implicitly[5](https://doi.org/10.48550/arxiv.2009.03167) |\n\n## How it works\n\nAn e-process for a null hypothesis is a nonnegative stochastic process \\( (E_{t}) \\) satisfying \\( E_{P}[E_{\\tau}] \\le 1 \\) for every stopping time τ and every distribution P in the null.[5](https://doi.org/10.48550/arxiv.2009.03167) • [9](https://arxiv.org/abs/2604.19353) For a composite null, an e-process reports the minimum wealth across simultaneous betting games against each distribution P in the null. An e-value is a nonnegative random variable whose expected value under the null hypothesis is at most 1; an infimum of betting scores or a realized-to-expected-value ratio is a valid e-value only under suitable conditions.[1](https://arxiv.org/html/2210.01948v2)\n\nThe probabilistic engine is Ville's inequality: under the null, P(sup_t E_t ≥ c) ≤ 1/c for any stopping time, which delivers anytime-valid type-I control without α-spending or sample-size commitments.[9](https://arxiv.org/abs/2604.19353) • [10](https://arxiv.org/html/2602.04146v5) Ville's inequality converts e-processes into sequential tests or confidence sequences, and it is the central instrument for translating them into practical tests.[1](https://arxiv.org/html/2210.01948v2) • [9](https://arxiv.org/abs/2604.19353)\n\nTwo consequences define the framework. First, if M is an e-process for P, then \\( 1/(\\max_{s \\le t} M_{s}) \\) is an anytime-valid p-value: such a p-value satisfies \\( P(p_{\\tau} \\le u) \\le u \\) for any stopping time τ and any u in [0,1], so with probability at least 1 − u it never drops below u, and stopping decisions based on its current value never violate type-I error control.[1](https://arxiv.org/html/2210.01948v2) Second, a (1 − α)-confidence sequence is a sequence of sets (C_t) with P(∀t ≥ 1: φ(P) ∈ C_t) ≥ 1 − α, that is, coverage at every stopping time; ordinary confidence sets require the sample size or stopping time to be fixed in advance of seeing any data.[1](https://arxiv.org/html/2210.01948v2) • [5](https://doi.org/10.48550/arxiv.2009.03167)\n\n## How it is done\n\nConstruction routes include mixture likelihood ratios, betting-based e-processes, and universal inference, an e-process construction of the form used by Wasserman, Ramdas and Balakrishnan (2020) that is not a test (super)martingale, showing that e-processes are more general than test martingales.[1](https://arxiv.org/html/2210.01948v2) • [11](https://doi.org/10.1073/pnas.1922664117)\n\nIn practice, monitoring is simple: the same threshold \\( E_{t} \\ge 1/\\alpha \\) controls type-I error under arbitrary peeking, which simplifies adaptive clinical trial monitoring compared to prespecified monitoring schedules.[6](https://arxiv.org/html/2602.06379v1) [Confidence](https://www.edgepedia.org/confidence) sequences are obtained by inverting a test martingale: a standard design uses a prequential plug-in test martingale thresholded at 1/α, and Ville's inequality implies P{θ ∉ CI_t for some t ≥ 0} ≤ α.[12](https://nejsds.nestat.org/journal/NEJSDS/article/73/text)\n\n## Origin\n\nThe mathematical ingredients are old. The use of nonnegative martingales and supermartingales arises from wealths of gambling strategies and from sequential tests based on likelihood ratios.[13](https://ar5iv.labs.arxiv.org/html/2304.01163) The e-variable was later analyzed by Shafer et al. (2011) and others.[14](https://safestatistics.com/wp-content/uploads/2025/01/SafeTestingOUPWebVersie-2.pdf)\n\nThe modern framework took shape in 2020. Universal inference was reported by [Larry Wasserman](https://www.edgepedia.org/larry-a-wasserman), Aaditya Ramdas, and Sivaraman Balakrishnan in the [Proceedings of the National Academy of Sciences](https://www.edgepedia.org/proceedings-of-the-national-academy-of-sciences) in 2020.[11](https://doi.org/10.1073/pnas.1922664117) The e-process concept was reported by [Aaditya Ramdas](https://www.edgepedia.org/aaditya-ramdas) and colleagues in 2020, in arXiv work showing that admissible anytime-valid inference must rely on nonnegative martingales.[5](https://doi.org/10.48550/arxiv.2009.03167)\n\n## Variants\n\nThe vocabulary distinguishes several related objects. An e-value is a single nonnegative statistic with null expectation at most one; a standard e-value is not anytime-valid, exhibiting its error bound only at fixed times, which does not suffice for sequential settings.[1](https://arxiv.org/html/2210.01948v2) • [5](https://doi.org/10.48550/arxiv.2009.03167) An e-process is its sequential generalization. Always-valid p-values are constructed via sequential tests of power one, tests that do not accept the null in finite time, with stopping when the p-value crosses level α implementing the corresponding sequential test.[2](https://arxiv.org/pdf/1512.04922) Confidence sequences are sequences of confidence intervals built from the first samples with a uniform (simultaneous) coverage guarantee.[15](https://stat.cmu.edu/~aramdas/talks/JHU24.pdf) Safe testing and game-theoretic statistics frame inference as betting against the null, a principle discussed in depth for point nulls by Shafer and for composite nulls by Grünwald, De Heide and Koolen and by Waudby-Smith and Ramdas.[1](https://arxiv.org/html/2210.01948v2)\n\n## Applications\n\nThe methodology has been implemented in a large-scale commercial A/B testing platform to analyze hundreds of thousands of experiments, where fixed-horizon p-values are unreliable under continuous monitoring.[2](https://arxiv.org/pdf/1512.04922) In clinical research, e-value based tests illustrated on the SWEPIS trial, which was stopped early for harm, would have given sufficient evidence to stop for harm after the same number of events.[3](https://ar5iv.labs.arxiv.org/html/2106.02693) The anytime-valid logrank test provides type-I error guarantees under optional stopping without specifying a maximum sample size or stopping rule, and e-process-based analyses allow extending existing trials, starting new trials, and meta-analyses while retaining type-I error control under continuous monitoring and early stopping, capabilities classical p-value analyses lack.[12](https://nejsds.nestat.org/journal/NEJSDS/article/73/text) SAVI also responds to needs in living meta-analysis and bandit experiments.[1](https://arxiv.org/html/2210.01948v2)\n\n## Limitations and alternatives\n\nAnytime-validity has a price. In a standard normal location setting testing \\( \\mu = 0 \\) against \\( \\mu = .3 \\) with \\( N = 100 \\) observations, the z-test rejects with probability 91% while the anytime-valid sequential test rejects at any time \\( n \\le N \\) with probability just 79%; if the alternative is learned sequentially via the maximum likelihood estimator, the sequential test's power is just 47%.[7](https://academic.oup.com/jrsssb/article/88/4/1366/8493290) The width cost is bounded: the discrete mixture confidence sequence stays within a factor of two of the fixed-sample CLT interval width over five orders of magnitude in time, shrinking at a \\( 1/\\sqrt{t} \\) rate ignoring log factors, with an \\( O(t^{-1/2} \\log\\log t) \\) asymptotic rate that matches the lower bound implied by the law of the iterated logarithm.[8](https://par.nsf.gov/servlets/purl/10251927) The cost at a known horizon can be eliminated: for any valid test based on N observations, an anytime-valid sequential test can be constructed that matches it after N observations.[7](https://academic.oup.com/jrsssb/article/88/4/1366/8493290)\n\nOther limitations are structural. The evidence against a null as measured by a test martingale or e-process may decrease as more data are collected, since gamblers may lose money even with favorable odds.[1](https://arxiv.org/html/2210.01948v2)\n\nCompared with classical sequential analysis, the most popular sequential test remains the Sequential Probability Ratio Test (SPRT).[7](https://academic.oup.com/jrsssb/article/88/4/1366/8493290) Ville's inequality delivers type-I control without α-spending or sample-size commitments, in direct contrast with group-sequential alpha-spending designs.[10](https://arxiv.org/html/2602.04146v5)\n\n## References\n\n1. [Game-Theoretic Statistics and Safe Anytime-Valid Inference](https://arxiv.org/html/2210.01948v2)\n2. [Always Valid Inference: Continuous Monitoring of A/B Tests](https://arxiv.org/pdf/1512.04922)\n3. [Generic E-Variables for Exact Sequential k-Sample Tests that allow for Optional Stopping](https://ar5iv.labs.arxiv.org/html/2106.02693)\n4. [Aaditya Ramdas, SAVI slides (ICML 2025)](https://stat.cmu.edu/~aramdas/icml25/ramdas1.pdf)\n5. [Ramdas, Aaditya and colleagues (2020). Admissible anytime-valid sequential inference must rely on nonnegative martingales. arXiv (Cornell University).](https://doi.org/10.48550/arxiv.2009.03167)\n6. [E-values for Adaptive Clinical Trials: Anytime-Valid Monitoring in Practice](https://arxiv.org/html/2602.06379v1)\n7. [Anytime validity is free: inducing sequential tests (JRSS-B)](https://academic.oup.com/jrsssb/article/88/4/1366/8493290)\n8. [Time-uniform, nonparametric, nonasymptotic confidence sequences (Howard et al.)](https://par.nsf.gov/servlets/purl/10251927)\n9. [Asymptotic e-processes](https://arxiv.org/abs/2604.19353)\n10. [Bayes, E-values, and Testing](https://arxiv.org/html/2602.04146v5)\n11. [Larry Wasserman, Aaditya Ramdas, Sivaraman Balakrishnan (2020). Universal inference. Proceedings of the National Academy of Sciences.](https://doi.org/10.1073/pnas.1922664117)\n12. [The Anytime-Valid Logrank Test: Error Control Under Continuous Monitoring with Unlimited Horizon](https://nejsds.nestat.org/journal/NEJSDS/article/73/text)\n13. [The extended Ville's inequality for nonintegrable nonnegative supermartingales](https://ar5iv.labs.arxiv.org/html/2304.01163)\n14. [Safe Testing (book, OUP web version)](https://safestatistics.com/wp-content/uploads/2025/01/SafeTestingOUPWebVersie-2.pdf)\n15. [Aaditya Ramdas tutorial talk (JHU 2024)](https://stat.cmu.edu/~aramdas/talks/JHU24.pdf)\n\n---\n*Topic: Encyclopedia › Physical world and mathematics › Mathematics and statistics › Statistics and probability › Statistical inference, estimation, sampling, and testing › Hypothesis testing › Sequential tests and stopping-based inference*\n\n*Initially written Sep 29, 2026 · Edited: Oct 15, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [ "https://stat.cmu.edu/~aramdas/icml25/ramdas1.pdf", "https://stat.cmu.edu/~aramdas/talks/JHU24.pdf" ], "url": "https://www.edgepedia.org/anytime-valid-inference", "markdown_url": "https://www.edgepedia.org/anytime-valid-inference.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Anytime-valid inference\", Edgepedia (EdgeChat), https://www.edgepedia.org/anytime-valid-inference. Edgepedia Community License 1.0.", "credit_md": "\"[Anytime-valid inference](https://www.edgepedia.org/anytime-valid-inference)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/anytime-valid-inference](https://www.edgepedia.org/anytime-valid-inference). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Anytime-valid inference\", Edgepedia (EdgeChat), https://www.edgepedia.org/anytime-valid-inference. Edgepedia Community License 1.0.", "speakable": "Anytime-valid inference, also called safe anytime-valid inference, is a statistical framework using e-processes and confidence sequences whose error guarantees hold at any stopping time, useful in A/B testing." }