{ "id": "epdy8rp34t", "slug": "expert-elicitation", "title": "Expert elicitation", "updated": "2026-09-30", "topic_path": [ { "id": "physical", "label": "Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/topics/physical" }, { "id": "physical.science", "label": "General science and scientific practice", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.science" }, { "id": "physical.science.research_methods_design", "label": "Research methods and experimental design", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.science.research_methods_design" }, { "id": "physical.science.research_methods_design.systematic-reviews-and-evidence-synthesis", "label": "Systematic reviews and evidence synthesis", "api_url": "https://www.edgepedia.org/api/v1/topics/physical.science.research_methods_design.systematic-reviews-and-evidence-synthesis" } ], "geo": [ { "id": "geo.nongeo.t1946.physical.science.research_methods_design", "label": "Non-geographic · 1946 to 2000: Research methods and experimental design", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.physical.science.research_methods_design", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t1946", "label": "Non-geographic · 1946 to 2000", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946" }, { "id": "geo.nongeo.t1946.physical", "label": "Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.physical" }, { "id": "geo.nongeo.t1946.physical.science", "label": "General science and scientific practice", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.physical.science" }, { "id": "geo.nongeo.t1946.physical.science.research_methods_design", "label": "Research methods and experimental design", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t1946.physical.science.research_methods_design" } ] }, { "id": "geo.nongeo.t2001.physical", "label": "Non-geographic · 2001 to 2020: Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.physical", "path": [ { "id": "geo.nongeo", "label": "Non-geographic", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo" }, { "id": "geo.nongeo.t2001", "label": "Non-geographic · 2001 to 2020", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001" }, { "id": "geo.nongeo.t2001.physical", "label": "Physical world and mathematics", "api_url": "https://www.edgepedia.org/api/v1/geo/geo.nongeo.t2001.physical" } ] } ], "excerpt": "Expert elicitation is a structured method for synthesizing expert judgments into probability distributions when data are sparse or absent, used in health care, natural hazards, and forecasting.", "snippet": "Expert elicitation is a structured method for synthesizing expert judgments into probability distributions when data are sparse or absent, used in health care, natural hazards, and forecasting.", "node": "physical.science.research_methods_design.systematic-reviews-and-evidence-synthesis", "markdown": "# Expert elicitation\n\nExpert elicitation is a structured method for synthesizing the judgments of domain experts into probability distributions when empirical data are sparse or nonexistent. The usual output is a subjective probability density function (PDF) expressing an expert's belief about an uncertain quantity, produced through a procedure designed to minimize the inherent biases of subjective judgment.[1](https://www.rivm.nl/bibliotheek/rapporten/630004001.pdf) [Reference](https://www.edgepedia.org/reference) guides take practitioners step by step through eliciting and analyzing expert judgment for products, systems, and situations where measurements or test results are sparse or absent.[2](https://epubs.siam.org/doi/book/10.1137/1.9780898718485) The Sheffield Elicitation Framework (SHELF) is a package of documents, templates, and software for eliciting probability distributions for uncertain quantities from a group of experts.[3](https://shelf.sites.sheffield.ac.uk/)\n\n| Key fact | Detail |\n|---|---|\n| Output | Subjective probability distributions (PDFs) for quantities where data are insufficient or unattainable[1](https://www.rivm.nl/bibliotheek/rapporten/630004001.pdf) |\n| Named protocols | SHELF, modified Delphi, Cooke's classical method, IDEA, and the MRC reference protocol[4](https://www.ispor.org/heor-resources/good-practices/article/recommendations-on-the-use-of-structured-expert-elicitation-protocols-for-healthcare-decision-making) |\n| Group size | Around 4 to 8 experts, large enough to cover the range of opinion[5](https://www.tonyohagan.co.uk/academic/pdf/ElicSubjSci.pdf) |\n| Validation record | 33 classical-model applications from 2006 to March 2015; fewer than one-third of individual experts were statistically accurate[6](https://ideas.repec.org/a/oup/renvpo/v12y2018i1p113-132..html); post-2006 data extended to 530 experts assessing 580 calibration variables[7](https://www.sciencedirect.com/science/article/pii/S0169207020300959) |\n| Performance scoring | Calibration score (probability that score differences arose by chance) combined with an information score (distribution concentration)[8](https://www.york.ac.uk/media/che/documents/Structured%20expert%20elicitation%20for%20healthcare%20decision%20making%20A%20practical%20guide.pdf) |\n| Aggregation modes | Behavioural (consensus discussion, as in SHELF and Delphi) versus mathematical (opinion pooling or performance weighting, as in Cooke's method)[5](https://www.tonyohagan.co.uk/academic/pdf/ElicSubjSci.pdf) |\n| Application fields | Natural hazards, environmental management, food safety, health care, security and counterterrorism, economic and geopolitical forecasting, and risk and reliability analysis[9](https://www.journalslibrary.nihr.ac.uk/hta/HTA25370) |\n\n## How it works\n\nStructured protocols exist because unstructured consultation is vulnerable to cognitive shortcuts. Design guidance targets biases including anchoring, availability, and representativeness,[10](https://www.nvtag.nl/wp-content/uploads/2025/10/Ijzerman_NVTAG_SEE.pdf) and structured protocols are used to improve the transparency, accuracy, and consistency of quantitative judgments while limiting the effect of heuristics and biases.[4](https://www.ispor.org/heor-resources/good-practices/article/recommendations-on-the-use-of-structured-expert-elicitation-protocols-for-healthcare-decision-making)\n\nProtocols divide on how individual judgments become one answer. Behavioural aggregation, used in SHELF and Delphi-style methods, has experts discuss and agree a consensus distribution; mathematical aggregation pools individual distributions with Bayesian methods, opinion pooling, or Cooke's method.[8](https://www.york.ac.uk/media/che/documents/Structured%20expert%20elicitation%20for%20healthcare%20decision%20making%20A%20practical%20guide.pdf) No normative theory has been established for combining expert opinions into a single consensus distribution, so combination methods are compared empirically, by calibration and sharpness, rather than theoretically.[11](https://pmc.ncbi.nlm.nih.gov/articles/PMC7996321/)\n\n## How it is done\n\nA typical workflow runs as follows. First, the quantities to be elicited are defined precisely; O'Hagan's staged protocol treats precise definition as a prerequisite before any judgments are collected.[12](http://www.tonyohagan.co.uk/academic/pdf/probspec.pdf) A 1992 risk-assessment procedure lists the remaining steps: selection of experts, selection and definition of issues, preparation for probability elicitation, elicitation, post-elicitation processing of judgments, and documentation.[13](https://ascelibrary.org/doi/10.1061/%28ASCE%290733-9402%281992%29118%3A2%28136%29) Panels of around 4 to 8 experts are recommended; larger numbers usually extend discussion without adding knowledge.[5](https://www.tonyohagan.co.uk/academic/pdf/ElicSubjSci.pdf)\n\nElicitation formats differ by protocol. SHELF asks for a median with quartiles or tertiles in a structured sequence designed to minimize anchoring bias; the Cooke method typically asks for 5th, 50th, and 95th percentiles.[5](https://www.tonyohagan.co.uk/academic/pdf/ElicSubjSci.pdf) The facilitator then shows experts their judgments in ways designed to provoke discussion, and revised distributions are elicited.[12](http://www.tonyohagan.co.uk/academic/pdf/probspec.pdf) A probability distribution is fitted to the elicited summaries, a stage that often blurs into elicitation itself because the choice of summaries depends on the chosen distributional form.[14](https://www.stat.cmu.edu/tr/tr808/tr808.pdf)\n\nIn Cooke's classical method, experts also answer seed questions whose true values are known to the facilitator but not to the experts; seeds come from future measurements, unpublished measurements, unfamiliar information in standard datasets, or combinations of datasets.[8](https://www.york.ac.uk/media/che/documents/Structured%20expert%20elicitation%20for%20healthcare%20decision%20making%20A%20practical%20guide.pdf) The product of statistical accuracy and informativeness gives a combined score that, after normalization, sets each expert's weight in the pooled decision maker; experts below a calibration cutoff are dropped.[15](https://repository.tudelft.nl/file/File_efead35d-dfa4-4723-b516-f152e5ccd3bb) • [16](https://pmc.ncbi.nlm.nih.gov/articles/PMC4034232/)\n\n## Origin\n\nThe [Delphi method](https://www.edgepedia.org/delphi-method) is credited to Olaf Helmer, who published \"Analysis of the Future: The Delphi Method\" in 1967 through the Defense Technical Information Center.[17](https://link.springer.com/chapter/10.1007/978-981-96-8357-4_1) The method was first reported in a classified document on the estimation of bombing requirements, and Dalkey and Helmer published a report on the method in 1963.[17](https://link.springer.com/chapter/10.1007/978-981-96-8357-4_1) Dalkey identified three defining features: anonymous response, iteration with controlled feedback, and a statistical group response.[17](https://link.springer.com/chapter/10.1007/978-981-96-8357-4_1) In Delphi, experts give single point estimates that can later be compared with observed values; distributional elicitation methods differ in producing full probability distributions.[18](https://filelist.tudelft.nl/EWI/Over%20de%20faculteit/Afdelingen/Applied%20Mathematics/uitzoeken/Applied%20Probability/Risk/Download/eur18820.pdf)\n\nThe Classical Model was presented in Roger M. Cooke's 1991 work of that name.[19](https://doi.org/10.1093/oso/9780195064650.003.0012) The IDEA protocol is documented in a 2017 practical guide by Victoria Hemming and colleagues in Methods in Ecology and [Evolution](https://www.edgepedia.org/evolution).[20](https://doi.org/10.1111/2041-210x.12857) A UK government technical report describes SHELF as a set of probabilistic elicitation tools associated with the Department of Probability and [Statistics](https://www.edgepedia.org/statistics) at the [University of Sheffield](https://www.edgepedia.org/university-of-sheffield),[21](https://assets.publishing.service.gov.uk/media/5a7576e8ed915d6faf2b3314/The_probabilistic_elicitation_of_subjective_data_WEBSITE.pdf) and the framework is described in methodological tutorials by Tony O'Hagan.[5](https://www.tonyohagan.co.uk/academic/pdf/ElicSubjSci.pdf)\n\n## Variants\n\nThe five protocols most often used differ in individual versus group elicitation, presence of feedback, fixed versus variable interval methods, aggregation by consensus or mathematics, and [distribution fitting](https://www.edgepedia.org/distribution-fitting).[10](https://www.nvtag.nl/wp-content/uploads/2025/10/Ijzerman_NVTAG_SEE.pdf)\n\n**Delphi** is an iterative survey with feedback over successive rounds, allowing experts to revise opinions toward consensus.[9](https://www.journalslibrary.nihr.ac.uk/hta/HTA25370) Named variants include Policy Delphi, Dissensus Delphi, Argument Delphi, and Disaggregative Policy Delphi, some of which do not aim for consensus.[17](https://link.springer.com/chapter/10.1007/978-981-96-8357-4_1)\n\n**Cooke's classical method** requires no interaction between experts; experts are empirically tested on seed questions and weighted by a combination of statistical accuracy and informativeness.[4](https://www.ispor.org/heor-resources/good-practices/article/recommendations-on-the-use-of-structured-expert-elicitation-protocols-for-healthcare-decision-making) **SHELF** uses behavioral aggregation: private individual judgments, then discussion to agree a single consensus distribution. Sources name this construct differently, as a \"rational impartial observer\"[5](https://www.tonyohagan.co.uk/academic/pdf/ElicSubjSci.pdf) or a \"rational independent observer\".[9](https://www.journalslibrary.nihr.ac.uk/hta/HTA25370)\n\n**IDEA** (Investigate, Discuss, Estimate, Aggregate) is a two-round version of Delphi with facilitated discussion after the first round.[5](https://www.tonyohagan.co.uk/academic/pdf/ElicSubjSci.pdf) Experts first investigate the questions and clarify their meanings, then give private best-guess point estimates with credible intervals in Round 1.[20](https://doi.org/10.1111/2041-210x.12857) It combines elements of the modified Delphi and Cooke's method, with optional seed-variable weighting at the final mathematical aggregation.[4](https://www.ispor.org/heor-resources/good-practices/article/recommendations-on-the-use-of-structured-expert-elicitation-protocols-for-healthcare-decision-making) On aggregation, three guidelines recommend a linear opinion pool equally weighting all experts; only Cooke and Goossens recommend differential weighting via seed-question performance.[9](https://www.journalslibrary.nihr.ac.uk/hta/HTA25370)\n\n## Applications\n\nStructured expert elicitation has been used in natural hazards, environmental management, food safety, health care, security and counterterrorism, economic and geopolitical forecasting, and risk and reliability analysis.[9](https://www.journalslibrary.nihr.ac.uk/hta/HTA25370) In health technology assessment, a SHELF application elicited longer-term survival of multiple myeloma patients treated with [CAR T-cell therapy](https://www.edgepedia.org/car-t-cell-therapy) through a web-based application, and an EFSA-style modified Delphi elicited distributions from three melanoma experts over two rounds.[4](https://www.ispor.org/heor-resources/good-practices/article/recommendations-on-the-use-of-structured-expert-elicitation-protocols-for-healthcare-decision-making) In natural resource management, the IDEA protocol was applied to estimating 14 future abiotic and biotic events on the [Great Barrier Reef](https://www.edgepedia.org/great-barrier-reef) with 76 participants randomly allocated to eight groups.[22](https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0198468) In that study, aggregating diverse individual judgments into pooled group judgments almost always outperformed individuals, and a modified Delphi approach that removed linguistic ambiguity further improved judgments.[22](https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0198468)\n\n## Limitations and alternatives\n\nEarly criticism has persisted: Sackman concluded that conventional Delphi is basically an unreliable and scientifically unvalidated technique, and Hill and Fowles criticized its reliability, validity, vague questions, and undefined expertise.[17](https://link.springer.com/chapter/10.1007/978-981-96-8357-4_1) Expert elicitation can also be abused in support of public policy decisions, motivating methodological critiques of when it is appropriate.[16](https://pmc.ncbi.nlm.nih.gov/articles/PMC4034232/) A mixed-methods review of guidelines found a lack of consistency across them and a lack of empirical evidence supporting most recommendations.[9](https://www.journalslibrary.nihr.ac.uk/hta/HTA25370) On reporting, an ISPOR task force concluded that many studies using structured elicitation have not documented their approach well and urges transparency;[4](https://www.ispor.org/heor-resources/good-practices/article/recommendations-on-the-use-of-structured-expert-elicitation-protocols-for-healthcare-decision-making) the York practical guide provides a detailed list of items that should be reported.[8](https://www.york.ac.uk/media/che/documents/Structured%20expert%20elicitation%20for%20healthcare%20decision%20making%20A%20practical%20guide.pdf)\n\nPerformance-weighting has an unresolved empirical comparison with simple pooling. Clemen's examination of 14 studies using the classical method concluded that its overall out-of-sample performance appears no better than equal weights, with equal weights showing less variability and better accuracy.[16](https://pmc.ncbi.nlm.nih.gov/articles/PMC4034232/) The validation record is nonetheless substantial: 530 experts assessing 580 calibration variables post-2006,[7](https://www.sciencedirect.com/science/article/pii/S0169207020300959) and 6864 expert uncertainty estimates from 49 classical-model studies scored under five rules (CRPS, Kolmogorov-Smirnov, Cramer-von Mises, Anderson Darling, and chi-square).[15](https://repository.tudelft.nl/file/File_efead35d-dfa4-4723-b516-f152e5ccd3bb) Against alternatives, crowdsourcing trades expertise for a large number of contributors, whereas expert judgmental forecasting uses a small number of independent high-quality forecasts; both enlarge the prediction space for combination.[11](https://pmc.ncbi.nlm.nih.gov/articles/PMC7996321/)\n\nRecent developments center on standardization and new tools. A systematic review notes growing emphasis on standardizing best practices, reflected in the 2023/2024 ISPOR report, alongside web-based elicitation platforms enabling remote expert engagement and increased integration of elicitation into Bayesian analytical frameworks.[23](https://link.springer.com/article/10.1007/s40273-026-01628-x) LLM-based proposals have appeared: Scalable Delphi runs a Delphi-style structured risk estimation process with multiple large language model personas chosen to reflect diverse perspectives, mirroring the diversity sought in human Delphi panels.[24](https://t-lorenz.com/publications/scalable-delphi/scalable-delphi.pdf)\n\n## References\n\n1. [RIVM report 630004001 Expert Elicitation](https://www.rivm.nl/bibliotheek/rapporten/630004001.pdf)\n2. [Eliciting and Analyzing Expert Judgment: A Practical Guide (SIAM)](https://epubs.siam.org/doi/book/10.1137/1.9780898718485)\n3. [The Sheffield Elicitation Framework (SHELF)](https://shelf.sites.sheffield.ac.uk/)\n4. [Recommendations on the Use of Structured Expert Elicitation Protocols for Healthcare Decision Making: A Good Practices Report of an ISPOR Task Force](https://www.ispor.org/heor-resources/good-practices/article/recommendations-on-the-use-of-structured-expert-elicitation-protocols-for-healthcare-decision-making)\n5. [Expert Knowledge Elicitation: Subjective but Scientific (Tony O'Hagan)](https://www.tonyohagan.co.uk/academic/pdf/ElicSubjSci.pdf)\n6. [Expert Elicitation: Using the Classical Model to Validate Experts' Judgments (Review of Environmental Economics and Policy, 2018)](https://ideas.repec.org/a/oup/renvpo/v12y2018i1p113-132..html)\n7. [Expert forecasting with and without uncertainty quantification and weighting: What do the data say? (International Journal of Forecasting; excerpts merged from author-site copy, doi:10.1016/j.ijforecast.2020.06.007)](https://www.sciencedirect.com/science/article/pii/S0169207020300959)\n8. [Structured expert elicitation for healthcare decision making: A practical guide (University of York, CHE)](https://www.york.ac.uk/media/che/documents/Structured%20expert%20elicitation%20for%20healthcare%20decision%20making%20A%20practical%20guide.pdf)\n9. [Developing a reference protocol for structured expert elicitation in health-care decision-making: a mixed-methods study (NIHR HTA; excerpts merged from NCBI Bookshelf copy)](https://www.journalslibrary.nihr.ac.uk/hta/HTA25370)\n10. [Expert elicitation methods for health care decision making (NVTAG presentation)](https://www.nvtag.nl/wp-content/uploads/2025/10/Ijzerman_NVTAG_SEE.pdf)\n11. [Aggregating predictions from experts: a review of statistical methods, experiments, and applications](https://pmc.ncbi.nlm.nih.gov/articles/PMC7996321/)\n12. [Overview, elaboration techniques and their application to a mechanistic model of carbon fluxes (O'Hagan)](http://www.tonyohagan.co.uk/academic/pdf/probspec.pdf)\n13. [Acquisition of Expert Judgment: Examples from Risk Assessment (ASCE, 1992)](https://ascelibrary.org/doi/10.1061/%28ASCE%290733-9402%281992%29118%3A2%28136%29)\n14. [Statistical Methods for Eliciting Probability Distributions (CMU technical report)](https://www.stat.cmu.edu/tr/tr808/tr808.pdf)\n15. [Continuous Distributions and Measures of Statistical Accuracy for Structured Expert Judgment (TU Delft repository)](https://repository.tudelft.nl/file/File_efead35d-dfa4-4723-b516-f152e5ccd3bb)\n16. [Use (and abuse) of expert elicitation in support of decision making for public policy (PNAS)](https://pmc.ncbi.nlm.nih.gov/articles/PMC4034232/)\n17. [Origins and Uses of the Delphi Method (Springer chapter)](https://link.springer.com/chapter/10.1007/978-981-96-8357-4_1)\n18. [EUR report (TU Delft) on expert judgment vs Delphi](https://filelist.tudelft.nl/EWI/Over%20de%20faculteit/Afdelingen/Applied%20Mathematics/uitzoeken/Applied%20Probability/Risk/Download/eur18820.pdf)\n19. [Roger M Cooke (1991). The Classical Model. .](https://doi.org/10.1093/oso/9780195064650.003.0012)\n20. [Victoria Hemming and colleagues (2017). A practical guide to structured expert elicitation using the IDEA protocol. Methods in Ecology and Evolution.](https://doi.org/10.1111/2041-210x.12857)\n21. [Technical report on the probabilistic elicitation of subjective data (UK government)](https://assets.publishing.service.gov.uk/media/5a7576e8ed915d6faf2b3314/The_probabilistic_elicitation_of_subjective_data_WEBSITE.pdf)\n22. [Eliciting improved quantitative judgements using the IDEA protocol: A case study in natural resource management (PLOS One)](https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0198468)\n23. [Applications of Structural Expert Elicitations for Economic Evaluations: A Systematic Review Update (PharmacoEconomics)](https://link.springer.com/article/10.1007/s40273-026-01628-x)\n24. [Scalable Delphi: Large Language Models for Structured Risk Estimation](https://t-lorenz.com/publications/scalable-delphi/scalable-delphi.pdf)\n\n---\n*Topic: Encyclopedia › Physical world and mathematics › General science and scientific practice › Research methods and experimental design › Systematic reviews and evidence synthesis*\n\n*Initially written Sep 29, 2026 · Reviewed: Sep 30, 2026 · Edited: Sep 30, 2026 · Last review: Sep 30, 2026*\n\n*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*\n\nLicense: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license\n", "same_as": [], "url": "https://www.edgepedia.org/expert-elicitation", "markdown_url": "https://www.edgepedia.org/expert-elicitation.md", "license": { "name": "Edgepedia Community License 1.0", "url": "https://www.edgepedia.org/edgepedia/license", "summary": "Free with credit, commercial use included. AI training is open to everyone. For other uses, organizations over USD 100M in revenue or 100M monthly users license separately.", "spdx": "LicenseRef-Edgepedia-Community-1.0" }, "credit": "\"Expert elicitation\", Edgepedia (EdgeChat), https://www.edgepedia.org/expert-elicitation. Edgepedia Community License 1.0.", "credit_md": "\"[Expert elicitation](https://www.edgepedia.org/expert-elicitation)\", Edgepedia (EdgeChat), [https://www.edgepedia.org/expert-elicitation](https://www.edgepedia.org/expert-elicitation). [Edgepedia Community License 1.0](https://www.edgepedia.org/edgepedia/license).", "credit_html": "\"Expert elicitation\", Edgepedia (EdgeChat), https://www.edgepedia.org/expert-elicitation. Edgepedia Community License 1.0.", "speakable": "Expert elicitation is a structured method for synthesizing expert judgments into probability distributions when data are sparse or absent, used in health care, natural hazards, and forecasting." }