OpenLB
Open Library of Bioscience
Advanced Search
Medicine
Feb 17, 2023;102 (7): e32829.

Use of Delphi in health sciences research: A narrative review.

Zhida Shang
Author Information
  1. Zhida Shang: Faculty of Medicine and Health Sciences, McGill University, Montreal, QC, Canada. ORCID
PMID: 36800594 DOI: 10.1097/MD.0000000000032829

Abstract

The use of the Delphi technique is prevalent across health sciences research, and it is used to identify priorities, reach consensus on issues of importance and establish clinical guidelines. Thus, as a form of expert opinion research, it can address fundamental questions present in healthcare. However, there is little guidance on how to conduct them, resulting in heterogenous Delphi studies and methodological confusion. Therefore, the purpose of this review is to introduce the use of the Delphi method, assess the application of the Delphi technique within health sciences research, discuss areas of methodological uncertainty and propose recommendations. Advantages of the use of Delphi include anonymity, controlled feedback, flexibility for the choice of statistical analysis, and the ability to gather participants from geographically diverse areas. Areas of methodological uncertainty worthy of further discussion broadly include experts and data management. For experts, the definition and number of participants remain issues of contention, while there are ongoing difficulties with expert selection and retention. For data management, there are issues with data collection, defining consensus and methods of data analysis, such as percent agreement, central tendency, measures of dispersion, and inferential statistics. Overall, the use of Delphi addresses important issues present in health sciences research, but methodological issues remain. It is likely that the aggregation of future Delphi studies will eventually pave the way for more comprehensive reporting guidelines and subsequent methodological clarity.

References

  1. Watson R. Quantitative research. Nurs Stand. 2015;29:44–8.
  2. Polit DF, Beck CT. Nursing Research: Generating and Assessing Evidence for Nursing Practice. Lippincott Williams & Wilkins; 2008.
  3. Sidani S. Health Intervention Research: Understanding Research Design and Methods. London: SAGE Publications; 2014.
  4. Curtis E, Comiskey C, Dempsey O. Correlational research: Importance and use in nursing and health research. Nurs Res. 2015;6:20–5.
  5. Coughlan M, Cronin P, Ryan F. Survey research: process and limitations. Int J Ther Rehabil. 2009;16:9–15.
  6. Privitera GJ. Research Methods for the Behavioral Sciences. SAGE Publications; 2018.
  7. Story DA, Tait AR. Survey research. Anesthesiology. 2019;130:192–202.
  8. Cope DG. Using electronic surveys in nursing research. Oncol Nurs Forum. 2014.
  9. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32:1008–15.
  10. Kalaian S, Kasim RM. Terminating sequential Delphi survey data collection. Pract Assess Res. 2012;17:1–11.
  11. Shariff N. Utilizing the Delphi survey approach: a review. J Nurs Care Qual. 2015;4:246–51.
  12. Baethge C, Goldbeck-Wood S, Mertens S. SANRA – a scale for the quality assessment of narrative review articles. Res Integr Peer Rev. 2019;4:5.
  13. Dalkey NC. Analysis of the Future: the Delphi Method. Santa Monica. California: RAND Corporation; 1967.
  14. McKenna HP. The Delphi technique: a worthwhile research approach for nursing?. J Adv Nurs. 1994;19:1221–5.
  15. Rowe G, Wright G. The Delphi technique as a forecasting tool: issues and analysis. Int J Forecast. 1999;15:353–75.
  16. Falzarano M, Zipp GP. Seeking consensus through the use of the Delphi technique in health sciences research. J Allied Health. 2013;42:99–105.
  17. McPherson S, Reese C, Wendler MC. Methodology update: Delphi studies. Nurs Res. 2018;67:404–10.
  18. Hsu C-C, Sandford BA. The Delphi technique: making sense of consensus. Pract Assess Res. 2007;12:10.
  19. Thangaratinam S, Redman CW. The Delphi technique. Obstet Gynaecol. 2005;7:120–5.
  20. Humphrey-Murto S, Varpio L, Gonsalves C, et al. Using consensus group methods such as Delphi and Nominal Group in medical education research. Med Teach. 2017;39:14–9.
  21. Harvey N, Holmes CA. Nominal group technique: an effective method for obtaining group consensus. Int J Nurs Pract. 2012;18:188–94.
  22. McMillan SS, King M, Tully MP. How to use the nominal group and Delphi techniques. Int J Clin Pharm. 2016;38:655–62.
  23. Mannes AE, Soll JB, Larrick RP. The wisdom of select crowds. J Pers Soc Psychol. 2014;107:276–99.
  24. Surowiecki J. The Wisdom of Crowds. Anchor; 2005.
  25. Jorm AF. Using the Delphi expert consensus method in mental health research. Aust N Z J Psychiatry. 2015;49:887–97.
  26. Delbecq AL, Van de Ven AH, Gustafson DH. Group Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes. Scott, Foresman; 1975.
  27. Bradley F, Schafheutle EI, Willis SC, Noyce PR. Changes to supervision in community pharmacy: pharmacist and pharmacy support staff views. Health Soc Care Community. 2013;21:644–54.
  28. Humphrey-Murto S, Varpio L, Wood TJ, et al. The use of the Delphi and other consensus group methods in medical education research: a review. Acad Med. 2017;92:1491–8.
  29. Fitch K, Bernstein SJ, Aguilar MD, et al. The RAND/UCLA Appropriateness Method User’s Manual. CA: Rand Corp Santa Monica; 2001.
  30. Foth T, Efstathiou N, Vanderspank-Wright B, et al. The use of Delphi and Nominal Group Technique in nursing education: a review. Int J Nurs. 2016;60:112–20.
  31. Boulkedid R, Abdoul H, Loustau M, et al. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One. 2011;6:e20476.
  32. Diamond IR, Grant RC, Feldman BM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol. 2014;67:401–9.
  33. Varndell W, Fry M, Lutze M, et al. Use of the Delphi method to generate guidance in emergency nursing practice: a systematic review. Int Emerg Nurs. 2020;100867.
  34. Asselin M, Harper M. Revisiting the Delphi technique: implications for nursing professional development. J Nurses Prof Dev. 2014;30:11–5.
  35. Trevelyan EG, Robinson PN. Delphi methodology in health research: how to do it?. Eur J Integr Med. 2015;7:423–8.
  36. Morisset J, Johannson KA, Jones KD, et al. Identification of diagnostic criteria for chronic hypersensitivity pneumonitis. An international modified Delphi survey. Am J Respir Crit. 2018;197:1036–44.
  37. Feo R, Conroy T, Jangland E, et al. Towards a standardised definition for fundamental care: a modified Delphi study. J Clin Nurs. 2018;27:2285–99.
  38. Toronto C. Considerations when conducting e-Delphi research: a case study. Nurse Res. 2017;25:10–5.
  39. Turoff M. The design of a policy Delphi. Technol Forecast Soc Change. 1970;2:149–71.
  40. Turoff M, Linstone HA. The Delphi method-techniques and applications. 2002. Available at: http://www.foresight.pl/assets/downloads/publications/Turoff_Linstone.pdf .
  41. de Loë RC, Melnychuk N, Murray D, et al. Advancing the state of policy Delphi practice: a systematic review evaluating methodological evolution, innovation, and opportunities. Technol Forecast Soc Change. 2016;104:78–88.
  42. Donohoe H, Stellefson M, Tennant B. Advantages and limitations of the e-Delphi technique: implications for health education researchers. Am J Health Educ. 2012;43:38–46.
  43. Gill FJ, Leslie GD, Grech C, et al. Using a web-based survey tool to undertake a Delphi study: application for nurse education research. Nurse Educ Today. 2013;33:1322–8.
  44. Ingham-Broomfield JR. A nurses’ guide to the hierarchy of research designs and evidence. Aust J Adv Nurs. 2016;33:38–43.
  45. Schmalz U, Spinler S, Ringbeck J. Lessons learned from a two-round Delphi-based scenario study. MethodsX. 2021;8:101179.
  46. Isla D, González-Rojas N, Nieves D, et al. Treatment patterns, use of resources, and costs of advanced non-small-cell lung cancer patients in Spain: results from a Delphi panel. Clin Transl Oncol. 2011;13:460–71.
  47. Ab Latif R, Dahlan A, Mulud ZA, et al. The Delphi technique as a method to obtain consensus in health care education research. EIMJ. 2017;9.
  48. Schneider Z, Whitehead D, LoBiondo-Wood G, et al. Nursing and Midwifery Research: Methods and Appraisal for Evidence Based Practice. Elsevier; 2016.
  49. Devaney L, Henchion M. Who is a Delphi “expert?” Reflections on a bioeconomy expert selection procedure from Ireland. Futures. 2018;99:45–55.
  50. Keeney S, Hasson F, McKenna HP. A critical review of the Delphi technique as a research methodology for nursing. Int J Nurs. 2001;38:195–200.
  51. Manzano-García G, Ayala J-C. Insufficiently studied factors related to burnout in nursing: results from an e-Delphi study. PLoS One. 2017;12:e0175352.
  52. Cheung KL, de Ruijter D, Hiligsmann M, et al. Exploring consensus on how to measure smoking cessation. A Delphi study. BMC Public Health. 2017;17.
  53. Salgado TM, Fedrigon A, Omichinski DR, et al. Identifying medication management smartphone app features suitable for young adults with developmental disabilities: Delphi consensus study. JMIR mHealth uHealth. 2018;6.
  54. Grisham T. The Delphi technique: a method for testing complex and multifaceted topics. Int J Manag. 2009.
  55. Bloor M, Sampson H, Baker S, et al. Useful but no Oracle: reflections on the use of a Delphi group in a multi-methods policy research study. Qual Res. 2015;15:57–70.
  56. Cuhls K, Kuwahara T. Outlook for Japanese and German Future Technology: Comparing Technology Forecast Surveys. Springer Science & Business Media; 2012.
  57. Paul CL. A modified delphi approach to a new card sorting methodology. J Usabil Stud. 2008;4:7–30.
  58. Akins RB, Tolson H, Cole BR. Stability of response characteristics of a Delphi panel: application of bootstrap data expansion. BMC Med Res Methodol. 2005;5:37.
  59. Powell C. The Delphi technique: myths and realities. J Adv Nurs. 2003;41:376–82.
  60. Vernon W. The Delphi technique: a review. Int J Ther Rehabil. 2009;16:69–76.
  61. Skulmoski GJ, Hartman FT, Krahn J. The Delphi method for graduate research. J Inf Technol. 2007;6:1–21.
  62. Keeney S, McKenna H, Hasson F. The Delphi Technique in Nursing and Health Research. John Wiley & Sons; 2011.
  63. Hejblum G, Ioos V, Vibert J-F, et al. A web-based Delphi study on the indications of chest radiographs for patients in ICUs. Chest. 2008;133:1107–12.
  64. Stedman RC, Connelly NA, Heberlein TA, et al. The end of the (research) world as we know it?. Understanding and coping with declining response rates to mail surveys. Soc Nat Resour. 2019;32:1139–54.
  65. Campbell RM, Venn TJ, Anderson NM. Cost and performance tradeoffs between mail and internet survey modes in a nonmarket valuation study. J Environ Manage. 2018;210:316–27.
  66. Giannarou L, Zervas E. Using Delphi technique to build consensus in practice. Int J Bus Sci Appl Manag. 2014;9:65–82.
  67. Chyung SY, Roberts K, Swanson I, et al. Evidence-based survey design: the use of a midpoint on the Likert scale. Perform Improv. 2017;56:15–23.
  68. Johns R. One size doesn’t fit all: selecting response scales for attitude items. J Elect Public Opin Parties. 2005;15:237–64.
  69. Kulas JT, Stachowski AA. Respondent rationale for neither agreeing nor disagreeing: person and item contributors to middle category endorsement intent on Likert personality indicators. J Res Pers. 2013;47:254–62.
  70. Gargon E, Crew R, Burnside G, et al. Higher number of items associated with significantly lower response rates in COS Delphi surveys. J Clin Epidemiol. 2019;108:110–20.
  71. Becker GE, Roberts T. Do we agree?. Using a Delphi technique to develop consensus on skills of hand expression. J Hum Lact. 2009;25:220–5.
  72. Grant S, Booth M, Khodyakov D. Lack of preregistered analysis plans allows unacceptable data mining for and selective reporting of consensus in Delphi studies. J Clin Epidemiol. 2018;99:96–105.
  73. Keeney S, Hasson F, McKenna H. Consulting the oracle: ten lessons from using the Delphi technique in nursing research. J Adv Nurs. 2006;53:205–12.
  74. von der Gracht HA. Consensus measurement in Delphi studies: review and implications for future quality assurance. Technol Forecast Soc Change. 2012;79:1525–36.
  75. Holey EA, Feeley JL, Dixon J, et al. An exploration of the use of simple statistics to measure consensus and stability in Delphi studies. BMC Med Res Methodol. 2007;7:52.
  76. Gnatzy T, Warth J, von der Gracht H, et al. Validating an innovative real-time Delphi approach - a methodological comparison between real-time and conventional Delphi studies. Technol Forecast Soc Change. 2011;78:1681–94.
  77. Birko S, Dove ES, Özdemir V. Evaluation of nine consensus indices in Delphi foresight research and their dependency on Delphi survey characteristics: a simulation study and debate on Delphi design and interpretation. PLoS One. 2015;10:e0135162.
  78. Dajani JS, Sincoff MZ, Talley WK. Stability and agreement criteria for the termination of Delphi studies. Technol Forecast Soc Change. 1979;13:83–90.
  79. Green B, Jones M, Hughes D, et al. Applying the Delphi technique in a study of GPs’ information requirements. Health Soc Care Community. 1999;7:198–205.
  80. Griffiths SV, Conway DH, Javier BF, et al. What are the optimum components in a care bundle aimed at reducing post-operative pulmonary complications in high-risk patients?. Perioper Med. 2018;7:7.
  81. Huijben JA, Wiegers EJA, de Keizer NF, et al. Development of a quality indicator set to measure and improve quality of ICU care for patients with traumatic brain injury. Crit Care. 2019;23:95.

MeSH Term

Humans
Delphi Technique
Consensus
Medicine
Research Design
Review Journal Article
Article has an altmetric score of 7

Word Cloud

Created with Highcharts 10.0.0DelphiissuesmethodologicalusehealthsciencesresearchdatatechniqueconsensusguidelinesexpertpresentstudiesreviewareasuncertaintyincludeanalysisparticipantsexpertsmanagementremainprevalentacrossusedidentifyprioritiesreachimportanceestablishclinicalThusformopinioncanaddressfundamentalquestionshealthcareHoweverlittleguidanceconductresultingheterogenousconfusionThereforepurposeintroducemethodassessapplicationwithindiscussproposerecommendationsAdvantagesanonymitycontrolledfeedbackflexibilitychoicestatisticalabilitygathergeographicallydiverseAreasworthydiscussionbroadlydefinitionnumbercontentionongoingdifficultiesselectionretentioncollectiondefiningmethodspercentagreementcentraltendencymeasuresdispersioninferentialstatisticsOveralladdressesimportantlikelyaggregationfuturewilleventuallypavewaycomprehensivereportingsubsequentclarityUseresearch:narrative

Similar Articles

Cited By (60)