Extended reality for biomedicine.

Jie Yuan, Sohail S Hassan, Jiaojiao Wu, Casey R Koger, René R Sevag Packard, Feng Shi, Baowei Fei, Yichen Ding
Author Information
  1. Jie Yuan: Department of Bioengineering, Erik Jonsson School of Engineering and Computer Science, The University of Texas at Dallas, Richardson, TX, United States. ORCID
  2. Sohail S Hassan: Department of Bioengineering, Erik Jonsson School of Engineering and Computer Science, The University of Texas at Dallas, Richardson, TX, United States. ORCID
  3. Jiaojiao Wu: Department of Research and Development, Shanghai United Imaging Intelligence Co., Ltd., Shanghai, China. ORCID
  4. Casey R Koger: Department of Bioengineering, Erik Jonsson School of Engineering and Computer Science, The University of Texas at Dallas, Richardson, TX, United States. ORCID
  5. René R Sevag Packard: Division of Cardiology, Department of Medicine, David Geffen School of Medicine, University of California Los Angeles, Los Angeles, CA, United States. ORCID
  6. Feng Shi: Department of Research and Development, Shanghai United Imaging Intelligence Co., Ltd., Shanghai, China. ORCID
  7. Baowei Fei: Department of Bioengineering, Erik Jonsson School of Engineering and Computer Science, The University of Texas at Dallas, Richardson, TX, United States. ORCID
  8. Yichen Ding: Department of Bioengineering, Erik Jonsson School of Engineering and Computer Science, The University of Texas at Dallas, Richardson, TX, United States. ORCID

Abstract

Extended reality (XR) refers to an umbrella of methods that allows users to be immersed in a three-dimensional (3D) or a 4D (spatial + temporal) virtual environment to different extents, including virtual reality (VR), augmented reality (AR), and mixed reality (MR). While VR allows a user to be fully immersed in a virtual environment, AR and MR overlay virtual objects over the real physical world. The immersion and interaction of XR provide unparalleled opportunities to extend our world beyond conventional lifestyles. While XR has extensive applications in fields such as entertainment and education, its numerous applications in biomedicine create transformative opportunities in both fundamental research and healthcare. This Primer outlines XR technology from instrumentation to software computation methods, delineating the biomedical applications that have been advanced by state-of-the-art techniques. We further describe the technical advances overcoming current limitations in XR and its applications, providing an entry point for professionals and trainees to thrive in this emerging field.

References

  1. Ann R Coll Surg Engl. 2000 Mar;82(2):88-94 [PMID: 10743423]
  2. J Endourol. 2019 Apr;33(4):283-288 [PMID: 30460860]
  3. Curr Pain Headache Rep. 2018 Jun 15;22(8):53 [PMID: 29904806]
  4. Am J Transl Res. 2017 Sep 15;9(9):3867-3880 [PMID: 28979666]
  5. Front Genet. 2021 Oct 28;12:764170 [PMID: 34777482]
  6. Sci Eng Ethics. 2016 Feb;22(1):1-29 [PMID: 25552240]
  7. Lancet. 2011 May 14;377(9778):1693-702 [PMID: 21571152]
  8. Nat Methods. 2020 Nov;17(11):1100-1102 [PMID: 32958921]
  9. Front Hum Neurosci. 2019 Jul 11;13:244 [PMID: 31354460]
  10. Sci Rep. 2022 Jul 30;12(1):13137 [PMID: 35907935]
  11. J Clin Neurosci. 2017 Jan;35:1-4 [PMID: 28137372]
  12. Plast Reconstr Surg Glob Open. 2015 Aug 10;3(8):e479 [PMID: 26495192]
  13. JACC Basic Transl Sci. 2018 Jun 25;3(3):420-430 [PMID: 30062228]
  14. Nat Methods. 2017 Oct;14(10):995-1002 [PMID: 28825703]
  15. Nat Commun. 2021 Sep 29;12(1):5712 [PMID: 34588436]
  16. J Neuroeng Rehabil. 2016 Aug 09;13(1):70 [PMID: 27503215]
  17. Expert Rev Neurother. 2008 Feb;8(2):215-33 [PMID: 18271709]
  18. Radiographics. 1992 Jan;12(1):127-9 [PMID: 1734458]
  19. J Oral Maxillofac Surg. 2018 May;76(5):1065-1072 [PMID: 29104028]
  20. Comput Methods Programs Biomed. 2020 Jul;190:105385 [PMID: 32062090]
  21. Comput Methods Programs Biomed. 2020 Apr;187:105099 [PMID: 31601442]
  22. Eur J Psychotraumatol. 2019 Aug 19;10(1):1654782 [PMID: 31489138]
  23. Front Hum Neurosci. 2019 Jun 19;13:210 [PMID: 31275126]
  24. IEEE Trans Vis Comput Graph. 2018 Apr;24(4):1496-1505 [PMID: 29543167]
  25. IEEE Trans Pattern Anal Mach Intell. 2004 Nov;26(11):1452-8 [PMID: 15521493]
  26. Sensors (Basel). 2017 Oct 24;17(10): [PMID: 29064457]
  27. Ann Biomed Eng. 2018 Dec;46(12):2177-2188 [PMID: 30112710]
  28. Nat Commun. 2022 Nov 2;13(1):6566 [PMID: 36323677]
  29. Sci Rep. 2018 Sep 12;8(1):13657 [PMID: 30209261]
  30. Nat Commun. 2018 Nov 1;9(1):4562 [PMID: 30385830]
  31. Light Sci Appl. 2021 Oct 25;10(1):216 [PMID: 34697292]
  32. J Cardiothorac Vasc Anesth. 2019 Mar;33(3):732-741 [PMID: 30340952]
  33. Med Image Anal. 2017 Jan;35:633-654 [PMID: 27744253]
  34. Nature. 2019 Nov;575(7783):473-479 [PMID: 31748722]
  35. Plast Reconstr Surg Glob Open. 2020 Jun 09;8(6):e2909 [PMID: 32766059]
  36. Nat Commun. 2022 Jul 15;13(1):4128 [PMID: 35840566]
  37. Neurosurg Focus. 2021 Aug;51(2):E21 [PMID: 34333463]
  38. PLoS Comput Biol. 2020 Mar 31;16(3):e1007747 [PMID: 32231351]
  39. Protein Sci. 2018 Jan;27(1):95-102 [PMID: 28815765]
  40. Nat Methods. 2020 Mar;17(3):343-351 [PMID: 32123394]
  41. J Mol Biol. 2018 Oct 19;430(21):4028-4035 [PMID: 29949752]
  42. JAMA Netw Open. 2020 Dec 1;3(12):e2031217 [PMID: 33369660]
  43. Psychiatriki. 2016 Oct-Dec;27(4):276-286 [PMID: 28114091]
  44. JMIR Mhealth Uhealth. 2019 Apr 26;7(4):e10967 [PMID: 31025950]
  45. IEEE Trans Med Imaging. 2012 Mar;31(3):563-75 [PMID: 22042148]
  46. JAMA Otolaryngol Head Neck Surg. 2020 Mar 1;146(3):286-290 [PMID: 31917412]
  47. Nat Neurosci. 2021 Aug;24(8):1065-1070 [PMID: 34183867]
  48. IEEE Trans Vis Comput Graph. 2021 May;27(5):2545-2554 [PMID: 33755568]
  49. Commun Biol. 2022 Feb 14;5(1):130 [PMID: 35165405]
  50. Int J Technol Assess Health Care. 2019;35(5):373-378 [PMID: 31452469]
  51. Sci Rep. 2019 Apr 29;9(1):6616 [PMID: 31036828]
  52. Light Sci Appl. 2022 May 30;11(1):161 [PMID: 35637183]
  53. Virtual Augment Mixed Real (2020). 2020;12191:341-356 [PMID: 34327520]
  54. Arch Otolaryngol Head Neck Surg. 1999 Nov;125(11):1275-8 [PMID: 10555704]
  55. Nat Commun. 2019 Aug 2;10(1):3474 [PMID: 31375678]
  56. IEEE Trans Image Process. 2005 Oct;14(10):1570-82 [PMID: 16238062]
  57. Comput Methods Programs Biomed. 2018 Jul;161:39-44 [PMID: 29852966]
  58. Ophthalmology. 2022 Jan;129(1):77-85 [PMID: 34534556]
  59. Protein Sci. 2021 Jan;30(1):70-82 [PMID: 32881101]
  60. J Clin Psychiatry. 2007 Nov;68(11):1639-47 [PMID: 18052556]
  61. Nat Methods. 2020 Nov;17(11):1097-1099 [PMID: 33046895]
  62. Front Psychol. 2022 Mar 21;13:747456 [PMID: 35386898]
  63. Eur Spine J. 2012 Jan;21(1):23-30 [PMID: 21842236]
  64. J Biomed Opt. 2017 Jun 1;22(6):65005 [PMID: 28657107]
  65. Nurse Educ Today. 2021 Jun;101:104868 [PMID: 33798987]
  66. Arch Plast Surg. 2017 May;44(3):179-187 [PMID: 28573091]
  67. JACC Clin Electrophysiol. 2020 Aug;6(8):1023-1025 [PMID: 32819517]
  68. BMC Bioinformatics. 2019 Feb 15;20(1):78 [PMID: 30767777]
  69. J Neuroeng Rehabil. 2018 Mar 27;15(1):29 [PMID: 29587853]
  70. Cell. 2020 Dec 23;183(7):2041-2042 [PMID: 33357402]
  71. Trials. 2016 Feb 02;17:60 [PMID: 26833396]
  72. Minim Invasive Ther Allied Technol. 2010;19(1):35-41 [PMID: 20095896]
  73. Vis Comput Ind Biomed Art. 2020 Sep 16;3:21 [PMID: 32954214]
  74. Surg Innov. 2020 Feb;27(1):88-100 [PMID: 31514682]
  75. J Am Acad Orthop Surg Glob Res Rev. 2022 Apr 01;6(4): [PMID: 35467580]
  76. Plast Reconstr Surg. 2017 Nov;140(5):1066-1070 [PMID: 29068946]
  77. Can J Neurol Sci. 1997 Nov;24(4):302-6 [PMID: 9398976]
  78. Spine J. 2017 Sep;17(9):1215-1229 [PMID: 28428081]
  79. Med Image Anal. 2021 Oct;73:102193 [PMID: 34371440]
  80. Nat Rev Neurol. 2020 Aug;16(8):409-425 [PMID: 32591756]
  81. IEEE Comput Graph Appl. 2012 Sep-Oct;32(5):22-8 [PMID: 24806984]
  82. Surg Oncol Clin N Am. 2019 Jan;28(1):31-44 [PMID: 30414680]
  83. Med Image Anal. 2017 Dec;42:60-88 [PMID: 28778026]
  84. IEEE Trans Pattern Anal Mach Intell. 2022 Jul;44(7):3523-3542 [PMID: 33596172]
  85. Med Image Anal. 2022 Apr;77:102361 [PMID: 35168103]
  86. IEEE Trans Vis Comput Graph. 2017 Apr;23(4):1302-1311 [PMID: 28129166]
  87. IEEE Comput Graph Appl. 2005 Nov-Dec;25(6):6-9 [PMID: 16315470]
  88. Nat Commun. 2021 Apr 23;12(1):2432 [PMID: 33893283]
  89. JCI Insight. 2017 Nov 16;2(22): [PMID: 29202458]
  90. J Exp Orthop. 2021 Jul 21;8(1):52 [PMID: 34287721]
  91. Nat Rev Methods Primers. 2021;1: [PMID: 35663461]
  92. J Eye Mov Res. 2019 Apr 05;12(1): [PMID: 33828721]
  93. ASAIO J. 2021 Dec 1;67(12):e211-e215 [PMID: 34261876]
  94. Cell Rep Med. 2021 Jul 21;2(7):100348 [PMID: 34337564]
  95. Stud Health Technol Inform. 2012;173:85-91 [PMID: 22356963]
  96. Cochrane Database Syst Rev. 2015 Sep 09;(9):CD010198 [PMID: 26352008]
  97. Int J Environ Res Public Health. 2022 Mar 29;19(7): [PMID: 35409751]
  98. Eur Heart J Cardiovasc Imaging. 2016 Aug;17(8):845-9 [PMID: 27283456]
  99. J Clin Med. 2019 Mar 05;8(3): [PMID: 30841509]
  100. IEEE J Transl Eng Health Med. 2020 Jul 03;8:1900810 [PMID: 32742821]
  101. Spine (Phila Pa 1976). 2022 Jun 15;47(12):865-872 [PMID: 35132049]
  102. Comput Aided Surg. 2005 Mar;10(2):93-9 [PMID: 16298920]
  103. J Clin Neurosci. 2019 Apr;62:14-20 [PMID: 30642663]
  104. Int J Med Robot. 2020 Apr;16(2):e2067 [PMID: 31867864]
  105. Sci Rep. 2020 Jan 20;10(1):707 [PMID: 31959895]
  106. Cell. 2020 Dec 10;183(6):1600-1616.e25 [PMID: 33248024]
  107. J Digit Imaging. 2020 Feb;33(1):111-120 [PMID: 31396777]
  108. J Behav Ther Exp Psychiatry. 2008 Sep;39(3):250-61 [PMID: 17720136]
  109. Int J Med Robot. 2012 Mar;8(1):45-56 [PMID: 22069247]
  110. J Cardiothorac Vasc Anesth. 2019 Mar;33(3):742-743 [PMID: 30292388]
  111. Int Forum Allergy Rhinol. 2017 Oct;7(10):1006-1013 [PMID: 28719062]
  112. JACC Cardiovasc Imaging. 2022 Mar;15(3):519-532 [PMID: 34656478]
  113. IEEE Trans Med Imaging. 2000 Jan;19(1):62-9 [PMID: 10782620]
  114. Plast Reconstr Surg. 2019 Feb;143(2):647-651 [PMID: 30688914]
  115. Neurosurgery. 2012 Sep;71(1 Suppl Operative):32-42 [PMID: 22233921]
  116. J Digit Imaging. 2011 Aug;24(4):640-64 [PMID: 20714917]
  117. Neural Netw. 2015 Jan;61:85-117 [PMID: 25462637]
  118. Surg Endosc. 2014 Sep;28(9):2569-76 [PMID: 24671353]
  119. J Healthc Eng. 2021 Feb 9;2021:6611196 [PMID: 33628402]

Grants

  1. I01 BX004558/BLRD VA
  2. R01 HL140325/NHLBI NIH HHS
  3. R01 CA156775/NCI NIH HHS
  4. UL1 TR001881/NCATS NIH HHS
  5. R01 CA204254/NCI NIH HHS
  6. R21 CA231911/NCI NIH HHS
  7. R56 HL158569/NHLBI NIH HHS
  8. R00 HL148493/NHLBI NIH HHS

Word Cloud

Created with Highcharts 10.0.0realityXRvirtualapplicationsExtendedmethodsallowsimmersedenvironmentVRARMRworldopportunitiesbiomedicinerefersumbrellausersthree-dimensional3D4Dspatial+temporaldifferentextentsincludingaugmentedmixeduserfullyoverlayobjectsrealphysicalimmersioninteractionprovideunparalleledextendbeyondconventionallifestylesextensivefieldsentertainmenteducationnumerouscreatetransformativefundamentalresearchhealthcarePrimeroutlinestechnologyinstrumentationsoftwarecomputationdelineatingbiomedicaladvancedstate-of-the-arttechniquesdescribetechnicaladvancesovercomingcurrentlimitationsprovidingentrypointprofessionalstraineesthriveemergingfield

Similar Articles

Cited By (9)