The Impact of Virtual Humans on Psychosomatic Medicine.

Kate Loveys, Mark Sagar, Michael Antoni, Elizabeth Broadbent
Author Information
  1. Kate Loveys: From the Department of Psychological Medicine (Loveys, Broadbent), The University of Auckland; Soul Machines Ltd (Loveys, Sagar); Auckland Bioengineering Institute (Sagar), The University of Auckland, Auckland, New Zealand; and Center for Psycho-Oncology Research (Antoni), University of Miami, Coral Gables, Florida. ORCID

Abstract

OBJECTIVE: Virtual humans are likely to enhance the delivery of health care over the next decade. Virtual humans are artificially intelligent computer agents with hyperrealistic, autonomously animated embodiments based on affective computing techniques. Virtual humans could be programmed to screen for health conditions, triage patients, and deliver health interventions, with appropriate facial expressions and body gestures, functioning as a supplement to human care. This article provides a perspective on the implications of virtual humans for behavioral and psychosomatic medicine, and health psychology.
METHODS: A narrative review was conducted to integrate observations and findings from research on virtual humans from 91 articles in this multidisciplinary area.
RESULTS: Virtual humans can be used for multimodal behavior analysis of patients, individualized tailoring of interventions, and detection of changes of psychological and behavioral measures over time. Virtual humans can also pair the scalability of a website with the interactivity and relational skills of a human tele-therapist. Research is beginning to show the acceptability, feasibility, and preliminary effectiveness of virtual humans in a range of populations. Virtual humans can be easily tailored in terms of their appearance, voice, and language, and may be adapted to fit the characteristics of a patient population or hard-to-reach groups. If co-designed with these communities, virtual humans may help to promote health care engagement and improve outcomes.
CONCLUSIONS: Virtual humans can engage and motivate patients, and deliver personalized psychological and behavioral health care. This article provides an overview of the potential impact of virtual humans on psychosomatic medicine and discusses ethical implications.

References

  1. The World Bank. Universal Health Coverage Data [Internet] . Washington, DC: The World Bank; 2021. Available at: https://datatopics.worldbank.org/universal-health-coverage/ . Accessed March 27, 2022.
  2. Liu JX, Goryakin Y, Maeda A, Bruckner T, Scheffler R. Global health workforce labor market projections for 2030. Hum Resour Health 2017;15:11.
  3. Drennan VM, Ross F. Global nurse shortages: the facts, the impact and action for change. Br Med Bull 2019;130:25–37.
  4. World Health Organization. Universal Health Coverage (UHC) [Internet] . Geneva, Switzerland: World Health Organization; 2021. Available at: https://www.who.int/news-room/fact-sheets/detail/universal-health-coverage-(uhc) . Accessed March 27, 2022.
  5. Batterham RW, Hawkins M, Collins PA, Buchbinder R, Osborne RH. Health literacy: applying current concepts to improve health services and reduce health inequalities. Public Health 2016;132:3–12.
  6. Allen H, Wright BJ, Harding K, Broffman L. The role of stigma in access to health care for the poor. Milbank Q 2014;92:289–318.
  7. World Health Organization. Tracking Universal Health Coverage: 2021 Global Monitoring Report [Internet] . Geneva, Switzerland: World Health Organization; 2021. Available at: https://cdn.who.int/media/docs/default-source/world-health-data-platform/events/tracking-universal-health-coverage-2021-global-monitoring-report_uhc-day.pdf?sfvrsn=fd5c65c6_5&download=true . Accessed March 27, 2022.
  8. Argulian E. Anticipating the “second wave” of health care strain in the COVID-19 pandemic. JACC Case Rep 2020;2:845–6.
  9. Cassell J. Embodied conversational agents: representation and intelligence in user interfaces. AI Magazine 2001;22:67.
  10. Bickmore TW, Picard RW. Establishing and maintaining long-term human-computer relationships. ACM Trans Comput Hum Interact 20051;12:293–327.
  11. Brave S, Nass C, Hutchinson K. Computers that care: investigating the effects of orientation of emotion exhibited by an embodied computer agent. Int J Hum Comput Stud. 2005;62:161–78.
  12. Bott N, Wexler S, Drury L, Pollak C, Wang V, Scher K, et al. A protocol-driven, bedside digital conversational agent to support nurse teams and mitigate risks of hospitalization in older adults: case control pre-post study. J Med Internet Res 2019;21:e13440.
  13. Sagar M, Seymour M, Henderson A. Creating connection with autonomous facial animation. Commun ACM 2016;59:82–91.
  14. Wu J, Ghosh S, Chollet M, Ly S, Mozgai S, Scherer S. Nadia: neural network driven virtual human conversation agents. In: Proceedings of the 18th International Conference on Intelligent Virtual Agents, IVA 2018. New York, NY: ACM; 2018;173–8. Available at: https://dl.acm.org/doi/abs/10.1145/3267851.3267860 . Accessed July 24, 2023. [DOI: 10.1145/3267851.3267860]
  15. DiPaola S, Yalçin ÖN. A multi-layer artificial intelligence and sensing based affective conversational embodied agent. In: 2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW). Cambridge, United Kingdom: IEEE; 2019;91–2. Available at: https://ieeexplore.ieee.org/abstract/document/8925291 . Accessed July 24, 2023.
  16. Chattopadhyay D, Ma T, Sharifi H, Martyn-Nemeth P. Computer-controlled virtual humans in patient-facing systems: systematic review and meta-analysis. J Med Internet Res 2020;22:e18839.
  17. Mystakidis S. Metaverse. Encyclopedia 2022;2:486–97.
  18. Park SM, Kim YG. A metaverse: taxonomy, components, applications, and open challenges. IEEE Access 2022;10:4209–51.
  19. Thomason J. MetaHealth—how will the metaverse change health care? J Metaverse 2021;1:13–6.
  20. Picard RW. Affective Computing . Cambridge, MA: MIT Press; 2000.
  21. Poria S, Cambria E, Bajpai R, Hussain A. A review of affective computing: from unimodal analysis to multimodal fusion. Inform Fusion 2017;37:98–125.
  22. Umematsu T, Sano A, Taylor S, Tsujikawa M, Picard RW. Forecasting stress, mood, and health from daytime physiology in office workers and students. Annu Int Conf IEEE Eng Med Biol Soc 2020;2020:5953–7.
  23. Morency LP, Mihalcea R, Doshi P. Towards multimodal sentiment analysis: harvesting opinions from the web. In: Proceedings of the 13th International Conference on Multimodal Interfaces. Alicante, Spain. New York, NY: Association for Computing Machinery (ACM); 2011:169–76.
  24. Loveys K, Crutchley P, Wyatt E, Coppersmith G. Small but mighty: affective micropatterns for quantifying mental health from social media language. In: Proceedings of the Fourth Workshop on Computational Linguistics and Clinical Psychology—From Linguistic Signal to Clinical Reality. Vancouver, Canada. Stroudsburg, PA: Association for Computational Linguistics (ACL); 2017:85–95.
  25. Eichstaedt JC, Weidman AC. Tracking fluctuations in psychological states using social media language: a case study of weekly emotion. Eur J Pers 2020;34:845–58.
  26. Cavallo F, Semeraro F, Fiorini L, Magyar G, Sinčák P, Dario P. Emotion modelling for social robotics applications: a review. J Bionic Eng 2018;15:185–203.
  27. Marci CD, Ham J, Moran E, Orr SP. Physiologic correlates of perceived therapist empathy and social-emotional process during psychotherapy. J Nerv Ment Dis 2007;195:103–11. doi:10.1097/01.nmd.0000253731.71025.fc. [DOI: 10.1097/01.nmd.0000253731.71025.fc]
  28. Coulehan JL, Platt FW, Egener B, Frankel R, Lin CT, Lown B, et al. ‘Let me see if I have this right…’ Words that help build empathy. Ann Intern Med 2000;135:221–7. doi:10.7326/0003-4819-135-3-200108070-00022. [DOI: 10.7326/0003-4819-135-3-200108070-00022]
  29. Zachariae R, Pedersen CG, Jensen AB, Ehrnrooth E, Rossen PB, von der Maase H. Association of perceived physician communication style with patient satisfaction, distress, cancer-related self-efficacy, and perceived control over the disease. Br J Cancer 2003;88:658–65. doi:10.1038/sj.bjc.6600798. [DOI: 10.1038/sj.bjc.6600798]
  30. Johanson DL, Ahn HS, Broadbent E. Improving interactions with healthcare robots: a review of communication behaviours in social and healthcare contexts. Int J Soc Robot 2021;13:1835–50.
  31. Broadbent E, Johanson D, Shah J. A new model to enhance robot-patient communication: applying insights from the medical world. In: Salichs MA, Broadbent E, He H, Wagner AR, editors. Social Robotics—10th International Conference, ICSR 2018, Qingdao, China. LNAI 11357. 2018:308–17. doi:10.1007/978-3-030-05204-1_30.
  32. Rizzo AS, Shilling R. Clinical virtual reality tools to advance the prevention, assessment, and treatment of PTSD. Eur J Psychotraumatol 2017;8(suppl 5):1414560.
  33. Rathbone AL, Prescott J. The use of mobile apps and sms messaging as physical and mental health interventions: systematic review. J Med Internet Res 2017;19:e295. doi:10.2196/jmir.7740. [DOI: 10.2196/jmir.7740]
  34. Borghouts J, Eikey E, Mark G, De Leon C, Schueller SM, Schneider M, et al. Barriers to and facilitators of user engagement with digital mental health interventions: systematic review. J Med Internet Res 2021;23:e24387.
  35. Rizzo AA, Lucas GM, Gratch J, Stratou G, Morency LP, Chavez K, et al. Automatic behavior analysis during a clinical interview with a virtual human. MMVR 2016;220:316–22.
  36. Broadbent E, Loveys K, Sagar M, Donkin L, Antoni M. Using digital humans to deliver stress management to patients with cancer. Int J Behav Med 2021;28(Suppl 1):S147–8.
  37. Kocaballi AB, Berkovsky S, Quiroz JC, Laranjo L, Tong HL, Rezazadegan D, et al. The personalization of conversational agents in health care: systematic review. J Med Internet Res 2019;21:e15360. doi:10.2196/15360. [DOI: 10.2196/15360]
  38. Loveys K, Antoni M, Donkin L, Sagar M, Xu W, Broadbent E. Effects of cognitive behavioral stress management delivered by a virtual human, teletherapy, and an e-manual on psychological and physiological outcomes in adult women: an experimental test. Multimodal Technol Interact 2022;6:99.
  39. Loveys K, Antoni M, Donkin L, Sagar M, Broadbent E. Comparing the feasibility and acceptability of a virtual human, teletherapy, and an e-manual in delivering a stress management intervention to distressed adult women: pilot study. JMIR Form Res 2023;7:e42390.
  40. Bickmore TW, Pfeifer LM, Paasche-Orlow MK. Using computer agents to explain medical documents to patients with low health literacy. Patient Educ Couns 2009;75:315–20.
  41. Burton C, Szentagotai Tatar A, McKinstry B, et al. Pilot randomised controlled trial of Help4Mood, an embodied virtual agent-based system to support treatment of depression. J Telemed Telecare 2016;22:348–55.
  42. Vardoulakis LP, Ring L, Barry B, Sidner CL, Bickmore T. Designing relational agents as long term social companions for older adults. In: Intelligent Virtual Agents: 12th International Conference, IVA 2012; Santa Cruz, CA; September, 12–14, 2012. Proceedings 12. Berlin, Germany: Springer; 2012:289–302.
  43. Loveys K, Sagar M, Pickering I, Broadbent E. A digital human for delivering a remote loneliness and stress intervention to at-risk younger and older adults during the COVID-19 pandemic: randomized pilot trial. JMIR Ment Health 2021;8:e31586. Shamekhi A, Bickmore T. Breathe Deep: A Breath-Sensitive Interactive Meditation Coach. Int Conf Pervasive Comput Technol Healthc 2018; New York, NY.
  44. Shamekhi A, Bickmore T. Breathe deep: a breath-sensitive interactive meditation coach. In: Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Healthcare, 2018 May 21–24. New York, NY: Association for Computing Machinery; 2018:108–17.
  45. Ellis T, Latham NK, DeAngelis TR, Thomas CA, Saint-Hilaire M, Bickmore TW. Feasibility of a virtual exercise coach to promote walking in community-dwelling persons with Parkinson disease. Am J Phys Med Rehabil 2013;92:472–85; quiz 482-5.
  46. Utami D, Bickmore T, Nikolopoulou A, Paasche-Orlow M. Talk about death: end of life planning with a virtual agent. International Conference on Intelligent Virtual Agents. Cham, Switzerland: Springer; 2017:441–50.
  47. Walker R, Drakeley S, Julce C, Sidduri N, Bickmore T, Skouteris H, et al. Assessing the potential of a virtual patient advocate to provide preconception care and health advice to women living in Australia. J Dev Orig Health Dis 2021;12:300–8.
  48. Kimani E, Bickmore T, Trinh H, Ring L, Paasche-Orlow MK, Magnani JW. A smartphone-based virtual agent for atrial fibrillation education and counseling. In: Traum D, Swartout W, Khooshabeh P, Kopp S, Scherer S, Leuski A, editors. Intelligent Virtual Agents. Cham, Switzerland: Springer; 2016:120–7.
  49. Zhou S, Bickmore T. Automating cancer genetic counseling with an adaptive pedagogical agent. In: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, CHI EA 2021. New York, NY: ACM; 2021;1–7. Available at: https://doi.org/10.1145/3411763.3451675 . Accessed July 24, 2023. [DOI: 10.1145/3411763.3451675]
  50. Bickmore TW, Kimani E, Trinh H, Pusateri A, Paasche-Orlow MK, Magnani JW. Managing chronic conditions with a smartphone-based conversational virtual agent. Intell Virtual Agents 2018;119–24.
  51. Bickmore TW, Mitchell SE, Jack BW, Paasche-Orlow MK, Pfeifer LM, O’Donnell J. Response to a relational agent by hospital patients with depressive symptoms. Interact Comput 2010;22:289–98.
  52. Wang C, Paasche-Orlow MK, Bowen DJ, Cabral H, Winter MR, Cunningham TN, et al. Utility of a virtual counselor (VICKY) to collect family health histories among vulnerable patient populations: a randomized controlled trial. Patient Educ Couns 2021;104:979–88.
  53. Bickmore T, Rubin A, Simon S. Substance use screening using virtual agents: towards automated Screening, Brief Intervention, and Referral to Treatment (SBIRT). Intell Virtual Agents 2020;1–7.
  54. Bickmore T, Bukhari L, Vardoulakis LP, Paasche-Orlow M, Shanahan C. Hospital buddy: a persistent emotional support companion agent for hospital patients. In: International Conference on Intelligent Virtual Agents. Santa Cruz, CA: Springer; 2012:492–5.
  55. O’Leary TK, Stowell E, Kimani E, Parmar D, Olafsson S, Hoffman J, et al. Community-based cultural tailoring of virtual agents. Intell Virtual Agents 2020;1–8.
  56. Bickmore T, Trinh H, Asadi R, Olafsson S. Safety first: conversational agents for health care. In: Studies in Conversational UX Design. Cham, Switzerland: Springer; 2018:33–57.
  57. Parsons TD, Kenny P, Ntuen CA, Pataki CS, Pato MT, Rizzo AA, et al. Objective structured clinical interview training using a virtual human patient. Stud Health Technol Inform 2008;132:357–62.
  58. Rizzo AA, Lange B, Buckwalter JG, Forbell E, Kim J, Sagae K, et al. An intelligent virtual human system for providing healthcare information and support. Stud Health Technol Inform 2011;163:503–9.
  59. DeVault D, Artstein R, Benn G, Dey T, Fast E, Gainer A, et al. SimSensei Kiosk: a virtual human interviewer for healthcare decision support. Proc Int Joint Conf Auton Agents Multiagent Syst 2014;1061–8.
  60. DeVault D, Georgila K, Artstein R, Morbini F, Traum D, Scherer S, et al. Verbal indicators of psychological distress in interactive dialogue with a virtual human. In: Proceedings of the SIGDIAL 2013 Conference. Metz, France. New York, NY: Association for Computing Machinery (ACM); 2013:193–202.
  61. Sagar M, Bullivant D, Robertson P, Efimov O, Jawed K, Kalarot R, et al. A neurobehavioural framework for autonomous animation of virtual human faces. In: SIGGRAPH Asia 2014 Autonomous Virtual Humans and Social Robot for Telepresence. 2014:1–10.
  62. Antoni MH. Stress Management Intervention for Women with Breast Cancer. Washington, DC: American Psychological Association. 2003.
  63. World Health Organization. Meet Florence, WHO’s Digital Health Worker Who Can Help You Quit Tobacco [Internet] . Geneva, Switzerland: World Health Organization; 2021. Available at: https://www.euro.who.int/en/health-topics/disease-prevention/tobacco/news/news/2021/2/meet-florence,-whos-digital-health-worker-who-can-help-you-quit-tobacco . Accessed April 11, 2022.
  64. Gold A, Gross D. Deploying machine learning to improve mental health. MIT News [Internet] 2022. Available at: https://news.mit.edu/2022/deploying-machine-learning-improve-mental-health-rosalind-picard-0126 . Accessed March 23, 2022.
  65. Sagar M, Broadbent E. Participatory medicine: model based tools for engaging and empowering the individual. Interface Focus 2016;6:20150092.
  66. Jones ASK, Ellis CJ, Nash M, Stanfield B, Broadbent E. Using animation to improve recovery from acute coronary syndrome: a randomized trial. Ann Behav Med 2016;50:108–18.
  67. Perera AI, Thomas MG, Moore JO, Faasse K, Petrie KJ. Effect of a smartphone application incorporating personalized health-related imagery on adherence to antiretroviral therapy: a randomized clinical trial. AIDS Patient Care STDS 2014;28:579–86.
  68. Tworek JK, Jamniczky HA, Jacob C, Hallgrímsson B, Wright B. The LINDSAY Virtual Human Project: an immersive approach to anatomy and physiology. Anat Sci Educ 2013;6:19–28.
  69. Hidayat NA, Megantoro P, Yurianta A, Sofiah A, Aldhama SA, Effendi YA. The application of instrumentation system on a contactless robotic triage assistant to detect early transmission on a COVID-19 suspect. Ind J Electr Eng Comput Sci 2021;22:1334–44.
  70. Choi A, Lovett AW, Kang J, Lee K, Choi L. Mobile applications to improve medication adherence: existing apps, quality of life and future directions. Adv Pharmacol Pharmacy 2015;3:64–74.
  71. Bickmore TW, Utami D, Matsuyama R, Paasche-Orlow MK. Improving access to online health information with conversational agents: a randomized controlled experiment. J Med Internet Res 2016;18:e1. doi:10.2196/jmir.5239. [DOI: 10.2196/jmir.5239]
  72. Wang C, Bickmore T, Bowen DJ, Norkunas T, Campion M, Cabral H, et al. Acceptability and feasibility of a virtual counselor (VICKY) to collect family health histories. Genet Med 2015;17:822–30.
  73. King AC, Bickmore TW, Campero MI, Pruitt LA, Yin JL. Employing virtual advisors in preventive care for underserved communities: results from the COMPASS study. J Health Commun 2013;18:1449–64. doi:10.1080/10810730.2013.798374. [DOI: 10.1080/10810730.2013.798374]
  74. Roser M, Ritchie H, Ortiz-Ospina E. Internet [Internet]. Our World in Data 2015. Available at: https://ourworldindata.org/internet . Accessed April 11, 2022.
  75. Park KM, Ku J, Choi SH, Jang HJ, Park JY, Kim SI, et al. A virtual reality application in role-plays of social skills training for schizophrenia: a randomized, controlled trial. Psychiatry Res 2011;189:166–72.
  76. Marcos-Pablos S, González-Pablos E, Martín-Lorenzo C, Flores LA, Gómez-García-Bermejo J, Zalama E. Virtual avatar for emotion recognition in patients with schizophrenia: a pilot study. Front Hum Neurosci 2016;10:421.
  77. Bickmore TW, Puskar K, Schlenk EA, Pfeifer LM, Sereika SM. Maintaining reality: relational agents for antipsychotic medication adherence. Interact Comput 2010;22:276–88.
  78. ter Stal S, Kramer LL, Tabak M, op den Akker H, Hermens H. Design features of embodied conversational agents in eHealth: a literature review. Int J Hum Comput Stud 2020;138:102409.
  79. Loveys K, Sebaratnam G, Sagar M, Broadbent E. The effect of design features on relationship quality with embodied conversational agents: a systematic review. Int J Soc Robot. 2020;12:1293–312.
  80. Loveys K, Hiko C, Sagar M, Zhang X, Broadbent E. “I felt her company”: a qualitative study on factors affecting closeness and emotional support seeking with an embodied conversational agent. Int J Hum Comput Stud. 2022;102771:102771.
  81. Liew TW, Tan SM. Social cues and implications for designing expert and competent artificial agents: a systematic review. Telemat Inform 2021;65:101721.
  82. Trinh H, Shamekhi A, Kimani E, Bickmore TW. Predicting user engagement in longitudinal interventions with virtual agents. Intell Virtual Agents 2018;9–16.
  83. West M, Kraut R, Chew HE. I’d Blush If I Could: Closing Gender Divides in Digital Skills Through Education [Internet] . Paris, France: UNESCO; 2019. Available at: https://unesdoc.unesco.org/ark:/48223/pf0000367416.page=1 . Accessed March 27. 2022.
  84. Yang S, Lee J, Sezgin E, Bridge J, Lin S. Clinical advice by voice assistants on postpartum depression: cross-sectional investigation using Apple Siri, Amazon Alexa, Google Assistant, and Microsoft Cortana. JMIR Mhealth Uhealth 2021;9:e24045.
  85. Chen S-C, Moyle W, Jones C, Petsky H. A social robot intervention on depression, loneliness, and quality of life for Taiwanese older adults in long-term care. Int Psychogeriatr 2020;32:981–91. doi:10.1017/S1041610220000459. [DOI: 10.1017/S1041610220000459]
  86. Xie T, Pentina I. Attachment theory as a framework to understand relationships with social chatbots: a case study of replika. In: Proceedings of the 55th Hawaii International Conference on System Sciences. Hawaii; January 4, 2022.
  87. Cruz Rivera S, Liu X, Chan AW, Denniston AK, Calvert MJ. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. Lancet Digit Health 2020;2:e549–60.
  88. Liu X, Rivera SC, Moher D, Calvert MJ, Denniston AK. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. BMJ 2020;370, m3164.
  89. Tsai W, Lu Q. Culture, emotion suppression and disclosure, and health. Soc Pers Psychol Compass 2018;12:e12373.
  90. Chentsova-Dutton YE, Chu JP, Tsai JL, Rottenberg J, Gross JJ, Gotlib IH. Depression and emotional reactivity: variation among Asian Americans of East Asian descent and European Americans. J Abnorm Psychol 2007;116:776–85.
  91. Mascarenhas S, Degens N, Paiva A, Prada R, Hofstede GJ, Beulens A, et al. Modeling culture in intelligent virtual agents. Auton Agent Multi Agent Syst 2016;30:931–62.

MeSH Term

Humans
Psychosomatic Medicine
Health Promotion

Word Cloud

Created with Highcharts 10.0.0humansVirtualhealthvirtualcarecanpatientsbehavioraldeliverinterventionshumanarticleprovidesimplicationspsychosomaticmedicinepsychologicalmayOBJECTIVE:likelyenhancedeliverynextdecadeartificiallyintelligentcomputeragentshyperrealisticautonomouslyanimatedembodimentsbasedaffectivecomputingtechniquesprogrammedscreenconditionstriageappropriatefacialexpressionsbodygesturesfunctioningsupplementperspectivepsychologyMETHODS:narrativereviewconductedintegrateobservationsfindingsresearch91articlesmultidisciplinaryareaRESULTS:usedmultimodalbehavioranalysisindividualizedtailoringdetectionchangesmeasurestimealsopairscalabilitywebsiteinteractivityrelationalskillstele-therapistResearchbeginningshowacceptabilityfeasibilitypreliminaryeffectivenessrangepopulationseasilytailoredtermsappearancevoicelanguageadaptedfitcharacteristicspatientpopulationhard-to-reachgroupsco-designedcommunitieshelppromoteengagementimproveoutcomesCONCLUSIONS:engagemotivatepersonalizedoverviewpotentialimpactdiscussesethicalImpactHumansPsychosomaticMedicine

Similar Articles

Cited By