Natural language processing and entrustable professional activity text feedback in surgery: A machine learning model of resident autonomy.
Christopher C Stahl, Sarah A Jung, Alexandra A Rosser, Aaron S Kraut, Benjamin H Schnapp, Mary Westergaard, Azita G Hamedani, Rebecca M Minter, Jacob A Greenberg
Author Information
Christopher C Stahl: Department of Surgery, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA.
Sarah A Jung: Department of Surgery, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA.
Alexandra A Rosser: Department of Surgery, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA.
Aaron S Kraut: Department of Emergency Medicine, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA.
Benjamin H Schnapp: Department of Emergency Medicine, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA.
Mary Westergaard: Department of Emergency Medicine, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA.
Azita G Hamedani: Department of Emergency Medicine, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA.
Rebecca M Minter: Department of Surgery, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA.
Jacob A Greenberg: Department of Surgery, University of Wisconsin School of Medicine and Public Health, Madison, WI, USA. Electronic address: greenbergj@surgery.wisc.edu.
BACKGROUND: Entrustable Professional Activities (EPAs) contain narrative 'entrustment roadmaps' designed to describe specific behaviors associated with different entrustment levels. However, these roadmaps were created using expert committee consensus, with little data available for guidance. Analysis of actual EPA assessment narrative comments using natural language processing may enhance our understanding of resident entrustment in actual practice. METHODS: All text comments associated with EPA microassessments at a single institution were combined. EPA-entrustment level pairs (e.g. Gallbladder Disease-Level 1) were identified as documents. Latent Dirichlet Allocation (LDA), a common machine learning algorithm, was used to identify latent topics in the documents associated with a single EPA. These topics were then reviewed for interpretability by human raters. RESULTS: Over 18 months, 1015 faculty EPA microassessments were collected from 64 faculty for 80 residents. LDA analysis identified topics that mapped 1:1 to EPA entrustment levels (Gammas >0.99). These LDA topics appeared to trend coherently with entrustment levels (words demonstrating high entrustment were consistently found in high entrustment topics, word demonstrating low entrustment were found in low entrustment topics). CONCLUSIONS: LDA is capable of identifying topics relevant to progressive surgical entrustment and autonomy in EPA comments. These topics provide insight into key behaviors that drive different level of resident autonomy and may allow for data-driven revision of EPA entrustment maps.
