Dense surface reconstruction using a learning-based monocular vSLAM model for laparoscopic surgery.

James Yu, Kelden Pruitt, Nati Nawawithan, Brett A Johnson, Jeffrey Gahan, Baowei Fei
Author Information
  1. James Yu: Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX.
  2. Kelden Pruitt: Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX.
  3. Nati Nawawithan: Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX.
  4. Brett A Johnson: Department of Urology, University of Texas Southwestern Medical Center, Dallas, TX.
  5. Jeffrey Gahan: Department of Urology, University of Texas Southwestern Medical Center, Dallas, TX.
  6. Baowei Fei: Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX.

Abstract

Augmented reality (AR) has seen increased interest and attention for its application in surgical procedures. AR-guided surgical systems can overlay segmented anatomy from pre-operative imaging onto the user's environment to delineate hard-to-see structures and subsurface lesions intraoperatively. While previous works have utilized pre-operative imaging such as computed tomography or magnetic resonance images, registration methods still lack the ability to accurately register deformable anatomical structures without fiducial markers across modalities and dimensionalities. This is especially true of minimally invasive abdominal surgical techniques, which often employ a monocular laparoscope, due to inherent limitations. Surgical scene reconstruction is a critical component towards accurate registrations needed for AR-guided surgery and other downstream AR applications such as remote assistance or surgical simulation. In this work, we utilize a state-of-the-art (SOTA) deep-learning-based visual simultaneous localization and mapping (vSLAM) algorithm to generate a dense 3D reconstruction with camera pose estimations and depth maps from video obtained with a monocular laparoscope. The proposed method can robustly reconstruct surgical scenes using real-time data and provide camera pose estimations without stereo or additional sensors, which increases its usability and is less intrusive. We also demonstrate a framework to evaluate current vSLAM algorithms on non-Lambertian, low-texture surfaces and explore using its outputs on downstream tasks. We expect these evaluation methods can be utilized for the continual refinement of newer algorithms for AR-guided surgery.

Keywords

References

  1. Am J Surg. 1994 Jan;167(1):35-9; discussion 39-41 [PMID: 8311138]
  2. Endoscopy. 2001 Apr;33(4):358-66 [PMID: 11315900]
  3. JMIR Mhealth Uhealth. 2019 Apr 26;7(4):e10967 [PMID: 31025950]
  4. Med Ref Serv Q. 2012;31(2):212-8 [PMID: 22559183]
  5. Endoscopy. 2004 Jan;36(1):48-51 [PMID: 14722855]
  6. Surg Technol Int. 2011 Dec;21:147-55 [PMID: 22504984]
  7. Med Image Anal. 2021 Jul;71:102058 [PMID: 33930829]
  8. Med Image Anal. 2017 Apr;37:66-90 [PMID: 28160692]
  9. J Am Assoc Gynecol Laparosc. 2003 Nov;10(4):469-73 [PMID: 14738630]
  10. Arch Surg. 2012 Dec;147(12):1114-22 [PMID: 23248013]
  11. Front Robot AI. 2021 Aug 30;8:716007 [PMID: 34527706]

Grants

  1. R01 CA204254/NCI NIH HHS
  2. R01 CA288379/NCI NIH HHS

Word Cloud

Created with Highcharts 10.0.0surgicalreconstructionsurgeryAR-guidedcanmonocularvSLAMusingrealityARpre-operativeimagingstructuresutilizedmethodswithoutlaparoscopedownstream3DcameraposeestimationsalgorithmsAugmentedseenincreasedinterestattentionapplicationproceduressystemsoverlaysegmentedanatomyontouser'senvironmentdelineatehard-to-seesubsurfacelesionsintraoperativelypreviousworkscomputedtomographymagneticresonanceimagesregistrationstilllackabilityaccuratelyregisterdeformableanatomicalfiducialmarkersacrossmodalitiesdimensionalitiesespeciallytrueminimallyinvasiveabdominaltechniquesoftenemploydueinherentlimitationsSurgicalscenecriticalcomponenttowardsaccurateregistrationsneededapplicationsremoteassistancesimulationworkutilizestate-of-the-artSOTAdeep-learning-basedvisualsimultaneouslocalizationmappingalgorithmgeneratedensedepthmapsvideoobtainedproposedmethodrobustlyreconstructscenesreal-timedataprovidestereoadditionalsensorsincreasesusabilitylessintrusivealsodemonstrateframeworkevaluatecurrentnon-Lambertianlow-texturesurfacesexploreoutputstasksexpectevaluationcontinualrefinementnewerDensesurfacelearning-basedmodellaparoscopicMRISLAMaugmenteddeeplearningimage-guidedlaparoscopyneuralnetworks

Similar Articles

Cited By