OpenLB
Open Library of Bioscience
Advanced Search
Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery
Apr, 2025;172 (4): 1435-1441.

Neural Radiance Fields (NeRF) for 3D Reconstruction of Monocular Endoscopic Video in Sinus Surgery.

Jeremy S Ruthberg, Randall Bly, Nicole Gunderson, Pengcheng Chen, Mahdi Alighezi, Eric J Seibel, Waleed M Abuzeid
Author Information
  1. Jeremy S Ruthberg: Department of Otolaryngology-Head and Neck Surgery, University of Washington, Seattle, Washington, USA. ORCID
  2. Randall Bly: Department of Otolaryngology-Head and Neck Surgery, University of Washington, Seattle, Washington, USA.
  3. Nicole Gunderson: Department of Mechanical Engineering, University of Washington, Seattle, Washington, USA.
  4. Pengcheng Chen: Department of Mechanical Engineering, University of Washington, Seattle, Washington, USA.
  5. Mahdi Alighezi: School of Medicine, University of Washington, Seattle, Washington, USA.
  6. Eric J Seibel: Department of Mechanical Engineering, University of Washington, Seattle, Washington, USA.
  7. Waleed M Abuzeid: Department of Otolaryngology-Head and Neck Surgery, University of Washington, Seattle, Washington, USA.
PMID: 39791932 DOI: 10.1002/ohn.1105

Abstract

OBJECTIVE: To validate the use of neural radiance fields (NeRF), a state-of-the-art computer vision technique, for rapid, high-fidelity 3-dimensional (3D) reconstruction in endoscopic sinus surgery (ESS).
STUDY DESIGN: An experimental cadaveric pilot study.
SETTING: Academic medical center.
METHODS: Complete bilateral endoscopic sinus surgery was performed on 3 cadaveric specimens, followed by postsurgical nasal endoscopy using a 0° rigid endoscope. NeRF was utilized to generate 3D reconstructions from the monocular endoscopic video feed. Reconstructions were calibrated, scaled, and then co-registered to postoperative computed tomography (CT) image sets to assess accuracy. Reconstruction error was determined by comparing ethmoid sinus measurements on NeRF reconstructions and CT image sets.
RESULTS: NeRF-based 3D scene reconstructions were successfully generated and co-registered to corresponding CT images for 5 out of 6 cadaveric nasal cavity sides. The mean reconstruction errors and standard error of the mean (SEM) for ethmoid length and height were 0.17 (SEM 0.59) and 0.70 (SEM 0.44) mm, respectively.
CONCLUSION: NeRF demonstrates significant potential for dynamic, high-fidelity 3D surgical field reconstruction in ESS, offering submillimeter accuracy comparable to postoperative CT data in cadaveric specimens. This innovative approach may ultimately augment dynamic real-time intraoperative navigation through co-registration of the 3D reconstruction with preoperative imaging to potentially reduce the risk of injury to critical structures, optimize surgical completeness and, thereby, improve surgical outcomes. Further refinement and validation in live surgical settings are necessary to fully realize its clinical utility.

Keywords

3D reconstruction computer vision computer‐assisted surgery endoscopic sinus surgery neural radiance fields skull‐base surgery

References

  1. Stankiewicz JA, Lal D, Connor M, Welch K. Complications in endoscopic sinus surgery for chronic rhinosinusitis: a 25‐year experience. Laryngoscope. 2011;121(12):2684‐2701. doi:10.1002/lary.21446
  2. Gore MR, Ebert CS, Zanation AM, Senior BA. Beyond the “central sinus”: radiographic findings in patients undergoing revision functional endoscopic sinus surgery. Int Forum Allergy Rhinol. 2013;3(2):139‐146. doi:10.1002/alr.21079
  3. Rawlings BA, Han JK. Level of complete dissection of the ethmoid sinuses with a computed tomographic image guidance system. Ann Otol Rhinol Laryngol. 2010;119(1):17‐21. doi:10.1177/000348941011900103
  4. Kim TT, Drazin D, Shweikeh F, Pashman R, Johnson JP. Clinical and radiographic outcomes of minimally invasive percutaneous pedicle screw placement with intraoperative CT (O‐arm) image guidance navigation. Neurosurg Focus. 2014;36(3):E1. doi:10.3171/2014.1.FOCUS13531
  5. Cuddy K, Khatib B, Bell RB, et al. Use of intraoperative computed tomography in craniomaxillofacial trauma surgery. J Oral Maxillofac Surg. 2018;76(5):1016‐1025. doi:10.1016/j.joms.2017.12.004
  6. Mitchell MB, Labadie RF. Cost‐effectiveness of intraoperative CT scanning in cochlear implantation in fee‐for‐service and bundled payment models. Ear Nose Throat J. 2022;101(4):NP164‐NP168. doi:10.1177/0145561320952192
  7. Jackman AH, Palmer JN, Chiu AG, Kennedy DW. Use of intraoperative CT scanning in endoscopic sinus surgery: a preliminary report. Am J Rhinol. 2008;22(2):170‐174. doi:10.2500/ajr.2008.22.3153
  8. Chen L, Tang W, John NW, Wan TR, Zhang JJ. SLAM‐based dense surface reconstruction in monocular minimally invasive surgery and its application to augmented reality. Comput Methods Programs Biomed. 2018;158:135‐146. doi:10.1016/j.cmpb.2018.02.006
  9. Liu X, Li Z, Ishii M, Hager GD, Taylor RH, Unberath M. SAGE: SLAM with appearance and geometry prior for endoscopy. 2022 International Conference on Robotics and Automation (ICRA), Philadelphia, PA. IEEE; 2022:5587‐5593. doi:10.1109/ICRA46639.2022.9812257
  10. Liu X, Sinha A, Ishii M, et al. Dense depth estimation in monocular endoscopy with self‐supervised learning methods. IEEE Trans Med Imaging. 2020;39(5):1438‐1447. doi:10.1109/TMI.2019.2950936
  11. Liu X, Stiber M, Huang J, et al. Reconstructing sinus anatomy from endoscopic video— towards a radiation‐free approach for quantitative longitudinal assessment. In: Martel AL, Abolmaesumi P, Stoyanov D, et al. Medical Image Computing and Computer Assisted Intervention— MICCAI 2020. Springer International Publishing; 2020:3‐13.
  12. Mildenhall B, Srinivasan PP, Tancik M, Barron JT, Ramamoorthi R, Ng R. NeRF: Representing scenes as neural radiance fields for view synthesis. Commun ACM. 2021;65(1):99‐106. doi:10.1145/3503250
  13. Qin Z, Qian K, Liang S, Zheng Q, Peng J, Tai Y. Neural radiance fields‐based multi‐view endoscopic scene reconstruction for surgical simulation. Int J Computer Assisted Radiol Surg. 2024;19(5):951‐960. doi:10.1007/s11548-024-03080-8
  14. Batlle VM, Montiel JMM, Fua P, Tardós JD. LightNeuS: neural surface reconstruction in endoscopy using illumination decline. In: Greenspan H, Madabhushi A, Mousavi P, Salcudean S, Duncan J, Syeda‐Mahmood T, Taylor R, eds. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. Springer Nature; 2023:502‐512.
  15. Shi Y, Lu B, Liu JW, Li M, Shou MZ. ColonNeRF: high‐fidelity neural reconstruction of long colonoscopy. arXiv. 2023. https://arxiv.org/abs/2312.02015
  16. Chen P, Lewis A, Speich JR, Porter MP, Seibel EJ. Enabling rapid and high‐quality 3D scene reconstruction in cystoscopy through neural radiance fields. SPIE Med Imag. 2023;12928:350‐359.
  17. Song W, Zheng H, Tu D, Liang C, He L. Oral‐3Dv2: 3D oral reconstruction from panoramic X‐ray imaging with implicit neural representation. arXiv. 2023. https://arxiv.org/abs/2312.02015
  18. Wang Y, Long Y, Fan SH, Dou Q. Neural rendering for stereo 3D reconstruction of deformable tissues in robotic surgery. arXiv. 2022. doi:10.1007/978-3-031-16449-1_41
  19. Chen P, Li W, Gunderson N, et al. Hybrid NeRF‐stereo vision: pioneering depth estimation and 3D reconstruction in endoscopy. arXiv. 2024. https://arxiv.org/abs/2410.04041
  20. O'Brien WT, Hamelin S, Weitzel EK. The preoperative sinus CT: avoiding a “CLOSE” call with surgical complications. Radiology. 2016;281(1):10‐21. doi:10.1148/radiol.2016152230
  21. Fedorov A, Beichel R, Kalpathy‐Cramer J, et al. 3D slicer as an image computing platform for the quantitative imaging network. Magn Reson Imaging. 2012;30(9):1323‐1341. doi:10.1016/j.mri.2012.05.001
  22. Pinter C, Lasso A, Fichtinger G. Polymorph segmentation representation for medical image computing. Comput Methods Programs Biomed. 2019;171:19‐26. doi:10.1016/j.cmpb.2019.02.011
  23. Musy PY, Kountakis SE. Anatomic findings in patients undergoing revision endoscopic sinus surgery. Am J Otolaryngol. 2004;25(6):418‐422. doi:10.1016/j.amjoto.2004.06.002
  24. Baban MIA, Mirza B, Castelnuovo P. Radiological and endoscopic findings in patients undergoing revision endoscopic sinus surgery. Surg Radiol Anat. 2020;42(9):1003‐1012. doi:10.1007/s00276-020-02427-5
  25. Schmale IL, Vandelaar LJ, Luong AU, Citardi MJ, Yao WC. Image‐guided surgery and intraoperative imaging in rhinology: clinical update and current state of the art. Ear Nose Throat J. 2021;100(10):NP475‐NP486. doi:10.1177/0145561320928202
  26. Barbour MC, Amin SN, Friedman SD, et al. Surface reconstruction of the pediatric larynx via structure from motion photogrammetry: a pilot study. Otolaryngol Head Neck Surg. 2024;170:1195‐1199. doi:10.1002/ohn.635
  27. Chen P, Gunderson NM, Lewis A, Speich JR, Porter MP, Seibel EJ. Enabling rapid and high‐quality 3D scene reconstruction in cystoscopy through neural radiance fields. In: Rettmann ME, Siewerdsen JH, eds. Medical Imaging 2024: Image‐Guided Procedures, Robotic Interventions, and Modeling. SPIE; 2024:56. doi:10.1117/12.3000828
  28. Fu Z, Jin Z, Zhang C, et al. Visual‐electromagnetic system: a novel fusion‐based monocular localization, reconstruction, and measurement for flexible ureteroscopy. Int J Med Robot. 2021;17(4):e2274. doi:10.1002/rcs.2274
  29. Bartholomew RA, Zhou H, Boreel M, et al. Surgical navigation in the anterior skull base using 3‐dimensional endoscopy and surface reconstruction. JAMA Otolaryngol Head Neck Surg. 2024;150(4):318‐326. doi:10.1001/jamaoto.2024.0013
  30. Richter A, Steinmann T, Rosenthal JC, Rupitsch SJ. Advances in real‐time 3D reconstruction for medical endoscopy. J Imaging. 2024;10(5):120. doi:10.3390/jimaging10050120
  31. Douglas JE, Patel TD, Rullan‐Oliver BE, Kohanski MA, Palmer JN, Adappa ND. Novel intraoperative fast anatomic mapping as teaching adjunct in endoscopic sinus surgery. Int Forum Allergy Rhinol. 2022;12(12):1575‐1577. doi:10.1002/alr.23046
  32. BrainLab. Curve navigation: curve into digital surgery. 2024. Accessed February 29, 2024. https://www.brainlab.com/surgery-products/overview-platform-products/curve-image-guided-surgery/
  33. Chen P, Gong C, Lewis A, et al. Real‐time flexible endoscope navigation within bladder phantom having sparse non‐distinct features is enhanced with robotic control. In: Linte CA, Siewerdsen JH, eds. Medical Imaging 2022: Image‐Guided Procedures, Robotic Interventions, and Modeling. SPIE; 2022:11. doi:10.1117/12.2611306
  34. American Academy of Otolaryngology–Head and Neck Surgery. Intra‐Operative Use of Computer Aided Surgery. 2021. Accessed December 26, 2024. https://www.entnet.org/resource/position-statement-intra-operative-use-of-computer-aided-surgery/

Grants

  1. R25 DC021791/NIDCD NIH HHS

MeSH Term

Humans
Imaging, Three-Dimensional
Cadaver
Pilot Projects
Endoscopy
Tomography, X-Ray Computed
Paranasal Sinuses
Journal Article

Word Cloud

Created with Highcharts 10.0.03DNeRFreconstructionsurgeryendoscopicsinuscadavericCT0surgicalreconstructionsSEMneuralradiancefieldscomputervisionhigh-fidelityESSspecimensnasalco-registeredpostoperativeimagesetsaccuracyReconstructionerrorethmoidmeandynamicOBJECTIVE:validateusestate-of-the-arttechniquerapid3-dimensionalSTUDYDESIGN:experimentalpilotstudySETTING:AcademicmedicalcenterMETHODS:Completebilateralperformed3followedpostsurgicalendoscopyusing0°rigidendoscopeutilizedgeneratemonocularvideofeedReconstructionscalibratedscaledcomputedtomographyassessdeterminedcomparingmeasurementsRESULTS:NeRF-basedscenesuccessfullygeneratedcorrespondingimages56cavitysideserrorsstandardlengthheight17597044 mmrespectivelyCONCLUSION:demonstratessignificantpotentialfieldofferingsubmillimetercomparabledatainnovativeapproachmayultimatelyaugmentreal-timeintraoperativenavigationco-registrationpreoperativeimagingpotentiallyreduceriskinjurycriticalstructuresoptimizecompletenesstherebyimproveoutcomesrefinementvalidationlivesettingsnecessaryfullyrealizeclinicalutilityNeuralRadianceFieldsMonocularEndoscopicVideoSinusSurgerycomputer‐assistedskull‐base

Similar Articles

Cited By