Neural Radiance Fields for High-Fidelity Soft Tissue Reconstruction in Endoscopy.

Jinhua Liu, Yongsheng Shi, Dongjin Huang, Jiantao Qu
Author Information
  1. Jinhua Liu: Shanghai Film Academy, Shanghai University, Shanghai 200072, China. ORCID
  2. Yongsheng Shi: Shanghai Film Academy, Shanghai University, Shanghai 200072, China. ORCID
  3. Dongjin Huang: Shanghai Film Academy, Shanghai University, Shanghai 200072, China.
  4. Jiantao Qu: Shanghai Film Academy, Shanghai University, Shanghai 200072, China.

Abstract

The advancement of neural radiance fields (NeRFs) has facilitated the high-quality 3D reconstruction of complex scenes. However, for most NeRFs, reconstructing 3D tissues from endoscopy images poses significant challenges due to the occlusion of soft tissue regions by invalid pixels, deformations in soft tissue, and poor image quality, which severely limits their application in endoscopic scenarios. To address the above issues, we propose a novel framework to reconstruct high-fidelity soft tissue scenes from low-quality endoscopic images. We first construct an EndoTissue dataset of soft tissue regions in endoscopic images and fine-tune the Segment Anything Model (SAM) based on EndoTissue to obtain a potent segmentation network. Given a sequence of monocular endoscopic images, this segmentation network can quickly obtain the tissue mask images. Additionally, we incorporate tissue masks into a dynamic scene reconstruction method called Tensor4D to effectively guide the reconstruction of 3D deformable soft tissues. Finally, we propose adopting the image enhancement model EDAU-Net to improve the quality of the rendered views. The experimental results show that our method can effectively focus on the soft tissue regions in the image, achieving higher fidelity in detail and geometric structural integrity in reconstruction compared to state-of-the-art algorithms. Feedback from the user study indicates high participant scores for our method.

Keywords

References

  1. Int J Comput Assist Radiol Surg. 2014 Mar;9(2):283-93 [PMID: 24037504]
  2. Sci Data. 2023 Feb 6;10(1):75 [PMID: 36746950]
  3. Med Image Anal. 2021 May;70:101994 [PMID: 33611053]
  4. Int J Med Robot. 2022 Aug;18(4):e2396 [PMID: 35318786]
  5. Sci Data. 2020 Aug 28;7(1):283 [PMID: 32859981]
  6. Med Image Anal. 2021 May;70:102002 [PMID: 33657508]
  7. Annu Int Conf IEEE Eng Med Biol Soc. 2022 Jul;2022:3843-3848 [PMID: 36085823]
  8. Comput Biol Med. 2023 Jun;160:106961 [PMID: 37156222]
  9. IEEE Trans Med Imaging. 2021 May;40(5):1450-1460 [PMID: 33556005]
  10. IEEE Trans Image Process. 2013 Mar;22(3):1032-41 [PMID: 23144035]
  11. Med Image Anal. 2023 Dec;90:102975 [PMID: 37804586]
  12. Med Image Anal. 2021 Dec;74:102222 [PMID: 34543913]
  13. Nat Commun. 2024 Jan 22;15(1):654 [PMID: 38253604]
  14. Med Image Comput Comput Assist Interv. 2014;17(Pt 2):438-45 [PMID: 25485409]
  15. J Healthc Eng. 2017;2017:4037190 [PMID: 29065595]
  16. Sensors (Basel). 2024 Sep 12;24(18): [PMID: 39338670]
  17. Sci Rep. 2024 Jan 23;14(1):2032 [PMID: 38263232]

Grants

  1. 2023-X-036/the Science and Technology Projects of National Archives Administration of China
  2. 2021016/the Shanghai Talent Development Funding of China

MeSH Term

Humans
Endoscopy
Algorithms
Imaging, Three-Dimensional
Neural Networks, Computer
Image Processing, Computer-Assisted

Word Cloud

Created with Highcharts 10.0.0tissuesoftreconstructionimagesimageendoscopic3DregionssegmentationmethodneuralradiancefieldsNeRFsscenestissuesqualityproposeEndoTissueobtainnetworkcaneffectivelyadvancementfacilitatedhigh-qualitycomplexHoweverreconstructingendoscopyposessignificantchallengesdueocclusioninvalidpixelsdeformationspoorseverelylimitsapplicationscenariosaddressissuesnovelframeworkreconstructhigh-fidelitylow-qualityfirstconstructdatasetfine-tuneSegmentAnythingModelSAMbasedpotentGivensequencemonocularquicklymaskAdditionallyincorporatemasksdynamicscenecalledTensor4DguidedeformableFinallyadoptingenhancementmodelEDAU-Netimproverenderedviewsexperimentalresultsshowfocusachievinghigherfidelitydetailgeometricstructuralintegritycomparedstate-of-the-artalgorithmsFeedbackuserstudyindicateshighparticipantscoresNeuralRadianceFieldsHigh-FidelitySoftTissueReconstructionEndoscopydynamics

Similar Articles

Cited By

No available data.