SonoNERFs: Neural Radiance Fields Applied to Biological Echolocation Systems Allow 3D Scene Reconstruction through Perceptual Prediction.

Wouter Jansen, Jan Steckel
Author Information
  1. Wouter Jansen: Cosys-Lab, Faculty of Applied Engineering, University of Antwerp, 2020 Antwerpen, Belgium. ORCID
  2. Jan Steckel: Cosys-Lab, Faculty of Applied Engineering, University of Antwerp, 2020 Antwerpen, Belgium. ORCID

Abstract

In this paper, we introduce SonoNERFs, a novel approach that adapts Neural Radiance Fields (NeRFs) to model and understand the echolocation process in bats, focusing on the challenges posed by acoustic data interpretation without phase information. Leveraging insights from the field of optical NeRFs, our model, termed SonoNERF, represents the acoustic environment through Neural Reflectivity Fields. This model allows us to reconstruct three-dimensional scenes from echolocation data, obtained by simulating how bats perceive their surroundings through sound. By integrating concepts from biological echolocation and modern computational models, we demonstrate the SonoNERF's ability to predict echo spectrograms for unseen echolocation poses and effectively reconstruct a mesh-based and energy-based representation of complex scenes. Our work bridges a gap in understanding biological echolocation and proposes a methodological framework that provides a first-order model of how scene understanding might arise in echolocating animals. We demonstrate the efficacy of the SonoNERF model on three scenes of increasing complexity, including some biologically relevant prey-predator interactions.

Keywords

References

  1. Cognition. 1989 Nov;33(1-2):155-99 [PMID: 2691182]
  2. Radiology. 1992 Nov;185(2):607-10 [PMID: 1410382]
  3. Proc Natl Acad Sci U S A. 2020 Dec 8;117(49):31134-31141 [PMID: 33229524]
  4. Brain Behav Evol. 2018;91(2):97-108 [PMID: 29874652]
  5. Biomimetics (Basel). 2023 Sep 19;8(5): [PMID: 37754187]
  6. Proc Biol Sci. 2007 Apr 7;274(1612):905-12 [PMID: 17251105]
  7. PLoS One. 2010 Aug 16;5(8):e11893 [PMID: 20808438]
  8. Proc Natl Acad Sci U S A. 2020 Nov 17;117(46):29229-29238 [PMID: 33139550]
  9. Elife. 2015 Sep 01;4: [PMID: 26327624]
  10. Proc Natl Acad Sci U S A. 2020 Jan 21;117(3):1367-1374 [PMID: 31907314]
  11. PLoS One. 2013;8(1):e54076 [PMID: 23365647]
  12. J Acoust Soc Am. 1998 Aug;104(2 Pt 1):1101-10 [PMID: 9714928]
  13. Nature. 2001 Dec 13;414(6865):742-5 [PMID: 11742397]
  14. Front Physiol. 2013 Apr 05;4:65 [PMID: 23576990]
  15. PLoS Comput Biol. 2021 Jun 28;17(6):e1009052 [PMID: 34181643]
  16. J Comp Physiol A. 1985 Nov;157(5):687-97 [PMID: 3837108]
  17. PLoS Comput Biol. 2021 Dec 16;17(12):e1009706 [PMID: 34914700]
  18. J Exp Biol. 2023 Mar 1;226(5): [PMID: 36806421]
  19. Proc Biol Sci. 2013 Jan 16;280(1754):20122830 [PMID: 23325775]
  20. Biol Cybern. 2008 Jun;98(6):587-95 [PMID: 18491168]
  21. J Acoust Soc Am. 1993 Nov;94(5):2691-712 [PMID: 8270744]
  22. J Comp Physiol A. 1990 Feb;166(4):449-70 [PMID: 2332837]
  23. J Acoust Soc Am. 2008 Oct;124(4):2123-32 [PMID: 19062853]
  24. Biomed Opt Express. 2024 Feb 14;15(3):1651-1667 [PMID: 38495696]
  25. J Acoust Soc Am. 1976 Jan;59(1):97-106 [PMID: 1249325]
  26. Curr Biol. 2019 Aug 19;29(16):2731-2736.e3 [PMID: 31378617]
  27. Curr Biol. 2015 Jul 20;25(14):1911-6 [PMID: 26166777]
  28. Biol Lett. 2011 Jun 23;7(3):436-9 [PMID: 21270023]
  29. Commun Biol. 2021 Nov 10;4(1):1275 [PMID: 34759372]
  30. Naturwissenschaften. 2005 Jun;92(6):272-6 [PMID: 15871000]

Word Cloud

Created with Highcharts 10.0.0NeuralFieldsmodelecholocationRadiancebatsscenesNeRFsacousticdataSonoNERFreconstructbiologicaldemonstrateunderstandingecholocatingpaperintroduceSonoNERFsnovelapproachadaptsunderstandprocessfocusingchallengesposedinterpretationwithoutphaseinformationLeveraginginsightsfieldopticaltermedrepresentsenvironmentReflectivityallowsusthree-dimensionalobtainedsimulatingperceivesurroundingssoundintegratingconceptsmoderncomputationalmodelsSonoNERF'sabilitypredictechospectrogramsunseenposeseffectivelymesh-basedenergy-basedrepresentationcomplexworkbridgesgapproposesmethodologicalframeworkprovidesfirst-orderscenemightariseanimalsefficacythreeincreasingcomplexityincludingbiologicallyrelevantprey-predatorinteractionsSonoNERFs:AppliedBiologicalEcholocationSystemsAllow3DSceneReconstructionPerceptualPredictionReflectancebiosonarsonarultrasoundsensing

Similar Articles

Cited By