HDMF: Hierarchical Data Modeling Framework for Modern Science Data Standards.

Advanced Search

Andrew J Tritt, Oliver Rübel, Benjamin Dichter, Ryan Ly, Donghe Kang, Edward F Chang, Loren M Frank, Kristofer Bouchard

Author Information

Andrew J Tritt: Computational Research Division, Lawrence Berkeley National Laboratory, Berkeley, CA, USA.
Oliver Rübel: Computational Research Division, Lawrence Berkeley National Laboratory, Berkeley, CA, USA.
Benjamin Dichter: Biological Systems and Engineering, Lawrence Berkeley National Laboratory, Berkeley, CA, USA.
Ryan Ly: Computational Research Division, Lawrence Berkeley National Laboratory, Berkeley, CA, USA.
Donghe Kang: Computer Science and Engineering, Ohio State University, Columbus, OH, USA.
Edward F Chang: Department of Neurological Surgery and the Center for Integrative Neuroscience, University of California, San Francisco, San Francisco, CA, USA.
Loren M Frank: Howard Hughes Medical Institute, Kavli Institute for Fundamental Neuroscience, Department of Physiology, University of California, San Francisco, San Francisco, CA.
Kristofer Bouchard: Biological Systems and Engineering, Lawrence Berkeley National Laboratory, Berkeley, CA, USA.

PMID: 34632466 DOI: 10.1109/bigdata47090.2019.9005648

A ubiquitous problem in aggregating data across different experimental and observational data sources is a lack of software infrastructure that enables flexible and extensible standardization of data and metadata. To address this challenge, we developed HDMF, a hierarchical data modeling framework for modern science data standards. With HDMF, we separate the process of data standardization into three main components: (1) data modeling and specification, (2) data I/O and storage, and (3) data interaction and data APIs. To enable standards to support the complex requirements and varying use cases throughout the data life cycle, HDMF provides object mapping infrastructure to insulate and integrate these various components. This approach supports the flexible development of data standards and extensions, optimized storage backends, and data APIs, while allowing the other components of the data standards ecosystem to remain stable. To meet the demands of modern, large-scale science data, HDMF provides advanced data I/O functionality for iterative data write, lazy data load, and parallel I/O. It also supports optimization of data storage via support for chunking, compression, linking, and modular data storage. We demonstrate the application of HDMF in practice to design NWB 2.0 [13], a modern data standard for collaborative science across the neurophysiology community.

HDF5 data formats data modeling data standards neurophysiology

Nat Methods. 2012 Sep;9(9):854-5 [PMID: 22936162]
Anal Chem. 2013 Nov 5;85(21):10354-61 [PMID: 24087878]
Sci Data. 2016 Mar 15;3:160018 [PMID: 26978244]
Front Neuroinform. 2016 Nov 04;10:48 [PMID: 27867355]

R24 MH116922/NIMH NIH HHS

Journal Article

The Neurodata Without Borders ecosystem for neurophysiological data science.A Perspective on Neuroscience Data Standardization with Neurodata Without Borders.A perspective on neuroscience data standardization with Neurodata Without Borders.

OpenLB
Open Library of Bioscience