aws-s3-integrity-check: an open-source bash tool to verify the integrity of a dataset stored on Amazon S3.
Sonia Garc��a-Ruiz, Regina Hertfelder Reynolds, Melissa Grant-Peters, Emil Karl Gustavsson, Aine Fairbrother-Browne, Zhongbo Chen, Jonathan William Brenton, Mina Ryten
Author Information
Sonia Garc��a-Ruiz: Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK. ORCID
Regina Hertfelder Reynolds: Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK. ORCID
Melissa Grant-Peters: Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK. ORCID
Emil Karl Gustavsson: Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK. ORCID
Aine Fairbrother-Browne: Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK. ORCID
Zhongbo Chen: Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK. ORCID
Jonathan William Brenton: Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK. ORCID
Mina Ryten: Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK. ORCID
Amazon Simple Storage Service (Amazon S3) is a widely used platform for storing large biomedical datasets. Unintended data alterations can occur during data writing and transmission, altering the original content and generating unexpected results. However, no open-source and easy-to-use tool exists to verify end-to-end data integrity. Here, we present , a user-friendly, lightweight, and reliable bash tool to verify the integrity of a dataset stored in an Amazon S3 bucket. Using this tool, we only needed ���114 min to verify the integrity of 1,045 records ranging between 5 bytes and 10 gigabytes and occupying ���935 gigabytes of the Amazon S3 cloud. Our tool also provides file-by-file on-screen and log-file-based information about the status of each integrity check. To our knowledge, this tool is the only open-source one that allows verifying the integrity of a dataset uploaded to the Amazon S3 Storage quickly, reliably, and efficiently. The tool is freely available for download and use at https://github.com/SoniaRuiz/aws-s3-integrity-check and https://hub.docker.com/r/soniaruiz/aws-s3-integrity-check.
