OpenLB
Open Library of Bioscience
Advanced Search
PloS one
2021;16 (6): e0251194.

A computational reproducibility study of PLOS ONE articles featuring longitudinal data analyses.

Heidi Seibold, Severin Czerny, Siona Decke, Roman Dieterle, Thomas Eder, Steffen Fohr, Nico Hahn, Rabea Hartmann, Christoph Heindl, Philipp Kopper, Dario Lepke, Verena Loidl, Maximilian Mandl, Sarah Musiol, Jessica Peter, Alexander Piehler, Elio Rojas, Stefanie Schmid, Hannah Schmidt, Melissa Schmoll, Lennart Schneider, Xiao-Yin To, Viet Tran, Antje Völker, Moritz Wagner, Joshua Wagner, Maria Waize, Hannah Wecker, Rui Yang, Simone Zellner, Malte Nalenz
Author Information
  1. Heidi Seibold: Department of Statistics, LMU Munich, Munich, Germany. ORCID
  2. Severin Czerny: Department of Statistics, LMU Munich, Munich, Germany.
  3. Siona Decke: Department of Statistics, LMU Munich, Munich, Germany.
  4. Roman Dieterle: Department of Statistics, LMU Munich, Munich, Germany.
  5. Thomas Eder: Department of Statistics, LMU Munich, Munich, Germany.
  6. Steffen Fohr: Department of Statistics, LMU Munich, Munich, Germany.
  7. Nico Hahn: Department of Statistics, LMU Munich, Munich, Germany.
  8. Rabea Hartmann: Department of Statistics, LMU Munich, Munich, Germany.
  9. Christoph Heindl: Department of Statistics, LMU Munich, Munich, Germany.
  10. Philipp Kopper: Department of Statistics, LMU Munich, Munich, Germany.
  11. Dario Lepke: Department of Statistics, LMU Munich, Munich, Germany.
  12. Verena Loidl: Department of Statistics, LMU Munich, Munich, Germany.
  13. Maximilian Mandl: Department of Statistics, LMU Munich, Munich, Germany.
  14. Sarah Musiol: Department of Statistics, LMU Munich, Munich, Germany. ORCID
  15. Jessica Peter: Department of Statistics, LMU Munich, Munich, Germany.
  16. Alexander Piehler: Department of Statistics, LMU Munich, Munich, Germany.
  17. Elio Rojas: Department of Statistics, LMU Munich, Munich, Germany.
  18. Stefanie Schmid: Department of Statistics, LMU Munich, Munich, Germany.
  19. Hannah Schmidt: Department of Statistics, LMU Munich, Munich, Germany.
  20. Melissa Schmoll: Department of Statistics, LMU Munich, Munich, Germany.
  21. Lennart Schneider: Department of Statistics, LMU Munich, Munich, Germany. ORCID
  22. Xiao-Yin To: Department of Statistics, LMU Munich, Munich, Germany.
  23. Viet Tran: Department of Statistics, LMU Munich, Munich, Germany.
  24. Antje Völker: Department of Statistics, LMU Munich, Munich, Germany.
  25. Moritz Wagner: Department of Statistics, LMU Munich, Munich, Germany.
  26. Joshua Wagner: Department of Statistics, LMU Munich, Munich, Germany.
  27. Maria Waize: Department of Statistics, LMU Munich, Munich, Germany.
  28. Hannah Wecker: Department of Statistics, LMU Munich, Munich, Germany.
  29. Rui Yang: Department of Statistics, LMU Munich, Munich, Germany.
  30. Simone Zellner: Department of Statistics, LMU Munich, Munich, Germany.
  31. Malte Nalenz: Department of Statistics, LMU Munich, Munich, Germany.
PMID: 34153038 DOI: 10.1371/journal.pone.0251194

Abstract

Computational reproducibility is a corner stone for sound and credible research. Especially in complex statistical analyses-such as the analysis of longitudinal data-reproducing results is far from simple, especially if no source code is available. In this work we aimed to reproduce analyses of longitudinal data of 11 articles published in PLOS ONE. Inclusion criteria were the availability of data and author consent. We investigated the types of methods and software used and whether we were able to reproduce the data analysis using open source software. Most articles provided overview tables and simple visualisations. Generalised Estimating Equations (GEEs) were the most popular statistical models among the selected articles. Only one article used open source software and only one published part of the analysis code. Replication was difficult in most cases and required reverse engineering of results or contacting the authors. For three articles we were not able to reproduce the results, for another two only parts of them. For all but two articles we had to contact the authors to be able to reproduce the results. Our main learning is that reproducing papers is difficult if no code is supplied and leads to a high burden for those conducting the reproductions. Open data policies in journals are good, but to truly boost reproducibility we suggest adding open code policies.

References

  1. PLoS One. 2018 Mar 9;13(3):e0194060 [PMID: 29522532]
  2. CPT Pharmacometrics Syst Pharmacol. 2019 Apr;8(4):205-210 [PMID: 30697975]
  3. PLoS One. 2018 Dec 18;13(12):e0209290 [PMID: 30562401]
  4. PLoS One. 2014 Nov 18;9(11):e112653 [PMID: 25405342]
  5. PLoS One. 2017 Jan 27;12(1):e0170120 [PMID: 28129336]
  6. PLoS One. 2020 May 27;15(5):e0233107 [PMID: 32459806]
  7. PLoS One. 2017 Mar 8;12(3):e0172213 [PMID: 28273162]
  8. PLoS One. 2015 Mar 17;10(3):e0120685 [PMID: 25781924]
  9. PLoS One. 2017 Feb 2;12(2):e0170891 [PMID: 28151986]
  10. PLoS One. 2015 Aug 28;10(8):e0136788 [PMID: 26317426]
  11. PLoS One. 2019 May 15;14(5):e0216125 [PMID: 31091251]
  12. PLoS One. 2016 Sep 29;11(9):e0162949 [PMID: 27685944]
  13. Psychol Methods. 2021 Oct;26(5):527-546 [PMID: 33180514]
  14. R Soc Open Sci. 2021 Apr 21;8(4):201925 [PMID: 33996122]
  15. R Soc Open Sci. 2021 Jan 6;8(1):201494 [PMID: 33614084]
  16. Proc Natl Acad Sci U S A. 2018 Mar 13;115(11):2584-2589 [PMID: 29531050]
  17. PLoS One. 2015 Apr 28;10(4):e0123812 [PMID: 25919116]
  18. Psychol Sci. 2011 Nov;22(11):1359-66 [PMID: 22006061]
  19. PLoS One. 2016 Jun 10;11(6):e0157148 [PMID: 27285824]
  20. PLoS One. 2018 Jan 10;13(1):e0190643 [PMID: 29320564]
  21. PLoS One. 2018 Jul 6;13(7):e0199789 [PMID: 29979709]

MeSH Term

Computational Biology
Data Analysis
Humans
Longitudinal Studies
Publications
Reproducibility of Results
Research Design
Software
Journal Article Research Support, Non-U.S. Gov't
Article has an altmetric score of 97

Word Cloud

Created with Highcharts 10.0.0articlesdataresultscodereproducereproducibilityanalysislongitudinalsourcesoftwareableopenstatisticalsimpleanalysespublishedPLOSONEusedonedifficultauthorstwopoliciesComputationalcornerstonesoundcredibleresearchEspeciallycomplexanalyses-suchdata-reproducingfarespeciallyavailableworkaimed11InclusioncriteriaavailabilityauthorconsentinvestigatedtypesmethodswhetherusingprovidedoverviewtablesvisualisationsGeneralisedEstimatingEquationsGEEspopularmodelsamongselectedarticlepartReplicationcasesrequiredreverseengineeringcontactingthreeanotherpartscontactmainlearningreproducingpaperssuppliedleadshighburdenconductingreproductionsOpenjournalsgoodtrulyboostsuggestaddingcomputationalstudyfeaturing

Similar Articles

Cited By