OpenLB
Open Library of Bioscience
Advanced Search
Research synthesis methods
Nov, 2024;15 (6): 872-895.

Checking the inventory: Illustrating different methods for individual participant data meta-analytic structural equation modeling.

Lennert J Groot, Kees-Jan Kan, Suzanne Jak
Author Information
  1. Lennert J Groot: University of Amsterdam, Amsterdam, The Netherlands. ORCID
  2. Kees-Jan Kan: University of Amsterdam, Amsterdam, The Netherlands. ORCID
  3. Suzanne Jak: University of Amsterdam, Amsterdam, The Netherlands. ORCID
PMID: 39138520 DOI: 10.1002/jrsm.1735

Abstract

Researchers may have at their disposal the raw data of the studies they wish to meta-analyze. The goal of this study is to identify, illustrate, and compare a range of possible analysis options for researchers to whom raw data are available, wanting to fit a structural equation model (SEM) to these data. This study illustrates techniques that directly analyze the raw data, such as multilevel and multigroup SEM, and techniques based on summary statistics, such as correlation-based meta-analytical structural equation modeling (MASEM), discussing differences in procedures, capabilities, and outcomes. This is done by analyzing a previously published collection of datasets using open source software. A path model reflecting the theory of planned behavior is fitted to these datasets using different techniques involving SEM. Apart from differences in handling of missing data, the ability to include study-level moderators, and conceptualization of heterogeneity, results show differences in parameter estimates and standard errors across methods. Further research is needed to properly formulate guidelines for applied researchers looking to conduct individual participant data MASEM.

Keywords

IPD meta‐analysis meta‐analytic structural equation modeling raw data synthesis structural equation modeling

References

  1. Munafò MR, Nosek BA, Bishop DV, et al. A manifesto for reproducible science. Nat Hum Behav. 2017;1(1):1‐9.
  2. Borgman CL. The conundrum of sharing research data. J Am Soc Inform Sci Technol. 2012;63(6):1059‐1078.
  3. Borgman CL. Big Data, Little Data, no Data: Scholarship in the Networked World. MIT Press; 2017.
  4. Glass GV. Meta‐analysis at 25. 2000 http://web.archive.org/web/20100616123146/http://glass.ed.asu.edu:80/gene/papers/meta25.html
  5. Chalmers I, Hedges LV, Cooper H. A brief history of research synthesis. Eval Health Prof. 2002;25(1):12‐37.
  6. Hussong AM, Curran PJ, Bauer DJ. Integrative data analysis in clinical psychology research. Annu Rev Clin Psychol. 2013;9:61‐89.
  7. Glasziou P, Altman DG, Bossuyt P, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014;383(9913):267‐276.
  8. Crowther MJ, Riley RD, Staessen JA, Wang J, Gueyffier F, Lambert PC. Individual patient data meta‐analysis of survival data using Poisson regression models. BMC Med Res Methodol. 2012;12(1):1‐14.
  9. Riley RD, Tierney JF, Stewart LA. Individual Participant Data Meta‐Analysis: A Handbook for Healthcare Research. John Wiley & Sons; 2021b.
  10. Stewart LA, Parmar MK. Meta‐analysis of the literature or of individual patient data: is there a difference? Lancet. 1993;341(8842):418‐422.
  11. McArdle JJ, Prescott CA, Hamagami F, Horn JL. A contemporary method for developmental‐genetic analyses of age changes in intellectual abilities. Dev Neuropsychol. 1998;14(1):69‐114.
  12. Hofer SM, Piccinin AM. Integrative data analysis through coordination of measurement and analysis protocol across independent longitudinal studies. Psychol Methods. 2009;14(2):150. doi:10.1037/a0015566
  13. Hagger MS, Cheung MW‐L, Ajzen I, Hamilton K. Perceived behavioral control moderating effects in the theory of planned behavior: a meta‐analysis. Health Psychol. 2022;41:155‐167. doi:10.1037/hea0001153
  14. Ajzen I. From intentions to actions: a theory of planned behavior. Action Control. Springer; 1985:11‐39.
  15. Ajzen I. The theory of planned behavior. Handbook of Theories of Social Psychology. Vol 1. SAGE Publications; 2012:438‐459.
  16. Bandura A. Social Foundations of Thought and Action. Vol 1986. Prentice‐Hall, Inc.; 1986.
  17. Ajzen I. The theory of planned behavior. Organ Behav Hum Decis Process. 1991;50(2):179‐211. doi:10.1016/0749‐5978(91)90020‐T
  18. De Leeuw J, Kreft I. Random coefficient models for multilevel analysis. J Educ Stat. 1986;11(1):57‐85.
  19. Hox JJ, Moerbeek M, van de Schoot R. Multilevel Analysis: Techniques and Applications. 3rd ed. Routledge; 2018.
  20. Burton P, Gurrin L, Sly P. Extending the simple linear regression model to account for correlated responses: an introduction to generalized estimating equations and multi‐level mixed modelling. Stat Med. 1998;17(11):1261‐1291.
  21. De Leeuw J, Kreft IG. Questioning multilevel models. J Educ Behav Stat. 1995;20(2):171‐189.
  22. Barcikowski RS. Statistical power with group mean as the unit of analysis. J Educ Stat. 1981;6(3):267‐285.
  23. Williams RL. A note on robust variance estimation for cluster‐correlated data. Biometrics. 2000;56(2):645‐646. http://www.jstor.org/stable/2677014
  24. Simpson EH. The interpretation of interaction in contingency tables. J R Stat Soc B Methodol. 1951;13(2):238‐241. http://www.jstor.org/stable/2984065
  25. Sprenger J, Weinberger N. Simpson's paradox. In: Zalta EN, ed. The Stanford Encyclopedia of Philosophy (Summer 2021). Metaphysics Research Lab, Stanford University; 2021.
  26. Cameron AC, Miller DL. A practitioner's guide to cluster‐robust inference. J Human Resour. 2015;50(2):317‐372.
  27. White H. Asymptotic Theory for Econometricians. Academic Press; 1984.
  28. Hox JJ, Moerbeek M, van de Schoot R. Multilevel factor models. Multilevel Analysis: Techniques and Applications. 3rd ed. Routledge; 2018.
  29. Hox JJ, Moerbeek M, van de Schoot R. Multilevel path models. Multilevel Analysis: Techniques and Applications. 3rd ed. Routledge; 2018.
  30. Meuleman B, Billiet J. A monte carlo sample size study: how many countries are needed for accurate multilevel sem? Surv Res Methods. 2009;3(1):45‐58.
  31. Maas CJ, Hox JJ. Sufficient sample sizes for multilevel modeling. Methodology. 2005;1(3):86‐92.
  32. McNeish D, Stapleton LM. Modeling clustered data with very few clusters. Multivar Behav Res. 2016;51(4):495‐518.
  33. Hox JJ, Moerbeek M, van de Schoot R. Assumptions and robust estimation methods. Multilevel Analysis: Techniques and Applications. 3rd ed. Routledge; 2018.
  34. Hedges LV, Vevea JL. Fixed‐and random‐effects models in meta‐analysis. Psychol Methods. 1998;3(4):486.
  35. Muthén BO. Latent variable modeling in heterogeneous populations. Psychometrika. 1989;54:557‐585.
  36. Muthén BO. Multilevel covariance structure analysis. Sociol Methods Res. 1994;22(3):376‐398.
  37. Schmidt WH. Covariance structure analysis of the multivariate random effects model [Doctoral dissertation]. The University of Chicago. 1969.
  38. Hox JJ. Multilevel Analysis: Techniques and Applications. Lawrence Erlbaum Associates; 2002.
  39. Ryu E, West SG. Level‐specific evaluation of model fit in multilevel structural equation modeling. Struct Eq Modeling. 2009;16(4):583‐601.
  40. Lüdtke O, Marsh HW, Robitzsch A, Trautwein U. A 2× 2 taxonomy of multilevel latent contextual models: accuracy–bias trade‐offs in full and partial error correction models. Psychol Methods. 2011;16(4):444.
  41. Riley RD, Jackson D, White IR. Multivariate meta‐analysis using IPD. In: Riley RD, Jackson D, White IR, eds. Individual Participant Data Meta‐Analysis: A Handbook for Healthcare Research. Wiley & Sons; 2021a:311‐346.
  42. Becker BJ, Wu M‐J. The synthesis of regression slopes in meta‐analysis. Stat Sci. 2007;22(3):414‐429.
  43. Valentine JC, Pigott TD, Rothstein HR. How many studies do you need?: a primer on statistical power for meta‐analysis. J Educ Behav Stat. 2010;35(2):215‐247. doi:10.3102/1076998609346961
  44. Cheung MW‐L, Cheung SF. Random‐effects models for meta‐analytic structural equation modeling: review, issues, and illustrations. Res Synth Methods. 2016;7(2):140‐155. doi:10.1002/jrsm.1166
  45. Hunter JE, Hamilton MA. The advantages of using standardized scores in causal analysis. Hum Commun Res. 2002;28(4):552‐561.
  46. Sheng Z, Kong W, Cortina JM, Hou S. Analyzing matrices of meta‐analytic correlations: current practices and recommendations. Res Synth Methods. 2016;7(2):187‐208. doi:10.1002/jrsm.1206
  47. Becker BJ, Schram C. Examining explanatory models through research synthesis. In: Cooper H, Hedges LV, eds. The Handbook of Research Synthesis. Russel Sage Foundation; 1994:357‐381.
  48. Cheung MW‐L. Fixed‐and random‐effects meta‐analytic structural equation modeling: examples and analyses in R. Behav Res Methods. 2014;46(1):29‐40. doi:10.3758/s13428‐013‐0361‐y
  49. Cheung MW‐L, Chan W. Meta‐analytic structural equation modeling: a two‐stage approach. Psychol Methods. 2005;10(1):40‐64.
  50. Viswesvaran C, Ones DS. Theory testing: combining psychometric meta‐analysis and structural equation modeling. Pers Psychol. 1995;48(4):865‐885. doi:10.1111/j.1744‐6570.1995.tb01784.x
  51. Jak S, Cheung MW‐L. Meta‐analytic structural equation modeling with moderating effects on SEM parameters. Psychol Methods. 2020;25(4):430‐455.
  52. Jak S, Li H, Kolbe L, de Jonge H, Cheung MW‐L. Meta‐analytic structural equation modeling made easy: a tutorial and web application for one‐stage MASEM. Res Synth Methods. 2021;12(5):590‐606.
  53. RStudio Team. RStudio: Integrated Development Environment for R. RStudio, PBC; 2022 http://www.rstudio.com/
  54. R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing; 2020.
  55. Rosseel Y. Lavaan: an R package for structural equation modeling. J Stat Softw. 2012;48(2):1‐36. doi:10.18637/jss.v048.i02
  56. Viechtbauer W. Conducting meta‐analyses in R with the metafor package. J Stat Softw. 2010;36(3):1‐48. doi:10.18637/jss.v036.i03
  57. Cheung MW‐L. Metasem: an R package for meta‐analysis using structural equation modeling. Front Psychol. 2015b;5(1521):1‐7. doi:10.3389/fpsyg.2014.01521
  58. Cole VT, Hussong AM, Gottfredson NC, Bauer DJ, Curran PJ. Informing harmonization decisions in integrative data analysis: exploring the measurement multiverse. Prev Sci. 2023;24:1595‐1607. doi:10.1007/s11121‐022‐01466‐1
  59. Yuan K, Bentler P. Three likelihood‐based methods for mean and covariance structure analysis with non‐normal missing data. Sociol Methods. 2000;30(1):165‐200.
  60. McArdle JJ, McDonald RP. Some algebraic properties of the reticular action model for moment structures. Br J Math Stat Psychol. 1984;37(2):234‐251.
  61. Steiger JH. Structural model evaluation and modification: an interval estimation approach. Multivar Behav Res. 1990;25(2):173‐180. doi:10.1207/s15327906mbr2502\_4
  62. Bentler PM. EQS Structural Equations Program Manual. Vol 6. Multivariate Software; 1995.
  63. Bentler PM. Comparative fit indexes in structural models. Psychol Bull. 1990;107(2):238‐246. doi:10.1037/0033‐2909.107.2.238
  64. Hu L‐t, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Model Multidiscip J. 1999;6(1):1‐55.
  65. Hu L‐t, Bentler PM. Fit indices in covariance structure modeling: sensitivity to underparameterized model misspecification. Psychol Methods. 1998;3(4):424.
  66. Steiger JH, Lind J. Statistically based tests for the number of common factors. Paper Presented at the Annual Meeting of the Psychometric Society, Iowa Cyty, 1980. 1980.
  67. Hamaker E. The curious case of the cross‐sectional correlation. Multivar Behav Res. 2022;1‐12. https://www.tandfonline.com/doi/full/10.1080/00273171.2022.2155930
  68. Jolani S, Debray TP, Koffijberg H, van Buuren S, Moons KG. Imputation of systematically missing predictors in an individual participant data meta‐analysis: a generalized approach using mice. Stat Med. 2015;34(11):1841‐1863.
  69. Resche‐Rigon M, White IR. Multiple imputation by chained equations for systematically and sporadically missing multilevel data. Stat Methods Med Res. 2018;27(6):1634‐1649.
  70. Schminkey DL, von Oertzen T, Bullock L. Handling missing data with multilevel structural equation modeling and full information maximum likelihood techniques. Res Nurs Health. 2016;39(4):286‐297.
  71. Hazlett C, Wainstein L. Understanding, choosing, and unifying multilevel and fixed effect approaches. Polit Anal. 2022;30(1):46‐65.
  72. Tipton E, Pustejovsky JE, Ahmadi H. A history of meta‐regression: technical, conceptual, and practical developments between 1974 and 2018. Res Synth Methods. 2019;10(2):161‐179.
  73. Cheung MW‐L. Meta‐Analysis: A Structural Equation Modeling Approach. 1st ed. John Wiley & Sons; 2015a.
  74. Costello AB, Osborne J. Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. Pract Assess Res Eval. 2005;10(1):7.
  75. MacCallum RC, Zhang S, Preacher KJ, Rucker DD. On the practice of dichotomization of quantitative variables. Psychol Methods. 2002;7(1):19.
  76. Hox JJ, Moerbeek M, van de Schoot R. The multilevel approach to meta‐analysis. Multilevel Analysis: Techniques and Applications. 3rd ed. Routledge; 2018.
  77. Bell A, Jones K. Explaining fixed effects: random effects modeling of time‐series cross‐sectional and panel data. Polit Sci Res Methods. 2015;3(1):133‐153. doi:10.1017/psrm.2014.7
  78. Schurer S, Yong J. Personality, well‐being and the marginal utility of income: What can we learn from random coefficient models? 2012.
  79. Simmonds M, Stewart G, Stewart LA. A decade of individual participant data meta‐analyses: a review of current practice. Contemp Clin Trials. 2015;45:76‐83.
  80. Bosnjak M, Ajzen I, Schmidt P. The theory of planned behavior: selected recent advances and applications. Eur J Psychol. 2020;16(3):352.
  81. Montazemi AR, Qahri‐Saremi H. Factors affecting adoption of online banking: a meta‐analytic structural equation modeling study. Inf Manag. 2015;52(2):210‐226.
  82. Scherer R, Siddiq F, Tondeur J. The technology acceptance model (tam): a meta‐analytic structural equation modeling approach to explaining teachers' adoption of digital technology in education. Comput Educ. 2019;128:13‐35.
  83. Cooper H, Patall E. The relative benefits of meta‐analysis conducted with individual participant data versus aggregated data. Psychol Methods. 2009;14(2):165‐176. doi:10.1037/a0015565
  84. Pigott T. Advances in Meta‐Analysis. Springer Science & Business Media; 2012.

Grants

  1. VI.Vidi.201.009/NWO

MeSH Term

Humans
Algorithms
Computer Simulation
Data Interpretation, Statistical
Meta-Analysis as Topic
Models, Statistical
Multilevel Analysis
Reproducibility of Results
Research Design
Software
Journal Article

Word Cloud

Created with Highcharts 10.0.0datastructuralequationrawmodelingSEMtechniquesdifferencesstudyresearchersmodelMASEMdatasetsusingdifferentmethodsindividualparticipantResearchersmaydisposalstudieswishmeta-analyzegoalidentifyillustratecomparerangepossibleanalysisoptionsavailablewantingfitillustratesdirectlyanalyzemultilevelmultigroupbasedsummarystatisticscorrelation-basedmeta-analyticaldiscussingprocedurescapabilitiesoutcomesdoneanalyzingpreviouslypublishedcollectionopensourcesoftwarepathreflectingtheoryplannedbehaviorfittedinvolvingAparthandlingmissingabilityincludestudy-levelmoderatorsconceptualizationheterogeneityresultsshowparameterestimatesstandarderrorsacrossresearchneededproperlyformulateguidelinesappliedlookingconductCheckinginventory:Illustratingmeta-analyticIPDmeta‐analysismeta‐analyticsynthesis

Similar Articles

Cited By

No available data.