OpenLB
Open Library of Bioscience
Advanced Search
Behavior research methods
08, 2022;54 (4): 1559-1579.

Estimation and statistical inferences of variance components in the analysis of single-case experimental design using multilevel modeling.

Haoran Li, Wen Luo, Eunkyeng Baek, Christopher G Thompson, Kwok Hap Lam
Author Information
  1. Haoran Li: Department of Educational Psychology, Texas A&M University, 4225 TAMU, College Station, TX, 77843-4225, USA. haoranli@tamu.edu. ORCID
  2. Wen Luo: Department of Educational Psychology, Texas A&M University, 4225 TAMU, College Station, TX, 77843-4225, USA.
  3. Eunkyeng Baek: Department of Educational Psychology, Texas A&M University, 4225 TAMU, College Station, TX, 77843-4225, USA.
  4. Christopher G Thompson: Department of Educational Psychology, Texas A&M University, 4225 TAMU, College Station, TX, 77843-4225, USA.
  5. Kwok Hap Lam: Department of Educational Psychology, Texas A&M University, 4225 TAMU, College Station, TX, 77843-4225, USA.
PMID: 34508288 DOI: 10.3758/s13428-021-01691-6

Abstract

Multilevel models (MLMs) can be used to examine treatment heterogeneity in single-case experimental designs (SCEDs). With small sample sizes, common issues for estimating between-case variance components in MLMs include nonpositive definite matrix, biased estimates, misspecification of covariance structures, and invalid Wald tests for variance components with bounded distributions. To address these issues, unconstrained optimization, model selection procedure based on parametric bootstrap, and restricted likelihood ratio test (RLRT)-based procedure are introduced. Using simulation studies, we compared the performance of two types of optimization methods (constrained vs. unconstrained) when the covariance structures are correctly specified or misspecified. We also examined the performance of a model selection procedure to obtain the optimal covariance structure. The results showed that the unconstrained optimization can avoid nonpositive definite issues to a great extent without a compromise in model convergence. The misspecification of covariance structures would cause biased estimates, especially with small between case variance components. However, the model selection procedure was found to attenuate the magnitude of bias. A practical guideline was generated for empirical researchers in SCEDs, providing conditions under which trustworthy point and interval estimates can be obtained for between-case variance components in MLMs, as well as the conditions under which the RLRT-based procedure can produce acceptable empirical type I error rate and power.

Keywords

Monte Carlo simulation Restricted likelihood ratio tests Single-case experimental design Variance components

References

  1. Alen, E., Grietens, H., & Van den Noortgate, W. (2009). Meta-analysis of single-case studies: An illustration for the treatment of anxiety disorders. Unpublished manuscript, Department of Educational Science and Psychology, University of Leuven.
  2. Baek, E., Beretvas, S. N., Van den Noortgate, W., & Ferron, J. M. (2020). Brief research report: Bayesian versus REML estimations with noninformative priors in multilevel single-case data. The Journal of Experimental Education, 88(4), 698−710. https://doi.org/10.1080/00220973.2018.1527280 [DOI: 10.1080/00220973.2018.1527280]
  3. Baek, E. K., & Ferron, J. M. (2013). Multilevel models for multiple-baseline data: Modeling across-participant variation in autocorrelation and residual variance. Behavior Research Methods, 45(1), 65−74. https://doi.org/10.3758/s13428-012-0231-z [DOI: 10.3758/s13428-012-0231-z]
  4. Baek, E., & Ferron, J. M. (2020). Modeling heterogeneity of the level-1 error covariance matrix in multilevel models for single-case data. Methodology, 16(2), 166−185. https://doi.org/10.5964/meth.2817 [DOI: 10.5964/meth.2817]
  5. Barlow, D. H., Nock, M. K., & Hersen, M. (2009). Single case experimental designs: Strategies for studying behavior change (3rd ed.). Allyn & Bacon.
  6. Carsey, T. M., & Harden, J. J. (2013). Monte Carlo simulation and resampling methods for social science. Sage Publications.
  7. Chung, Y., Gelman, A., Rabe-Hesketh, S., Liu, J., & Dorie, V. (2015). Weakly informative prior for point estimation of covariance matrices in hierarchical models. Journal of Educational and Behavioral Statistics, 40(2), 136−157. https://doi.org/10.3102/1076998615570945 [DOI: 10.3102/1076998615570945]
  8. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Erlbaum.
  9. Davison, A. C., & Hinkley, D. V. (1997). Bootstrap methods and their application. Cambridge University Press. [DOI: 10.1017/CBO9780511802843]
  10. Demidenko, E. (2013). Mixed models: theory and applications with R. John Wiley & Sons.
  11. Denis, J., Van den Noortgate, W., & Maes, B. (2011). Self-injurious behavior in people with profound intellectual disabilities: A meta-analysis of single-case studies. Research in Developmental Disabilities, 32, 911−923. https://doi.org/10.1016/j.ridd.2011.01.014 [DOI: 10.1016/j.ridd.2011.01.014]
  12. Ferron, J. M., Bell, B. A., Hess, M. R., Rendina-Gobioff, G., & Hibbard, S. T. (2009). Making treatment effect inferences from multiple-baseline data: The utility of multilevel modeling approaches. Behavior Research Methods, 41(2), 372−384. https://doi.org/10.3758/BRM.41.2.372 [DOI: 10.3758/BRM.41.2.372]
  13. Ferron, J. M., Farmer, J. L., & Owens, C. M. (2010). Estimating individual treatment effects from multiple-baseline data: A Monte Carlo study of multilevel-modeling approaches. Behavior Research Methods, 42(4), 930−943. https://doi.org/10.3758/BRM.42.4.930 [DOI: 10.3758/BRM.42.4.930]
  14. Gill, J., & King, G. (2004). What to do when your Hessian is not invertible: Alternatives to model respecification in nonlinear estimation. Sociological Methods & Research, 33(1), 54−87. https://doi.org/10.1177/0049124103262681 [DOI: 10.1177/0049124103262681]
  15. Hembry, I., Bunuan, R., Beretvas, S. N., Ferron, J. M., & Van den Noortgate, W. (2015). Estimation of a nonlinear intervention phase trajectory for multiple-baseline design data. The Journal of Experimental Education, 83(4), 514−546. https://doi.org/10.1080/00220973.2014.907231 [DOI: 10.1080/00220973.2014.907231]
  16. Hoffman, L. (2015). Longitudinal analysis: Modeling within-person fluctuation and change. Routledge.
  17. Joo, S. H., & Ferron, J. M. (2019). Application of the within-and between-series estimators to non-normal multiple-baseline data: Maximum likelihood and Bayesian approaches. Multivariate Behavioral Research, 54(5), 666−689. https://doi.org/10.1080/00273171.2018.1564877 [DOI: 10.1080/00273171.2018.1564877]
  18. Joo, S. H., Ferron, J. M., Moeyaert, M., Beretvas, S. N., & Van den Noortgate, W. (2019). Approaches for specifying the level-1 error structure when synthesizing single-case data. The Journal of Experimental Education, 87(1), 55−74. https://doi.org/10.1080/00220973.2017.1409181 [DOI: 10.1080/00220973.2017.1409181]
  19. Kenward, M. G., & Roger, J. H. (1997). Small sample inference for fixed effects from restricted maximum likelihood. Biometrics, 53(3), 983−997. https://doi.org/10.2307/2533558 [DOI: 10.2307/2533558]
  20. Kiernan, K. (2018). Insights into using the GLIMMIX procedure to model categorical outcomes with random effects. Cary: SAS Institute Inc. paper SAS2179.
  21. Kokina, A., & Kern, L. (2010). Social Story interventions for students with autism spectrum disorders: A meta-analysis. Journal of Autism and Developmental Disorders, 40, 812– 826. https://doi.org/10.1007/s10803-009-0931-0 [DOI: 10.1007/s10803-009-0931-0]
  22. Kratochwill, T. R., & Levin, J. R. (2010). Enhancing the scientific credibility of single-case intervention research: Randomization to the rescue. Psychological Methods, 15(2), 124−144. https://doi.org/10.1037/14376-003 [DOI: 10.1037/14376-003]
  23. Kratochwill, T. R., & Levin, J. R. (2014). Single-case intervention research: Methodological and statistical advances. American Psychological Association.
  24. Levin, J. R., Ferron, J. M., & Gafurov, B. S. (2018). Comparison of randomization-test procedures for single-case multiple-baseline designs. Developmental Neurorehabilitation, 21(5), 290−311. https://doi.org/10.1080/17518423.2016.1197708 [DOI: 10.1080/17518423.2016.1197708]
  25. Manolov, R., & Moeyaert, M. (2017). Recommendations for choosing single-case data analytical techniques. Behavior Therapy, 48(1), 97−114. https://doi.org/10.1016/j.beth.2016.04.008 [DOI: 10.1016/j.beth.2016.04.008]
  26. Matuschek, H., Kliegl, R., Vasishth, S., Baayen, H., & Bates, D. (2017). Balancing type I error and power in linear mixed models. Journal of Memory and Language, 94, 305−315. https://doi.org/10.1016/j.jml.2017.01.001 [DOI: 10.1016/j.jml.2017.01.001]
  27. McNeish, D., & Bauer, D. J. (2020). Reducing incidence of nonpositive definite covariance matrices in mixed effect models. Multivariate Behavioral Research. Advance online publication. https://doi.org/10.1080/00273171.2020.1830019
  28. Michiels, B., Tanious, R., De, T. K., & Onghena, P. (2020). A randomization test wrapper for synthesizing single-case experiments using multilevel models: A Monte Carlo simulation study. Behavior Research Methods, 52(2), 654−666. https://doi.org/10.3758/s13428-019-01266-6 [DOI: 10.3758/s13428-019-01266-6]
  29. Moeyaert, M., Ferron, J. M., Beretvas, S. N., & Van den Noortgate, W. (2014a). From a single-level analysis to a multilevel analysis of single-case experimental designs. Journal of School Psychology, 52(2), 191−211. https://doi.org/10.1016/j.jsp.2013.11.003 [DOI: 10.1016/j.jsp.2013.11.003]
  30. Moeyaert, M., Ugille, M., Ferron, J. M., Beretvas, S. N., & Van den Noortgate, W. (2013a). Modeling external events in the three-level analysis of multiple-baseline across-participants designs: A simulation study. Behavior Research Methods, 45(2), 547−559. https://doi.org/10.3758/s13428-012-0274-1 [DOI: 10.3758/s13428-012-0274-1]
  31. Moeyaert, M., Ugille, M., Ferron, J. M., Beretvas, S. N., & Van den Noortgate, W. (2013b). Three-level analysis of standardized single-case experimental data: Empirical validation. Multivariate Behavior Research, 48, 719−748. https://doi.org/10.1080/00220973.2012.745470 [DOI: 10.1080/00220973.2012.745470]
  32. Moeyaert, M., Ugille, M., Ferron, J. M., Beretvas, S. N., & Van den Noortgate, W. (2013c). The three-level synthesis of standardized single-subject experimental data: A Monte Carlo simulation study. Multivariate Behavioral Research, 48(5), 719−748. https://doi.org/10.1080/00273171.2013.816621 [DOI: 10.1080/00273171.2013.816621]
  33. Moeyaert, M., Ugille, M., Ferron, J. M., Beretvas, S. N., & Van den Noortgate, W. (2014b). Three-level analysis of single-case experimental data: Empirical validation. The Journal of Experimental Education, 82(1), 1−21. https://doi.org/10.1080/00220973.2012.745470 [DOI: 10.1080/00220973.2012.745470]
  34. Moeyaert, M., Ugille, M., Ferron, J. M., Beretvas, S. N., & Van den Noortgate, W. (2016). The misspecification of the covariance structures in multilevel models for single-case data: A Monte Carlo simulation study. The Journal of Experimental Education, 84(3), 473−509. https://doi.org/10.1080/00220973.2015.1065216 [DOI: 10.1080/00220973.2015.1065216]
  35. Moeyaert, M., Rindskopf, D., Onghena, P., & Van den Noortgate, W. (2017). Multilevel modeling of single-case data: A comparison of maximum likelihood and Bayesian estimation. Psychological Methods, 22(4), 760−778. https://doi.org/10.1037/met0000136 [DOI: 10.1037/met0000136]
  36. Pinheiro J., Bates D., DebRoy S., Sarkar D., & R Core Team (2019). nlme: Linear and Nonlinear Mixed Effects Models. R package version 3.1-142. https://CRAN.R-project.org/package=nlme
  37. Pinheiro, J. C., & Bates, D. M. (1996). Unconstrained parametrizations for variance-covariance matrices. Statistics and Computing, 6(3), 289−296. https://doi.org/10.1007/BF00140873 [DOI: 10.1007/BF00140873]
  38. Pustejovsky, J. E., Swan, D. M., & English, K. W. (2019). An examination of measurement procedures and characteristics of baseline outcome data in single-case research. Behavior Modification, Advance online publication. https://doi.org/10.1177/0145445519864264
  39. Satterthwaite, F. E. (1946). An approximate distribution of estimates of variance components. Biometrics Bulletin, 2(6), 110−114. https://doi.org/10.2307/3002019 [DOI: 10.2307/3002019]
  40. SAS Institute. (2017). SAS/STAT® 14.3 user’s guide.
  41. Self, S. G., & Liang, K.-Y. (1987). Asymptotic properties of maximum likelihood estimators and likelihood ratio tests under nonstandard conditions. Journal of the American Statistical Association, 82(398), 605−610. https://doi.org/10.2307/2289471 [DOI: 10.2307/2289471]
  42. Shadish, W. R., Kyse, E. N., & Rindskopf, D. M. (2013a). Analyzing data from single-case designs using multilevel models: New applications and some agenda items for future research. Psychological Methods, 18(3), 385−405. https://doi.org/10.1037/a0032964 [DOI: 10.1037/a0032964]
  43. Shadish, W. R., & Rindskopf, D. M. (2007). Methods for evidence-based practice: Quantitative synthesis of single-subject designs. New Directions for Evaluation, 2007(113), 95−109. https://doi.org/10.1002/ev.217 [DOI: 10.1002/ev.217]
  44. Shadish, W. R., Rindskopf, D. M., Hedges, L. V., & Sullivan, K. J. (2013b). Bayesian estimates of autocorrelations in single-case designs. Behavior Research Methods, 45(3), 813−821. https://doi.org/10.3758/s13428-012-0282-1 [DOI: 10.3758/s13428-012-0282-1]
  45. Shadish, W. R., & Sullivan, K. J. (2011). Characteristics of single-case designs used to assess intervention effects in 2008. Behavior Research Methods, 43(4), 971−980. https://doi.org/10.3758/s13428-011-0111-y [DOI: 10.3758/s13428-011-0111-y]
  46. Shogren, K. A., Fagella-Luby, M. N., Bae, S. J., & Wehmeyer, M. L. (2004). The effect of choice-making as an intervention for problem behavior. Journal of Positive Behavior Interventions, 6, 228−237. https://doi.org/10.1177/10983007040060040401 [DOI: 10.1177/10983007040060040401]
  47. Sideridis, G. D., & Greenwood, C. R. (1997). Is human behavior autocorrelated? An empirical analysis. Journal of Behavioral Education, 7(3), 273−293. https://doi.org/10.1023/A:1022895805201 [DOI: 10.1023/A]
  48. Stram, D. O., & Lee, J. W. (1994). Variance components testing in the longitudinal mixed effects model. Biometrics, 50(4) 1171−1177. https://doi.org/10.2307/2533455 [DOI: 10.2307/2533455]
  49. Tsai, M. Y., & Hsiao, C. K. (2008). Computation of reference Bayesian inference for variance components in longitudinal studies. Computational Statistics, 23(4), 587−604. https://doi.org/10.1007/s00180-007-0100-x [DOI: 10.1007/s00180-007-0100-x]
  50. Van den Noortgate, W., & Onghena, P. (2003a). Combining single-case experimental data using hierarchical linear models. School Psychology Quarterly, 18(3), 325−346. https://doi.org/10.1521/scpq.18.3.325.22577 [DOI: 10.1521/scpq.18.3.325.22577]
  51. Van Den Noortgate, W., & Onghena, P. (2003b). Hierarchical linear models for the quantitative integration of effect sizes in single-case research. Behavior Research Methods, Instruments, & Computers, 35(1), 1−10. https://doi.org/10.3758/BF03195492 [DOI: 10.3758/BF03195492]
  52. Van den Noortgate, W., & Onghena, P. (2007). The aggregation of single-case results using hierarchical linear models. The Behavior Analyst Today, 8(2), 196−209. https://doi.org/10.1037/h0100613 [DOI: 10.1037/h0100613]
  53. Wang, S., Cui, Y., & Parrila, R. (2011). Examining the effectiveness of peer-mediated and video-modeling social skills interventions for children with autism spectrum disorders: A meta-analysis in single-case research using HLM. Research in Autism Spectrum Disorders, 5, 562–569. https://doi.org/10.1016/j.rasd.2010.06.023 [DOI: 10.1016/j.rasd.2010.06.023]
  54. West, B. T., Welch, K. B., & Galecki, A. T. (2014). Linear mixed models: a practical guide using statistical software. Chapman and Hall/CRC. [DOI: 10.1201/b17198]
  55. Wiencierz, A., Greven, S., & Küchenhoff, H. (2011). Restricted likelihood ratio testing in linear mixed models with general error covariance structure. Electronic Journal of Statistics, 5, 1718−1734. https://doi.org/10.1214/11-EJS654 [DOI: 10.1214/11-EJS654]
  56. Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3), 255–278. https://doi.org/10.1016/j.jml.2012.11.001

MeSH Term

Computer Simulation
Humans
Likelihood Functions
Models, Statistical
Multilevel Analysis
Research Design
Journal Article
Article has an altmetric score of 2

Word Cloud

Created with Highcharts 10.0.0componentsvarianceprocedurecancovariancemodelMLMsexperimentalissuesestimatesstructuresunconstrainedoptimizationselectionsingle-caseSCEDssmallbetween-casenonpositivedefinitebiasedmisspecificationtestslikelihoodratiosimulationperformanceempiricalconditionsdesignMultilevelmodelsusedexaminetreatmentheterogeneitydesignssamplesizescommonestimatingincludematrixinvalidWaldboundeddistributionsaddressbasedparametricbootstraprestrictedtestRLRT-basedintroducedUsingstudiescomparedtwotypesmethodsconstrainedvscorrectlyspecifiedmisspecifiedalsoexaminedobtainoptimalstructureresultsshowedavoidgreatextentwithoutcompromiseconvergencecauseespeciallycaseHoweverfoundattenuatemagnitudebiaspracticalguidelinegeneratedresearchersprovidingtrustworthypointintervalobtainedwellRLRT-basedproduceacceptabletypeerrorratepowerEstimationstatisticalinferencesanalysisusingmultilevelmodelingMonteCarloRestrictedSingle-caseVariance

Similar Articles

Cited By