OpenLB
Open Library of Bioscience
Advanced Search
Behavior research methods
02, 2020;52 (1): 177-192.

MultiSCED: A tool for (meta-)analyzing single-case experimental data with multilevel modeling.

Lies Declercq, Wilfried Cools, S Natasha Beretvas, Mariola Moeyaert, John M Ferron, Wim Van den Noortgate
Author Information
  1. Lies Declercq: Faculty of Psychology and Educational Sciences, imec - ITEC, KU Leuven, Leuven, Belgium. lies.declercq@kuleuven.be.
  2. Wilfried Cools: Faculty of Psychology and Educational Sciences, imec - ITEC, KU Leuven, Leuven, Belgium.
  3. S Natasha Beretvas: Department of Educational Psychology, University of Texas, Austin, TX, 78712, USA.
  4. Mariola Moeyaert: Department of Educational Psychology and Methodology, University at Albany-State University of New York, Albany, NY, USA.
  5. John M Ferron: Department of Educational Measurement and Research, University of South Florida, Tampa, FL, 33620, USA.
  6. Wim Van den Noortgate: Faculty of Psychology and Educational Sciences, imec - ITEC, KU Leuven, Leuven, Belgium.
PMID: 30972557 DOI: 10.3758/s13428-019-01216-2

Abstract

The MultiSCED web application has been developed to assist applied researchers in behavioral sciences to apply multilevel modeling to quantitatively summarize single-case experimental design (SCED) studies through a user-friendly point-and-click interface embedded within R. In this paper, we offer a brief introduction to the application, explaining how to define and estimate the relevant multilevel models and how to interpret the results numerically and graphically. The use of the application is illustrated through a re-analysis of an existing meta-analytic dataset. By guiding applied researchers through MultiSCED, we aim to make use of the multilevel modeling technique for combining SCED data across cases and across studies more comprehensible and accessible.

Keywords

Multilevel analysis R SCED Shiny Single-case experimental design

References

  1. Auerbach, C., & Schudrich, W. Z. (2013). SSD for R: A comprehensive statistical package to analyze single-system data. Research on Social Work Practice, 23(3), 346–353. https://doi.org/10.1177/1049731513477213 [DOI: 10.1177/1049731513477213]
  2. Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4), 390–412. https://doi.org/10.1016/j.jml.2007.12.005 [DOI: 10.1016/j.jml.2007.12.005]
  3. Baek, E. K., & Ferron, J. M. (2013). Multilevel models for multiple-baseline data: Modeling across-participant variation in autocorrelation and residual variance. Behavior Research Methods, 45(1), 65–74. https://doi.org/10.3758/s13428-012-0231-z [DOI: 10.3758/s13428-012-0231-z]
  4. Baek, E. K., et al. (2016). Using visual analysis to evaluate and refine multilevel models of single-case studies. Journal of Special Education, 50(1), 18–26. https://doi.org/10.1177/0022466914565367 [DOI: 10.1177/0022466914565367]
  5. Bates, D. (2006). lmer, p-values and all that. https://stat.ethz.ch/pipermail/r-help/2006-May/094765.html
  6. Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01 . [DOI: 10.18637/jss.v067.i01]
  7. Bolker, B., et al. (2018). GLMM FAQ. http://bbolker.github.io/mixedmodels-misc/glmmFAQ.html
  8. Bulté, I., & Onghena, P. (2013). The single-case data analysis package: Analysing single-case experiments with R software. Journal of Modern Applied Statistical Methods, 12(2), 450–478. https://doi.org/10.1017/CBO9781107415324.004 [DOI: 10.1017/CBO9781107415324.004]
  9. Chang, W., et al. (2017). Shiny: Web application framework for R. https://cran.r-project.org/package=shiny
  10. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale: Erlbaum.
  11. Declercq, L., et al. (2018). Analysis of single-case experimental count data using the linear mixed effects model: A simulation study. Behavior Research Methods. https://doi.org/10.3758/s13428-018-1091-y [DOI: 10.3758/s13428-018-1091-y]
  12. Dibley, S., & Lim, L. (1999). Providing choice making opportunities within and between daily school routines. Journal of Behavioral Education, 9 (2), 117–132. https://doi.org/10.1023/A:1022888917128 [DOI: 10.1023/A]
  13. Ferron, J. M., et al. (2009). Making treatment effect inferences from multiple-baseline data: The utility of multilevel modeling approaches. Behavior Research Methods, 41(2), 372–384. https://doi.org/10.3758/BRM.41.2.372 [DOI: 10.3758/BRM.41.2.372]
  14. Ferron, J. M., Farmer, J. L., & Owens, C. M. (2010). Estimating individual treatment effects from multiple-baseline data: A Monte Carlo study of multilevel-modeling approaches. Behavior Research Methods, 42(4), 930–943. https://doi.org/10.3758/BRM.42.4.930 [DOI: 10.3758/BRM.42.4.930 https]
  15. Fisch, G. S. (2001). Evaluating data from behavioral analysis: Visual inspection or statistical models? Behavioural Processes, 54(1-3), 137–154. https://doi.org/10.1016/S0376-6357(01)00155-3 [DOI: 10.1016/S0376-6357(01)00155-3]
  16. Jamshidi, L., et al. (2018). Review of single-subject experimental design meta-analyses and reviews: 1985-2015. Submitted for publication.
  17. Joo, S. H., et al. (2017). Approaches for specifying the level-1 error structure when synthesizing single-case data. Journal of Experimental Education, 0973, 1–20. https://doi.org/10.1080/00220973.2017.1409181 [DOI: 10.1080/00220973.2017.1409181 https]
  18. Kenward, M. G., & Roger, J. H. (1997). Small sample inference for fixed effects from restricted maximum likelihood. Biometrics, 53(3), 983–997. https://doi.org/10.2307/2533558 [DOI: 10.2307/2533558]
  19. Kern, L., et al. (2001). Choice of task sequence to reduce problem behaviors. Journal of Positive Behavior Interventions, 3(1), 3– 10. https://doi.org/10.1177/109830070100300102 . http://search.ebscohost.com/login.aspx?direct=true%7B%5C&%7Ddb=psyh%7B%5C&%7DAN=2001-14259-001%7B%5C&%7Dsite=ehostlive [DOI: 10.1177/109830070100300102]
  20. Littell, R. C., et al. (2007). SAS for mixed models. 2nd ed., 2. Cary: SAS Institute.
  21. Luke, S. G. (2017). Evaluating significance in linear mixed-effects models in R. Behavior Research Methods, 49(4), 1494–1502. https://doi.org/10.3758/s13428-016-0809-y [DOI: 10.3758/s13428-016-0809-y]
  22. Manolov, R., & Solanas, A. (2013). A comparison of mean phase difference and generalized least squares for analyzing single-case data. Journal of School Psychology, 51(2), 201–215. https://doi.org/10.1016/j.jsp.2012.12.005 [DOI: 10.1016/j.jsp.2012.12.005]
  23. Manolov, R., & Rochat, L. (2015). Further developments in summarising and meta-analysing single-case data: An illustration with neurobehavioural interventions in acquired brain injury. Neuropsychological Rehabilitation, 25 (5), 637–662. https://doi.org/10.1080/09602011.2015.1064452 [DOI: 10.1080/09602011.2015.1064452]
  24. Manolov, R., & Moeyaert, M. (2017). How can single-case data be analyzed? Software resources, tutorial, and reflections on analysis. Behavior Modification, 41(2), 179–228. https://doi.org/10.1177/0145445516664307 [DOI: 10.1177/0145445516664307]
  25. Manolov, R., & Solanas, A. (2018). Quantifying differences between conditions in single-case designs: Possible analysis and meta-analysis. Developmental Neurorehabilitation, 21(4), 238–252. https://doi.org/10.3109/17518423.2015.1100688 [DOI: 10.3109/17518423.2015.1100688]
  26. Moes, D. R. (1998). Integrating choice-making opportunities within teacher-assigned academic tasks to facilitate the performance of children with autism. Journal of the Association for Persons with Severe Handicaps, 23(4), 319–328. [DOI: 10.2511/rpsd.23.4.319]
  27. Moeyaert, M., et al. (2013). The three-level synthesis of standardized single-subject experimental data: A Monte Carlo simulation study. Multivariate Behavioral Research, 48(5), 719–748. https://doi.org/10.1080/00273171.2013.816621 [DOI: 10.1080/00273171.2013.816621]
  28. Moeyaert, M., et al. (2014). From a single-level analysis to a multilevel analysis of single-case experimental designs. Journal of School Psychology, 52(2), 191–211. https://doi.org/10.1016/j.jsp.2013.11.003 [DOI: 10.1016/j.jsp.2013.11.003]
  29. Moeyaert, M., et al. (2015). Estimating intervention effects across different types of single-subject experimental designs: Empirical illustration. School Psychology Quarterly, 30(1), 50–63. https://doi.org/10.1037/spq0000068 [DOI: 10.1037/spq0000068]
  30. Moeyaert, M., et al. (2017). Multilevel modeling of single-case data: A comparison of maximum likelihood and Bayesian estimation. Psychological Methods, 22(4), 760–778. https://doi.org/10.1037/met0000136 [DOI: 10.1037/met0000136]
  31. Nagler, E., Rindskopf, D. M., & Shadish, W. R. (2008). Analyzing data from small N designs using multilevel models: A procedural handbook. Unpublished manuscript.
  32. Onghena, P., & Edgington, E. S. (2005). Customization of pain treatments: Single-case design and analysis. The Clinical Journal of Pain, 21(1), 56–68. https://doi.org/10.1097/00002508-200501000-00007 [DOI: 10.1097/00002508-200501000-00007]
  33. Owens, C. M., & Ferron, J. M. (2012). Synthesizing single-case studies: A Monte Carlo examination of a three-level meta-analytic model. Behavior Research Methods, 44(3), 795–805. https://doi.org/10.3758/s13428-011-0180-y [DOI: 10.3758/s13428-011-0180-y]
  34. Parsonson, B. S., & Baer, D. M. (1992). The visual analysis of data, and current research into the stimuli controlling it. In T.R. Kratochwill, J.R. Levin, & N. J. Hillsdale (Eds.) Single-case research design and analysis: New directions for psychology and education. Erlbaum. Chap. 2 (pp. 15–38).
  35. Peugh, J. L., & Enders, C. K. (2005). Using the SPSS mixed procedure to fit cross-sectional and longitudinal multilevel models. Educational and Psychological Measurement, 65(5), 717–741. https://doi.org/10.1177/0013164405278558 [DOI: 10.1177/0013164405278558]
  36. Pinheiro, J., et al. (2018). nlme: Linear and nonlinear mixed effects models. R package version 3. pp. 1-137. https://CRAN.R-project.org/package=nlme
  37. Pustejovsky, J. E. (2016) scdhlm: Estimating hierarchical linear models for single-case designs. R package version 0.3. Austin: University of Texas. http://github.com/jepusto/scdhlm
  38. R Core Team (2013). R: A language and environment for statistical computing. Vienna, Austria. http://www.r-project.org/
  39. Rasbash, J., et al. (2009). A user’s guide to MLwiN, Version 2.10, pp. 1–296. http://www.bristol.ac.uk/media-library/sites/cmm/migrated/documents/mlwin-userman-09.pdf
  40. Raudenbush, S. W., Bryk, A. S., & Congdon, R. (2013). HLM 7.01 for Windows [computer software]. Skokie, IL.
  41. Rindskopf, D. M., & Ferron, J. M. (2014). Using multilevel models to analyze single-case design data. In R.T. Kratochwill, & J.R. Levin (Eds.) Single-case intervention research: Methodological and statistical advances. School psychology series. Washington, DC, US: American Psychological Association, pp. 221–246. https://doi.org/10.1037/14376-008
  42. Rodabaugh, E., & Moeyaert, M. (2017). Multilevel modeling of single-case data: An introduction and tutorial for the applied researcher. In NERA Conference Proceedings 2017 8.
  43. Romaniuk, C., et al. (2002). The influence of activity choice on problem behaviors maintained by escape versus attention. Journal of Applied Behavior Analysis, 35(4), 349–362. https://doi.org/10.1901/jaba.2002.35-349 . http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1284398/ [DOI: 10.1901/jaba.2002.35-349]
  44. Shadish, W. R. (2014). Statistical analyses of single-case designs: The shape of things to come. Current Directions in Psychological Science, 23(2), 139–146. https://doi.org/10.1177/0963721414524773 [DOI: 10.1177/0963721414524773]
  45. Shadish, W. R., & Sullivan, K. J. (2011). Characteristics of single-case designs used to assess intervention effects in 2008. Behavior Research Methods, 43(4), 971–980. https://doi.org/10.3758/s13428-011-0111-y [DOI: 10.3758/s13428-011-0111-y]
  46. Shadish, W. R., Rindskopf, D. M., & Hedges, L. V. (2008). The state of the science in the meta-analysis of single-case experimental designs. Evidence-Based Communication Assessment and Intervention, 2(3), 188–196. https://doi.org/10.1080/17489530802581603 . http://search.ebscohost.com/login.aspx?direct=true%7B%5C&%7Dlang=zh-tw%7B%5C&%7Dsite=ehost-live [DOI: 10.1080/17489530802581603]
  47. Shadish, W. R., Kyse, E. N., & Rindskopf, D. M. (2013). Analyzing data from single-case designs using multilevel models: New applications and some agenda items for future research. Psychological Methods, 18(3), 385–405. https://doi.org/10.1037/a0032964 [DOI: 10.1037/a0032964]
  48. Shogren, K. A., Faggella-Luby, M. N., Bae, S. J., & Wehmeyer, M. L. (2004). The effect of choice-making as an intervention for problem behavior: A meta-analysis. Journal of Positive Behavior Interventions, 6(4), 228–237. https://doi.org/10.1177/10983007040060040401 [DOI: 10.1177/10983007040060040401]
  49. Ugille, M., et al. (2012). Multilevel meta-analysis of single-subject experimental designs: A simulation study. Behavior Research Methods, 44(4), 1244–1254. https://doi.org/10.3758/s13428-012-0213-1 [DOI: 10.3758/s13428-012-0213-1]
  50. Valentine, J. C., et al. (2016). Between-case standardized mean difference effect sizes for single-case designs: A primer and tutorial using the scdhlm web application. Oslo, Norway: The Campbell Collaboration. https://doi.org/10.4073/cmdp.2016.1 . https://campbellcollaboration.org [DOI: 10.4073/cmdp.2016.1]
  51. Van den Noortgate, W., et al. (2014). Meta-analysis of multiple outcomes: A multilevel approach. Behavior Research Methods, 47(4), 1274–1294. https://doi.org/10.3758/s13428-014-0527-2 [DOI: 10.3758/s13428-014-0527-2]
  52. Van den Noortgate, W., & Onghena, P. (2003a). Combining single-case experimental data using hierarchical linear models. School Psychology Quarterly, 18(3), 325–346. https://doi.org/10.1521/scpq.18.3.325.22577 [DOI: 10.1521/scpq.18.3.325.22577 https]
  53. Van den Noortgate, W., & Onghena, P. (2003b). Hierarchical linear models for the quantitative integration of effect sizes in single-case research. Behavior Research Methods, Instruments, and Computers, 35(1), 1–10. https://doi.org/10.3758/BF03195492 [DOI: 10.3758/BF03195492]
  54. Van den Noortgate, W., & Onghena, P. (2008). A multilevel meta-analysis of single-subject experimental design studies. Evidence-Based Communication Assessment and Intervention, 2 (3), 142–151. https://doi.org/10.1080/17489530802505362 [DOI: 10.1080/17489530802505362]
  55. Welham, S. J., & Thompson, R. (1997). Likelihood ratio tests for fixed model terms using residual maximum likelihood. Journal of the Royal Statistical Society, Series B: Statistical Methodology, 59(3), 701–714. https://doi.org/10.2307/2346019 [DOI: 10.2307/2346019]
  56. Wickham, H. (2009) ggplot2: Elegant graphics for data analysis. New York: Springer. ISBN: 978-0-387-98140-6. http://ggplot2.org . [DOI: 10.1007/978-0-387-98141-3]
  57. Wickham, H. (2011). The split-apply-combine strategy for data analysis. Journal of Statistical Software, 40 (1), 1–29. http://www.jstatsoft.org/v40/i01/ [DOI: 10.18637/jss.v040.i01]

MeSH Term

Behavioral Sciences
Multilevel Analysis
Research Design
Journal Article
Article has an altmetric score of 9

Word Cloud

Created with Highcharts 10.0.0multilevelapplicationmodelingexperimentalSCEDMultiSCEDappliedresearcherssingle-casedesignstudiesRusedataacrosswebdevelopedassistbehavioralsciencesapplyquantitativelysummarizeuser-friendlypoint-and-clickinterfaceembeddedwithinpaperofferbriefintroductionexplainingdefineestimaterelevantmodelsinterpretresultsnumericallygraphicallyillustratedre-analysisexistingmeta-analyticdatasetguidingaimmaketechniquecombiningcasescomprehensibleaccessibleMultiSCED:toolmeta-analyzingMultilevelanalysisShinySingle-case

Similar Articles

Cited By