OpenLB
Open Library of Bioscience
Advanced Search
Psychometrika
12, 2024;89 (4): 1230-1260.

Asymptotically Correct Person Fit z-Statistics For the Rasch Testlet Model.

Zhongtian Lin, Tao Jiang, Frank Rijmen, Paul Van Wamelen
Author Information
  1. Zhongtian Lin: Financial Industry Regulatory Authority, Washington, USA. lzt713@gmail.com. ORCID
  2. Tao Jiang: Cambium Assessment, Washington, USA.
  3. Frank Rijmen: Cambium Assessment, Washington, USA.
  4. Paul Van Wamelen: Cambium Assessment, Washington, USA.
PMID: 39153026 DOI: 10.1007/s11336-024-09997-y

Abstract

A well-known person fit statistic in the item response theory (IRT) literature is the statistic (Drasgow et al. in Br J Math Stat Psychol 38(1):67-86, 1985). Snijders (Psychometrika 66(3):331-342, 2001) derived , which is the asymptotically correct version of when the ability parameter is estimated. However, both statistics and other extensions later developed concern either only the unidimensional IRT models or multidimensional models that require a joint estimate of latent traits across all the dimensions. Considering a marginalized maximum likelihood ability estimator, this paper proposes and , which are extensions of and , respectively, for the Rasch testlet model. The computation of relies on several extensions of the Lord-Wingersky algorithm (1984) that are additional contributions of this paper. Simulation results show that has close-to-nominal Type I error rates and satisfactory power for detecting aberrant responses. For unidimensional models, and reduce to and , respectively, and therefore allows for the evaluation of person fit with a wider range of IRT models. A real data application is presented to show the utility of the proposed statistics for a test with an underlying structure that consists of both the traditional unidimensional component and the Rasch testlet component.

Keywords

statistic IRT Person fit Rasch testlet model

References

Albers, C. J., Meijer, R. R., & Tendeiro, J. N. (2016). Derivation and applicability of asymptotic results for multiple subtests person-fit statistics. Applied Psychological Measurement, 40(4), 274–288. [DOI: 10.1177/0146621615622832]
Bedrick, E. J. (1997). Approximating the conditional distribution of person fit indexes for checking the Rasch model. Psychometrika, 62(2), 191–199. [DOI: 10.1007/BF02295274]
Bradlow, E. T., Wainer, H., & Wang, X. (1999). A Bayesian random effects model for testlets. Psychometrika, 64(2), 153–168. [DOI: 10.1007/BF02294533]
Cai, L. (2015). Lord-Wingersky algorithm version 2.0 for hierarchical item factor models with applications in test scoring, scale alignment, and model fit testing. Psychometrika, 80(2), 535–559. [DOI: 10.1007/s11336-014-9411-3]
Chen, H. (2013). Testlet Effects on Standardized Log-likelihood Person Fit Index to Detect Aberrant Responses for the IRT Testlet Model (Doctoral dissertation, University of Missouri–Columbia).
De La Torre, J., & Deng, W. (2008). Improving person-fit assessment by correcting the ability estimate and its reference distribution. Journal of Educational Measurement, 45(2), 159–177. [DOI: 10.1111/j.1745-3984.2008.00058.x]
Drasgow, F., Levine, M. V., & Williams, E. A. (1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38(1), 67–86. [DOI: 10.1111/j.2044-8317.1985.tb00817.x]
Glas, C. A. W., & Dagohoy, A. V. T. (2007). A person fit test for IRT models for polytomous items. Psychometrika, 72(2), 159–180. [DOI: 10.1007/s11336-003-1081-5]
Gorney, K., Sinharay, S., Eckerly, C. (2024). Efficient corrections for standardized person-fit statistics. Psychometrika, 1–23.
Hong, M., Lin, L., & Cheng, Y. (2021). Asymptotically corrected person fit statistics for multidimensional constructs with simple structure and mixed item types. Psychometrika, 86(2), 464–488. [DOI: 10.1007/s11336-021-09756-3]
Karabatsos, G. (2003). Comparing the aberrant response detection performance of thirty-six person-fit statistics. Applied Measurement in Education, 16(4), 277–298. [DOI: 10.1207/S15324818AME1604_2]
Liou, M., & Chang, C. H. (1992). Constructing the exact significance level for a person fit statistic. Psychometrika, 57(2), 169–181. [DOI: 10.1007/BF02294503]
Lord, F. M., & Wingersky, M. S. (1984). Comparison of IRT true-score and equipercentile observed-score “equatings’’. Applied Psychological Measurement, 8(4), 453–461. [DOI: 10.1177/014662168400800409]
Magis, D., Raîche, G., & Béland, S. (2012). A didactic presentation of Snijders’s lz* index of person fit with emphasis on response model selection and ability estimation. Journal of Educational and Behavioral Statistics, 37(1), 57–81. [DOI: 10.3102/1076998610396894]
Meijer, R. R., & Sijtsma, K. (2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25(2), 107–135. [DOI: 10.1177/01466210122031957]
Molenaar, I. W., & Hoijtink, H. (1990). The many null distributions of person fit indices. Psychometrika, 55(1), 75–106. [DOI: 10.1007/BF02294745]
Nering, M. L. (1995). The distribution of person fit using true and estimated person parameters. Applied Psychological Measurement, 19(2), 121–129. [DOI: 10.1177/014662169501900201]
New Hampshire Department of Education (2019). New hampshire statewide assessment system 2018-2019 annual technical report volume 1. https://www.education.nh.gov/sites/g/files/ehbemt326/files/inline-documents/sonh/nhsas-v1-tech-report-2018-19.pdf
Reise, S. P. (1995). Scoring method and the detection of person misfit in a personality assessment context. Applied Psychological Measurement, 19(3), 213–229. [DOI: 10.1177/014662169501900301]
Rijmen, F., Turhan, A., Jiang, T. (2018). An item response theory model for next generation of science standards assessments. National Council of Measurement in Education Annual Conference, New York, NY.
Rupp, A. A. (2013). A systematic review of the methodology for person fit research in item response theory: Lessons about generalizability of inferences from the design of simulation studies. Psychological Test and Assessment Modeling, 55(1), 3.
Seo, D. G., & Weiss, D. J. (2013). lz Person-fit index to identify misfit students with achievement test data. Educational and Psychological Measurement, 73(6), 994–1016. [DOI: 10.1177/0013164413497015]
Sinharay, S. (2015). Assessment of person fit for mixed-format tests. Journal of Educational and Behavioral Statistics, 40(4), 343–365. [DOI: 10.3102/1076998615589128]
Sinharay, S. (2016). Asymptotically correct standardization of person-fit statistics beyond dichotomous items. Psychometrika, 81(4), 992–1013. [DOI: 10.1007/s11336-015-9465-x]
Sireci, S. G., Thissen, D., & Wainer, H. (1991). On the reliability of testlet-based tests. Journal of Educational Measurement, 28(3), 237–247. [DOI: 10.1111/j.1745-3984.1991.tb00356.x]
Snijders, T. A. (2001). Asymptotic null distribution of person fit statistics with estimated person parameter. Psychometrika, 66(3), 331–342. [DOI: 10.1007/BF02294437]
van Krimpen-Stoop, E. M., & Meijer, R. R. (1999). The null distribution of person-fit statistics for conventional and adaptive tests. Applied Psychological Measurement, 23(4), 327–345. [DOI: 10.1177/01466219922031446]
von Davier, M., & Molenaar, I. W. (2003). A person-fit index for polytomous Rasch models, latent class models, and their mixture generalizations. Psychometrika, 68(2), 213–228. [DOI: 10.1007/BF02294798]
Wainer, H., & Lukhele, R. (1997). How reliable are TOEFL scores? Educational and Psychological Measurement, 57(5), 741–758. [DOI: 10.1177/0013164497057005002]
Wainer, H., & Thissen, D. (1996). How is reliability related to the quality of test scores? What is the effect of local dependence on reliability? Educational Measurement: Issues and Practice, 15(1), 22–29. [DOI: 10.1111/j.1745-3992.1996.tb00803.x]
Wainer, H., & Wang, X. (2000). Using a new statistical model for testlets to score TOEFL. Journal of Educational Measurement, 37(3), 203–220. [DOI: 10.1111/j.1745-3984.2000.tb01083.x]
Wang, W. C., & Wilson, M. (2005). The Rasch testlet model. Applied Psychological Measurement, 29(2), 126–149. [DOI: 10.1177/0146621604271053]
Xia, Y., & Zheng, Y. (2018). Asymptotically normally distributed person fit indices for detecting spuriously high scores on difficult items. Applied Psychological Measurement, 42(5), 343–358. [DOI: 10.1177/0146621617730391]
Yen, W. M. (1993). Scaling performance assessments: Strategies for managing local item dependence. Journal of Educational Measurement, 30(3), 187–213. [DOI: 10.1111/j.1745-3984.1993.tb00423.x]

MeSH Term

Humans
Psychometrics
Models, Statistical
Algorithms
Likelihood Functions
Computer Simulation
Journal Article

Word Cloud

Similar Articles

Cited By