Examination of the Effects of Different Missing Data Techniques on Item Parameters Obtained by CTT and IRT
Author :
Abstract
The purpose of this study is to examine the effect of different missing data techniques on the item parameters estimated for Classical Test Theory (CTT) and Item Response Theory (IRT) comparatively through simulated and real data sets. For this purpose, data sets with missing completely at random pattern with different sample sizes and missing data rates have been generated. Item parameters based on CTT and IRT are estimated after applying different missing data techniques (listwise deletion, regression imputation and expectation – maximization). Estimated parameters were compared with the parameters of complete data sets. The same procedure was performed on the data of the PM3 subtest in the PISA 2012 application. It is found that higher difficulty and lower discrimination values are obtained with listwise deletion while accurate item difficulties can be obtained with regression imputation and expectation - maximization algorithm, but the same situation is not valid for item discriminations. It is also seen that missing data at a ratio of %2 can lead to serious problems even if they have missing completely at random pattern.
Keywords
Abstract
The purpose of this study is to examine the effect of different missing data techniques on the item parameters estimated for Classical Test Theory (CTT) and Item Response Theory (IRT) comparatively through simulated and real data sets. For this purpose, data sets with missing completely at random pattern with different sample sizes and missing data rates have been generated. Item parameters based on CTT and IRT are estimated after applying different missing data techniques (listwise deletion, regression imputation and expectation – maximization). Estimated parameters were compared with the parameters of complete data sets. The same procedure was performed on the data of the PM3 subtest in the PISA 2012 application. It is found that higher difficulty and lower discrimination values are obtained with listwise deletion while accurate item difficulties can be obtained with regression imputation and expectation - maximization algorithm, but the same situation is not valid for item discriminations. It is also seen that missing data at a ratio of %2 can lead to serious problems even if they have missing completely at random pattern.
Keywords
- Akbaş, U. (2014). Farklı örneklem büyüklüklerinde ve kayıp veri örüntülerinde ölçeklerin psikometrik özelliklerinin kayıp veri baş etme teknikleri ile incelenmesi. Yayınlanmamış doktora tezi, Ankara Üniversitesi, Eğitim Bilimleri Enstitüsü, Ankara.
- Allison, P. D. (2002). Missing data. Newbury Park, CA: Sage.
- Alpar, R. (2011). Uygulamalı çok değişkenli istatistiksel yöntemler. Ankara: Detay Yayıncılık.
- Bal, C. (2003). Çok gruplu veri setlerinde eksik gözlem sorununun çözümlenmesi ve sağlık alanında bir uygulama. Yayınlanmamış doktora tezi, Osmangazi Üniversitesi, Sağlık Bilimleri Enstitüsü, Eskişehir.
- Bernaards, C. A. and Sijtsma, K. (1999). Factor analysis of multidimensional polytomous item response data suffering from ignorable item nonresponse, Multivariate Behavioral Research. 34(3), 277 – 313. doi:10.1207/S15327906MBR3403_1
- Buuren, van S. (2013). Flexible imputation of missing data. New York: Chapman & Hall/CRC Press.
- Can, S. (2003). The analyses of secondary education institutions student selection and placement test’s verbal section with respect to item response theory models. Unpublished master thesis, Middle East Technical University, Department of Educational Sciences, Ankara.
- Çakıcı Eser, D. (2015). Çok boyutlu madde tepki kuramının farklı modellerinden çeşitli koşullar altında kestirilen parametrelerin incelenmesi. Yayınlanmamış doktora tezi, Hacettepe Üniversitesi, Eğitim Bilimleri Enstitüsü, Ankara.
- Çelen, Ü. (2008). Klasik test kuramı ve madde test kuramına dayalı olarak geliştirilen iki testin psikometrik özelliklerinin karşılaştırılması. Yayınlanmamış doktora tezi, Ankara Üniversitesi, Eğitim Bilimleri
- Çüm, S. and Gelbal, S. (2015). Kayıp veriler yerine yaklaşık değer atamada kullanılan farklı yöntemlerin model veri uyumu üzerindeki etkisi. Mehmet Akif Ersoy Üniversitesi Eğitim Fakültesi Dergisi. 35, 87-111.
- Demir, E. (2013). Kayıp verilerin varlığında iki kategorili puanlanan maddelerden oluşan testlerin psikometrik özelliklerinin incelenmesi. Yayınlanmamış doktora tezi, Ankara Üniversitesi, Eğitim Bilimleri Enstitüsü, Ankara.
- Doğanay Erdoğan, B. (2012). Çoklu atama yöntemlerinin Rasch modelleri için performansının benzetim çalışması ile incelenmesi. Yayınlanmamış doktora tezi, Ankara Üniversitesi, Sağlık Bilimleri Enstitüsü, Ankara.
- Enders, C. K. (2010). Applied missing data analysis. New York: The Guilford Press.
- Enders, C. K. (2003). Using the expectation maximization algorithm to estimate coefficient alpha for scales with item-level missing data. Psychological Methods. 8(3), 322-337. doi:10.1037/1082-989X.8.3.322
- Enders, C. K. (2004). The impact of missing data on sample reliability estimates: İmplications for reliability reporting practices. Educational and Psychological Measurement. 64(3), 419-436. doi:10.1177/0013164403261050
- Finch, H. (2008). Estimation of item response theory parameters in the presence of missing data. Journal of Educational Measurement. 45(3), 225-245.
- Field, A. (2005). Discovering statistics using SPSS. Sage.
- Ginkel, van J. R., Ark, van der L. A. and Sijstma, K. (2007). Multiple imputation for item scores when test data are factorially complex. British Journal of Mathematical and Statistical Psychology, 60, 315 - 337. doi:10.1348/000711006X117574
- Graham. J. W. (2012). Missing data analysis and design. Springer.
- Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E. and Tatham, R. L. (2006). Multivariate data analysis. Pearson – Prentice Hall.
- Hambleton, R. K., Swaminathan, H. and Rogers, H. (1991). Fundamentals of item response theory. Newbury Park CA: Sage.
- Han, K. T. (2007). Wingen: Windows software that generates IRT parameters and item responses. Applied Psychological Measurement, 31(5), 457-459.
- Heerwegh, D. (2005). Web Surveys. Explaining and Reducing Unit Nonresponse, Item nonresponse and partial nonresponse. Yayınlanmamış doktora tezi, Katholieke Universiteit Leuven Faculteit Sociale Wetenschappen, Leuven, Belçika.
- Holman, R. and Glas, C. A. W. (2005). Modelling non‐ignorable missing‐data mechanisms with item response theory models. British Journal of Mathematical and Statistical Psychology, 58, 1–17. doi:10.1348/000711005X47168
- Karasar, N. (2007). Bilimsel araştırma yöntemi: kavramlar, ilkeler, teknikler. Ankara: Nobel Yayın Dağıtım.
- Köse, İ. A. and Öztemur, B. (2014). Kayıp veri ele alma yöntemlerinin t-testi ve anova parametreleri üzerine etkisinin incelenmesi. Abant İzzet Baysal Üniversitesi Eğitim Fakültesi Dergisi, 14(1), 400-412. doi:10.17240/aibuefd.2014.14.1-5000091519
- Little, R. J. A. and Rubin, D. B. (1987). Statistical analysis with missing data. John Wiley & Sons.
- Little, R. J. A. (1988). A test of missing completely at random for multivariate data with missing values. Journal of the American Statistical Association, 83, 1198-1020. doi: 10.2307/2290157.
- Leeuw, E. D. Hox, J. and Huisman, M. (2003). Prevention and treatment of item nonresponse. Journal of Official Statistics. 19(2), 153-176.
- Nartgün, Z. (2015). Comparison of various methods used in solving missing data problems in terms of psychometric features of scales and measurement results under different missing data conditions. International Online Journal of Educational Sciences, 7(4), 252 – 265. doi: 10.15345/iojes.2015.04.017
- OECD. (2014). PISA technical report. OECD Publishing.
- Özer Özkan, Y. (2012). Öğrenci başarılarının belirlenmesi (öbbs) sınavından klasik test kuramı, tek boyutlu ve çok boyutlu madde tepki kuramı modelleri ile kestirilen başarı puanlarının karşılaştırılması. Yayınlanmamış doktora tezi, Ankara Üniversitesi, Eğitim Bilimleri Enstitüsü, Ankara.
- Roth, P. L. (1994). Missing data: A conceptual review for applied psychologists. Personnel Psychology, 47(3), 537-560. doi: 10.1111/j.1744-6570.1994.tb01736.x
- Rubin, D. B. (1976). Inference and missing data. Biometrika 63, 581-592. doi:10.1093/biomet/63.3.581
- Schafer. J. L. and Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147 – 177. doi: 10.1037/1082-989X.7.2.147
- Schafer, J. L. (1997). Analysis of incomplete multivariate data. New York: Chapman & Hall/CRC.
- Sijtsma, K. and Ark, van der, L. A. (2003). Investigation and treatment of missing item scores in test and questionnaire data. Multivariate Behavioral Research. 38(4), 505-528. doi:10.1207/s15327906mbr3804_4
- SPSS (2007). SPSS missing values 17.0. SPSS Inc.
- Şahin Kürşad, M. (2014). Sıklıkla kullanılan kayıp veri yöntemlerinin betimsel istatistik güvenirlik ve geçerlik açısından karşılaştırılması. Yayınlanmamış yüksek lisans tezi, Abant İzzet Baysal Üniversitesi, Eğitim Bilimleri Enstitüsü, Bolu.
- Tabachnick, B. G. and Fidell, L. S. (1996). Using multivariate statistics. Harper Collins College Publishers.