Analysis of influence of the item total scores on the levels of ability of respondents in the dichotomous Rasch model based on the weighted maximum likelihood method

Purpose of the study. Rasch model is often used in processing test results. However, when using this model and the maximum likelihood method (ML), the estimates the levels of ability of respondents depend only on the number of correctly performed test items and do not depend on the difficulties of the items. The purpose of the research is to analyze the influence of the difficulties of the items on the levels of abilities of the respondents based on the weighted maximum likelihood method (WML). To obtain the weights of the WML, the item total scores are used. Materials and methods. The analysis of the influence of the difficulties of the items on the levels of abilities of the respondents is investigated using the dichotomous table obtained when testing the knowledge of 19 respondents in the course “Fundamentals of Electronics”. Indicator variables of 16 test items were used. For items, we calculate the item total scores that determine their items difficulties. The weighting coefficients of the used WML depend on the item total scores and on the coefficient of influence K. When K = 0, WML convert into ML. As K increases from 0 to 2, the weighting coefficients increase and it becomes possible to analyze in detail the influence of the difficulties of the items on the respondents’ ability levels. To calculate the parameters of the Rasch model based on WML, programs (M-files) for the MATLAB environment and Ministep (Winsteps) are used. Results. The use of WML with weighting coefficient obtained on the basis of the item total scores of the difficulties of the items allowed us to further differentiate the levels of respondents’ abilities in the dichotomous Rasch model. The results of the analysis performed using the data of the test on electronics show that, ceteris paribus, new levels of person’s abilities increase if respondents perform difficult items and, conversely, the respondent’s ability levels decrease if respondents perform light items. At the same time, the difficulty levels of the items practically do not change. As a rule, the greater the coefficient of influence K, the more different the estimation of abilities of respondents, obtained on the basis of WML, from the estimation on the basis of ML. However, there are respondents whose ability level does not change or change slightly when the coefficient K is increased from 0 to 2. For the data of the test on electronics with a coefficient K ≤ 1, the original order of respondents in their ability levels calculated on the basis of ML is preserved. With an increased coefficient of influence K ≥ 1,5, new levels of ability, calculated using WML, cause a change in the order of distribution of respondents according to ability levels. Calculations performed using the MATLAB package are confirmed by data obtained using the Winsteps program. Differences without extreme respondents do not exceed 0.01 logit with the maximum value of the coefficient K equal to 2. Conclusion. On the basis of WML, a method is proposed for taking into account the influence of the difficulties of items on the levels of respondents’ abilities in the Rasch dichotomous model when using the item total scores. The results of the analysis performed using the data of the test on electronics show that in this case we will obtain a differentiation of the levels of abilities of the respondents who score the same points. Note that the results obtained using WML and using the data of the test on electronics do not reject the data obtained on the basis of the classical dichotomous Rasch model and ML. The results obtained on the basis of WML, allow to refine the levels of abilities of the respondents, obtained on the basis of ML.

1. Neyman YU.M., KHlebnikov V.A. Vvedeniye v teoriyu modelirovaniya i parametrizatsii pedagogicheskikh testov = Introduction to the theory of modeling and parameterization of pedagogical tests. Moscow: Prometheus; 2000. 169 p. (In Russ.)

2. Kim V.S. Testirovaniye uchebnykh dostizheniy. Monografiya = Testing educational achievements. Monograph. Ussuriysk: Publisher UGPI; 2007. 214 p. (In Russ.)

3. Avanesov V: Item Response Theory: Basic Concepts and Provisions. Pedagogicheskiye izmereniya = Pedagogical Measurements. 2007; 2: 3–28. (In Russ.)

4. CHelyshkova M.B. Teoriya i praktika konstruirovaniya pedagogicheskikh testov: Ucheb. posobiye = Theory and practice of constructing pedagogical tests: tutorial. Moscow: Logos; 2002. 432 p. (In Russ.)

5. Maslak A.A. Teoriya i praktika izmereniya latentnykh peremennykh v obrazovanii = Theory and practice of measuring latent variables in education. Moscow: Yurayt; 2016. 256 p. (In Russ.)

6. Popov A.P. Critical analysis of parametric models of Rush and Birnbaum. In: Materialy 4-y NMK «Innovatsionnyye metody i sredstva otsenki kachestva obrazovaniya» = Proceedings of the 4th NMC «Innovative methods and tools for assessing the quality of education.» Moscow: MGUP; 2006: 231-235. (In Russ.)

7. SHrayfel’ I.S., Eliseyev I.N. Theoretical substantiation of a single iterative process of joint quantitative assessment of the difficulties of assignments and levels of student training. Sibirskiy zhurnal vychislitel’noy matematiki = Siberian Journal of Computational Mathematics. 2016; 19; 1: 107-123. (In Russ.)

8. Serbin V.I. Management of the learning process using a Markov model. Prikaspiyskiy zhurnal: upravleniye i vysokiye tekhnologii = Caspian Journal: Management and High Technologies. 2013; 4 (24): 29–35. (In Russ.)

9. Semov A.M., Semova M.A., KHlebnikov V.A. Edinyy iteratsionnyy protsess sovmestnoy kolichestvennoy otsenki trudnostey zadaniy i urovney podgotovlennosti uchastnikov testirovaniya. Trudy tsentra testirovaniya, vypusk 2 = A unified iterative process of joint quantitative assessment of the difficulties of tasks and levels of preparedness of testing participants. Works of the testing center, issue 2. Moscow: Prometheus Publishing House, 1999: 54–60. (In Russ.)

10. Hu F., Zidek J.V. The weighted likelihood. The Canadian Journal of Statistics. 2002; 30; 3: 347–371. DOI: 10.2307/3316141.

11. Simakhin V.A., Investigation of weighted maximum likelihood estimates. Vestnik Kurganskogo gosudarstvennogo universiteta. Seriya: Tekhnicheskiye nauki = Bulletin of Kurgan State University. Series: Engineering. 2011; 6: 72–77. (In Russ.)

12. Warm T.A. Weighted Likelihood Estimation of Ability in Item Response Theory. Psychometrika. 1989; 54: 427–450. DOI: 10.1007/BF02294627.

13. Linacre J.M. The Efficacy of Warm’s Weighted Mean Likelihood Estimate (WLE) Correction to Maximum Likelihood Estimate (MLE) Bias. Rasch Measurement Transactions. 2009; 23; 1: 1188–1189

14. John M. Linacre. A User’s Guide to WINSTEPS, MINISTEP. Rasch-Model Computer Programs; 2017. [Internet] URL: http://www.winsteps. com/winman/copyright.htm (Cited: 07.04.2019).

15. Tao, J., Shi, N., & Chang, H. Item-weighted likelihood method for ability estimation in tests composed of both dichotomous and polytomous items. Journal of Educational and Behavioral Statistics. 2012; 37; 2: 298-315. DOI: 10.3102/1076998610393969.

16. Kuchumov A. I. Elektronika i skhemotekhnika. Uchebnoye posobiye = Electronics and circuitry. Tutorial. Moscow: Helios ARV; 2011. 336 p. (In Russ.)

17. MatLab. Fminunc function [Internet] URL: https://www.mathworks.com/help/optim/ug/ fminunc.html?s_tid=srchtitle (Cited: 07.04.2019).

18. Alekseyev M.V., Belov E.B., Kitayev N.P., Kuchumov A.I. ferences in assessing the levels of preparedness of extreme respondents using the classical theory of testing and the dichotomous Rasch model. Informatizatsiya obrazovaniya i nauki = Informatization of education and science. 2018. 2(38): 99-107. (In Russ.)

19. Wright B.D. Estimating measures for extreme scores. Rasch Measurement Transactions. 1998. 12 (2): 632-633.

20. Parameter MEAN in table 14.1 when using IWEIGHT. [Internet] URL: http://raschforum. boards.net/thread/796/parameter-mean-table-using-iweight (Cited: 07.04.2019).