Off-campus UNL users: To download campus access dissertations, please use the following link to log into our proxy server with your NU ID and password. When you are done browsing please remember to return to this page and log out.

Non-UNL users: Please talk to your librarian about requesting this dissertation through interlibrary loan.

Accuracy and robustness of diagnostic methods: Comparing performance across domain score, multidimensional item response, and diagnostic categorization models

Tzu-Yun Chin, University of Nebraska - Lincoln

Abstract

Developers and users of educational and psychological tests often have the desire to report scores from sub-scales, in addition to total test scores. However, sub-scales are usually short and the psychometric properties of their scores may prohibit them from being used for making important interpretations or decisions. As a response, a number of diagnostic methods have been proposed to improve sub-scale score quality. In this Monte Carlo simulation study, three diagnostic methods – Wainer augmented method (AUG), multidimensional three-parameter logistic model (M3PL), and log-linear cognitive diagnosis model (LCDM) were evaluated by their classification accuracies. Two additional research questions were also investigated: the robustness of LCDM against attribute heterogeneity and the population invariance property of LCDM parameter estimates. Subscale length, attribute correlations, attribute homogeneity, and average examinee ability were systematically manipulated. The results suggested that AUG and M3PL performed similarly with regards to classification accuracy in most cases. When compared to AUG and M3PL, LCDM had lower percent agreement, kappa, and sensitivity, especially when the examinee abilities were higher than what the items targeted. However, LCDM had better specificity when the examinee abilities were higher than the items targeted. The results of this study also showed that the LCDM estimates were not population invariant although LCDM might be viewed as robust against attribute heterogeneity. In conclusion, due to both its classification accuracy performance and its ease of implementation, AUG may be advised over M3PL or LCDM in practice for reporting mastery classification along sub-scales if the assessment conditions are similar to those simulated in this study.

Subject Area

Educational tests & measurements|Educational evaluation|Quantitative psychology

Recommended Citation

Chin, Tzu-Yun, "Accuracy and robustness of diagnostic methods: Comparing performance across domain score, multidimensional item response, and diagnostic categorization models" (2011). ETD collection for University of Nebraska-Lincoln. AAI3482677.
https://digitalcommons.unl.edu/dissertations/AAI3482677

Share

COinS