Graphically Modeling Student Knowledge: A Bayesian Network Approach

Abstract

It is commonly believed that diagnostic information that are instructionally relevant and educationally meaningful would help students identify remediation learning paths and assist teachers customize their instruction according to students’ knowledge gaps (DiBello et al., 2006). The Bayesian Network (BN) approach operationalizes cognitive diagnosis in a novel way of releasing some constraints of Cognitive Diagnostic Models (CDMs) and catering to a need of complex modeling of attribute structure and polytomous attributes. It also naturally relates to the assessment framework of Evidence-Centered Design (ECD). However, empirical studies fall short of systematically addressing the utility, uniqueness, and value of using BN in analyzing diagnostic assessment data. In this study, I first conducted simulations to examine the performance of the BN approach across assessment scenarios of different sample sizes, test lengths, Q matrix complexities, and attribute types. Second, I evaluated the mastery classification accuracies of the BN approach when various amount of information on the structure in attributes is provided. Third, I compared the utility and the performance of both BN and CDM in terms of mastery classification accuracies across different assessment scenarios. Finally, I applied BN to analyze a dataset with dichotomous attributes from Trends in International Mathematics and Science Study (TIMSS) and a dataset with polytomous attributes. The results supported that BN can yield model parameters with acceptable accuracy for formative diagnostic assessments under various conditions, namely different test lengths, sample sizes, Q matrix complexities, and attribute types. BN can also provide adequate estimation results when partial information on the attribute structure is provided. The comparison of BN and a CDM model highlights the flexibility of BN in handling different assessment types. Finally, the real data analyses showcased the diagnostic reports on student performance levels based on the BN approach.