Abstract
In cognitive diagnosis modeling, the attributes required for each item are specified in the Q-matrix. The traditional way of constructing a Qmatrix based on expert opinion is inherently subjective, consequently resulting in serious validity concerns. The current study proposes a new validation method under the deterministic inputs, noisy "and" gate (DINA) model to empirically validate attribute specifications in the Q-matrix. In particular, an iterative procedure with a modified version of the sequential search algorithm is introduced. Simulation studies are conducted to compare the proposed method with existing parametric and nonparametric methods. Results show that the new method outperforms the other methods across the board. Finally, the method is applied to real data using fraction-subtraction data.
Abstract
ARTICLE HISTORY
INTRODUCTION
Cognitive diagnosis models (CDMs) require a Q-matrix (Tatsuoka, 1983) to identify the specific subset of attributes measured by each item. The entry qjk in row j and column k of the Q-matrix is 1 if the ℎ attribute is required to correctly answer item j, and 0 otherwise. Due to its nature, constructing a Q-matrix is usually subjective, which has raised serious validity concerns among researchers. For instance, the estimation of model parameters, and ultimately the accuracy of attribute classifications may be negatively affected by including or omitting multiple q-entries in the Q-matrix (de la Torre, 2011; Rupp & Templin, 2008) . However, the Q-matrix is usually assumed to be correct once specified by domain experts. This assumption is generally made because until recently, few well-established methods have become available to detect misspecifications in the Q-matrix (Chiu, 2013; de la Torre, 2008; Rupp & Templin, 2008) , particularly when general CDMs are involved (de la Torre & Chiu, 2016; Liu, Xu, & Ying, 2012; Terzi, 2017) . Any analysis, such as model-fit evaluation, that does not check the correctness of the Q-matrix, becomes questionable.
INTRODUCTION
These concerns have led to developments of some statistical methods for validating the appropriateness of Q-matrix specifications. One of the earlier studies on the Q-matrix validation was introduced by de la Torre (2008) for the deterministic inputs, noisy "and" gate (DINA; Haertel, 1989; Junker & Sijtsma, 2001 ) model. It is an empirically based δ-method that defines the correct q-vector for each item. In doing so, the discrimination index of item j, , is estimated. The index is the difference in the probabilities of correct responses between examinees who have mastered the required attributes and those who have not. Using the δ-method, two algorithms were discussed in de la Torre (2008) . However, the algorithms have some limitations. As noted by de la Torre (2008) , an incorrect Q-matrix because of over-and under-specifications of attributes can cause bias in parameter estimation. This issue cannot be completely addressed by the algorithms because they usually choose q-vectors with all attributes specified. For one of the algorithms, the sequential search algorithm (SSA), it is also not clear what cut-off values should be used in practice because it could vary depending on many conditions, such as changes in sample sizes, test lengths, item qualities, and amount of misspecifications, all of which were fixed in de la Torre (2008)'s paper. It should also be noted that the algorithm was not implemented iteratively, meaning that the validation method stops after one full iteration even if changes are made in the provisional Q-matrix.
INTRODUCTION
Another method, the Q-matrix refinement method (QRM), was proposed by (Chiu, 2013 ) based on a nonparametric classification procedure (Chiu & Douglas, 2013) . This method aims to minimize the residual sum of squares (RSS) between the observed and ideal responses among all the possible q-vectors given a Q-matrix. The RSS is used to identify any misspecified q-entries for an item. In the algorithm, the item vector with the highest RSS gets replaced by the one having the lowest RSS. The process is repeated iteratively until the convergence criterion is met. Due to its nature as a nonparametric method, it neither relies on the estimation of model parameters nor makes any assumptions other than those made by the CDM itself (Chiu, 2013) . However, if the underlying model is known, parametric methods should provide more powerful results particularly when N is large.
INTRODUCTION
DeCarlo (2011) introduced a model-based approach using a Bayesian extension of the DINA model. In this method, possible misspecified entries in the Q-matrix were identified in advance. Then, these entries were treated as random (Bernoulli) variables and estimated with the rest of the model parameters. Limitations of this method are that it is computationally timeconsuming and any misspecified q-entries have to be identified in advance. Unlike DeCarlo (2011)'s study, Liu et al. (2012) proposed a data-driven approach in that any expert involvement in Q-matrix design is not required for identifying misspecified entries in the Q-matrix. However, when unknown guessing parameters exist, the identifiability of the Q-matrix can be difficult.
INTRODUCTION
Recently, de la Torre and Chiu (2016) developed a discrimination index, as an extension of the empirically based δ-method (de la Torre, 2008), using the G-DINA model. This index can be applied under a wider class of CDMs. However, the findings of the study were limited to the fixed sample size and test length. Moreover, the index does not determine optimal ε values that prevent q-entries from over-or under-specifications, and the procedure is not iterative, meaning that it stops further identifying attribute specifications after the first round of validation step.
INTRODUCTION
The purpose of this current study is to introduce an iterative procedure in conjunction with a modified version of the SSA, and is called iterative modified SSA (IMSSA). The new method aims to make three crucial contributions to the Q-matrix validation literature. First, using simulation, an approximation was made to generally define an empirically based a cutoff value applicable across all conditions. Second, the search algorithm only focuses on singleattribute specifications so that it can eliminate additional complications that could happen due to q-vectors with more than single-attribute specifications. Third, the algorithm is implemented iteratively, such that, if any q-vectors are changed in the previous iteration, a new calibration is carried out using the updated Q-matrix as the provisional Q-matrix. The iterative algorithm aims to alleviate negative effects of any misspecified attribute specifications given in the preceding iteration. In this present study, iterative and non-iterative algorithms were compared to examine if an iterative algorithm can further identify and correct misspecifications in succeeding iterations.
INTRODUCTION
Given the purpose, the rest of the paper consists of the following sections: First is a brief background on the DINA model, Q-matrix refinement method, and exhaustive and sequential search algorithms. Second is a presentation of the new method proposed in this paper. This is followed by simulation study design and results. Then, real data analysis and its results are introduced. Finally, the paper concludes with a discussion and conclusion for future studies.
BACKGROUND 2.1. The DINA Model
The DINA model has been commonly used in many studies (e.g., de la Torre & Douglas, 2004 , 2008 de la Torre, 2009a; DeCarlo, 2011; Kuo, Pai, & de la Torre, 2016; Liu, Ying, & Zhang, 2015; Park & Lee, 2014; Rupp & Templin, 2008) . This study focuses on the DINA model because of its more straightforward interpretations, smaller sample size requirements for accurate parameter estimation (Rojas, de la Torre, & Olea, 2012) , and flexibility for extension to more general cognitive diagnostic models. The DINA model is an example of a conjunctive model for dichotomously scored test items, where all required attributes of an item should be mastered by examinees before an examinee can be expected to correctly answer the item. Nonmastery of one or more required attributes for an item is equivalent to nonmastery of all required attributes. Let examinee i's binary attribute vector be denoted by = { }. The item response function of the model is defined as:
BACKGROUND 2.1. The DINA Model
which is the probability of answering an item j correctly by examinees with the attribute pattern , is the response of examinee i (i = 1, 2, …, N) to item j (j = 1, 2, …, J), and is the ideal response computed as:
BACKGROUND 2.1. The DINA Model
an indicator of whether all of the required attributes associated with item j have been mastered by examinee i.
Q-Matrix Refinement Method
The RSS of item j across all examinees is defined as:
Q-Matrix Refinement Method
where ij X and ij  are the observed and ideal item responses of examinee i to item j, respectively, is the latent proficiency class m, and N is the number of examinees. Note that the index j of in Equation 3 was replaced by m because ideal item responses are class-specific, meaning that every examinee in the same latent class is assumed to have the same ideal response to an item (Chiu, 2013) .
Exhaustive Search Algorithm
The exhaustive search algorithm (ESA) for Q-matrix validation computes for all = 2 -1 possible q-vectors for each j item (de la Torre, 2008) . The q-vector that gives the largest difference in the probabilities of correct response between examinees who have all the required attributes ( = 1) and those who do not have ( = 0) among all the possible attribute patterns is chosen as the correct q-vector for item j. However, the algorithm becomes impractical when K is reasonably large. Additionally, the ESA has the tendency to choose overspecified q-vectors (de la Torre, 2008).
Sequential Search Algorithm
The sequential search algorithm (SSA), in comparison to the ESA, is considered more efficient because it does not require the comparisons of for all the possible attribute patterns. More specifically, is computed for ( + 1) − ( 2 + )/2 q-vectors for item j, where is the number of attributes required for item j (de la Torre, 2008 As stated earlier, estimation that involves some misspecified q-vectors can affect the quality of parameter estimation (Rupp & Templin, 2008 ) and this in turn affects the accuracy of the validation method. Similarly, the noise due to the stochastic nature of the response process makes it possible to obtain a q-vector with more attribute specifications than necessary.
Sequential Search Algorithm
Especially using real data can cause ̂( +1) >̂( ) or the reverse, resulting in over-or underspecifications, respectively. A suggested solution is to assign ε, which is a minimum increment in the discrimination index of the item before an additional attribute can be included, as in,
THE PROPOSED METHOD 3.1. An Iterative Method for Empirically-Based Q-Matrix Validation
This study introduces an iterative procedure in conjunction with a modified version of the SSA, and is called iterative modified SSA (IMSSA). The IMSSA differs from the SSA in two respects. First, the IMSSA determines required attribute specifications based on only the single-attribute q-vectors. Similar to the empirically based δ-method (de la Torre, 2008), the IMSSA starts by estimating the item parameters via an empirical Bayesian implementation of the expected-maximization (EM) algorithm (de la Torre, 2009b) using a provisional Q-matrix. The K numbers of ̂s corresponding to the single-attribute q-vectors (i.e., 1 ) are then estimated and ordered from the highest to the lowest. The correct attribute specification is determined based on the proportion of ̂ * 1 relative to the maximum ̂( max)
THE PROPOSED METHOD 3.1. An Iterative Method for Empirically-Based Q-Matrix Validation
because it corresponds to the best suggested attribute specification. The noise due to the use of the estimated posterior distribution should be controlled so as to not cause any over-or under-specifications. That can be done by using a cut-off point denoted by
THE PROPOSED METHOD 3.1. An Iterative Method for Empirically-Based Q-Matrix Validation
(1) , which represent the minimum ratio between single-attribute qvectors and the best single-attribute q-vector corresponding to ̂( max)
1
. Specifically, if ̂2 1 is considerably smaller than ̂( max) 1 (i.e., ̂2 1 /̂( max) 1 <
1
(1) , the required attribute would be an attribute specified in the single-attribute q-vector corresponding to ̂( max)
1
; if not, the attribute specifications in the first two q-vectors are chosen. It continues by checking the ratio ̂3 1 /̂( max)
1
. If the ratio is larger than
1
(1) , the attribute specification in the third q-vector is also added on the top of the previous two specifications, and it continues; otherwise, the process is terminated. The ratio between ̂ * 1 and ̂( max) 1 was determined based on some preliminary findings, and the values of (1) , the cut-off point, were defined using simulated response data.
1
At this point, an example can be helpful to lay out the rationale as to how the study determines the correctness of attribute specifications based on the ratio of ̂s to the maximum ̂. For illustration purposes, we considered two items, each with a misspecified attribute specification. In practice, the provisional Q-matrix may not have entirely correct specifications. However, data based on parameter estimates using the provisional Q-matrix can be generated. The ̂-computation for the simulated data can be monitored, which can allow us to define extreme changes in the ratio of ̂s .
1
Examples of ̂ * 1 computations for the simulated data can help determine whether or not the algorithm could identify correct specifications. Assume that K = 5. Table 1 displays examples of items that have over-and under-specifications. In the first misspecification, the qvector (1,0,0,0,0)′ is over-specified as in (1,0,1,0,0)′. The EM estimation is carried out with the latter q-vector, and ̂s of single-attribute q-vectors are estimated and sorted from the highest to the lowest. The result suggests that the correct attribute specification is only 1 (̂( max) 1 = .41) due to a large drop in ̂2 1 (i.e., ̂2 1 /̂( max) 1 = .15 <
1
(1) ), in that a value of (1) will be determined later. A similar result is also observed for an item that has been under-specified. The misspecification appears as (1,0,0,0,0)′ from the correct vector of (1,1,0,0,0)′ in the righthand side of Table 1 . The ratio of the second ̂2 1 to the maximum ̂( max) 1 shows a small drop (i.e., ̂2 1 /̂( max) 1 = .73 > (1) ); however, the next ratio is rather small (i.e., ̂3 1 /̂( max) 1 = .13 < (1) ). Therefore, the attributes in the first two single-attribute q-vectors are accurately specified (i.e., 1 and 2 ). Note that the criterion is similar to the method proposed by de la Torre and Chiu (2016) , which is the proportion of variance accounted for (PVAF) by a particular q-vector relative to the maximum ̂2 that is achieved when all the attributes are specified (i.e., (1,1,1,1,1)′). However, the criterion in this study is not exactly the same, because it is relative to the best attribute specification, not the attribute vector with all the attributes specified.
1
Second, the IMSSA becomes more efficient than the original SSA because ̂ is not computed beyond single-attribute vectors. As such, the maximum number of comparisons for the new algorithm is K, which is considerably smaller than SSA (i.e., ( + 1) − ( 2 + )/2) and ESA (i.e., 2 − 1), where K is the total number of attributes and is the number of attributes being measured by item j. For example, let = 10 and = 3. The maximum number of comparisons is 10 for the IMSSA, 34 for the SSA, and 1023 for the ESA. Thus, using the IMSSA can lessen complications associated with multiple search steps. In summary, examining the proportion of ̂ by a particular single-attribute q-vector to the maximum ̂ using a provisional q-vector could suggest which attributes should be specified --̂ of required attributes are considerably larger compared to ̂ of other attributes. 
SIMULATION STUDY DESIGN
To evaluate the viability of the proposed method, two simulation studies were conducted with the following goals: (1) to determine an optimal (1) value, which could be generalized across the conditions; and (2) to compare the effectiveness of different validation methods with an iterative and noniterative algorithm. For each simulation condition, 100 datasets were replicated using the DINA model with the following factors: sample sizes (N = 1,000 and 2,000), test lengths (J = 15 and 30), item qualities ( = = 0.1, 0.2, and 0.3), and amount of misspecifications (5% and 10%). In this study, the three sets of item qualities were considered similar to Hou, de la Torre, and Nandakumar (2014). In each condition, 100 misspecified Qmatrices were generated, which contain 5% or 10% randomly misspecified q-entries. Two constraints were imposed on altering the q-vectors, namely, the misspecified q-vectors cannot have more than two-attribute misspecifications, and at least one attribute should be specified as 1. For example, if a Q-matrix has 10% misspecifications for J = 30 and K = 5, 15 of 150 entries were randomly altered by producing over-or under-specified q-vectors, where almost eight to 15 q-vectors are misspecified. In doing so, the study was able to focus on the impact of the amount of misspecifications rather than the type of misspecifications. It should be noted that the true Q-matrices in Table 2 for J = 15 and 30 are related in two ways. Each attribute is measured six and 12 times when J = 15 and 30, respectively, and there are equal numbers of 1-, 2-, and 3-attribute q-vectors in the each Q-matrix. Finally, the attribute profiles were generated from a uniform distribution in that all the possible attribute patterns were generated with equal probabilities from a multinomial distribution.
SIMULATION STUDY DESIGN
To define an optimal (1) value for the IMSSA, the item quality was generated from Unif (0.05,0.45) . Based on the results of a pilot study, the performance of the proposed method was examined given
SIMULATION STUDY DESIGN
(1) values in the range 0.10 to 0.90, with an increment of 0.1. After defining an optimal value, the second simulation study was conducted to compare the five validation procedures: IMSSA, MSSA, ESA, SSA, and QRM. These methods were compared based on the proportions of correctly identifying attribute specifications at the vector level. The code to implement the IMSSA, MSSA, ESA, and SSA was written in Ox (Doornik, 2009) , whereas, the NPCD R package (Zheng & Chiu, 2015) was used (R Core Team, 2014) for the QRM analyses. Note. Items with * are used for J = 15.
FINDINGS 5.1. Simulation Study I
In the first simulation study, the performance of the IMSSA was observed to define an optimal (1) value which can be used under all conditions. Focusing in the range 0.10 to 0.90, values were derived based on the highest proportions of correctly identifying attribute specifications on average throughout all conditions, as shown in Table 3 . When
FINDINGS 5.1. Simulation Study I
(1) = 0.50 and 0.60, 92% of the q-vectors were correctly identified on average, which is the highest proportions of recovery across all
FINDINGS 5.1. Simulation Study I
(1) values. Thus, (1) was nwilliams@example.net in the second simulation study. Note. Numbers in bold are the highest proportions of recovery for each condition. Table 4 shows results reported at the vector level, which are divided into two, with and without iterative algorithms. Among the methods with a non-iterative algorithm, the MSSA outperformed the others for each simulation condition considered in this study. In addition, the SSA (76%) provided better recovery than the ESA (74%) on average across all the conditions. As noted in de la Torre (2008), the SSA procedure was originally proposed to be a more efficient algorithm that does not require computing for the 2 − 1 possible q-vectors; however, results based on the ESA and SSA did not show considerable differences, which was only 2% on average. In general across the conditions, the average recovery of the MSSA was 89% of the q-vectors; whereas, the average recoveries of the ESA and SSA were 74% and 76%, respectively. In particular, recovery based on the MSSA was 15% and 13% higher than that of the ESA and SSA, respectively. Therefore, even though an iterative algorithm was not implemented in the MSSA, we could state that the modified version of the SSA (MSSA) improved the recovery in comparison to the SSA.
Simulation Study II
Specifically in comparing the iterative methods (i.e., IMSSA and QRM) under the high quality item, the QRM worked usually equally well as or better than the IMSSA. In this quality of items, both methods had perfect or above 97% of recovery. In continuing the comparison of the IMSSA and QRM under the medium quality item, both methods had recovery of attribute specifications above 99% when J = 30. The lowest recovery was 69% for the QRM and 86% for the IMSSA. When data were generated from the low quality item, the IMSSA (81%) had 9% more recovery than the QRM (72%) on average. The QRM only outperformed the IMSSA under four conditions, where the proportions of recovery differed only by 1% to 2% when the item qualities were medium (i.e., N = 1,000, J = 30 with 5% and 10% misspecifications) and high (i.e., N = 1,000, J = 15 with 10% misspecifications and N = 2,000, J = 15 with 10% misspecifications), respectively. Other than these differences, the IMSSA provided a better overall recovery than the QRM.
Simulation Study II
It is interesting to report that the performance of the QRM was equally well or worse when the sample size was doubled. For example, when the item quality was low under a condition where N = 1,000, J = 30 with 5% misspecifications, doubling the sample size to N = 2,000 resulted in the recovery dropping from 85% to 83%. In contrast, considering the same conditions, the recovery improved from 70% to 77% for the ESA, from 73% to 79% for SSA, from 88% to 92% for MSSA, and from 89% to 93% for the IMSSA. However, doubling the test items from 15 to 30, the recovery increased for all the methods. This finding can indicate that doubling the test length can lead to better improvement in recovery more than doubling the sample size.
Simulation Study II
Similarly, with regards to the difference in recovery rates due to the amount of misspecifications within the same conditions (i.e., N and J), a larger test length provided a smaller gap than a larger sample size. That is, recovery differences between 5% and 10% misspecifications were higher with a larger sample size than a longer length test. For example, among the non-iterative methods when N = 1,000 and J = 15 under the high quality item, recovery differences between 5% and 10% misspecifications were 22%, 21%, and 9% for the SSA, ESA, and MSSA, respectively, which dropped to 9%, 8%, and 0% when J = 30 holding the sample size constant. However, doubling the sample size with a fixed test length did not change the recovery differences that much, which was only 20%, 19%, and 9% for the SSA, ESA, and MSSA, respectively. In taking the amount of misspecifications into account for the non-iterative methods, doubling the test length had a considerably positive impact on the recovery than doubling the sample size.
Simulation Study II
For the iterative methods, again, doubling the test length decreased the difference in recovery rates between 5% and 10% misspecified Q-matrices. Under the same conditions, when N = 1,000 and J = 15, it was 1% for the QRM (i.e., 100 -99 = 1) and 3% for the IMSSA (i.e., 100 -97 = 13). However, that gap was smaller when J = 30 than N = 2,000. The difference substantially dropped for both methods after doubling the test length with a constant sample size. Therefore, based on these findings, it can be stated that doubling the test length substantially improved the recovery for both iterative methods and decreased the recovery differences due to a different amount of misspecifications. In summary, the proposed MSSA and IMSSA worked much better than the other methods. That is, after averaging the proportions of recovery across the conditions (i.e., N, J, item qualities, and amount of misspecifications), recovery based on the IMSSA (92%) and MSSA (89%) was 5% and 2% higher than that of the QRM (87%), respectively, and rather larger than the ESA and SSA. Note that the number of iterations in the iterative procedures was usually between two and three, and did not go beyond four.
REAL DATA ANALYSIS 6.1. Data
In addition to the simulation study, real data were analyzed to investigate the applicability of the method. The fraction-subtraction data (Tatsuoka, 1984) with 536 middle school students' responses to 12 fraction subtraction problems were examined. The four attributes for this dataset are: (a) performing a basic fraction subtraction operation, (b) simplifying/reducing, (c) separating a whole number from fraction, and (d) borrowing one from a whole number to fraction. The 12 items with the corresponding attribute specifications and ̂ values are shown in Table 5 . Note. 1 -performing a basic fraction subtraction operation; 2 -simplifying/reducing; 3 -separating a whole number from fraction; and 4 -borrowing one from a whole number to fraction.
REAL DATA ANALYSIS 6.1. Data
Note that the data set of Tatsuoka (1984) has been one of the most commonly examined real data designed for cognitively diagnostic assessment (Chiu, 2013; Chiu & Köhn, 2015; de la Torre, 2008; de la Torre & Chiu, 2016; DeCarlo, 2011) . In CDM analyses, one of the main concerns is the completeness of the Q-matrix. Unfortunately, the fraction-subtraction data do not appear to have a complete Q-matrix. It was demonstrated by Chiu, Douglas, and Li (2009) that a complete Q-matrix should identify all possible attribute patterns and require each attribute to be represented by at least one single-attribute vector. This issue has been further discussed with the original data (see Table 4 on pp. 615, Chiu, 2013; DeCarlo, 2011) or subsets of it (see de la Torre, 2008; de la Torre & Chiu, 2016) . The incompleteness of the Q-matrix in this dataset occurs because of the fact that only 58 of 256 (K = 8; Chiu, 2013) and 10 of 32 (K = 5; Chiu & Köhn, 2015) possible attribute patterns can be identified by the items, meaning that multiple classes may be merged (Chiu, 2013) . Therefore, results of this data analysis should be interpreted with caution.
Results
For the IMSSA, ̂ * 1 statistic and ̂ * 1 /̂( ) 1 ratios for 12 items are reported in Table 6 , and the suggested Q-matrix is further shown in Table 7 . Given the results in the first simulation study, the (1) values were nwilliams@example.net and 0.60. Note. * indicates a suggested attribute specification, (1) = 0.50. The results of the fraction-subtraction data obtained from the IMSSA were compared to the QRM. The IMSSA suggested attribute changes in seven items (i.e., items 1, 3, 4, 7, 8, 9 , and 10) when
Results
(1) = 0.50; whereas, the QRM suggested attribute changes in three items (i.e., items 1, 10, and 11). Based on the IMSSA, the result indicated that item 1 (i.e., ) should require two more attributes (i.e., 2 and 3 ) in addition to 1 . This suggestion may have occurred because this item requires more than just 1 , performing a basic fraction subtraction problem. Another suggestion was for item 3 (i.e., ) required 2 in addition to 1 and 3 . Note that another strategy for solving the problem in one of these four itemsborrowing one from a whole number to fraction, performing a basic fraction, and simplifying/reducing -happens to give the correct answer. The following example shows another strategy to solve item 9: Another attribute suggestion (i.e., 3 ) was for item 7 (i.e., ) on the top of 1 and 2 . Similar to the preceding example, a different strategy -separating a whole number from fraction, performing a basic fraction subtraction operation, and simplifying/reducing -could also give the correct answer to item 7, as in, In applying the QRM, Chiu (2013) found that item 4, which appears as item 2 in this study, did not require the possession of 3 to be correctly answered. In contrast, the QRM in this study suggested that 3 was necessary. An explanation could be because of the fact that Chiu (2013) used 20 items with 8 attributes. Whereas, the IMSSA indicated that the mastery of the third attribute was required to answer item 2 correctly. The QRM also suggested to include and exclude 2 in items 10 and 11, respectively.
Results
As demonstrated by the examples, a deeper analysis is needed. The IMSSA has more 1s than the QRM that can be controlled by adjusting the cut-offs. The cut-off values defined in the simulation study do not perfectly fit to the real data analysis in this case because it did not have a complete Q-matrix. The latter values were just approximations based on the conditions defined in the simulation study. Further discussions about multiple strategies in cognitive diagnosis using the fraction subtraction data can be found in de la Torre and Douglas (2008), Hou and Mislevy (1996) . Other reasons could be because the fraction subtraction data have fewer number of items and attributes than the simulation study. Also note that when
Results
(1) was nwilliams@example.net, three items presented different attribute specifications (i.e., items 1, 2, and 3). 3 in item 1, and 2 , 3 , and 4 in item 2 were altered to 0s; however attribute specifications in item 3 was consistent with the Q-matrix given for the data.
DISCUSSION AND CONCLUSION
CDMs aim to classify the attribute mastery or nonmastery of examinees, and the Qmatrix is needed for specifying required attributes for each item in a test. The importance of revising attribute specifications in the Q-matrix should not be underestimated due to the inherent subjectivity of domain experts, consequently resulting in serious validity concerns.
DISCUSSION AND CONCLUSION
The IMSSA for Q-matrix validation presented in this study aimed to extend the SSA (de la Torre, 2008) in several ways. First, it offered a more efficient solution as it only examines the first K single-attribute q-vectors. Second, in addition to less number of computational requirements, an iterative algorithm was included in the method to decrease negative effects of any misspecified attribute specification given in the previous iteration. And, third, an approximation was made to generally define optimal cut-off values applicable across the specific set of conditions.
DISCUSSION AND CONCLUSION
In this work, three methods without an iterative algorithm were compared to two methods with an iterative algorithm. Among the noniterative methods, the MSSA reported better results, which had higher recovery than the QRM on average across all the factors. As expected, the results showed that the IMSSA worked much better than the noniterative methods. According to the simulation studies, the IMSSA showed promising improvements in Q-matrix validation that could enhance the estimation of model parameters, model-data fit analyses, and ultimately, the accuracy of attribute-classifications.
DISCUSSION AND CONCLUSION
Using a 3.50-GHz I7 computer, it took the code the least amount of time to run the validation procedures for MSSA, followed by IMSSA, ESA, SSA, and QRM. For instance, it took 1.64, 3.11, 9.89, 24.35, and 30.00 minutes using MSSA, IMSSA, ESA, SSA, and QRM procedures, respectively, for 100 iterations under the condition in that N = 2,000, J = 30, and medium quality items with 10% misspecifications in the Q-matrix.
DISCUSSION AND CONCLUSION
This present study had some limitations. For instance, the number of attributes was assumed to be known and fixed to K = 5. It would be interesting to investigate the method by relaxing this assumption. The findings of this study were based on the attribute structure generated from a uniform distribution. The performance of the methods should be investigated under a condition where attributes were generated from a higher order distribution (de la Torre & Douglas, 2004) . Also, in addition to the δ-statistic used in this study, other statistics can be carried out for Q-matrix validation. This study should also be extended to make it applicable to a wider class of CDMs such as the G-DINA model (de la Torre, 2011 ). This will obviate the need to assume the specific CDMs involved. Finally, this method should be applied to other real data sets (e.g., Akbay, Terzi, Kaplan, & Karaaslan, 2018 ) with a complete Q-matrix so that further insights can be gained on how the proposed method could work in practice.
ORCID


{'paper_id': '67850680', 'title': 'An Iterative Method for Empirically-Based Q-Matrix Validation', 's2_url': 'https://api.semanticscholar.org/CorpusID:67850680', 'source': 'S2ORC'}