Abstract
It is a commonly accepted assumption by educational researchers and practitioners that an underlying longitudinal achievement construct exists across grades in K-12 achievement tests. This assumption provides the necessary assurance to measure and interpret student growth over time. However, evidence is needed to determine whether the achievement construct remains consistent or shifts over grades or time. The current investigative study uses a multiple-indicator, latent-growth modelling (MLGM) approach to examine the longitudinal achievement construct and its invariance for the measures of academic progress® (MAP®), a computerised adaptive test in reading and mathematics. The results of the analyses from ten states suggest that with repeated measures, the construct of both MAP reading and mathematics remained consistent at different time points. The findings support the achievement construct's invariance throughout different grades or time points and provide empirical evidence for measuring student growth.
Introduction
Most current achievement tests can be characterised by their test algorithm and administration platforms. Two major testing algorithms are commonly employed in achievement tests: one is the linear test, in which all students answer all test items. Another is the computerised adaptive test (CAT), in which different students may get different items. During adaptive testing, each item that administrates to an individual student adapts to his or her provisional achievement ability by selecting the most appropriate item difficulty. The most common platforms for achievement tests are paper and pencil and computer. When the linear test algorithm is used in computer platforms, the test is called a 'computer-based test'; when the CAT algorithm used in computer platforms, the test is a 'CAT'.
Introduction
For decades, achievement tests delivered either in linear or CAT algorithms have been constructed to provide formative or summative measures about students' achievement status at grade level (for example, Amy's third-grade mathematics test score in Fall 2005), academic growth over time from longitudinal data (for example, Amy's third-grade mathematics test score in Fall 2005 and her seventh-grade mathematics score in Fall 2009) or achievement across grades from cross-sectional data (for example, Amy's third-grade, Johnny's fourth-grade and Tim's fifth-grade mathematics scores in Fall 2005) (Hamilton et al., 2008; Patz, 2007; Smith and Yen, 2006; Yen, 2007 Yen, , 2009 ). In recent educational reforms, assessing individual students' growth has been required for high-stakes decisions by state and federal education policy, such as the race to the top initiatives (RTTT) (U.S. Department of Education, 2009). These requirements have put tremendous pressures on states and testing companies to develop high-quality and high-utility assessment systems. As RTTT states: "(Student) growth may be measured by a variety of approaches, but any approach used must be statistically rigorous and based on student achievement data, and may also include other measures of student learning in order to increase the construct validity and generalizability of the information" [U.S. Department of Education, (2009), p.37812] . CATs are considered more effective than linear tests for measuring individual students' growth over time (Way et al., 2010) . The advantages of CATs over linear tests include shorter tests, immediate student score reporting, higher reliability and measurement accuracy (Kingsbury and Weiss, 1983; Lord, 1977; Thissen and Mislevy, 1990) , cost savings and multiple testing opportunities for formative and interim assessments (Way, 2006) .
Introduction
Measuring individual student growth has two fundamental requirements (Bergman et al., 1991; Betebenner and Linn, 2010; Doran and Cohen, 2005; Linn, 1993; Mislevy, 1992; Williamson et al., 1991) . First, there are multiple measures of achievement construct along the growth trajectory. Second, the achievement construct should be invariant from different grade levels or points in time.
Introduction
This means that in order to measure student growth, achievement tests must satisfy two necessary conditions: First, there must be a continuous construct that the tests are designed to measure over grades or time; i.e., the tests are scored on a longitudinal construct. Second, the construct measured by multiple tests must be invariant, or constant, across grade levels and time; i.e., the construct does not shift over grade levels or time (Wang and Jiao, 2009 ).
Introduction
Many researchers have investigated whether achievement tests satisfy these two assumptions from the perspectives of content and vertical scaling (Cizek, 2005; Linn, 2001; Lissitz, 2006; Martineau et al., 2007; Wang and Jiao, 2009; Wise, 2004) . The study by Wang and Jiao (2009) provided empirical evidence to support construct invariance across grades for a vertically scaled norm-referenced test. However, few studies examine the longitudinal achievement construct and construct invariance across grade levels and points in time with CAT. Achievement construct invariance is an important validity issue. According to the Standards for Educational and Psychological Testing [American Educational Research Association (AERA), American Psychological Association (APA) & National Council on Measurement in Education (NCME), 1999], validity is the most important consideration in test development and evaluation. Validity refers to the degree to which empirical evidence and theoretical rationale support the inferences and actions based on test scores (Messick, 1989) or the degree to which evidence and theory support the interpretations of test scores for the proposed use of tests (American Educational Research Association, American Psychological Association & National Council on Measurement in Education, 1999) . Interpretations of test results that measure achievement ability or traits are subject to many validity threats, including two major threats: construct-under representation (CUR) and construct-irrelevant variance (CIV) (Messick, 1984) . The construct of a test is a theoretical representation of the underlying traits, concepts, attributes, processes or structures the test is designed to measure, and it directly relates to test validity (Cronbach, 1971; Messick, 1989) . Five sources of validity evidence specified in the Standards include a test content b response process c internal structure d relations to other variables e consequences of testing.
Introduction
CUR and CIV could be identified in the process of test development, administration and use of test results. This study focuses on validity evidence of internal structure and the invariance of the internal structure of achievement tests across grades and/or over time.
Introduction
Factorial validity can be a valuable component of validity evidence (Guilford, 1946; Messick, 1995) . Although evidence of a test's internal structure is routinely reported in many test technical reports or manuals from state assessment programmes and test companies, construct invariance is rarely addressed from a longitudinal perspective when test results are routinely used to determine student growth. Many test publishers have used the first-order latent growth curve model (FLGM) (Bollen and Curran, 2006; McArdle et al., 2009; Muthén, 1995) in longitudinal studies of student growth, based on the assumption that there is a continuous test construct. Unfortunately, little attention has been drawn to examine the assumption in practice. The second-order latent growth model, or the multiple-indicator latent growth model (MLGM), is rarely used to evaluate the longitudinal construct validity for adaptive test achievement based on CAT. There are two advantages of MLGM over FLGM (Bollen and Curran, 2006; Ferrer et al., 2008; Muthén and Muthén, 2007; Sayer and Cumsille, 2001) . First, MLGM can test, instead of making assumptions, whether the same latent construct is measured at each point in time or grade level. Second, it accounts for important information of the psychometric properties on the indicators. MLGM can be used to evaluate factorial invariance across points in time and determine whether the same latent construct is measured across time or grade levels so as to assure that changes of test scores quantify achievement growth rather than the shift of the achievement test construct.
Introduction
FLGM uses total scores as an observable variable or indictor in the longitudinal achievement growth analysis, while MLGM uses observed indicators either at the item level or the cluster of items level (testlet, subtest, goal score, etc.). The choice of the level of an indicator in MLGM or in general factor analysis has a significant effect on evaluating the construct, because item cluster involves averaging item scores and using summed scores as observed variables in analyses (Bandalos, 2002; Bandalos and Finney, 2001; Hall et al., 1999; Little et al., 2002; Nasser and Wisenbaker, 2003) . Some reasons to use clusters of items are to reduce the problems of non-normality, to have fewer free parameters to be estimated compared to the number of observations and to improve data-model fit. Arguments state that using clusters of items increases the chance of combining items that truly measure multiple dimensions and therefore results in severe bias. Overall, clustering items is a commonly used technique based upon theoretical rationale.
Introduction
Compared with linear tests, it is almost impossible to use indicators at the item level for adaptive tests because each student receives different items tailored to his or her ability. Using unique test forms with CATs makes evaluating longitudinal achievement constructs complicated. Compared with linear tests, CATs present two major challenges in using observable variables for MLGM. First, observable variables are different across persons at the item level. Second, even though the observable variables are the same at the cluster level, the context of the observable variables is different; i.e., the same subtest score may consist of different items for the same content. One possible solution to the first problem is to conduct a confirmatory factor analysis at the item level on the entire item bank. However, the drawback is the large amount of missing response data in the dataset. For example, the missing rate for the measure of academic progress (MAP, Northwest Evaluation Association, 2011) is around 98% for the reading and mathematics tests. Besides, the data are very sparse, because the ratio of the test length to the size of the item bank is about 1:50 . The commonly used imputation method (Rubin, 1987) may statistically help solve the problem, but it cannot deal with the missing data issue from the content perspective. Another possible solution for improvement is to treat items as nested inside person within the framework of generalisability theory, which provides information on parcel scores reliability.
Introduction
The purpose of this study is to investigate the longitudinal achievement constructs of a standardised, large-scale CAT in reading and mathematics across ten states.
Method

Data source and participants
All data used in this study were collected from the measures of academic progress (MAP) assessment system from Spring 2009 through Spring 2011. The MAP reading and mathematics tests were administered to students in grades 3-10 across 50 US states. The data analysis only focused on the ten states (Colorado, Illinois, Indiana, Kansas, Kentucky, Michigan, Minnesota, South Carolina, Washington and Wisconsin) that supplied the largest samples among the 50 states. . Each state sampled the corresponding population under the constraints that students must have five academic calendar records (five-wave) from grade 5. Student demographic data is collected legally and ethically through the Master Services Agreement with each NWEA partner. And at no time is personally identifiable information (PII) released without the written consent and permission of the associated partner. All data collection, storage and use in studies meets the requirements of the Family Educational Rights and Privacy Act (FERPA, U.S. Department of Education, 2012). Prior to each test administration, NWEA partners roster students who will take the MAP, and it is through this rostering process that student PII is provided (student first, last and middle name; date of birth, grade, gender and ethnicity). This data assists in properly identifying and tracking students from one test administration to another in order to assign the correct test scores, goal data and other relevant test characteristics.
Data source and participants
Although this paper does not focus on the effect of heterogeneity of samples on parameter estimates of the MLGM, a previous study (Zhang and Wang, 2012) found statistically significant different rates of change by gender, using across-grade cohorts with a non-linear hierarchical linear model (HLM). Table 1 and Table 2 list the frequency distributions and percentages of reading and mathematics for sample states (due to limited space) across five-wave by state, gender and ethnicity. 
Instruments
MAP has been published by Northwest Evaluation Association ™ (NWEA ™ ) since 1976, and all MAP tests are computerised and presented adaptively. The purpose of MAP is to provide educators with information to inform teaching and learning in reading, language usage, mathematics and science (NWEA, 2011) . The MAP tests align with the content standards of each state by assembling a customised item pool to measure the specific standards. The variation in the items pools selected across states is reflected in different state tests in terms of the number of goals shown in the Table 3 . For example, the MLGM for the South Carolina reading test will consist of three goals, and the MLGM for the Michigan mathematics test will consist of six goals.
Instruments
Unlike state assessment programmes used to report proficiency under NCLB, MAP tests allow certain off-grade items to be used for on-grade assessment for the purpose of measuring growth. Test algorithms survey the pool within each content standard goal or strand level to assure the content domain coverage. The marginal reliabilities of test scores are consistently in the low-to mid-0.90s range across grades, tests and states (Northwest Evaluation Association, 2011) . MAP items are calibrated with the Rasch model (Rasch, 1961) , and all MAP tests are vertically scaled (Northwest Evaluation Association, 2011).
Instruments
In the process of item selection, all items administered to each student must satisfy the content requirements of each test to ensure the content validity of the test. Table 3 lists test length (fixed-length CAT) and numbers of goals (subtests) for reading and mathematics for the 10 states. A sample of content specifications for reading in Colorado and mathematics in Indiana is shown in Table 4 . Due to the uniqueness of CAT test forms, the observed variables used in this study are goal scores (item clusters) under the assumption that the item cluster for each goal area contains homogeneous content across students. All goal scores are scale scores on the same metric across goals and time periods. Statistics, data analysis, and probability
14%
Problem solving 14% Figure 1 The MLGM at five time points with linear growth structure and invariance of factor loadings Figure 2 The MLGM at five time points with quadratic growth structure and invariance of factor loadings
Multiple-indicator, latent-growth model (MLGM)
MLGM is a multivariate extension of FLGM (Bollen and Curran, 2006; Ferrer et al., 2008; McDonald, 1985; McArdle, 1988; Muthén, 1991; Tisak and Meredith, 1990) . If y jti denote the observed variables (goals) for individual i, indicator j and time point t, and let η ti denote a latent variable construct, the level-1 model for measurement part is:
Multiple-indicator, latent-growth model (MLGM)
where τ jt is intercept for the j th indicator in the t th time period, λ jt is the factor loading for the j th indicator at the t th time point, and ε jti is the random error for the i th individual in the t th time point and the j th indicator. Level-1 models for a latent variable with both linear and quadratic growth are:
Multiple-indicator, latent-growth model (MLGM)
Quadratic growth
Multiple-indicator, latent-growth model (MLGM)
where ζ ti is the random normal error for the i th individual in the t th time point; η 0i , η 1i and η 2i are the intercept, slope and quadratic of latent factors, respectively, for individual i; 
Multiple-indicator, latent-growth model (MLGM)
where ζ 0i , ζ 1i and ζ 2i are normal random errors; α 0 , α 1 and α 2 are latent means of intercept, slope and quadratic terms for individual i and w i is weight. All MAP tests have been tested as one latent factor . Figure 1 and Figure 2 present the MLGM with linear and quadratic latent growth.
Measurement invariance of using MLGM
The measurement invariance (Drasgow, 1987; Ferrer et al., 2008; Meredith, 1993) evaluated in this study is the invariance across time points of testing. Although values of manifest variables are different across time in longitudinal studies, they should be on the same measurement scale to derive an equal definition of a latent construct across time. Widaman and Reise (1997) classified two types of factorial invariance as non-metric (configural) and metric. Configural invariance (CI) refers to the same indicators of the latent construct. The metric factorial invariance has three hierarchical levels, which are categorised as weak invariance (WI), where the factor loading of each indicator is invariant over time; strong invariance (SI), where the factor loading and intercept of each indicator are invariant over time; and strict factorial invariance (SFI), where the factor loading, intercept and unique variance of each indicator are invariant over time. Sayer and Cumsille (2001) showed that the SFI is unlikely to hold because heterogeneous variance across time is often observed. In this study, only CI, WI and SI are analysed. The invariance tested is summarised in three conditions: the CI, the WI that can be expressed as equation (9) and the SI that can be expressed in equation (10) and equation (11):
Measurement invariance of using MLGM
where λ 1 = 1, α 0 = 0 and variances of ε jti and ζ ti may vary over time. And for structural differences, the mean of η ti and variance of η ti vary over time.
Measurement invariance of using MLGM
Several well-known goodness-of-fit (GOF) indices were used to evaluate the model fit. They are 1 absolute indices that include chi-square χ 2 and standardised root mean square residual (SRMR) 2 incremental indices that include the comparative fit index (CFI) and Tucker-Lewis Index (TLI) 3 parsimony index, the root mean square error of approximation (RMSEA).
Measurement invariance of using MLGM
For nested models that include a different shape of growth (e.g., linear versus quadratic), both Akaike's Information Criterion (AIC; Akaike, 1987) and Bayesian Information Criterion (BIC) (Schwartz, 1978) are obtained for each model tested. According to Raftery (1995) , the values of BIC difference (BIC of quadratic model -BIC of linear model) ranging from 0 to 2 are interpreted as weak evidence for quadratic model, values of 2 to 6 are interpreted as moderate, values of 6 to 10 are interpreted as strong and values > 10 are interpreted as very strong. For model comparisons with increased constraints, the χ 2 value also provides the basis of comparison with the previously fitted model in addition to AIC and BIC. χ 2 is not considered best in practice because it is sample-size dependent. A non-significant difference in χ 2 values between nested models reveals that all equality constraints hold across time. Therefore, the measurement model remains invariant across groups as the constraints are increased. A significant χ 2 value does not necessarily indicate a departure from invariance when the sample size is large. Hu and Bentler (1999) recommended using combinations of GOF indices to obtain a robust evaluation for model-data fit in structural equation modelling. The cut-off criterion values of a good model-fit that they recommended are CFI > 0.95, TLI > 0.95, RMSEA < 0.06 and SRMR < 0.08. It is worth noting that many researchers (Marshet al., 2005; Marsh, 2007; Sayer and Cumsille, 2001 ) consider the GOF criteria from Hu and Bentler too restrictive. All analyses are conducted in Mplus 5.1 (Muthén and Muthén, 2007) .
Analytic approach
The different MLGM models have been used for different state MAP test results because different states have different test specification or test blueprints. For each state, both linear and quadratic MLGM models are fitted to MAP test data. All missing data are handled by using Mplus default and according to Muthén et al. (1978) , it is preferable to pairwise deletion (LISTWISE = ON missing data is dealt using pairwise deletion) because pairwise deletion requires missing completely at random (MCAR), not missing at random (MAR). Figure 3 and Figure 4 illustrate a sample of observed and estimated individual quadratic growth based on MGLM across five-wave academic terms. Table 5 and Table 6 display the summaries of GOF indexes of MLGM data fit for linear and quadratic growth in reading and mathematics across states. All values of the fit indexes satisfy the Hu and Bentler (1999) criteria in both content areas and show that each model fits the data extremely well across states, with one exception in Indiana for mathematics. The overall results suggest that both linear and quadratic MLGMs are reasonably good models for MAP tests in ten states. For AIC, the lower value (positive or negative) indicates a better fit than the higher value. The results show that the quadratic model fits data better than the linear model in the nested modelling comparison. For BIC, all differences between quadratic and linear models are greater than 10 in both reading and mathematics, which indicate that the quadratic model fits the data better than the linear model. The statistically significant χ 2 difference between the quadratic and linear models provides additional evidence to support the conclusion that the quadratic model is a better fit for the data than the linear model. It is also important to note that both the linear and quadratic models show that the longitudinal achievement construct underlying achievement measures equally well in growth. Table 7 and Table 8 present the summaries of GOF indexes with nested linear MLGM that was used for measurement invariance across five waves in reading and mathematics. Nearly all fit indexes satisfied Hu and Bentler's criteria in both reading and mathematics tests across states except Indiana. Some SRMRs seemed to be slightly above Hu and Bentler's criteria. In evaluating measurement invariance, the simple model is a restricted model and the complex model is an unrestricted model. The effect of constraints imposed on the less restricted model can be evaluated by using the difference of χ 2 (Δχ 2 ) for nested model comparisons, because the degree of freedom is equal to the difference in the degrees of freedom of two models. Results indicate that all χ 2 increases (Δχ 2 ) are statistically significant for evaluating the differences of invariance between unrestricted and restricted models. As χ 2 becomes statistically significant, a more complex model should be chosen. However, the limitations of the χ 2 test are the sample size dependency (Cheung and Rensvold, 2002) and the difference of other GOF indexes (such as CFI) as adjuncts to the χ 2 statistic, which can also be used to assess model fit. According to Cheung and Rensvold (2002) , if the difference of CFI (ΔCFI) is less than 0.01 between the two models, the simple model is not worse than the complex model. The value of all ΔCFIs less than 0.01 in both tests indicates that constrained parameters are invariant across time.
Results

Results of MLGM

Results of invariance of MLGM
In summary, the results of analyses in this study provide clear support to the CI, WI and SI for all tests except the Indiana mathematics test. These results suggest that longitudinal constructs of MAP tests are well defined for measuring student achievement growth. 
Discussion
Since the factor structure of a test is directly related to the construct validity interpretation of the test at a particular point in time, the longitudinal factor structure at different points in time is crucial for the longitudinal construct validity interpretation to measure student growth. The achievement constructs of a test at a particular time, grade level or semester calendar is well studied and reported in practice for given purposes and related interpretation of test scores. Although many standardised achievement tests in large-scale assessments report test scores on a vertical scale for student growth and group achievement trends, few studies reported the longitudinal achievement construct. Many researchers are interested in whether the longitudinal achievement construct remains the same over time or shifts from time to time from content standard and vertical scaling perspectives. A few studies have focused on validation of longitudinal achievement construct using the MLGM approach, especially for studies based on CAT longitudinal data. First, this study examined the hypothesis of shapes of latent construct across time. As shown in Table 5 , the shape of the growth fits both linear and quadratic MLGM well; however, quadratic growth has slight advantages over linear growth in terms of fit statistics. This means that between two competing interpretations of longitudinal constructs across grades, quadratic growth makes more real sense because, in general, the rate of changes of student achievement across grades are not constant. From longitudinal perspective, for example, lower grades growth is always faster than higher grades growth across time for both reading and mathematics achievement tests.
Discussion
Second, the present study tested the hypothesis of factorial invariance of MAP reading and mathematics tests over time. The evidence collected in the study shows that with repeated measures, the construct of both reading and mathematics remained consistent at different points in time, which supports the internal structure of MAP design for intended purposes. The evidence also suggest that there are not only configure and WI, but also SI of the longitudinal construct in MAP reading and mathematics tests across different states (except the Indiana mathematics test), which supports valid interpretations of student growth.
Discussion
The current study utilised the advantages of MLGM over FLGM to investigate the longitudinal construct of achievement test. No longitudinal achievement constructs studies using MLGM have been done on CAT data across states. The major difference between FLGM and MLGM is that the FLGM uses total scores as observable variable in analysis, while the MLGM uses item cluster (or item) as observable variable. However, since the study was based on longitudinal data, students who missed one test were excluded from the sample, which might introduce sample bias.
Discussion
In summary, this study underscores the importance of empirical evidence in validating longitudinal achievement constructs to support the interpretation of student growth. In particular, the study explored the feasibility of assessing the internal structure of MAP tests using CAT data. The results support consistent and reasonable interpretations of the MAP reading and mathematics tests across academic calendar years used by different states. This study carries the validation process beyond a traditional construct validation process in which validation evidence is usually collected at one point in time only, but used to support the longitudinal achievement construct for student growth. It is indeed important to investigate the longitudinal achievement construct to ensure that the same construct is measured over time for a valid interpretation of student achievement growth. We strongly recommend that achievement test publishers and users continue investigating the longitudinal achievement construct and construct invariance over time in the near future to support valid interpretations of student academic growth.
Limitations of the study
First, although the current study takes the advantages of MLGM over FLGM, the study could not avoid using item parcels or clustering items as indicators of the MLGM for CAT data. The effect of using aggregated indicators vs. individual items is unknown and item parcels' interpretability has not been sufficiently examined with respect to content. Second, it is well-known fact that heterogeneity in sample may bias the model estimates (Muthén, 1989) . All samples used in this study are not homogeneous with respect to at least gender and ethnicity, and the impact of heterogeneity on model parameter estimates is unknown.

{'paper_id': '145675331', 'title': 'Validation of longitudinal achievement constructs of vertically scaled computerised adaptive tests: a multiple-indicator, latent-growth modelling approach', 's2_url': 'https://api.semanticscholar.org/CorpusID:145675331', 'source': 'S2ORC'}