the receiver operating characteristic ( roc ) curve plots two accuracy measures of tests ( the false - positive rate and the true - positive rate ) , which are frequently used to measure and compare the accuracy of different diagnostic methods . it frequently depends on covariates such as gender and age as in the hearing impairment example in dodd and pepe.1 to explain the effects of covariates , the covariate - dependent roc models are studied extensively in the literature . two most common approaches are as follows : ( i ) one that introduces covariate - dependent error distributions2 and ( ii ) the other that directly quantifies the covariate effect on the roc curve.1,38 the clustered outcomes are multiple measurements of a diagnostic test from a single subject ( or cluster ) . they typically occur when subjects are repeatedly tested over time and are naturally correlated to each other . the random - effects model is introduced to explain the correlation among observations within a cluster . in particular , the location - scale model , where the location and scale parameters are modeled with random effects to explain the correlation among observations within a cluster , is popularly used.9,10 on another direction , there are efforts to directly model the area under the roc curve ( auc ) . dodd and pepe1 introduced the generalized estimation equation ( gee ) framework to model the auc with covariates . recently , lim et al.12 extended the study of dodd and pepe1 to incorporate the clustered outcomes by considering a wider class of gee weights and proposed a procedure to choose the optimal weights to minimize the variance of the regression estimator of interest . despite many scholastic discussions on the roc regression model , we have a limited number of models for the clustered ordinal outcomes and practitioners still have difficulties in using them . in this article , we propose a simple regression model ( a nonlinear mixed - effects model ) for the clustered ordinal test results with covariates . the proposed model is originally from the proportional hazard model for grouped - survival ( gs ) time data by hedeker et al.13 we find that the proposed model defines a new type of location - scale model and also a lehman family of roc curves , where the lehman family was proposed and extensively studied by gnen and heller.15 finally , due to the formulation of the gs time ( or gs in short ) model , our model can be estimated by fitting a nonlinear mixed - effects model , which is supported by many common statistical software , including sas and spss . the remainder of the article is composed of four sections . in the model section , we introduce the model we proposed and discuss its connection to the existing models . in the numerical study section , we present a numerical study to assess the performance of the proposed method . we apply the model to comparing different magnetic resonance imaging ( mri ) methods to detect the presence of hepatic metastases in the data example section . let yij be an ordinal marker value with k categories of the ith cluster ( or subject ) and the jth measurement ( diagnostic result ) , for i = 1 , 2 , , n , and j = 1,2 ,  , ni . let xij be a p - dimensional covariate vector and vi be a random effect to explain correlation of observations in the ith cluster . let dij be an indicator variable of the true diagnostic class , where dij = 0 and dij = 1 imply that the true class of the ( i , j)th observation is normal ( negative ) class and tumor ( positive ) class , respectively . the regression model we propose is , for k = 1 , 2 ,  , k , as follows : 
 g{pij(k|vi;xij , dij)}=0k+dij+xijt+dijxijt+vi,(1)where pij(k|vi ; xij , dij ) = p(yij  k|vi ; xij , dij ) and the link function g( ) is a monotone increasing function from ( 0,1 ) to r. the model ( 1 ) is known as the grouped - time survival model by hedeker et al.13 the model assumes that the true survival time tij is observed as a categorical response yij defined by 
 yij=k=1kki(tijik ) , where ik , k = 1 , 2 ,  , k , are predetermined disjoint and exhaustive intervals on [ 0, ) . for example , the ordinal outcomes in the next section are the medical judgments by experts based on the size or number of tumors . in applying the gs model to the data , the continuous severity measures of a disease ( for example , the size of tumors ) are considered as the true survival times and the ordinal diagnostic outcomes are read as the observed gs times . by reading the ordinal results as gs times , we have a handy and interesting class of nonlinear random - effects models for the roc curves of correlated categorical diagnostic results . in particular , the model we propose is also closely related to the several models in the previous literature . first , the model ( 1 ) with the link function as g(p ) = log(log p ) defines an extension of the lehmann family of the roc curves to the model with random effects . suppose x is a covariate vector attached to the test results y. the lehmann family of the roc curves is defined as a collection of the roc curves of the form : 
 r(u;x)=s1(s01(u;x))=uexp(+xt),(2)where 
 s0(t;x)=s0(t)exp(+xt)ands1(t;x)=s0(t;x)exp(+xt).(3 ) here , s0(t ) is the survival function ( = 1  cumulative distribution function ) of the outcome of normal subject with x = 0 . it assumes that the survival functions of normal and diseased subjects have the proportional hazard specification on the covariates . the hazard rate at ( the outcome value ) t is the instantaneous rate that we have a diagnostic outcome value at t when its value is known to be not < t. the proportional hazard means that the covariates are multiplicatively related to the hazard rate . given the cluster - specific random effect v , our model specifies 
 s0(t|v;x)=s0(t)exp((t)+xt+v)ands1(t|v;x)=s0(t|v;x)exp(+xt),(4)and its roc curves forms the same lehmann family ( 2 ) mentioned earlier . in addition , if we further assume that log vi is distributed as a gamma distribution , a simple algebra shows that the marginal model ( integrating out vi ) also defines the same lehmann family . second , the model ( 1 ) is closely related to the location - scale roc model introduced in pepe ( page 151 in chapter 6.3),16 that is 
 p(yijk|vi;xij , dij)=s0(0k+dij+xijt+dijxijt+vi).(5 ) the model in ( 4 ) tells that , given vi , the survival function of yij is 
 logs(t|vi;xij , dij)=exp{(t)+dij+xijt+dijxijt+vi}logs0(t),(6)and it defines the location - scale family mentioned earlier if log s0(t ) = exp(t ) . in addition , our model ( 4 ) assumes that the intercept is random in s0(t|v;x ) , and all other coefficient vectors are fixed , not cluster - specific random . this simplification makes the cluster - specific roc curve be the same with ( 2 ) . our primary goal of the roc analysis is the comparison of different diagnostic methods . to do it , we introduce dichotomous covariates x indicating the choice of methods and test their coefficients in model ( 1 ) . in our motivating example ( the example followed in the next section ) , we have three imaging methods by two readers ( two medical doctors who read the images ) and we use the three - dimensional covariate vector with two levels ( 0 or 1 ) , x = ( xr , xm2 , xm3 ) , where xr = 1 implies the outcome is recorded by the second reader , xm2 = 1 implies the outcome is from the second imaging method , and xm3 = 1 implies the outcome is from the third imaging method . finally , the proposed model ( 1 ) is a nonlinear mixed - effects model , which is well studied in the literature . many refined approximations to the likelihood function of the model are proposed and encoded into common statistical packages , including sas and spss . the comparison of different diagnosis methods could be done by testing the regression coefficients of the covariates , indicating the choice of diagnostic methods . this feature is illustrated in details by analyzing a real data in the next section . we refer readers to davidian and giltinan17 for the details of the nonlinear mixed - effects model . in this section , we conduct a small numerical study to access the performance of the proposed gs model . the study considers one sample problem to estimate and test the effectiveness of a single diagnostic method with the auc instead of comparing the rocs of two or more diagnostic methods . thus , we have the diagnostic outcomes of two populations , say the normal and cancer populations , by a given diagnostic method . the number of subjects from each population is set as 25 ( n = 50 (= 25 + 25 ) ) and 50 ( n = 100 (= 50 + 50 ) ) . the number of repeated observations per subject is set as m = 2 and m = 4 . the ordinal data are generated by categorizing exponentially distributed random variables as follows . for the ith subject of the normal population , we generate m continuous repeatedly measurements from the exponential distribution with the rate  = 0.1vi , where log vi is from normal distribution with mean 0 and variance 
 v2 . for the subject i in the cancer population , the repeated measurements are from the exponential distribution with the rate  so that 
 =0.1viexp(),(7)where log vi is again from normal distribution with mean 0 and variance 
 v2 . the variance 
 v2 is considered as 1 and 3 , where 
 v2=3 introduces higher correlation among observations within a subject than 
 v2=1 . we transform them to the ordinal data ( with five levels indexed by k = 0 , 1 , 2 , 3 , 4 ) using pre - decided grids , which are the five quantiles of the exponential distribution with  = 0.1 . in ( 7 ) ,  = 0 implies that there is no difference in the distributions of diagnostic outcomes of normal and cancer populations . in the study , we vary  = 0 , 0.2 , 0.4 , 0.6 , 0.8 , 1.0 and consider the powers in testing h0 :  = 0 ( versus  > 0 ) as a measure of effectiveness of the procedure . to test the hypothesis , we apply the proposed gs model with link function 
 log{log(1pij(k|vi;dij))}=0k+dij+vi,(8)where di is the indicator variable for the cancer population ( it has value 1 if the ith subject is from the cancer population , otherwise , 0 ) and pij(k|vi ; dij ) = p(yij # k|vi ; di ) for k = 0 , 1 , 2 , 3 , 4 . the hypothesis is tested by the z - statistic 
 z=^/{var^(^)}1/2 , and both 
 ^ and 
 var^(^ ) are obtained using the nlmixed procedure in sas version 9.3 . for the comparison , we consider the auc estimate based on the mw statistic , which is popularly used in practice . here , the mw statistic does not take into account the within - cluster correlation and treats all observations as independent samples . the sample auc for the ordinal data is computed as the mw statistic with ties as 
 u=1ndnndi=1ndj=1nnduij,(9)where uij = 1 , if 
 yjd > yjnd ; uij = 1/2 , if 
 yid = ylnd ; and uij = 0 , if 
 yid < y . here , nd is the total number of observations from the disease population and nnd is that from the normal population . in our case , nd = nnd = n  m. the asymptotic variance of the auc under the assumption of independence of all outcomes is given as follows : 
 var^(u)=ndnnd12{nd+nndk=15sk(sk21)(nd+nnd)(nd+nnd1)},(10)where sk is the number of observations whose scores are j for k = 1 ,  , 5 . the test for non - effectiveness of the diagnostic method is done using the standardized statistics 
 zmw = u/{var^(u)}1/2 , which follows approximately the standard normal distribution under the null . we generate 1000 data sets for each case of 
 v2=1,3 ( the variance of random effects ) and m = 2 , 4 ( the number of repetitions with a subject ) and evaluate the empirical power by counting the number of rejections among 1000 data sets . the empirical powers of the gs method and mw statistic are plotted in figure 1 . in the figure , mw ( size corrected )  is the empirical size correction of mw test , where 100(1  )th empirical percentile of the mw statistic for  = 0 is used as the critical value , instead of z/2 , the 100(1  /2)th percentile of the standard normal distribution . mw ( size corrected )  is simply added as a reference and is not applicable in practice because we do not have those statistics from the null . figure 1 shows that the size of mw - based test ( the power when  = 0 ) is biased significantly and its magnitude increases as either 
 v2 increases or the number of repetition m increases . both increases of 
 v2 and m imply the increase in the correlation among repeated observations within a subject ( or cluster ) . on the other hand , the size of the proposed gs model - based test is approximately at the aimed level 0.05 , regardless of 
 v2 and m. the powers of gs - method are comparable to the size - corrected mw test in all cases considered . the liver is the most frequent site of metastases from various extrahepatic malignancies , and determining the presence of hepatic metastases is important in order to provide the optimal plan for patients who are candidates for surgery and in order to assess prognosis after initial treatment . the data analyzed in this article are the records of patients who underwent liver mri with separate acquisition of double contrast enhancement between november 2005 and june 2006 . the data are collected from the database of the radiology department at asan medical center in seoul , republic of korea . the data record the test results of the 106 focal liver lesions from 36 patients 
 ( n=36andi=1nfi=106 ) , where fi denotes the number of the focal hepatic lesions from the ith subject . we take a picture of each lesion with three different methods : mri with multihance ( set a ) , mri with resovist ( set b ) , and the combination of the original mri with resovist and dynamic mri ( set c ) . two readers determine the possibility of malignancy of the detected lesions using a 5-point confidence rating scale ( definitely not = 0 , probably not = 1 , possibly = 2 , probably = 3 , and definitely = 4 ) . then , each lesion has six ratings ; the combination of two readers ( two medical and three imaging methods ) and a subject has six x fi(= ni ) ratings in total . the goal of this experiment is comparing the performance of three picturing methods and two readers . more details on the data including how the data are corrected are available from the study of hong et al.14 a summary of the data is presented in table 1 . let yij be the jth rating of the ith subject having an ordinal integer mark value between 0 and 4 , for i = 1 , 2 ,  , 36 and j = 1 , 2 , , ni ; let dij be the binary variable , which equals to 1 if the ( i , j)th rating is from the disease ( positive ) group in truth , otherwise 0 . for the ( i , j)th rating , let xij = ( xijr , xij2 , xij3 ) be the three - dimensional covariate vector of indicator variables , where xijr = 1 if the rating is done by the first reader , xij2 = 1 if the rating is on the mri by set b , and xij3 = 1 if it is by set c. we apply the model ( 1 ) with the complementary loglog link , so that we have 
 log{log(1pij(k|vi;xij , dij))}=0k+dij+xijt+dijxijt+vi,(11)where pij ( k|vi ; xij , dij ) = p ( yij  k|vi ; xij , dij ) ,  = ( r , 2 , 3 ) and  = ( r , 2 , 3 ) . let vi be a random effect from the ith subject and 
 vi~iidn(0,2 ) . here ,  measures the difference in outcomes between benign and metastasis lesions ; r , 2 , and 3 measure the difference between two readers , the difference in imaging methods between set a and set b , and the difference between set a and set c , respectively ; r is the interaction effect between the reader and existence of disease ; and 2 and 3 are the interaction effects between the imaging methods and the existence of disease . here , the non - zero  implies that the readers and the methods perform differently between the normal and disease groups . table 2 displays the parameter estimates calculated from the data , and figure 1 plots the estimated roc curves for the six combinations of readers and mri methods . the results tell that the mri methods and the reader do not perform differently for normal liver lesions . however , for the tumor lesions , the mri method set c performs better than set a ( p - value = 0.0044 ) . in addition , the p - value for jointly testing h0 : 2 = 3 = 0 is 0.0160 and that for testing h0 : 2  3 = 0 is 0.0490 . this implies that the mri method set c performs better than any of set a and set b for tumor lesions . in tumor groups the covariate - specific roc curve r(u|x , d ) from ( 11 ) is in the form of 
 r(u;x)=uexp(+xt)(12)and its auc is a(x ) = 1/{1 + exp( + x)}. figure 1 plots the estimated roc curves along with their empirical roc curves . here , the empirical roc curves do not take into account the correlation among outcomes within a cluster . for both readers , the curves for set c are higher than set a and set b , indicating that taking pictures of lesion with combination of original mri with resovist and dynamic mri method is superior to use only a single mri method ( fig . 2 ) . table 3 summarizes the estimates of the aucs for the combinations of a reader and an imaging method . the standard errors ( ses ) of the model - based estimates of the auc are obtained by the delta method . we also report their empirical estimates without taking account the within - cluster correlation of outcomes by obuchowski.11 the formulas of empirical estimates can also be found from pepe ( chapter 6.3).16 the aucs may not be always sensible to detect the differences for specific covariates , regardless of whether they are model based or empirical , since they are functional forms of many other components as given : 
 a(x;,r,2,3)=1/{1+exp(+xt)}.(13 ) on the other hand , the proposed regression model can test the contribution from each covariate separately . to be specific , in our example , if we want to find the performance difference between imaging methods of set a and set c for reader 1 , the ( model - based ) auc estimates are 0.914 and 0.969 , respectively . the 95% confidence interval for auc of set c is ( 0.793 , 1 ) , which overlaps the confidence interval of auc for set a , ( 0.876 , 0.952 ) . however , the test based on the proposed regression model makes it possible to test the significance for particular parameter . for example , the p - value of test h0 : 3 = 0 is 0.0044 , and it indicates the existence of significant interaction between sets ( a and c ) and disease groups ( disease and non - disease ) at  = 0.05 ; this implies that the imaging methods a and c perform differently in detecting the cancer . in this article , we propose a new roc regression model for clustered ordinal outcomes . the new model views the ordinal outcomes as gs times and uses the grouped - time survival model to define the regression model of the roc curve . it is shown that the proposed model is closely related with many existing models including the lehman family and the location - scale family of the roc curves and further provides their extensions to the random - effects models . our proposed model has an additional advantage of being easily programmed in many standard statistical packages , which makes it easy to use and interpret . in summary , the model proposed in this article provides a flexible exploratory tool for identifying covariate effects on the roc curve with clustered ordinal outcomes .
in this article , we propose a regression model to compare the performances of different diagnostic methods having clustered ordinal test outcomes . 
 the proposed model treats ordinal test outcomes ( an ordinal categorical variable ) as grouped - survival time data and uses random effects to explain correlation among outcomes from the same cluster . to compare different diagnostic methods , 
 we introduce a set of covariates indicating diagnostic methods and compare their coefficients . 
 we find that the proposed model defines a lehmann family and can also introduce a location - scale family of a receiver operating characteristic ( roc ) curve . 
 the proposed model can easily be estimated using standard statistical software such as sas and spss . 
 we illustrate its practical usefulness by applying it to testing different magnetic resonance imaging ( mri ) methods to detect abnormal lesions in a liver .