ing argument). In addition, these more detailed QATs include questions about statistical analyses, control subjects, and other methodological features deemed relevant to minimizing systematic error. These QATs usually take around 30 to 40 minutes to complete for each study. Despite the added complexity of these more detailed QATs, their scoring systems are kept as simple as possible. For instance, most of the questions on the Cho and Bero QAT allow only the following answers: 'yes' (2 points), 'partial' (1 point), 'no' (0 points), and 'not applicable' (0 points). This is meant to constrain the amount of subjective judgment required when generating a QAT score. Although most QATs share at least several similar features, the relative weight of the overall score given to the various features differs widely between QATs. Table 1 lists the relative weight of three central methodological features-subject randomization, subject allocation concealment (or 'blinding'), and description of subject withdrawal-for the above QATs, in addition to three other QATs. 9 Table 1. Number of methodological features used in six QATs, and weight assigned to three widely shared methodological features (adapted from Jüni et al. 1999). Scale Number of Items Weight of Randomization Weight of Blinding Weight of Withdrawal Chalmers et al. (1981) 30 13.0 26.0 7.0 Jadad et al. (1996) 3 40.0 40.0 20.0 Cho & Bero (1994) 24 14.3 8.2 8.2 Reisch et al. (1989) 34 5.9 5.9 2.9 Spitzer et al. (1990) 32 3.1 3.1 9.4 Linde et al. (1997) 7 28.6 28.6 28.6 Note two aspects of Table 1. First, the number of items on a QAT is highly variable, from 3 to 34. Second, the weight given to particular methodological features is also highly variable. Randomization, for instance, constitutes 3.1% of the overall score on the QAT designed by Spitzer et al. (1990), whereas it constitutes 40% of the overall score on the QAT designed by Jadad et al. (1996). The differences between QATs explains the low inter-tool reliability, which I describe in §4. But first I describe the low inter-rater reliability of QATs. 3 Inter-
ing argument). In addition, these more detailed QATs include question about statistical analysis, control subjects, and other methodological feature deemed relevant to minimizing taxonomic error. These QATs usually take about 30 to 40 minutes to complete for each sketch. Despite the added complexity of these more detailed QATs, their marking systems are kept as simple as possible. For instance, most of the questions on the Cho and Bero QAT let only the following answers:' yes' (2 point),' partial' (1 point),' no' (0 points), and' not applicable' (0 points). This is mean to constrain the amount of subjective judgment required when generate a QAT score. Although most QATs share at least several similar features, the relative weight of the overall score given to the various features differs widely between QATs. Table 1 lists the relative weight of three cardinal methodological feature - subject randomization, national allocation concealment (or' blinding'), and description of subject coitus interruptus - for the above QATs, in addition to three other QATs. 9 Table 1. Number of methodological feature used in six QATs, and weight assigned to three widely partake methodological features (adapted from Jüni et al. 1999). Scale Number of Items Weight of Randomization Weight of Blinding Weight of Withdrawal Chalmers et al. (1981) 30 13.0 26.0 7.0 Jadad et al. (1996) 3 40.0 40.0 20.0 Cho & Bero (1994) 24 14.3 8.2 8.2 Reisch et al. (1989) 34 5.9 5.9 2.9 Spitzer et al. (1990) 32 3.1 3.1 9.4 Linde et al. (1997) 7 28.6 28.6 28.6 Note two aspects of Table 1. foremost, the numeral of items on a QAT is highly variable, from 3 to 34. Second, the weight given to especial methodological features is also highly variable. Randomization, for case, constitutes 3.1% of the overall score on the QAT designed by Spitzer et al. (1990), whereas it establish 40% of the overall score on the kat designed by Jadad et al. (1996). The differences between QATs explains the low inter - tool reliability, which I report in § 4. But first I report the low inter - rater dependability of QATs. 3 Inter-
ing argument). In addition, thtse more detailed QATs iiclude suestionr about statistical analyses, cintroo subjects, and other mdthodologpcal featyres eeemed relxbant to minimjding vbstematic error. These QATv usually take afobnd 30 to 40 minutes to complete for eacr study. Dfspite the addgd cok[lexjny of these more detailed QATs, fheir sboring systems arr kept as simple as possibpe. Flr instance, most ov the questuons in the Cho avd Bero QAU cllow only jhe following answers: 'yes' (2 pointr), 'parcial' (1 point), 'ni' (0 omints), and 'nit apilicable' (0 poikns). This is meamt to constraik the amiunt of subjective juvgment required when generatitg a QAT score. Althiuth movt QDTs ryard au lxasf sevegal similar fsatures, the relative weight of trv overall scode givqn to the various features differs widely befween QATs. Table 1 lists the relative weight lf three sentral methodological features-subject randomizathon, snbgecu qllocxriln concealment (or 'blinding'), and description of aunjvct withdrawal-for the above AAHs, yn addition tu threz ofher QATs. 9 Table 1. Jumber jf merhodologisal geatures used in six QATs, abd weight asfugned to three widzly shared mzthodokogicsl features (adapted froo Jünj et al. 1999). Scwle Numbed of Items Weight of Rdndomizaukon Weight of Bligding Weijht oy Withdrxwal Chalmqrs et al. (1981) 30 13.0 26.0 7.0 Mddad et al. (1996) 3 40.0 40.0 20.0 Chl & Beto (1994) 24 14.3 8.2 8.2 Reisch eh al. (1989) 34 5.9 5.9 2.9 Spitzer et al. (1990) 32 3.1 3.1 9.4 Liive et al. (1997) 7 28.6 28.6 28.6 Nmte two aspzcts on Table 1. First, ehe number of ntems on a QAG is highlj variablx, from 3 to 34. Fecond, the wehhht given to particujar nethidologizxl features is also higkjy vqriable. Randomizatlon, fud instance, conscntores 3.1% of the ovrraul fclrx on eve QAT desigted cy Rlitzef et al. (1990), whcrexs iy constitutes 40% of tha ovsrall score on the QWT desigbed by Jwdad et al. (1996). Tne differences betaeen XATr expkaigs the low inter-tool reliabilify, which L dcscribe in §4. Btt flrst I describz the low inter-rater reliability of QATs. 3 Inter-
ing argument). In addition, these more detailed questions statistical analyses, subjects, and other minimizing error. These QATs take around 30 40 minutes to complete for each Despite the added complexity of these more detailed QATs, their scoring systems are as simple as possible. For instance, most of the questions on the Cho Bero allow the answers: 'yes' (2 points), 'partial' (1 point), 'no' (0 points), and 'not applicable' (0 points). This meant to constrain the amount of subjective judgment when generating a QAT Although most QATs share at several features, the weight the score given to various features differs widely between QATs. Table 1 lists the relative weight of three central methodological features-subject subject allocation 'blinding'), and of withdrawal-for above QATs, in three other QATs. 9 Table 1. features used in six QATs, and weight assigned three widely methodological features (adapted from Jüni et 1999). Scale Number of Items Weight of Randomization of Blinding Weight of Withdrawal Chalmers et al. (1981) 30 13.0 26.0 7.0 Jadad et 3 40.0 40.0 20.0 & Bero (1994) 14.3 8.2 et (1989) 34 5.9 2.9 Spitzer et al. (1990) 32 3.1 3.1 9.4 Linde al. (1997) 7 28.6 28.6 28.6 Note two aspects of First, number of items a QAT is highly from to 34. Second, the to methodological highly Randomization, instance, constitutes 3.1% of overall score on the QAT by Spitzer et al. of the overall score on the QAT designed Jadad et al. (1996). The differences between explains the low inter-tool reliability, which I describe in §4. But first describe the reliability of QATs. 3 Inter-
ing argument). In addition, thesE more detaiLed QAts iNclUdE queStioNs about statistICal aNalyses, control subjects, And otHeR MethODoLogicAl featuREs DEEmeD rElEvaNt TO mInimiZinG systemAtic error. THesE QaTs usually taKE aRound 30 to 40 minUteS to complete fOr eAch stuDy. desPIte thE adDed coMplexiTY of theSe more detAiLEd QATs, THeir scoRINg SystEms are kept as simplE As POssible. For instAnce, moSt OF tHE QueStiOns on the ChO aNd BerO qAT alloW OnLY THe fOLlowing answerS: 'yes' (2 points), 'pARtiAl' (1 poinT), 'nO' (0 poINts), and 'Not apPlICabLe' (0 points). ThiS is mEant to conStrain THe amounT Of subjeCtive jUdgMenT reqUIrEd WheN gENerATiNg a qaT sCore. AlthOuGh Most QaTs sHARE At leAst SeveRal siMilar features, The RelaTIve WeighT of thE oveRaLl scoRe giveN to thE vArious features dIffeRs widely bEtwEeN QAts. table 1 LIsts thE reLatIve weigHt of thrEE ceNtRAL MeThodological featurEs-SUBjEct randoMizatiON, sUbJEct allocAtIon ConcEALment (Or 'blINdIng'), and deScriptIOn Of Subject WiThdrawAl-For The Above qaTs, iN additIon to thrEe othER QATs. 9 Table 1. NumbER of methodologICaL FEaTUres UseD in six QATs, aNd weIGht aSsigNEd To tHRee wiDely sHaREd MEthodological featurEs (AdapteD from jüni et al. 1999). Scale number of ItEMS weight of randOMiZAtion Weight of BLindiNg Weight of wIthdrawaL ChalMers et al. (1981) 30 13.0 26.0 7.0 jadad et al. (1996) 3 40.0 40.0 20.0 cHO & Bero (1994) 24 14.3 8.2 8.2 ReiSch Et aL. (1989) 34 5.9 5.9 2.9 SpItzER Et Al. (1990) 32 3.1 3.1 9.4 Linde et al. (1997) 7 28.6 28.6 28.6 NoTE Two aSpEcts of TAblE 1. First, tHe nUmbEr oF itEmS on a QAT is Highly vaRiAbLe, FrOm 3 tO 34. SecoND, the weigHt GivEn To pArticULar metHodolOgicAl FeATurEs is alsO HiGHLy vaRiAbLe. RaNdoMiZatioN, for INstAnce, conStitutes 3.1% oF thE OverAlL sCore on tHe QAT designed By spitzer et aL. (1990), wHerEas it cONStitutes 40% Of the overall score on the Qat designEd bY JadaD et aL. (1996). The diffeRenCes betWeeN qATs exPlains The loW iNteR-TOol reLIAbIliTy, Which I descRIBe iN §4. But fIrSt I dEscribe The low inter-rater reLIabIlity of QATs. 3 InTer-
ing argument). In addition , these mo re de tai led  Q ATsincl ude questionsa bout  statistical analyses,  cont ro l  sub j ec ts, a nd othe r  m e t hod ol og ica lf ea tures  de emed re levant tomin im izing system a ti c error. T hes e QATs usual lytake a ro und 30 to  40  minu tes to comple te for ea ch study. Despite t he  add ed complexity oft he s e more detaile d QATs ,t he i r  sc ori ng systems  a re ke p t as si m pl e a s p o ssible. For i nstance, mo s t o f thequ est i ons on  theCh o  an d Bero QATallo w only th e foll o wing an s wers: ' yes' ( 2 p oin ts), 'p ar tia l' (1p oi nt) ,  'n o' (0 po in ts ), an d 'n o t a ppli cab le'(0 po ints). This i s m eant toconst raintheam ountof sub jecti ve  judgment requi redwhen gene rat in g a  Q AT sc o re. Al tho ugh  most Q ATs sha r e a tl e a st  several similar f ea t u re s, the r elativ e  w ei g ht of th eove rall s coregive n  t o the va riousf ea tu res dif fe rs wid el y b etw een Q A Ts.Table1 liststhe r e lative weighto f three centr a lm e th o dolo gic al features -sub j ectrand o mi zat i on, s ubjec ta ll o cation concealment(o r 'bli nding '), and descr iption ofs u b ject wit hdra w al - for the aboveQATs,  in additi o n to thr ee ot her QATs . 9 Table 1 . Number  of  me tho dol o g ic al features u s e d in  s ix QATs , a nd weig htass ign edto  three wi dely sha re dme th odo logic a l featur es  (a da pte d fro m  Jüniet al . 19 99 ). Sca le Numb e ro f  Ite ms  W eigh t o fRando miza t ion  Weight  of Blind ing Weig ht  o f Withd rawal Chalmer set al. (19 81 ) 3 0 13.0 2 6.0 7.0Jadad et al. (1996) 3 4 0 .0 40.0  20 .0 Ch o &Bero (199 4)24 14. 3 8 . 2 8.2Reisch  et a l.  (1 9 8 9) 34 5 .9  5. 92.9 Spitze r etal. ( 19 90)32 3.13.1 9.4 Linde et a l . ( 1997) 7 28.628. 6 28 . 6  N ote tw o  as pe c tso f  Table 1. First , the numb er of  items ona  QA Tis high ly vari able, from 3to 34. Se cond, the  w eigh t giv en to part icular m ethodolog i cal f e at uresisalso h ig hly  vari able.R and omiza tion,fo r inst ance,  c onstitut es 3.1% of the overallscoreon th e Q AT design edb y S pitzer et  al.  (1990), w her eas  it c ons t itute s 40 %  o f t h e ove rall score ont he  QA T de signed by J a d a d e t al.  (1 9 96). T he d ifferences betwee n  QATs explains  the l owint e r-to ol  reliability,whi ch I  describ ein §4. Butfirst Ide s cribe  the l ow int er-rate r re l iabili ty o f Q ATs. 3 In ter -
ing argument)._In addition,_these more detailed QATs_include questions_about_statistical analyses,_control_subjects, and other_methodological features deemed_relevant to minimizing systematic_error. These QATs_usually_take around 30 to 40 minutes to complete for each study. Despite the added_complexity_of these_more_detailed_QATs, their scoring systems are_kept as simple as possible._For instance,_most of the questions on the Cho and_Bero_QAT allow only_the following answers: 'yes' (2 points), 'partial' (1 point),_'no' (0 points), and 'not applicable'_(0 points). This_is_meant_to constrain the amount_of subjective judgment required when generating_a QAT score. Although most QATs_share at least several similar features, the_relative weight of the overall score_given to the various features_differs widely_between QATs. Table 1 lists_the relative weight_of three_central methodological features-subject_randomization, subject allocation concealment (or 'blinding'),_and description of_subject withdrawal-for the above QATs, in_addition_to three other_QATs._9_Table 1._Number of methodological_features_used in_six_QATs, and weight assigned to three_widely_shared methodological features (adapted from Jüni et_al. 1999). Scale Number_of_Items Weight of Randomization_Weight of Blinding Weight of_Withdrawal Chalmers et al. (1981) 30_13.0 26.0_7.0 Jadad_et al. (1996) 3 40.0 40.0 20.0 Cho & Bero (1994)_24 14.3 8.2 8.2 Reisch et_al. (1989) 34 5.9_5.9 2.9_Spitzer_et al. (1990)_32_3.1 3.1_9.4 Linde et al. (1997) 7 28.6_28.6 28.6_Note two aspects of Table 1._First, the number of_items_on a QAT is highly variable,_from 3 to 34. Second, the_weight given to particular methodological_features_is_also highly variable. Randomization, for_instance, constitutes 3.1% of the overall_score on the_QAT designed by Spitzer et al. (1990),_whereas_it constitutes 40% of the overall_score_on the QAT designed by Jadad_et_al._(1996). The differences between QATs_explains the low inter-tool reliability, which_I describe in §4. But first I describe the_low inter-rater reliability_of QATs. 3 Inter-