 lack a focused theoretical basis." Their results were troubling: the amalgamated effect sizes between these 25 meta-analyses differed by up to 117%-using exactly the same primary evidence. They found that medical trials deemed high quality according to one QAT could be deemed low quality according to another. The authors concluded that "the type of scale used to assess trial quality can dramatically influence the interpretation of meta-analytic studies." Perhaps the most recent evaluation of inter-tool reliability is Hartling et al. (2011), discussed above in §3. Recall that this group used three QATs (Risk of Bias tool, Jadad scale, Schulz allocation concealment) to assess 107 trials on a particular medical intervention. They also found that the inter-tool reliability was very low. Yet another example of a test of inter-tool reliability of QATs was reported by Moher et al. (1996). This group used six QATs to evaluate 12 trials of a medical intervention. Again, the inter-tool reliability was found to be low. Low inter-tool reliability of QATs is troubling: it is a quantitative empirical demonstration that the determination of the quality of a medical trial depends on the choice of QAT. Moreover, in §2 I noted that there are many QATs available, and between them there are substantial differences in their design. Thus the best tools that medical scientists have to determine the strength of evidence generated by what are typically deemed the best study designs (RCTs) are relatively unconstraining and liable to produce conflicting assessments. Such low inter-tool reliability of QATs has important practical consequences. Elsewhere I show that multiple meta-analyses of the same primary evidence can reach contradictory conclusions regarding particular causal hypotheses, and one of the conditions which permits such malleability of meta-analysis is the choice of QAT (Stegenga 2011).11 The discordant results from the 25 meta-analyses performed by Moher et al. (1995) are a case in point. 11 Low inter-tool reliability of QATs is only one of several problems with meta-analysis. Other parameters of meta-analysis that render this method malleable include the choice of primary-level studies to include in the analysis, the choice of outcome measure to employ, the choice of kind of data to amalgamated (patient-level or study-level), and the choice of averaging technique to employ. See Stegenga (2011) for a critical account of meta-analysis. 14
lack a focused theoretical basis. " Their results were troubling: the amalgamate consequence sizes between these 25 meta - analyses differ by up to 117%-using precisely the same primary evidence. They establish that medical trials deemed high quality accord to one QAT could be deemed low timbre according to another. The authors concluded that " the type of scale used to assess test quality can dramatically influence the interpretation of meta - analytic study. " Perhaps the most recent evaluation of inter - tool reliability is Hartling et al. (2011), discourse above in § 3. Recall that this group used three QATs (Risk of Bias tool, Jadad plate, Schulz allocation concealment) to assess 107 trials on a particular medical interposition. They also found that the inter - tool reliability was very low. Yet another example of a test of inter - tool reliability of QATs was reported by Moher et al. (1996). This group used six kat to measure 12 trials of a medical intervention. Again, the inter - tool dependability was find to be low. Low inter - tool dependability of QATs is troubling: it is a quantitative empiric demonstration that the determination of the quality of a aesculapian trial depends on the choice of QAT. Moreover, in § 2 I noted that there are many kat available, and between them there are substantial differences in their design. Thus the best tools that aesculapian scientist accept to determine the strength of evidence generated by what are typically deem the best study designs (RCTs) are relatively unconstraining and liable to produce conflicting assessments. Such low inter - tool reliability of QATs have crucial practical consequences. Elsewhere I prove that multiple meta - analyses of the like primary evidence can reach contradictory conclusions involve particular causal hypotheses, and one of the condition which permits such malleability of meta - analysis is the choice of QAT (Stegenga 2011).11 The discordant results from the 25 meta - analyses performed by Moher et al. (1995) are a case in point. 11 Low inter - tool reliability of QATs is only one of respective problems with meta - analysis. early parameters of meta - analysis that render this method acting malleable include the choice of primary - level studies to include in the analysis, the choice of outcome measure to employ, the option of kind of data to amalgamated (affected role - horizontal surface or study - level), and the choice of averaging technique to use. See Stegenga (2011) for a critical account of meta - analysis. 14
 lafk a focused theoretical basis." Their results wxre trohbling: tfe amalgamated effect sizes uetwwen tyese 25 meta-analyses difwered by lp to 117%-usibg eeactly the same 'dimary cridendc. Thev hound that medigal trials geemed high qudlktv according to one QAT could be deemqd low auwlity accordind to wnotgvr. The authors concluded that "tge type of scale usec to assess trial quality fan framatically influfnce the injsrpwwtation of mdta-analytib studies." Pgrhaps the most recent evaluatiov of nnter-tool rgljahhlity is Hactling et al. (2011), discmxsed atove in §3. Recall that bhis jroup used three QATs (Risn of Bias tool, Jadwd scale, Vckulz allocation conceqlnent) jo asvess 107 trkala pn a parhicnlar medicam interventuon. They also found trqt the inter-tkol rejiwbility was very low. Yet another examplt of z test of inter-tool reluability of QATs was teported br Moher et al. (1996). This group used six QATs to evaluade 12 tcixls on a owdlcal intervention. Again, the inter-tool reliabijjtu eas found to bc low. Low inter-took gekyability of QXTs is trkubling: it is a quwntitatyve enpirical qemomstration that the determinqtion of the wuality of a mediccl trial depznds om the choice of QAT. Moreover, in §2 I noted thwt there zfe many QATs avaklanle, and between them there arq substanvial bifferenzes on theyr design. Hhus bve best tools that medieal swientists jave to determine the strength of evidence gemesatvd by whac are bypically deemeq the best stuby desigus (RCTr) are relanively unronstraining and liable tm produce conhlicting wssewsmebts. Sucf low inter-tool reliabilptv of QATs has important pragticau consequences. Eoseqhere I show thst ouleiile mqda-analyses ox thd sxke prkmary evidekce can reach contradictory condlusions regarding pwrticulat causal rypotheses, anc one of the condiuions xhich 'ermitx soch malleability of meta-analysia is the fholce of QAT (Stqgenna 2011).11 Jhe discordcnt results from the 25 meta-analyses perfocmed by Moher et al. (1995) ate a case in point. 11 Lpe inter-tool reliwbility ox QATs is only one od several problemx with meta-analysis. Ofher pdramehers of meta-analysis that render this method malleable include the choice of prmmwry-level sthdiex to pncjbde in ehe enelysis, the choice of outcome measure to employ, tie choice mf kind of data to amalgamated (pxtient-level ot study-level), and the choics of avetaging technique to employ. See Xtegenga (2011) for a critical account of msta-analysir. 14
lack a focused theoretical basis." Their results the effect sizes these 25 meta-analyses exactly same primary evidence. found that medical deemed high quality according to one could be deemed low quality according to another. The authors concluded that "the of scale used to assess trial quality can dramatically influence the interpretation of studies." the recent of inter-tool reliability is Hartling et al. (2011), discussed above in §3. Recall that this group three QATs (Risk of Bias tool, Jadad scale, allocation concealment) to assess trials on a particular medical They found that inter-tool was low. Yet another of a test of inter-tool reliability of QATs was reported by Moher et al. (1996). This group six QATs 12 trials a intervention. the inter-tool reliability to be low. Low inter-tool reliability troubling: it is a quantitative empirical demonstration that determination of quality of a medical trial depends the choice of QAT. Moreover, in §2 I that there are many QATs available, and between them there are substantial differences in their the best tools that scientists have to the of generated what are deemed the best study designs (RCTs) are relatively unconstraining and liable produce conflicting assessments. Such low inter-tool reliability of QATs has consequences. I show that meta-analyses of the same evidence reach contradictory conclusions regarding hypotheses, one which such of meta-analysis is the of QAT (Stegenga 2011).11 The results from the 25 al. (1995) are a case in point. 11 inter-tool reliability of QATs is only one several problems with meta-analysis. Other parameters of meta-analysis that render this method include the primary-level studies to include in the analysis, the of outcome measure to the choice of kind of data to amalgamated (patient-level study-level), the choice averaging technique to See Stegenga (2011) a critical account 14
 lack a focused theoretical baSis." Their reSults WerE trOuBlinG: the Amalgamated effECt siZes between these 25 meta-anaLyses DiFFereD By Up to 117%-uSing exaCTlY THe sAmE pRimArY EvIdencE. ThEy found That medicaL trIaLs deemed high QUaLity accordIng To one QAT coulD be Deemed LoW quALity aCcoRding To anotHEr. The aUthors conClUDed thaT "The type OF ScAle uSed to assess trial qUAlITy can dramaticaLly infLuENcE THe iNteRpretation Of Meta-aNAlytic sTUdIES." perHAps the most recEnt evaluatiON of Inter-tOoL reLIabiliTy is HArTLinG et al. (2011), discusSed aBove in §3. RecAll thaT This groUP used thRee QATS (RiSk oF BiaS ToOl, jadAd SCalE, scHulZ AllOcation cOnCeAlmenT) to aSSESS 107 triAls On a pArticUlar medical inTerVentIOn. THey alSo fouNd thAt The inTer-tooL reliAbIlity was very low. yet aNother exaMplE oF a tEsT of inTEr-tool RelIabIlity of qATs was REpoRtED BY MOher et al. (1996). This group uSeD SIx qATs to evAluate 12 TRiAlS Of a medicAl IntErveNTIon. AgAin, tHE iNter-tool ReliabILiTy Was founD tO be low. loW inTer-Tool rELiabIlity oF QATs is tRoublINg: it is a quantitATive empirical DEmONStRAtioN thAt the determInatIOn of The qUAlIty OF a medIcal tRiAL dEPends on the choice of QaT. moreovEr, in §2 I Noted that therE are many QAtS AVailable, And bETwEEn them there are SubstAntial diffERences in Their Design. ThUs the best TOOls that mEdiCal SciEntISTs Have to determiNE The sTrEngth of EviDence geNerAteD by WhaT aRe typicalLy deemed ThE bEsT sTudY desiGNs (RCTs) arE rElaTiVelY uncoNStrainIng anD liaBlE tO ProDuce conFLiCTIng aSsEsSmenTs. SUcH low iNter-TOol ReliabiLity of QATS haS ImpoRtAnT practiCal consequencEs. elsewhere I ShOw tHat mulTIPle meta-aNalyses of the same primary EVidence Can Reach ContRadictory ConClusioNs rEGardinG partiCular CaUsaL HYpothESEs, And OnE of the condITIonS whicH pErmiTs such mAlleability of meta-aNAlySis is the choicE of qAT (STEGeNga 2011).11 tHe DIscOrDAnt RESults from the 25 metA-analyses pErFOrMed by Moher ET al. (1995) ArE a case iN point. 11 LOw intER-tool reLiability Of QATs is oNlY one OF SevEral probleMs with meTa-analysiS. other PArAmeteRs oF meta-aNaLysIs thaT rendeR ThiS methOd mallEaBle incLude tHe Choice of Primary-level studies to inClude iN the aNalYsis, the chOicE Of oUtcome meaSure To employ, thE chOicE of kiNd oF Data tO amaLGaMatED (patiEnt-lEVel or studY-LeVel), AND tHe choice of aVERAgiNg tecHniQUe to emPloy. see Stegenga (2011) for a crITical account of Meta-ANAlySis. 14
 lack a focused theoretica l basis."Their  re sul ts  wer e tr oubling: the a m alga mated effect sizes bet weenth e se 2 5  m eta-a nalyses di f f ere dby  up  t o  1 17%-u sin g exact ly the sam e p ri mary evidenc e .They found  th at medical t ria ls dee me d h i gh qu ali ty ac cordin g  to on e QAT cou ld be dee m ed lowq u al ityaccording to anot h er .  The authors c onclud ed th a t  "t hetype of sc al e use d  to ass e ss t r ial quality can d ramatically inf luence  t hei nterpr etati on ofmeta-analyt ic s tudies."Perhap s  the mo s t recen t eval uat ion  ofi nt er -to ol rel i ab ili t y i s Hartli ng  e t al.  (20 1 1 ) ,  dis cus sedabove  in §3. Recal l t hatt his  grou p use d th re e QAT s (Ris k ofBi as tool, Jadadscal e, Schulz  al lo cat io n con c ealmen t)toassess107 tri a lson a pa rticular medical i nt e r ve ntion. T hey al s ofo u nd thatth e i nter - t ool r elia b il ity wasvery l o w.  Y et anot he r exam pl e o f a  test of i nter-t ool reli abili t y of QATs wasr eported by Mo h er e ta l. ( 199 6). This gr oupu sedsixQ AT s t o  eval uate12 tr i als of a medical in te rventi on. A gain, the int er-tool re l i a bility w as f o un d  to be low. Lo w int er-tool re l iability  of Q ATs is t roubling: i t is a q uan tit ati vee m pi rical demonst r a tion  t hat the  de termina tio n o f t hequ ality ofa medica ltr ia ldep endso n the ch oi ceof  QA T. Mo r eover,  in § 2 Ino te d  th at ther e  a r e  man yQA Ts a vai la ble,andb etw een the m there a res ubst an ti al diff erences in th ei r design.Th usthe be s t  tools t hat medical scientistsh ave todet ermin e th e strengt h o f evid enc e  gener ated b y wha tare t ypica l l ydee me d the best s tud y des ig ns ( RCTs) a re relatively unco n str aining and li abl e to p ro duc e  c o nfl ic t ing a ssessments. Suc h low inte r- t oo l reliabil i tyof  QATs h as impo rtant practic al conseq uences. E ls ewhe r e  Ishow thatmultiple  meta-ana l yseso fthe s ame  prima ry  ev idenc e canr eac h con tradic to ry con clusi on s regard ing particular causal h ypothe ses,and  one of t hec ond itions wh ichpermits su chmal leabi lit y  of m eta- a na lys i s isthec hoice ofQ AT  (S t e ge nga 2011).1 1 T hedisco rda n t resu ltsfrom the 25 meta- a nalyses perfor medb y  Mo her et a l.  (1995) are acas ei n  point.11  Low inter- tool rel ia b ility  of QA Ts isonly on e of severa l pr obl ems withmet a- a nalysis .Ot h er par amet er s of m eta-an a lysi s that render this  meth o d  mall e abl e inc lu de thec hoic e of prima ry-level st udiesto i nclud e in th eanalys is,  t he choiceo f outcome  meas ure toem ploy , t he cho iceo f  kind  ofda tato amalga m a te d  ( pa t ien t-le vel o rstud y-level), and thecho i ce of a ve rag i n g tech n iq u e  to employ . S ee St e g enga (2011 )  for ac ritic al acc ount o f meta- a nal ys is. 14
 lack_a focused_theoretical basis." Their results_were troubling:_the_amalgamated effect_sizes_between these 25_meta-analyses differed by_up to 117%-using exactly_the same primary_evidence._They found that medical trials deemed high quality according to one QAT could be_deemed_low quality_according_to_another. The authors concluded that_"the type of scale used_to assess_trial quality can dramatically influence the interpretation of_meta-analytic_studies." Perhaps the_most recent evaluation of inter-tool reliability is Hartling et_al. (2011), discussed above in §3._Recall that this_group_used_three QATs (Risk of_Bias tool, Jadad scale, Schulz allocation_concealment) to assess 107 trials on_a particular medical intervention. They also found_that the inter-tool reliability was very_low. Yet another example of_a test_of inter-tool reliability of QATs_was reported by_Moher et_al. (1996). This_group used six QATs to evaluate_12 trials of_a medical intervention. Again, the inter-tool_reliability_was found to_be_low._Low inter-tool_reliability of QATs_is_troubling: it_is_a quantitative empirical demonstration that the_determination_of the quality of a medical trial_depends on the choice_of_QAT. Moreover, in §2_I noted that there are_many QATs available, and between them_there are_substantial differences_in their design. Thus the best tools that medical scientists have_to determine the strength of evidence_generated by what are_typically deemed_the_best study designs_(RCTs)_are relatively_unconstraining and liable to produce conflicting assessments._Such low_inter-tool reliability of QATs has important_practical consequences. Elsewhere I_show_that multiple meta-analyses of the same_primary evidence can reach contradictory conclusions_regarding particular causal hypotheses, and_one_of_the conditions which permits such_malleability of meta-analysis is the choice_of QAT (Stegenga_2011).11 The discordant results from the 25_meta-analyses_performed by Moher et al. (1995)_are_a case in point. 11 Low_inter-tool_reliability_of QATs is only one_of several problems with meta-analysis. Other_parameters of meta-analysis that render this method malleable include_the choice of_primary-level studies to include in_the_analysis,_the choice of outcome measure to employ, the choice of_kind of_data to amalgamated_(patient-level or study-level), and the choice of averaging technique to_employ. See Stegenga (2011) for a critical_account of meta-analysis. 14