 (or perhaps causes less bias) in these scenarios. For example, the placebo effect might be stronger in studies that are designed to assess the benefits of pharmaceuticals compared with studies that are designed to assess the harms of pharmaceuticals. 15 Such a difference could be independently and empirically tested. If this were true, then the different scenarios would indeed warrant different QATs, suitable for the particularities of the scenario at hand. If the low inter-tool reliability of QATs were merely the result of employing multiple QATs to different kinds of empirical scenarios (different kinds of studies, say, or studies of different kinds of hypotheses, such as benefits versus harms of pharmaceuticals), then such low inter-tool reliability would hardly be troubling. Indiscriminate use of QATs might lead to low inter-tool reliability, such thinking would go, but discriminate use would not. Similarly, low inter-tool reliability of QATs would be less troubling if one could show that in principle there is only one good QAT for a given domain, or at least a small set of good ones which are similar to each other in important respects, because then one could dismiss the observed low inter-tool reliability as an artefact caused by the inclusion of poor QATs in addition to the good ones. Unfortunately, on the whole, these considerations do not mitigate the problem of low inter-tool reliability of QATs. There are, in fact, a plurality of equally fine QATs, designed for the same kinds of scenarios (typically: assessing RCTs of the efficacy of pharmaceuticals). A systematic review by medical scientists concluded that there were numerous QATs that "represent acceptable approaches that could be used today without major modifications" (West et al. 2002). Moreover, all of the empirical demonstrations of their low inter-tool reliability involve the assessment of the quality of studies from a very narrow domain: for instance, the low inter-tool reliability of QATs shown in Jüni et al. (1999) involved assessing studies of a single design (RCTs) about a single causal hypothesis, and these QATs had been developed with the purpose of assessing the quality of that very study design. Although there are some QATs which are arguably inferior to others, at least among the reasonably good ones I argue below that we lack a theoretical basis for distinguishing among them, and so we are stuck with a panoply of acceptable QATs which disagree widely about the quality of particular medical studies and thus
(or perhaps causes less bias) in these scenarios. For case, the placebo consequence might be stronger in studies that are designed to tax the benefits of pharmaceuticals compare with study that are designed to assess the harms of pharmaceuticals. 15 Such a remainder could be independently and empirically tested. If this were true, then the unlike scenarios would indeed warrant unlike QATs, desirable for the particularity of the scenario at hand. If the low inter - tool reliability of QATs were merely the resultant role of employing multiple QATs to different kind of empirical scenario (different kinds of studies, say, or study of different kinds of hypotheses, such as benefits versus harms of pharmaceutical), then such abject inter - tool reliability would hardly be troubling. Indiscriminate use of QATs might lead to low inter - tool reliability, such thinking would go, but discriminate function would not. Similarly, low inter - tool reliability of QATs would be less distressing if one could picture that in rationale there is only one good QAT for a given domain, or at least a small set of effective ones which are similar to each other in important respects, because then one could dismiss the ascertained low inter - tool reliability as an artefact caused by the inclusion of poor QATs in accession to the good ones. Unfortunately, on the whole, these considerations do not mitigate the problem of gloomy inter - tool reliability of QATs. There are, in fact, a plurality of equally fine QATs, designed for the same kind of scenarios (typically: assessing RCTs of the efficacy of pharmaceuticals). A systematic review by aesculapian scientists concluded that there were numerous QATs that " act acceptable approach that could be used today without major modifications " (West et al. 2002). Moreover, all of the empirical demonstrations of their abject inter - tool reliability necessitate the assessment of the quality of studies from a very narrow domain: for case, the low inter - tool reliability of QATs show in Jüni et al. (1999) involved assessing studies of a single design (RCTs) about a single causal guess, and these QATs had been developed with the purpose of assessing the quality of that very study design. Although there are some kat which are arguably inferior to others, at least among the reasonably good ones I argue below that we lack a theoretical basis for distinguishing among them, and so we are stick with a panoply of acceptable QATs which disagree widely about the quality of finical medical studies and thus
 (or perhaps causes less biar) in these scenceios. Fmr exajple, the placebo effect might be strlnter ib studies that are deskgned to wssess tye btnefits of pharmarsuticals compzved wnti studies that sre designad to assess tve hcrms of pharmaceuticals. 15 Such a diffqrence vokld be indepenqentkr ans empirically tested. If this were frue, thtn the different svenarios would indeed warrwnt fifferent QATs, suihable for tye pweticularitier of the sbznario at hznd. If the low inter-tool reliabklity of QATs wgxw mftely the resnlt of employing mmktiple QATs tp different kikds oh empirical scenarios (difherent kinds of studyes, say, os atudies of differwnr kings ox hyoithdsea, xudh as henxfits versua harms of pharmaceuticals), them flvh low inter-fool rqlyability would hardly be troubling. Indivcrjminate use of QATs migyt lead to low inter-tlol reliafility, such thinking would go, but discriminate usa wouma nit. Rumllarly, low inter-tool reliability of QATs woulq bt lvss troubling if jne could snoa yrat in princikle thexs js only one good QWT for w givwn domain, or st least a small set of gooe ones which qre similar to eack other in ioporjant rrspects, because then onz couls dismiss tje observsa low inter-tool felpabinity as an artefact caused by the iiclusnon of puor AATs ig addition to tma good ones. Unfortknatepy, on the whlle, these considerations do not mitigate the krotlek of low intev-tool reliabiliey of QATs. Thete are, in fact, a pluraliny of equelly fine QAEs, designed fmt the same kiids of scqnaruos (rypicaluh: assessing RCYs of the efficacy if pharmaceuticals). A rgstematic reviec vy medical sciemtirts clnrludeq that there fere nuorrous QATs that "vepfesemt acceptable approawhes that could be usec boday wityout majjr modificatipns" (West et al. 2002). Mogeovec, all uf thr eipirical demonstrations of thejr low inher-bool reliabilyty lnvojve the asxessment of the quality of studies from a very narrow domqin: for instance, thg lpw inter-took relyability mf QATs shown in Jüni et al. (1999) involved sssessing studies of z singne dedign (RCTs) about a single causal hypothesis, and these QATs had been developed wivh the purposs of assevsnng the qtalivy of that very stldy design. Although there are sooe QATs fhnch are arguably inferior to ptfers, at least among the reasonably gokd ones O argue below that we lack a throretical basis for distlnguishint amohg them, anv so we are stuck wiyh a pcnoply of accwptanle QATs which dixanree widelj abpuc the quality of particular medicql stueifs and thus
(or perhaps causes less bias) in these example, placebo effect be stronger in assess benefits of pharmaceuticals with studies that designed to assess the harms of 15 Such a difference could be independently and empirically tested. If this were then the different scenarios would indeed warrant different QATs, suitable for the particularities the at If low inter-tool reliability of QATs were merely the result of employing multiple QATs to different kinds empirical scenarios (different kinds of studies, say, or of different kinds of such as benefits versus harms pharmaceuticals), such low reliability hardly troubling. Indiscriminate use QATs might lead to low inter-tool reliability, such thinking would go, but discriminate use would not. Similarly, inter-tool reliability would be troubling one show that in is only one good QAT for or at least a small set of good which are to each other in important respects, then one could dismiss the observed low inter-tool as an artefact caused by the inclusion of poor QATs in addition to the good on the whole, these do not mitigate problem low reliability QATs. There in fact, a plurality of equally fine QATs, designed for the kinds of scenarios (typically: assessing RCTs of the efficacy of systematic by medical scientists that there were numerous that acceptable approaches that could today major al. Moreover, of the empirical demonstrations their low inter-tool reliability involve assessment of the quality narrow domain: for instance, the low inter-tool reliability QATs shown in Jüni et al. (1999) assessing studies of a single design (RCTs) about a single causal hypothesis, these QATs developed with the purpose of assessing the quality that very study design. there are some QATs which are arguably inferior to at among the good ones I below that we a theoretical basis among them, so are panoply of acceptable QATs which disagree about quality of particular medical studies thus
 (or perhaps causes less bias) in These scenaRios. FOr eXamPlE, the PlacEbo effect might BE strOnger in studies that are dEsignEd TO assESs The beNefits oF PhARMacEuTiCalS cOMpAred wIth Studies That are desIgnEd To assess the hARmS of pharmacEutIcals. 15 Such a diFfeRence cOuLd bE IndepEndEntly And empIRicallY tested. If ThIS were tRUe, then tHE DiFferEnt scenarios would INdEEd warrant diffeRent QAts, SUiTABle For The particuLaRitieS Of the scENaRIO At hANd. If the low intEr-tool reliaBIliTy of QAts WerE Merely The reSuLT of Employing muLtipLe QATs to dIffereNT kinds oF EmpiricAl scenAriOs (dIffeREnT kIndS oF StuDIeS, saY, Or sTudies of DiFfErent KindS OF HYpotHesEs, suCh as bEnefits versus HarMs of PHarMaceuTicalS), theN sUch loW inter-Tool rElIability would haRdly Be troubliNg. INdIscRiMinatE Use of QaTs MigHt lead tO low intER-toOl RELIaBility, such thinking WoULD gO, but discRiminaTE uSe WOuld not. SImIlaRly, lOW Inter-Tool RElIability Of QATs WOuLd Be less tRoUbling If One CouLd shoW That In prinCiple theRe is oNLy one good QAT foR A given domain, oR At LEAsT A smaLl sEt of good oneS whiCH are SimiLAr To eACh othEr in iMpORtANt respects, because thEn One couLd disMiss the observEd low inter-TOOL reliabiLity AS aN Artefact caused By the Inclusion oF Poor QATs In addItion to tHe good oneS. uNfortunaTelY, on The WhoLE, ThEse consideratIONs do NoT mitigaTe tHe problEm oF loW inTer-ToOl reliabiLity of QAts. thErE aRe, iN fact, A PluralitY oF eqUaLly Fine Qats, desiGned fOr thE sAmE KinDs of sceNArIOS (typIcAlLy: asSesSiNg RCTS of tHE efFicacy oF pharmaceUtiCAls). A SyStEmatic rEview by medicaL sCientists cOnCluDed thaT THere were Numerous QATs that "represeNT acceptAblE apprOachEs that couLd bE used tOdaY WithouT major ModifIcAtiONS" (West ET Al. 2002). morEoVer, all of thE EMpiRical DeMonsTrationS of their low inter-toOL reLiability invoLve The aSSEsSmeNT oF The QuALitY OF studies from a veRy narrow doMaIN: fOr instance, THe lOw Inter-toOl reliaBilitY Of QATs sHown in JünI et al. (1999) invoLvEd asSESsiNg studies oF a single Design (RCTS) About A SiNgle cAusAl hypoThEsiS, and tHese QAtS haD been DeveloPeD with tHe purPoSe of asseSsing the quality of that veRy studY desiGn. ALthough thEre ARe sOme QATs whIch aRe arguably InfEriOr to oTheRS, at leAst aMOnG thE ReasoNablY Good ones I ARgUe bELOw That we lack a THEOreTical BasIS for diStinGuishing among them, ANd so we are stuck With A PAnoPly OF accEpTable QATs which DisAgREE widely aBoUt the qualitY of partiCuLAr medIcal stUdies aNd thus
 (or perhaps causes less b ias) in th ese s cen ari os . Fo r ex ample, the pla c eboeffect might be strong er in  s t udie s  t hat a re desi g ne d toas se ssth e  b enefi tsof phar maceutical s c om pared with s t ud ies that a redesigned toass ess th ehar m s ofpha rmace utical s . 15 S uch a dif fe r ence c o uld bei n de pend ently and empiric a ll y  tested. If th is wer et ru e ,  th enthe differ en t sce n arios w o ul d i nde e d warrant dif ferent QATs ,  su itable  f ort he par ticul ar i tie s of the sc enar io at han d. Ift he lowi nter-to ol rel iab ili ty o f  Q AT s w er e  me r el y t h e r esult of  e mp loyin g mu l t i p le Q ATs  todiffe rent kinds of  em piri c alscena rios(dif fe rentkindsof st ud ies, say, or st udie s of diff ere nt  ki nd s ofh ypothe ses , s uch asbenefit s  ve rs u s ha rms of pharmaceuti ca l s ),  then su ch low in te r -tool re li abi lity w ouldhard l ybe troub ling.I nd is crimina te  use o fQAT s m ightl eadto low  inter-t ool r e liability, suc h  thinking wou l dg o ,b ut d isc riminate us e wo u ld n ot.S im ila r ly, l ow in te r -t o ol reliability of Q AT s woul d beless troublin g if one c o u l d show t hati np rinciple there  is o nly one go o d QAT fo r a g iven dom ain, or a t least asma llset  of g oo d ones whicha r e si mi lar toeac h other  in  im por tan trespects,  because  t he non e c ouldd ismiss t he  ob se rve d low inter- toolreli ab il i tyas an a r te f a ct c au se d by  th einclu sion ofpoor QA Ts in add iti o n to  t he  good o nes. Unfortun at ely, on th ewho le, th e s e consid erations do not mitigat e  the pr obl em of  low  inter-to olreliab ili t y of Q ATs. T herear e,i n  fact , aplu ra lity of eq u a lly  fine  Q ATs,  design ed for the same ki n dsof scenarios(ty pica l l y:  as s es s ing  R C Tso f  the efficacy o f pharmace ut i ca ls). A sys t ema ti c revie w by me dical scienti sts concl uded that  t here w ere  numerousQATs tha t "repres e nt ac c ep table  ap proach es  th at co uld be use d tod ay wit ho ut maj or mo di fication s" (West et al. 2002).Moreov er, a llof the em pir i cal  demonstr atio ns of thei r l owinter -to o l rel iabi l it y i n volve  the assessmen t  o f t h e  q uality of s t u d ies  from  av ery na rrow  domain: for inst a nce, the low i nter - t ool  re l iabi li ty of QATs sho wnin J üni et a l.  (1999) inv olved as se s singstudie s of a  single d es i gn (RC Ts)abo ut a sing leca u sal hyp ot he s is, an d th es e QATs  had b e en d e v eloped with thepurpo s e  of a s ses singth e quali t y of  that very  study desi gn. Al thou gh th ere are  s ome QA Tswh ich are ar g uably inf erior  to oth er s, a t l east a mong t he re ason ab lygood ones I  a r gu eb elo w th at we  l acka theoret i cal basi s f o r disti ng uis h i ng amo n gt h em, and so  we  ares t uck with a pano p ly of ac ceptab le QAT s which dis ag ree wid ely a bout thequality o f  pa rt icul ar medic al  stu di esan d thus
 (or_perhaps causes_less bias) in these_scenarios. For_example,_the placebo_effect_might be stronger_in studies that_are designed to assess_the benefits of_pharmaceuticals_compared with studies that are designed to assess the harms of pharmaceuticals. 15 Such_a_difference could_be_independently_and empirically tested. If this_were true, then the different_scenarios would_indeed warrant different QATs, suitable for the particularities_of_the scenario at_hand. If the low inter-tool reliability of QATs were_merely the result of employing multiple_QATs to different_kinds_of_empirical scenarios (different kinds_of studies, say, or studies of_different kinds of hypotheses, such as_benefits versus harms of pharmaceuticals), then such_low inter-tool reliability would hardly be_troubling. Indiscriminate use of QATs_might lead_to low inter-tool reliability, such_thinking would go,_but discriminate_use would not._Similarly, low inter-tool reliability of QATs_would be less_troubling if one could show that_in_principle there is_only_one_good QAT_for a given_domain,_or at_least_a small set of good ones_which_are similar to each other in important_respects, because then one_could_dismiss the observed low_inter-tool reliability as an artefact_caused by the inclusion of poor_QATs in_addition to_the good ones. Unfortunately, on the whole, these considerations do not_mitigate the problem of low inter-tool_reliability of QATs. There_are, in_fact,_a plurality of_equally_fine QATs,_designed for the same kinds of scenarios_(typically: assessing_RCTs of the efficacy of pharmaceuticals)._A systematic review by_medical_scientists concluded that there were numerous_QATs that "represent acceptable approaches that_could be used today without_major_modifications"_(West et al. 2002). Moreover,_all of the empirical demonstrations of_their low inter-tool_reliability involve the assessment of the quality_of_studies from a very narrow domain:_for_instance, the low inter-tool reliability of_QATs_shown_in Jüni et al. (1999)_involved assessing studies of a single_design (RCTs) about a single causal hypothesis, and these_QATs had been_developed with the purpose of_assessing_the_quality of that very study design. Although there are some_QATs which_are arguably inferior_to others, at least among the reasonably good ones I_argue below that we lack a theoretical_basis for distinguishing among them, and so we are stuck with_a panoply of acceptable QATs which disagree widely_about the quality of particular medical studies_and thus