 determination of the epistemic significance of some piece of evidence by appealing to the poor inter-rater and intertool reliability of QATs is not merely an argument from disagreement. If it were, then the standard objection would simply note that the mere fact of disagreement about a particular subject does not imply that there is no correct or uniquely best view on the subject. Although different QATs disagree about the strength of evidence from a particular trial, this does not imply that there is no true or best view regarding the strength of evidence from this particular trial-goes the standard objection-since the best QATs might agree with each other about the evidence from this trial, and even more ambitiously, agreement or disagreement among QATs would be irrelevant if we just took into account the quality assessment of this particular trial by the uniquely best QAT. The burden that this objection faces is the identification of the single best QAT or at least the set of good ones (and then hope that multiple users of the best QAT will have high inter-rater reliability, or that the 18 set of good QATs will have high inter-tool reliability). As noted in §4, medical scientists involved in the development and assessment of QATs claim that there are simply a plurality of decent QATs that differ from one another in arbitrary respects. More fundamentally, we lack a theory of scientific inference that would allow us to referee between the most sophisticated QATs. Recall the different weightings of the particular methodological features assessed in QATs, noted in Table 1. Another way to state the burden of the'mere argument by disagreement' objection is that to identify the best QATs, one would have to possess a principled method of determining the optimal weights for the methodological features included on a QAT. That we do not presently have such a principled method is an understatement. Consider this compelling illustration of the arbitrariness involved in the assignment of weights to methodological features in QATs. Cho and Bero (1994) employed three different algorithms for weighting the methodological features of their QAT (discussed in §2). Then they tested the three weighting algorithms for their effect on quality scores of medical trials, and their effect on the inter-rater reliability of such scores. They selected for further use-with no principled basis- the weighting algorithm that had the highest inter-rater reliability. Cho and Bero explicitly admitted that nothing beyond the higher inter-rater reliability warranted the choice of this
determination of the epistemic significance of some piece of evidence by appealing to the hapless inter - rater and intertool dependability of QATs is not merely an argument from disagreement. If it were, then the standard expostulation would just note that the mere fact of discrepancy about a particular subject does not imply that there embody no correct or uniquely best scene on the subject. Although different kat disagree about the strength of evidence from a particular trial, this does not imply that there is no true or best position regarding the strength of evidence from this particular test - goes the standard objection - since the best QATs might agree with each early about the evidence from this trial, and even more ambitiously, agreement or discrepancy among QATs would be irrelevant if we just took into account the quality judgment of this particular trial by the uniquely best QAT. The burden that this expostulation faces is the identification of the single best QAT or at least the set of good ones (and then hope that multiple users of the best QAT will suffer high inter - rater reliability, or that the 18 hardening of good QATs will have high inter - tool dependability). As noted in § 4, aesculapian scientists involved in the development and appraisal of QATs claim that there are simply a plurality of decent QATs that differ from one another in arbitrary respects. More fundamentally, we lack a theory of scientific inference that would permit us to referee between the about advanced kat. Recall the different weightings of the particular methodological features assessed in QATs, noted in Table 1. Another way to submit the burden of the'mere argument by disagreement' objection is that to name the best QATs, one would have to possess a principled method of determine the optimal weight for the methodological features include on a QAT. That we do not presently have such a principled method acting is an understatement. Consider this compelling illustration of the arbitrariness necessitate in the assignment of weight to methodological features in QATs. Cho and Bero (1994) employed three different algorithms for weighting the methodological features of their QAT (discussed in § 2). Then they tested the three weighting algorithm for their effect on quality score of medical trials, and their effect on the inter - rater reliability of such scores. They selected for further use - with no principled basis- the weight algorithm that had the highest inter - rater reliability. Cho and Bero explicitly admitted that nothing beyond the higher inter - rater reliability warranted the choice of this
 dehermination of the episttmic significance of somx piece of eviddnce by appealing to the pooc inrer-rauvr and intertool relixbility ov QATs iw nou merely an argumxht from disagdcemenc. Mf it were, then the standdrd objection fojlb simply note that the mere fact of qisagrermfnt about a patticukwr shbject does not imply that there ia no cogrect or uniquely best view on the subject. Wlthlugh different QATd disagree qboue the strengtf of evidence from a pzrticular trial, this does not ioply chat there us no jrue or best view regarding tmv strencth of rvidence from bhis 'artucular trial-goes the vtandard objection-fince the bzst QATs might agree qirh eawh odher qbojt uhe ebidencf fcom this trjal, and eveb more ambitiously, sgwvrment or diszgreemqne among QATs would be irrelevant if we busf took into account the quality assessment ov this pawticular trial by the uniquely best QAT. The burdet thav ghiw jcheftion faces is the identification of the singjs next QAT or at lcast the set of gopd omgs (and then hoke that mumtiple users of thf best ZAT wull have righ inter-rater reliability, or rhat the 18 sen of good QATs will hare high intex-tool teliabolity). As noted in §4, medieal scjentists ingolved in ghe development xnd avsessmenu of QATs claim thwt there ere snmply a olurslity jf decent EATs bvat differ from onf anojher it arbitrarj respects. More fundamentally, wx lack a theoty mf vcientifnc infcrence that woujd allow us to referez betwden the moat sophmsticated QAEs. Recall the fifferent wemghtings jf tye pqrticulxf methodologicsl featurvs asswssed in QATs, notev in Fable 1. Another ccy to state the birddn jf tie'merq argument by disxgrdrment' objection ls ghat to identify the besd QAFs, one would have yo possess a princypled method pf determining the optikal weignts for the methodological featurss includfd jn a QAT. Thae we do not preszntly have such a principled method is ai understatement. Considwr this compelling nlkustration pf thq arbitrasiness involved in tye assignment of eeights to methodologjcal faaturfs in QATs. Cho and Bero (1994) employed three different algorithms for weightint the methodologiczl fraturvs jy their QAT (viscussed in §2). Them they tested the three weightiig algoritvmf for their effect on qualitu rcores of medkcal trials, and their efrect on yhe inter-rater reliability of sich scores. They selected for furrher hse-with no principled basis- yhe wenghtint algoruthm that had the hibhcst inter-rwter rzliability. Cho and Bero explicitlt admirtfd that nothing ueyond the hlgmer injer-rater reliability wardantsf the choice of thos
determination of the epistemic significance of some evidence appealing to poor inter-rater and not an argument from If it were, the standard objection would simply note the mere fact of disagreement about a particular subject does not imply that is no correct or uniquely best view on the subject. Although different QATs about strength evidence a particular trial, this does not imply that there is no true or best view regarding strength of evidence from this particular trial-goes the objection-since the best QATs agree with each other about evidence this trial, even ambitiously, or disagreement among would be irrelevant if we just took into account the quality assessment of this particular trial by uniquely best burden that objection is identification of the QAT or at least the set (and then hope that multiple users of the QAT will high inter-rater reliability, or that the set of good QATs will have high inter-tool As noted in §4, medical scientists involved in the development and assessment of QATs claim are simply a plurality decent QATs that from another arbitrary More fundamentally, lack a theory of scientific inference that would allow us to between the most sophisticated QATs. Recall the different weightings of methodological assessed in QATs, in Table 1. Another to the burden of the'mere disagreement' is the QATs, would have to possess principled method of determining the weights for the methodological That we do not presently have such a method is an understatement. Consider this compelling of the arbitrariness involved in the assignment of weights to methodological features QATs. Cho (1994) employed three different algorithms for weighting the features of their QAT in §2). Then they tested the three weighting algorithms their on quality of medical trials, their effect on inter-rater reliability of They selected further no weighting algorithm that had the highest reliability. and Bero explicitly admitted that beyond inter-rater reliability warranted the choice of this
 determination of the epistemIc significAnce oF soMe pIeCe of EvidEnce by appealinG To thE poor inter-rater and inteRtool ReLIabiLItY of QAts is not MErELY an ArGuMenT fROm DisagReeMent. If iT were, then tHe sTaNdard objectiON wOuld simply NotE that the mere FacT of disAgReeMEnt abOut A partIcular SUbject Does not imPlY That thERe is no cORReCt or Uniquely best view oN ThE Subject. AlthougH diffeReNT QatS diSagRee about thE sTrengTH of evidENcE FROm a PArticular triaL, this does noT ImpLy that ThEre IS no truE or beSt VIew Regarding thE strEngth of evIdence FRom this PArticulAr triaL-goEs tHe stANdArD obJeCTioN-SiNce THe bEst QATs mIgHt Agree With EACH OtheR abOut tHe eviDence from this TriAl, anD EveN more AmbitIousLy, AgreeMent or DisagReEment among QATs wOuld Be irrelevAnt If We jUsT took INto accOunT thE qualitY assessMEnt Of THIS pArticular trial by thE uNIQuEly best QaT. The bURdEn THat this oBjEctIon fACEs is tHe idENtIficatioN of the SInGlE best QAt oR at leaSt The Set Of gooD Ones (And theN hope thaT multIPle users of the bESt QAT will have HIgH INtER-ratEr rEliability, oR thaT The 18 sEt of GOoD QAtS will Have hIgH InTEr-tool reliability). As NoTed in §4, mEdicaL scientists inVolved in thE DEVelopmenT and ASsESsment of QATs clAim thAt there are SImply a plUraliTy of deceNt QATs thaT DIffer froM onE anOthEr iN ARbItrary respectS. mOre fUnDamentaLly, We lack a TheOry Of sCieNtIfic inferEnce that WoUlD aLlOw uS to reFEree betwEeN thE mOst SophiSTicateD QATs. recaLl ThE DifFerent wEIgHTIngs Of ThE parTicUlAr metHodoLOgiCal featUres assesSed IN QATS, nOtEd in TabLe 1. Another way tO sTate the burDeN of The'merE ARgument bY disagreement' objection iS That to iDenTify tHe beSt QATs, one WouLd have To pOSsess a PrinciPled mEtHod OF DeterMINiNg tHe Optimal weiGHTs fOr the MeThodOlogicaL features included oN A QAt. That we do not pResEntlY HAvE suCH a PRinCiPLed METhod is an understAtement. ConSiDEr This compelLIng IlLustratIon of thE arbiTRarinesS involved In the assiGnMent OF WeiGhts to methOdologicAl featureS In QATS. chO and BEro (1994) EmployEd ThrEe difFerent ALgoRithmS for weIgHting tHe metHoDologicaL features of their QAT (discUssed iN §2). Then TheY tested thE thREe wEighting aLgorIthms for thEir EffEct on QuaLIty scOres OF mEdiCAl triAls, aND their effECt On tHE InTer-rater relIABIliTy of sUch SCores. THey sElected for further USe-with no princiPled BASis- The WEighTiNg algorithm thaT haD tHE Highest iNtEr-rater reliAbility. CHo ANd BerO expliCitly aDmitted THAt NOthing BeyoNd tHe higher iNteR-rATer reliAbIlITy warrAnteD tHe choiCe of thIS
 determination of the epis temic sign ifica nce  of  s omepiec e of evidenceb y ap pealing to the poor in ter-r at e r an d  i ntert ool rel i ab i l ity  o fQAT si snot m ere ly an a rgument fr omdi sagreement.I fit were, t hen  the standar d o bjecti on  wo u ld si mpl y not e that the me re fact o fd isagre e ment ab o u ta pa rticular subjectd oe s  not imply tha t ther ei sn o  co rre ct or uniq ue ly be s t viewo nt h e  su b ject. Althoug h different QAT s disa gr eea bout t he st re n gth  of evidenc e fr om a part icular trial,t his doe s notimp lythat th er e i sn o t r ue  or bes t view r eg ar dingthes t r e ngth  of  evi dence  from this pa rti cula r  tr ial-g oes t he s ta ndard  objec tion- si nce the best QA Ts m ight agre e w it h e ac h oth e r abou t t heevidenc e fromt his  t r i a l,  and even more amb it i o us ly, agre emento rdi s agreemen tamo ng Q A T s wou ld b e  i rrelevan t if w e  j us t tookin to acc ou ntthe  qual i ty a ssessm ent of t his p a rticular trial by the unique l yb e st QAT.  Th e burden th at t h is o bjec t io n f a ces i s the  i d en t ification of the si ng le bes t QAT  or at leastthe set of g o od ones(and th e n hope that mu ltipl e users of the best  QATwill hav e high in t e r-raterrel iab ili ty, o rthat the 18 s e t  ofgo od QATs  wi ll have  hi ghint er- to ol reliab ility).As  n ot ed  in  §4,m edical s ci ent is tsinvol v ed inthe d evel op me n t a nd asse s sm e n t of  Q AT s cl aim  t hat t here are  simply  a plural ity of d ec en t QATsthat differ f ro m one anot he r i n arbi t r ary resp ects. More fundamentall y , we la cka the oryof scient ifi c infe ren c e that  would  allo wust o  refe r e ebet we en the mos t sop histi ca tedQATs. R ecall the differen t  we ightings of t hepart i c ul arm et h odo lo g ica l features assess ed in QATs ,n ot ed in Tabl e  1.  A notherway tostate the bur den of th e'mere ar gu ment b y d isagreemen t' objec tion is t h at to id entif y t he bes tQAT s, on e woul d  ha ve to  posse ss  a pri ncipl ed  methodof determining the opti mal we ights  fo r the met hod o log ical feat ures  includedona Q AT. T hat we do  not pr ese n tly h aves uch a pri n ci ple d me thod is anu n d ers tatem ent .  Consi derthis compelling i l lustration ofthea r bit rar i ness  i nvolved in the  as si g n ment ofwe ights to me thodolog ic a l fea turesin QAT s. Choa n dB ero (1 994)  em ployed th ree  d i fferent  a lg o rithms  for  w eighti ng the meth o d ological feature s oft h eir Q A T ( discu ss ed in § 2 ). T hen they t ested the t hree w eigh tingalgorit hm s forthe ir  effect on quality s cores  of med ic al t ria ls, an d th e i r eff ecton  th e inter-r a t er re li a bil ityof su ch  sco res. They selected  fo r  furthe ruse - w ith no pr i n cipled bas is-  thew e ighting al g orit h mt hat h ad the  highe st inte r -ra te r relia bil i t y. Cho an d Bero ex p lic it ly a dmittedth at n ot hin gbeyond the high er inter -rater r e l iabili tyw arra n ted th echoice of th is
 determination_of the_epistemic significance of some_piece of_evidence_by appealing_to_the poor inter-rater_and intertool reliability_of QATs is not_merely an argument_from_disagreement. If it were, then the standard objection would simply note that the mere_fact_of disagreement_about_a_particular subject does not imply_that there is no correct_or uniquely_best view on the subject. Although different QATs_disagree_about the strength_of evidence from a particular trial, this does not_imply that there is no true_or best view_regarding_the_strength of evidence from_this particular trial-goes the standard objection-since_the best QATs might agree with_each other about the evidence from this_trial, and even more ambitiously, agreement_or disagreement among QATs would_be irrelevant_if we just took into_account the quality_assessment of_this particular trial_by the uniquely best QAT. The_burden that this_objection faces is the identification of_the_single best QAT_or_at_least the_set of good_ones_(and then_hope_that multiple users of the best_QAT_will have high inter-rater reliability, or that_the 18 set of_good_QATs will have high_inter-tool reliability). As noted in_§4, medical scientists involved in the_development and_assessment of_QATs claim that there are simply a plurality of decent QATs_that differ from one another in_arbitrary respects. More fundamentally,_we lack_a_theory of scientific_inference_that would_allow us to referee between the most_sophisticated QATs._Recall the different weightings of the_particular methodological features assessed_in_QATs, noted in Table 1. Another_way to state the burden of_the'mere argument by disagreement' objection_is_that_to identify the best QATs,_one would have to possess a_principled method of_determining the optimal weights for the methodological_features_included on a QAT. That we_do_not presently have such a principled_method_is_an understatement. Consider this compelling_illustration of the arbitrariness involved in_the assignment of weights to methodological features in QATs._Cho and Bero_(1994) employed three different algorithms_for_weighting_the methodological features of their QAT (discussed in §2). Then_they tested_the three weighting_algorithms for their effect on quality scores of medical trials,_and their effect on the inter-rater reliability_of such scores. They selected for further use-with no principled basis-_the weighting algorithm that had the highest inter-rater_reliability. Cho and Bero explicitly admitted that_nothing beyond the higher_inter-rater_reliability warranted the choice of this