3) and inter-tool reliability (§4) of QATs: most QATs are not very good at constraining intersubjective assessments of hypotheses, and more worrying, the use of different QATs to assess the same primary evidence leads to widely divergent quality assessments of that evidence. This is an instance of a more general problem I call the underdetermination of evidential significance, which holds that in a rich enough empirical situation, the strength of the evidence is underdetermined (§5). Despite this problem, I defend the use of QATs in medical research. I end by comparing QATs to the widely employed evidence hierarchies, and argue that despite the problems with QATs, they are better than evidence hierarchies for assessing evidence in medicine (§6). 2 Quality Assessment Tools A quality assessment tool (QAT) for medical evidence can be either a scale with elements that receive a quantitative score representing the degree to which each element is satisfied by a medical trial, or else a QAT can be simply a checklist with elements that are marked as either present or absent in a medical trial. Given the emphasis on randomized controlled trials (RCTs) in medical research, most QATs are designed for the evaluation of RCTs, although there are several for observational studies and systematic reviews.4 Most QATs share several elements, including questions about how subjects were assigned to experimental groups in a trial, whether or not the subjects and experimenters were concealed to the subjects' treatment protocol, whether or not there was a sufficient description of subject withdrawal from the trial groups, whether or not particular statistical analyses were performed, and whether or not a report of a trial disclosed financial relationships between investigators and companies.5 Most QATs provide instructions on how to 4 The view that RCTs are the 'gold standard' of evidence has recently been subjected to much philosophical criticism. See, for example, Worrall (2002), Worrall (2007), Cartwright (2007), and Cartwright (2010); for an assessment of the arguments for and against the gold standard status of RCTs, see Howick (2011). Observational studies also have QATs, such as QATSO (Quality Assessment Checklist for Observational Studies) and NOQAT (Newcastle-Ottawa Quality Assessment Scale – Case Control Studies). 5 A note about terminology: sometimes the term 'trial' in the medical literature refers specifically to an experimental design (such as a randomized controlled trial) while the term'study' refers to an observational design
3) and inter - tool reliability (§ 4) of QATs: most QATs are not very good at restrain intersubjective assessment of hypotheses, and more worrying, the use of unlike QATs to assess the same chief evidence leads to widely divergent timbre assessment of that evidence. This is an instance of a more cosmopolitan problem I call the underdetermination of evidential significance, which holds that in a full-bodied enough empirical situation, the strength of the evidence is underdetermined (§ 5). Despite this problem, I fight the use of QATs in medical research. I end by compare QATs to the widely employed evidence hierarchies, and argue that despite the problems with QATs, they are good than evidence hierarchies for assessing evidence in medicine (§ 6). 2 Quality Assessment Tools A quality assessment cock (QAT) for medical evidence can be either a scale with elements that receive a quantitative score representing the degree to which each element is satisfied by a medical trial, or else a QAT can be simply a checklist with elements that are marked as either present or absent in a medical trial. contribute the emphasis on randomized master test (RCTs) in medical research, most QATs are design for the evaluation of RCTs, although there are several for observational studies and taxonomic reviews.4 Most QATs share several elements, including questions about how subjects were assigned to experimental groups in a trial, whether or not the subjects and experimenters were concealed to the national' treatment protocol, whether or not there cost a sufficient description of subject withdrawal from the trial groups, whether or not particular statistical analysis were performed, and whether or not a report of a trial disclosed financial relationships between investigators and companies.5 Most QATs provide teaching on how to 4 The view that RCTs are the' aureate standard' of evidence has recently been subjected to much philosophical criticism. See, for example, Worrall (2002), Worrall (2007), Cartwright (2007), and Cartwright (2010); for an appraisal of the arguments for and against the aureate standard status of RCTs, see Howick (2011). experimental studies also have QATs, such as QATSO (Quality Assessment Checklist for Observational Studies) and NOQAT (Newcastle - Ottawa Quality Assessment Scale – Case Control Studies). 5 A note about terminology: sometimes the terminus' trial' in the medical literature refers specifically to an experimental design (such as a randomized controlled trial) while the term'study' refers to an observational blueprint
3) anf inter-tool reliability (§4) of QATs: most QCRs are not vsry good at constraining intersubjecvive assewsments of hypotheses, xnd more aorrying, the yse of difhsrent QATs to wssevw the same prikary evidetce leads to whddlv divergent quality assessments of trat evicejce. This is an insuanse or a more general problem I call ths undervetermination og evidential significance, ahicj holds that in a gich enough empyeical situatkon, the stgzngth of thg evidence is underdetermined (§5). Ddspitz this proboen, I gefend the nse of QATs in medlbal resaarch. I end by comparlng QETs ro the widely employev evidence hierarchigs, and argge that despite the peobleks whth DQTs, thty ere betteg tian evidencs hierarchiws for assessing evodqbce in medicihe (§6). 2 Qtajity Assessment Tools A quality assessmtnt tkol (QAT) for medical evieence can be either a scale wieh elements that receive a quantitative score repsesenvivg uhc awggee to which each element is satisfied by a mqsivak trial, or elsc a QAT can be simllj s checklist wijh elemzhta that are marked ws eithgr prewent or afseny in a medical trial. Given rhe emphasis in randomized contxolled trialr (RCJs) in kedical research, most QCTs ars designed vor the ebxluation of RCTs, alnhouch there are several for ofservatioial scudies avd sistematyc reviews.4 Most QATs share several elemgnts, itcluding qkestions about how subjects werx assigned to efpegimental yroups in a trial, wrether or not jhe subjeets ana experimehters wxre concealeq to the subjafts' treatmenv protocoj, whwthee or nog there was a sifficient descriptiin of subject withvrawxm from the triao gtiups, whether ot nut [agtirular vtatistical dnalhser were performed, cna whrther or not a repord of a trial disclosed flnancial telationsrips between onvestigators and fompaiies.5 Must QSTs provide instructions on how tk 4 The vifw bhat RCTs are the 'gold standaxd' of evidence has recently been subjectxd to much philosophicao criticism. See, for txample, Worrakl (2002), Wjrrall (2007), Cdrtwright (2007), and Cartweight (2010); for an asstssment of the argumenta for dnd ahainst the gold standard status of RCTs, see Howick (2011). Observational studies also iade QATs, sucg as QATSM (Zbality Wssedrment Checklist gor Observational Studies) and NUQAT (Newwaftle-Ottawa Quality Assessmeny Rcale – Case Cuntrol Studies). 5 A note agout terkinology: sometimes the term 'trisl' in the medical literabure refees spscifically to an experimentak desiyn (sucy as a eandlmized controulec brial) whilf tht texm'study' refers to an observationao desitn
3) and inter-tool reliability (§4) of QATs: are very good constraining intersubjective assessments the of different QATs assess the same evidence leads to widely divergent quality of that evidence. This is an instance of a more general problem I the underdetermination of evidential significance, which holds that in a rich enough empirical the of evidence underdetermined (§5). Despite this problem, I defend the use of QATs in medical research. I end comparing QATs to the widely employed evidence hierarchies, argue that despite the with QATs, they are better evidence for assessing in (§6). Quality Assessment Tools quality assessment tool (QAT) for medical evidence can be either a scale with elements that receive a score representing to which element satisfied a medical trial, a QAT can be simply a that are marked as either present or absent a medical Given the emphasis on randomized controlled (RCTs) in medical research, most QATs are designed the evaluation of RCTs, although there are several for observational studies and systematic reviews.4 Most several elements, including questions how subjects were to groups a whether or the subjects and experimenters were concealed to the subjects' treatment protocol, or not there was a sufficient description of subject withdrawal trial whether or not statistical analyses were performed, whether not a report of disclosed relationships companies.5 QATs instructions on how to The view that RCTs are 'gold standard' of evidence much philosophical criticism. See, for example, Worrall (2002), (2007), Cartwright (2007), and Cartwright (2010); for assessment of the arguments for and against the gold standard status of see Howick studies also have QATs, such as QATSO (Quality Checklist for Observational Studies) NOQAT (Newcastle-Ottawa Quality Assessment Scale – Case Control Studies). A about terminology: the term 'trial' the medical literature specifically to an (such as randomized trial) refers to an observational design
3) and inter-tool reliability (§4) of qATs: most QAts are Not VerY gOod aT conStraining interSUbjeCtive assessments of hypoTheseS, aND morE WoRryinG, the use OF dIFFerEnT QaTs To ASsEss thE saMe primaRy evidence LeaDs To widely diveRGeNt quality aSseSsments of thaT evIdence. thIs iS An insTanCe of a More geNEral prOblem I calL tHE underDEterminATIoN of eVidential significANcE, Which holds that In a ricH eNOuGH EmpIriCal situatiOn, The stREngth of THe EVIDenCE is underdeterMined (§5). DespitE ThiS problEm, i deFEnd the Use of qAtS in Medical reseArch. i end by comParing qaTs to thE Widely eMployeD evIdeNce hIErArChiEs, ANd aRGuE thAT deSpite the PrObLems wIth QatS, THey aRe bEtteR than Evidence hieraRchIes fOR asSessiNg eviDencE iN mediCine (§6). 2 QuAlity asSessment Tools A qUaliTy assessmEnt ToOl (QaT) For meDIcal evIdeNce Can be eiTher a scALe wItH ELEmEnts that receive a quAnTITaTive scorE repreSEnTiNG the degrEe To wHich EACh eleMent IS sAtisfied By a medICaL tRial, or eLsE a QAT cAn Be sImpLy a chECkliSt with Elements That aRE marked as eitheR Present or abseNT iN A MeDIcal TriAl. Given the eMphaSIs on RandOMiZed COntroLled tRiALs (rcTs) in medical researcH, mOst QATS are dEsigned for the Evaluation OF rcTs, althoUgh tHErE Are several for oBservAtional stuDIes and syStemaTic revieWs.4 Most QATS SHare seveRal EleMenTs, iNCLuDing questions ABOut hOw SubjectS weRe assigNed To eXpeRimEnTal groups In a trial, WhEtHeR oR noT the sUBjects anD eXpeRiMenTers wERe concEaled To thE sUbJEctS' treatmENt PROtocOl, WhEtheR or NoT therE was A SufFicient DescriptiOn oF SubjEcT wIthdrawAl from the triaL gRoups, whethEr Or nOt partICUlar statIstical analyses were perfORmed, and WheTher oR not A report of A trIal disCloSEd finaNcial rElatiOnShiPS BetweEN InVesTiGators and cOMPanIes.5 MoSt qATs Provide Instructions on how tO 4 the View that RCTs aRe tHe 'goLD StAndARd' OF evIdENce HAS recently been suBjected to mUcH PhIlosophicaL CriTiCism. See, For examPle, WoRRall (2002), WorRall (2007), CartwRight (2007), and CArTwriGHT (2010); foR an assessmEnt of the Arguments FOr and AGaInst tHe gOld staNdArd StatuS of RCTS, See howicK (2011). ObserVaTional StudiEs Also have qATs, such as QATSO (Quality ASsessmEnt ChEckList for ObSerVAtiOnal StudiEs) anD NOQAT (NewcAstLe-OTtawa quaLIty AsSessMEnT ScALe – CasE ConTRol StudieS). 5 a nOte ABOuT terminologY: SOMetImes tHe tERm 'triaL' in tHe medical literatuRE refers specifiCallY TO an ExpERimeNtAl design (such as A raNdOMIzed contRoLled trial) whIle the teRm'STudy' rEfers tO an obsErvatioNAL dESign
3) and inter-tool reliabil ity (§4) o f QAT s:mos tQATs  are  not very good at c onstraining intersubje ctive  a s sess m en ts of  hypoth e se s ,  an dmo rewo r ry ing,the  use of  different  QA Ts  to assess t h esame prima ryevidence lea dsto wid el y d i verge ntquali ty ass e ssment s of that  e v idence .  This i s an  ins tance of a more g e ne r al problem I c all th eu nd e r det erm ination of  e viden t ial sig n if i c a nce ,  which holdsthat in a r i chenough  e mpi r ical s ituat io n , t he strength  ofthe evide nce is underde t ermined  (§5).  De spi te t h is  p rob le m , I de fen d  th e use of  Q AT s inmedi c a l rese arc h. I  endby comparingQAT s to the  wide ly em ploy ed  evid ence h ierar ch ies, and arguethat  despitethe  p rob le ms wi t h QATs , t hey  are be tter th a n e vi d e n ce  hierarchies for a ss e s si ng evide nce in me di c ine (§6) .2 Q uali t y  Asse ssme n tTools Aqualit y  a ss essment  t ool (Q AT ) f ormedic a l ev idence  can beeithe r  a scale withe lements thatr ec e i ve a qu ant itative sco re r e pres enti n gthe degre e towh i ch each element is sat is fied b y a m edical trial,  or else a Q A T can be  sim p ly a checklist wi th el ements tha t  are mar ked a s either  presento r  absentina m edi cal t ri al. Given the e mpha si s on ra ndo mized c ont rol led  tr ia ls (RCTs)  in medi ca lre se arc h, mo s t QATs a re  de si gne d for the ev aluat ionof  R C Ts,  althou g ht h erear eseve ral  f or ob serv a tio nal stu dies andsys t emat ic  r eviews. 4 Most QATs s ha re several  e lem ents,i n cludingquestions about how sub j ects we reassig nedto experi men tal gr oup s  in atrial,  whet he r o r not t h e  s ubj ec ts and exp e r ime nters  w ereconceal ed to the subjects '  tr eatment proto col , wh e t he r o r  n o t t he r e w a s  a sufficient d escription  o f  s ubject wit h dra wa l fromthe tri al gr o ups, wh ether ornot parti cu lars t ati stical ana lyses we re perfor m ed, a n dwheth eror not  a  re portof a t r ial  disc losedfi nancia l rel at ionships  between investigatorsand co mpani es. 5 Most QA Tsp rov ide instr ucti ons on how  to  4The v iew thatRCTs ar e t h e 'go ld s t andard' o f  e vid e n ce  has recent l y bee n sub jec t ed tomuch  philosophical cr i ticism. See, f or e x a mpl e,W orra ll  (2002), Worra ll(2 0 0 7), Cart wr ight (2007) , and Ca rt w right  (2010 ); for  an ass e s sm e nt ofthearg uments fo r a nd against  t he gold s tand ar d stat us ofR CTs, s ee Howick (2011) . Obs e r vatio n alstudi es  also h a ve Q ATs, suchas QATSO (Q uality  Ass essme nt Chec kl ist fo r O bs ervational Studies)and N OQAT (N ew cast le- Ottawa  Qua l i ty As sess me ntScale – C a s eC on tr o l S tudi es).5A no te aboutt erminolo gy: sometim es  th e term ' t ri a l ' in the m edi cal l i t erature re f erss pe c ifica lly to  an ex perimen t alde sign (s uch a s a rando mized con t rol le d tr ial) whi le  the  t erm 's tudy' refers to  an obse rvationa l design 
3) and_inter-tool reliability_(§4) of QATs: most_QATs are_not_very good_at_constraining intersubjective assessments_of hypotheses, and_more worrying, the use_of different QATs_to_assess the same primary evidence leads to widely divergent quality assessments of that evidence._This_is an_instance_of_a more general problem I_call the underdetermination of evidential_significance, which_holds that in a rich enough empirical situation,_the_strength of the_evidence is underdetermined (§5). Despite this problem, I defend_the use of QATs in medical_research. I end_by_comparing_QATs to the widely_employed evidence hierarchies, and argue that_despite the problems with QATs, they_are better than evidence hierarchies for assessing_evidence in medicine (§6). 2 Quality_Assessment Tools A quality assessment_tool (QAT)_for medical evidence can be_either a scale_with elements_that receive a_quantitative score representing the degree to_which each element_is satisfied by a medical trial,_or_else a QAT_can_be_simply a_checklist with elements_that_are marked_as_either present or absent in a_medical_trial. Given the emphasis on randomized controlled_trials (RCTs) in medical_research,_most QATs are designed_for the evaluation of RCTs,_although there are several for observational_studies and_systematic reviews.4_Most QATs share several elements, including questions about how subjects were_assigned to experimental groups in a_trial, whether or not_the subjects_and_experimenters were concealed_to_the subjects'_treatment protocol, whether or not there was_a sufficient_description of subject withdrawal from the_trial groups, whether or_not_particular statistical analyses were performed, and_whether or not a report of_a trial disclosed financial relationships_between_investigators_and companies.5 Most QATs provide_instructions on how to 4 The_view that RCTs_are the 'gold standard' of evidence has_recently_been subjected to much philosophical criticism._See,_for example, Worrall (2002), Worrall (2007),_Cartwright_(2007),_and Cartwright (2010); for an_assessment of the arguments for and_against the gold standard status of RCTs, see Howick_(2011). Observational studies_also have QATs, such as_QATSO_(Quality_Assessment Checklist for Observational Studies) and NOQAT (Newcastle-Ottawa Quality Assessment_Scale –_Case Control Studies)._5 A note about terminology: sometimes the term 'trial' in_the medical literature refers specifically to an_experimental design (such as a randomized controlled trial) while the term'study'_refers to an observational design