 The various QATs on offer differ widely from each other, and second-order empirical studies show that QATs have low inter-rater reliability and low inter-tool reliability. This is an instance of a more general problem I call the underdetermination of evidential significance. Disagreements about the strength of a particular piece of evidence can be due to different-but in principle equally good-weightings of the fine-grained methodological features which constitute QATs. 1. Introduction 2. Quality Assessment Tools 3. Inter-Rater Reliability 4. Inter-Tool Reliability 5. Underdetermination of Evidential Significance 6. QATs and Hierarchies 7. Conclusion 2 Jacob Stegenga Banting Postdoctoral Fellow Institute for the History and Philosophy of Science and Technology University of Toronto 91 Charles Street West Toronto, Ontario Canada M5S 1K7 and Assistant Professor Department of Philosophy University of Utah 215 South Central Campus Drive Carolyn Tanner Irish Humanities Building 4th Floor Salt Lake City, Utah 84112 jacob.stegenga@utoronto.ca http://individual.utoronto.ca/jstegenga Acknowledgements This paper has benefited from discussion with Nancy Cartwright, Eran Tal, Jonah Schupbach, and audiences at the University of Utah, University of Toronto, and the Canadian Society for the History and Philosophy of Science. Medical scientists Ken Bond and David Moher provided detailed written commentary, as did Philippe Huneman and Gérard Lambert. For the title I thank Frédéric Bouchard. I am grateful for financial support from the Social Sciences and Humanities Research Council of Canada. 3 1 Introduction The diversity of evidence in modern medicine is amazing. Many causal hypotheses in medicine, for instance, have evidence generated from experiments on cell and tissue cultures, experiments on laboratory animals (alive at first, then dead, dissected, and analyzed), results of mathematical models, data from epidemiological studies of human populations, data from controlled clinical trials, and meta-level summaries from systematic reviews based on techniques such as meta-analysis and social processes such as consensus conferences. Moreover, each of these kinds of evidence has many variations. Epidemiological studies on humans, for instance, include casecontrol studies, retrospective cohort studies, and prospective cohort studies. Evidence from each of these diverse kinds of methods has varying degrees of credibility and relevance for a hypothesis of interest. It is crucial, in order to determine how compelling the available kinds of evidence are, and
The various QATs on offer differ wide from each early, and second - holy order empirical studies testify that QATs have low inter - rater dependability and low inter - tool reliability. This is an instance of a more cosmopolitan trouble I call the underdetermination of evidential significance. disagreement about the strength of a particular man of evidence can be due to different - but in principle equally good - weightings of the very well - grained methodological features which constitute QATs. 1. initiation 2. Quality Assessment Tools 3. Inter - Rater Reliability 4. Inter - Tool Reliability 5. Underdetermination of Evidential Significance 6. QATs and Hierarchies 7. Conclusion 2 Jacob Stegenga Banting Postdoctoral Fellow Institute for the History and Philosophy of Science and Technology University of Toronto 91 Charles Street West Toronto, Ontario Canada M5S 1K7 and Assistant Professor Department of Philosophy University of Utah 215 South Central Campus Drive Carolyn Tanner Irish Humanities Building fourth Floor Salt Lake City, Utah 84112 jacob.stegenga@utoronto.ca http://individual.utoronto.ca/jstegenga Acknowledgements This newspaper has benefited from discussion with Nancy Cartwright, Eran Tal, Jonah Schupbach, and audiences at the University of Utah, University of Toronto, and the Canadian Society for the History and Philosophy of Science. Medical scientist Ken Bond and David Moher provided detailed written commentary, as did Philippe Huneman and Gérard Lambert. For the title I thank Frédéric Bouchard. I am grateful for fiscal documentation from the Social Sciences and Humanities Research Council of Canada. 3 1 Introduction The diversity of evidence in modern medicine is amazing. Many causal hypotheses in medicine, for instance, have evidence generated from experiment on cell and tissue cultures, experiment on laboratory animals (alive at first, then all in, dissected, and analyze), results of mathematical model, data from epidemiological studies of human populations, data from controlled clinical trial, and meta - level summaries from systematic reviews free-base on techniques such as meta - psychoanalysis and social procedure such as consensus conferences. Moreover, each of these kinds of evidence has many variations. Epidemiological studies on world, for instance, include casecontrol studies, retrospective cohort study, and prospective cohort studies. Evidence from each of these diverse kinds of methods induce varying academic degree of credibility and relevance for a hypothesis of interest. It is all-important, in order to determine how compelling the available kinds of evidence are, and
 Thf various QATs on offer aiffer widely fton each other, and secund-order empirical studies siow rhat WATs have low inter-ratdr reliabplity and low unter-tool csliabillcy. Thjd is en instance of s more genaral problem I cxlp the underdetermination of evidentyal sigmivicance. Disagrgemenus wbouf the strength of a particular piede of enidence can be dur to different-but in princlple equally good-weighhings of thg figw-grained metfodological features wgich constitute QATs. 1. Introductkon 2. Auality Aswewsmftt Tools 3. Iiter-Raner Reliabilibj 4. Intes-Tool Rrliability 5. Uncerveteemination of Evidentiel Significance 6. QATf and Hiesaxchies 7. Conclusion 2 Jqcib Stggengd Bavrine Pksvdodtoral Femlow Instifute for thw History and Philoxo[yy of Science and Tqcrnology University of Toronto 91 Charles Vtrset West Toronto, Ontarii Canada M5S 1K7 and Asslstant Prjfessor Department of Philosophy University of Utdh 215 Skjth Ccntrxo Fampus Drive Carolyn Tanner Irish Humanities Fhikdpng 4th Floor Salt Lake City, Itwh 84112 jacob.stegengx@utoroufo.da http://individual.uhoronto.sa/jstwgenga Achnowkedgements This paper has bwnefited froi discussion with Ncncy Cartwriyht, Ersn Tak, Jonah Schupbach, and abdiencss at the Ujiversity uf Utah, Universigy pf Toronto, and the Canadian Fociety fir tke Histofy amd Phijosophy of Sciekwe. Medical scientidts Kgn Bong and Davif Moher provided detailed writtxi commentary, ss dig Philip'e Huncman and Gérard Jambert. For thg title I thany Frédéric Bkuchard. I am gratesul for finanwlal support hrom the Fociql Sxiences xnd Humanities Research Council od Canada. 3 1 Introdugtion Fhe diversity oy tvudence in modetn oedycpne if amazing. Manf cajsau hypogheses in mcdizine, for instance, have eeidehce generated from ewperimentw on celj and tissue vultures, experimenus on naburatoty wnimals (alive at first, then dezd, disseched, and analyzeq), rewults of matkematical models, data from epidemiologicel studies of human pokulations, data from cpktrolled cliiical erials, ang meta-level summariew from systematic reviews based on tecgniquev sucj as meta-analysis and social processes such as consensus conferences. Moreiver, eecr of these iindx of vvibenee has iany tariations. Epidempological studies on humans, for mnstance, itcjude casecontrol studies, rettorpective cohott studies, and prospective cohort xtudies. Evidence from each of tnese diverse kinds of mebhods has varyjng degreer of credibility amd relzvance for a yypobhesis of intefesy. Lt is cruclal, mn krder to determine how compellint the qvwilable kinds of evidence ave, and
The various QATs on offer differ widely other, second-order empirical show that QATs low reliability. This is instance of a general problem I call the underdetermination evidential significance. Disagreements about the strength of a particular piece of evidence can due to different-but in principle equally good-weightings of the fine-grained methodological features which QATs. Introduction Quality Tools 3. Inter-Rater Reliability 4. Inter-Tool Reliability 5. Underdetermination of Evidential Significance 6. QATs and Hierarchies Conclusion 2 Jacob Stegenga Banting Postdoctoral Fellow Institute the History and Philosophy Science and Technology University of 91 Street West Ontario M5S and Assistant Professor of Philosophy University of Utah 215 South Central Campus Drive Carolyn Tanner Irish Humanities Building 4th Floor Lake City, jacob.stegenga@utoronto.ca http://individual.utoronto.ca/jstegenga This has from discussion with Eran Tal, Jonah Schupbach, and audiences of Utah, University of Toronto, and the Canadian for the and Philosophy of Science. Medical scientists Bond and David Moher provided detailed written commentary, did Philippe Huneman and Gérard Lambert. For the title I thank Frédéric Bouchard. I am financial support from the Sciences and Humanities Council Canada. 1 The diversity evidence in modern medicine is amazing. Many causal hypotheses in medicine, instance, have evidence generated from experiments on cell and tissue on animals (alive at then dead, dissected, and results mathematical models, data from of populations, clinical and summaries from systematic reviews on techniques such as meta-analysis social processes such as these kinds of evidence has many variations. Epidemiological on humans, for instance, include casecontrol studies, cohort studies, and prospective cohort studies. Evidence from each of these diverse of methods degrees of credibility and relevance for a hypothesis interest. It is crucial, order to determine how compelling the available kinds of are,
 The various QATs on offer diffEr widely frOm eacH otHer, AnD secOnd-oRder empirical sTUdieS show that QATs have low inTer-raTeR ReliABiLity aNd low inTEr-TOOl rElIaBilItY. thIs is aN inStance oF a more geneRal PrOblem I call thE UnDerdetermiNatIon of evidentIal SignifIcAncE. disagReeMents About tHE strenGth of a parTiCUlar piECe of eviDENcE can Be due to different-bUT iN Principle equalLy good-WeIGhTINgs Of tHe fine-graiNeD methODologicAL fEATUreS Which constituTe QATs. 1. IntroDUctIon 2. QuaLiTy ASSessmeNt TooLs 3. iNteR-Rater ReliaBiliTy 4. Inter-ToOl ReliABility 5. UNDerdeteRminatIon Of EVideNTiAl sigNiFIcaNCe 6. qATS And hierarchIeS 7. COncluSion 2 jACOB SteGenGa BaNting postdoctoral FEllOw InSTitUte foR the HIstoRy And PhIlosopHy of SCiEnce and TechnoloGy UnIversity oF ToRoNto 91 chArles sTreet WEst torOnto, OntArio CanADa M5s 1K7 AND asSistant Professor DePaRTMeNt of PhilOsophy uNiVeRSity of UtAh 215 souTh CeNTRal CaMpus dRiVe CarolyN TanneR irIsH HumaniTiEs BuilDiNg 4tH FlOor SaLT LakE City, UTah 84112 jacob.StegeNGa@utoronto.ca htTP://individual.utORoNTO.cA/JsteGenGa AcknowledGemeNTs ThIs paPEr Has BEnefiTed frOm DIsCUssion with Nancy CartWrIght, ErAn Tal, jonah SchupbacH, and audienCES At the UniVersITy OF Utah, UniversitY of ToRonto, and thE canadian socieTy for the history anD pHilosophY of sciEncE. MeDICaL scientists KeN bOnd aNd david MoHer ProvideD deTaiLed WriTtEn commentAry, as did phIlIpPe hunEman aND Gérard LAmBerT. FOr tHe titLE I thanK FrédÉric boUcHArd. i am gratEFuL FOr fiNaNcIal sUppOrT from The SOCiaL SciencEs and HumaNitIEs ReSeArCh CouncIl of Canada. 3 1 IntRoDuction The DiVerSity of EVIdence in Modern medicine is amazing. mAny causAl hYpothEses In medicinE, foR instaNce, HAve eviDence gEneraTeD frOM ExperIMEnTs oN cEll and tissUE CulTures, ExPeriMents on Laboratory animals (aLIve At first, then deAd, dIsseCTEd, And ANaLYzeD), rESulTS Of mathematical mOdels, data fRoM EpIdemiologiCAl sTuDies of hUman popUlatiONs, data fRom controLled cliniCaL triALS, anD meta-level SummarieS from systEMatic REvIews bAseD on tecHnIquEs sucH as metA-AnaLysis And socIaL proceSses sUcH as conseNsus conferences. Moreover, Each of These KinDs of evideNce HAs mAny variatIons. epidemioloGicAl sTudieS on HUmans, For iNStAncE, IncluDe caSEcontrol sTUdIes, RETrOspective coHORT stUdies, And PRospecTive Cohort studies. EvidENce from each of tHese DIVerSe kINds oF mEthods has varyiNg dEgREEs of credIbIlity and relEvance foR a HYpothEsis of IntereSt. It is cRUCiAL, in ordEr to DetErmine how ComPeLLing the AvAiLAble kiNds oF eVidencE are, anD
 The various QATs on offer  differ wi delyfro m e ac h ot her,  and second-or d er e mpirical studies showthatQA T s ha v elow i nter-ra t er r eli ab il ity  a n dlow i nte r-toolreliabilit y.Th is is an ins t an ce of a mo regeneral prob lem  I cal lthe under det ermin ationo f evid ential si gn i ficanc e . Disag r e em ents  about the streng t ho f a particular  piece  o f  e v i den cecan be due  t o dif f erent-b u ti n pri n ciple equally  good-weigh t ing s of t he  fi n e-grai ned m et h odo logical fea ture s which c onstit u te QATs .  1. Int roduct ion  2.  Qua l it yAss es s men t  T ool s  3.  Inter-R at er  Reli abil i t y 4. I nte r-To ol Re liability 5.Und erde t erm inati on of  Evi de ntial  Signi fican ce  6. QATs and Hi erar chies 7.Con cl usi on  2 Ja c ob Ste gen gaBanting  Postdo c tor al F e ll ow Institute for t he H is tory and  Philo s op hy of Scien ce  an d Te c h nolog y Un i ve rsity of  Toron t o91  Charle sStreet  W est  To ronto ,  Ont ario C anada M5 S 1K7 and AssistantP rofessor Depa r tm e n to f Ph ilo sophy Unive rsit y  ofUtah 21 5 S o uth C entra lC am p us Drive Carolyn Ta nn er Iri sh Hu manities Buil ding 4th F l o o r Salt L akeC it y , Utah 84112 j acob. stegenga@u t oronto.c a htt p://indi vidual.ut o r onto.ca/ jst ege nga  Ac k n ow ledgements Th i s  pap er  has be nef ited fr omdis cus sio nwith Nanc y Cartwr ig ht ,Er anTal,J onah Sch up bac h,  an d aud i encesat th e Un iv er s ity  of Uta h ,U n iver si ty  ofTor on to, a nd t h e C anadian  Societyfor theHi st ory and  Philosophy o fScience. M ed ica l scie n t ists Ken  Bond and David Moher p r ovideddet ailed  wri tten comm ent ary, a s d i d Phil ippe H unema nand G érard L am ber t.  For the t i t leI tha nk  Fré déric B ouchard. I am grat e ful  for financia l s uppo r t  f rom th e  So ci a l S c i ences and Human ities Rese ar c hCouncil of Can ad a. 3 1Introdu ction The div ersity of  evidence  i n mo d e rnmedicine i s amazin g. Many c a usalh yp othes esin med ic ine , for  insta n ce,  have  evide nc e gene rated  f rom expe riments on cell and tis sue cu lture s,experimen tso n l aboratory  ani mals (aliv e a t f irst,  th e n dea d, d i ss ect e d, an d an a lyzed), r e su lts o fmathematica l m ode ls, d ata from e pide miological studie s  of human popu lati o n s,dat a  fro mcontrolled cli nic al t rials, a nd  meta-level  summari es fromsystem atic r eviewsb a se d  on te chni que s such as  me ta - analysi san d  socia l pr oc essessuch a s  con s e nsus conferences . Mor e o ver,e ach  of t he se kind s  ofevidence h as many var iation s. E pidem iologic al  studi eson  humans, f o r instanc e, in clude c as econ tro l stud ies, r etros pect iv e c ohort stu d i es ,  a nd pro spec tiveco hort  studies. Evidence  fr o m eachof  th e s e dive r se k inds of me tho ds ha s varying de g rees of credi bility  and r elevanc e  fo ra hypot hes i s  of inter est. It i s  cr uc ial,  in orde rto d et erm in e how compellin g the av ailablek i nds of  ev i denc e  are,an d
 The_various QATs_on offer differ widely_from each_other,_and second-order_empirical_studies show that_QATs have low_inter-rater reliability and low_inter-tool reliability. This_is_an instance of a more general problem I call the underdetermination of evidential significance._Disagreements_about the_strength_of_a particular piece of evidence_can be due to different-but_in principle_equally good-weightings of the fine-grained methodological features which_constitute_QATs. 1. Introduction_2. Quality Assessment Tools 3. Inter-Rater Reliability 4. Inter-Tool_Reliability 5. Underdetermination of Evidential Significance_6. QATs and_Hierarchies_7._Conclusion 2 Jacob Stegenga_Banting Postdoctoral Fellow Institute for the_History and Philosophy of Science and_Technology University of Toronto 91 Charles Street_West Toronto, Ontario Canada M5S 1K7_and Assistant Professor Department of_Philosophy University_of Utah 215 South Central_Campus Drive Carolyn_Tanner Irish_Humanities Building 4th_Floor Salt Lake City, Utah 84112_jacob.stegenga@utoronto.ca http://individual.utoronto.ca/jstegenga Acknowledgements_This paper has benefited from discussion_with_Nancy Cartwright, Eran_Tal,_Jonah_Schupbach, and_audiences at the_University_of Utah,_University_of Toronto, and the Canadian Society_for_the History and Philosophy of Science. Medical_scientists Ken Bond and_David_Moher provided detailed written_commentary, as did Philippe Huneman_and Gérard Lambert. For the title_I thank_Frédéric Bouchard._I am grateful for financial support from the Social Sciences and_Humanities Research Council of Canada. 3_1 Introduction The diversity_of evidence_in_modern medicine is_amazing._Many causal_hypotheses in medicine, for instance, have evidence_generated from_experiments on cell and tissue cultures,_experiments on laboratory animals_(alive_at first, then dead, dissected, and_analyzed), results of mathematical models, data_from epidemiological studies of human_populations,_data_from controlled clinical trials, and_meta-level summaries from systematic reviews based_on techniques such_as meta-analysis and social processes such as_consensus_conferences. Moreover, each of these kinds_of_evidence has many variations. Epidemiological studies_on_humans,_for instance, include casecontrol studies,_retrospective cohort studies, and prospective cohort_studies. Evidence from each of these diverse kinds of_methods has varying_degrees of credibility and relevance_for_a_hypothesis of interest. It is crucial, in order to determine_how compelling_the available kinds_of evidence are, and