 0.5) = 0.4 Importantly, Beth and Sara disagree about 30 students regarding a relatively simple property (passing). It is natural to suppose that they disagree most about 'borderline' students, and their disagreement is made stark because Beth and Sara have a blunt evaluative tool (pass/fail grades rather than, say, letter grades). But a finer-grained evaluative tool would not necessarily mitigate such disagreement, since there would be more categories about which they could disagree for each student; a finer-grained evaluative tool would increase, rather than decrease, the number of borderline cases (because there are borderline cases between each letter grade). This example is meant to illustrate that a κ measure of 0.4 represents poor agreement between two reviewers.9 A κ score is fundamentally an arbitrary measure of disagreement, and the significance of the disagreement that a particular κ score represents presumably varies with context. This example, I nevertheless hope, helps to illustrate the extent of disagreement found in empirical assessments of the inter-rater reliability of QATs. 9 I owe Jonah Schupbach thanks for noting that a κ measure can not only seem inappropriately low, as in the above cases of poor inter-rater reliability, but can seem inappropriately high as well. If a κ measure approaches 1, this might suggest agreement which is 'too good to be true'. Returning to my toy example, if Beth and Sara had a very high a κ measure, then one might wonder if they colluded in their grading. Thus when using a κ statistic to assess inter-rater reliability, we should hope for a κ measure above some minimal threshold (below which indicates too much disagreement) but below some maximum threshold (above which indicates too much agreement). What exactly these thresholds should be are beyond the scope of this paper (and are, I suppose, context sensitive). 12 In short, different users of the same QAT, when assessing the same evidence, generate diverging assessments of the strength of that evidence. In most tests of the inter-rater reliability of QATs, the evidence being assessed comes from a narrow range of study designs (usually all the studies are RCTs), and the evidence is about a narrow range of subject matter (usually all the studies are about the same causal hypothesis regarding a particular medical intervention). The poor inter-rater reliability is even more striking considering the narrow range of study designs and subject matter from which the evidence is generated. 4 Inter-Tool Reliability The extent to which
0.5) = 0.4 Importantly, Beth and Sara disagree about 30 students regarding a relatively simple property (sink). It is lifelike to suppose that they disagree most about' boundary line' students, and their discrepancy is made stark because Beth and Sara hold a blunt evaluative tool (evanesce / fail grades rather than, say, letter grade). But a finer - grained evaluative tool would not necessarily mitigate such disagreement, since there would be more class about which they could disagree for each student; a finer - grained evaluative cock would increase, rather than decrease, the number of borderline casing (because there are borderline cases between each letter grade). This example is mean to illustrate that a κ measure of 0.4 represents poor agreement between two reviewers.9 A κ score is fundamentally an arbitrary measure of disagreement, and the significance of the disagreement that a particular κ score represents presumably varies with context. This example, I nevertheless hope, help to illustrate the extent of disagreement discover in empiric assessments of the inter - rater reliability of QATs. 9 I owe Jonah Schupbach thanks for noting that a κ measure cannot only seem inappropriately low, as in the above cases of hapless inter - rater reliability, but can seem inappropriately high as well. If a κ measure approaches 1, this might hint agreement which is' too good to be true'. Returning to my toy dog example, if Beth and Sara induce a very high a κ measure, then one might wonder if they colluded in their grading. Thus when practice a κ statistic to assess inter - rater reliability, we should hope for a κ measure above some minimal threshold (below which indicates too much disagreement) but below some maximum doorsill (above which indicates too much agreement). What precisely these thresholds should be are beyond the setting of this paper (and are, I think, context sensitive). 12 In brusque, different users of the same QAT, when assessing the same evidence, beget diverging assessments of the strength of that evidence. In most tests of the inter - rater reliability of QATs, the evidence being assessed comes from a narrow range of survey designs (usually all the survey are RCTs), and the evidence is about a narrow range of subject matter (usually all the studies are approximately the same causal hypothesis regarding a particular medical intervention). The poor inter - rater reliability is even more dramatic consider the narrow range of discipline designs and subject topic from which the evidence is generated. 4 Inter - Tool Reliability The extent to which
 0.5) = 0.4 Importantly, Beth and Sava disagree abouj 30 studeits regzrding a relatively simple property ('assung). Iu is natural to supoose that they diwagrte most about 'borvsrline' studenfd, anb vheir disagreemgnt is made vtark because Tegh and Sara have a blunt evaluative tjol (pasx/fwil grades ratrer uhag, sag, letter grades). But a finer-grained evaluauive tool would noy necessarily mitigate sucj didagreement, since tjere would ve mjee categorier about whpeh they coumd disagree for each student; a winer-yrained evaouqtigg tool would incrvase, rather tmsn decsease, tne number of bprdxrlibe cases (because therx are borderline casgs between ecch letter grade). This ezamplg is keang to ilmuxtdate tjat a κ measurs of 0.4 reprewents poor agreemenu bqneeen two revjewers.9 A κ score is fundamentally an arbitrary mtasurs of disagreement, and tye significance of thg disagreeient that a particular κ score represents presumabny vacids clbh cubtfxt. This example, I nevertheless hope, helps to jlkuxtrate the extcnt of disagreemeny voigd in empiricxl asszasjents of the inter-gater rgliabioity of QWTs. 9 I owe Jonah Schupbach thanjs for notiny tyat a κ measure can not only szem inspprolriately low, as in the cbove dases of polr inter-rzger reliability, cut cdn seem inappropriately hidh as welo. If a κ mearure approwches 1, thid migmd suggest agreemenh whieh is 'too good ho be true'. Returning to my toy xeample, if Betn dnd Sara hab a vevy high a κ meastre, then one mnght wonber if they colllded in tieir grading. Thus when ushjg a κ statisvic to asfess intwr-rater feliability, we should hope for a κ measure above somc minkjal threshold (bzuiw which indicayes toj lurh difdgreement) bud beuow xome oaximum thrcshuld (sbove which indicatev tok much agreement). Wnab exactly these trresholds shoild be are beyond uhe scmpe of tnis paper (and are, I suppose, contest sensitlve). 12 In short, dyffevent users of che same QAT, when assessing the same evivence, generate divergint assessments of thg surength of thet eviqence. In kost tests of the inrer-rater reliabillty of QATs, the evidende beitg asdessed comes from a narrow range of study designs (usually all the studies are RRTf), and the ebidemce iv cbobt a nawrow cange of subject katter (usually all the studies ere about dhz same causal hypothesis regatdkng a particuuar medical intervention). The poot inter-rater reliability is evem more striking considerlng the nqrrow range of rtudy designs and xubjecc mattwr from whigh the evidencd ix nenerated. 4 Inttr-Tokl Reliability The extent to whixh
0.5) = 0.4 Importantly, Beth and Sara 30 regarding a simple property (passing). that disagree most about students, and their is made stark because Beth and have a blunt evaluative tool (pass/fail grades rather than, say, letter grades). But finer-grained evaluative tool would not necessarily mitigate such disagreement, since there would be categories which could for each student; a finer-grained evaluative tool would increase, rather than decrease, the number of borderline (because there are borderline cases between each letter This example is meant illustrate that a κ measure 0.4 poor agreement two A score is fundamentally arbitrary measure of disagreement, and the significance of the disagreement that a particular κ score represents presumably with context. I nevertheless helps illustrate extent of disagreement empirical assessments of the inter-rater reliability I owe Jonah Schupbach thanks for noting that κ measure not only seem inappropriately low, as the above cases of poor inter-rater reliability, but seem inappropriately high as well. If a κ measure approaches 1, this might suggest agreement 'too good to be Returning to my example, Beth Sara a very a κ measure, then one might wonder if they colluded in grading. Thus when using a κ statistic to assess inter-rater should for a κ above some minimal threshold which too much disagreement) but maximum (above much What these thresholds should be beyond the scope of this (and are, I suppose, different users of the same QAT, when assessing same evidence, generate diverging assessments of the of that evidence. In most tests of the inter-rater reliability of QATs, evidence being from a narrow range of study designs (usually the studies are RCTs), the evidence is about a narrow range of subject (usually the studies about the same hypothesis regarding a medical intervention). The reliability is more considering of study designs and subject matter which evidence is generated. 4 Inter-Tool The which
 0.5) = 0.4 Importantly, Beth and Sara disAgree about 30 StudeNts RegArDing A relAtively simple pROperTy (passing). It is natural to SuppoSe THat tHEy DisagRee most ABoUT 'BorDeRlIne' StUDeNts, anD thEir disaGreement is MadE sTark because BETh And Sara havE a bLunt evaluatiVe tOol (pasS/fAil GRades RatHer thAn, say, lETter grAdes). But a fInER-grainED evaluaTIVe Tool Would not necessariLY mITigate such disaGreemeNt, SInCE TheRe wOuld be more CaTegorIEs about WHiCH THey COuld disagree fOr each studeNT; a fIner-grAiNed EValuatIve toOl WOulD increase, raTher Than decreAse, the NUmber of BOrderliNe caseS (beCauSe thERe ArE boRdERliNE cAseS BetWeen each LeTtEr graDe). ThIS EXAmplE is MeanT to ilLustrate that a Κ meAsurE Of 0.4 rEpresEnts pOor aGrEemenT betweEn two ReViewers.9 A κ score iS funDamentallY an ArBitRaRy meaSUre of dIsaGreEment, anD the sigNIfiCaNCE Of The disagreement thaT a PARtIcular κ sCore rePReSeNTs presumAbLy vArieS WIth coNtexT. thIs examplE, I neveRThElEss hope, HeLps to iLlUstRatE the eXTent Of disaGreement Found IN empirical asseSSments of the inTEr-RATeR ReliAbiLity of QATs. 9 I Owe JONah SChupBAcH thANks foR notiNg THaT A κ measure can not only SeEm inapProprIately low, as in The above caSES Of poor inTer-rATeR Reliability, but Can seEm inappropRIately hiGh as wEll. If a κ mEasure appROAches 1, thiS miGht SugGesT AGrEement which is 'TOO gooD tO be true'. retUrning tO my Toy ExaMplE, iF Beth and SAra had a vErY hIgH a Κ meAsure, THen one miGhT woNdEr iF they COlludeD in thEir gRaDiNG. ThUs when uSInG A Κ staTiStIc to AssEsS inteR-ratER reLiabiliTy, we shoulD hoPE for A κ MeAsure abOve some minimaL tHreshold (beLoW whIch indICAtes too mUch disagreement) but below SOme maxiMum ThresHold (Above whicH inDicateS toO Much agReemenT). What ExActLY These THReShoLdS should be aRE BeyOnd thE sCope Of this pAper (and are, I suppose, COntExt sensitive). 12 IN shOrt, dIFFeRenT UsERs oF tHE saME qAT, when assessinG the same evIdENcE, generate dIVerGiNg assesSments oF the sTRength oF that evidEnce. In mosT tEsts OF The Inter-rater ReliabilIty of QATs, THe eviDEnCe beiNg aSsesseD cOmeS from A narroW RanGe of sTudy deSiGns (usuAlly aLl The studiEs are RCTs), and the evidence Is abouT a narRow Range of suBjeCT maTter (usualLy alL the studieS arE abOut thE saME causAl hyPOtHesIS regaRdinG A particulAR mEdiCAL iNtervention). tHE PooR inteR-raTEr reliAbilIty is even more striKIng considering The nARRow RanGE of sTuDy designs and suBjeCt MATter from WhIch the evideNce is genErATed. 4 InTer-TooL ReliaBility THE ExTEnt to wHich
 0.5) = 0.4 Importantly, B eth and Sa ra di sag ree  a bout  30students regar d inga relatively simple pr opert y( pass i ng ). It  is nat u ra l tosu pp ose  t h at  they  di sagreemost about  'b or derline' stu d en ts, and th eir  disagreemen t i s made  s tar k  beca use  Beth  and S a ra hav e a blunt  e v aluati v e tool( p as s/fa il grades rathert ha n , say, lettergrades ). Bu t a f ine r-grainedev aluat i ve tool wo u l d  no t  necessarilymitigate su c h d isagre em ent ,  since  ther ew oul d be more c ateg ories abo ut whi c h theyc ould di sagree  fo r e achs tu de nt;  a fin e r- gra i ned  evaluat iv etoolwoul d i n crea se,  rat her t han decrease,  th e nu m ber  of b order line  c ases(becau se th er e are borderlin e ca ses betwe enea chle tterg rade).  Th isexample  is mea n t t oi l l us trate that a κ mea su r e  o f 0.4 re presen t spo o r agreem en t b etwe e n  tworevi e we rs.9 A κ  score is  f undamen ta lly an  a rbi tra ry me a sure  of di sagreeme nt, a n d the signific a nce of the di s ag r e em e nt t hat  a particul ar κ scor e re p re sen t s pre sumab ly va r ies with context. T hi s exam ple,I nevertheles s hope, he l p s  to illu stra t et he extent of d isagr eement fou n d in emp irica l assess ments oft h e inter- rat errel iab i l it y of QATs. 9I oweJo nah Sch upb ach tha nks  fo r n oti ng  that a κ  measure  c an  n ot  on ly se e m inappr op ria te lylow,a s in t he ab oveca se s  of  poor i n te r - rate rre liab ili ty , but  can see m inapp ropriatel y h i gh a swe ll. Ifa κ measure a pp roaches 1,  t his  might s uggest a greement which is 'toog ood tobetrue' . Re turning t o m y toyexa m ple, i f Beth  andSa rah a d a v e r yhig ha κ measur e ,  th en on emigh t wonde r if they colluded intheir grading . T husw h en  us i ng a κ  s t ati s t ic to assess in ter-raterre l ia bility, we sho ul d hopefor a κ  meas u re abov e some mi nimal thr es hold ( bel ow which i ndicates  too much disag r ee ment)  bu t belo wsom e max imum t h res hold(above  w hich i ndica te s too mu ch agreement). What exa ctly t hesethr esholds s hou l d b e are bey ondthe scopeofthi s pap er( and a re,I  s upp o se, c onte x t sensiti v e) . 1 2 In  short, dif f e r ent  user s o f  the s ameQAT, when assessi n g the same evi denc e ,  ge ner a te d iv erging assessm ent so f  the str en gth of that  evidenc e. In mo st tes ts ofthe int e r -r a ter re liab ili ty of QAT s,th e  eviden ce  b e ing as sess ed  comes  froma  nar r o w range of study  desi g n s (us u all y all  t he stud i es a re RCTs),and the evi denceis a bouta narro wrangeofsu bject matt e r (usuall y all  the st ud iesare  about  the s ame c ausa lhyp othesis r e g ar d in ga  pa rtic ularme dica l interve n tion). T hep oor int er -ra t e r reli a bi l i ty is even  mo re st r i king consi d erin g  t h e nar row ra nge of  studyd esi gn s and s ubj e c t matterfrom whic h  th eevid ence isge nera te d.4Inter-Tool Reli abilityThe exte n t  to wh ich 
 0.5)_= 0.4_Importantly, Beth and Sara_disagree about_30_students regarding_a_relatively simple property_(passing). It is_natural to suppose that_they disagree most_about_'borderline' students, and their disagreement is made stark because Beth and Sara have a_blunt_evaluative tool_(pass/fail_grades_rather than, say, letter grades)._But a finer-grained evaluative tool_would not_necessarily mitigate such disagreement, since there would be_more_categories about which_they could disagree for each student; a finer-grained evaluative_tool would increase, rather than decrease,_the number of_borderline_cases_(because there are borderline_cases between each letter grade). This_example is meant to illustrate that_a κ measure of 0.4 represents poor_agreement between two reviewers.9 A κ_score is fundamentally an arbitrary_measure of_disagreement, and the significance of_the disagreement that_a particular_κ score represents_presumably varies with context. This example,_I nevertheless hope,_helps to illustrate the extent of_disagreement_found in empirical_assessments_of_the inter-rater_reliability of QATs._9_I owe_Jonah_Schupbach thanks for noting that a_κ_measure can not only seem inappropriately low,_as in the above_cases_of poor inter-rater reliability,_but can seem inappropriately high_as well. If a κ measure_approaches 1,_this might_suggest agreement which is 'too good to be true'. Returning to_my toy example, if Beth and_Sara had a very_high a_κ_measure, then one_might_wonder if_they colluded in their grading. Thus when_using a_κ statistic to assess inter-rater reliability,_we should hope for_a_κ measure above some minimal threshold_(below which indicates too much disagreement)_but below some maximum threshold_(above_which_indicates too much agreement). What_exactly these thresholds should be are_beyond the scope_of this paper (and are, I suppose,_context_sensitive). 12 In short, different users_of_the same QAT, when assessing the_same_evidence,_generate diverging assessments of the_strength of that evidence. In most_tests of the inter-rater reliability of QATs, the evidence_being assessed comes_from a narrow range of_study_designs_(usually all the studies are RCTs), and the evidence is_about a_narrow range of_subject matter (usually all the studies are about the same_causal hypothesis regarding a particular medical intervention)._The poor inter-rater reliability is even more striking considering the narrow_range of study designs and subject matter from_which the evidence is generated. 4 Inter-Tool_Reliability The extent to_which