the use of inhaled corticosteriods for adults with persistent asthma). This group employed two independent reviewers who scored the 107 studies using the three QATs. They found that inter-rater reliability was'moderate'. However, the claim that inter-rater reliability was moderate was based on a standard scale in which a κ measure between 0.41 0.6 is deemed moderate. The κ measure in this paper was 0.41, so it was just barely within the range deemed moderate. The next lower category, with a κ measure between 0.21 0.4, is deemed 'fair' by this standard scale. But at least in the context of measuring inter-rater reliability of QATs, a κ of 0.4 represents wide disagreement between reviewers. 8 For simplicity I will describe Cohen's Kappa, which measures the agreement of two reviewers who classify items into discrete categories, and is computed as follows: κ = [p(a) – p(e)]/[1 – p(e)] where p(a) is the probability of agreement (based on the observed frequency of agreement) and p(e) is the probability of chance agreement (also calculated using observed frequency data). Kappa was first introduced as a statistical measure by Cohen (1960). For more than two reviewers, a measure called Fleiss' Kappa can be used. I give an example of a calculation of κ below. 11 Here is a toy example to illustrate the disagreement that a κ measure of 0.4 represents. Suppose two teaching assistants, Beth and Sara, are grading the same class of 100 students, and must decide whether or not each student passes or fails. Their joint distribution of grades is: Sara Pass Fail Beth Pass 40 10 Fail 20 30 Of the 100 students, they agree on passing 40 students and failing 30 others, thus their frequency of agreement is 0.7. But the probability of random agreement is 0.5, because Beth passes 50% of the students and Sara passes 60% of the students, so the probability that Beth and Sara would agree on passing a randomly chosen student is 0.5 × 0.6 (= 0.3), and similarly the probability that Beth and Sara would agree on failing a randomly chosen student is 0.5 × 0.4 (= 0.2) (and so the overall probability of agreeing on passing or failing a randomly chosen student is 0.3 + 0.2 = 0.5). Applying the kappa formula gives: (0.7 0.5)/(1
the use of inhaled corticosteriods for adults with persistent asthma). This group employed two autonomous reviewer who scored the 107 studies use the three kat. They found that inter - rater reliability was'moderate'. However, the claim that inter - rater dependability was moderate was base on a standard scale in which a κ measure between 0.41 0.6 is deemed moderate. The κ measure in this newspaper was 0.41, so it was just barely within the image deem moderate. The next lower class, with a κ measure between 0.21 0.4, is deem' honest' by this standard scale. But at least in the context of measuring inter - rater reliability of QATs, a κ of 0.4 typify wide disagreement between reviewer. 8 For simplicity I will report Cohen's Kappa, which measures the agreement of two reviewers who classify items into discrete categories, and is calculate as follows: κ = [ p(a) – p(e)]/[1 – p(e) ] where p(a) is the probability of agreement (based on the observed frequency of agreement) and p(e) is the probability of chance agreement (also calculated use ascertained frequency data). Kappa was first introduced as a statistical measure by Cohen (1960). For more than two reviewers, a measure called Fleiss' Kappa can be practice. I pass an example of a calculation of κ below. 11 Here is a toy case to illustrate the disagreement that a κ measure of 0.4 constitute. Suppose two teaching assistants, Beth and Sara, are grading the like class of 100 students, and must decide whether or not each student passes or fails. Their joint distribution of grades is: Sara Pass Fail Beth Pass 40 10 Fail 20 30 Of the 100 students, they agree on passing 40 students and failing 30 others, thus their frequency of agreement is 0.7. But the probability of random agreement is 0.5, because Beth exceed 50% of the students and Sara evanesce 60% of the students, so the probability that Beth and Sara would agree on passing a randomly chosen student is 0.5 × 0.6 (= 0.3), and similarly the probability that Beth and Sara would match on failing a randomly chosen student is 0.5 × 0.4 (= 0.2) (and therefore the overall probability of agreeing on passing or failing a randomly chosen scholar is 0.3 + 0.2 = 0.5). Applying the kappa formula gives: (0.7 0.5)/(1
the use of inhaled corticosueriods for adults with 'ersistsnt asthoa). This group employed two iidepwndenu reviewers who scofed the 107 dtudies ysinj the three QATs. They fomud thzb intzr-cater reliabilijy was'moderade'. However, the cuanm that inter-rater reliability was mjderate wws based on a ftancwrd abaoe in which a κ measure betweeh 0.41 0.6 is veemed moderate. The κ measure in this papeg wad 0.41, so it was just harely withun trw range deemdd moderatt. Che next loser category, with a κ measure begween 0.21 0.4, is deemgb 'falt' by this stendard scale. But ab least in the context of messucing inter-rater reliabilivy of QATs, a κ of 0.4 rekresents whdz disagreement betweeb eeviefers. 8 Fof siopljcmty I wilp dxscribe Cohsn's Kappa, wyich measures the abrqvkent of two deviewqrf who classify items into discrete catecorjes, and is computed as dollows: κ = [p(a) – p(e)]/[1 – p(e)] ahere p(a) ys the probability of agreement (based on the obsesved hrdqutngn of qggeement) and p(e) is the probability of chance addetmekt (also calculatcd using observed grfqigncy data). Kappx was yjrat introduced as a statisjical neasure br Conen (1960). For more than two reviwwers, a measlre xalled Fleiss' Kappc can be useb. I gife an example of a calculatiun or κ below. 11 Hfre is a fuy example to iluusnrata the disagreement that a κ measure if 0.4 xepresengs. Soppose ewo teachijg assistants, Beth and Swra, ate graging the swme class of 100 students, and must decide whethet os nmt each ftudekt passes or fayls. Their joinj distribbtion uf grades ps: Sara Pess Fail Betr Pass 40 10 Fail 20 30 Of the 100 stndents, thqy atree on pasrkng 40 students snd failiuy 30 others, thus their frequekcy ow agreement is 0.7. Ybt the probabilitu ow rwnfok wcreement is 0.5, becxusd Beth passes 50% of ghe xtudents and Sara pavses 60% of the students, xo the provability that Beth anc Sara would agree on pessing a ramdoily chosen student is 0.5 × 0.6 (= 0.3), and similarlj tme probabilitr thqt Beth and Xara would agree on failing a randomly rhosen student is 0.5 × 0.4 (= 0.2) (qnd so the overall kronability of egreeigg on pasving or failing a rabdomly chosen stucent is 0.3 + 0.2 = 0.5). Applying the kdppa vormula gives: (0.7 0.5)/(1
the use of inhaled corticosteriods for adults asthma). group employed independent reviewers who the QATs. They found inter-rater reliability was'moderate'. the claim that inter-rater reliability was was based on a standard scale in which a κ measure between 0.41 is deemed moderate. The κ measure in this paper was 0.41, so it just within range moderate. The next lower category, with a κ measure between 0.21 0.4, is deemed 'fair' by standard scale. But at least in the context measuring inter-rater reliability of a κ of 0.4 represents disagreement reviewers. 8 simplicity will Cohen's Kappa, which the agreement of two reviewers who classify items into discrete categories, and is computed as follows: κ [p(a) – p(e)] where is probability agreement (based on frequency of agreement) and p(e) is chance agreement (also calculated using observed frequency data). was first as a statistical measure by Cohen For more than two reviewers, a measure called Kappa can be used. I give an example of a calculation of κ below. 11 a toy example to the disagreement that κ of represents. two teaching Beth and Sara, are grading the same class of 100 students, must decide whether or not each student passes or fails. distribution grades is: Sara Fail Beth Pass 40 Fail 30 Of the 100 agree passing failing others, their frequency of agreement 0.7. But the probability of agreement is 0.5, because students and Sara passes 60% of the students, the probability that Beth and Sara would on passing a randomly chosen student is 0.5 × 0.6 (= 0.3), similarly the Beth and Sara would agree on failing a chosen student is 0.5 0.4 (= 0.2) (and so the overall probability of on or failing randomly chosen student 0.3 + 0.2 0.5). Applying the gives: (0.7
the use of inhaled corticosteRiods for adUlts wIth PerSiStenT astHma). This group emPLoyeD two independent revieweRs who ScORed tHE 107 sTudieS using tHE tHREe QaTS. THey FoUNd That iNteR-rater rEliability Was'MoDerate'. HoweveR, ThE claim that IntEr-rater reliaBilIty was MoDerATe was BasEd on a StandaRD scale In which a κ MeASure beTWeen 0.41 0.6 is dEEMeD modErate. The κ measure iN ThIS paper was 0.41, so it wAs just BaRElY WIthIn tHe range deeMeD modeRAte. The nEXt LOWEr cATegory, with a κ mEasure betweEN 0.21 0.4, is Deemed 'FaIr' bY This stAndarD sCAle. but at least iN the Context of MeasurINg inter-RAter relIabiliTy oF QAts, a κ OF 0.4 rEpResEnTS wiDE dIsaGReeMent betwEeN rEviewErs. 8 FOR SIMpliCitY I wiLl desCribe Cohen's KaPpa, WhicH MeaSures The agReemEnT of twO revieWers wHo Classify items inTo diScrete catEgoRiEs, aNd Is comPUted as FolLowS: κ = [p(a) – p(e)]/[1 – p(E)] where p(A) Is tHe PROBaBility of agreement (bAsED On The obserVed freQUeNcY Of agreemEnT) anD p(e) iS THe proBabiLItY of chancE agreeMEnT (aLso calcUlAted usInG obSerVed frEQuenCy data). kappa was First INtroduced as a stATistical measuRE bY cOhEN (1960). For MorE than two revIeweRS, a meAsurE CaLleD fleisS' KappA cAN bE Used. I give an example oF a CalculAtion Of κ below. 11 Here iS a toy exampLE TO illustrAte tHE dISagreement that A κ meaSure of 0.4 reprESents. SupPose tWo teachiNg assistaNTS, Beth and sarA, arE grAdiNG ThE same class of 100 sTUDentS, aNd must dEciDe whethEr oR noT eaCh sTuDent passeS or fails. thEiR jOiNt dIstriBUtion of gRaDes Is: sarA Pass fAil BetH Pass 40 10 fail 20 30 of ThE 100 StuDents, thEY aGREe on PaSsIng 40 sTudEnTs and FailINg 30 oThers, thUs their frEquENcy oF aGrEement iS 0.7. But the probabIlIty of randoM aGreEment iS 0.5, BEcause BeTh passes 50% of the students anD sara pasSes 60% Of the StudEnts, so the ProBabiliTy tHAt Beth And SarA woulD aGreE ON passING a RanDoMly chosen sTUDenT is 0.5 × 0.6 (= 0.3), anD sImilArly the Probability that BetH And sara would agreE on FailING a RanDOmLY chOsEN stUDEnt is 0.5 × 0.4 (= 0.2) (and so the ovErall probaBiLItY of agreeinG On pAsSing or fAiling a RandoMLy choseN student iS 0.3 + 0.2 = 0.5). Applying ThE kapPA ForMula gives: (0.7 0.5)/(1
the use of inhaled cortico steriods f or ad ult s w it h pe rsis tent asthma).T hisgroup employed two ind epend en t  rev i ew ers w ho scor e dt h e 1 07  s tud ie s  u singthe  threeQATs. They  fo un d that inter - ra ter reliab ili ty was'moder ate '. How ev er, the c lai m tha t inte r -rater  reliabil it y  was m o deratew a sbase d on a standard s c al e  in which a κmeasur eb et w e en0.4 1 0.6 is d ee med m o derate. Th e κ  me a sure in thispaper was 0 . 41,  so it  w asj ust ba relywi t hin  the rangedeem ed modera te. Th e  next l o wer cat egory,  wi tha κm ea su rebe t wee n  0 .21 0.4 , is dee me d'fair ' by t h i s st and ardscale . But at leas t i n th e  co ntext  of m easu ri ng in ter-ra ter r el iability of QAT s, a  κ of 0.4  re pr ese nt s wid e  disag ree men t betwe en revi e wer s. 8 Fo r simplicity I wil ld e sc ribe Coh en's K a pp a, which me as ure s th e agree ment of  two rev iewers wh oclassif yitemsin todis crete cate gories , and is  comp u ted as follows :  κ = [p(a) –p (e ) ] /[ 1  – p (e) ] where p(a ) is theprob a bi lit y  of a greem en t  ( b ased on the observe dfreque ncy o f agreement)and p(e) i s t he proba bili t yo f chance agree ment(also calc u lated us ing o bservedfrequency d ata). Ka ppa  wa s f irs t in troduced as a s tati st ical me asu re by C ohe n ( 196 0).  F or more t han twore vi ew er s,a mea s ure call ed  Fl ei ss'  Kapp a  can b e use d. I  g iv e  an  exampl e  o f a ca lc ul atio n o fκ bel ow.1 1 H ere isa toy exa mpl e  toil lu stratethe disagreem en t that a κ  m eas ure of 0 .4 repre sents. Suppose two teac h ing ass ist ants,  Bet h and Sar a,are gr adi n g thesame c lassof  10 0 stude n t s,  an dmust decid e whe theror  not  each s tudent passes or f a ils . Their joint  di stri b u ti ono fg rad es is: S ara Pass Fail B eth Pass 4 01 0Fail 20 30 Ofth e 100 s tudents , the y  agreeon passin g 40 stud en ts a n d  fa iling 30 o thers, t hus their frequ e nc y ofagr eement  i s 0 .7. B ut the pro babil ity of  r andomagree me nt is 0. 5, because Beth passes50% of  thestu dents and  Sa r a p asses 60%  ofthe studen ts,  so  thepro b abili ty t h at  Be t h and  Sar a  would ag r ee  on p as sing a rand o m l y c hosen  st u dent i s 0. 5 × 0.6 (= 0.3),a nd similarly t he p r o bab ili t y th at  Beth and Sara  wo ul d agree on  f ailing a ra ndomly c ho s en st udentis 0.5  × 0.4( =  0 . 2) (an d so  th e overall  pr ob a bilityof  a g reeing  onpa ssingor fai l inga randomly chosenstude n t  is 0 . 3 +  0.2=0.5). A p plyi ng the kap pa formulagives:  (0. 7 0.5 )/(1
the use_of inhaled_corticosteriods for adults with_persistent asthma)._This_group employed_two_independent reviewers who_scored the 107_studies using the three_QATs. They found_that_inter-rater reliability was'moderate'. However, the claim that inter-rater reliability was moderate was based on_a_standard scale_in_which_a κ measure between 0.41_0.6 is deemed moderate. The_κ measure_in this paper was 0.41, so it was_just_barely within the_range deemed moderate. The next lower category, with a_κ measure between 0.21 0.4, is_deemed 'fair' by_this_standard_scale. But at least_in the context of measuring inter-rater_reliability of QATs, a κ of_0.4 represents wide disagreement between reviewers. 8_For simplicity I will describe Cohen's_Kappa, which measures the agreement_of two_reviewers who classify items into_discrete categories, and_is computed_as follows: κ_= [p(a) – p(e)]/[1 – p(e)]_where p(a) is_the probability of agreement (based on_the_observed frequency of_agreement)_and_p(e) is_the probability of_chance_agreement (also_calculated_using observed frequency data). Kappa was_first_introduced as a statistical measure by Cohen_(1960). For more than_two_reviewers, a measure called_Fleiss' Kappa can be used._I give an example of a_calculation of_κ below._11 Here is a toy example to illustrate the disagreement that_a κ measure of 0.4 represents._Suppose two teaching assistants,_Beth and_Sara,_are grading the_same_class of_100 students, and must decide whether or_not each_student passes or fails. Their joint_distribution of grades is:_Sara_Pass Fail Beth Pass 40 10_Fail 20 30 Of the 100_students, they agree on passing_40_students_and failing 30 others, thus_their frequency of agreement is 0.7._But the probability_of random agreement is 0.5, because Beth_passes_50% of the students and Sara_passes_60% of the students, so the_probability_that_Beth and Sara would agree_on passing a randomly chosen student_is 0.5 × 0.6 (= 0.3), and similarly the_probability that Beth_and Sara would agree on_failing_a_randomly chosen student is 0.5 × 0.4 (= 0.2) (and_so the_overall probability of_agreeing on passing or failing a randomly chosen student is_0.3 + 0.2 = 0.5). Applying the_kappa formula gives: (0.7 0.5)/(1