title: Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) Checklist Clinical validation of a non-heteronormative version of the Social Interaction Anxiety Scale (SIAS) The use of research questionnaires with hearing impaired adults: online vs. paper-and-pencil administration The feasibility and validity of ambulatory self-report of psychotic symptoms using a smartphone software application Development of a methodological PubMed search filter for finding studies on measurement properties of measurement instruments Comparison of Internet-Based and Paper-Based Questionnaires in Taiwan Using Multisample Invariance Approach Methodological Issues in Internet-Mediated Research: A Randomized Comparison of Internet Versus Mailed Questionnaires A Comparison of Psychometric Properties Between Internet and Paper Versions of Two Depression Instruments (BDI-II and MADRS-S) Administered to Clinic Patients
abstract: Background The COSMIN checklist is a tool for evaluating the method ological quality of studies on measurement properties of health-related patient-reported outcomes . The aim of this study is to determine the inter-rater agreement and reliability of each item score of the COSMIN checklist ( n = 114 ) . Methods 75 articles evaluating measurement properties were r and omly selected from the bibliographic data base compiled by the Patient-Reported Outcome Measurement Group , Oxford , UK . Raters were asked to assess the method ological quality of three articles , using the COSMIN checklist . In a one-way design , percentage agreement and intraclass kappa coefficients or quadratic-weighted kappa coefficients were calculated for each item . Results 88 raters participated . Of the 75 selected articles , 26 articles were rated by four to six participants , and 49 by two or three participants . Overall , percentage agreement was appropriate ( 68 % was above 80 % agreement ) , and the kappa coefficients for the COSMIN items were low ( 61 % was below 0.40 , 6 % was above 0.75 ) . Reasons for low inter-rater agreement were need for subjective judgement , and accustom to different st and ards , terminology and definitions . Conclusions Results indicated that raters often choose the same response option , but that it is difficult on item level to distinguish between articles . When using the COSMIN checklist in a systematic review , we recommend getting some training and experience , completing it by two independent raters , and reaching consensus on one final rating . Instructions for using the checklist are improved Introduction Despite welcomed changes in societal attitudes and practice s towards sexual minorities , instances of heteronormativity can still be found within healthcare and research . The  Social Interaction Anxiety Scale ( SIAS  ) is a valid and reliable self-rating scale of social anxiety , which includes one item ( number 14 ) with an explicit heteronormative assumption about the respondent´s sexual orientation . This heteronormative phrasing may confuse , insult or alienate sexual minority respondents . A clinical ly vali date d version of the SIAS featuring a non-heteronormative phrasing of item 14 is thus needed . Methods 129 participants with diagnosed social anxiety disorder , enrolled in an Internet-based intervention trial , were r and omly assigned to responding to the SIAS featuring either the original or a novel non-heteronormative phrasing of item 14 , and then answered the other item version . Within-subject , correlation between item versions was calculated and the two scores were statistically compared . The two items ’ correlations with the other SIAS items and other psychiatric rating scales were also statistically compared . Results Item versions were highly correlated and scores did not differ statistically . The two items ’ correlations with other measures did not differ statistically either . Conclusions The SIAS can be revised with a non-heteronormative formulation of item 14 with psychometric equivalence on item and scale level . Implication s for other psychiatric instruments with heteronormative phrasings are discussed Background When evaluating hearing rehabilitation , it is reasonable to use self-report question naires as outcome measure . Question naires used in audiological research are developed and vali date d for the paper- and -pencil format . As computer and Internet use is increasing , st and ardized question naires used in the audiological context should be evaluated to determine the viability of the online administration format . The aim of this study was to compare administration of question naires online versus paper- and pencil of four st and ardised question naires used in hearing research and clinic . We included the Hearing H and icap Inventory for the Elderly ( HHIE ) , the International Outcome Inventory for Hearing Aids ( IOI-HA ) , Satisfaction with Amplification in Daily Life ( SADL ) , and the Hospital Anxiety and Depression Scale ( HADS ) . Methods A cross-over design was used by r and omly letting the participants complete the question naires either online or on paper . After 3 weeks the participants filled out the same question naires again but in the other format . A total of 65 hearing-aid users were recruited from a hearing clinic to participate on a voluntary basis and of these 53 completed both versions of the question naires . Results A significant main effect of format was found on the HHIE ( p < 0.001 ) , with participants reporting higher scores on the online format than in the paper format . There was no interaction effect . For the other question naires were no significant main or interaction effects of format . Significant correlations between the two ways of presenting the measures was found for all question naires ( p<0.05 ) . The results from reliability tests showed Cronbachs α ’s above .70 for all four question naires and differences in Cronbachs α between administration formats were negligible . Conclusions For three of the four included question naires the participants ’ scores remained consistent across administrations and formats . For the fourth included question naire ( HHIE ) a significant difference of format with a small effect size was found . The relevance of the difference in scores between the formats depends on which context the question naire is used in . On balance , it is recommended that the administration format remain stable across assessment points Background Semi-structured interview scales for psychosis are the gold st and ard approach to assessing psychotic and other symptoms . However , such assessment s have limitations such as recall bias , averaging , insensitivity to change and variable interrater reliability . Ambulant , real-time self-report assessment devices may hold advantages over interview measures , but it needs to be shown that the data thus collected are valid , and the collection method is acceptable , feasible and safe . We report on a monitoring system for the assessment of psychosis using smartphone technology . The primary aims were to : i ) assess validity through correlations of item responses with those on widely accepted interview assessment s of psychosis , and ii ) examine compliance to the procedure in individuals with psychosis of varying severity . Methods A total of 44 participants ( acute or remitted DSM-4 schizophrenia and related disorders , and prodromal ) completed 14 branching self-report items concerning key psychotic symptoms on a touch-screen mobile phone when prompted by an alarm at six pseudo-r and om times , each day , for one week . Face to face PANSS and CDS interviews were conducted before and after the assessment period blind to the ambulant data . Results Compliance as defined by completion of at least 33 % of all possible data -points over seven days was 82 % . In the 36 compliant participants , 5 items ( delusions , hallucinations , suspiciousness , anxiety , hopelessness ) showed moderate to strong ( rho 0.6 - 0.8 ) associations with corresponding items from interview rating scales . Four items showed no significant correlation with rating scales : each was an item based on observable behaviour . Ambulant ratings showed excellent test-retest reliability and sensitivity to change . Conclusions Ambulatory monitoring of symptoms several times daily using smartphone software applications represents a feasible and valid way of assessing psychotic phenomena for research and clinical management purpose s. Further evaluation required over longer assessment periods , in clinical trials and service setting Objectives For the measurement of  patient-reported outcomes , such as ( health-related ) quality of life , often many measurement instruments exist that intend to measure the same construct . To facilitate instrument selection , our aim was to develop a highly sensitive search filter for finding studies on measurement properties of measurement instruments in PubMed and a more precise search filter that needs less abstract s to be screened , but at a higher risk of missing relevant studies . Methods A r and om sample of 10,000 PubMed records ( 01 - 01 - 1990 to 31 - 12 - 2006 ) was used as a gold st and ard . Studies on measurement properties were identified using an exclusion filter and h and search ing . Search terms were selected from the relevant records in the gold st and ard as well as from 100 systematic review s of measurement properties and combined based on sensitivity and precision . The performance of the filters was tested in the gold st and ard as well as in two validation sets , by calculating sensitivity , precision , specificity , and number needed to read . Results We identified 116 studies on measurement properties in the gold st and ard . The sensitive search filter was able to retrieve 113 of these 116 studies ( sensitivity 97.4 % , precision 4.4 % ) . The precise search filter had a sensitivity of 93.1 % and a precision of 9.4 % . Both filters performed very well in the validation sets . Conclusion The use of these search filters will contribute to evidence -based selection of measurement instruments in all medical fields This study examines whether the Internet-based question naire is psychometrically equivalent to the paper-based question naire . A r and om sample of 2,400 teachers in Taiwan was divided into experimental and control groups . The experimental group was invited to complete the electronic form of the Chinese version of Center for Epidemiologic Studies Depression Scale ( CES-D ) placed on the Internet , whereas the control group was invited to complete the paper-based CES-D , which they received by mail . The multi sample invariance approach , derived from structural equation modeling ( SEM ) , was applied to analyze the collected data . The analytical results show that the two groups have equivalent factor structures in the CES-D. That is , the items in CES-D function equivalently in the two groups . Then the e quality of latent mean test was performed . The latent means of " depressed mood , " " positive affect , " and " interpersonal problems " in CES-D are not significantly different between these two groups . However , the difference in the " somatic symptoms " latent means between these two groups is statistically significant at alpha = 0.01 . But the Cohen 's d statistics indicates that such differences in latent means do not apparently lead to a meaningful effect size in practice . Both CES-D question naires exhibit equal validity , reliability , and factor structures and exhibit a little difference in latent means . Therefore , the Internet-based question naire represents a promising alternative to the paper-based question naire Background The majority of Internet-mediated studies use measures developed as paper- and -pencil measures or face-to-face-delivered material . Previous research suggests that the equivalence between online and offline measures must be demonstrated rather than assumed . Objective The objective of this study was to explore the equivalence 4 measures completed in an online or offline setting . Methods A sample of students ( n = 1969 ) was r and omly assigned to complete 4 popular scales ( the SF-12v2 , the Hospital Anxiety and Depression Scale ( HADS ) , the Fatigue Symptom Inventory , and a single-item fatigue measure ) either online or by mail survey ( pencil and paper ) . The response rate was 52.51 % ( n = 1034 ) and comparable between the online and offline groups . Results Significant differences were noted in fatigue levels between the online and offline group ( P = .01 ) as measured by the Fatigue Symptom Inventory , with the online sample demonstrating higher levels of fatigue . Equivalency was noted for the SF-12v2 , the Hospital Anxiety and Depression Scale , and the single-item fatigue measure . Internal consistency was high except for the SF-12v2 . The SF-12v2 may not be an ideal measure to use for remote administration . Conclusions Equivalency of the Hospital Anxiety and Depression Scale ( HADS ) and the Physical Component Score and Mental Component Score of the SF-12v2 for online and offline data were demonstrated . Equivalency was not demonstrated for the Fatigue Symptom Inventory . Explanations for the difference in fatigue score between the online and offline sample s are unclear . Research that seeks to match sample s and control for extraneous online and offline variables is called for , along with exploration of factors that may mediate the completion of question naires or alter the respondents ’ relationship with the same , to enhance progress in this area Background Self-report measures can guide clinical decisions and are useful when evaluating treatment outcomes . However , many clinicians do not use self-report measures systematic ally in their clinical practice . Internet-based question naires could facilitate administration , but the psychometric properties of the online version of an instrument should be explored before implementation . The recommendation from the International Test Commission is to test the psychometric properties of each question naire separately . Objective Our objective was to compare the psychometric properties of paper- and -pencil versions and Internet versions of two question naires measuring depressive symptoms . Methods The 87 participating patients were recruited from primary care and psychiatric care within the public health care system in Sweden . Participants completed the  Beck Depression Inventory ( BDI-II )  and the  Montgomery-Åsberg Depression Rating Scale — Self-rated ( MADRS-S )  , both on paper and on the Internet . The order was r and omized to control for order effects . Symptom severity in the sample ranged from mild to severe depressive symptoms . Results Psychometric properties of the two administration formats were mostly equivalent . The internal consistency was similar for the Internet and paper versions , and significant correlations were found between the formats for both MADRS-S ( r = .84 ) and the BDI-II ( r = .89 ) . Differences between paper and Internet total scores were not statistically significant for either question naire nor for the MADRS-S question dealing with suicidality ( item 9 ) when analyzed separately . The score on the BDI-II question about suicidality ( item 9 ) was significantly lower when administered via the Internet compared with the paper score , but the difference was small ( effect size , Cohen ’s [ d ] = 0.14 ) . There were significant main effects for order of administration on both question naires and significant interaction effects between format and order . This should not , however , pose a problem in clinical use as long as the administration format is not changed when repeated measurements are made . Conclusions The MADRS-S can be transferred to online use without affecting the psychometric properties in a clinical ly meaningful way . The full BDI-II also seems to retain its properties when transferred ; however , the item measuring suicidality in the Internet version needs further investigation since it was associated with a lower score in this study . The use of online question naires offers clinicians a more practical way of measuring depressive symptoms and has the potential to save re sources
target: For common instruments such as the CES-D , MADRS-S and HADS there is mounting evidence for adequate psychometric properties .
background: Background Online question naires for measuring common mental health disorders such as depression and anxiety disorders are increasingly used .
  The psychometrics of several pen- and -paper question naires have been re-examined for online use and new online instruments have been developed and tested for validity as well .
  This study aims to review and synthesis e the literature on this subject and provide a framework for future research .