Developing core outcome sets for clinical trials: issues to consider

Paula R Williamson
Douglas G Altman
Jane M Blazeby
Mike Clarke
Declan Devane
Elizabeth Gargon
Peter Tugwell
Developing core outcome sets for clinical trials: issues to consider
C O M M E N T A R Y Open AccessCore outcome setOutcome reporting biasClinical trialsSystematic reviewMethodologyConsensus
The selection of appropriate outcomes or domains is crucial when designing clinical trials in order to compare directly the effects of different interventions in ways that minimize bias. If the findings are to influence policy and practice then the chosen outcomes need to be relevant and important to key stakeholders including patients and the public, health care professionals and others making decisions about health care. There is a growing recognition that insufficient attention has been paid to the outcomes measured in clinical trials. These issues could be addressed through the development and use of an agreed standardized collection of outcomes, known as a core outcome set, which should be measured and reported, as a minimum, in all trials for a specific clinical area. Accumulating work in this area has identified the need for general guidance on the development of core outcome sets. Key issues to consider in the development of a core outcome set include its scope, the stakeholder groups to involve, choice of consensus method and the achievement of a consensus.ABSTRACTStructured summary2 Provide a structured summary including, as applicable: background, objectives, data sources, participant eligibility criteria, study methods, results, limitations, conclusions, and implications of key findings.

Background

Clinical trials seek to evaluate whether an intervention is effective. This is determined by comparing outcomes that are chosen to reflect beneficial and harmful effects. Outcomes may be specific (for example, progression of loss of eyesight in patients with retinal degeneration) or they may be a broad construct used to demonstrate effects of an intervention on one aspect of health (for example, pain). Selection of appropriate outcomes or domains is crucial when designing clinical trials to compare directly the effects of different interventions in ways that minimize bias. If the findings are to influence policy and practice then the chosen outcomes need to be relevant and important to key stakeholders including patients and the public, health care professionals and others making decisions about health care.

There is a growing recognition that insufficient attention has been paid to the outcomes measured in clinical trials. Difficulties caused by heterogeneity in outcome measurement are well-known to systematic reviewers. For example, the most accessed and the top cited Cochrane Reviews in 2009 [1] all describe problems due to inconsistencies in the outcomes reported in trials. Furthermore, empirical research provides strong evidence that outcome reporting bias, defined as the results-based selection for publication of a subset of the original measured outcome variables, is an important problem in randomized trials [2] that affects the conclusions in a substantial proportion of Cochrane reviews [3]. Outcome reporting bias is likely to affect systematic reviews more widely, as well as published research in general.

These issues could be addressed through the development and use of an agreed standardized collection of outcomes, known as a core outcome set (COS), which should be measured and reported in all trials for a specific clinical area [4]. These sets do not imply that outcomes in a particular trial should be restricted to those in the COS. Rather, there is an expectation that the core outcomes will always be collected and reported, and that researchers will continue to explore other outcomes. In most trials, the primary outcome would be expected to be one of those contained in the COS. If a COS is not implemented in a particular trial, the researchers should explain their decision in the trial protocol and subsequent report. Similarly, if the primary outcome for a particular trial is not within the COS, then the relevance and importance of the chosen outcome should be explained. This approach would enhance the value of evidence synthesis by reducing heterogeneity in reported outcomes between trials and reducing the risk of outcome reporting bias, since trial reports will always include presentation of their findings for the core outcomes, as a minimum. Statistical power would be increased because fewer studies would have to be omitted from meta-analyses. The GRADE (Grading of Recommendations Assessment, Development and Evaluation) group (http://www.gradeworkinggroup.org) recognize the need to identify a relevant set of core outcomes, and recommends that up to seven patient-important outcomes are listed in the 'Summary of Findings' tables in systematic reviews. This is supported by Cochrane Reviews of the effects of healthcare interventions [5,6] and by the World Health Organization (WHO) in developing guideline recommendations [7].

An important rationale for COS is that outcomes reported for trials may not reflect endpoints that are meaningful for health service users. Examples exist where patients identified an outcome important to them as a group that might not have been considered by practitioners on their own [8][9][10][11][12]. Recognition of the importance of incorporating health service user opinion in COS development is increasing, but involvement has been limited to date.

The most notable work relating to outcome standardization has been conducted by the OMERACT (Outcome Measures for Rheumatology Clinical Trials) collaboration since 1992, which advocates the use of COS in clinical trials in rheumatology [13]. More than 50 other groups have been working on COS in specific areas of health care, including pain [14], maternity care [15] and some pediatric specialties [16].

The COMET (Core Outcome Measures in Effectiveness Trials) Initiative [17] brings together researchers interested in the development, application and promotion of COS, derived using rigorous consensus methods, for effectiveness trials. COMET aims to collate and stimulate the development of relevant resources, both applied and methodological, to facilitate exchange of ideas and information, to work with patients, the public and their representatives to develop material to improve health service user engagement, and to foster methodological research in the area of COS. Data on relevant individual studies, both published and ongoing, are being included in a free, publically available internet-based resource. This is a unique resource, which is updated periodically, and which should serve to minimize duplication of effort in the development of COS.

As each COS is developed, it will be important to agree on an appropriate instrument or definition for each included outcome. Examples of lack of validated instruments and standardized definitions abound in the literature. For example, fewer than 20% of 906 different outcomes measured in breast reconstruction surgery trials were defined or measured with a validated tool [18,19].

Accumulating work in this area has identified the need for general guidance on the development of COS. We identify here key issues to consider and methodological decisions to be made, referencing illustrative examples. Study protocols, in which key decisions are documented regarding the choices made in the process of COS development, are emerging, for example in the MOMENT study (The management of otitis media with effusion in children with cleft palate: a feasibility study http://www. hta.ac.uk/project/2555.asp) and the development of COS for surgery for colorectal and esophageal cancer, morbid obesity and breast reconstruction.

Main text

Suggested approach for development of a core outcome set Scope

The specific area of health or healthcare to which the COS is to apply needs to be described, with details of health condition, population and types of interventions. The COS may be developed to encompass all stages or severity of a health condition or it may be focused on a particular disease category. For example, in colorectal cancer, a COS may be developed for all patients or it may focus on patients with metastatic disease. Similarly, the core set may be developed for all treatment types or for a particular intervention (for example, COS may be created for use in all trials of interventions to treat morbid obesity or for bariatric surgery alone).

Identifying existing knowledge

One of the difficulties in this area of research is how to identify studies that have already been done (or are underway) to develop COS. As part of the COMET Initiative, a searchable database has been developed [17]. This enables researchers to check for existing or ongoing work before embarking on a new project, thus minimizing duplication of effort.

A review of previous trials [20] or systematic reviews in the area can provide evidence of need for a COS and also identify a potential list of outcomes. Systematic reviewers are starting to use the outcome matrix recommended by the ORBIT project [3] to display the outcomes reported in the eligible studies. This matrix may demonstrate the inconsistency of outcomes measured to date in addition to potential outcome reporting bias. A review of studies other than clinical trials (for example, observational research into harms) may also identify additional outcomes, such as rare endpoints, that would be worthy of consideration for inclusion in the COS.

Stakeholder involvement

Key stakeholders may include patients and the public, health care practitioners, regulators, industry representatives, and researchers. Bringing diverse stakeholders together to try to reach a consensus is increasingly wellaccepted as the future of collaborative, influential research. An important example of this is the work of the James Lind Alliance in determining important questions about treatments where uncertainty remains [21]. Decisions regarding the stakeholder groups to be involved and the target number from each group will be dependent upon the particular scope of the COS as well as upon existing knowledge and practical feasibility considerations. These decisions should be documented and explained in the study protocol. Consideration should be given to potential conflicts of interest within the group developing the COS (for example, the developers of measurement instruments in the area of interest or those whose work is focused on a specific outcome) [22].

Few COS studies to date have involved patients or the public, yet those that have done so have identified outcomes that were not previously identified by the other stakeholders [4,5,[23][24][25][26]. Methods for identifying patients or their representatives include clinics, patient societies, advocacy groups, and care giver support groups. To achieve representativeness, it may be helpful to consider approaching practitioners with the support of professional bodies.

Consensus methods

The first step in the process is typically to develop consensus about 'what' to measure. The 'how' and 'when' to measure are usually later in the process but may be determined by consensus methods as well. Groups that first establish consensus about what concepts to measure may subsequently conclude that there is a single measurement instrument for an outcome in the COS that is supported by sufficient evidence to recommend its use. However they may identify that gaps in outcome measurement exist, either because there is no ideal instrument for a particular concept or because the evidence base for existing instruments is of limited quality. These limitations may make it difficult to identify which of several potential measures may be preferable for use in the relevant context. The COSMIN (COnsensusbased Standards for the selection of health Measurement Instruments, www.cosmin.nl) checklist [27] can be used as a tool for developing studies of the validity and reliability of measurement instruments because it describes the necessary design requirements for the assessment of those measurement properties. In addition, feasibility of measurement is a further consideration [13].

Methods used in previous studies to elicit opinions and to develop consensus about important outcomes include expert panel meetings [28], Delphi surveys [22], Nominal Group Techniques [13], focus groups [24], individual interviews [25] and individual questionnaires [5]. Considerations concerning the choice of method include the need to build a true consensus with methodological rigor, strategies to ensure that a diverse range of opinions are heard, and factors such as financial and carbon costs that might limit the practicality of face-to-face meetings. It is important to ensure that views from all key stakeholder groups are included when making the final decision regarding the COS. This can be achieved through anonymous voting facilitated through email or keypads at meetings.

It is necessary to decide what information about possible outcomes should be given to stakeholder participants before a consensus exercise begins. A literature review of relevant studies showing the outcomes clinicians value and report most frequently, which may include information about methods to measure the identified outcomes and when to measure them, has been proposed [18][19][20]. This should be combined with outcomes deemed to be important to health service users if such work has been undertaken previously. If consensus participants are shown a list of potential outcomes, it is generally recommended that they be given the opportunity to propose the inclusion of additional items, especially as the literature may not include outcomes associated with the most recent treatments available. Because this has the potential to result in a long list of items, criteria for determining inclusion of items to be considered in the consensus exercise may be needed. For example, Devane et al. [15] required new items to have been suggested by at least two participants. If it is felt that the sharing of a list of outcomes at the outset of the consensus process may bias responses, open questioning may be preferred. Techniques to do this include administering questionnaires [5], focus groups [24] and in-depth interviews [25], to determine outcomes important to patients. However this may lead to stakeholders not considering areas previously deemed important, and subsequent questions to prompt consideration of specific outcomes may be warranted.

Researchers should consider the potential impact of the following methodological decisions on the final results: group composition, questioning technique, the information participants receive to inform their answers, whether or not responses are anonymous, how the group participants interacted with or influenced each other, the medium of the interaction, attrition bias, analysis which can miss or overstate the importance of certain outcomes, and the way in which consensus is reached. A single heterogeneous consensus panel comprising the various stakeholders may be deemed appropriate for particular areas of healthcare whereas separate panels for different stakeholder groups followed by work to integrate the multiple perspectives may be more appropriate for others. A variety of methods have been used to date in published and ongoing studies, particularly related to the inclusion of the patient perspective. The proportion of patients and health service users chosen in a Delphi survey may depend upon the clinical setting; for example, for breast reconstructive surgery, which is an optional procedure undertaken for cosmetic purposes, the involvement of patients is more critical than in other settings. A research project has recently started within the COMET Initiative to assess the effectiveness of these different methods.

Consideration should be given in advance to the criteria that will be used to determine when consensus has been achieved. A review of the reporting of Delphi studies to develop COS demonstrated poor reporting of the methods used [29]. As an example definition, in a Delphi survey participants may be asked to score each outcome from a long list using the scale proposed by the GRADE group [http://www.gradeworkinggroup.org], in which 1 to 3 signifies an outcome of limited importance, 4 to 6 important but not critical, and 7 to 9 critical. A number of rounds may be held in which responses are summarized and fed back to individuals, allowing them to change their score in light of the group's opinion. Consensus regarding whether an outcome should be in the COS could be defined as 70% or more of the respondents scoring it 7 to 9 and fewer than 15% scoring it as 1 to 3. Consensus that an outcome should not be included in the COS could, for example, be defined as 70% or more scoring it as 1 to 3 and fewer than 15% scoring it as 7 to 9. All other score distributions would be taken to indicate lack of agreement for inclusion of a given outcome in the COS. The rationale for these thresholds is that acceptance that consensus has been reached for an outcome to be included in the COS requires agreement by the majority regarding the critical importance of the outcome, with only a small minority considering it to have little or no importance. Likewise, for consensus to have been reached that an outcome should not be in the COS requires agreement by the majority that the outcome is of little or no importance, with only a small minority considering it to be critically important. Whereas choice of thresholds is somewhat subjective, specification of the definition in the study protocol should reduce the risk that researchers will define consensus post-hoc in a way that would bias the conclusions toward their own beliefs.

Consideration should be given in advance to the possibility that consensus may not be achieved. For example, different stakeholders may disagree about the inclusion of specific outcomes in the COS. This might lead to a decision to recommend the smaller COS, about which there is consensus.

Achieve global consensus

To compare and contrast all research in a topic area, a COS must be applicable and adopted across international settings and across relevant disciplines. Expert panels and conference workshops have been used to achieve international consensus [13].

Review with feedback and updating as necessary

Opportunities to review the COS periodically are important as a form of validation, to ensure outcomes are still relevant and important, to allow the chance to add new outcomes, to evaluate how successful implementation has been and to engage further stakeholders as appropriate. The question of who should be developing and reviewing COS (for example, professional organizations or groups of trialists in particular therapeutic areas) needs careful consideration to ensure continued support for the activity.

Implementation of core outcome set

To increase COS uptake, it is recommended that developers consider engagement with the relevant Cochrane Review Groups, clinical guideline developers, research funders, journal editors, regulators such as research ethics committees, and trial registries. For example, the NIHR Health Technology Assessment funding body in the UK has recently added the following statement to its application form:

'Details should include justification of the use of outcome measures where a legitimate choice exists between alternatives.

-Where established Core Outcomes exist they should be included amongst the list of outcomes unless there is good reason to do otherwise. Please see The COMET Initiative website at www.comet-initiative.org to identify whether Core Outcomes have been established.' Potential barriers and cost implications of implementing the COS should be considered.

Clear and transparent presentation

Reporting standards for studies specifically using Delphi methods to achieve consensus about a COS have been recommended [29]. We propose the checklist in Table 1 to improve the reporting quality of studies to develop consensus around domain concepts or what to measure more generally.

This includes declaration of potential conflicts of interests within the COS development study team to avoid concerns that individuals with vested interests; for example, the developers of measurement instruments in the area of interest may have overtly or covertly manipulated the process of consensus development.

Registration with COMET

One of the aims of the COMET Initiative is to provide a means of identifying existing, ongoing and planned COS studies. COS developers should be encouraged to register their project with COMET. The following information about the scope and methods used will be recorded in the database for existing and ongoing work:

1. Clinical areas for which the outcome domains or outcomes are being considered, identifying both primary disease and types of intervention. Identify the report as a study to develop a COS.

INTRODUCTION

Rationale 3

Describe the rationale for the development of a COS in the context of what is already known. This may include a review of outcomes in previous trials or systematic reviews.

Objectives 4

Provide an explicit statement of questions being addressed with reference, as applicable, to: health condition, population, and types of intervention(s).

METHODS

Protocol and registration 5

Indicate if a study protocol exists, and where it can be accessed (for example, web address) Eligibility criteria 6 Specify participant eligibility criteria, including stakeholder group, the rationale for involving them, and how participants were identified and sampled.

Information sources 7 Describe all information sources (for example, systematic review, databases with dates of coverage, contact with study authors) provided to participants before the start of and during the consensus process. If no information on previously measured outcomes is provided, this should be clearly stated together with details of the method for obtaining information on outcomes of importance from the participants.

Consensus process 8

Describe method to determine consensus and the rationale. A checklist for reporting Delphi methods applied to the development of COS has previously been recommended [29].

Describe how outcomes will be scored during the consensus exercise, and how scores will be summarized across participants during each stage of the consensus process.

Definition of consensus 10 Clearly describe any pre-determined definition of consensus. Describe procedure for determining how outcomes will be included or excluded from consideration at each stage of the consensus process.

RESULTS

Participants 11

Give the total number of participants invited and the number involved in each aspect of the study. Give the proportion of each type of participant from the various stakeholder groups involved. Present any data collected on participant characteristics.

Results of the consensus process 12 As a minimum, provide a comprehensive list of all the outcomes that participants agreed should be included in the core set. Describe a measure of group response and distribution of response for each outcome considered during the process.

DISCUSSION

Summary of evidence 13 Summarize the main findings regarding the level of consensus and the content of the COS. Consider its relevance to key groups e.g. patients and the public, healthcare providers, and policy makers, and any potential barriers to implementation.

Limitations 14 Discuss limitations in terms of stakeholder and geographical coverage. Describe methods used for assessing risk of bias, in relation to information provided to participants beforehand, attrition, any lack of anonymity, etc.

Conclusions

15 Provide a general interpretation of the results in the context of other evidence, and implications for future research.

Funding and editorial independence

Funding 16 Describe sources of funding and the role of the funder in the study.

Conflicts of interest 17 Describe any conflicts of interest within the study team, for example researchers who have developed an outcome measurement instrument applicable to the scope of the COS.

2. Method of development to be used for the COS, including the information participants will receive beforehand to inform their answers, and the methods to be used to reduce any list of possible outcomes. 3. People and organizations involved in identifying and selecting the outcome domains or outcomes, recording how the relative contributions will be used to define the COS. 4. The geographical setting(s) of the study.

Details of any associated publications, including the final report describing whether consensus has been reached, and the outcome domains and outcomes that were included in the COS, will be recorded in the database. Currently, the database includes studies ranging from the development of a specific COS using formal consensus methods, to a systematic review of outcomes measured in clinical trials in a particular area that may inform future COS development. Articles are classified according to the methods used therein.

Discussion

Various methods have been used to develop a COS and it is uncertain which are most suitable, accurate and efficient. There is limited empirical evidence regarding whether different methods lead to similar or different conclusions. One example where consensus work has been undertaken in two different ways is in pediatric asthma. The American Thoracic Society/European Respiratory Society employed an expert panel approach [28], whereas other researchers combined results from a Delphi survey with clinicians and interviews with parents and children. The results were overlapping but not identical [10].

A systematic review of all COS development projects is ongoing, extending a previous review of pediatric studies [16]. This involves a survey of and interviews with COS developers in order to understand their choice of approach, to describe their experience of COS development and implementation, and to identify priority areas for methodological research. It is evident that the reporting of COS development studies requires improvement [29], and as the number of COS studies increases it is important to identify a minimum amount of information to be included in future publications. It is to be expected that the proposed checklist (Table 1) will develop over time, in a similar way to the CONSORT statement [30], based on accumulating evidence. A quality assessment tool will also be developed, using accepted methods [31].

The methods for COS development need to be sufficiently rigorous so as to avoid those situations in which a COS is recommended but subsequently found to be deficient in some way because of the methods used to develop it. Establishing a COS using rigorous methods will also help in setting the threshold for changing it, which arguably should be high, as it is in OMERACT [13]. For example, the original OMERACT core set in rheumatoid arthritis was established without direct patient input; but focus groups were held at OMERACT 6 (in 2002), the first OMERACT meeting that patients were invited to attend. Supported by a previous email survey, fatigue and sleep were identified as missing from the OMERACT core set, which only included pain, function, joint counts, global assessments and a blood test. The experience of fatigue has been reported by a large proportion of people with rheumatoid arthritis, and it is often the most important problem for individual patients. A systematic analysis of these patient focus group discussions revealed three overarching themes: fatigue is overwhelming and different from normal tiredness; it permeates every sphere of life; and selfmanagement is variable, but professional support is rare. Over the next two years a systematic search for articles measuring fatigue discovered twenty-three scales, six of which had sufficient evidence of validity to pass most of the OMERACT filter for truth, discrimination and feasibility [13]. Further work to demonstrate responsiveness was then undertaken and in 2006 fatigue was endorsed as an additional core outcome by a vote at the OMER-ACT 8 meeting [32].

As noted above, another important consideration is the question of who should be developing and reviewing COS (for example, professional organizations or groups of trialists in particular therapeutic areas). The collation and review of empirical evidence concerning dissemination methods and strategies to encourage uptake are other important aspects of the work that need to be considered early in the process.

Conclusion

The COMET Initiative provides a focus for the continued development of a framework for outcome measurement, first in relation to domains and outcomes within domains, subsequently in terms of definitions and measurement instruments, and finally in relation to the timing of measurement. There is an increasing awareness of the need for greater attention to be given to the outcomes measured in clinical trials, in terms of standardization and reporting. Consideration of the issues and the checklist described should help with the development, reporting and implementation of COS.

Abbreviations COMET: Core Outcome Measures in Effectiveness Trials; CONSORT: CONsolidated Standards Of Reporting Trials; COS: core outcome set; COSMIN: COnsensus-based Standards for the selection of health Measurement INstruments; GRADE: Grading of Recommendations Assessment, Development and Evaluation; MOMENT: The management of otitis media with effusion (OME) in children with cleft palate: a feasibility study; NIHR: National Institute for Health Research; ORBIT: Outcome

Table 1
1Checklist of the items that groups should consider when reporting the development of a COS of domain concepts (that is, 'what' to measure)Section/topic
#
Checklist item

TITLE

Title
1

http://www.trialsjournal.com/content/13/1/132
AcknowledgmentsWe are grateful to Dr Maarten Boers and Dr Ian Sinha for helpful comments on an earlier draft of this article, and to the reviewers for helpful comments on the submitted manuscript. This work was supported by a UK Medical Research Council Hub for Trials Methodology Research Network grant G0800792. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.Competing interestsThe authors declare that they have no competing interests.Authors' contributions PRW conceived of the article, and wrote the first draft of the manuscript. DGA helped to revise the draft manuscript. JMB helped to revise the draft manuscript. MC helped to revise the draft manuscript. DD helped to revise the draft manuscript. EG helped to revise the draft manuscript. PT helped to revise the draft manuscript. All authors read and approved the final manuscript.Submit your next manuscript to BioMed Central and take full advantage of:
Reporting Bias in Trials; OMERACT: Outcome MEasures for Rheumatology Clinical Trials; WHO: World Health Organization. Reporting Bias in Trials; OMERACT: Outcome MEasures for Rheumatology Clinical Trials; WHO: World Health Organization.

The Cochrane Library. D: Impact Of Tovey, Cochrane, Tovey D: Impact of Cochrane Reviews [editorial], The Cochrane Library.; 2010. http://www.thecochranelibrary.com/details/editorial/756937/The-Impact-of- Cochrane-Reviews-by-Dr-David-Tovey.html.

Systematic review of the empirical evidence of study publication bias and outcome reporting bias. K Dwan, D G Altman, J A Arnaiz, J Bloom, A W Chan, E Cronin, E Decullier, P J Easterbrook, E Von Elm, C Gamble, D Ghersi, J P Ioannidis, J Simes, P R Williamson, PLoS One. 33081Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E, Decullier E, Easterbrook PJ, von Elm E, Gamble C, Ghersi D, Ioannidis JP, Simes J, Williamson PR: Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One 2008, 3:e3081.

J J Kirkham, K M Dwan, D G Altman, C Gamble, S Dodd, R Smyth, P R Williamson, The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. 340365Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, Williamson PR: The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 2010, 340:c365.

Standardising outcomes for clinical trials and systematic reviews. M Clarke, Trials. 839Clarke M: Standardising outcomes for clinical trials and systematic reviews. Trials 2007, 8:39.

User testing and stakeholder feedback contributed to the development of understandable and useful Summary of Findings tables for Cochrane reviews. S E Rosenbaum, C Glenton, H K Nylund, A D Oxman, J Clin Epidemiol. 63Rosenbaum SE, Glenton C, Nylund HK, Oxman AD: User testing and stakeholder feedback contributed to the development of understandable and useful Summary of Findings tables for Cochrane reviews. J Clin Epidemiol 2010, 63:607-619.

Summary-of-findings tables in Cochrane reviews improved understanding and rapid retrieval of key information. S E Rosenbaum, C Glenton, A D Oxman, J Clin Epidemiol. 63Rosenbaum SE, Glenton C, Oxman AD: Summary-of-findings tables in Cochrane reviews improved understanding and rapid retrieval of key information. J Clin Epidemiol 2010, 63:620-626.

WHO: WHO Handbook for Guideline Development. WHO: WHO Handbook for Guideline Development. http://wwwwhoint/hiv/ topics/mtct/grc_handbook_mar2010_1pdf.

Patient involvement in health research: a contribution to a systematic review on the effectiveness of treatments for degenerative ataxias. P Serrano-Aguilar, M M Trujillo-Martin, J M Ramos-Goni, V Mahtani-Chugani, L Perestelo-Perez, Posada-De La Paz, M , Soc Sci Med. 69Serrano-Aguilar P, Trujillo-Martin MM, Ramos-Goni JM, Mahtani-Chugani V, Perestelo-Perez L, Posada-de la Paz M: Patient involvement in health research: a contribution to a systematic review on the effectiveness of treatments for degenerative ataxias. Soc Sci Med 2009, 69:920-925.

Identifying the clinical domains of fibromyalgia: contributions from clinician and patient Delphi exercises. P J Mease, L M Arnold, L J Crofford, D A Williams, I J Russell, L Humphrey, L Abetz, S A Martin, Arthritis Rheum. 59Mease PJ, Arnold LM, Crofford LJ, Williams DA, Russell IJ, Humphrey L, Abetz L, Martin SA: Identifying the clinical domains of fibromyalgia: contributions from clinician and patient Delphi exercises. Arthritis Rheum 2008, 59:952-960.

Development of a core outcome set for clinical trials in childhood asthma-a survey of clinicians, parents and young people. I Sinha, R Gallagher, P R Williamson, R L Smyth, Trials. 13103Sinha I, Gallagher R, Williamson PR, Smyth RL: Development of a core outcome set for clinical trials in childhood asthma-a survey of clinicians, parents and young people. Trials 2012, 13:103.

Incorporating the patient perspective into outcome assessment in rheumatoid arthritisprogress at OMERACT 7. J R Kirwan, S E Hewlett, T Heiberg, R A Hughes, M Carr, M Hehir, T K Kvien, P Minnock, S P Newman, E M Quest, E Taal, J Wale, J Rheumatol. 32Kirwan JR, Hewlett SE, Heiberg T, Hughes RA, Carr M, Hehir M, Kvien TK, Minnock P, Newman SP, Quest EM, Taal E, Wale J: Incorporating the patient perspective into outcome assessment in rheumatoid arthritis- progress at OMERACT 7. J Rheumatol 2005, 32:2250-2256.

A bibliography of research reports about patients' , clinicians' and researchers' priorities for new research. S Oliver, J Gray, London: James Lind AllianceOliver S, Gray J: A bibliography of research reports about patients' , clinicians' and researchers' priorities for new research. London: James Lind Alliance; 2006.

OMERACT: an international initiative to improve outcome measurement in rheumatology. P Tugwell, M Boers, P Brooks, L Simon, V Strand, L Idzerda, Trials. 838Tugwell P, Boers M, Brooks P, Simon L, Strand V, Idzerda L: OMERACT: an international initiative to improve outcome measurement in rheumatology. Trials 2007, 8:38.

D C Turk, R H Dworkin, R R Allen, N Bellamy, N Brandenburg, D B Carr, C Cleeland, R Dionne, J T Farrar, B S Galer, D J Hewitt, A R Jadad, N P Katz, L D Kramer, D C Manning, C G Mccormick, M P Mcdermott, P Mcgrath, S Quessy, B A Rappaport, J P Robinson, M A Royal, L Simon, J W Stauffer, W Stein, J Tollett, J Witter, Core outcome domains for chronic pain clinical trials: IMMPACT recommendations. Pain. 106Turk DC, Dworkin RH, Allen RR, Bellamy N, Brandenburg N, Carr DB, Cleeland C, Dionne R, Farrar JT, Galer BS, Hewitt DJ, Jadad AR, Katz NP, Kramer LD, Manning DC, McCormick CG, McDermott MP, McGrath P, Quessy S, Rappaport BA, Robinson JP, Royal MA, Simon L, Stauffer JW, Stein W, Tollett J, Witter J: Core outcome domains for chronic pain clinical trials: IMMPACT recommendations. Pain 2003, 106:337-345.

Evaluating maternity care: a core set of outcome measures. D Devane, C M Begley, M Clarke, D Horey, C Oboyle, Birth. 34Devane D, Begley CM, Clarke M, Horey D, Oboyle C: Evaluating maternity care: a core set of outcome measures. Birth 2007, 34:164-172.

A systematic review of studies that aim to determine which outcomes to measure in clinical trials in children. I Sinha, L Jones, R L Smyth, P R Williamson, PLoS Med. 596Sinha I, Jones L, Smyth RL, Williamson PR: A systematic review of studies that aim to determine which outcomes to measure in clinical trials in children. PLoS Med 2008, 5:e96.

. Comet The, Initiative, The COMET Initiative. http://www.comet-initiative.org.

Reporting clinical outcomes of breast reconstruction: a systematic review. S Potter, A Brigic, P F Whiting, S J Cawthorn, K N Avery, J L Donovan, J M Blazeby, J Natl Canc Inst. 103Potter S, Brigic A, Whiting PF, Cawthorn SJ, Avery KN, Donovan JL, Blazeby JM: Reporting clinical outcomes of breast reconstruction: a systematic review. J Natl Canc Inst 2011, 103:31-46.

Assessment of cosmesis after breast reconstruction surgery: a systematic review. S Potter, D Harcourt, S Cawthorn, R Warr, N Mills, D Havercroft, J Blazeby, Ann Surg Oncol. 18Potter S, Harcourt D, Cawthorn S, Warr R, Mills N, Havercroft D, Blazeby J: Assessment of cosmesis after breast reconstruction surgery: a systematic review. Ann Surg Oncol 2011, 18:813-823.

Outcomes in clinical trials of inhaled corticosteroids for children with asthma are narrowly focussed on short term disease activity. I P Sinha, P R Williamson, R L Smyth, PLoS One. 46276Sinha IP, Williamson PR, Smyth RL: Outcomes in clinical trials of inhaled corticosteroids for children with asthma are narrowly focussed on short term disease activity. PLoS One 2009, 4:e6276.

The James Lind Alliance Guidebook. K Cowan, S Oliver, Oxford: James Lind Alliance4th editionCowan K, Oliver S: The James Lind Alliance Guidebook. 4th edition. Oxford: James Lind Alliance; 2012.

Harmonizing Outcome Measurements in Eczema Delphi panel. Core outcome domains for controlled trials and clinical recordkeeping in eczema: international multiperspective Delphi consensus process. J Schmitt, S Langan, T Stamm, H C Williams, J Invest Dermatol. 131Schmitt J, Langan S, Stamm T, Williams HC: Harmonizing Outcome Measurements in Eczema Delphi panel. Core outcome domains for controlled trials and clinical recordkeeping in eczema: international multiperspective Delphi consensus process. J Invest Dermatol 2011, 131:623-630.

Patient perspectives on the impact of fibromyalgia. L M Arnold, L J Crofford, P J Mease, S M Burgess, S C Palmer, L Abetz, S A Martin, Patient Educ Counsel. 73Arnold LM, Crofford LJ, Mease PJ, Burgess SM, Palmer SC, Abetz L, Martin SA: Patient perspectives on the impact of fibromyalgia. Patient Educ Counsel 2008, 73:114-120.

What outcomes from pharmacologic treatments are important to people with rheumatoid arthritis? Creating the basis of a patient core set. T Sanderson, M Morris, M Calnan, P Richards, S Hewlett, Arthritis Care Res. 62Sanderson T, Morris M, Calnan M, Richards P, Hewlett S: What outcomes from pharmacologic treatments are important to people with rheumatoid arthritis? Creating the basis of a patient core set. Arthritis Care Res Hoboken 2010, 62:640-646.

Outcomes from the Patient Perspective Workshop at OMERACT 6. J Kirwan, T Heiberg, S Hewlett, R Hughes, T Kvien, M Ahlmèn, M Boers, P Minnock, K Saag, B Shea, M Suarez Almazor, E Taal, J Rheumatol. 30Kirwan J, Heiberg T, Hewlett S, Hughes R, Kvien T, Ahlmèn M, Boers M, Minnock P, Saag K, Shea B, Suarez Almazor M, Taal E: Outcomes from the Patient Perspective Workshop at OMERACT 6. J Rheumatol 2003, 30:868-872.

Exploring Impact: Public involvement in NHS, public health and social care research. Eastleigh: INVOLVE. K Staley, Staley K: Exploring Impact: Public involvement in NHS, public health and social care research. Eastleigh: INVOLVE; 2009.

The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. L B Mokkink, C B Terwee, D L Patrick, J Alonso, P W Stratford, D Knol, L M Bouter, H C De Vet, Qual Life Res. 19Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol D, Bouter LM, de Vet HC: The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res 2010, 19:539-549.

statement: asthma control and exacerbations: standardizing endpoints for clinical asthma trials and clinical practice. H K Reddel, D R Taylor, E D Bateman, L-P Boulet, H A Boushey, W W Busse, T B Casale, P Chanez, P L Enright, P G Gibson, J C De Jongste, Ham Kerstjens, S C Lazarus, M L Levy, P M O'byrne, M R Partridge, I D Pavord, M R Sears, P J Sterk, S W Stoloff, S D Sullivan, S J Szefler, M D Thomas, S E Wenzel, Am J Respir Crit Care Med. 180European Respiratory SocietyTask Force on Asthma Control and Exacerbations: An officialReddel HK, Taylor DR, Bateman ED, Boulet L-P, Boushey HA, Busse WW, Casale TB, Chanez P, Enright PL, Gibson PG, de Jongste JC, Kerstjens HAM, Lazarus SC, Levy ML, O'Byrne PM, Partridge MR, Pavord ID, Sears MR, Sterk PJ, Stoloff SW, Sullivan SD, Szefler SJ, Thomas MD, Wenzel SE, American Thoracic Society/European Respiratory Society Task Force on Asthma Control and Exacerbations: An official American Thoracic Society/European Respiratory Society statement: asthma control and exacerbations: standardizing endpoints for clinical asthma trials and clinical practice. Am J Respir Crit Care Med 2009, 180:59-99.

Using the delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. I P Sinha, R L Smyth, P R Williamson, PLoS Med. 81000393Sinha IP, Smyth RL, Williamson PR: Using the delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS Med 2011, 8:1000393.

. Consort The, Statement, The CONSORT Statement. http://www.consort-statement.org/consort- statement/.

The AGREE Collaboration: Appraisal of Guidelines for Research & Evaluation (AGREE) Instrument. The AGREE Collaboration: Appraisal of Guidelines for Research & Evaluation (AGREE) Instrument; http://www.agreecollaborationorg.

Patient perspective: reasons and methods for measuring fatigue in rheumatoid arthritis. J R Kirwan, S Hewlett, 10.1186/1745-6215-13-132J Rheumatol. 34Kirwan JR, Hewlett S: Patient perspective: reasons and methods for measuring fatigue in rheumatoid arthritis. J Rheumatol 2007, 34:1171-1173. doi:10.1186/1745-6215-13-132

Developing core outcome sets for clinical trials: issues to consider. Williamson, Trials. 13132Cite this article asCite this article as: Williamson et al.: Developing core outcome sets for clinical trials: issues to consider. Trials 2012 13:132.


A New Website for Cartographic Resources -CartoMundi

Jean-Luc Arnaud firstname.lastname@example.com
National Center for Scientific Research
Maison méditerranéenne des sciences de l'hommeAix-en-Provence

A New Website for Cartographic Resources -CartoMundi
iSSN: 1435-5205. P271-275CC8BB88D7A03F3FBA982FBE277D30E91CNRSADONISMMSHkey mapgraphic indexmap seriescollective catalogueGISdigitised maps
For about two years, the Maison Méditerranéenne des Sciences de l'Homme -MMSH -in Aix-en-Provence has been developing a new website -CartoMundidedicated to cartographic documents.This site is organised around a tool able to identify and select documents according to the geographical zone they really represent.Each series is described by a unique notice and a key map.CartoMundi is a collective project; it consists of an open network of members.It will offer easy access to largely unknown documents and should encourage the use of cartography, both as a resource and as a research subject.

The Maison Méditerranéenne des Sciences de l'Homme -MMSH -in Aix-en-Provence, numbering ten research centres in human and social sciences specialises in the Mediterranean region, and has a cross-disciplinary research topic dedicated to spatial representations.Its goal is to develop a long-term pool of competences for the use and the analysis of cartographic and iconographic documents.Its activities are organised in three main fields: research, training and production of research tools.Within this framework, a new website -CartoMundi -dedicated to cartographic documents, is being developed.This programme finds its origin in the gap noticeable between metadata usually used to describe cartographic documents and the questions emanating from users who are looking for places.To reduce the gap, CartoMundi suggests adding to databases a geographic field which will be able to answer these questions.This field allows building a list of documents which represent a given place (punctual or territorial) identified though a simple click on a world map.

A Profuse Production with an Inadequate Recording System

Most of the maps and plans of towns and territories produced since the end of the 18th century are organised in series.Each series represents a specific place (town, region or country) at a specific time, on a relatively detailed scale, considering the size of the covered perimeter, which then needs to be published on very numerous sheets.For example, the map of France published by the Institut Géographique National numbers no less than one thousand sheets.Generally, these plans and maps have been produced by local, regional or national public authorities and their main goal is either the knowledge of and control over the territory for war purposes (the army is a great map producer) or the ability to plan infrastructure at every level, to record land property, or to calculate taxes (through cadastral maps).These documents are standardised and have not given rise to much interest among historians of cartography.

Nevertheless, it is obvious that these series are of exceptional value for academic work.They are not only important for retrospective geographical research, but for a vast number of studies in which researchers are curious about the spatial localisation of the phenomena they are analysing.However it is very difficult for researchers to use these documents.Their number renders their consultation difficult: finding your way around the 230 square metres of the map of France is no easy task.Furthermore, due to the number of sheets, the available catalogues do not render the territorial segmentation adequately.

Unlike periodicals, maps and plans series do not have a specific mode of cataloguing.The international system currently in use is adapted from the book cataloguing system: one entry corresponds to one document.This principle is relevant for individual maps, but it is not operative for map series because it is unable to indicate the geographical zone represented by each sheet, and cannot describe their editorial relation between the sheets.This drawback has two neg-ative implications for the use of the document.Whatever the field of research, the documents are difficult to use.Moreover, the standard mode of cataloguing is often neglected by the map curators themselves who prefer to keep printed files and key maps rather than to launch themselves in the long and tedious task of computerising data with an unsatisfactory result for users.

A New Tool -Changing Scale

In the light of this situation, it appeared necessary to develop a specific tool for map series -a tool able to identify and select documents according to the geographical zone they really represent.The most efficient method is well known: it is the key map or graphic index.For each series, the sheets' grid is printed on a base map with enough reference points to plot out the zone represented by each sheet.This method is not new, but until recently it was difficult to computerise.Thus, these key maps are often considered as a local appendix to the collections.On the contrary, in the system developed by CartoMundi, the graphic indexes are central.This entails a transformation of the cataloguing system: the unit is no longer a sheet but a series, itself described by a notice and a graphic index.

This method was first developed by the MMSH within an experiment on the mapping of Mediterranean towns and countries: La Cartothèque Méditerranéenne.This bilingual website, launched in September 2006, catalogues several thousands of maps depicting mainly the South and East of the Mediterranean basin, through the description of about 200 cartographic series.First of all, this index is independent from the place of storage of the maps and plans: each series is described fully from incomplete collections held by various libraries.Secondly, it is also a collective catalogue which allows an unlimited number of partners to report and to update the content of their collection.

Several French institutions have shown a great deal of interest in this project, especially the maps and plans department of the Bibliothèque nationale de France, the Institut géographique national, the Comité français de cartographie, the Réseau des bibliothèques universitaires de France, and the European network of social sciences research centres Ramses 2 .Moreover, the Centre national de la recherche scientifique (CNRS) has selected CartoMundi for its project ADONIS aimed to improve access to computerised data.

Functioning through Stages and Networks

CartoMundi consists in an open network of members, and its development follows four main stages.

Setting up the Project

The project combines a geographic information system with a data base management system and a graphic interface of data restitution.Housed in the MMSH in Aix-en-Provence, it is developed by the MMSH with the help of the ADONIS project and the TELEMME research centre.The project is supervised by two teams of experts: representatives of the Bibliothèque nationale de France, of the Comité français de cartographie and of the Institut géographique national who are concerned with the documentation, and scholars of several research centres who deal with computerised and geomatic cataloguing.

Building the Index

This stage is dedicated to the generic description of map series.The MMSH, with the help of the Ramses 2 project, takes care of the maps and plans representing the Eastern and Southern Mediterranean countries.For the other parts of the world, it is necessary to develop partnerships with institutions with sufficient means to invest in the widening of the geographic scope covered by the project.National libraries, geographical societies, map libraries associations, map producing institutions, local or regional authorities, or universities could develop such a partnership.

Collective Catalogue

The collective catalogue is designed to include any interested institution, as long as it holds cartographic resources organised in series in libraries open to the public, and wishes to bring an added value to their collection through an innovative description mode.CartoMundi offers an interface through which, for each series of maps described in the index, they can enter online the content of their collection.

Digitised Reproductions

In the last few years digitised maps have grown in numbers which makes the navigation between files gradually more difficult.In this case again, the graphic index is the most efficient tool to select and visualise the documents.CartoMundi offers two different services.From its website, it puts online its own digitised documents or documents belonging to its members.It also develops links to other websites which offer online map series without any navigation tool between sheets.

CartoMundi seems to have interested map curators for its cataloguing system and management of documents which are simple to use.Moreover, it is also a tool for documentation, developed by scholars, for research purposes.It will offer easy access to documents which have been largely unknown, and should encourage the use of cartography, both as an information resource and as a research subject.In this way, it will contribute to one of the main objectives of the research programme: space, representation and use of the MMSH.

Websites Referred to in the Text