Quality indicators (QIs) are used in assessing the quality of healthcare. Evidence-based clinical practice guidelines (CPGs) are relevant sources for generating QIs. In this context, QIs are important tools to assess the implementation of guideline recommendations. However, the methodological approaches to guideline-based QI development vary considerably.
In Germany, the guideline classification scheme of the AWMF (German Association of the Scientific Medical Societies) differentiates between S1-, S2k-, S2e-, and S3-CPGs depending on the methodological approach. Thus, S3-CPGs are consensus- and evidence-based CPGs and have the highest methodological standard in Germany. An analysis of the status quo of reported QIs in S3-CPGs found 35 current S3-CPGs, which report 372 different QIs.
Currently, there is no gold standard for the development of guideline-based QIs. To our knowledge, no studies have investigated to what extent guideline-based QIs from different CPGs that are related to the same topic are consistent. The objective of this study is to compare guideline-based QIs and their underlying methodological approaches of German S3-CPGs with those of topic-related international CPGs.
Based on the previous identified German S3-CPGs (n = 35), which report quality indicators, we will conduct systematic searches in the guidelines databases of G-I-N (Guidelines International Network) and NGC (National Guideline Clearinghouse) to identify international CPGs matching the topics of the S3-CPGs. If necessary, we will search additionally the websites of the particular CPG providers for separate documents with regard to QIs. We will include evidence-based CPGs which report QIs. Reported QIs as well as methods of development and the rationale for QIs will be extracted and compared with those of the S3-CPGs.
This study will be part of the project “Systematic analysis of the translation of guideline recommendations into quality indicators and development of an evidence- and consensus-based standard,” supported by the German Research Association (DFG). The results of this analysis will feed into a subsequent qualitative study, which will consist of structured interviews with developers of international CPGs. Further, the results will be considered in a consensus study on standards of the translation of guideline recommendations into quality indicators in Germany.
Quality measurement and improvement play an important role in healthcare. For this purpose, quality indicators (QIs) can be used. There is no clear-cut definition of a QI. According to Lawrence and Frede, a QI is a “measurable element of practice performance for which there is evidence or consensus that it can be used to assess the quality, and hence change in the quality, of care provided” . The Joint Commission on Accreditation of Healthcare Organizations (JCAHO) defines QIs as “[…] quantitative measures that can be used to monitor and evaluate the quality of important governance, management, clinical, and support functions that affect patient outcomes” . To be deemed as trustworthy and useful, QIs have to satisfy different criteria, such as relevance, validity, reliability, feasibility, and target group orientation [3–6]. To meet the high methodological requirements on QIs, they should be based where possible on scientific evidence and developed in a systematic and transparent way [7, 8].
As evidence-based clinical practice guidelines (CPGs) are designed to reflect current best practice, they are relevant sources for generating QIs [7, 9]. The term “guideline-based QIs” indicates in particular QIs that are either generated from already available CPGs or coupled with the process of CPG development . Besides assessing the quality of healthcare, these are important tools to assess the implementation of guideline recommendations [11–13]. However, the methodological approaches to guideline-based QI development vary considerably .
In Germany, the AWMF (German Association of the Scientific Medical Societies) provides the methodological framework for the development of CPGs by the scientific medical societies. The guideline classification scheme of the AWMF differentiates between S1-, S2k-, S2e-, and S3-CPGs depending on the methodological approach . Thus, S1-CPGs are based on an informal consensus building. In S2k-CPGs, a formal consensus method is applied in a representative panel, and S2e-CPGs include a systematic approach for literature searching as well as selection and appraisal of evidence. S3-CPGs comprise both the requirements for S2k-CPGs and those for S2e-CPGs and thus have the highest methodological standard in Germany. An analysis of the status quo of reported QIs in S3-CPGs from 2013 found 34 S3-CPGs, which report 394 different QIs (including measures of quality which are labeled such as “quality criteria” or “quality measure”) . For example, the S3-CPG “Diagnostics, treatment and follow-up care of malignant ovarial tumors” comprises 12 QIs, one of them regarding counseling by a social service (numerator: number of patients with counseling by a social service; denominator: all patients with an initial diagnosis of ovarian cancer and treatment in a clinical institution) . In the S3-CPG “Long-Term Opioid-Use in Non-Cancer Pain,” three QIs are stated, such as the QI “number of patients with somatoform pain disorder who are treated with an opioid” . An update (search up to 2016) of this analysis (not yet published) found 35 current S3-CPGs, which report 372 different QIs. Four S3-CPGs were developed by the National Program for Disease Management Guidelines (DMG), 15 by the German Guideline Program in Oncology (GGPO), and 16 by various medical societies. Particularly, the CPGs of the DMG and GGPO have a broad scope and cover various areas of medical care. For these CPGs, the development of guideline-based QIs is obligatory [11–13].
Although a working group of the Guidelines International Network (G-I-N) recently proposed a set of reporting standards for guideline-based performance measures , there is currently no gold standard for the development of guideline-based QIs [10, 19]. To our knowledge, no studies have investigated to what extent guideline-based QIs from different CPGs are consistent. Our hypothesis is that QIs from S3-CPGs are in many cases not corresponding with QIs of topic-related international CPGs.
The objective of this study is to compare guideline-based QIs and their underlying methodological approaches of the 35 previously identified German S3-CPGs with those of topic-related international CPGs.
Our study is not registered with PROSPERO as we will not report health-related outcomes.
Based on the previously identified S3-CPGs which report QIs, we will conduct systematic searches in the guidelines databases of G-I-N and NGC (National Guideline Clearinghouse) to identify international CPGs matching the topics of the S3-CPGs. The search strategies will include suitable keywords relating to the clinical topics and as appropriate truncations as well as Boolean operators. In cases where we cannot identify topical eligible guidelines, we will screen the websites of CPG providers additionally, whereby the searches will be tailored to the structure and capabilities of the websites. Furthermore, we will crosscheck the reference lists of the S3-CPGs and the international CPGs eligible for inclusion in the analysis.
In cases where topical eligible CPGs comprise neither QIs nor links to QIs, we will search the websites of the particular CPG providers for separate documents with regard to QIs that are explicitly linked with the particular CPG.
One reviewer will screen the titles of records, and the full texts of those deemed eligible for inclusion will be retrieved. In the next step, the screening of full texts will be conducted by one reviewer and checked by another. The reasons for exclusion will be documented, and any disagreements will be resolved through discussion and consensus.
The records will be uploaded and managed using Microsoft Excel.
In cases where no eligible CPG matching the topic of a S3-CPG can be found, we will exclude the particular CPG from analysis.
The extractions will be conducted by one reviewer and checked by another, any disagreements will be resolved through discussion and consensus.
Two reviewers will perform the quality assessment independently. In case of two or more points of difference in the appraisal of the two reviewers, disagreement will be resolved through discussion and consensus. A domain score will be calculated by summing up the scores of the individual items and by standardizing the total as the percentage of the maximum possible score for the domain (4 (strongly agree) × 7 (items) × 2 (appraisers)) .
Reviewers who have been involved in the development of the included CPGs will not participate in their quality assessment.
Data synthesis will contain a descriptive analysis and a tabular comparison of the QIs of the included CPGs and those of the S3-CPGs for each clinical topic and when applicable for each underlying recommendation. We will collect the number of CPGs that give information to the QI-development group, the methods of QI-development, as well as the rationale and intended purpose of QI. On the basis of reported QIs, we will collect the number of QI for which quality objectives and measurement properties are reported as well as the number of QI that are explicitly or implicitly based on guideline recommendations.
For each matched pair of CPGs, we will compare the suggested QIs and assess if the QIs agree, disagree, or if they are not comparable. We will assign QIs on the same topic either to the category “not different/slightly different” or “different.” QIs that are not comparable will be extracted under the category “QI only defined in the international respectively the S3-CPG. For each category, we will collect the number of QIs respectively QI-pairs. Furthermore, the methods for QI-development will be summarized narratively.
This study will be part of the project “Systematic analysis of the translation of guideline recommendations into quality indicators and development of an evidence- and consensus-based standard,” supported by the German Research Association (DFG). It will be the second systematic analysis in the overall project. The results of this analysis will feed into a subsequent qualitative study which will consist of structured interviews with developers, methodologists, and users of international guidelines. Both studies intend to deliver additional information to existing research on methods for the development of guideline-based QIs [10, 18]. For the analysis of possible differences between QIs from different CPGs, we will consider existing guidelines or rather QI development manuals of the respective guideline organization.
An overview of the overall project is shown in Fig. 1.
This protocol adheres to the “Preferred Reporting Items for Systematic Review and Meta-Analysis-Protocols (PRISMA-P)” . As PRISMA-P aims to guide the development of protocols for systematic reviews evaluating therapeutic efficacy, we deviated from the original checklist by omitting items (e.g., outcomes and prioritization) due to the methodological focus of our planned systematic review (see Additional file 2 for the completed PRISMA-P checklist).
The results of our study will be considered in the last phases of the overall project, namely a consensus-study on standards of the translation of guideline recommendations into quality indicators.
We thank Max Geraedts (University of Marburg) for reviewing the final draft of the protocol.
German Research Association (DFG) (grant no. NE 385/23-1); the DFG was not involved in developing the protocol.
MB participated in the design of the study, wrote the first draft of the protocol, and revised the manuscript. JB participated in the design of the study, helped to draft the manuscript, and has been involved in revising the manuscript. MN participated in the design of the study and has been involved in revising the manuscript. SD has been involved in revising the manuscript. JS participated in the design of the study and has been involved in revising the manuscript. EN participated in the design of the study and has been involved in revising the manuscript. DP participated in the design of the study, helped to draft the manuscript, and has been involved in revising the manuscript. All authors read and approved the final manuscript. MB is a guarantor.
MN was involved in the development of several S3-CPGs that are considered in this review. MB was involved in the preparation of the evidence report for one CPG that is considered in this review. MB, JB, MN, JS, EN, and DP were involved in the development of several S3-CPGs that are not considered in this review. DS and MS declare no competing interests (financial and non-financial).
Kelley Edward, Jermy H. Health care quality indicators project conceptual framework paper. 2006: OECD Health Working Papers: 23, https://www.oecd.org/els/health-systems/36262363.pdf. Accessed 08 May 2017.
National Quality Forum, Measure evaluation criteria and guidance for evaluating measures for endorsement. 2016: http://www.qualityforum.org/Measuring_Performance/Endorsed_Performance_Measures_Maintenance.aspx. Assessed 08 May 2017.
Institute of Medicine (U.S.). Committee on redesigning health insurance performance measures, P., and performance improvement programs, performance measurement: accelerating improvement (pathways to quality health care series). Washington: The National Academies Press; 2006. https://www.nap.edu/. Accessed 08 May 2017.
German Guideline Program in Oncology (GGPO). Development of guideline based quality indicators. 2013: http://leitlinienprogramm-onkologie.de/uploads/tx_sbdownloader/QIDP_GGPO_2013.pdf. Accessed 08 May 2017.
German Medical Association (GMA), National Association of Statutory Health Insurance Physicians (NASHIP), and Association of the Scientific Medical Societies (AWMF), National Programme for Disease Management Guidelines. Method Report. 4th edition. 2010 http://www.versorgungsleitlinien.de/methodik/reports. Accessed 08 May 2017.
Arbeitsgemeinschaft der Wissenschaftlichen Medizinischen Fachgesellschaften (AWMF)- Ständige Kommission Leitlinien, AWMF-Regelwerk „Leitlinien". 2012: http://www.awmf.org/leitlinien/awmf-regelwerk.html. Accessed 08 May 2017.
Leitlinienprogramm Onkologie (Deutsche Krebsgesellschaft, Deutsche Krebshilfe, AWMF). S3-Leitlinie Diagnostik, Therapie und Nachsorge maligner Ovarialtumoren, Langversion 2.0. AWMF Registrierungsnummer: 032-035OL. 2016: http://leitlinienprogramm-onkologie.de/Leitlinien.7.0.html. Assessed 21 June 2017.
Deutsche Schmerzgesellschaft. S3 - Leitlinie „Langzeitanwendung von Opioiden bei nicht tumorbedingten Schmerzen - "LONTS". AWMF Registernummer: 145/003. 2015: http://www.awmf.org. Assessed 19 June 2017.
World Health Organization, List of Member States by WHO Region and Mortality Stratum. http://www.who.int/choice/demography/mortality_strata/en/. Accessed 20 Oct 2017.
National Institute for Health and Care Excellence (NICE) (2016). The NICE menu of general practice and clinical commissioning group indicators. https://www.nice.org.uk/Media/Default/Standards-and-indicators/indicator-menu-update-aug-16.pdf. Assessed 08 Nov 2017.
German Association of the Scientific Medical societies (AWMF) and Agency for Quality in Medicine (ÄZQ), German Instrument for Methodological Guideline Appraisal 2008: http://www.delbi.de/. Accessed 08 May 2017.