Meryl Brod,Laura Tesler Waldman,Thomas Sparre,Anne Kirstine Busk
Development and Validation of the Diabetes Pen Experience Measure: A New Patient-Reported Outcome Measure
insulin pen,patient-reported outcome,diabetes,injection device
Background: Satisfaction with insulin-delivery devices has been shown to improve treatment adherence, translating into better glycemic control. The Diabetes Pen Experience Measure (DPEM) is a new patient-reported outcome measure to evaluate patients’ experience when using an injection device. Methods: The DPEM was developed using literature review and concept elicitation interviews with clinical experts and patients. This led to a theoretical model and a draft measure of the diabetes pen experience, which was refined following cognitive debriefing. Validation entailed a web-based, noninterventional survey; psychometric analyses conducted according to a statistical analysis plan; and refinement and finalization of the DPEM and theoretical model. Results: In total, 42 patients participated in concept elicitation interviews. Analysis of the qualitative interviews resulted in a preliminary theoretical model. Based on this model, DPEM items were generated; the preliminary version of the DPEM contained 30 items. Following cognitive debriefing, the validation-ready version comprised 28 items. These were later reduced to 7 higher-order items owing to ceiling/floor effects. In total, 300 patients participated in the web-based validation study. The item statistics were all adequate. Item-to-item correlations were good. Item-to-total correlations displayed acceptable associations between each item against the rest of the items, with correlations of 0.68 to 0.79. The internal consistency was adequate, with a Cronbach’s alpha of 0.91. The DPEM is scored by summing the 7 item scores and transforming the sum onto a 100-point scale. Conclusion: The evidence presented supports the use of the DPEM in clinical trials to evaluate the patients’ experience with diabetes injection devices.
Development and Validation of the Diabetes Pen Experience Measure: A New Patient-Reported Outcome Measure

Satisfaction with insulin-delivery devices has been shown to improve treatment adherence, translating into better glycemic control. The Diabetes Pen Experience Measure (DPEM) is a new patient-reported outcome measure to evaluate patients’ experience when using an injection device.

The DPEM was developed using literature review and concept elicitation interviews with clinical experts and patients. This led to a theoretical model and a draft measure of the diabetes pen experience, which was refined following cognitive debriefing. Validation entailed a web-based, noninterventional survey; psychometric analyses conducted according to a statistical analysis plan; and refinement and finalization of the DPEM and theoretical model.

In total, 42 patients participated in concept elicitation interviews. Analysis of the qualitative interviews resulted in a preliminary theoretical model. Based on this model, DPEM items were generated; the preliminary version of the DPEM contained 30 items. Following cognitive debriefing, the validation-ready version comprised 28 items. These were later reduced to 7 higher-order items owing to ceiling/floor effects. In total, 300 patients participated in the web-based validation study. The item statistics were all adequate. Item-to-item correlations were good. Item-to-total correlations displayed acceptable associations between each item against the rest of the items, with correlations of 0.68 to 0.79. The internal consistency was adequate, with a Cronbach’s alpha of 0.91. The DPEM is scored by summing the 7 item scores and transforming the sum onto a 100-point scale.

The evidence presented supports the use of the DPEM in clinical trials to evaluate the patients’ experience with diabetes injection devices.

Diabetes (type 1 and type 2) is a global public health burden that affects more than 460 million people worldwide. Diabetes is associated with an increased risk of macrovascular and microvascular complications. Intensive blood-glucose control with oral glucose-lowering medications or injectable therapies, such as insulin treatments and glucagon-like peptide 1 receptor agonists (GLP-1 RA), has been shown to reduce the risk of diabetic complications. In particular, the Diabetes Control and Complications Trial (DCCT) and its ongoing long-term, observational, follow-up Epidemiology of Diabetes Interventions and Complications (EDIC) study have demonstrated that early intensive insulin intervention aiming to achieve glycemia as close to the nondiabetic range as is safely possible reduced early-stage and advanced microvascular and cardiovascular complications of type 1 diabetes. Similarly, the UK Prospective Diabetes Study 33 (UKPDS 33) has shown that intensive blood-glucose control with sulfonylureas or insulin reduced the risk of microvascular complications in patients with type 2 diabetes. However, for many patients with diabetes, adequate glycemic control and, consequently, improvements in clinical outcomes are not achieved. Poor treatment adherence is one of the main factors leading to inadequate glycemic control, which has been associated with increased physician, emergency room, and hospital visits, as well as increased mortality. Issues contributing to poor treatment adherence include inconvenience, fear of injections, and complicated treatment regimens. Satisfaction with insulin-delivery devices has been shown to positively affect attitudes toward insulin therapy and improve treatment adherence, which in turn could result in better glycemic control, and, ultimately, in reduced long-term diabetic complications. Several patient-reported outcome (PRO) measures have been developed to assess treatment satisfaction in patients with diabetes. However, few of these PRO measures are specific to pens. Here, we report the development and validation of the Diabetes Pen Experience Measure (DPEM), a new PRO measure to evaluate patients’ experience of using an injection device for the treatment of diabetes. Development of the DPEM followed the scientific process outlined in the 2009 U.S. Food and Drug Administration (FDA) guidance for industry for the development of PRO measures.

Following current guidelines for the development of PRO measures, such as the 2009 U.S. FDA guidance, the concept elicitation (CE) phase of the DPEM development included (1) a targeted literature review and CE interviews with clinical experts and patients; (2) generation of a draft theoretical model of the diabetes pen experience in patients and items reflecting the domain; (3) refinement of the measure based on cognitive debriefing (CD); (4) preliminary validation analysis; (5) further refinement of the measure based on preliminary validation analysis, reevaluation of the CE interview data, and further CD; and (6) finalization of the validation-ready DPEM. The psychometric validation study entailed the following: (1) development of a validation battery, (2) usability testing of a web-based validation survey, (3) conducting the web-based validation study, (4) refinement of the DPEM based on findings, and (5) finalization of the DPEM and theoretical model.

A literature review conducted using PubMed provided the basis for developing the semi-structured interview guides used in interviews with clinical experts and patients, respectively. CE data were collected iteratively in the U.S. from clinical experts via individual telephone interviews and from patients via in-person focus groups and individual telephone interviews. Completed interviews were used to guide and inform subsequent interviews. Issues raised by experts and patients in earlier interviews were further explored and either confirmed or rejected based on the content of the additional interviews. Eligible clinical experts were in current practice and either: (1) an endocrinologist treating patients with diabetes for at least 5 years and spending at least 50% of their time caring for patients in a clinical setting, (2) a primary care physician treating patients with diabetes for at least 10 years, or (3) a certified diabetes nurse educator treating patients with diabetes for at least 5 years. Eligible patients must have been aged 18 years or older, received a diagnosis of type 1 or type 2 diabetes, and been using insulin and/or GLP-1 RA delivered via a pen device for at least 3 months prior to the interview. Patients using an insulin pump, receiving any injectable medication for a medical condition other than diabetes, or having a history of cognitive impairment or other mental health issue affecting their ability to participate in a focus group were excluded. Verbal informed consent was obtained from all participants before the interviews took place. Focus group and telephone interview transcripts were analyzed with qualitative research methods using an adapted grounded theory approach and Dedoose software (http://www.dedoose.com). Interview transcripts were coded in the order the interviews had been conducted for the clinician and patient populations, respectively. Emerging concepts were added to the coding scheme as they arose, and earlier transcripts were iteratively reviewed to apply the new codes as appropriate. The codes were then organized into the major themes and subthemes that emerged from the analysis. Following the analysis, the research team held a 2-day item-generation meeting. During this meeting, the qualitative data findings were reviewed, and a preliminary theoretical model of the functions and features of diabetes pens was developed; the model identified proximal and distal impacts of these features on well-being and diabetes management, and mediators and moderators of these. Based on the draft theoretical model, items for the DPEM were generated, using lay terminology as much as possible, to reflect the areas of importance for the functions and features domain as well as for the proximal impacts. The criteria for item identification were as follows: (1) patient endorsement of at least 15% for aspects of using an injection pen that they considered to be bothersome or difficult, or between 10% and 14% if considered conceptually important by the research team, (2) clinician endorsement of at least 20%, unless it was an aspect of using an injection pen with which they may have lacked direct experience (e.g., pen jamming or needle bending), (3) analyst/interviewer rating of whether the aspect of using an injection pen was important to patients based on the qualitative findings and/or interviewer impressions, (4) analyst/interviewer rating of whether the aspect of using an injection pen was important to using it properly, and (5) the pen feature or characteristic was not a distal impact. To confirm the relevance, clarity, and appropriateness of the DPEM items, a CD was conducted in an independent sample of adults with diabetes who were using at least one injection pen for treatment of their diabetes and met the same eligibility criteria as the focus-group sample. Based on the CD interviews, the validation-ready version of the DPEM was developed. CDs were conducted in blocks of respondents who were interviewed individually. Edits were made between each block based on comments made during the interviews. The interviews followed a semi-structured interview guide with questions about the instructions, items, response options, formatting, and amount of time required to complete the DPEM employing a “think aloud” method, as well as verbal probing as needed.

Psychometric validation of the DPEM was conducted in 3 stages. The web-based validation study was a noninterventional survey in patients with type 1 or type 2 diabetes who were using an injection device to administer their diabetes medication, were aged 18 years or older, and had Internet access. A survey battery was developed for this web-based survey that included questions relating to demographic and disease characteristics, the DPEM measure, and other PRO measures needed to validate the DPEM (Treatment Related Impact Measure for Diabetes—Device [TRIM-DD], Diabetes Injection Device Experience Questionnaire [DID-EQ], and Diabetes Treatment Satisfaction Questionnaire (status version) [DTSQ]). These PRO measures were used with permission from the respective copyright holders. A noninterventional, in-person usability study in patients with type 1 or type 2 diabetes of the web-based validation survey was conducted prior to the validation study to ensure that the web survey functioned properly and was acceptable to patients. The analyses used to evaluate the performance of the DPEM items were in accordance with classical psychometric theory, and modern measurement theory (ie, Rasch measurement theory [RMT]) methods. DPEM items were evaluated using information from the following analyses: floor effect (when too many patients endorse the worst response option, which can then only improve) and ceiling effect (when too many patients endorse the best possible response option, which can then only get worse), item-to-item correlations, item-to-total correlations, factor analyses, reliability estimation, and item parameters from RMT analyses. Given the study design (cross-sectional, web-based survey), test–retest reproducibility, and sensitivity to change were not assessed. All statistical tests used a significance level of 0.05 (2-sided) unless otherwise noted. Statistical tests involving multiple comparisons (eg, analysis of variance [ANOVA] models with multiple groups) included Scheffé post hoc tests, which adjust for multiple comparisons and reduce the possibility of type I error. Statistics were conducted using SPSS Statistics software (IBM, Armonk, NY). Convergent validity was assessed by examining the magnitude of correlations between the hypothesized domain items and the TRIM-DD, DID-EQ, and DTSQ overall items of satisfaction, ease of use, and convenience. Convergent validity is supported when the measure scores are substantially correlated (≥0.40) with items or measures evaluating similar concepts. Additionally, the final single-domain DPEM total score was used to examine correlations with the a priori hypothesized domain relationships. To assess known-groups validity (the ability of the scale to discriminate between expected known relationships), the following hypotheses were tested using ANOVA tests at a P < .05 level: H01, those who are using an injection pen compared with those using a vial/syringe will have a better DPEM total score, H02, those who (1) use pens which require greater preparation, (2) take longer to prepare the pen and inject their medication will score worse on the DPEM “ease of use” domain, and H03, those who (1) have had diabetes for a longer duration, (2) consider themselves experienced pen users, (3) are well trained to use their injection device will score better on the DPEM “confidence” domain.

Five healthcare providers from across the U.S.participated in the CE interviews: 2 endocrinologists, 1 primary care physician, and 2 diabetes nurse educators. Three were female and 2 were male. The providers had been practicing their respective disciplines for a mean of 13.8 (range: 5-18) years and reported seeing an average of 158 patients with diabetes per month (range: 90-200 patients). Three worked in a private practice/outpatient clinic, 1 in a hospital, and 1 at a university-based practice. In total, 42 patients participated in CE interviews; 32 participated in the 4 focus groups, and 10 participated in one-to-one in-depth telephone interviews. The demographic characteristics of these participants are shown in Table 1. Nine major themes emerged from the analysis of the clinicians and patients interviews: (1) learning to use the pen, (2) preparing the pen, (3) injecting the dose, (4) disposal, (5) general pen characteristics and features, (6) impact on diabetes management, (7) quality of life when using an injection pen, (8) psychological impact of using an injection pen, and (9) costs associated with using an injection pen. For each theme, major subconcepts and endorsements are presented in Supplementary Tables 1 to 9. Thematic saturation, defined as the point when no additional meaningful themes had emerged, was assessed for the 4 focus groups and the 10 patients telephone interviews in the order in which they occurred. After the second focus group, 88% of the concepts had been discussed, and by the third focus group, 97% of the concepts had been captured. Based on the qualitative data analysis, a preliminary theoretical model was developed, and items for the DPEM were generated to reflect the areas of importance for the functions and features domain as well as for proximal impacts. The preliminary version of the DPEM contained 30 items.

In total, 14 adults participated in the one-to-one CD interviews. Nine were male (64.3%) and the mean age was 43.5 (range: 19-59) years (Table 1). The individual CD interviews were divided into 3 blocks. The first 2 blocks included 3 patients each and the third included 5 patients, to ensure adequate representation of different types of injectable devices. Three additional interviews were subsequently conducted in a fourth block to refine the instructions. Through these brief, confirmatory interviews, DPEM items were refined to improve readability and relevance, resulting in a reduction of the number of items, and minor changes were made to the instructions. After the CD, a decision was made to add two more overall items to the DPEM so that each domain had a corresponding overall item. The added items were “Overall, how confident are you that your pen functions properly?” and “Overall, how confident are you that you use your pen properly?”. The validation-ready version of the DPEM had 28 items and reflected three domains: ease of use, confidence, and overall.

A review of the preliminary data from an initial validation study revealed floor and ceiling effects for several items. Floor and ceiling effects were defined as a frequency of more than 50% for “not at all” or “extremely” responses, respectively. The following issues were identified as probable causes for the floor and ceiling effects: (1) some items had low endorsement rates, (2) some participants endorsed certain concepts as “bothersome,” but relatively small proportions further reported them to be difficult, (3) some concepts were not relevant to all pens, and (4) some concepts pertained to issues that occurred infrequently.

Based on these preliminary findings, the qualitative CE data were reevaluated, and the research team held a second item-generation meeting to review the findings. Subsequently, the DPEM was refined to eliminate the floor and ceiling effects. Specifically, the minimum endorsement rate required for inclusion of items about using an injection pen that the patients found bothersome or difficult as a major concept was raised from 15% to 25% of the patients. Items that did not meet this new endorsement rate were either deleted or moved into a single higher-order concept. Minor revisions were also made to the instructions. The validation-ready revised version of the DPEM had 7 items and underwent a CD with 12 adults; 6 were male (50.0%) and the mean age was 48.0 (range: 20-72) years (Table 1). Following the CD, no further modifications to the wording of the items were found to be necessary and only minor revisions were made to the instructions. In addition, the term “injection pen” was changed to “injection device” because the questions were considered relevant to vial and syringe users. The validation-ready, revised version of the DPEM consisted of 7 items and reflected 3 domains: ease of use, confidence, and overall. Each of the items listed in the DPEM contains a 5-point Likert-type response scale that differs depending on the type of question.

Development of the DPEM continued with further quantitative testing and evaluation from a noninterventional, web-based survey study. The psychometric analysis process included developing the PRO validation battery, conducting usability testing of the PRO battery, conducting the psychometric web-based validation study, and refining and finalizing the DPEM. In total, 12 adults participated in usability-testing interviews; 7 were male (58.3%) and the mean age was 61.0 (range: 27-77) years. Understandability and relevance of the “injection device” term incorporated after refinement of the DPEM were also assessed. After changes to the instructions by adding in parentheses “pen or vial and syringe,” consensus was reached among interviewees that all sections in the DPEM were clearly worded, understandable, and relevant to the participants’ diabetes treatment. In total, 300 patients in the U.S. with type 1 or type 2 diabetes participated in the web-based survey. The mean age was 44.7 (range: 19-83) years, and 66.3% were female (Table 1). The item statistics were all adequate. Based on the RMT analysis, 2 items were disordered, that is, the threshold values between adjacent pairs of response options were disordered by magnitude. One of these items had a ceiling effect of 53.7%. The distributions for the other items were as expected. Item-to-item correlations were good, with 2 similarly framed items (ease of use) having correlations of greater than 0.70 (Table 2). In addition, item-to-total correlations displayed acceptable associations between each item against the rest of the items as a total score (excluding that item), with correlations ranging from 0.68 to 0.79 (Supplementary Table 10). Finally, the internal consistency of the DPEM was adequate, with a Cronbach’s alpha of 0.91. The original conceptual framework indicated the potential to have domains of “ease of use,” “confidence,” and “overall.” However, factor analyses resulted in a single component representing overall pen experience. The resulting single factor was then referred to as the “DPEM total score.” This score was transformed to facilitate interpretation, and the DPEM items and total score were computed so that a higher score indicated a better pen experience. The DPEM is scored by summing the 7 items and transforming the sum onto a 100-point scale using the following formula: The distribution of the DPEM total scores was similar to that of the other PROs assessed in the validation battery (Supplementary Figure 1), and the scores tended to be in the high range of the scale, indicating an overall positive experience. Lastly, all convergent (Table 3) and known-groups validity hypotheses (Supplementary Figures 2–9) were supported. The resulting final theoretical model of the DPEM is presented in Figure 1, and the final conceptual framework is presented in Figure 2.

Treatment adherence in patients with type 1 or type 2 diabetes is crucial to achieve adequate glycemic control and improvements in long-term clinical outcomes. Given the association between insulin treatment adherence and patients’ satisfaction with injection devices, it is essential to have a reliable tool to accurately assess patients’ experiences with specific treatments, dose regimens, and injection devices. Understanding patients’ experience is pivotal to improving the simplicity and convenience of therapies and injection devices and, ultimately, to maximize the success of treatments. The newly developed PRO measure DPEM was designed to assess patients’ experiences with most pens for the administration of glucose-lowering drugs. It was developed and validated following the U.S. FDA guidance for the development of PRO measures. The final DPEM contains 7 items and a single domain. The frequency distribution of the scores for each item largely revealed acceptable floor and ceiling effects (Supplementary Table 10). The validation study produced a satisfactory Cronbach’s alpha coefficient of reliability for the DPEM total score. The validity of the DPEM total score scale as well as each individual item was confirmed using well-established PRO measures. The DPEM was able to discriminate between pen versus vial/syringe use, length of time to prepare and administer the injections, length of time with diabetes, experience and training in the use of an injection device, and satisfaction with a device. Although there are some overlapping items in the DID-EQ, TRIM-DD, and the DPEM, those PRO measures differ in several aspects; as such they might be used differently or in distinct patient populations in clinical practice. For example, the DID-EQ measure was developed in patients receiving non-insulin injectable medication, whereas the DPEM was developed in patients receiving any types of injectable glucose-lowering medication. Furthermore, the DID-EQ contains items on issues that the patient population in the present study did not find difficult or bothersome (such as portability and needle size) and also includes items that are related to treatment rather than device. Finally, the response scales and the way the “Overall” questions are phrased vary between measures. In the DPEM, patients are directly asked how easy or difficult and how convenient it is to use their device, and how satisfied they are with their device. In contrast, in the DID-EQ, patients are asked to rate their level of agreement with statements indicating their device is easy to use, convenient, and that they are satisfied overall. The results presented here should be considered in the context of study limitations. As this was a cross-sectional, web-based study, test–retest reproducibility, and sensitivity to change could not be assessed and will have to be assessed in future validation studies. Given the brevity of the DPEM and the important concepts contained within the items, all items must be completed to compute the total score.

The cumulative evidence on content and construct validity of the DPEM gathered from the CE and the psychometric validation study phases of this research supports the use of the DPEM in clinical trials for diabetes to assess the patient’s experience with diabetes injection devices.

Click here for additional data file. Supplemental material, sj-docx-1-dst-10.1177_19322968221079396 for Development and Validation of the Diabetes Pen Experience Measure (DPEM): A New Patient-Reported Outcome Measure by Meryl Brod, Laura Tesler Waldman, Thomas Sparre and Anne Kirstine Busk in Journal of Diabetes Science and Technology