Empowering smokers with a web-assisted tobacco intervention to use prescription smoking cessation medications: a feasibility trial

Peter Selby
Sarwar Hussain
Sabrina Voci
Laurie Zawertailo
Empowering smokers with a web-assisted tobacco intervention to use prescription smoking cessation medications: a feasibility trial
10.1186/s13012-015-0329-7R E S E A R C H
Background: Varenicline and bupropion, efficacious smoking cessation medications, have had suboptimal impact due to barriers at the patient, practitioner and system level. This study explored the feasibility of a web-assisted tobacco intervention offering free prescription smoking cessation medication by mail if the smoker visited a physician for authorization. Methods: Adult Ontarians, smoking at least 10 cigarettes daily, intending to quit within 30 days, with no contraindications to bupropion or varenicline were eligible. After an online assessment, eligible participants received an electronic personalized printable prescription form for a 12-week course of varenicline or bupropion to bring to a physician within 3 weeks for authorization, if appropriate. The physician's office faxed prescriptions to an online pharmacy that couriered medication to the patient following medication counselling by telephone. Weekly motivational emails were sent during treatment. Participants were asked to complete follow-up questionnaires online at 7, 11, 15 and 41 weeks after enrollment.Results: In total, 1214 individuals submitted an online assessment from April to September 2010 and 73.6 % (95 % confidence interval (CI) = 71.1-76.1 %; n = 893) were eligible. At least 65.8 % (95 % CI = 62.7-68.9 %; n = 588) of eligible participants subsequently visited a physician and 58.7 % (95 % CI = 55.5-61.9 %; n = 524) received medication (50.6 % varenicline [n = 265] and 49.4 % bupropion [n = 259]). Reasons for not filling a prescription were failure to visit a physician (80.1 %; 95 % CI = 73.8-86.5 %; n = 121), physician not prescribing the medication (15.9 %; 95 % CI = 10.1-21.7 %; n = 24) or other reasons (4.0 %; 95 % CI = 0.9-7.1 %; n = 6). Follow-up response rate was 66.7 % (95 % CI = 63.7-69.8 %; n = 596). Minimal issues were encountered with printing the prescription or medication delivery. Conclusions: This study establishes the feasibility of using the Internet and free medication to enable smokers to engage physicians to treat this addiction. Implementation of this intervention can be scaled up by leveraging existing healthcare systems to treat smokers on a population level. Further evaluation in a randomized controlled trial is necessary.Trial registration: ClinicalTrials.gov Identifier NCT01023659

Background

Each year, approximately six million people worldwide die prematurely due to tobacco-related diseases [1]. For each person who dies from a smoking-related disease, approximately 30 more people have at least one serious medical condition caused by smoking [2]. The majority of smokers want to quit [3,4] but find it difficult [5]. Each year, close to half of smokers in North America make a quit attempt [4,6], but without assistance only 3-5 % achieve long-term abstinence [7]. Several smoking cessation medications have been shown to improve long-term abstinence rates [8,9], but despite their proven efficacy, approximately two-thirds of smokers report not having used medication in their most recent quit attempt [10].

Barriers to the use of these medications exist at the patient, system and practitioner level. Patient-level barriers include poor awareness, perceived lack of effectiveness, low desirability including concern regarding adverse effects, and poor accessibility of treatment [11,12]. Accessibility issues include anticipated difficulties obtaining a prescription from a physician and perceived high cost of medication [11]. Full coverage of the cost of smoking cessation medications has been proven to increase utilization and improve quit rates [13,14]. Advice by a healthcare provider to quit smoking has also been associated with increased use of smoking cessation treatments [15]. Though evidence-based clinical practice guidelines recommend that every smoker be advised to quit and assisted if ready to make a quit attempt [16,17], a recent Canadian survey found that of the smokers who visited a physician in the previous 12 months, only 56 % received advice to quit and 34 % received information about smoking cessation aids [4].

Passive dissemination of guideline recommendations is generally ineffective in changing practitioner behaviour, and more intensive (and costly) training is generally necessary to alter practice [18]. Strategies to promote implementation of smoking cessation guidelines are most often targeted to clinicians [19]. Healthcare professionals who have received training for delivery of smoking cessation interventions are more likely to perform cessation-related tasks such as asking patients to set a quit date, offer counselling and provide self-help materials [20]. However, a Cochrane review found no demonstrated effect of training on provision of smoking cessation medications [20]. Therefore, new methods are required to enhance the prescribing of these evidencebased medications, which almost double to triple the odds of long-term cessation [8]. There is evidence that interventions targeted to patients can also change clinician practice and improve care [19,21]. Patientmediated interventions are delivered to patients with the aim of changing the attitudes, knowledge, skills and/or behaviour of the healthcare provider via patient-provider interaction [22]. Empowering patients to take an active role in their treatment in this manner may help foster greater autonomous motivation to use smoking cessation medications, which is associated with increased medication use, and in turn, higher levels of abstinence [23].

Large-scale distribution of free over-the-counter nicotine replacement therapies (NRT) mailed to smokers wanting to quit has had demonstrated success and reduces the barrier of geographic access to health services [24,25]. However, regulatory restrictions require a visit to a prescriber to obtain a prescription for bupropion or varenicline. The current study was conducted to test the feasibility of a web-assisted tobacco intervention designed to increase the use of prescription smoking cessation medications by having patients prompt a physician to prescribe medication. Smokers motivated to quit were directly engaged online and were required to visit their physician within 3 weeks with a pre-populated form that was converted to a prescription upon physician signature in order to receive a free 12-week supply of bupropion or varenicline, mailed to them by an online pharmacy. The offer of free medication motivated the patient to initiate the dialogue with their physician. The online pharmacy was able to courier medication to any jurisdiction to increase the reach of the intervention and make it more accessible to those living in rural and remote areas. Given that this design is innovative and involves multiple steps with different providers, it was important to establish feasibility prior to conducting a larger-scale randomized clinical trial [26,27]. We hypothesized that this intervention would be a logistically feasible approach to provide prescription medication to a large number of smokers over a wide geographic area.

Methods

Study design

An open-label trial was conducted to test the feasibility of distributing by mail a free 12-week supply of bupropion SR (Zyban) or varenicline (Champix, Chantix) to smokers following a visit to a physician of their choice for authorization of the prescription, in conjunction with weekly motivational emails and with a 41-week follow-up period to assess quit outcomes. The study was approved by the Research Ethics Board at the Centre for Addiction and Mental Health and was registered as a clinical trial (ClinicalTrials.gov Identifier NCT01023659).

Participant recruitment

Adult smokers from Ontario, Canada, interested in quitting could enroll at the study website (www.stopstudy.ca; see Additional file 1). Study information, including an electronic copy of a recruitment poster that could be printed and posted, was provided to healthcare professionals from public health units, community health centres, family health teams and community pharmacies across Ontario. Physicians were also informed about the study via the Ontario College of Family Physicians newsletter.

Inclusion criteria required that participants be 18 years of age or older, residents of Ontario, smoke at least 10 cigarettes per day, had smoked daily for at least the past 3 months, had smoked at least 100 cigarettes in their lifetime and intended to quit smoking within the next 30 days. Exclusion criteria were self-reported: history of brain injury or seizure(s); pregnancy, lactation or risk of becoming pregnant; allergy to bupropion or varenicline; current use of monoamine oxidase inhibitors (MAOIs), thioridazine, varenicline or bupropion; severe liver impairment; or history of anorexia and/or bulimia. The inclusion criteria were to mimic efficacy trial inclusion criteria. The exclusion criteria were those listed as absolute exclusions in the product monograph for bupropion and varenicline at the time of the study.

Sample size

The number of participants enrolled in each group was determined by the quantity of each medication available for this study, considered sufficient for achieving the primary goal of determining the logistic feasibility of the proposed distribution method. With a sample size of approximately 500, we could estimate a physician visit rate of 50 % to within a 95 % confidence interval of ±4 %.

Intervention

A diagram outlining the participant flow is provided in Fig. 1.

Safety considerations

A number of checks were incorporated into the design to ensure that distribution was executed safely. The first check was excluding participants who reported any one of several contraindications to either medication during the initial online assessment; this also minimized resources from being wasted due to unnecessary physician appointments. Should an individual have met eligibility criteria by either deliberately or inadvertently submitting an incorrect response, the physician acted as a secondary check to screen for contraindications prior to prescribing the medication. Finally, the pharmacist acted as third safety check by screening for potential drug interactions and fraudulent prescriptions.

Online self-assessment and automated eligibility determination Prior to being asked to provide any personal information, participants were required to provide informed consent by selecting the "Yes" option to indicate their agreement with the statement of consent. The website presented frequently asked questions and contact information of study personnel if individuals had additional questions about study participation. The initial assessment questions were designed to assess eligibility and collect information on demographic and baseline smoking characteristics. Participants were required to provide an email address at which they could be contacted regarding their eligibility and for follow-up. Individuals were not able to submit more than one assessment using the same email address. Upon submitting the initial assessment, an automated process informed participants via email whether or not they were eligible. Those not eligible were provided with a list of smoking cessation resources they could access for assistance with quitting.

Personalized prescription form

Eligible smokers were immediately sent two documents via email (see Additional file 2) with instructions to print and bring to their physician within 3 weeks of study enrollment, after which the prescription would be void. We estimated that 3 weeks would allow time to get an appointment with a physician and still receive medication within the 30-day period during which participants were planning to quit. The documents were as follows:

(1) a letter to the physician that provided information about the study and (2) a personalized prescription form for study medication. The patient name and address fields were automatically filled in on the prescription form based on information entered in the initial assessment. The prescription form provided the option to select either bupropion (150 mg once daily for 3 days, then 150 mg twice daily for the remainder of 12 weeks) or varenicline (0.5 mg once daily for 3 days, 0.5 mg twice daily for 4 days, then 1.0 mg twice daily for the remainder of 12 weeks). Information about the potential side effects and risks of each medication were presented to the participant prior to them providing informed consent. Given that the quantity of each medication was limited, when the supply of one type of medication was depleted, the option of selecting either medication was removed from the website and on the prescription form. The email to eligible participants also contained personal login information for the study website where they could access both documents at any time.

Patient visit to physician

In order to receive medication, participants were required to schedule a visit with a physician of their choice to whom they would provide both study documents for review, clarification and final determination of eligibility. The letter explicitly stated that the physician had full discretion to prescribe either medication to the patient and that "the study intends to fully defer to the patientdoctor relationship and thus leave the patient under your clinical care" (see Additional file 2). There were no incentives provided to the physician but they were free to bill the public healthcare system (Ontario Health Insurance Plan) for the visit as per their respective eligibility and compensation model. To prescribe either medication, the physician could checkmark the appropriate box on the prescription form to indicate the type of medication (if a choice was still available) and authorize the form by providing the date, their name, signature and registration number. If either medication was prescribed, the physician's office was required to fax the signed prescription form to the online pharmacy (www.pharmacy.ca). The form included explicit instructions and the pharmacy name, fax and phone number.

Online pharmacy

The pharmacist verified the authenticity of the prescription and contacted the physician's office if any of the information on the form was incomplete or missing as per their usual prescription verification practices. In accordance with the Ontario College of Pharmacists' standard of practice, each participant was contacted by a pharmacist when dispensing medication to discuss possible allergies, concomitant medications and provide relevant medication counselling. Participants were cautioned regarding the possibility of drowsiness and to avoid driving or operating machinery until they were reasonably certain that the medication did not affect their mental alertness or physical coordination. The pharmacist asked the  participant to confirm their address and then couriered the medication to the participant directly. An online pharmacy allowed participants to receive their medication without the additional step of visiting a pharmacy in person for dispensing, thereby eliminating a potential barrier.

Behavioural support

All participants deemed eligible after completing the initial assessment, regardless of whether they received medication or not, were sent a weekly motivational email for 12 weeks. Emails commenced at 3 weeks after study enrollment, the date by which prescriptions had to be submitted for medication dispensing. See Table 1 for examples of email messages sent. Each medication was accompanied by a self-help booklet provided by the manufacturer that included smoking cessation tips and how to access web-based or telephone counselling support.

Participant follow-up

Collection of follow-up data was attempted via email for all participants at 7, 11, 15 and 41 weeks after enrollment. We estimated that medications would be started approximately 3 weeks after enrollment. Monthly contact for the 12-week period thereafter was to monitor any side effects and assess quit success. Follow-up at 41 weeks after enrollment (approximately 6 months after the end of treatment) was to assess long-term quit success. At each follow-up, participants were screened for suicidal ideation because of post-marketing concerns regarding both bupropion and varenicline. Participants who screened positive were advised to stop their medication and contact their physician immediately or report to their nearest Emergency Department. One reminder email was sent to non-respondents. Participants were also provided a telephone number with the option of completing the survey via telephone.

Outcome measures

The primary feasibility outcome was that at least 50 % of eligible smokers planning to quit within 30 days would report being able to schedule an appointment with a physician before the prescription expired at 3 weeks after enrollment. A 2008 survey found that 68 % of all adults in Canada report being able to get an appointment with a physician within 7 days and 89 % within 30 days for routine care [28]. Based on this finding, we estimated that at least 75 % of participants would be able to schedule an appointment with a physician within 3 weeks after enrollment. Taking into consideration that potential delays in calling the physician's office (e.g. varying levels of motivation and failure to remember) and conflict with other responsibilities and priorities (e.g. employment and childcare) may lower this number, we set 50 % as the threshold for feasibility. A low Creating a smoke-free environment is important during your quit attempt. Make a decision not to smoke in your home and vehicle and ask others to do the same. If your entire home cannot go smoke-free, explore areas where you can restrict smoking. At work, avoid smoking areas during your breaks. Making your physical environment smoke-free can help reinforce your decision to quit smoking.

2 Support systems are important during any big change. Identify all of the positive supports in your life and tell them you are quitting smoking and need their support. Also identify any negative influences who may not want you to quit and figure out how you are going to deal with them during this time. Take advantage of other supports available to you, such as Smoker's Helpline, websites, your doctors or other healthcare providers. Surrounding yourself with positive and supportive people can help you quit and stay quit.

3

Slips and lapses are a part of the quitting process and can be common. Use any slip or lapse as a learning experience. Identify what happened, how you could have prevented the situation, and what you can do if you're in the situation again. Use these experiences to re-assess your quit plan and then try quitting again. It is important that you realize your quit attempt is not over; refocus and restart immediately after your lapse. Remember, quitting smoking is a process not an event and may take several attempts before you get it right.

4

One of the benefits of quitting smoking is the amount of money you save. The price of a pack of cigarettes is about C$8; so, that means if you smoked about 15 cigarettes every day you would save about C$570 in 3 months (enough to purchase a new 32-inch flat-screen TV) or save C$2180 in 1 year (enough for a long vacation abroad). Therefore, take advantage of quitting smoking and reward yourself. 6 Your smoking may be associated with certain people, places, or things. These can act as triggers for you to want to smoke. Identify your personal triggers and think about how you will deal with them. For example, change your day-to-day routine or find alternative activities to smoking. Problem-solving ahead of time can help you deal with these situations when they arise and help you quit and stay quit. 8 There are many good reasons why people want to quit smoking. Sometimes it's easy to forget why you wanted to quit in the first place. Write down your personal reasons for quitting and use them as reminders when things seem tough. Your reasons may change over time so review your list regularly. Reminding yourself of all the reasons you want to quit can help you stay focused on achieving your goal.

proportion of physician visits was considered acceptable in light of the negligible costs associated with enrolling and following up with participants who do not visit a physician. The following cut-off points were also set in order to establish the logistic feasibility of the distribution method prior to proceeding with a larger study: (1) ability to recruit a sample of approximately 500 participants within 6 months without using paid advertisements, based approximately on the recruitment rate from a study we conducted distributing free NRT that required at least one visit to a pharmacy [29], assuming one quarter as many individuals would be interested in prescription cessation medications compared to NRT [30] and a lower eligibility rate due to a greater number of contraindications associated with bupropion and varenicline; (2) eligibility rate of at least 70 %, similar to comparable randomized trials comparing bupropion and varenicline [31,32]; (3) at least 45 % of eligible participants would receive medication, based on the assumption that at least 50 % are able to visit a physician, and of these, almost all would receive a prescription due to the exclusion of those with contraindications at the initial assessment; and (4) no more than 7 days would be typically required to reach the participant via telephone for medication counselling and to confirm shipping address, operationalized as the median number of days between filling the prescription and sending the medication out for delivery, permitting most participants to receive medication within 30 days. Additional feasibility data collected included reasons for not scheduling a visit with a physician, frequency of medication delivery issues, and the average number of days between study enrollment and prescription dispensing and delivery. Logistic feasibility outcome data were collected from participants at each follow-up survey (to maximize the probability of having feasibility data for each participant, given loss to follow-up) and also extracted from pharmacy records. Where a participant provided conflicting responses at different follow-up time points with respect to feasibility outcomes, the earliest response is reported based on the assumption that recall would be less accurate as time elapsed.

Acceptability of the motivational email component of the intervention was assessed by asking participants at each follow-up throughout the 12-week treatment period to rate the helpfulness of the emails on a fivepoint scale from 1 (not helpful at all) to 5 (very helpful). Responses provided later in treatment replaced ratings provided earlier in treatment, where applicable. Participants were also asked throughout and at the end-oftreatment follow-up how much of the 12-week supply of medication they had used. This was done because completion of treatment is a predictor of successful quitting [33]. Participants were asked to select from several response options provided, which were collapsed into a smaller number of categories for analysis (none, less than 1 week, 1 to less than 4 weeks, 4 to less than 8 weeks, 8 to less than 12 weeks, all 12 weeks). As above, responses provided further along in treatment replaced earlier responses, where applicable.

Treatment outcome was assessed in order to obtain an estimate of treatment effect to inform a sample size calculation for a future randomized trial. The outcome variable related to the effectiveness of the intervention was 7-day point prevalence of abstinence from smoking at 41 weeks after enrollment, approximately 6 months after the individual completed their 12-week course of medication. Seven-day point prevalence abstinence was defined as not having smoked even a puff over the previous 7 days.

Statistical analysis

Comparisons on baseline variables were made using chisquare tests of association for categorical variables and t tests for continuous variables to detect any differences between participants who received medication and those who did not, as well as between those who received bupropion versus varenicline. Respondents and participants lost to follow-up at 41 weeks were similarly compared on baseline characteristics to identify potential attrition bias, and a comparison on baseline demographic characteristics was made for participants who were able to visit a physician versus those who were not. A one-sample chi-square test was used to compare the observed versus expected frequencies of smokers in each geographic region based on population estimates derived from the 2009-2010 Canadian Community Health Survey [34].

Data screening prior to analysis revealed that the proportion of missing data values at baseline was highest for annual household income (8 %) and employment status (3 %); all other baseline variables had less than 1 % missing data. Cases with missing data were deleted pairwise in all bivariate analyses. To attempt to minimize attrition bias, multiple imputation was used to address the missing data for the analysis of the primary smoking cessation outcome at 41-week follow-up; findings are presented alongside a complete case analysis. Multiple imputation by chained equations was implemented using Stata 12 (mi impute chained) to generate 20 imputed datasets, with a burn-in period of 10 iterations. An inclusive strategy was used to impute data in an attempt to maximize efficiency and reduce bias [35]; thus, additional auxiliary variables from the baseline questionnaire were included in the imputation model (e.g. number of cigarettes per day, the longest previous quit attempt, education level and employment status). Multiple imputation analyses were conducted with Stata 12 [36]; all other analyses were conducted with SPSS 15.0 [37].

Level of significance was set at p < 0.05, with Bonferroni adjustment to correct for multiple comparisons on baseline variables.

Results

Participants and recruitment

A total of 1214 unique individuals submitted an initial assessment between April and September 2010 (see Fig. 1). To manage and ensure availability of the limited supply of medication, enrollment was closed periodically for varying lengths of time to monitor how many eligible participants were filling prescriptions for each medication. Enrollment was highest in the first few months, Participant baseline characteristics are presented in Table 2. Participants who received either medication were older, had been smoking daily for longer and were less likely to report that all or most of their friends smoked. There were no statistically significant differences at baseline between participants who received bupropion versus varenicline. The proportion of the sample from each geographic region differed significantly from the proportions expected based on population estimates of smoking prevalence in each region [34], χ 2 (df = 4) = 167.02, p < 0.001. In particular, the proportion of smokers from the city of Toronto was lower than expected ( Table 3). Respondents were older and had a higher annual household income; while they had been smoking daily for a greater number of years, they also reported longer periods of previous abstinence from smoking, reported a longer time to first cigarette in the morning and had a smaller proportion of friends who smoked.

Feasibility and acceptability outcomes

Among the entire sample with follow-up data, 79.7 % (95 % CI = 76.1-83.3 %; n = 475) were able to get an appointment with a physician, 5.9 % (95 % CI = 3.8-8.0 %; n = 35) chose not to see a physician and 14.1 % (95 % CI = 11.0-17.2 %; n = 84) were unable to get an appointment with a physician in the required 3-week time period. An additional 113 participants without follow-up data who received medication could be assumed to have visited a physician. Thus, at least 588 of all 893 eligible participants (65.8 %; 95 % CI = 62.7-68.9 %) visited a physician, though a somewhat higher proportion is probable due to nonresponse at follow-up by participants who had visited a physician but did not receive medication. Excluding those who chose not to make an appointment, those unable to visit a physician did not significantly differ from those who were able to visit a physician on any demographic characteristic except age, with those unable to get an appointment being younger than those able to get an appointment, 38.35 (SD = 12.11) vs. 44.14 (SD = 12.38), p < 0.001.

Of those eligible for study medication, 58.7 % (95 % CI = 55.5-61.9 %; n = 524) received either prescription medication. Almost equal proportions received varenicline (50.6 %; n = 265) or bupropion (49.4 %; n = 259). All medication packages were successfully delivered, with only 1.1 % (95 % CI = 0.2-2.1 %; n = 6) requiring address clarification or reshipment. When a choice of medication was offered, participants chose varenicline (85.1 %; 95 % CI = 80.0-90.1 %; n = 165) more often than bupropion (14.9 %; 95 % CI = 9.9-20.0 %; n = 29). Once the available supply of varenicline was depleted at the end of May, only bupropion was available.

Among those who received medication, 6.3 % (95 % CI = 4.2-8.4 %; n = 33) were initially ineligible but had altered their responses on resubmission to subsequently become eligible. The most frequent reason (more than one could apply) among these individuals for being ineligible on original submission was not planning to quit within 30 days (54.5 %; 95 % CI = 37.6-71.5 %; n = 18), followed by contraindication to one of the medications (27.3 %; 95 % CI = 12.1-42.5 %; n = 9), not currently smoking daily and/or at least 10 cigarettes per day (15.2 %; 95 % CI = 2.9-27.4 %; n = 5) and technical error (6.1 %; 95 % CI = 0.0-14.2 %; n = 2). Contraindications initially reported included an allergy to either medication, severe liver impairment, history of brain injury and current use of bupropion, varenicline, thioridazine or an MAOI.

Follow-up data were not available and therefore reasons for not receiving either medication were unknown for 59.1 % (95 % CI = 54.1-64.1 %; n = 218) of the 369 participants that did not receive medication. Of the 151  Table 4. The most common reason reported was not being able to arrange an appointment within 3 weeks. A small number of patients did not have a family physician or were unable to access or print the prescription form. While most patients who did not visit a physician were unable to, some voluntarily chose not to visit a physician, such as those who changed their mind about quitting or were not interested in bupropion once varenicline was no longer available. The median number of days between study enrollment and having a prescription filled was 10 (IQR 5-16) and

for having it sent out for delivery was [14][15][16][17][18][19][20][21]. A median of 2 days (IQR 1-5) was required after filling the prescription to reach the participant for medication counselling and to confirm shipping address.

The amount of the medication supply used was reported by 57.6 % (95 % CI = 53.4-61.9 %; n = 302) of participants that received medication, who reported using the following amounts: none, 2.6 % (95 % CI = 0.8-4.5 %; n = 8); less than 1 week, 4.0 % (95 % CI = 1.8-6.2 %; n = 12); 1 to less than 4 weeks, 27. Based on an informal collection of data by the pharmacy that was reported to investigators at the end of the study, several issues arose that the pharmacist needed to resolve in order to fill a prescription, which had the potential to cause delays. Such issues included the following: need to follow-up with physicians at the request of patients when signed prescriptions were not faxed to the pharmacy, additional counselling required when physicians did not provide information to their patient regarding the medication and needing to call a physician for recommendation when a contraindication was reported by the participant. Though a precise frequency of these occurrences was not provided, the average processing time suggests that these issues did not regularly cause extended delays. The pharmacy also reported having received expired prescription forms that could not be filled and the form was revised to clearly list the expiry date. Several patients (less than 10) called the pharmacy to request the contact information of a physician that would participate in the study because their physician had declined to participate. These patients were referred to the College of Physicians and Surgeons of Ontario to find physicians accepting new patients.

Cost analysis

The cost per patient that received medication was C$409.69. The cost per quitter ranged from approximately C$1003 (based on an overall quit rate of 24.0 % at 41 weeks among respondents only) to C$1835 (based on a quit rate of 13.1 % assuming non-respondents at follow-up were still smoking). The cost increased by approximately C$33.70 for each patient seen by a physician if they billed the government insurance plan for their services.

Discussion

The current study establishes the feasibility of using a patient-driven model that engages smokers directly for Table 4 Reasons reported at follow-up for not visiting a physician to discuss prescription for study medication Reason n Percentage (95 % CI) of those that did not visit a physician (n = 121)

Unable to see physician

Could not get an appointment within 3 weeks delivery of evidence-based smoking cessation treatment including prescription pharmacotherapy and behavioural intervention. The majority of participants learned about the study through family and friends or formal communication about the study targeted to participants (e.g. posters and study website) rather than through healthcare providers, demonstrating that the decision to use and prescribe medication to quit smoking was often prompted by the patient themselves. To our knowledge, few physicians declined to participate and only a small proportion of participants who visited a physician were not prescribed the medication, suggesting a high level of acceptability of the intervention among physicians. This also suggests that screening and excluding participants based on contraindications to either medication successfully reduced unnecessary visits to the physician by those for whom the medication was not suitable. A subsample of participants reported having learned about the study from a physician, though it is unknown if this was due to study recruitment efforts directed at healthcare providers or if the impact on practice extended such that physicians who had heard about the study from patients then recommended medication to other patients. Further research is needed to determine to what extent this type of patient-mediated intervention has the capacity to change physician practice across patients and over time.

The intervention was able to reach participants from a wide range of communities across Ontario, and the only region appreciably under-represented was the city of Toronto, possibly due to a higher level of access to services. The majority of the sample that responded to follow-up was able to visit a physician to have the prescription signed within the required time period. The only demographic characteristic predictive of whether an appointment with a physician could be scheduled was age; greater difficulty getting an appointment may be due to lower rates of having a regular medical doctor among younger Canadians [38]. Very few technical difficulties were reported with accessing or printing the prescription, pharmacists were able to reach participants by telephone in a timely manner for medication counselling, and all medication packages were successfully delivered. The amount of medication used varied, though reasons for using less than the 12-week supply was not available. While some likely stopped the medication due to adverse effects or perceived lack of efficacy, some participants may have started the medication later than the study's assumed start date for treatment and may ultimately have used the entire supply.

For the most part, safety checks were successful and the majority who reported contraindications did not go on to receive medication. Though nine individuals who initially reported medical contraindications resubmitted an assessment and did go on to receive medication, we cannot presume that the medication was prescribed in error. For example, contraindications for bupropion and varenicline were assessed jointly and thus an individual may have been prescribed the medication to which they had no contraindication. Furthermore, a physician with a more detailed knowledge of the patient's medical history may have used their clinical judgment and deemed the medication acceptable. Individuals who initially reported using one of bupropion, varenicline, thioridazine or an MAOI may have been seeking a free supply of medication they were already using. A list of medication contraindications may be beneficial to include in the documentation provided to physicians to further ensure they screen for all possible contraindications rather than rely solely on the decision of the study regarding eligibility for medication or on the pharmacist.

More often, altering responses to fit non-medical study eligibility criteria (e.g. target quit date) occurred. These resubmissions were identified post hoc by means of a thorough examination of the personal information provided (i.e. names and birth date). Ideally, automated a priori procedures would be in place to block resubmissions and prevent prescriptions from being generated and printed. In theory, this could be accomplished by programming to detect and prohibit multiple submissions based on an identifier, such as participant name or Internet protocol (IP) address [39]. However, there are inherent limitations with such strategies. Identifying information that is manually entered is easy to alter, which we observed in many resubmissions. While a correct name is needed to fill a prescription and not as liable to falsification, we did encounter unique individuals with the same first and last names. Therefore, blocking submissions based on a combination of two separate identifiers would be beneficial, to minimize the chance of preventing legitimate submissions. Limiting submissions based on IP address or by using cookies also has limitations [39], including preventing individuals in the same household from participating [39] and trying to quit smoking simultaneously. However, in the current study, it is also possible that alteration of quit date may have demonstrated that the offer of free medication was a motivator for change, prompting an earlier quit attempt, rather than an attempt to falsify information.

Overall, findings from the current study support the feasibility of using this distribution method in a future larger trial to establish the effectiveness of this intervention for improving quit rates. A randomized trial is required in order to conclude that either medication was superior to the control condition, particularly given that the reasons for not receiving medication were largely unreported due to low response rate in that group. Given the possibility of selection bias, randomization is also necessary to make conclusions regarding the relative effectiveness of bupropion versus varenicline. While there was clearly a strong preference for varenicline in the current sample, this was likely due to the relative novelty of the medication at the time in Canada. Estimates of treatment effect obtained in the current study will be used to ensure that a larger randomized trial is sufficiently powered.

One factor to consider for future implementation is the length of time provided before the prescription is considered void. Extending the prescription expiry date may be warranted as not being able to make an appointment within the required 3-week time period was the most common reason for not having visited a physician, though overall it was not reported frequently. However, further research is needed to determine whether extending the period does in fact lead to an appreciable increase in visits to a physician. Extending the initial period may require adjustments to the methodology to take into account greater variability between participants in the length of time between enrollment and the start of treatment. One option may be to have the pharmacist log in online to indicate when a prescription has been filled to activate the start of motivational emails and determine the timing of follow-ups rather than being based on enrollment date. In addition, consideration on how to manage supply and capacity to fill prescriptions is required in future implementation of this method. While the current study managed the limited medication supply by periodically closing down enrollment due to uncertainty regarding the proportion of eligible participants that would fill a prescription, further pilot testing may be beneficial as this rate is likely to differ in other jurisdictions, using other recruitment methods etc. Placing a limit on the number of enrollments per day or week is a potential alternative strategy to shutting enrollment down completely. That adjunct motivational emails were not predominantly rated very helpful is similar to previous findings [40]. Additional research may determine how to improve the benefit to participants by altering the content or format of the messages or making these emails an optional component of the intervention.

Some limitations are noted with respect to the design of this distribution method. While online access affords an efficient, cost-effective and convenient format for assessing eligibility and enrolling in a program, its reach is limited to those who have access to the Internet and thus may be less accessible to lower-income households. In 2012, 83 % of Canadian households had access to the Internet at home; however, this proportion drops to 58 % in homes with household incomes in the lowest quartile (≤$30,000/year) [41]; however, this may be mitigated by access to the Internet outside the home such as in public libraries and community centres. Reach is also poorer in rural areas [41] and among older adults, who are less likely to be online [42].

Additional limitations of the current feasibility study existed. The response rates were very poor during the 12-week treatment period, and future studies may benefit by providing an incentive to improve response rate. The response rate did considerably increase at 41-week follow-up without any change to our methodology; the reason for this is unclear though we speculate that seasonality may partially account for this finding as surveys throughout treatment were largely conducted during the summer months. The response rate among those who did not receive medication remained relatively low though and thus the reasons for not receiving medication remains unknown for a large proportion of this group. This loss to follow-up may have led to underestimation of barriers to participation and additional barriers that went unreported may exist. Furthermore, significant differences between respondents and non-respondents indicate the possibility of bias in the reported outcomes. Findings may also not be generalizable to settings without a publicly funded healthcare system and universal access to a physician. Abstinence was not biochemically verified in the current study. A future trial designed to assess effectiveness of this intervention could verify abstinence by collecting saliva samples through the mail to determine salivary cotinine levels [43]. Finally, without knowing the number of smokers to whom information about the study was disseminated, the fact that we were able to recruit 500 eligible smokers within 6 months is not an entirely informative indicator of the success of our recruitment efforts. That is, it does not tell us what proportion who heard about the study attempted to enroll.

While an Internet-based pharmacy was shown to be an effective and efficient means to distribute prescription medication over a wide geographic area in the current study, recent legislative changes have authorized pharmacists in some jurisdictions to prescribe varenicline and bupropion for smoking cessation. This presents the option of a pharmacist distributing medication directly to the patient without the need to visit a physician. The availability of pharmacists, often without the need for a scheduled appointment, could enhance the accessibility of the intervention and dispensing in-person would eliminate delivery costs. The feasibility of this method of distribution is supported by our previous experience facilitating pharmacy-based smoking cessation interventions offering NRT and counselling [29].

Conclusions

The current study supports the feasibility of an innovative method of distributing free prescription medications to smokers whereby those interested in quitting complete an online assessment, bring a prescription to their physician to assess the suitability of the medication, which is then dispensed and delivered by an Internet-based pharmacy. It demonstrates that using Internet-based pharmacies with the capability to dispense by mail can be used to distribute prescription medications for smoking cessation to smokers across vast geographic areas, including both major urban centres and smaller underserved rural communities to reduce geographic disparity in access to services. This method of distribution maintains some of the advantages of Internet-based treatment delivery, including increased efficiency and lower cost, but overcomes a limitation by integrating in-person clinical contact with a physician and brief telephone counselling with a pharmacist. The current model can readily be scaled up for state-or nationwide implementation, including in jurisdictions where cessation medication costs are covered by governmental health agencies, in an effort to enhance smoking cessation rates at a population-wide level.

Additional files

Additional file 1: Screenshot of study website (home page). A screenshot of the home page of the study website (stopstudy.ca) during recruitment (July 2009). (PNG 130 KB) Additional file 2: Letter to physician and prescription form.

Letter to the physician that provided information about the study and personalized prescription form for study medication that were personalized based on information provided in the initial assesment and sent electronically to participants to print and bring to their physician within three weeks of enrollment. (PDF 177 KB)

Competing interests

Salary support was provided by the Ontario Ministry of Health and Long-term Care to all authors. Medications for this study were purchased through a commercial buyer and Biovail (the manufacturer of bupropion) at a discounted price. No funding was received from the manufacturers of these medications for design, data collection, analysis or write up of the current feasibility study. PS has received honoraria for consultancy in an advisory capacity to Pfizer Inc. (manufacturer of varenicline), unrelated to this study. Subsequent to the conduct of this pilot trial, LZ and PS received a grant for a randomized trial from the Global Research Awards in Nicotine Dependence (GRAND), which is funded by Pfizer Inc.

Authors' contributions PS and LZ conceived of the study. PS, LZ and SH designed the study and wrote the protocol. SV and SH analysed the data and wrote the first draft of the manuscript. All authors contributed to revisions and approved the final version of the manuscript.

Fig. 1
1Flow chart of recruitment and follow-up. EOT end of treatment (15 weeks after study enrollment)

with 80 % recruitment by the end of June. The median number of eligible participants that enrolled in the study on days that enrollment was open was 12 (interquartile range (IQR) 6-26) until the end of June and dropped to 4 (IQR 3-6) thereafter. Participants reported hearing about the study from the following sources: friends or family, 39.2 % (95 % confidence interval (CI) = 36.0-42.4 %; n = 348); physician, 16.8 % (95 % CI = 14.3-19.3 %; n = 149); public health unit, 15.1 % (95 % CI = 12.8-17.5 %; n = 134); other healthcare professional/ organization, 5.6 % (95 % CI = 4.1-7.2 %; n = 50); media (newspaper, television and radio), 12.6 % (95 % CI = 10.4-14.8 %; n = 112); the Internet, 4.1 % (95 % CI = 2.8-5.4 %; n = 36); poster or flyer, 2.6 % (95 % CI = 1.5-3.6 %; n = 23); others, 3.2 % (95 % CI = 2.0-4.3 %; n = 28); and don't know/refuse, 0.8 % (95 % CI = 0.2-1.4 %; n = 7). Of the 1214 that submitted an initial assessment, 893 participants were eligible (73.6 %; 95 % CI = 71.1-76.1 %) and successfully enrolled in the study. Individuals were excluded for the following reasons: not planning to quit within 30 days, 17.6 % (95 % CI = 15.5-19.8 %; n = 214); smoked fewer than 10 cigarettes/day, 4.5 % (95 % CI = 3.4-5.7 %; n = 55); did not smoke daily over the past 3 months, 2.1 % (95 % CI = 1.3-2.9 %; n = 25); did not smoke cigarettes, 0.6 % (95 % CI = 0.2-1.0 %; n = 7); had smoked fewer than 100 cigarettes in their lifetime, 0.1 % (95 % CI = 0.0-0.2 %; n = 1); pregnant, 0.4 % (95 % CI = 0.1-0.8 %; n = 5); history of brain injury, 0.8 % (95 % CI = 0.3-1.3 %; n = 10), seizure, 1.4 % (95 % CI = 0.7-2.1 %; n = 17), bulimia or anorexia, 0.2 % (95 % CI = 0.0-0.4 %; n = 2) or allergic reaction to bupropion or varenicline, 0.7 % (95 % CI = 0.2-1.1 %; n = 8); or current use of bupropion, varenicline, MAOIs or thioridazine, 1.5 % (95 % CI = 0.8-2.2 %; n = 18).

6.0 % [95 % CI = 4.4-7.5 %; n = 53] vs. 17.5 %) and the proportion of smokers from Southwestern Ontario was higher (25.1 % [95 % CI = 22.2-27.9 %; n = 223] vs. 13.7 %); differences were minimal (<5 %) in the remaining regions.

Follow-up response ratesThe response rates at follow-up were as follows: 19.6 % (95 % CI = 17.0-22.2 %; n = 175) at 7 weeks; 18.7 % (95 % CI = 16.1-21.3 %; n = 167) at 11 weeks; 33.0 % (95 % CI = 29.9-36.1 %; n = 295) at 15 weeks (end of treatment); and 54.5 % (95 % CI = 51.3-57.8 %; n = 487) at 41 weeks. The response rate at 41-week follow-up among those that did not receive medication was substantially lower (37.1 %; 95 % CI = 32.2-42.1 %; n = 137) compared to the bupropion (71.4 %; 95 % CI = 65.9-76.9 %; n = 185) and varenicline (62.3 %; 95 % CI = 56.4-68.1 %; n = 165) groups. Overall, 66.7 % (95 % CI = 63.7-69.8 %; n = 596) of the sample had follow-up data for at least one time point. Several differences emerged between respondents and those lost to follow-up at 41 weeks (see

CI = 21.5-31.5 %; n = 80); 8 to less than 12 weeks, 24.5 % (95 % CI = 19.7-29.4 %; n = 74); all 12 weeks, 14.6 % (95 % CI = 10.6-18.5 %; n = 44); and don't know/ refuse, 0.7 % (95 % CI = 0.0-1.6 %; n = 2).

Ratings of the perceived helpfulness of the weekly motivational emails were available for 44.1 % (95 % CI = 40.9-47.4 %; n = 394) of participants. The mean rating on a five-point scale was 2.78 (SD = 1.36). Ratings varied widely; while 42.4 % (95 % CI = 37.5-47.3 %; n = 167) rated the emails as not helpful at all or low on helpfulness (1 or 2), 27.7 % (95 % CI = 23.2-32.1 %; n = 109) selected the midpoint on the scale and 29.9 % (95 % CI = 25.4-34.5 %; n = 118) rated the emails helpful (4 or 5). Smoking cessation outcomes The 7-day point prevalence abstinence rate among respondents at 41-week follow-up was 16.2 % (95 % CI = 9.9-22.2 %; n = 22) for the group that did not receive medication, 30.3 % (95 % CI = 23.3-37.3 %; n = 50) for the varenicline group and 24.3 % (95 % CI = 18.1-30.5 %; n = 45) for the bupropion group. The abstinence rates among those who received bupropion was 20.7 % (95 % CI = 5.9-35.4 %; n = 6) for those who had a choice of either medication and 25.0 % (95 % CI = 18.2-31.8 %; n = 39) for those who did not have a choice; this difference was not statistically significant (p = 0.65). Abstinence rates were very similar when missing data were replaced using multiple imputation; the abstinence rate at 41-week follow-up was 17.3 % (95 % CI = 14.1-20.5 %) for the group that did not receive medication, 31.0 % (95 % CI = 27.0-35.0 %) for the varenicline group, and 23.8 % (95 % CI = 20.2-27.4 %) for the bupropion group. Those who received bupropion and had a choice of medication had a quit rate of 21.5 % (95 % CI = 6.6-36.5 %) and those without a choice had a quit rate of 24.3 % (95 % CI = 17.6-31.0 %); this difference was again not statistically significant (p = 0.70).

Reasons reported during follow-up interviews at 7, 11, 15 and 41 weeks. Where the same participant reported differing reasons at separate time points, the reason provided at the earliest follow-up is reported

Table 1
1Examples of weekly motivational email messagesWeekly tip
number

Tip

1

Table 2
2Participant characteristics at baselineCharacteristic
Received medication
p a
Type of medication
p a

No (n = 365) Yes (n = 522)
Bupropion (n = 258) Varenicline (n = 264)

Female, n (%)
196 (53.7)
307 (58.8)
0.130
150 (58.1)
157 (59.5)
0.758

Age (years), mean (SD)
38.6 (12.0)
44.2 (12.5)
<0.001 44.4 (12.2)
44.0 (12.8)
0.729

Education level below high school diploma, n (%)
94 (25.9)
117 (22.4)
0.232
59 (22.9)
58 (22.0)
0.806

Annual household income ≤$20,000, n (%)
164 (48.8)
209 (42.9)
0.095
116 (46.4)
93 (39.2)
0.111

Employed, n (%) b
198 (54.2)
274 (52.5)
0.642
136 (52.7)
138 (52.3)
0.647

Geographic region, n (%)
0.495
0.418

Northern Ontario
46 (12.6)
50 (9.5)
26 (10.0)
24 (9.1)

Toronto
23 (6.3)
30 (5.7)
19 (7.3)
11 (4.2)

Central Ontario (excluding Toronto)
147 (40.3)
237 (45.2)
109 (42.1)
128 (48.3)

Eastern Ontario
57 (15.6)
76 (14.5)
37 (14.3)
39 (14.7)

Southwestern Ontario
92 (25.2)
131 (25.0)
68 (26.3)
63 (23.8)

Cigarettes per day, n (%)
0.365
0.764

10-19
35.9 (131)
32.2 (168)
31.4 (81)
33.0 (87)

20-29
51.0 (186)
51.9 (271)
51.6 (133)
52.3 (138)

30+
13.2 (48)
15.9 (83)
17.1 (44)
14.8 (39)

Time to first cigarette in the morning ≤5 min, n (%)
51.0 (186)
46.0 (240)
0.291
47.3 (122)
44.7 (118)
0.817

Years since started smoking daily, mean (SD)
24.2 (12.0)
29.9 (12.4)
<0.001 29.7 (12.1)
30.0 (12.7)
0.835

Fewer than three past quit attempts, n (%) c
220 (60.3)
270 (51.7)
0.012
140 (54.3)
118 (45.7)
0.251

Confidence in quitting (1-10), mean (SD)
7.5 (2.1)
7.8 (2.1)
0.019
7.8 (2.2)
7.8 (2.0)
0.865

Previous use of smoking cessation medication, n (%) d 186 (51.0)
284 (54.4)
0.311
142 (55.0)
142 (53.8)
0.774

Longest period of previous abstinence from
smoking <1 week, n (%)

115 (31.5)
149 (28.5)
0.342
75 (29.1)
74 (28.0)
0.793

All or most friends smoke, n (%)
180 (49.3)
164 (31.4)
<0.001 80 (31.0)
84 (31.8)
0.749

Lives with other smoker(s), n (%)
219 (60.0)
279 (53.4)
0.053
140 (54.3)
139 (52.7)
0.712

History of drug or alcohol problem, n (%)
55 (15.1)
67 (12.8)
0.342
40 (15.5)
27 (10.2)
0.072

N varies due to missing data. All baseline data are missing for six participants

a

The Bonferroni-adjusted threshold for significance was p < 0.003

b

Employed full-time, part-time or self-employed

c

Abstained from smoking for at least 24 h in an attempt to quit smoking

d

Nicotine patch, gum, inhaler, varenicline, bupropion

Table 3
3Comparison of baseline characteristics for respondents and participants lost to follow-up at 41 weeksCharacteristics
Enrolled (N = 887) a

Respondents (n = 505)
Lost to follow-up at 41 weeks (n = 382)
p b

Female, n (%)
287 (59.3)
216 (53.6)
0.088

Age (years), mean (SD)
43.95 (12.47)
39.46 (12.35)
<0.001

Education level below high school diploma, n (%)
98 (20.3)
113 (28.1)
0.007

Annual household income ≤$20,000, n (%)
178 (39.7)
195 (52.0)
<0.001

Employed, n (%) c
256 (54.2)
216 (55.1)
0.799

Geographic region, n (%)
0.980

Northern Ontario
62 (12.8)
54 (13.4)

Greater Toronto Area (GTA)
52 (10.7)
41 (10.2)

Central Ontario (excluding GTA)
119 (24.6)
104 (25.8)

Eastern Ontario
96 (19.8)
75 (18.6)

Southwestern Ontario
155 (32.0)
129 (32.0)

Cigarettes per day, n (%)
0.218

10-19
174 (36.0)
125 (31.0)

20-29
245 (50.6)
212 (52.6)

30+
65 (13.4)
66 (16.4)

Time to first cigarette in the morning ≤5 min, n (%)
209 (43.2)
217 (53.8)
0.002

Years since started smoking daily, mean (SD)
27.19 (12.27)
22.80 (12.34)
<0.001

Fewer than three past quit attempts, n (%) d
253 (52.3)
237 (58.8)
0.051

Confidence in quitting (1-10), mean (SD)
7.74 (2.13)
7.60 (2.07)
0.303

Previous use of smoking cessation medication, n (%) e
262 (54.1)
208 (51.6)
0.454

Longest period of previous abstinence from smoking <1 week, n (%)
120 (24.8)
144 (35.7)
<0.001

All or most friends smoke, n (%)
157 (32.4)
187 (46.4)
<0.001

Lives with other smoker(s), n (%)
258 (53.3)
240 (59.6)
0.062

History of drug or alcohol problem, n (%)
58 (12.0)
64 (15.9)
0.093

a

N varies due to missing data. All baseline data were missing for six participants

b

The Bonferroni-adjusted threshold for significance was p < 0.003

c

Employed full-time, part-time or self-employed

d

Abstained from smoking for at least 24 h in an attempt to quit smoking

e

Nicotine patch, gum, inhaler, varenicline, bupropion

AcknowledgementsFunding was provided by the Ontario Ministry of Health and Long-term Care.Author
World Health Organization. WHO report on the global tobacco epidemic, 2013: enforcing bans on tobacco advertising, promotion and sponsorship. Geneva, SwitzerlandWorld Health OrganizationWorld Health Organization. WHO report on the global tobacco epidemic, 2013: enforcing bans on tobacco advertising, promotion and sponsorship. Geneva, Switzerland: World Health Organization; 2013.

Smoking and tobacco use: fast facts. Centers for Disease Control and PreventionAtlanta, GACenters for Disease Control and Prevention. Smoking and tobacco use: fast facts. Atlanta, GA: Centers for Disease Control and Prevention. 2014. http://www.cdc.gov/Tobacco/data_statistics/fact_sheets/fast_facts/ index.htm. Accessed May 20 2014.

Quitting smoking among adults-United States. Centers for Disease Control and Prevention. 6044Morb Mortal Wkly Rep.Centers for Disease Control and Prevention. Quitting smoking among adults-United States, 2001-2010. Morb Mortal Wkly Rep. 2011;60(44):1513-9.

Tobacco use in Canada: patterns and trends. J L Reid, D Hammond, V L Rynard, R Burkhalter, Waterloo, ONPropel Centre for Population Health Impact, University of WaterlooReid JL, Hammond D, Rynard VL, Burkhalter R. Tobacco use in Canada: patterns and trends, 2014 edition. Waterloo, ON: Propel Centre for Population Health Impact, University of Waterloo; 2014.

Prevalence and factors associated with difficulty and intention to quit smoking in Switzerland. P Marques-Vidal, J Melich-Cerveira, F Paccaud, G Waeber, P Vollenweider, J Cornuz, BMC Public Health. 111227Marques-Vidal P, Melich-Cerveira J, Paccaud F, Waeber G, Vollenweider P, Cornuz J. Prevalence and factors associated with difficulty and intention to quit smoking in Switzerland. BMC Public Health. 2011;11(1):227.

Interventions to increase smoking cessation at the population level: how much progress has been made in the last two decades?. S-H Zhu, M Lee, Y-L Zhuang, A Gamst, T Wolfson, Tob Control. 212Zhu S-H, Lee M, Zhuang Y-L, Gamst A, Wolfson T. Interventions to increase smoking cessation at the population level: how much progress has been made in the last two decades? Tob Control. 2012;21(2):110-8.

Shape of the relapse curve and long-term abstinence among untreated smokers. J R Hughes, J Keely, S Naud, Addiction. 991Hughes JR, Keely J, Naud S. Shape of the relapse curve and long-term abstinence among untreated smokers. Addiction. 2004;99(1):29-38.

Pharmacological interventions for smoking cessation: an overview and network meta-analysis. K Cahill, S Stevens, R Perera, T Lancaster, Cochrane Database Syst Rev. 59329Cahill K, Stevens S, Perera R, Lancaster T. Pharmacological interventions for smoking cessation: an overview and network meta-analysis. Cochrane Database Syst Rev. 2013;5:CD009329.

Effectiveness of smoking cessation interventions among adults: a systematic review of reviews. V Lemmens, A Oenema, I K Knut, J Brug, Eur J Cancer Prev. 176Lemmens V, Oenema A, Knut IK, Brug J. Effectiveness of smoking cessation interventions among adults: a systematic review of reviews. Eur J Cancer Prev. 2008;17(6):535-44.

Use of smoking-cessation treatments in the United States. S Shiffman, S E Brockwell, J L Pillitteri, J G Gitchell, Am J Prev Med. 342Shiffman S, Brockwell SE, Pillitteri JL, Gitchell JG. Use of smoking-cessation treatments in the United States. Am J Prev Med. 2008;34(2):102-11.

Understanding why smokers do not want to use nicotine dependence medications to stop smoking: qualitative and quantitative studies. F Vogt, S Hall, T M Marteau, Nicotine Tob Res. 108Vogt F, Hall S, Marteau TM. Understanding why smokers do not want to use nicotine dependence medications to stop smoking: qualitative and quantitative studies. Nicotine Tob Res. 2008;10(8):1405-13.

Do smokers know how to quit? Knowledge and perceived effectiveness of cessation assistance as predictors of cessation behaviour. D Hammond, P W Mcdonald, G T Fong, R Borland, Addiction. 998Hammond D, McDonald PW, Fong GT, Borland R. Do smokers know how to quit? Knowledge and perceived effectiveness of cessation assistance as predictors of cessation behaviour. Addiction. 2004;99(8):1042-8.

Healthcare financing systems for increasing the use of tobacco dependence treatment. A A Reda, D Kotz, S Evers, C P Van Schayck, Cochrane Database Syst Rev. 64305Reda AA, Kotz D, Evers S, van Schayck CP. Healthcare financing systems for increasing the use of tobacco dependence treatment. Cochrane Database Syst Rev. 2012;6:CD004305.

A pragmatic, randomized, controlled study evaluating the impact of access to smoking cessation pharmacotherapy coverage on the proportion of successful quitters in a Canadian population of smokers motivated to quit (ACCESSATION). P Selby, G Brosky, P Oh, V Raymond, C Arteaga, S Ranger, BMC Public Health. 141433Selby P, Brosky G, Oh P, Raymond V, Arteaga C, Ranger S. A pragmatic, randomized, controlled study evaluating the impact of access to smoking cessation pharmacotherapy coverage on the proportion of successful quitters in a Canadian population of smokers motivated to quit (ACCESSATION). BMC Public Health. 2014;14(1):433.

Under-use of smoking-cessation treatments: results from the National Health Interview Survey. V E Cokkinides, E Ward, A Jemal, M J Thun, Am J Prev Med. 281Cokkinides VE, Ward E, Jemal A, Thun MJ. Under-use of smoking-cessation treatments: results from the National Health Interview Survey, 2000. Am J Prev Med. 2005;28(1):119-22.

Treating tobacco use and dependence: 2008 update-clinical practice guideline. M C Fiore, C R Jaén, T B Baker, W C Bailey, N L Benowitz, S J Curry, Public Health Service, US Department of Health and Human Services. Fiore MC, Jaén CR, Baker TB, Bailey WC, Benowitz NL, Curry SJ, et al. Treating tobacco use and dependence: 2008 update-clinical practice guideline. 2008. Rockville, MD: Public Health Service, US Department of Health and Human Services; 2008.

Canadian smoking cessation clinical practice guideline. Toronto, Canada: Centre for Addiction and Mental Health. Can-Adaptt, CAN-ADAPTT. Canadian smoking cessation clinical practice guideline. Toronto, Canada: Centre for Addiction and Mental Health; 2011.

Changing provider behavior: an overview of systematic reviews of interventions. J M Grimshaw, L Shirran, R Thomas, G Mowatt, C Fraser, L Bero, Med Care. 39Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L, et al. Changing provider behavior: an overview of systematic reviews of interventions. Med Care. 2001;39:II2-II45.

Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. L A Bero, R Grilli, J M Grimshaw, E Harvey, A D Oxman, M A Thomson, Br Med J. 3177156Bero LA, Grilli R, Grimshaw JM, Harvey E, Oxman AD, Thomson MA. Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. Br Med J. 1998;317(7156):465-8.

Training health professionals in smoking cessation. K V Carson, M E Verbiest, M R Crone, M P Brinn, A J Esterman, W J Assendelft, Cochrane Database Syst Rev. 5214Carson KV, Verbiest ME, Crone MR, Brinn MP, Esterman AJ, Assendelft WJ, et al. Training health professionals in smoking cessation. Cochrane Database Syst Rev. 2012;5:CD000214.

Changing physician performance: a systematic review of the effect of continuing medical education strategies. D A Davis, M A Thomson, A D Oxman, R B Haynes, JAMA. 2749Davis DA, Thomson MA, Oxman AD, Haynes RB. Changing physician performance: a systematic review of the effect of continuing medical education strategies. JAMA. 1995;274(9):700-5.

Knowledge translation in health care: moving from evidence to practice. D Stacey, S Hill, Straus S, Tetroe J, Graham IDJohn Wiley & SonsChichester, UK; LtdPatient-direct and patient-mediated KT interventionsStacey D, Hill S. Patient-direct and patient-mediated KT interventions. In: Straus S, Tetroe J, Graham ID, editors. Knowledge translation in health care: moving from evidence to practice. Chichester, UK: John Wiley & Sons, Ltd; 2013. p. 197-211.

Testing a self-determination theory intervention for motivating tobacco cessation: supporting autonomy and competence in a clinical trial. G C Williams, H A Mcgregor, D Sharp, C Levesque, R W Kouides, R M Ryan, Health Psychol. 251Williams GC, McGregor HA, Sharp D, Levesque C, Kouides RW, Ryan RM, et al. Testing a self-determination theory intervention for motivating tobacco cessation: supporting autonomy and competence in a clinical trial. Health Psychol. 2006;25(1):91-101.

Reach and effectiveness of mailed nicotine replacement therapy for smokers: 6-month outcomes in a naturalistic exploratory study. L Zawertailo, R Dragonetti, S J Bondy, J C Victor, P Selby, Tob Control. 2234Zawertailo L, Dragonetti R, Bondy SJ, Victor JC, Selby P. Reach and effectiveness of mailed nicotine replacement therapy for smokers: 6-month outcomes in a naturalistic exploratory study. Tob Control. 2013;22(3):e4.

Reach, efficacy, and cost-effectiveness of free nicotine medication giveaway programs. K M Cummings, B Fix, P Celestino, S Carlin-Menter, O' Connor, R Hyland, A , J Public Health Manag Pract. 121Cummings KM, Fix B, Celestino P, Carlin-Menter S, O'Connor R, Hyland A. Reach, efficacy, and cost-effectiveness of free nicotine medication giveaway programs. J Public Health Manag Pract. 2006;12(1):37-43.

The importance of piloting an RCT intervention. N Feeley, S Cossette, J Côté, M Héon, R Stremler, G Martorella, Can J Nurs Res. 412Feeley N, Cossette S, Côté J, Héon M, Stremler R, Martorella G, et al. The importance of piloting an RCT intervention. Can J Nurs Res. 2009;41(2):84-99.

A tutorial on pilot studies: the what, why and how. L Thabane, J Ma, R Chu, J Cheng, A Ismaila, L P Rios, BMC Med Res Methodol. 1011Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, et al. A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol. 2010;10(1):1.

Experiences with primary health care in Canada. Ottawa: Canadian Institute for Health Information. Canadian Institute for Health InformationCanadian Institute for Health Information. Experiences with primary health care in Canada. Ottawa: Canadian Institute for Health Information; 2009.

Effectiveness of pharmacist counseling combined with nicotine replacement therapy: a pragmatic randomized trial with 6987 smokers. M J Costello, B Sproule, J C Victor, S T Leatherdale, L Zawertailo, P Selby, Cancer Causes Control. 222Costello MJ, Sproule B, Victor JC, Leatherdale ST, Zawertailo L, Selby P. Effectiveness of pharmacist counseling combined with nicotine replacement therapy: a pragmatic randomized trial with 6987 smokers. Cancer Causes Control. 2011;22(2):167-80.

edition (v2). Waterloo, ON: Propel Centre for Population Health Impact. J Reid, D Hammond, University of WaterlooTobacco use in Canada: patterns and trendsReid J, Hammond D. Tobacco use in Canada: patterns and trends, 2009 edition (v2). Waterloo, ON: Propel Centre for Population Health Impact, University of Waterloo; 2009.

Varenicline, an α4β2 nicotinic acetylcholine receptor partial agonist vs sustained-release bupropion and placebo for smoking cessation: a randomized controlled trial. D Gonzales, S I Rennard, M Nides, C Oncken, S Azoulay, C B Billing, JAMA. 2961Gonzales D, Rennard SI, Nides M, Oncken C, Azoulay S, Billing CB, et al. Varenicline, an α4β2 nicotinic acetylcholine receptor partial agonist vs sustained-release bupropion and placebo for smoking cessation: a randomized controlled trial. JAMA. 2006;296(1):47-55.

Efficacy of varenicline, an α4β2 nicotinic acetylcholine receptor partial agonist, vs placebo or sustained-release bupropion for smoking cessation: a randomized controlled trial. D E Jorenby, J T Hays, N A Rigotti, S Azoulay, E J Watsky, K E Williams, JAMA. 2961Jorenby DE, Hays JT, Rigotti NA, Azoulay S, Watsky EJ, Williams KE, et al. Efficacy of varenicline, an α4β2 nicotinic acetylcholine receptor partial agonist, vs placebo or sustained-release bupropion for smoking cessation: a randomized controlled trial. JAMA. 2006;296(1):56-63.

Association between duration of use of pharmacotherapy and smoking cessation: findings from a national survey. M Siahpush, R A Shaikh, M Mccarthy, A S Kessler, M Tibbits, G K Singh, BMJ Open. 516229Siahpush M, Shaikh RA, McCarthy M, Kessler AS, Tibbits M, Singh GK. Association between duration of use of pharmacotherapy and smoking cessation: findings from a national survey. BMJ Open. 2015;5(1):e006229.

. Statistics Canada. Canadian Community Health Survey. Ottawa: Statistics CanadaStatistics Canada. Canadian Community Health Survey (CCHS), 2009-2010. Ottawa: Statistics Canada; 2011.

A comparison of inclusive and restrictive strategies in modern missing data procedures. L M Collins, J L Schafer, C-M Kam, Psychol Methods. 64Collins LM, Schafer JL, Kam C-M. A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychol Methods. 2001;6(4):330-51.

Stata Corp. Stata Statistical Software: Release 12. College Station: StataCorp LP. Stata Corp. Stata Statistical Software: Release 12. College Station: StataCorp LP; 2011.

Access to a regular medical doctor. Statistics Canada. Ottawa: Statistics CanadaStatistics Canada. Access to a regular medical doctor, 2013. Ottawa: Statistics Canada; 2014. http://www.statcan.gc.ca/pub/82-625-x/2014001/ article/14013-eng.htm. Accessed June 25 2015.

Human research and data collection via the Internet. M H Birnbaum, Annu Rev Psychol. 55Birnbaum MH. Human research and data collection via the Internet. Annu Rev Psychol. 2004;55:803-32.

Text2Quit: results from a pilot test of a personalized, interactive mobile health smoking cessation program. L C Abroms, M Ahuja, Y Kodl, L Thaweethai, J Sims, J P Winickoff, J Health Commun. 171Abroms LC, Ahuja M, Kodl Y, Thaweethai L, Sims J, Winickoff JP, et al. Text2Quit: results from a pilot test of a personalized, interactive mobile health smoking cessation program. J Health Commun. 2012;17 suppl 1:44-53.

Canadian Internet use survey. Ottawa: Statistics CanadaStatistics Canada. Canadian Internet use survey, 2012. Ottawa: Statistics Canada; 2013. http://www.statcan.gc.ca/daily-quotidien/131126/dq131126d- eng.htm. Accessed January 26, 2015.

Rates of Internet use, by age group, selected years. Ottawa: Statistics Canada. Statistics CanadaStatistics Canada. Rates of Internet use, by age group, selected years. Ottawa: Statistics Canada; 2009. http://www.statcan.gc.ca/pub/11-008-x/ 2009002/c-g/10910/c-g001-eng.htm. Accessed January 26, 2015.

Submit your next manuscript to BioMed Central and take full advantage of: • Convenient online submission • Thorough peer review • No space constraints or color figure charges • Immediate publication on acceptance • Inclusion in PubMed, CAS, Scopus and Google Scholar • Research which is freely available for redistribution. J A Cunningham, S T Leatherdale, P L Selby, R F Tyndale, L Zawertailo, V Kushnir, BMC Public Health. 111741Randomized controlled trial of mailed nicotine replacement therapy to Canadian smokers: study protocol. Submit your manuscript at www.biomedcentral.com/submitCunningham JA, Leatherdale ST, Selby PL, Tyndale RF, Zawertailo L, Kushnir V. Randomized controlled trial of mailed nicotine replacement therapy to Canadian smokers: study protocol. BMC Public Health. 2011;11(1):741. Submit your next manuscript to BioMed Central and take full advantage of: • Convenient online submission • Thorough peer review • No space constraints or color figure charges • Immediate publication on acceptance • Inclusion in PubMed, CAS, Scopus and Google Scholar • Research which is freely available for redistribution Submit your manuscript at www.biomedcentral.com/submit


Validity evidence and reliability of a simulated patient feedback instrument

Claudia Schlegel
Ulrich Woermann
Jan-Joost Rethans
Cees Van Der Vleuten
Validity evidence and reliability of a simulated patient feedback instrument
R E S E A R C H A R T I C L E Open Access
Background: In the training of healthcare professionals, one of the advantages of communication training with simulated patients (SPs) is the SP's ability to provide direct feedback to students after a simulated clinical encounter. The quality of SP feedback must be monitored, especially because it is well known that feedback can have a profound effect on student performance. Due to the current lack of valid and reliable instruments to assess the quality of SP feedback, our study examined the validity and reliability of one potential instrument, the 'modified Quality of Simulated Patient Feedback Form' (mQSF).Methods: Content validity of the mQSF was assessed by inviting experts in the area of simulated clinical encounters to rate the importance of the mQSF items. Moreover, generalizability theory was used to examine the reliability of the mQSF. Our data came from videotapes of clinical encounters between six simulated patients and six students and the ensuing feedback from the SPs to the students. Ten faculty members judged the SP feedback according to the items on the mQSF. Three weeks later, this procedure was repeated with the same faculty members and recordings. Results: All but two items of the mQSF received importance ratings of > 2.5 on a four-point rating scale. A generalizability coefficient of 0.77 was established with two judges observing one encounter.Conclusions:The findings for content validity and reliability with two judges suggest that the mQSF is a valid and reliable instrument to assess the quality of feedback provided by simulated patients.

Background

A major advantage of communication training with simulated patients (SPs) in the training of healthcare professionals is that SPs are able to provide feedback to students from a patient's perspective immediately after a simulated clinical encounter (SCE) [1][2][3]. Feedback is an important and valuable tool in interactive learning [4], and for our purposes, it may be defined as the provision of specific information on a student's performance relative to a specific performance standard [5], with the intention of improving the student's performance. High-quality feedback can have a profound effect on student performance [6] and is, therefore, vital to the overall effectiveness of a training sequence.

To assess the quality of SP feedback, a valid and reliable assessment instrument is needed. The only existing instrument, the "Maastricht Assessment of Simulated Patients" (MaSP) [7], has two subscales and assesses both the quality of SP feedback and the authenticity of SP performance during a simulated consultation; as a result, it is not detailed enough to assess the quality of SP feedback in-depth. At the same time, some items of the MaSP, e.g. "SP left the room between consultation and feedback", are too specific and irrelevant for institutions where the SP stays in the room between encounter and feedback.

In the "grey literature", we found another instrument, the "Quality of Simulated Patient Feedback (QSF) Form" [8], which was designed to help SP trainers to evaluate the quality of the oral SP feedback process and thus to determine whether SPs need more training in oral feedback. The detailed items of this instrument meet our needs, but the form has never been validated and there are no data on its reliability. The QSF is an 18-item questionnaire (Table 1) with a dichotomous checklist based on the tenets of basic feedback rules [9][10][11]. For our purpose, the dichotomous rating options were expanded to a four-point rating scale (c.f. the method section for reasons). The aim of our study, then, was to gather evidence on the validity and reliability of the mQSF when used to assess the quality of feedback provided by SPs.

Methods

The study was conducted at a school of nursing in Berne, Switzerland, with nursing students in their second of three curricular years. The two-step approach of the study consisted, first, of an evaluation of the evidence for content validity, and second, of a generalizability analysis to estimate the reliability of the instrument.

Forward-backward translation of the questionnaire

Since the study was conducted in a German-speaking country, the English QSF had to be translated into German. We used a forward-backward translation approach, which is recommended for translating test instruments [12]. Using this approach, a native speaker of the target language (in our case German) translated the instrument from the source language (English), and another person fluent in English then translated the text back from German into English. The original and the back-translated versions were then compared to ensure that the meaning and the nuances of the text were conserved.

Evidence for the content validity of the mQSF items

The content validity of the 18 mQSF items was ascertained by asking 25 medical and nursing education experts from Switzerland, Germany and Austria to rank the importance of each item on a four-point rating scale (1 = not at all important; 4 = very important), using an online survey tool. An even number of scale points (no "neutral" middle position) was used to force clear ratings. The experts were alumni of the Master of Medical Education Programme at the University of Berne, Switzerland, who were actively involved in SP programmes at their own institutions. They were also invited to comment on the mQSF, e.g. whether they thought additional items should be added.

Moreover, since the items were rated on an ordinal rating scale, both mean and median ratings were calculated. Further, Cronbach's α was calculated to ascertain homogeneity among raters. An item-total correlation was performed to check whether any item is inconsistent with the rest of the scale and would thus have to be discarded.

We considered the relevance of an item of the mQSF as most important. If the mean of such an item was below 2.5 we studied the item-correlation of that item in more detail and decided to withdrew that item if a negative item-total correlation was present.

Reliability of the mQSF

We were interested in the reliability of the quality of the SP feedback and of how the quality might be increased, e.g. by having more than one judge rating the quality. For this purpose, an analysis of generalizability (using Genova [13]) was used; reliability estimates were based on a partitioning into true and multiple sources of error variance. Six SPs were videotaped during eight clinical encounters with different students; at the end of each encounter, feedback was given by the SPs. One videotaped encounter per SP was randomly selected for assessment by ten faculty members who judged the feedbacks according to the mQSF items. The six SPs, four females and two males, had at least 1 year of experience in role-playing and giving feedback. Three SPs impersonated a case of acute postoperative pain after an open appendectomy and were instructed to act as if they were afraid that something had gone wrong during the operation. The other three SPs enacted the role of a patient in a consultation on oral anticoagulation therapy after aortic valve replacement; they were instructed to act as if they were indifferent toward the information they received. All SP clinical encounters used and recorded in this investigation were specifically designed for this purpose and in line with the heretofore-acquired curricular competences.

In the G-study, the quality of feedback given in these six encounters was rated by 10 judges (teachers from our institution who were trained in the use of the mQSF) using a rating scale for the mQSF that ranged from 1 (= strongly agree) to 4 (= strongly disagree). We expanded the originally dichotomous rating options to a four-point rating scale because we wanted to provide more subtle parameters for the assessment of SP performance in terms of qualitative holistic judgments [14]. Three weeks later, the procedure was repeated with the same ten teachers and the same six recorded SCEs. We thus had a fully-crossed Video (encounter) by Rater by Occasion (6×10×2) design in which we treated all facets as random.

In the subsequent decision-study (D-study), the facet "V" (video) of a CD-recorded clinical encounter was the object of measurement, whereas the number (n) of judges (facet J) and occasions (facet O) were varied (Figure 1).

Ethical considerations

Ethical approval was sought from the ethics committee of the State of Bern, Switzerland. Informed consent was obtained from all participating students and SPs. Participation in the study was completely voluntary. All participants were free to leave the study at any time without any repercussions. There was no financial compensation.

Results

Forward-backward translation

The comparison of the original source text of the QSF and the retranslated text revealed no major discrepancies.

Evidence of content validity of the mQSF items

Of the 25 experts invited to participate in the study, 14 completed the questionnaire (response rate 56%). The importance rates of the mQSF items from the experts were > 2.5 on a four-point rating scale for all but two items. The highest ratings were those for items 7, 8, and 10 (mean = 3.86 (SD = 0.53), median = 3.96), whereas the lowest ratings were those for items 12 (mean = 2.43 (SD = 1.28), median = 2.10) and 18 (mean = 2.43 (SD = 1.09), median 2.13) ( Table 1).

The items were largely, albeit not completely, homogeneous for all judges, with a Cronbach's α of 0.78. As for the item-total correlation, items 5 and 18 showed negative correlations with the overall score. The latter item was also rated with a mean < 2.5, which is why we excluded it. Without item 18, the internal consistency (Cronbach's α) increased from 0.78 to 0.80. None of the experts gave comments on adding new, or modifying existing, items.

Reliability of the mQSF

All simulated clinical encounters, including the feedback part, were conducted and recorded successfully.

The estimated variance components that resulted from the analysis of variance are given in Table 2. Most of the variance can be explained as systematic differences between videos (70%). Being the object of measurement, this represents the true score variation. The general error term is the largest source of error, followed by systematic variation between the raters (rater leniency/stringency; nearly 8%) and rater by video variation (rater leniency/ stringency for some videos, but not for others; 16%). All  occasion-related components are small, indicating a high intra-rater consistency. Using these variance components, we got a domainreferenced dependability coefficient of 0.633 was calculated by using one judge on one occasion using the formula expressing the composition of the sources of error variance divided by their respective sample size (n) (Figure 1).

The judge-related components are rather large, which means that sampling more judges would increase reliability. On the other hand, repeated judgments would hardly result in increased reliability, given the small variance component of 0.71 for facet O (occasion).

The following estimates of the dependability coefficient were calculated running D-studies for varying numbers of judges and occasions (Table 3).

Discussion

Evidence of content validity of the mQSF

Consistently positive expert ratings appear to support the conclusion that the mQSF has adequate content validity. Cronbach's α with 17 items was 0.80, which suggests a high degree of rater homogeneity.

Correlations among items were positive, except for items 5 and 18. However, these two items differed with regard to importance, which was rated quite high for item 5 but low for item 18 (Table 1). Item 5 ("SP gave first positive feedback") relates to an essential feedback rule (sandwich technique, [15]) which holds that starting with positive feedback creates an open mind-set in the recipient of the feedback. Item 18 ("SP thanked the student"), on the other hand, addresses a cultural peculiarity. Lauffs et al. (2008) [16] state that it is not only necessary to translate an instrument from one language into another, but also to adapt it culturally. At institutions where thanking students at the end of a feedback session is not customary, item 18 should indeed be removed from the QSF as we did.

Item 12 ("SP confirmed the feelings with student") had the lowest mean and median for importance ratings (Table 1), but showed the highest correlation with the overall score. Moreover, the importance of emphasising students' feelings has been underlined by Steinwachs (1992) [17] who stated that strong feelings of students should be addressed.

Reliability of the mQSF

The G-study shows that increasing the number of judges observing an SP giving oral feedback increases the generalizability coefficient. A realistic design would include one judge and one occasion. In our decision study, this yielded a generalizability coefficient of 0.63. Since individual judges are likely to be subjective in their judgments and can introduce substantial error variance, it seemed advisable to use more than one judge. In fact, with two judges, the generalizability coefficient increased from 0.63 to 0.77. This indicates that two judges should observe the same encounter if possible.

Limitations

A limitation of our study is that the original instrument was translated into another language. Translating an instrument always involves the risk that the original idea expressed in an item may not be conveyed fully and accurately. Cultural differences can also hamper accurate representation of item content. A further limitation is that only content validity was explored and not other types of validity, but we studied the content validity of this instrument because without content validity, other types of validity are meaningless.

While 14 raters seem sufficient to determine content validity [18], the generalizability study involved a rather low number of simulated clinical encounters; this was due to limited resources. However, a small sample size may be problematic with respect to representatively as it limits the generalizability to other settings and the confidence in the results of the G-study. Moreover, in our study, only one encounter, one case and one student per participating SP was rated. It would be interesting to explore case variability in SP feedback in future research.

Conclusions

The findings for content validity and reliability with two judges suggest that the mQSF is a valid and reliable instrument to assess the quality of feedback provided by simulated patients.

We recommend that more studies be conducted, with larger samples, more cases, and more students to corroborate the findings reported here. Such studies should include more encounters and compare results obtained with the mQSF with results obtained with the feedback quality component of the MaSP or other instruments for the assessment of feedback quality.

Figure 1
1The object of measurement for the D-study. Facet "V" (video), of a CD-recorded clinical encounter, number (n) of judges (facet J), occasions (facet O)

Table 1
1Experts' judgments of the importance of the QSF items16 The SP ensured that the student understood what she (the student) needed to work on 17 SP continued to ask student if she had questions until student said "no"No Item

Table 2
2Estimated variance componentsVariance components

Source
Estimate
Error
% of total Variance

V
79.390
43.905
70.33

J
8.714
6.279
7.72

O
0.714
1.035
0.63

VJ
18.198
5.957
16.12

VO
0.000
0.867
0.00

JO
2.069
2.276
1.83

VJO
18.363
3.788
16.27

V video, J judges, O occasions

Table 3
3Number of judges, occasions and reliability Schlegel et al. BMC Medical Education 2012, 12:6 http://www.biomedcentral.com/1472-6920/12/61 judge
2 judges
3 judges

1 occasion
0.63
0.77
0.83

2 occasions
0.68
0.81
0.86

3 occasions
0.70
0.83
0.88

Authors' contributions UW, JJR and CvdV supervised CS in the design, data collection and statistical analyze of the project. UW, JJR, CvdV assisted CS in interpretation of results, drafting the manuscript and critically evaluated earlier drafts of the manuscript. All authors read and approved the final manuscript.Competing interestsThe authors declare that they have no competing interests.
Using standardized patients to assass communication skills in medical and nursing education. A Ryan, A Walshe, R Gaffney, A Shanks, L Burgoyne, C Wiskin, BMC Med Educ. 1024Ryan A, Walshe A, Gaffney R, Shanks A, Burgoyne L, Wiskin C: Using standardized patients to assass communication skills in medical and nursing education. BMC Med Educ 2010, 10:(24):1-8.

Peer role-play and standardised patients in communication training: a comparative study on the student perspective on acceptability, realism, and perceived effect. H M Bosse, M Nickel, S Huwendiek, J Junger, J H Schultz, C Nikendei, BMC Med Educ. 1027Bosse HM, Nickel M, Huwendiek S, Junger J, Schultz JH, Nikendei C: Peer role-play and standardised patients in communication training: a comparative study on the student perspective on acceptability, realism, and perceived effect. BMC Med Educ 2010, 10:27.

Feedack by simulated patients in undergraduate medical education: a systemic review of the literature. L Bokken, T Linssen, A Scherpbier, Cpm Van Der Vleuten, J Rethans, Med Educ. 4311Bokken L, Linssen T, Scherpbier A, van der Vleuten CPM, Rethans J: Feedack by simulated patients in undergraduate medical education: a systemic review of the literature. Med Educ 2009, 43:(11):202-210.

Learning performance with interactive simulations in medical education: lessons learned from results of learning complex physiologicyl models with the HAEMOdynamics SIMulator. A Holzinger, M D Kickmeier, S Wassertheurer, M Hessinger, Comput Educ. 521Holzinger A, Kickmeier MD, Wassertheurer S, Hessinger M: Learning performance with interactive simulations in medical education: lessons learned from results of learning complex physiologicyl models with the HAEMOdynamics SIMulator. Comput Educ 2009, 52(1):292-301.

What is feedback in clinical education?. Jmm Van De Ridder, K M Stokking, W Mcgaghie, Otj Ten Cate, Med Educ. 42van de Ridder JMM, Stokking KM, McGaghie W, ten Cate OTJ: What is feedback in clinical education? Med Educ 2008, 42:189-197.

Focus on formative feedback. V J Shute, Rev Educ Res. 781Shute VJ: Focus on formative feedback. Rev Educ Res 2008, 78(1):153-189.

Assessing simulated patients in an educational setting: the MaSP (Maastricht Assessment of Simulated Patients). L A Wind, J Van Dalen, Amm Muijtjens, J Rethans, Med Educ. 38Wind LA, Van Dalen J, Muijtjens AMM, Rethans J: Assessing simulated patients in an educational setting: the MaSP (Maastricht Assessment of Simulated Patients). Med Educ 2004, 38:39-44.

Training to standardize feedback. W May, D Fisher, Prato, Italy: Monash University2nd International skills conferenceMay W, Fisher D: Training to standardize feedback. 2nd International skills conference Prato, Italy: Monash University; 2007.

One to one teaching and feedback. J Gordon, BMJ. 326Gordon J: One to one teaching and feedback. BMJ 2003, 326:543-545.

. C Claiborn, R Goodyear, P Horner, Feedback. Psychother, Theory, Res. 384Claiborn C, Goodyear R, Horner P: Feedback. Psychother, Theory, Res, Pract, Train 2001, 38(4):401-405.

Feedback and reflection: teaching methods for clinical settings. W Branch, A Paranajape, Acad Med. 7712Branch W, Paranajape A: Feedback and reflection: teaching methods for clinical settings. Acad Med 2002, 77(12):1185-1188.

Validation of the Italian version of the cystic fibrosis quality of life questionnaire (CFQoL), a disease specific measure for adults and adolescents with cystic fibrosis. F Monti, F Lupi, F Gobbi, F Agostini, A Miano, L Gee, J Abbott, J Cyst Fibros. Monti F, Lupi F, Gobbi F, Agostini F, Miano A, Gee L, Abbott J: Validation of the Italian version of the cystic fibrosis quality of life questionnaire (CFQoL), a disease specific measure for adults and adolescents with cystic fibrosis. J Cyst Fibros 2007.

R L Brennan, Elements of generalizability theory. Iowa CityAmerican College Testing ProgramBrennan RL: Elements of generalizability theory Iowa City,: American College Testing Program; 1983.

D L Streiner, G R Norman, Health Measurement. OxfordOxford University PressFourth editionStreiner DL, Norman GR: Health Measurement. Fourth edition. Oxford: Oxford University Press; 2008.

Feed me back. P Trembley, 1933Trembley P: Feed me back. IJO 2008, 19(2):33.

Corsscultural adaption of the Swedish version of readiness for interprofessional learning scale. M Lauffs, S Ponzer, F Saboonchi, K Lonka, U Hylin, A K Mattiasson, Med Educ. 42Lauffs M, Ponzer S, Saboonchi F, Lonka K, Hylin U, Mattiasson AK: Corss- cultural adaption of the Swedish version of readiness for interprofessional learning scale. Med Educ 2008, 42:405-411.

How to facilitate a debriefing. B Steinwachs, Simul Gaming. 232Steinwachs B: How to facilitate a debriefing. Simul Gaming 1992, 23: (2):186-195.

Validating Research Instruments Pensacola FL: National Nursing Staff Development Organization NNSDO. C Malmgreen, Malmgreen C: Validating Research Instruments Pensacola FL: National Nursing Staff Development Organization NNSDO; 2005.

Pre-publication history The pre-publication history for this paper can be accessed here. 10.1186/1472-6920-12-6Pre-publication history The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1472-6920/12/6/prepub doi:10.1186/1472-6920-12-6

Validity evidence and reliability of a simulated patient feedback instrument. Schlegel, BMC Medical Education. 126Cite this article asCite this article as: Schlegel et al.: Validity evidence and reliability of a simulated patient feedback instrument. BMC Medical Education 2012 12:6.