There is ample evidence that medical students are insufficiently trained in safe and effective prescribing.[1](#cpt1549-bib-0001){ref-type="ref"} A recent European multicenter study showed that only one in four final‐year medical students chose the most appropriate therapy in a case‐based examination and that roughly half of their prescriptions contained one or more errors.[2](#cpt1549-bib-0002){ref-type="ref"} Moreover, recently graduated doctors make more potentially hazardous prescribing errors than consultants.[3](#cpt1549-bib-0003){ref-type="ref"} Fortunately, the need for urgent changes in international clinical pharmacology and therapeutics (CPT) education is becoming increasingly apparent, and the first steps toward improvement have been taken. Key learning outcomes were identified in an international multicenter Delphi study, and the European Association for Clinical Pharmacology and Therapeutics (EACPT) Working Group on education has published a list of recommendations to harmonize and modernize CPT education.[4](#cpt1549-bib-0004){ref-type="ref"}, [5](#cpt1549-bib-0005){ref-type="ref"}, [6](#cpt1549-bib-0006){ref-type="ref"} One of these recommendations is for CPT teachers to use more (online) digital learning resources (DLRs) and preferably share these at a national or international level.[5](#cpt1549-bib-0005){ref-type="ref"} These resources are not limited to (online or offline) e‐learning programs, but may also include more innovative ways of teaching, such as podcasts, simulations, serious games, and virtual or augmented reality. Indeed, the possibility to distribute high‐quality content and to reuse this extensively, after only a single investment of time and money, is one of the advantages of DLRs over traditional teaching methods. Other potential advantages for teachers include the possibility to rapidly update, revise, and standardize the content and the use of learning management systems to track individual learner progress. DLRs also have potential advantages for learners, such as the possibility to alter study pace and revise content extensively, to use multimedia and interactive elements, and to study anytime and anywhere. On the other hand, first generation DLRs may also have significant drawbacks because of the lack of interaction with teachers and peers.[7](#cpt1549-bib-0007){ref-type="ref"} Studies of healthcare education have shown that DLRs increase knowledge compared with no education and are at least equal to traditional learning methods in terms of knowledge acquisition and learner satisfaction.[8](#cpt1549-bib-0008){ref-type="ref"}, [9](#cpt1549-bib-0009){ref-type="ref"} However, safe and effective prescribing is a more complex process, requiring the integration of cognitive skills (e.g., knowledge, problem‐solving, and decision making) and attitudes in a busy and potentially stressful workplace.[10](#cpt1549-bib-0010){ref-type="ref"} Whether DLRs are suitable to teach these components in an effective manner has been questioned.[7](#cpt1549-bib-0007){ref-type="ref"} Unfortunately, the effect of the diverse DLRs on safe prescribing remains largely unclear, and none of the existing review articles on digital health education have focused on prescribing. DLRs are often evaluated in terms of subjective satisfaction and knowledge acquisition only, and studies evaluating more clinically relevant outcomes are scarce because they are costly, time‐consuming, and limited by numerous confounding factors.[7](#cpt1549-bib-0007){ref-type="ref"}, [11](#cpt1549-bib-0011){ref-type="ref"} In order to help implement the EACPT working group recommendation on internationally distributed DLRs, more information about their effects on prescribing is required. Therefore, the aims of this review are to identify published forms of DLR used to teach rational prescribing and to assess their effect on learner‐related and clinically relevant outcomes, with a view to determining which elements are effective in teaching prescribing.

Literature Review {#cpt1549-sec-0002}
=================

The database search yielded 2,110 records, 125 of which were eligible on the basis of title and abstract. Fifty‐nine full‐text articles were eligible for inclusion.[12](#cpt1549-bib-0012){ref-type="ref"}, [13](#cpt1549-bib-0013){ref-type="ref"}, [14](#cpt1549-bib-0014){ref-type="ref"}, [15](#cpt1549-bib-0015){ref-type="ref"}, [16](#cpt1549-bib-0016){ref-type="ref"}, [17](#cpt1549-bib-0017){ref-type="ref"}, [18](#cpt1549-bib-0018){ref-type="ref"}, [19](#cpt1549-bib-0019){ref-type="ref"}, [20](#cpt1549-bib-0020){ref-type="ref"}, [21](#cpt1549-bib-0021){ref-type="ref"}, [22](#cpt1549-bib-0022){ref-type="ref"}, [23](#cpt1549-bib-0023){ref-type="ref"}, [24](#cpt1549-bib-0024){ref-type="ref"}, [25](#cpt1549-bib-0025){ref-type="ref"}, [26](#cpt1549-bib-0026){ref-type="ref"}, [27](#cpt1549-bib-0027){ref-type="ref"}, [28](#cpt1549-bib-0028){ref-type="ref"}, [29](#cpt1549-bib-0029){ref-type="ref"}, [30](#cpt1549-bib-0030){ref-type="ref"}, [31](#cpt1549-bib-0031){ref-type="ref"}, [32](#cpt1549-bib-0032){ref-type="ref"}, [33](#cpt1549-bib-0033){ref-type="ref"}, [34](#cpt1549-bib-0034){ref-type="ref"}, [35](#cpt1549-bib-0035){ref-type="ref"}, [36](#cpt1549-bib-0036){ref-type="ref"}, [37](#cpt1549-bib-0037){ref-type="ref"}, [38](#cpt1549-bib-0038){ref-type="ref"}, [39](#cpt1549-bib-0039){ref-type="ref"}, [40](#cpt1549-bib-0040){ref-type="ref"}, [41](#cpt1549-bib-0041){ref-type="ref"}, [42](#cpt1549-bib-0042){ref-type="ref"}, [43](#cpt1549-bib-0043){ref-type="ref"}, [44](#cpt1549-bib-0044){ref-type="ref"}, [45](#cpt1549-bib-0045){ref-type="ref"}, [46](#cpt1549-bib-0046){ref-type="ref"}, [47](#cpt1549-bib-0047){ref-type="ref"}, [48](#cpt1549-bib-0048){ref-type="ref"}, [49](#cpt1549-bib-0049){ref-type="ref"}, [50](#cpt1549-bib-0050){ref-type="ref"}, [51](#cpt1549-bib-0051){ref-type="ref"}, [52](#cpt1549-bib-0052){ref-type="ref"}, [53](#cpt1549-bib-0053){ref-type="ref"}, [54](#cpt1549-bib-0054){ref-type="ref"}, [55](#cpt1549-bib-0055){ref-type="ref"}, [56](#cpt1549-bib-0056){ref-type="ref"}, [57](#cpt1549-bib-0057){ref-type="ref"}, [58](#cpt1549-bib-0058){ref-type="ref"}, [59](#cpt1549-bib-0059){ref-type="ref"}, [60](#cpt1549-bib-0060){ref-type="ref"}, [61](#cpt1549-bib-0061){ref-type="ref"}, [62](#cpt1549-bib-0062){ref-type="ref"}, [63](#cpt1549-bib-0063){ref-type="ref"}, [64](#cpt1549-bib-0064){ref-type="ref"}, [65](#cpt1549-bib-0065){ref-type="ref"}, [66](#cpt1549-bib-0066){ref-type="ref"}, [67](#cpt1549-bib-0067){ref-type="ref"}, [68](#cpt1549-bib-0068){ref-type="ref"}, [69](#cpt1549-bib-0069){ref-type="ref"}, [70](#cpt1549-bib-0070){ref-type="ref"} Six additional articles were identified from the previously used database,[71](#cpt1549-bib-0071){ref-type="ref"}, [72](#cpt1549-bib-0072){ref-type="ref"}, [73](#cpt1549-bib-0073){ref-type="ref"}, [74](#cpt1549-bib-0074){ref-type="ref"}, [75](#cpt1549-bib-0075){ref-type="ref"}, [76](#cpt1549-bib-0076){ref-type="ref"} making a total of 65 articles. An overview of all included articles (with effect‐estimates when available) is presented in **Table** [S1](#cpt1549-sup-0001){ref-type="supplementary-material"}. The complete selection process, as per the Preferred Reporting Items for Systematic Reviews and Meta‐Analyses (PRISMA) guidelines, is shown in **Figure** [1](#cpt1549-fig-0001){ref-type="fig"}.

![Preferred Reporting Items for Systematic Reviews and Meta‐Analyses (PRISMA) flow diagram for study selection. Search strategies used for both databases are described in **Table** [S2](#cpt1549-sup-0002){ref-type="supplementary-material"}. CINAHL, Cumulative Index to Nursing and Allied Health Literature; ERIC, Education Resources Information Centre.](CPT-106-1236-g001){#cpt1549-fig-0001}

Digital learning resource characteristics {#cpt1549-sec-0003}
-----------------------------------------

**Table** [1](#cpt1549-tbl-0001){ref-type="table"} shows the characteristics of the digital teaching interventions. Most (*N* = 55; 85%) articles described the use of (online) e‐learning,[12](#cpt1549-bib-0012){ref-type="ref"}, [13](#cpt1549-bib-0013){ref-type="ref"}, [14](#cpt1549-bib-0014){ref-type="ref"}, [15](#cpt1549-bib-0015){ref-type="ref"}, [16](#cpt1549-bib-0016){ref-type="ref"}, [17](#cpt1549-bib-0017){ref-type="ref"}, [19](#cpt1549-bib-0019){ref-type="ref"}, [20](#cpt1549-bib-0020){ref-type="ref"}, [21](#cpt1549-bib-0021){ref-type="ref"}, [22](#cpt1549-bib-0022){ref-type="ref"}, [23](#cpt1549-bib-0023){ref-type="ref"}, [24](#cpt1549-bib-0024){ref-type="ref"}, [25](#cpt1549-bib-0025){ref-type="ref"}, [26](#cpt1549-bib-0026){ref-type="ref"}, [27](#cpt1549-bib-0027){ref-type="ref"}, [28](#cpt1549-bib-0028){ref-type="ref"}, [29](#cpt1549-bib-0029){ref-type="ref"}, [30](#cpt1549-bib-0030){ref-type="ref"}, [31](#cpt1549-bib-0031){ref-type="ref"}, [32](#cpt1549-bib-0032){ref-type="ref"}, [33](#cpt1549-bib-0033){ref-type="ref"}, [34](#cpt1549-bib-0034){ref-type="ref"}, [36](#cpt1549-bib-0036){ref-type="ref"}, [37](#cpt1549-bib-0037){ref-type="ref"}, [38](#cpt1549-bib-0038){ref-type="ref"}, [40](#cpt1549-bib-0040){ref-type="ref"}, [41](#cpt1549-bib-0041){ref-type="ref"}, [43](#cpt1549-bib-0043){ref-type="ref"}, [44](#cpt1549-bib-0044){ref-type="ref"}, [48](#cpt1549-bib-0048){ref-type="ref"}, [49](#cpt1549-bib-0049){ref-type="ref"}, [51](#cpt1549-bib-0051){ref-type="ref"}, [52](#cpt1549-bib-0052){ref-type="ref"}, [53](#cpt1549-bib-0053){ref-type="ref"}, [54](#cpt1549-bib-0054){ref-type="ref"}, [55](#cpt1549-bib-0055){ref-type="ref"}, [56](#cpt1549-bib-0056){ref-type="ref"}, [57](#cpt1549-bib-0057){ref-type="ref"}, [58](#cpt1549-bib-0058){ref-type="ref"}, [59](#cpt1549-bib-0059){ref-type="ref"}, [60](#cpt1549-bib-0060){ref-type="ref"}, [61](#cpt1549-bib-0061){ref-type="ref"}, [62](#cpt1549-bib-0062){ref-type="ref"}, [63](#cpt1549-bib-0063){ref-type="ref"}, [65](#cpt1549-bib-0065){ref-type="ref"}, [66](#cpt1549-bib-0066){ref-type="ref"}, [67](#cpt1549-bib-0067){ref-type="ref"}, [68](#cpt1549-bib-0068){ref-type="ref"}, [69](#cpt1549-bib-0069){ref-type="ref"}, [70](#cpt1549-bib-0070){ref-type="ref"}, [71](#cpt1549-bib-0071){ref-type="ref"}, [72](#cpt1549-bib-0072){ref-type="ref"}, [73](#cpt1549-bib-0073){ref-type="ref"}, [74](#cpt1549-bib-0074){ref-type="ref"}, [76](#cpt1549-bib-0076){ref-type="ref"} 11(20%) of which described e‐learning combined with traditional classroom teaching, making the intervention blended learning.[15](#cpt1549-bib-0015){ref-type="ref"}, [19](#cpt1549-bib-0019){ref-type="ref"}, [21](#cpt1549-bib-0021){ref-type="ref"}, [26](#cpt1549-bib-0026){ref-type="ref"}, [33](#cpt1549-bib-0033){ref-type="ref"}, [38](#cpt1549-bib-0038){ref-type="ref"}, [41](#cpt1549-bib-0041){ref-type="ref"}, [62](#cpt1549-bib-0062){ref-type="ref"}, [65](#cpt1549-bib-0065){ref-type="ref"}, [66](#cpt1549-bib-0066){ref-type="ref"}, [72](#cpt1549-bib-0072){ref-type="ref"}, [73](#cpt1549-bib-0073){ref-type="ref"} Eight articles (15%) combined e‐learning with some sort of digital communication channel, such as Web forums or chatrooms for interaction with peers and teachers.[17](#cpt1549-bib-0017){ref-type="ref"}, [21](#cpt1549-bib-0021){ref-type="ref"}, [28](#cpt1549-bib-0028){ref-type="ref"}, [36](#cpt1549-bib-0036){ref-type="ref"}, [40](#cpt1549-bib-0040){ref-type="ref"}, [51](#cpt1549-bib-0051){ref-type="ref"}, [63](#cpt1549-bib-0063){ref-type="ref"}, [66](#cpt1549-bib-0066){ref-type="ref"} Of the six articles (9%) describing a digital assessment, only one was not combined with e‐learning.[45](#cpt1549-bib-0045){ref-type="ref"} Nine articles (14%) described more unique learning interventions: gamified e‐learning,[18](#cpt1549-bib-0018){ref-type="ref"} a text‐based virtual patient,[35](#cpt1549-bib-0035){ref-type="ref"}, [64](#cpt1549-bib-0064){ref-type="ref"} an augmented reality virtual patient,[50](#cpt1549-bib-0050){ref-type="ref"} e‐learning with email case discussions,[75](#cpt1549-bib-0075){ref-type="ref"} a wiki‐like student drug formulary,[46](#cpt1549-bib-0046){ref-type="ref"} a drug‐dosage simulator,[39](#cpt1549-bib-0039){ref-type="ref"} a digital drug repository,[42](#cpt1549-bib-0042){ref-type="ref"} and informative podcasts.[47](#cpt1549-bib-0047){ref-type="ref"} Fifty‐one percent of interventions were aimed at postgraduate physicians,[12](#cpt1549-bib-0012){ref-type="ref"}, [13](#cpt1549-bib-0013){ref-type="ref"}, [15](#cpt1549-bib-0015){ref-type="ref"}, [18](#cpt1549-bib-0018){ref-type="ref"}, [19](#cpt1549-bib-0019){ref-type="ref"}, [20](#cpt1549-bib-0020){ref-type="ref"}, [23](#cpt1549-bib-0023){ref-type="ref"}, [24](#cpt1549-bib-0024){ref-type="ref"}, [27](#cpt1549-bib-0027){ref-type="ref"}, [28](#cpt1549-bib-0028){ref-type="ref"}, [29](#cpt1549-bib-0029){ref-type="ref"}, [30](#cpt1549-bib-0030){ref-type="ref"}, [31](#cpt1549-bib-0031){ref-type="ref"}, [33](#cpt1549-bib-0033){ref-type="ref"}, [34](#cpt1549-bib-0034){ref-type="ref"}, [36](#cpt1549-bib-0036){ref-type="ref"}, [40](#cpt1549-bib-0040){ref-type="ref"}, [42](#cpt1549-bib-0042){ref-type="ref"}, [50](#cpt1549-bib-0050){ref-type="ref"}, [54](#cpt1549-bib-0054){ref-type="ref"}, [55](#cpt1549-bib-0055){ref-type="ref"}, [58](#cpt1549-bib-0058){ref-type="ref"}, [59](#cpt1549-bib-0059){ref-type="ref"}, [60](#cpt1549-bib-0060){ref-type="ref"}, [62](#cpt1549-bib-0062){ref-type="ref"}, [64](#cpt1549-bib-0064){ref-type="ref"}, [65](#cpt1549-bib-0065){ref-type="ref"}, [67](#cpt1549-bib-0067){ref-type="ref"}, [68](#cpt1549-bib-0068){ref-type="ref"}, [71](#cpt1549-bib-0071){ref-type="ref"}, [72](#cpt1549-bib-0072){ref-type="ref"}, [73](#cpt1549-bib-0073){ref-type="ref"}, [74](#cpt1549-bib-0074){ref-type="ref"}, [75](#cpt1549-bib-0075){ref-type="ref"} 25% were aimed at nonmedical prescribers (all of whom were nurses),[15](#cpt1549-bib-0015){ref-type="ref"}, [17](#cpt1549-bib-0017){ref-type="ref"}, [18](#cpt1549-bib-0018){ref-type="ref"}, [21](#cpt1549-bib-0021){ref-type="ref"}, [22](#cpt1549-bib-0022){ref-type="ref"}, [29](#cpt1549-bib-0029){ref-type="ref"}, [35](#cpt1549-bib-0035){ref-type="ref"}, [44](#cpt1549-bib-0044){ref-type="ref"}, [47](#cpt1549-bib-0047){ref-type="ref"}, [58](#cpt1549-bib-0058){ref-type="ref"}, [61](#cpt1549-bib-0061){ref-type="ref"}, [62](#cpt1549-bib-0062){ref-type="ref"}, [63](#cpt1549-bib-0063){ref-type="ref"}, [64](#cpt1549-bib-0064){ref-type="ref"}, [66](#cpt1549-bib-0066){ref-type="ref"} 20% at medical students,[14](#cpt1549-bib-0014){ref-type="ref"}, [25](#cpt1549-bib-0025){ref-type="ref"}, [32](#cpt1549-bib-0032){ref-type="ref"}, [37](#cpt1549-bib-0037){ref-type="ref"}, [39](#cpt1549-bib-0039){ref-type="ref"}, [43](#cpt1549-bib-0043){ref-type="ref"}, [45](#cpt1549-bib-0045){ref-type="ref"}, [46](#cpt1549-bib-0046){ref-type="ref"}, [49](#cpt1549-bib-0049){ref-type="ref"}, [53](#cpt1549-bib-0053){ref-type="ref"}, [56](#cpt1549-bib-0056){ref-type="ref"}, [57](#cpt1549-bib-0057){ref-type="ref"}, [76](#cpt1549-bib-0076){ref-type="ref"} and 11% at pharmacy students.[14](#cpt1549-bib-0014){ref-type="ref"}, [38](#cpt1549-bib-0038){ref-type="ref"}, [48](#cpt1549-bib-0048){ref-type="ref"}, [51](#cpt1549-bib-0051){ref-type="ref"}, [52](#cpt1549-bib-0052){ref-type="ref"}, [69](#cpt1549-bib-0069){ref-type="ref"}, [70](#cpt1549-bib-0070){ref-type="ref"} Nine postgraduate courses (14%) were aimed at prescribers from more than one background;[15](#cpt1549-bib-0015){ref-type="ref"}, [18](#cpt1549-bib-0018){ref-type="ref"}, [29](#cpt1549-bib-0029){ref-type="ref"}, [58](#cpt1549-bib-0058){ref-type="ref"}, [61](#cpt1549-bib-0061){ref-type="ref"}, [62](#cpt1549-bib-0062){ref-type="ref"}, [63](#cpt1549-bib-0063){ref-type="ref"}, [64](#cpt1549-bib-0064){ref-type="ref"}, [71](#cpt1549-bib-0071){ref-type="ref"} one was an undergraduate course (the Australian National Prescribing Curriculum).[14](#cpt1549-bib-0014){ref-type="ref"} Elements such as anywhere, anytime availability and interaction were mostly present, but many interventions were not described in detail.

###### 

Characteristics of the digital learning resources

  Intervention characteristics     *N* (%)
  -------------------------------- -----------
  Type of learning intervention    
  E‐learning                       55 (84.6)
  Blended learning                 12 (18.5)
  Digital assessment               6 (9.2)
  Social learning                  9 (13.8)
  Other                            9 (13.8)
  Target population                
  Physicians                       49 (75.4)
  Nurse prescribers                16 (24.6)
  Pharmacists                      12 (18.5)
  Dentists                         4 (6.2)
  Education level                  
  Postgraduate                     44 (67.7)
  Final year                       4 (6.2)
  Undergraduate                    17 (26.2)
  Characteristics                  
  Compulsory                       
  Yes                              16 (24.6)
  No                               26 (40.0)
  Not reported                     23 (35.4)
  Anytime--anywhere availability   
  Yes                              42 (64.6)
  No                               4 (6.2)
  Not reported                     19 (29.2)
  Type of assessment               
  Formative                        20 (30.8)
  Summative                        12 (18.5)
  Not reported                     33 (50.8)
  Interactivity                    
  Yes                              41 (63.1)
  No                               3 (4.6)
  Not reported                     21 (32.3)
  Case oriented                    
  Yes                              34 (52.3)
  No                               2 (3.1)
  Not reported                     29 (44.6)

Multiple interventions and populations possible, percentages add up to \>100%. Others may not add up to 100% due to rounding of percentages.

John Wiley & Sons, Ltd

Research designs and study quality {#cpt1549-sec-0004}
----------------------------------

Ten articles (15%) had a descriptive design,[14](#cpt1549-bib-0014){ref-type="ref"}, [25](#cpt1549-bib-0025){ref-type="ref"}, [32](#cpt1549-bib-0032){ref-type="ref"}, [39](#cpt1549-bib-0039){ref-type="ref"}, [41](#cpt1549-bib-0041){ref-type="ref"}, [49](#cpt1549-bib-0049){ref-type="ref"}, [50](#cpt1549-bib-0050){ref-type="ref"}, [58](#cpt1549-bib-0058){ref-type="ref"}, [63](#cpt1549-bib-0063){ref-type="ref"}, [76](#cpt1549-bib-0076){ref-type="ref"} 11 (17%) a qualitative design,[13](#cpt1549-bib-0013){ref-type="ref"}, [15](#cpt1549-bib-0015){ref-type="ref"}, [19](#cpt1549-bib-0019){ref-type="ref"}, [20](#cpt1549-bib-0020){ref-type="ref"}, [21](#cpt1549-bib-0021){ref-type="ref"}, [30](#cpt1549-bib-0030){ref-type="ref"}, [35](#cpt1549-bib-0035){ref-type="ref"}, [46](#cpt1549-bib-0046){ref-type="ref"}, [47](#cpt1549-bib-0047){ref-type="ref"}, [57](#cpt1549-bib-0057){ref-type="ref"}, [66](#cpt1549-bib-0066){ref-type="ref"} and 44 (68%) a quantitative design.[12](#cpt1549-bib-0012){ref-type="ref"}, [16](#cpt1549-bib-0016){ref-type="ref"}, [17](#cpt1549-bib-0017){ref-type="ref"}, [18](#cpt1549-bib-0018){ref-type="ref"}, [22](#cpt1549-bib-0022){ref-type="ref"}, [23](#cpt1549-bib-0023){ref-type="ref"}, [24](#cpt1549-bib-0024){ref-type="ref"}, [26](#cpt1549-bib-0026){ref-type="ref"}, [27](#cpt1549-bib-0027){ref-type="ref"}, [28](#cpt1549-bib-0028){ref-type="ref"}, [29](#cpt1549-bib-0029){ref-type="ref"}, [31](#cpt1549-bib-0031){ref-type="ref"}, [33](#cpt1549-bib-0033){ref-type="ref"}, [34](#cpt1549-bib-0034){ref-type="ref"}, [36](#cpt1549-bib-0036){ref-type="ref"}, [37](#cpt1549-bib-0037){ref-type="ref"}, [38](#cpt1549-bib-0038){ref-type="ref"}, [40](#cpt1549-bib-0040){ref-type="ref"}, [42](#cpt1549-bib-0042){ref-type="ref"}, [43](#cpt1549-bib-0043){ref-type="ref"}, [44](#cpt1549-bib-0044){ref-type="ref"}, [45](#cpt1549-bib-0045){ref-type="ref"}, [48](#cpt1549-bib-0048){ref-type="ref"}, [51](#cpt1549-bib-0051){ref-type="ref"}, [52](#cpt1549-bib-0052){ref-type="ref"}, [53](#cpt1549-bib-0053){ref-type="ref"}, [54](#cpt1549-bib-0054){ref-type="ref"}, [55](#cpt1549-bib-0055){ref-type="ref"}, [56](#cpt1549-bib-0056){ref-type="ref"}, [59](#cpt1549-bib-0059){ref-type="ref"}, [60](#cpt1549-bib-0060){ref-type="ref"}, [61](#cpt1549-bib-0061){ref-type="ref"}, [62](#cpt1549-bib-0062){ref-type="ref"}, [64](#cpt1549-bib-0064){ref-type="ref"}, [65](#cpt1549-bib-0065){ref-type="ref"}, [67](#cpt1549-bib-0067){ref-type="ref"}, [68](#cpt1549-bib-0068){ref-type="ref"}, [69](#cpt1549-bib-0069){ref-type="ref"}, [70](#cpt1549-bib-0070){ref-type="ref"}, [71](#cpt1549-bib-0071){ref-type="ref"}, [72](#cpt1549-bib-0072){ref-type="ref"}, [73](#cpt1549-bib-0073){ref-type="ref"}, [74](#cpt1549-bib-0074){ref-type="ref"}, [75](#cpt1549-bib-0075){ref-type="ref"} Twenty‐six of the quantitative articles (59%) used a randomized[23](#cpt1549-bib-0023){ref-type="ref"}, [24](#cpt1549-bib-0024){ref-type="ref"}, [28](#cpt1549-bib-0028){ref-type="ref"}, [29](#cpt1549-bib-0029){ref-type="ref"}, [31](#cpt1549-bib-0031){ref-type="ref"}, [34](#cpt1549-bib-0034){ref-type="ref"}, [37](#cpt1549-bib-0037){ref-type="ref"}, [40](#cpt1549-bib-0040){ref-type="ref"}, [52](#cpt1549-bib-0052){ref-type="ref"}, [56](#cpt1549-bib-0056){ref-type="ref"}, [59](#cpt1549-bib-0059){ref-type="ref"}, [64](#cpt1549-bib-0064){ref-type="ref"}, [65](#cpt1549-bib-0065){ref-type="ref"}, [67](#cpt1549-bib-0067){ref-type="ref"}, [72](#cpt1549-bib-0072){ref-type="ref"}, [73](#cpt1549-bib-0073){ref-type="ref"}, [74](#cpt1549-bib-0074){ref-type="ref"}, [75](#cpt1549-bib-0075){ref-type="ref"} or nonrandomized[22](#cpt1549-bib-0022){ref-type="ref"}, [27](#cpt1549-bib-0027){ref-type="ref"}, [38](#cpt1549-bib-0038){ref-type="ref"}, [54](#cpt1549-bib-0054){ref-type="ref"}, [60](#cpt1549-bib-0060){ref-type="ref"}, [62](#cpt1549-bib-0062){ref-type="ref"}, [69](#cpt1549-bib-0069){ref-type="ref"}, [70](#cpt1549-bib-0070){ref-type="ref"} trial design; the remaining 18 articles (41%) either used no control or compared one group before and after the intervention.[12](#cpt1549-bib-0012){ref-type="ref"}, [16](#cpt1549-bib-0016){ref-type="ref"}, [17](#cpt1549-bib-0017){ref-type="ref"}, [18](#cpt1549-bib-0018){ref-type="ref"}, [26](#cpt1549-bib-0026){ref-type="ref"}, [33](#cpt1549-bib-0033){ref-type="ref"}, [36](#cpt1549-bib-0036){ref-type="ref"}, [42](#cpt1549-bib-0042){ref-type="ref"}, [43](#cpt1549-bib-0043){ref-type="ref"}, [44](#cpt1549-bib-0044){ref-type="ref"}, [45](#cpt1549-bib-0045){ref-type="ref"}, [48](#cpt1549-bib-0048){ref-type="ref"}, [51](#cpt1549-bib-0051){ref-type="ref"}, [53](#cpt1549-bib-0053){ref-type="ref"}, [55](#cpt1549-bib-0055){ref-type="ref"}, [61](#cpt1549-bib-0061){ref-type="ref"}, [68](#cpt1549-bib-0068){ref-type="ref"}, [71](#cpt1549-bib-0071){ref-type="ref"} The nature of the control group was diverse: five articles (11% of quantitative) compared the use of clinical practice guidelines to DLRs,[27](#cpt1549-bib-0027){ref-type="ref"}, [28](#cpt1549-bib-0028){ref-type="ref"}, [37](#cpt1549-bib-0037){ref-type="ref"}, [59](#cpt1549-bib-0059){ref-type="ref"}, [64](#cpt1549-bib-0064){ref-type="ref"} and six (14%) compared traditional face‐to‐face teaching with e‐learning.[22](#cpt1549-bib-0022){ref-type="ref"}, [34](#cpt1549-bib-0034){ref-type="ref"}, [38](#cpt1549-bib-0038){ref-type="ref"}, [54](#cpt1549-bib-0054){ref-type="ref"}, [60](#cpt1549-bib-0060){ref-type="ref"}, [65](#cpt1549-bib-0065){ref-type="ref"} Blended learning was compared with e‐learning in only two articles (5%).[38](#cpt1549-bib-0038){ref-type="ref"}, [62](#cpt1549-bib-0062){ref-type="ref"} There were also numerous less relevant control groups: eight articles (18%) compared the DLR to no education at all,[23](#cpt1549-bib-0023){ref-type="ref"}, [24](#cpt1549-bib-0024){ref-type="ref"}, [29](#cpt1549-bib-0029){ref-type="ref"}, [31](#cpt1549-bib-0031){ref-type="ref"}, [40](#cpt1549-bib-0040){ref-type="ref"}, [69](#cpt1549-bib-0069){ref-type="ref"}, [72](#cpt1549-bib-0072){ref-type="ref"}, [73](#cpt1549-bib-0073){ref-type="ref"} and four articles (9%) compared standard education to standard education with extra e‐learning.[52](#cpt1549-bib-0052){ref-type="ref"}, [56](#cpt1549-bib-0056){ref-type="ref"}, [70](#cpt1549-bib-0070){ref-type="ref"}, [75](#cpt1549-bib-0075){ref-type="ref"} The scores for the separate Medical Education Research Study Quality Instrument (MERSQI) domains are shown in **Table** [2](#cpt1549-tbl-0002){ref-type="table"}. The mean MERSQI was 11.6 ± 3.0; there were 8, 22, and 14, articles of low, medium, and high quality, respectively.

###### 

Study designs and quality

  Study characteristics                                        *N* (%)
  ------------------------------------------------------------ ------------
  Research design                                              
  Descriptive                                                  10 (15.4)
  Qualitative                                                  11 (16.9)
  Quantitative                                                 44 (67.7)
  MERSQI (*N* = 44 quantitative)                               
  Overall (mean ± SD)                                          11.6 ± 3.0
  Low (5.0--8.5)                                               8 (18.2)
  Medium (9.0--13.0)                                           22 (50.0)
  High (13.5--18.0)                                            14 (31.8)
  Study design (MERSQI score)                                  
  Single group, cross‐sectional (1)                            11 (25.0)
  Single group, before and after intervention (1.5)            7 (15.9)
  Prospective cohort (2)                                       8 (18.2)
  Randomized controlled trial (3)                              18 (40.9)
  Study outcomes -- highest Kirkpatrick level (MERSQI score)   
  Learner attitude -- Kirkpatrick level I (1)                  11 (25.0)
  Knowledge acquisition -- Kirkpatrick level II (1.5)          17 (38.6)
  Learner behavior -- Kirkpatrick level III (2)                10 (22.7)
  Patient outcomes -- Kirkpatrick level IV (3)                 6 (13.6)
  Sampling: Institutions (MERSQI score)                        
  Single institution (0.5)                                     22 (50.0)
  Two institutions (1)                                         5 (11.4)
  Three or more institutions (1.5)                             17 (38.6)
  Sampling: Response rate (MERSQI score)                       
  \<50% or not reported (0.5)                                  13 (29.5)
  50--74% (1)                                                  6 (13.6)
  ≥75% (1.5)                                                   25 (56.8)
  Type of data (MERSQI score)                                  
  Assessment by study participant (1)                          9 (20.5)
  Objective (3)                                                35 (79.5)
  Instrument validity (MERSQI score)                           
  Not applicable                                               1 (2.3)
  Content validity reported (1)                                12 (27.3)
  Internal structure reported (1)                              6 (13.6)
  Relationship with other variables reported (1)               7 (15.9)
  Data analysis: Sophistication (MERSQI score)                 
  Descriptive statistics only (1)                              8 (18.2)
  Beyond descriptive statistics (2)                            36 (81.8)
  Data analysis: Appropriate                                   
  Yes (1)                                                      34 (77.3)
  No (0)                                                       10 (22.7)

Instrument validity consists of three independently scored items (reported or not reported) and, therefore, does not add up to 100%. Others may not add up to 100% due to rounding of percentages.

MERSQI, Medical Education Research Study Quality Instrument.

John Wiley & Sons, Ltd

Head‐to‐head comparisons {#cpt1549-sec-0005}
------------------------

Four of five studies comparing DLRs to the use of clinical practice guidelines found no between‐group difference. The other article reported a significant increase in knowledge (8.4 ± 0.8 vs. 6.1 ± 1.3 on a 9‐question scale) and self‐reported competence (3.8 ± 0.6 vs. 3.5 ± 0.7 on a 10‐point Likert scale from incompetent to competent) in opioid prescribing after interactive web‐based training as compared with guideline only.[59](#cpt1549-bib-0059){ref-type="ref"} Articles directly comparing traditional teaching methods to DLRs reported equal effects in two,[38](#cpt1549-bib-0038){ref-type="ref"}, [54](#cpt1549-bib-0054){ref-type="ref"} higher knowledge test scores after interventions with e‐learning in two,[60](#cpt1549-bib-0060){ref-type="ref"}, [65](#cpt1549-bib-0065){ref-type="ref"} and a better performance on knowledge tests after a traditional lecture in one.[22](#cpt1549-bib-0022){ref-type="ref"} One article reported a 5% increase of prescribing errors made by physicians after e‐learning, but a decrease from 58% to 37% after pharmacist‐led feedback and targeted education.[34](#cpt1549-bib-0034){ref-type="ref"} The two articles comparing blended learning with nonblended DLRs found the intervention to be beneficial, but no relevant between‐group differences were found.

Patient outcomes (Kirkpatrick level IV) {#cpt1549-sec-0006}
---------------------------------------

Six articles reported outcomes at a patient level.[12](#cpt1549-bib-0012){ref-type="ref"}, [28](#cpt1549-bib-0028){ref-type="ref"}, [42](#cpt1549-bib-0042){ref-type="ref"}, [55](#cpt1549-bib-0055){ref-type="ref"}, [73](#cpt1549-bib-0073){ref-type="ref"}, [74](#cpt1549-bib-0074){ref-type="ref"} One found a relative risk of antibiotics being preferred by the patient of 0.48 (95% confidence interval 0.34--0.68) after an online training on shared decision making.[73](#cpt1549-bib-0073){ref-type="ref"} Another evaluated the effect of online training in the use of a point‐of‐care CRP test and online enhanced communications training, when these interventions were combined a relative risk of 0.38 (95% CI 0.25--0.55) was observed.[74](#cpt1549-bib-0074){ref-type="ref"} Both articles report no harm to patients. Three articles found no effects on prescribing errors of online trainings on the management of diabetes, elderly, and chronic opioid use.[12](#cpt1549-bib-0012){ref-type="ref"}, [28](#cpt1549-bib-0028){ref-type="ref"}, [55](#cpt1549-bib-0055){ref-type="ref"} One found no effect of a series of interventions, one of which was the introduction of a digital drug repository and e‐learning.[42](#cpt1549-bib-0042){ref-type="ref"}

Behavioral outcomes (Kirkpatrick level III) {#cpt1549-sec-0007}
-------------------------------------------

Eleven studies investigated behavioral change in learners.[24](#cpt1549-bib-0024){ref-type="ref"}, [29](#cpt1549-bib-0029){ref-type="ref"}, [34](#cpt1549-bib-0034){ref-type="ref"}, [56](#cpt1549-bib-0056){ref-type="ref"}, [60](#cpt1549-bib-0060){ref-type="ref"}, [62](#cpt1549-bib-0062){ref-type="ref"}, [65](#cpt1549-bib-0065){ref-type="ref"}, [68](#cpt1549-bib-0068){ref-type="ref"}, [72](#cpt1549-bib-0072){ref-type="ref"}, [74](#cpt1549-bib-0074){ref-type="ref"}, [75](#cpt1549-bib-0075){ref-type="ref"} Three found no clear benefit on prescribing errors as measured with medication chart reviews,[34](#cpt1549-bib-0034){ref-type="ref"}, [60](#cpt1549-bib-0060){ref-type="ref"}, [65](#cpt1549-bib-0065){ref-type="ref"} whereas the other eight reported positive effects, such as a reduction in antibiotic prescriptions,[24](#cpt1549-bib-0024){ref-type="ref"}, [72](#cpt1549-bib-0072){ref-type="ref"}, [74](#cpt1549-bib-0074){ref-type="ref"} the correct use of a Broselow pediatric dosing tape,[29](#cpt1549-bib-0029){ref-type="ref"} and improved legibility and completeness of written prescriptions.[62](#cpt1549-bib-0062){ref-type="ref"}

Knowledge outcomes (Kirkpatrick level II) {#cpt1549-sec-0008}
-----------------------------------------

All 19 articles that used knowledge acquisition as outcome reported the intervention to increase test scores as compared with preintervention testing or no education.[16](#cpt1549-bib-0016){ref-type="ref"}, [18](#cpt1549-bib-0018){ref-type="ref"}, [23](#cpt1549-bib-0023){ref-type="ref"}, [31](#cpt1549-bib-0031){ref-type="ref"}, [37](#cpt1549-bib-0037){ref-type="ref"}, [38](#cpt1549-bib-0038){ref-type="ref"}, [45](#cpt1549-bib-0045){ref-type="ref"}, [48](#cpt1549-bib-0048){ref-type="ref"}, [51](#cpt1549-bib-0051){ref-type="ref"}, [52](#cpt1549-bib-0052){ref-type="ref"}, [54](#cpt1549-bib-0054){ref-type="ref"}, [56](#cpt1549-bib-0056){ref-type="ref"}, [59](#cpt1549-bib-0059){ref-type="ref"}, [60](#cpt1549-bib-0060){ref-type="ref"}, [64](#cpt1549-bib-0064){ref-type="ref"}, [65](#cpt1549-bib-0065){ref-type="ref"}, [69](#cpt1549-bib-0069){ref-type="ref"}, [70](#cpt1549-bib-0070){ref-type="ref"}, [71](#cpt1549-bib-0071){ref-type="ref"} Three studies reported sustained effects on knowledge lasting up to 6 months.[23](#cpt1549-bib-0023){ref-type="ref"}, [31](#cpt1549-bib-0031){ref-type="ref"}, [59](#cpt1549-bib-0059){ref-type="ref"}

Learner satisfaction (Kirkpatrick level I) and self‐reported outcomes {#cpt1549-sec-0009}
---------------------------------------------------------------------

Twenty‐one articles reported the reaction of participants, mostly assessed using 5‐point Likert scales or open questions.[17](#cpt1549-bib-0017){ref-type="ref"}, [18](#cpt1549-bib-0018){ref-type="ref"}, [19](#cpt1549-bib-0019){ref-type="ref"}, [20](#cpt1549-bib-0020){ref-type="ref"}, [21](#cpt1549-bib-0021){ref-type="ref"}, [22](#cpt1549-bib-0022){ref-type="ref"}, [27](#cpt1549-bib-0027){ref-type="ref"}, [30](#cpt1549-bib-0030){ref-type="ref"}, [31](#cpt1549-bib-0031){ref-type="ref"}, [33](#cpt1549-bib-0033){ref-type="ref"}, [35](#cpt1549-bib-0035){ref-type="ref"}, [38](#cpt1549-bib-0038){ref-type="ref"}, [40](#cpt1549-bib-0040){ref-type="ref"}, [43](#cpt1549-bib-0043){ref-type="ref"}, [51](#cpt1549-bib-0051){ref-type="ref"}, [53](#cpt1549-bib-0053){ref-type="ref"}, [54](#cpt1549-bib-0054){ref-type="ref"}, [61](#cpt1549-bib-0061){ref-type="ref"}, [65](#cpt1549-bib-0065){ref-type="ref"}, [66](#cpt1549-bib-0066){ref-type="ref"}, [67](#cpt1549-bib-0067){ref-type="ref"} All results were positive, but no structural differences between DLRs and traditional teaching methods were identified. Thirteen studies found an increase in self‐reported confidence and knowledge on different domains of safe prescribing.[12](#cpt1549-bib-0012){ref-type="ref"}, [15](#cpt1549-bib-0015){ref-type="ref"}, [23](#cpt1549-bib-0023){ref-type="ref"}, [26](#cpt1549-bib-0026){ref-type="ref"}, [31](#cpt1549-bib-0031){ref-type="ref"}, [40](#cpt1549-bib-0040){ref-type="ref"}, [48](#cpt1549-bib-0048){ref-type="ref"}, [53](#cpt1549-bib-0053){ref-type="ref"}, [55](#cpt1549-bib-0055){ref-type="ref"}, [57](#cpt1549-bib-0057){ref-type="ref"}, [59](#cpt1549-bib-0059){ref-type="ref"}, [60](#cpt1549-bib-0060){ref-type="ref"}, [67](#cpt1549-bib-0067){ref-type="ref"}

Recommendations for future DLRs {#cpt1549-sec-0010}
-------------------------------

Recommendations for elements to include in future DLRs for safe prescribing were compiled on the basis of recurrent themes mentioned in qualitative reports and are shown in **Table** [3](#cpt1549-tbl-0003){ref-type="table"}.

###### 

Recommendations for future digital learning resources on safe and effective prescribing

  \#   Element to include in DLR                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Mentioned in
  ---- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  1    *Anytime*--*anywhere availability*. By far the most appreciated feature of DLRs is the possibility to study anytime and anywhere. It promotes just‐in‐time learning, puts the student in charge of his/her learning process, and enhances time efficiency. This is especially beneficial for postgraduates, because they often have little study time available because of their clinical duties and have external (e.g., continuous education points) or internal (e.g., interest in the topic) motivation to start learning.   13 articles[15](#cpt1549-bib-0015){ref-type="ref"}, [17](#cpt1549-bib-0017){ref-type="ref"}, [20](#cpt1549-bib-0020){ref-type="ref"}, [21](#cpt1549-bib-0021){ref-type="ref"}, [26](#cpt1549-bib-0026){ref-type="ref"}, [33](#cpt1549-bib-0033){ref-type="ref"}, [35](#cpt1549-bib-0035){ref-type="ref"}, [40](#cpt1549-bib-0040){ref-type="ref"}, [43](#cpt1549-bib-0043){ref-type="ref"}, [46](#cpt1549-bib-0046){ref-type="ref"}, [47](#cpt1549-bib-0047){ref-type="ref"}, [51](#cpt1549-bib-0051){ref-type="ref"}, [70](#cpt1549-bib-0070){ref-type="ref"}
  2    *Time demand*. The anytime--anywhere availability impinges on time scheduled for other activities, including social life. This may lead to motivational problems, and DLRs are often perceived as too long and time‐consuming. Openness about the required time investment and offering sufficient "protected" study time may help overcome these problems. Additionally, the use of bite‐size chunks of information is advocated, in accordance with the cognitive load theory.                                                 8 articles[17](#cpt1549-bib-0017){ref-type="ref"}, [20](#cpt1549-bib-0020){ref-type="ref"}, [21](#cpt1549-bib-0021){ref-type="ref"}, [22](#cpt1549-bib-0022){ref-type="ref"}, [44](#cpt1549-bib-0044){ref-type="ref"}, [53](#cpt1549-bib-0053){ref-type="ref"}, [66](#cpt1549-bib-0066){ref-type="ref"}, [70](#cpt1549-bib-0070){ref-type="ref"}
  3    *Learning pace and revision*. Associated with the anywhere−anytime availability is the possibility for students to alter pace, skip sections, and revise parts of the DLR extensively. This can be facilitated by structuring the DLR and offering clear navigational menus.                                                                                                                                                                                                                                                     6 articles [22](#cpt1549-bib-0022){ref-type="ref"}, [26](#cpt1549-bib-0026){ref-type="ref"}, [33](#cpt1549-bib-0033){ref-type="ref"}, [44](#cpt1549-bib-0044){ref-type="ref"}, [47](#cpt1549-bib-0047){ref-type="ref"}, [51](#cpt1549-bib-0051){ref-type="ref"}
  4    *The use of quizzes and game design principles*. Interactive elements, such as quizzes and other game design principles (e.g., competition with peers), are stimulating. Moreover, they help students to gauge their learning needs and may direct them to parts of the DLR that require revision.                                                                                                                                                                                                                               4 articles[21](#cpt1549-bib-0021){ref-type="ref"}, [44](#cpt1549-bib-0044){ref-type="ref"}, [47](#cpt1549-bib-0047){ref-type="ref"}, [69](#cpt1549-bib-0069){ref-type="ref"}
  5    *Feedback*. The possibility to offer direct feedback on (multiple choice) questions is found to be helpful, but is limited to generic pregenerated content that may sometimes be insufficient.                                                                                                                                                                                                                                                                                                                                   6 articles[22](#cpt1549-bib-0022){ref-type="ref"}, [35](#cpt1549-bib-0035){ref-type="ref"}, [43](#cpt1549-bib-0043){ref-type="ref"}, [53](#cpt1549-bib-0053){ref-type="ref"}, [57](#cpt1549-bib-0057){ref-type="ref"}, [69](#cpt1549-bib-0069){ref-type="ref"}
  6    *Contact with peers and teachers*. The lack of social interaction with peers and teachers is not easily overcome by using digital communication channels, such as Web forums and email, as these features remained largely unused.                                                                                                                                                                                                                                                                                               5 articles[15](#cpt1549-bib-0015){ref-type="ref"}, [22](#cpt1549-bib-0022){ref-type="ref"}, [36](#cpt1549-bib-0036){ref-type="ref"}, [51](#cpt1549-bib-0051){ref-type="ref"}, [66](#cpt1549-bib-0066){ref-type="ref"}
  7    *Content*. Generic, irrelevant, or unauthentic content should be avoided. Instead, there should be a clear connection with predefined and clearly communicated learning goals. The DLR is appreciated if acquired skills and knowledge are directly applicable.                                                                                                                                                                                                                                                                  12 articles[13](#cpt1549-bib-0013){ref-type="ref"}, [15](#cpt1549-bib-0015){ref-type="ref"}, [17](#cpt1549-bib-0017){ref-type="ref"}, [18](#cpt1549-bib-0018){ref-type="ref"}, [19](#cpt1549-bib-0019){ref-type="ref"}, [20](#cpt1549-bib-0020){ref-type="ref"}, [21](#cpt1549-bib-0021){ref-type="ref"}, [22](#cpt1549-bib-0022){ref-type="ref"}, [44](#cpt1549-bib-0044){ref-type="ref"}, [51](#cpt1549-bib-0051){ref-type="ref"}, [57](#cpt1549-bib-0057){ref-type="ref"}, [69](#cpt1549-bib-0069){ref-type="ref"}
  8    *Accessibility and design*. Limited accessibility due to low internet connectivity, other IT problems, or insufficient computer knowledge of the users is highly demotivating, as is an outdated or visually unattractive design. Some accessibility issues may be limited by offering on‐demand computer support.                                                                                                                                                                                                               10 articles[13](#cpt1549-bib-0013){ref-type="ref"}, [15](#cpt1549-bib-0015){ref-type="ref"}, [17](#cpt1549-bib-0017){ref-type="ref"}, [18](#cpt1549-bib-0018){ref-type="ref"}, [22](#cpt1549-bib-0022){ref-type="ref"}, [44](#cpt1549-bib-0044){ref-type="ref"}, [47](#cpt1549-bib-0047){ref-type="ref"}, [51](#cpt1549-bib-0051){ref-type="ref"}, [66](#cpt1549-bib-0066){ref-type="ref"}, [69](#cpt1549-bib-0069){ref-type="ref"}
  9    *Multimedia and learning styles*. The use of multimedia, such as audio podcasts, videos, or animations, can be very helpful in explaining challenging content. However, different users prefer different learning styles, and it advisable to present content in various ways.                                                                                                                                                                                                                                                   4 articles[18](#cpt1549-bib-0018){ref-type="ref"}, [20](#cpt1549-bib-0020){ref-type="ref"}, [22](#cpt1549-bib-0022){ref-type="ref"}, [44](#cpt1549-bib-0044){ref-type="ref"}
  10   *Web links*. Links to other resources (e.g., scientific articles, online drug formularies) may aid learning and are easily incorporated in the DLR. They can be distracting if too numerous or not entirely relevant.                                                                                                                                                                                                                                                                                                            5 articles[17](#cpt1549-bib-0017){ref-type="ref"}, [21](#cpt1549-bib-0021){ref-type="ref"}, [35](#cpt1549-bib-0035){ref-type="ref"}, [46](#cpt1549-bib-0046){ref-type="ref"}, [66](#cpt1549-bib-0066){ref-type="ref"}

DLR, digital learning resource; IT, information technology.

John Wiley & Sons, Ltd

Discussion {#cpt1549-sec-0011}
==========

To the best of our knowledge, this article is the first to systematically assess the effects of different types of digital teaching in the context of prescribing education. E‐learning, blended learning, and digital assessment are commonly used in prescribing education, and the effects on knowledge acquisition are comparable with those of traditional teaching methods. Additionally, DLRs seem to be effective in teaching skills associated with safe and effective prescribing, but whether this directly benefits the patient remains questionable. The most advantageous and disadvantageous elements, as identified from qualitative reports, are compiled into recommendations for best practice.

Types of DLR {#cpt1549-sec-0012}
------------

Novel techniques, such as augmented reality and serious games, are increasingly used for training healthcare providers. Perhaps unsurprisingly, these techniques focus primarily on teaching the visuospatial aspects of subjects, such as anatomy and surgery.[77](#cpt1549-bib-0077){ref-type="ref"}, [78](#cpt1549-bib-0078){ref-type="ref"} While Nifakos and Zary[50](#cpt1549-bib-0050){ref-type="ref"} describe the potential to combine virtual patients with augmented reality for teaching safe and effective prescribing, only two studies actually used a text‐based (not augmented) virtual patient for this purpose.[35](#cpt1549-bib-0035){ref-type="ref"}, [50](#cpt1549-bib-0050){ref-type="ref"}, [64](#cpt1549-bib-0064){ref-type="ref"} Bond *et al*.[18](#cpt1549-bib-0018){ref-type="ref"} reported the use of serious game design principles in their e‐learning program, but no serious games were identified. The use of an antibiotic dose‐response simulator for teaching dose regimens was described but was not assessed for effectiveness.[39](#cpt1549-bib-0039){ref-type="ref"} Most DLRs were produced and used locally. Only Australia (National Prescribing Curriculum[79](#cpt1549-bib-0079){ref-type="ref"}), England (SCRIPT and Prescribe[80](#cpt1549-bib-0080){ref-type="ref"}, [81](#cpt1549-bib-0081){ref-type="ref"}), and the Netherlands (Pscribe[82](#cpt1549-bib-0082){ref-type="ref"}) have nationally available prescribing e‐learning modules. An example of a more international DLR is the Teaching Resource Center (TRC), produced by the Centre of Human Drug Resource (CHDR) and Leiden University Medical Centre in the Netherlands.[83](#cpt1549-bib-0083){ref-type="ref"}, [84](#cpt1549-bib-0084){ref-type="ref"} Although this longitudinal program is used for teaching the six‐stepmodels of rational prescribing from the World Health Organization\'s Guide to good prescribing, it was not included in the current analyses because published articles on the TRC focused on basic pharmacology rather than on prescribing training. It is, however, a good example of how the free distribution of resources could lead to widespread and international adoption. Higher numbers of users will increase cost‐effectiveness, and it is known that sharing DLRs as open educational resources (free to use, modify, and distribute derivatives) enhances their timely improvement and overall quality.[85](#cpt1549-bib-0085){ref-type="ref"} Unfortunately, only one of the included articles mentioned sharing the DLR in a truly open manner.[43](#cpt1549-bib-0043){ref-type="ref"}

Effects of DLRs on safe and effective prescribing {#cpt1549-sec-0013}
-------------------------------------------------

A recent overview of review articles on medical education found that DLRs are at least as efficacious as traditional teaching methods when it comes to knowledge acquisition and student satisfaction.[8](#cpt1549-bib-0008){ref-type="ref"} However, these results were found for healthcare education in general and might not be applicable to prescribing education, because prescribing is a complex skill that is affected by the prescriber\'s attitude and by the workplace environment.[10](#cpt1549-bib-0010){ref-type="ref"} Moreover, the effects of DLRs on skills and patient outcomes are less evident and often reported for specific skills (e.g., intubation) only.[86](#cpt1549-bib-0086){ref-type="ref"}, [87](#cpt1549-bib-0087){ref-type="ref"} This review identified several effects of DLRs on diverse skills and attitudes associated with safe and effective prescribing, such as moderation of antibiotic use. Therefore, it seems that DLRs are efficacious in teaching safe and effective prescribing skills and attitudes. However, we did not find DLR use to be accompanied by a reduction in prescribing errors or by a benefit to patients. This is probably because the process of prescribing medication is multifactorial and aspects, such as workplace environment and workload, may influence the results.[88](#cpt1549-bib-0088){ref-type="ref"} Most studies assessing effects on knowledge acquisition only compared outcomes before and after the intervention or did not use a control. Unsurprisingly, these studies found positive effects of DLRs. Of the five studies directly comparing pharmacotherapeutic DLRs with traditional teaching methods, two favored DLRs, two had equivocal findings, and one favored traditional teaching methods. However, the effect of DLRs relative to guideline use and for other healthcare disciplines is well established, and, overall, DLRs are found to be no better or worse than traditional teaching methods.[8](#cpt1549-bib-0008){ref-type="ref"}, [9](#cpt1549-bib-0009){ref-type="ref"} Sikkens *et al*.[56](#cpt1549-bib-0056){ref-type="ref"} reported a mixture of knowledge and behavioral outcomes by examining students by means of a simulated pharmacotherapeutic consultation with a patient actor. This type of objective structured clinical examination (OSCE) may be the best surrogate we have to real‐life prescribing for undergraduate students. Interestingly, they found that students performed significantly better on the OSCE 6 months after the e‐learning intervention than did students who did not follow the e‐learning intervention. Several other articles reported sustained effects for up to 3 months, suggesting that DLRs have a long‐term effect on knowledge. Overall, learners are well satisfied with DLRs for safe and effective prescribing. However, a positive feeling about DLRs is no guarantee for their quality, and it is more interesting to find which elements of DLRs improve quality.[7](#cpt1549-bib-0007){ref-type="ref"}

Recommendations for the production of DLRs for safe prescribing {#cpt1549-sec-0014}
---------------------------------------------------------------

The Association for Medical Education in Europe (AMEE) guide \#32 on e‐learning in medical education highlights the potential advantages of DLRs, but provides relatively little information about which elements to include or avoid when producing a DLR.[89](#cpt1549-bib-0089){ref-type="ref"} In addition to providing a practical framework to create postgraduate medical DLRs, de Leeuw *et al*.[90](#cpt1549-bib-0090){ref-type="ref"}, [91](#cpt1549-bib-0091){ref-type="ref"}, [92](#cpt1549-bib-0092){ref-type="ref"} identified relevant elements on the basis of focus interviews and an international Delphi study. Although their research focused on postgraduate continuous medical education in general, the recommendations are likely to be relevant for (undergraduate and postgraduate) prescribing education as well. Unfortunately, owing to heterogeneous study designs, outcomes, and insufficient reporting of DLR characteristics (e.g., presence of cases and interactive quizzes), quantitative analyses were not feasible. However, on the basis of qualitative reports, we were able to compile a list of 10 recommendations for the future development of DLRs for safe and effective prescribing (**Table** [3](#cpt1549-tbl-0003){ref-type="table"}). These recommendations are in accordance with those previously identified for (postgraduate) medical education in general.[8](#cpt1549-bib-0008){ref-type="ref"}, [90](#cpt1549-bib-0090){ref-type="ref"} The most appreciated element is anytime−anywhere availability. Not only may this save the learner precious (travel) time, but the autonomy associated with this is a pillar of adult learning theory and stimulates just‐in‐time learning, improving the relevance and applicability of the content. On the other hand, the anytime−anywhere availability affects the learner\'s time scheduled for other (e.g., social) activities. This is one of the reasons why DLRs are often perceived as requiring too much time. Practical solutions exist (e.g., offering protected study time), but this does not address the underlying problem: a lack of motivation. Unsurprisingly, many of the other recommended elements (game design principles, multimedia, and directly applicable content) aim to increase the learner\'s motivation.[90](#cpt1549-bib-0090){ref-type="ref"} Many articles report the lack of peer and teacher interaction to be disadvantageous or mention the presence of such contact (with blended learning) as beneficial. Several DLRs tried to accommodate this by making digital communication possible. Unfortunately, many of these techniques were insufficiently used by the students. Therefore, it seems that while DLRs are complementary to traditional prescribing education, it remains questionable whether they will ever be able to replace face‐to‐face prescribing education.

Limitations {#cpt1549-sec-0015}
-----------

Undoubtedly, useful digital teaching tools are being used but not scientifically studied or published. This is evidenced by the numerous relevant conference abstracts that have never appeared as full‐text articles. Therefore, it seems that underreporting (and perhaps publication bias) may have limited the integrity of this review article. In order to thoroughly assess the types of DLR used for prescribing education, other research designs, such as (international) surveys among prescribing educators, may be valuable. The mean quality of the articles included in this review was low‐to‐moderate, but surprisingly slightly higher (difference 1.1, *P* = 0.03 compared with Brinkman *et al*.[1](#cpt1549-bib-0001){ref-type="ref"}) than that found in previous review studies.[1](#cpt1549-bib-0001){ref-type="ref"}, [93](#cpt1549-bib-0093){ref-type="ref"} This is most likely due to the nature of DLRs, making it relatively easy to perform multicenter trials with adequate response rates. Moreover, a high proportion of studies used a randomized controlled trial design, leading to high MERSQI scores.[94](#cpt1549-bib-0094){ref-type="ref"} However, the relevance of the control groups is not scored in the MERSQI, and an alarming number of studies compared the DLR with no education at all. The obvious findings of these articles are of limited value and outcomes may be easily biased due to the Hawthorne effect.[95](#cpt1549-bib-0095){ref-type="ref"} Although the search was compiled and performed with the assistance of an experienced medical information specialist, it is possible that we missed relevant literature because our search criteria were too strict. This is supported by the finding that a previously used database, not focused on prescribing education, contained six previously unidentified relevant articles. During the selection process, it seemed that our predefined eligibility criteria left some gray area as to what is considered "teaching safe and effective prescribing." However, consensus was reached by including up to three experts in the decision process. For example, although relevant to optimal medication use, we chose to exclude DLRs that teach drug dosage calculations to (unspecialized) nurses, because in many countries nurses are not licensed to prescribe medicine. These decisions limit the repeatability of our findings.

Conclusions {#cpt1549-sec-0016}
===========

E‐learning is the most‐used type of digital learning for safe and effective prescribing education and is efficacious in teaching undergraduate and postgraduate prescribers the required knowledge, skills, and attitudes. Although this may ultimately benefit patient care, direct effects on patient outcomes have not yet been established. Ten recommendations for the future development of pharmacotherapeutic DLRs are provided (**Table** [3](#cpt1549-tbl-0003){ref-type="table"}).

Implications for future research {#cpt1549-sec-0017}
--------------------------------

Now that we know DLRs can be used to teach safe and effective prescribing, future research should focus on their broad implementation in European (and perhaps worldwide) medical curricula. Because the included articles rarely reported DLRs to be openly shared between universities, future research should focus on improving the sharing of educational resources and on understanding why educators are hesitant to do so. Obviously, patient care is influenced by many factors other than educational interventions for prescribers. Therefore, it remains difficult to show the effects of such interventions on "hard" outcomes. However, our data show that it is feasible to achieve effects directly influencing the patient, and that this results in the most relevant articles. Therefore, we recommend future DLRs to be evaluated as high as possible on the Kirkpatrick pyramid of educational outcomes, preferably on the level of patient outcomes. In addition, future articles evaluating the effects of DLRs should compare these interventions with relevant control interventions rather than no education.

Search Strategy and Data Analysis {#cpt1549-sec-0018}
=================================

Databases PubMed, EMBASE, Education Resources Information Centre (ERIC), the Cumulative Index to Nursing and Allied Health Literature (CINAHL), and the Cochrane database (CENTRAL) were searched for articles on digital methods (e‐learning, online course, virtual reality, etc.) for teaching CPT or prescribing (prescribing, drug therapy, pharmacovigilance, etc.) to graduate or undergraduate prescribers (physicians, advance nurse practitioners, medical students, etc.). A complete list of the search terms used, as well as the complete searches, can be found in **Table** [S2](#cpt1549-sup-0002){ref-type="supplementary-material"}. The search in the first four databases was performed on July 30, 2018; the search in CENTRAL was added on March 6, 2019. No other date restrictions were used. Additionally, colleagues from the Centre for Population Health Sciences (CePHaS), Singapore, provided a previously compiled database (unpublished data) of the effects of e‐learning on healthcare education. This database was manually searched for articles relevant to prescribing education. The search details are presented in **Table** [S2](#cpt1549-sup-0002){ref-type="supplementary-material"}.

Study selection {#cpt1549-sec-0019}
---------------

Articles were included if they described or evaluated at least one DLR for teaching safe and effective prescribing to graduate and undergraduate medical and nonmedical prescribers (doctors, nurse specialists, dentists, and pharmacists). Exclusion criteria were: (i) articles not written in English or Dutch, (ii) not an original research article (e.g., conference abstracts) or unavailability of the fulltext, (iii) learning resources not using digital technologies, and (iv) DLRs aimed at prescribing other therapies than medications (e.g., dietary and transfusion). Articles were selected independently by two reviewers (M.B. and A.W.); discrepancies were discussed, and when no agreement was reached a third reviewer (J.T.) was consulted.

Definitions, data extraction, and study quality assessment {#cpt1549-sec-0020}
----------------------------------------------------------

**Table** [4](#cpt1549-tbl-0004){ref-type="table"} lists the definitions used to classify the DLRs. These classifications and other data were extracted into a spreadsheet independently by two reviewers (M.B. and A.W.) and cross‐checked between them. Differences were resolved by discussion. Research designs were categorized as descriptive (articles describing the production of a DLR without evaluation), qualitative (open‐ended survey questions or interviews, but no quantitative outcomes), or quantitative (studies using any kind of quantitative measurement, including Likert‐type surveys). The qualitative and quantitative results of mixed‐methods articles are presented separately, but the research design was scored on the quantitative part. The quality of the quantitative studies was assessed using the MERSQI.[94](#cpt1549-bib-0094){ref-type="ref"} The MERSQI consists of six domains. Per domain, a maximum of three points may be scored; the minimum varies, leading to a possible score of 5--18 points. Some domains may be scored nonapplicable, leading to a lower maximum score; this was corrected using the formula: MERSQI = (total score/maximum possible)\*18. Although no official cutoff values exist, scores of \<8.5, 8.5--13.5, and \>13.5 were used as cutoff for low, intermediate, and high quality, respectively, in line with previous articles.[93](#cpt1549-bib-0093){ref-type="ref"}, [96](#cpt1549-bib-0096){ref-type="ref"} The outcomes were grouped according to Kirkpatrick\'s pyramid model of educational outcomes.[97](#cpt1549-bib-0097){ref-type="ref"}

###### 

Definitions of digital learning resources

  Type of digital learning resource   Definition used for this review
  ----------------------------------- --------------------------------------------------------------------------------------------------------------------------------------------------------
  E‐learning                          Web‐based (online) or offline course offering a mixture of text, multimedia, and interactive elements
  Digital assessment                  Formative or summative examination, exceeding the use of simple interactive questions in e‐learning
  Social component                    (Mostly online) learning environment including any type of digital interaction with peers or teachers, such as Web forums, chatrooms, or email support
  Podcasts                            Informative digital audio files
  Blended learning                    Any type of digital learning resource combined with face‐to‐face teaching
  Virtual patient                     Patient simulator (may vary from simple text‐based to virtual/augmented reality)
  Virtual reality/patient             Interactive computer‐generated experience in a virtual environment
  Augmented reality                   Virtual elements projected on real‐world environment
  Serious/educational game            A video game with the purpose of training an individual

The definitions used for this review are adapted from van den Berg *et al*. and Piovesan *et al*.[98](#cpt1549-bib-0098){ref-type="ref"}, [99](#cpt1549-bib-0099){ref-type="ref"}

John Wiley & Sons, Ltd

Data analysis {#cpt1549-sec-0021}
-------------

Data were analyzed using SPSS (IBM SPSS Statistics for Windows, version 22.0; IBM, Armonk, NY) and Microsoft Excel 2016 (Redmond, WA). Owing to heterogeneity in study design and outcomes, only descriptive statistics were used.

Funding {#cpt1549-sec-0023}
=======

No funding was received for this work.

Conflict of Interest {#cpt1549-sec-0024}
====================

The authors declared no competing interests for this work.

Supporting information
======================

###### 

**Table S1.** Included articles.

###### 

Click here for additional data file.

###### 

**Table S2.** Search strategies.

###### 

Click here for additional data file.

The authors would like to thank René H.J. Otten, information specialist of the medical library, VU University Amsterdam for his assistance with the database searches. The authors thank Louise S. van Galen, Lorainne Tudor Car, and Josip Car from the Centre for Population Health Sciences (CePHaS), Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore, for the provision of a previously used database on e‐learning for health sciences. This research was conducted on behalf of the education working group of the European Association of Clinical Pharmacology and Therapeutics (EACPT).
