Background {#Sec1}
==========

Motivation {#Sec2}
----------

An important component of electronic health record systems is the use of clinical decision support (CDS) to improve medication safety \[[@CR1],[@CR2]\]. Preventable adverse drug reactions include those resulting from drug-drug interactions (DDIs) \[[@CR3],[@CR4]\]. Among other things, CDS systems leverage drug-drug interaction information to reduce the possibility of adverse drug events \[[@CR5],[@CR6]\]. While there are many sources of DDI information, and many commercially available systems which contain this information, there is limited consensus on what constitutes critical and non-critical DDI \[[@CR7]\].

One source of publicly available DDI information is contained in the National Drug File Reference Terminology (NDF-RT) and available through the NDF-RT application programming interface (API). Several applications developed for medication management take advantage of the API for checking interactions in medication lists (e.g., the iOS app "Dosage" for medication information and reminders). However, it was announced before the writing of this article that the DDI information would be soon removed from NDF-RT \[[@CR8]\]. Our search for another publicly available source of DDI information led us to evaluate the DrugBank data source as a possible replacement for the NDF-RT DDIs.

In this paper, our objective is to evaluate the DDI information in NDF-RT and DrugBank. More specifically, we contrast NDF-RT and DrugBank interactions against each other, and contrast both against a previously published reference set of critical DDIs \[[@CR9]\]. We also contrast NDF-RT and DrugBank in their ability to detect DDIs in a large prescription dataset. Finally, we compare the coverage of the reference set by DrugBank and NDF-RT to that of a commercial source.

Drug information sources {#Sec3}
------------------------

The following sections detail the characteristics of the drug information sources used in this research. We use RxNorm to harmonize drugs between the two sources of DDIs under investigation, NDF-RT and DrugBank.

### NDF-RT {#Sec4}

The National Drug File Reference Terminology (NDF-RT) is a resource developed by the Department of Veterans Affairs (VA) Veterans Health Administration, as an extension of the VA National Drug File \[[@CR10]\]. It is updated monthly. NDF-RT covers 7319 active moieties (level = ingredient). In addition to providing information about individual drugs (e.g., mechanism of action, physiologic effect, therapeutic intent), NDF-RT also provides a set of 10,831 drug-drug interactions (DDIs). DDIs are asserted at the ingredient level and accompanied by a mention of severity (significant or critical). For example, NDF-RT asserts a critical interaction between *omeprazole* and *clopidogrel* and asserts a significant interaction between *diltiazem* and *lovastatin*. The version used in this study is dated July 7, 2014 and was accessed through the NDF-RT API \[[@CR11]\]. (Provision of DDI information in NDF-RT was discontinued in November 2014).

### DrugBank {#Sec5}

Developed with funding from Genome Canada, DrugBank is a knowledge base containing "extensive biochemical and pharmacological information about drugs, their mechanisms and their targets" \[[@CR12]\]. DrugBank covers 7683 active moieties. Although not primarily developed for clinical use, DrugBank provides a set of 12,128 drug-drug interactions (DDIs), asserted at the ingredient level, along with a brief textual description of the interaction, and information about the possible molecular basis of the interaction (target-based, enzyme-based, transporter-based). For example, DrugBank asserts an interaction between *omeprazole* and *clopidogrel bisulfate*, described as "Omeprazole may decrease serum concentrations of the active metabolite(s) of clopidogrel. Clopidogrel prescribing information recommends avoiding concurrent use with omeprazole, due to the possibility that combined use may result in decreased clopidogrel effectiveness". The possible molecular basis of the interaction is reported to be enzyme-based or transporter-based. The version used in this study (4.0) was downloaded from the DrugBank website on July 1, 2014.

### RxNorm {#Sec6}

RxNorm is a standardized nomenclature for medications produced and maintained by the U.S. National Library of Medicine (NLM) \[[@CR13]\]. While NDF-RT is one of the drug information sources integrated in RxNorm, DrugBank is not. However, most ingredients from DrugBank are covered by RxNorm and RxNorm can be used to map drugs between DrugBank and NDF-RT. Moreover, RxNorm provides a rich network of relations among various types of drug entities. For example, RxNorm explicitly asserts that *clopidogrel bisulfate* is the "precise ingredient" of the ingredient *clopidogrel*, making it possible to normalize the various salts and esters of a drug to their base form. RxNorm also integrates the Anatomical Therapeutic Chemical (ATC) classification, which makes it possible to extract the ATC class for most drugs. For example ATC classifies *clopidogrel* as a *Platelet aggregation inhibitors excl. heparin*. The July 2014 version of RxNorm is used in this study and was accessed through the RxNorm API \[[@CR14]\].

Related work {#Sec7}
------------

Extracting drug-drug interaction (DDI) information from textual resources, such as the biomedical literature or structured product labels is an active field of research \[[@CR15]-[@CR18]\]. Other researchers predict DDIs from a variety of resources \[[@CR19]\].

As mentioned earlier, researchers have shown the benefit of integrating DDI information in CPOEs (e.g., \[[@CR1],[@CR2]\]). However, not all studies demonstrate improvement on medication safety, especially due to the large number of alerts produced by some systems \[[@CR20]\], which raises questions about the quality of the underlying DDI information.

Given the limited consensus across sources of DDI information \[[@CR7]\], researchers have proposed criteria for assessing high-priority DDIs \[[@CR21]\] and for calibrating CDS systems \[[@CR22]\]. An expert panel was convened to identify high-severity, clinically significant DDIs for the Office of the National Coordinator for Health Information Technology (ONC) as part of the *Meaningful Use* incentive program \[[@CR9]\]. Candidate DDIs were assessed by the panel based on a number of factors, including severity levels across medication knowledge bases, consequences of the interaction, predisposing factors, and availability of therapeutic alternatives. The resulting list contains 360 interacting pairs of individual drugs containing 86 unique drugs. This list will be referred to as the reference set of DDIs in the following sections.

Specific contribution {#Sec8}
---------------------

The specific contribution of our work is to contrast two publicly available sources of DDI information, NDF-RT and DrugBank, through an assessment of the overlap of their content, and of their coverage of a reference set of DDIs. Moreover, we compare the ability of these two sources to identify DDIs in a large prescription dataset, and contrast them against a commercial source. To the best of our knowledge, this is the first such comparative investigation of NDF-RT and DrugBank DDI information.

Methods {#Sec9}
=======

Our approach to evaluating drug-drug interaction (DDI) information in NDF-RT and DrugBank can be summarized as follows. We acquire the list of drug-drug interactions in NDF-RT and DrugBank, as well as a reference set of DDIs. We map all drugs from the three sets to RxNorm and further normalize them to ingredient entities. We then compare the lists of pairs of interacting drugs across sources in order to determine the shared coverage between NDF-RT and DrugBank, as well as the coverage of the reference set by both sources. We characterize the differences among DDI sets in terms of drug classes. We also compare the interactions detected with NDF-RT and DrugBank in a large prescription dataset. Finally, we compare the coverage of the reference set by DrugBank and NDF-RT to that of a commercial source.

Acquiring DDI information {#Sec10}
-------------------------

### NDF-RT {#Sec11}

We used the NDF-RT API \[[@CR11]\] to first extract the full set of DDIs (DRUG_INTERACTION_KIND concepts), then to extract each associated drug concept (level = ingredient) in the pair.

### DrugBank {#Sec12}

The DrugBank XML and schema definition files were downloaded from the DrugBank web site. We extracted the interaction data from the XML file and created a table of drug name pairs for the interacting drugs.

### Reference set {#Sec13}

The reference set of DDIs was created from the drug names listed in Table two of \[[@CR9]\] by associating each object drug with all corresponding precipitant drug(s) within a given interaction class. One pair involving a multi-ingredient drug (*azathioprine and mercaptopurine*) was eliminated, because multi-ingredient drugs are generally not consistently represented across sources.

Normalizing drugs in reference to RxNorm {#Sec14}
----------------------------------------

After obtaining the drug name pairs for interacting drugs, we mapped the drugs to RxNorm by retrieving the RxNorm identifiers (RxCUIs). For NDF-RT, the RxCUI is part of the drug's concept properties, so we used the NDF-RT API to extract the RxCUI from the drug properties. For DrugBank and the reference set, we used the RxNorm API \[[@CR7]\] to find the RxCUI from the drug name. More specifically, we used exact and normalized string matches to map drug names to RxNorm.

We then "normalized" to RxNorm ingredients those drugs which mapped to RxNorm entities. Some of the drugs corresponded to base ingredients (e.g., *doxacurium*), while others corresponded to salt forms that RxNorm classifies as "precise ingredients" (e.g., *doxacurium chloride*). In order to establish a consistent drug representation across all three data sets for comparison, we converted precise ingredients to RxNorm ingredients (e.g., *doxacurium chloride* to *doxacurium*) using the RxNorm API.

We eliminated from the comparison those pairs for which at least one of the drugs could not be found in RxNorm. For example, the DrugBank drug *cerivastatin*, which was withdrawn from the U.S. market in 2001, is not present in RxNorm.

Comparing interactions across sources {#Sec15}
-------------------------------------

Having normalized all drugs to RxNorm ingredients, we compared the lists from DDIs of NDF-RT and DrugBank in order to determine the similarities and unique features of each source. In addition, we compared each with the reference set of DDIs to determine how many of the reference DDIs each source covered.

Characterizing differences {#Sec16}
--------------------------

To determine if there was a pattern of missing interactions, we abstracted the pairs of interacting drugs into pairs of classes from the Anatomical Therapeutic Chemical (ATC) Classification System. We mapped the drugs directly to their 4^th^-level ATC classes using the RxNorm API and identified those pairs of ATC classes for which the proportion of DDIs in common between NDF-RT and DrugBank was low, using the Jaccard score. Of note, some NDF-RT and DrugBank drugs are not represented in ATC. So only DDIs for which both the object and precipitant drugs are present in ATC were analysed.

Comparing coverage of interactions from actual prescription data {#Sec17}
----------------------------------------------------------------

In order to assess the difference between NDF-RT and DrugBank interactions based on usage data, we collected drug pairs generated from actual patient prescription lists. Checking medication lists for drug-drug interactions represents a practical use case and is offered as a service by many medication list applications. It also provides some frequency of usage of the co-prescribed drug pairs extracted from the lists. The data was acquired from Symphony Health Solutions (<http://symphonyhealth.com/>). It included one year of prescription data from the Washington, D.C. area from July 1, 2011 through June 30, 2012. Prescription information included prescriber, de-identified patient information, and specific medication information, including the drug name and strength. The drug information was mapped into RxNorm clinical drugs. From a list of co-prescribed drugs, we generated all possible pairwise combinations within the drug list (ignoring the order of drugs in the pair, since the distinction between object and precipitant drugs is not required for DDI testing). We ignored topical drugs from the prescription list, because DDI information represented at the ingredient level in NDF-RT and DrugBank usually refers to interactions between systemic drugs. For example, we generated the following pair of RxNorm clinical drug, *24 HR Diltiazem Hydrocloride 360 MG Extended Release Oral Capsule (830795)* and *Lovastatin 20 MG Oral Tablet (197904)*. We mapped the clinical drugs to their active moieties in RxNorm (*Diltiazem Hydrocloride* and *Lovastatin*, respectively) and further normalized those to RxNorm ingredients, as we did with all object and precipitant drugs from the various DDI lists. Here, we normalized *Diltiazem Hydrocloride* to *Diltiazem*. In summary, we transformed the pair of RxNorm clinical drugs extracted from the prescription list into the pair of RxNorm ingredients (*Diltiazem*, *Lovastatin*) for comparison to the DDIs in NDF-RT and DrugBank.

Publicly available vs. commercial DDI sources {#Sec18}
---------------------------------------------

To compare the coverage of the reference set by DrugBank and NDF-RT to that of a commercial source, we investigated the Multum drug knowledge base through the interaction checker of the website Drugs.com (<http://www.drugs.com/>), against which we tested the 360 DDIs of the reference set.

Results {#Sec19}
=======

DDI information in NDF-RT, DrugBank and the reference set {#Sec20}
---------------------------------------------------------

Table [1](#Tab1){ref-type="table"} summarizes the number of DDIs in the three data sets.Table 1**DDI counts in the three datasetsData setDDI counts*Total from source**Mapped to RxNorm**Normalized***Reference360360360DrugBank121281176211552NDF-RT1083194529392

### Reference set {#Sec21}

The reference set contained 360 DDIs; all the drugs mapped to RxNorm and were all classified as ingredients. The 360 DDIs covered 86 RxNorm ingredients.

### DrugBank {#Sec22}

DrugBank contained 12,128 DDIs defined in the XML file. DDIs involving drugs with no mapping to RxNorm were discarded (418 DDIs involving 46 drugs). Analysis of all the discarded DDIs revealed several reasons why the DDIs were eliminated. Some DDIs involved drugs that were either withdrawn from public use (e.g., *cerivastatin, ephedra, heptabarbital*) or not approved by the U.S. Food and Drug Adminstration (e.g., *cinolazepam, carbetocin*) and those drugs could not be mapped to RxNorm. Additionally, 518 DrugBank DDIs were eliminated through the ingredient normalization process. For example, the DDIs containing *zuclopenthixol, zuclopenthixol acetate* and *zuclopenthixol deconoate* were normalized to produce a single set with *zuclopenthixol* as the ingredient, eliminating the redundant pairs containing the salt forms. The resulting 11,552 normalized DDIs covered 1153 RxNorm ingredients.

### NDF-RT {#Sec23}

NDF-RT contained 10,831 DDIs extracted from the data set. DDIs involving drugs with no mapping to RxNorm were discarded (1379 DDIs involving 38 drugs). Analysis of all the discarded DDIs revealed that some DDIs were associated with drugs which referenced obsolete RxNorm concepts, many of these vaccine drugs that were recently removed from RxNorm. Additionally, 60 NDF-RT DDIs were eliminated through the ingredient normalization process. The resulting 9,392 normalized DDIs covered 1079 RxNorm ingredients.

In the remainder of this paper, DDIs refer to pairs of object and precipitant drugs normalized to RxNorm ingredients. However, even after normalization to RxNorm ingredients, the coverage of drugs is not expected to be the same in NDF-RT and DrugBank. For example, vaccines and other biologicals are present in NDF-RT, but out of scope for DrugBank. When analysing DDIs across the two sources, breakdown by pharmacological classes will reflect such differences in drug coverage.

Comparing interactions across sources {#Sec24}
-------------------------------------

The matching DDIs between the three data sets are shown in Table [2](#Tab2){ref-type="table"}.Table 2**Matching DDIs across data setsData setNumber of matching DDIs*Reference**DrugBank**NDF-RT***Reference360211207DrugBank211115522801NDF-RT20728019392

### Overlap between DrugBank and NDF-RT {#Sec25}

Overall, the 2801 DDIs common to NDF-RT and DrugBank represent a 30% coverage rate of NDF-RT by DrugBank and a 24% coverage rate of DrugBank by NDF-RT. Example of common DDIs include *diltiazem/lovastatin* and *itraconazole/sirolimus*. Examples of DDIs in DrugBank only include *acebutolol/Insulin Lispro* and *metronidazole/terfenadine*. Examples of DDIs in NDF-RT only include *amiodarone/sotalol* and *meperidine/linezolid*.

When we only consider DDIs from the reference set, the overlap between DrugBank and NDF-RT is significantly higher. The 146 reference DDIs common to NDF-RT and DrugBank represent 71% of the 207 reference DDIs covered by NDF-RT and 69% of the 211 reference DDIs covered by DrugBank.

### Reference set coverage {#Sec26}

Table [3](#Tab3){ref-type="table"} shows the breakdown by the reference set groups of the DDI mapping for DrugBank and NDF-RT. DrugBank contained 211 DDIs (59%) from the reference set, compared with 207 DDIs (58%) for NDF-RT. There were 146 DDIs (42%) from the reference set which were both in the NDF-RT and DrugBank, including *diltiazem/lovastatin, simvastatin/amiodarone* and *atazanavir/omeprazole*. Conversely, there were 88 DDIs (24%) in the reference set which were not contained in either DrugBank or NDF-RT, including *irinotecan/indinavir, ketoconazole/ergonovine* and *milnacipran/selegiline*. DrugBank contained 65 DDIs from the reference set which were not in NDF-RT (e.g., *lovastatin/tipranavir* and *ramelteon/fluvoxamine*), and NDF-RT contained 61 DDIs from the reference set not in DrugBank (e.g., *fluoxetine/procarbazine* and *simvastatin/saquinavir*). DrugBank had coverage in all groups from the reference set, though only 100% coverage in two groups. NDF-RT had coverage of all but one group (\#22 *ramelteon-fluvoxamine*). The coverage of the reference set of drug-drug interactions by DrugBank and NDF-RT is available as Additional file [1](#MOESM1){ref-type="media"}.Table 3**Reference Set DDI and coverage in NDF-RT and DrugBankGrp \#DDI group description\# pairsNDF-RT matchesDrugBank matchesObject membersPrecipitant members**3Amphetamine and derivatives -- MAO inhibitors6030 (50%)30 (50%)Dexmethylphenidate, Dextroamphetamine, Methylphenidate, Lisdexamfetamine, Phendimetrazine, Pseudoephedrine, Amphetamine, Benzphetamine, Diethylproprion, Phentermine, Atomoxetine, MethamphetamineTranylcypromine, Phenelzine, Isocarboxazid, Procarbazine, Selegiline4Atazanavir -- Proton pump inhibitors (PPIs)55 (100%)5 (100%)AtazanavirOmeprazole, Lansoprazole, Pantoprazole, Rabeprazole, Esmoprazole8Fluoxetine - MOA inhibitors5539 (71%)43 (78%)Fluoxetine, Paroxetine, Citalopram, Escitalopram, Sertraline, Fluvoxamine, Duloxetine, Nefazodone, Desvenlafaxine, Milnacipran, VenlafaxineTranylcypromine, Phenelzine, Isocarboxazid, Procarbazine, Selegiline11Irinotecan -- Ketoconazole231 (4%)5 (22%)IrinotecanRitonavir, Nelfinavir, Atazanavir, Indinavir, Saquinavir, Amprenavir, Darunavir, Lopinavir, Tipranavir, Fosamprenavir, Clarithromycin, Erythromycin, Telithromycin, Amiodarone, Verapamil, Diltiazem, Ketoconazole, Itraconazole, Fluconazole, Voriconazole, Nefazodone, Aprepitant, Cimetidine16Narcotic analgesics -- MAO inhibitors3025 (83%)15 (50%)Meperidine, Methadone, Tapentadol, Fentanyl, Tramadol, DextromethorphanTranylcypromine, Phenelzine, Isocarboxazid, Procarbazine, Selegiline22Ramelteon-fluvoxamine40 (0%)2 (50%)RamelteonFluvoxamine, Amiodarone, Ticlopidine, Ciprofloxacin23Rifampin -- ritonavir6041 (68%)25 (42%)Bosentan, Rifapentine, Carbamazepine, Rifabutin, Rifampin, St. John's wortRitonavir, Nelfinavir, Atazanavir, Indinavir, Saquinavir, Amprenavir, Darunavir, Lopinavir, Tipranavir, Fosamprenavir25HMG Co-A reductase inhibitors -- protease inhibitors4038 (95%)33 (83%)Simvastatin, LovastatinRitonavir, Nelfinavir, Atazanavir, Indinavir, Saquinavir, Amprenavir, Darunavir, Lopinavir, Tipranavir, Clarithromycin, Erythromycin, Telithromycin, Amiodarone, Verapamil, Diltiazem, Tranylcypromine, Phenelzine, Isocarboxazid, Procarbazine, Selegiline27Telithromycin -- ergot alkaloids and derivatives6013 (22%)35 (58%)Ritonavir, Nelfinavir, Atazanavir, Indinavir, Saquinavir, Amprenavir, Darunavir, Lopinavir, Tipranavir, Clarithromycin, Erythromycin, Telithromycin, Ketoconazole, Itraconazole, VoriconazoleErgotamine, Methylergonovine, Dihydroergotamine, Ergonovine28Tizanidine -- ciprofloxacin76 (86%)6 (86%)TizanidineFluvoxamine, Amiodarone, Ticlopidine, Ciprofloxacin, Mexiletine, Propafenone, Zileuton30Tranylcypromine -- procarbazine11 (100%)1 (100%)TranylcypromineProcarbazine31Triptans -- MAO inhibitors158 (53%)11 (73%)Sumatriptan, Zolmitriptan, RizatriptanTranylcypromine, Phenelzine, Isocarboxazid, Moclobamide, Methylene blue**TOTAL207 (58%)211 (59%)**

Of the 86 drugs contained in the reference set, one (*moclobamide*, involved in 3 DDIs) did not exist in NDF-RT and two (*dexmethylphenidate*, involved in 5 DDIs and *methylene blue*, involved in 3 DDIs) did not exist in DrugBank. In addition, there were four other drugs *(bosentan, lopinavir, methadone* and *zileuton*) which were present in DrugBank, but not involved in any of the DDIs from the reference set. DrugBank did have DDIs for these four drugs outside of the reference set.

### Pharmacologic classes {#Sec27}

We mapped the DDIs from NDF-RT and DrugBank to 4^th^-level ATC drug classes to see if there were distinctive class differences between the two sets of DDIs. A small proportion of the drugs, such as *avanafil*, *lopinavir* and *zileuton* (14% in NDF-RT, 11% in DrugBank) were not represented in ATC and the corresponding DDIs were excluded from the analysis.

Table [4](#Tab4){ref-type="table"} shows the top frequency count of ATC classes for NDF-RT and DrugBank DDIs, along with the number of drugs from each class represented in the source of DDI. For example, NDF-RT has 401 DDIs involving 11 drugs for the *fluoroquinolone* class, while DrugBank has 335 interactions involving 14 drugs for this class. *Fluoroquinolones* is in the top-10 classes for the frequency of DDIs in NDF-RT, but not in DrugBank. Two classes, *protein kinase inhibitors* and *selective immunosuppressants*, seem underrepresented in DrugBank, while three classes, *anticholinesterases*, *hydantoin derivatives* and *non-selective monoamine reuptake inhibitors*, seem underrepresented in NDF-RT.Table 4**Top ATC Class Counts in NDF-RT and DrugBankNDF-RTDrugBankATC Class Name*DDI**drugs**DDI**drugs***Antibiotics**709**22**744**25Anticholinesterases254**411**7Fluoroquinolones**401**1133514Hydantoin derivatives2664**406**4Macrolides**443**6**502**9Non-selective monoamine reuptake inhibitors**311**10**736**10Other antidepressants**309**11**457**10Protease inhibitors**1130**11**751**11Protein kinase inhibitors**759**19**389**20Selective immunosuppressants**307**1315014Selective serotonin reuptake inhibitors**341**63806Triazole derivatives**427**4**502**4Vitamin K antagonists2923**427**3\*Boldface values indicate the ten top categories in each source.

Table [5](#Tab5){ref-type="table"} shows the ATC pairs containing the most DDIs from NDF-RT and DrugBank, but a low proportion of shared DDIs between the two sources (evidenced by the low Jaccard scores). The *Adrenergic and dopaminergic agents - Non-selective monoamine reuptake inhibitors* class pair, for example, has no NDF-RT DDIs and 78 DrugBank DDIs. Examples in this class pair include *dopamine/amitriptyline* and *norepinephrine/amoxipine*. Conversely, in the *Protease inhibitors - Protein kinase inhibitors* class pair, many more specific DDIs exist in NDF-RT, such as *boceprevir/bosutinib*, than in DrugBank. Finally, the *Beta blocking agents, non-selective - Sulfonamides, urea derivatives* class pair illustrates a situation where both sources share a majority of DDIs, but also have a significant number of specific DDIs. For example, *sotalol/glyburide* is common, but *sotalol/tolazamide* is specific to NDF-RT and *sotalol/gliclazide* is specific to DrugBank.Table 5**Top ATC Class PairsATC Class Pairtotal DDIsNDF-RT onlyDrugBank onlybothJaccard**Protease inhibitors - Protein kinase inhibitors1037210210.20Benzodiazepine derivatives - Protease inhibitors88552280.32Adrenergic and dopaminergic agents - Non-selective monoamine reuptake inhibitors7807800Antibiotics - Other quaternary ammonium compounds71358280.39Penicillins with extended spectrum - Tetracyclines6305850.08Protein kinase inhibitors - Triazole derivatives603210180.30Beta blocking agents, non-selective - Sulfonamides, urea derivatives561014320.57Macrolides - Protein kinase inhibitors562714150.27

### Comparing coverage of interactions from actual prescription data {#Sec28}

From the prescription dataset, almost 35.9 million pairs were extracted, representing 816,258 unique pairs of RxNorm clinical drugs. Restricted to systemic drugs, the dataset included 35.5 million pairs of clinical drugs (808,285 unique). Each clinical drug maps to at least one ingredient, and multi-ingredient drugs map to several ingredients, resulting in multiple ingredient pairs for a given pair of clinical drugs. For example, starting from the pair of clinical drugs *Primidone 250 MG Oral Tablet* and *Carbidopa 25 MG/Levodopa 100 MG Oral Tablet*, we generate the following two pairs of ingredients, (*Primidone*, *Carbidopa*) and (*Primidone*, *Levodopa*), because the clinical drug *Carbidopa 25 MG/Levodopa 100 MG Oral Tablet* maps to two ingredients, *Carbidopa* and *Levodopa*. After mapping of the clinical drugs to ingredients, there were 45.2 million pairs of co-prescribed drugs, ranging in frequency between 1 and 158,515 (median = 18). Of these, 808,285 pairs matched with NDF-RT DDIs (2153 unique), while 1,170,693 pairs matched with DrugBank DDIs (2823 unique). There were 382,833 pairs that matched with both NDF-RT and DrugBank (988 unique). There were 26,672 matches with the reference set (88 unique). Figure [1](#Fig1){ref-type="fig"} shows the frequency of the DDIs (in parenthesis) with the unique number of DDIs found in the prescription dataset.Figure 1Frequency of DDIs in prescription pairs.

Examples of frequently co-prescribed drugs identified as a DDI by both NDF-RT and DrugBank include *Diltiazem*/*Lovastatin* and *Clarithromycin*/*Simvastatin*. Co-prescribed drugs identified as a DDI by NDF-RT, but not DrugBank include *Amlodipine*/*Simvastatin* and *Diltiazem*/*Oxycodone*. Conversely, DDIs identified by DrugBank, but not NDF-RT, include *Amoxicillin*/*Ethinyl Estradiol* and *Hydrochlorothiazide*/*Insulin Lispro*. Table [6](#Tab6){ref-type="table"} shows the most frequently co-prescribed pairs recognized as a DDI by one or both of the sources.Table 6**Top DDI Pairs found in Prescription DataDrug pairFreqDrugBank DDINDF-RT DDI**Amlodipine-Simvastatin76980XGlipizide-Metoprolol15469XXatorvastatin-Fenofibrate14897XXFenofibrate-Simvastatin14060XXAlbuterol-Metoprolol14030XInsulin Glargine-Metoprolol13492XMetoprolol-Sertraline13311Xatorvastatin-Diltiazem13218XXDigoxin-Furosemide12958XLisinopril-Triamterene12399XXcyclobenzaprine-Tramadol12211XDiltiazem-Metoprolol11738XOxycodone-Sertraline11710XDiltiazem-Simvastatin11484XXAlprazolam-Fluoxetine11194XSertraline-Trazodone11066XAmoxicillin-Ethinyl Estradiol11044XLisinopril-Spironolactone10248XXClonazepam-Fluoxetine10149XFenofibrate-rosuvastatin9954XXClonidine-Metoprolol9827XXHydralazine-Metoprolol9556Xduloxetine-Trazodone9236Xclopidogrel-pantoprazole9177XEscitalopram-Metoprolol8396XLevofloxacin-Prednisone8335XEscitalopram-Oxycodone8228XCitalopram-Trazodone8081XCitalopram-Oxycodone8016XMetoprolol-salmeterol7703XAtenolol-Glipizide7597XXFluoxetine-Trazodone7579Xcarvedilol-Digoxin7517XFluoxetine-Oxycodone7508Xcarvedilol-Insulin Glargine7496XCitalopram-Metoprolol7474XEscitalopram-Trazodone7421XAlprazolam-Omeprazole7339X

### Publicly available vs. commercial DDI sources {#Sec29}

We were able to match 328 (91%) DDIs of the reference set to Multum DDIs, a significantly higher proportion compared to NDF-RT and DrugBank. The level of severity provided by Multum for these DDIs (on a 3-level scale *mild, moderate* or *major*) was consistent with the notion of "high-severity" claimed by the authors of the reference set, since 302 DDIs were rated major and the other 26 were rated moderate.

Discussion {#Sec30}
==========

In this section, we discuss some of the reasons for the limited overlap of DDI information among sources. We also analyse its implications for our interactions API.

Limited overlap between NDF-RT and DrugBank DDIs {#Sec31}
------------------------------------------------

### Findings {#Sec32}

One important finding of this investigation is the limited overlap (\<30%) between DrugBank and NDF-RT DDIs, although both sources cover roughly the same number of DDIs. We found 6591 DDIs specific to NDF-RT and 8751 specific to DrugBank, while only 2801 DDIs were common to both sources. Differences in scope (e.g., the lack of vaccine DDIs in DrugBank) account for only a small portion of the differences. Differential coverage persisted after abstracting DDIs to the corresponding drug classes from ATC. For example, NDF-RT had many more DDIs than DrugBank for *Protease inhibitor* drugs, and had much fewer DDIs for *Anticholinesterases* drugs. Likewise, many ATC class pairs had a small proportion of DDIs in common, evidenced by low Jaccard scores. For example, among the 56 DDIs for the *Macrolides - Protein kinase inhibitors* class pair, only 15 are common to NDF-RT and DrugBank (Jaccard = 0.27). The low overlap rate also applied to our more frequently prescribed drug pairs in the prescription data set. When we examined the co-prescribed drugs from the prescription data set, we found significant differences in identification of DDIs between the two sources for the most frequently co-prescribed pairs. Of the top 10 most frequently prescribed drug pairs identified as potential DDIs by DrugBank, only 5 were recognized as NDF-RT DDIs (see Table [6](#Tab6){ref-type="table"} for details).

### DDI severity {#Sec33}

We wondered if the differential coverage observed between NDF-RT and DrugBank could be due in part to different editorial guidelines for curating less severe DDIs, assuming the most severe DDIs would be covered more consistently by both sources. This did not seem to be the case since the 360 high-severity DDIs from the reference set are only partially covered by NDF-RT and DrugBank. While NDF-RT provides an indication of severity (*critical* or *significant*) for its DDIs, DrugBank does not. Nevertheless, we tested if DrugBank would cover a larger proportion of NDF-RT *critical* DDIs. In fact, DrugBank DDIs accounted for 31% of the total *critical* DDIs in NDF-RT and for 29% of the total *significant* DDIs. In other words, there is no difference in the coverage of NDF-RT by DrugBank in terms of DDI severity. (Even when correcting for the absence of vaccine DDIs in DrugBank, the overlapping DDIs accounted for just 40% of the *critical* DDIs in NDF-RT.)

### Differential coverage of DDIs among sources {#Sec34}

Our finding of limited overlap between NDF-RT and DrugBank should not be surprising given similar findings in studies comparing drug knowledge bases for critically important DDIs \[[@CR21]\]. There is no standardization for what constitutes a drug-drug interaction, and DDI curators have to consider a variety of drug and patient specific factors in their decision to include pairs of drugs in their DDI lists. These factors include the severity of the interaction, the probability of the interaction, patient characteristics (e.g., specific patient groups, such as elderly patients), and evidence supporting the interaction (quantity of evidence, as well as the quality of the evidence). The number of factors to consider makes this a complex and daunting task. Moreover, the maintenance of DDI lists itself is an issue, as additional evidence for or against DDIs becomes available over time. Finally, different strategies for prioritizing the various factors also accounts for some of the differences observed across sources.

Implications for the interactions API {#Sec35}
-------------------------------------

Overall, our investigation of NDF-RT and DrugBank as sources of DDIs for our API provides a mixed picture. Not only do they both provide incomplete coverage of the reference set (about 60% each), but their overlap is also limited (42%). A similar difference could be observed when we simulated interaction detection based on actual usage data. We also confirmed that a commercial source, Multum, provided a more systematic coverage of the reference set.

With the provision of DDI information being discontinued in NDF-RT in November 2014, we decided to use DrugBank as a replacement for our interactions API. This solution is far from ideal, because this investigation did not establish the accuracy of either source, but simply assessed differences among them. On the other hand, DDI determination is not an exact science and both NDF-RT and DrugBank provide valuable information to support medical decision. NDF-RT DDIs are associated with levels of severity, while DrugBank's come with a short description. In practice, the absence of severity levels in DrugBank is a disadvantage, as severity is an important consideration for determining when to alert physicians to a potential DDI. Severity is also a requirement when checking DDI in the context of the *Meaningful Use* incentive program.

It is important to keep in mind that no clinical decision system, as good as it is, can be a substitute for medical advice. Our API not only clearly discloses the origin of the information it provides, but also reminds our users to seek advice from health professionals before making decisions about their medications. In the future, we plan to perform a more systematic and qualitative investigation of publicly available and commercial DDI sources in order to better assess the differences among these sources.

Conclusions {#Sec36}
===========

This study is the first comparative investigation of DDI information in two publicly available sources, NDF-RT and DrugBank. We compared the two sources not only to themselves, but also to a reference set of DDIs, and assessed their ability to identify DDIs in a large prescription dataset. We also contrasted NDF-RT and DrugBank against the commercial source Multum.

This investigation confirms the limited overlap between DDI information between NDF-RT and DrugBank, even for the reference dataset. Additional research is required to determine which source is better, if any. Usage of any of these sources in clinical decision systems should clearly disclose these limitations.

Additional file {#Sec37}
===============

Additional file 1:**Coverage of the reference set of drug-drug interactions by DrugBank and NDF-RT.**

**Competing interests**

The authors declare that they have no competing interests.

**Authors' contributions**

LP downloaded, normalized and compared the main datasets. OB created and normalized the reference dataset. NB characterized the differences against ATC drug classes and the commercial source of DDI information. All the authors contributed to writing the manuscript. All the authors have read and approved the final manuscript.

This work was supported by the Intramural Research Program of the NIH, National Library of Medicine.
