more than 480,000 deaths per year are contributed to smoking , costing nearly $ 100 billion in direct medical care.1 only 30% of smokers have received evidence - based tobacco dependence treatment during a health care visit.2 to motivate clinicians and health care systems to identify patients who use tobacco and to provide them with evidence - based treatment , recording smoking status for people aged 13 years or older is one of the core criteria for meaningful use of electronic medical records ( emrs).2 presently , smoking status is generally recorded as a structured data field with the following values : current every day smoker ; current some day smoker ; former smoker ; never smoker ; smoker , current status unknown ; and unknown if ever smoked . with the increasing use of longitudinal emrs for clinical and translational research , extracting smoking information from emrs is crucial since smoking is a risk factor for many conditions . prior to meaningful use , smoking information available in emrs is frequently recorded in narrative text . natural language processing ( nlp ) can be used to extract smoking information,35 especially after the organization of the 2006 i2b2 nlp shared task.6 this task aims to evaluate the following two challenging questions : 1 )  what is the state - of - the - art in automatic de - identification of clinical data ?  and 2 )  how accurately can automatic methods evaluate the smoking status of patients based on their medical records ?  . both statistical- and pattern - based nlp techniques for example , the smoking module of ctakes used smoking - related keywords to extract sentences and then used support vector machines to classify sentences as current smoker , former smoker , smoker , or never smoker . the patient s current smoking information is then summarized from all existing clinical documents using heuristics . the portability of the smoking module in ctakes was examined on the vanderbilt university hospital s emr data where a different summarization rule yields better performance.7 structurally , smoking information can be recorded as the international classification of diseases , ninth revision ( icd-9 ) , tobacco use codes or patient - provided information ( ppi ) . the evaluation of icd-9 tobacco use codes indicates high precision but a low patient coverage rate in identifying ever smoker.8 some studies show that self - reported smoking status can be inconsistent.9,10 overall , the availability of structured smoking status is limited and the extraction of smoking information using nlp generally showed overall performance around 90% but is less accessible due to technical complexity . to date , there is little investigation on the comparison of smoking status information from three distinct sources : nlp , ppi , and icd-9 . additionally , smoking amount or strength information is also crucial , especially for diseases that are directly caused by smoking such as lung diseases . , we evaluated the performance of three sources of information ( nlp , ppi , and icd-9 ) , independently and jointly , in identifying smoking status and two sources of information ( nlp and ppi ) in identifying smoking strength . our study leveraged an existing lung cancer cohort for smoking status , amount , and strength information , which was manually chart - reviewed as a background comparison . on the nlp side , a pattern - based smoking information extraction module was implemented to extract smoking - related information . we then utilized heuristic rules to obtain smoking status - related information . on the other hand , icd-9- and ppi - based smoking information we describe the data set used , the algorithms developed , and performance assessment . in total , 561 patients aged between 15 and 45 years who were diagnosed with one of the 21 categories of lung cancer subtypes from 1997 to 2011 were manually reviewed to obtain smoking status and strength information.11 mayo clinic reached meaningful use stage 1 in 2011 . before stage 1 information including smoker age , year start , year stop , pack per day , and pack - years was manually recorded . date of diagnosis was incorporated to classify each patient in one of the five smoking status categories , including current ( every / some ) day smoker , former smoker , never smoker , smoker current status unknown , and unknown if ever smoked . former smoker refers to an individual who has smoked at least 100 cigarettes during his / her lifetime but does not currently smoke . never smoker is defined as an individual who has not smoked 100 or more cigarettes during his / her lifetime . smoking strength contains two categories , heavy smoker and light smoker , where heavy smoker smokes 10 cigarettes or more per day.12 clinical notes in text format for 561 patients were obtained from the mayo clinic emr . we extracted smoking - related information using medtagger13 , integrated with heuristic rules and medtime.14 medtime can parse temporal expression in text to the output of entity types such as  date  ,  time  , and information relevant to smoking status ( eg , current smoker , former smoker , never smoker)amount information  the amount of smoking ( eg , number of cigarettes per day)quit tense  the keyword quit and the corresponding tense ( eg , will quit , has quit , should quit)temporal expression or age  temporal information relevant to date , frequency ( eg , repeat two times a day ) , or duration ( eg , smoked for 10 years ) smoking status information relevant to smoking status ( eg , current smoker , former smoker , never smoker ) amount information  the amount of smoking ( eg , number of cigarettes per day ) quit tense  the keyword quit and the corresponding tense ( eg , will quit , has quit , should quit ) temporal expression or age  temporal information relevant to date , frequency ( eg , repeat two times a day ) , or duration ( eg , smoked for 10 years ) we then used the above smoking - related information to detect smoking status within 30 days after cancer diagnosis . the parsed information was then standardized by formatting date and smoking status into consistent formats and converting frequency , duration , and smoking amount to consistent units of measure . for example , for cases where date contains only year and month , the 15th day of the month was assumed . for smoking status , the co - occurrence of quit with current or past tense indication words  have  ,  had  ,  has  , and  is  was considered quitting smoking ( ie , former smoker ) . frequency and duration all information available for each patient was processed according to the order of clinical note record date . we only considered the paragraphs with section codes that belonged to  social and behavior history category or those paragraphs having the presence of pack per day , pack - years , or smoking status ( eg , former smoker or current smoker ) . information including pack per day , pack - years , and smoking duration was utilized to deduce the corresponding missing values not mentioned / captured from the original clinical notes . for example , given a patient smoked 1.5 pack per day for 6 years , it can be calculated that the patient smoked 9 pack - years . we traced each patient through the record date timeline to assign a smoking status for that record date . the general rule was that every patient starts with unknown smoking status on the first record date and switches status to nonsmoker , former smoker , or current smoker , based on the smoking - related information throughout the timeline . smoking status at the time of diagnosis was determined by using the smoking status that was recorded no later than 30 days after the diagnosis date . for each patient , smoking amount - related information was used to determine the smoking strength ( ie , heavy smoker [ 10 cigarettes per day ] ; light smoker [ < 10 cigarettes per day ] ) for each record date . the strength results from all record dates were summarized to determine the final smoking strength of a patient ( the smoking strength with the highest occurrence was used , or unknown if the occurrences are equal ) . patient - provided smoking information was obtained from structured ppi in emrs , where 49% of patients complete the structured data . to find smoking - related information , we limited the search criteria to retrieve only smoking - related ppi entries . the results were manually reviewed to keep entries that can be reasonably used to deduce smoking status or strength and remove irrelevant records such as  are you interested in more information about safety ( seat belts , smoke detectors , firearms ) ?  . to deduce smoking status , the records were filtered to keep those recorded no later than 30 days after the recorded diagnosis date . the diagnosis code extracted from hospital billing information was used to group patients into two categories of ever smoker or never smoker . specifically , patients with one or more occurrences of the icd-9 codes , tobacco use disorder ( 305.1 ) and history of tobacco use ( v15.82 ) , were considered ever smoker . , we evaluated the performance of automatically generated results using nlp , icd , ppi alone , and combinations including nlp + icd , nlp | icd , nlp + ppi , nlp | ppi , nlp + ppi + icd , and nlp | ppi | icd . for combinations , the  + rule considers a patient as ever smoker when any single strategy identifies the patient as ever smoker . nlp as the primary source of information and other source(s ) as a supplement when nlp failed to identify the status . at the second level , the nlp , ppi , and nlp | ppi - based results and manually reviewed smoking status were compared by converting both parts to standard smoking status type , which assigns each patient to one of the five smoking status categories and one of the two smoking strength categories . comparison results were presented as sensitivity , specificity , and accuracy of measurements . for nlp and ppi , patient coverage is defined as the proportion of patients whose smoking status / strength can be effectively determined . for diagnosis codes , patients with smoking - related icd-9 codes are categorized as ever smoker , otherwise never smoker . for each method , sensitivity , specificity , and accuracy are measured using patients with coverage ( ie , excluding patients without original documentation or whose smoking status and/or strength can not be effectively determined from existing documentation ) . the sensitivity is defined as 
 |true  positives||true  positives|+|false  negatives| , specificity is defined as 
 |true  negatives||true  negatives|+|false  positives| , and accuracy is defined as 
 |true  positives+true  negatives||true  positives|+|false  positives|+|true  negatives|+|false  negatives| . in total , 561 patients aged between 15 and 45 years who were diagnosed with one of the 21 categories of lung cancer subtypes from 1997 to 2011 were manually reviewed to obtain smoking status and strength information.11 mayo clinic reached meaningful use stage 1 in 2011 . before stage 1 , documentations of smoking status were meaningful in some cases . information including smoker age , year start , year stop , pack per day , and pack - years was manually recorded . date of diagnosis was incorporated to classify each patient in one of the five smoking status categories , including current ( every / some ) day smoker , former smoker , never smoker , smoker current status unknown , and unknown if ever smoked . former smoker refers to an individual who has smoked at least 100 cigarettes during his / her lifetime but does not currently smoke . never smoker is defined as an individual who has not smoked 100 or more cigarettes during his / her lifetime . smoking strength contains two categories , heavy smoker and light smoker , where heavy smoker smokes 10 cigarettes or more per day.12 clinical notes in text format for 561 patients were obtained from the mayo clinic emr . we extracted smoking - related information using medtagger13 , integrated with heuristic rules and medtime.14 medtime can parse temporal expression in text to the output of entity types such as  date  ,  time  , and information relevant to smoking status ( eg , current smoker , former smoker , never smoker)amount information  the amount of smoking ( eg , number of cigarettes per day)quit tense  the keyword quit and the corresponding tense ( eg , will quit , has quit , should quit)temporal expression or age  temporal information relevant to date , frequency ( eg , repeat two times a day ) , or duration ( eg , smoked for 10 years ) smoking status information relevant to smoking status ( eg , current smoker , former smoker , never smoker ) amount information  the amount of smoking ( eg , number of cigarettes per day ) quit tense  the keyword quit and the corresponding tense ( eg , will quit , has quit , should quit ) temporal expression or age  temporal information relevant to date , frequency ( eg , repeat two times a day ) , or duration ( eg , smoked for 10 years ) we then used the above smoking - related information to detect smoking status within 30 days after cancer diagnosis . the parsed information was then standardized by formatting date and smoking status into consistent formats and converting frequency , duration , and smoking amount to consistent units of measure . for example , for cases where date contains only year and month , the 15th day of the month was assumed . for smoking status , the co - occurrence of quit with current or past tense indication words  have  ,  had  ,  has  , and  is  was considered quitting smoking ( ie , former smoker ) . all information available for each patient was processed according to the order of clinical note record date . we only considered the paragraphs with section codes that belonged to  social and behavior history category or those paragraphs having the presence of pack per day , pack - years , or smoking status ( eg , former smoker or current smoker ) . information including pack per day , pack - years , and smoking duration was utilized to deduce the corresponding missing values not mentioned / captured from the original clinical notes . for example , given a patient smoked 1.5 pack per day for 6 years , it can be calculated that the patient smoked 9 pack - years . we traced each patient through the record date timeline to assign a smoking status for that record date . the general rule was that every patient starts with unknown smoking status on the first record date and switches status to nonsmoker , former smoker , or current smoker , based on the smoking - related information throughout the timeline . smoking status at the time of diagnosis was determined by using the smoking status that was recorded no later than 30 days after the diagnosis date . for each patient , smoking amount - related information was used to determine the smoking strength ( ie , heavy smoker [ 10 cigarettes per day ] ; light smoker [ < 10 cigarettes per day ] ) for each record date . the strength results from all record dates were summarized to determine the final smoking strength of a patient ( the smoking strength with the highest occurrence was used , or unknown if the occurrences are equal ) . patient - provided smoking information was obtained from structured ppi in emrs , where 49% of patients complete the structured data . to find smoking - related information , we limited the search criteria to retrieve only smoking - related ppi entries . the results were manually reviewed to keep entries that can be reasonably used to deduce smoking status or strength and remove irrelevant records such as  are you interested in more information about safety ( seat belts , smoke detectors , firearms ) ?  . to deduce smoking status , the records were filtered to keep those recorded no later than 30 days after the recorded diagnosis date . the diagnosis code extracted from hospital billing information was used to group patients into two categories of ever smoker or never smoker . specifically , patients with one or more occurrences of the icd-9 codes , tobacco use disorder ( 305.1 ) and history of tobacco use ( v15.82 ) , were considered ever smoker . , we evaluated the performance of automatically generated results using nlp , icd , ppi alone , and combinations including nlp + icd , nlp | icd , nlp + ppi , nlp | ppi , nlp + ppi + icd , and nlp | ppi | icd . for combinations , rule considers a patient as ever smoker when any single strategy identifies the patient as ever smoker . the  |  rule uses nlp as the primary source of information and other source(s ) as a supplement when nlp failed to identify the status . at the second level , the nlp , ppi , and nlp | ppi - based results and manually reviewed smoking status were compared by converting both parts to standard smoking status type , which assigns each patient to one of the five smoking status categories and one of the two smoking strength categories . comparison results were presented as sensitivity , specificity , and accuracy of measurements . for nlp and ppi , patient coverage is defined as the proportion of patients whose smoking status / strength can be effectively determined . for diagnosis codes , patients with smoking - related icd-9 codes are categorized as ever smoker , otherwise never smoker . for each method , sensitivity , specificity , and accuracy are measured using patients with coverage ( ie , excluding patients without original documentation or whose smoking status and/or strength can not be effectively determined from existing documentation ) . the sensitivity is defined as 
 |true  positives||true  positives|+|false  negatives| , specificity is defined as 
 |true  negatives||true  negatives|+|false  positives| , and accuracy is defined as 
 |true  positives+true  negatives||true  positives|+|false  positives|+|true  negatives|+|false  negatives| . the number of patients grouped by cancer subtypes and manually reviewed smoking status is shown in table 1 . we manually identified 206 ( 37% ) as never smokers and 355 ( 63% ) as ever smokers , of which 26% were current every / some day smokers , 15% were former smokers , and 22% were smokers with current status unknown . for smoking strength , we manually identified 52% as heavy smokers and 6% as light smokers , which accounts for 91% of patients with smoking history ( table 1 ) . at the first level patient coverage rates for nlp , icd-9 , and ppi were 88% , 100% , and 49% , respectively . nlp alone showed the best sensitivity ( 0.97 ) and accuracy ( 0.88 ) compared to icd-9 and ppi . icd-9 alone had the best specificity ( 0.98 ) but the worst sensitivity ( 0.25 ) . the performance of ppi alone was between icd-9 and nlp . for combinations , the nlp | ppi and nlp | ppi | icd-9 had the best performance by improving the patient coverage over nlp alone ( 88% for nlp to 96% and 100% for nlp | ppi and nlp | ppi | icd-9 , respectively ) . icd-9 generally did not provide additional improvement to nlp and nlp | ppi . using ppi or icd as the primary source supplemented with nlp generally showed poor performance ( data not shown ) . for 329 patients with smoking strength identified by manual review , nlp alone was able to identify 184 ( 56% ) patients with smoking strength - related information and achieved sensitivity of 0.74 , specificity of 0.88 , and accuracy of 0.75 . while ppi alone had poor patient coverage rate ( 23% ) , it was improved when combined with nlp . nlp | ppi together had 63% patient coverage rate , with sensitivity of 0.73 , specificity of 0.9 , and accuracy of 0.74 ( table 2 ) . at the second level of comparing standard smoking status types , nlp identified 263 ( 51% ) patients who have exactly the same detailed smoking status as in manual review ( table 3 ) . among them , 135 patients are current or former smokers , 4 are smokers with unknown current status , and 124 are never smokers . for the 248 ( 49% ) patients without exactly the same smoking status , the majority ( n = 172 ) was caused by the misidentification among current smoker , former smoker , and smoker with unknown current status . there were 8 cases where smokers were misidentified as never smokers and 53 cases where never smokers were misidentified as smokers . the performance of nlp ( 51% exact match ) was generally better than ppi . supplementing ppi to nlp manual review of 10 patients who were ever smokers in the gold standard but misidentified as never smokers by nlp showed that five patients had no smoking - related labels in the original clinical notes before diagnosis or within 30 days after diagnosis . four patients had at least one never or nonsmoker label possibly due to incorrect documentation . one patient had a former smoker label appeared 6 years after the diagnosis date . on the other hand , manual review of 10 never smokers misidentified as ever smokers by nlp indicated that two patients had never smoker tags coexisting with smoking amount tags . one patient had both a former smoker tag and more than one smoking frequency - or duration - related tags . a considerable number ( n = 83 ) of those classified as smoker with unknown current status were identified as current smoker by nlp . this is primarily due to our algorithm that considers a patient as current smoker if ever appeared in history and has no quit tag . while in manual review , ever smokers without a smoking quit year listed were considered as without known current status . however , the manual review is very subjective and it is not clear how the abstractor distinguished the smoking information that appeared 5 years ago and the smoking information that appeared a month ago . in our nlp algorithm , we simplified the algorithm so that ever smoker without a quit tag is considered as current smoker . in this study , we compared independent and joint performance of extracting smoking information through different sources including narrative text ( parsed by nlp ) , ppi , and icd-9 . results showed that nlp and ppi could provide adequate information for smoking detection , while icd-9 had almost no additional contribution . our analysis indicates that using nlp as the primary source of information , supplemented with other sources , has better performance , suggesting narrative text as a more reliable and comprehensive source for obtaining smoking - related information . while adding ppi information did not improve the sensitivity and specificity , it remarkably improved the patient coverage rate over nlp alone . this suggested that ppi , the readily available structured information , could be used as an extra source for more comprehensive patient coverage . smoking status was determined by using clinical information in emrs no later than 30 days after the diagnosis ( lung cancer ) date . the performance of nlp with respect to patient coverage is dependent on the availability of smoking information and the associated details . for example , the majority of the patients with no smoking information extracted by nlp had no smoking - related information in their original clinical notes . unlike previous studies that primarily focused on smoking status,4,15,16 our study features smoking strength detection . despite the incomplete original clinical notes , some missing values regarding smoking strength could be inferred . the final smoking strength was determined through longitudinal emrs for each patient , which maximized the possibility of detecting the true smoking strength . , about half of the patients did not provide smoking - related information and some of the remaining patients provided incomplete or contradictive information . we investigated the current smokers by gold standard , for example , one patient answered  no  to the question of  cigarettes current use  , while in clinical notes , a status of current smoker was recorded . the limitation of the study is that heuristic rules to obtain smoking status - related information in nlp were developed on the basis of existing research data from a lung cancer cohort , and the performance was only evaluated based on these data ; no cross evaluation was conducted . our results indicate nlp alone has the best sensitivity , and using joint sources of information primarily improved patient coverage over nlp alone . nlp is a relatively comprehensive and reliable resource for extracting both smoking status and strength information . the nlp rules we used are very generic , not specifically trained and not cancer specific .
objectivethe primary aim was to compare independent and joint performance of retrieving smoking status through different sources , including narrative text processed by natural language processing ( nlp ) , patient - provided information ( ppi ) , and diagnosis codes ( ie , international classification of diseases , ninth revision [ icd-9 ] ) . 
 we also compared the performance of retrieving smoking strength information ( ie , heavy / light smoker ) from narrative text and ppi.materials and methodsour study leveraged an existing lung cancer cohort for smoking status , amount , and strength information , which was manually chart - reviewed . on the nlp side , 
 smoking - related electronic medical record ( emr ) data were retrieved first . 
 a pattern - based smoking information extraction module was then implemented to extract smoking - related information . 
 after that , heuristic rules were used to obtain smoking status - related information . 
 smoking information was also obtained from structured data sources based on diagnosis codes and ppi . 
 sensitivity , specificity , and accuracy were measured using patients with coverage ( ie , the proportion of patients whose smoking status / strength can be effectively determined).resultsnlp alone has the best overall performance for smoking status extraction ( patient coverage : 0.88 ; sensitivity : 0.97 ; specificity : 0.70 ; accuracy : 0.88 ) ; combining ppi with nlp further improved patient coverage to 0.96 . 
 icd-9 does not provide additional improvement to nlp and its combination with ppi . for smoking strength , 
 combining nlp with ppi has slight improvement over nlp alone.conclusionthese findings suggest that narrative text could serve as a more reliable and comprehensive source for obtaining smoking - related information than structured data sources . 
 ppi , the readily available structured data , could be used as a complementary source for more comprehensive patient coverage .