Interpretation of Mammogram and Chest X-Ray Reports Using Deep Neural Networks - Preliminary Results
Hojjat Salehinejad Shahrokh Valaee Aren Mnatzakanian Tim Dowdell Joseph Barfett Errol Colak


























Interpretation of Mammogram and Chest Radiograph Reports Using Deep Neural Networks
    Hojjat Salehinejad,
        Shahrokh Valaee, Senior Member, IEEE,
        Aren Mnatzakanian,      
        Tim Dowdell,

        Joseph Barfett, and
        Errol Colak 


H. Salehinejad is with the Edward S. Rogers Sr. Department of Electrical & Computer Engineering, University of Toronto, Toronto, Canada, and Department of Medical Imaging, St. Michael's Hospital, University of Toronto, Toronto, Canada, e-mail: salehinejadh@smh.ca.
S. Valaee is with the Edward S. Rogers Sr. Department of Electrical & Computer Engineering, University of Toronto, Toronto, Canada, e-mail: valaee@ece.utoronto.ca.
A. Mnatzakanian, T. Dowdell, J. Barfett, and E. Colak are with the Department of Medical Imaging, St. Michael's Hospital, University of Toronto, Toronto, Canada, e-mail: {mnatzakaniana,dowdellt,barfettj,colake}@smh.ca.

    
========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================

pageStyleOne




Radiology reports are an important means of communication between radiologists and other physicians. These reports express a radiologist's interpretation of a medical imaging examination and are critical in establishing a diagnosis and formulating a treatment plan. In this paper, we propose a Bi-directional convolutional neural network (Bi-CNN) model for the interpretation and classification of mammograms based on breast density and chest radiographic radiology reports based on the basis of chest pathology. The proposed approach is a part of an auditing system that evaluates radiology reports to decrease incorrect diagnoses. Our study revealed that the proposed Bi-CNN outperforms convolutional neural network, random forest and support vector machine methods.



 




Breast density, chest radiograph, convolutional neural networks, mammography, radiology reports classification.


















 



§ INTRODUCTION



Radiology reports are an important means of communication between radiologists and other physicians <cit.>. These reports express a radiologist's interpretation of a medical imaging examination and are critical in establishing a diagnosis and formulating a treatment plan. 

Radiology is among the medical specialties with the highest rate of malpractice claims <cit.>. These claims can arise from a failure to communicate important findings, a failure of perception, lack of knowledge, and misjudgment. A failure to detect an abnormality on a medical imaging examination can lead to significant medical consequences for patients such as a delayed diagnosis. With a delayed or incorrect diagnosis, patients can present later with worsened symptoms and more advanced disease that may require more aggressive treatment or may be untreatable.




















In this paper, we propose an auditing system for radiologists that has two main components: a natural language processing (NLP) model to process and interpret a radiology report and a machine vision model that interprets the medical imaging examination. This auditing system reviews the radiologist's report and compares it with the interpretation of a machine vision model. The proposed system would notify the radiologist if there is a discrepancy between the two interpretations. Many investigators have aimed to develop machine vision models for the interpretation of medical imaging examinations, such as chest radiographs <cit.>, mammograms <cit.>, and head computerized tomography (CT) scans <cit.>. However, fewer attempts have been made for the NLP of radiology reports <cit.>. The focus of this paper is on the design and performance evaluation of the NLP component.

We propose a bi-directional convolutional neural network (Bi-CNN) for the NLP of radiology reports. The Bi-CNN will be trained independently for two distinct report data types: breast mammograms and chest radiographs. In particular, this model will classify the degree of breast density and the type of chest pathology based on the radiology report content. Our proposed NLP model differs from a keyword search algorithm. It is capable of interpreting and classifying a radiology report. The proposed Bi-CNN has two independent input channels, where the order of non-padded input to one channel is the reverse of the non-padded input to the other channel. Performance of the proposed model is compared with a single channel convolutional neural network (CNN), random forest (RF), and support vector machine (SVM) models and a comparative study is conducted. 

















§ RELATED WORK


The NLP models used for the classification and interpretation of radiology reports can be categorized into rule-based and machine learning models. Rule-based systems are usually built upon a series of “if-then” rules, which require an exhaustive search through documents. Such rules are typically designed by human experts. By contrast machine learning models do not require explicit rules. Instead, they try to train themselves iteratively and extract features from data. 



 §.§ Rule-based Approaches

A radiology report mining system generally has three main components. These include a medical finding extractor, a report and image retriever, and a text-assisted image feature
extractor <cit.>. For example, a set of hand-crafted semantic rules are presented in <cit.> for mining brain CT radiology reports. A natural language parser for medical reports is proposed in <cit.> that uses four-gram and higher order systems to assign a stability metric of a reference word within
a given sentence. This model is inspired from Field theory and uses a dependency tree that represents
the global minimum energy state of the system of words for
a given sentence in radiology reports <cit.>. A classifier using extracted keywords from clinical chest radiograph reports is developed in <cit.>. In <cit.>, a method is developed to identify patients with mediastinal findings related to inhalational anthrax. These methods are applied on different case studies with different sets of rules. Such methods are not flexible enough to be applied to case studies for which they were not designed, and hence are not transferable, and their performance comparison requires specific implementation and fine tuning of rules.




 §.§ Machine Learning Approaches

A NLP study for a large database of radiology reports was conducted in <cit.> based on two classes: positive and negative radiology findings. These classes have different patient attributes such as age groups, gender, and clinical indications <cit.>. A collection of 99 musculoskeletal radiology examinations were studied using a machine learning model versus a naive Bayes classification algorithm, followed by a SVM to detect limb fractures from radiology reports <cit.>. A CNN was used in <cit.> as a supervised tool to map from images
to label spaces. This system has an interactive component between supervised and unsupervised machine learning models to extract
and mine the semantic interactions of radiology images and
reports. Recurrent neural networks (RNN) along with the CNN have been used to interpret chest x-ray reports for a limited number of classes <cit.>. This CNN model learns from the annotated sequences produced by the RNN model. The RNN model with long short term memory (LSTM) is capable of learning long term dependencies between the annotations <cit.>. The model proposed in <cit.> uses a recurrent design to extract features from word representations. Compared with window-based neural networks, which uses a rolling window to extract features, this model generates less noise and is more stable. The max-pooling method distinguishes the words that have a major impact in the report.  
Machine learning models have also been developed for understanding medical reports in other languages, Korean being one example, using tools such as Naive Bayes classifiers,
maximum entropy, and SVMs <cit.>. 




 §.§ Learning Vector Representations of Words

Word2vec is a well-known model used for learning vector space representations of words. It produces a vector space from a large corpus of text with reduced dimensionality. This model assigns a word vector to each unique word in the corpus. Vectors with closer contexts are positioned in closer proximity in this space <cit.>. 

GloVe approach provides a global vector space representation of words. It uses global matrix factorization and local context windowing methods to construct a global log-bilinear regression model <cit.>. The word-word co-occurrence matrix of the words is a sparse matrix by nature. However, GloVe only uses the non-zero elements for training and does not consider individual context windows in a large corpus <cit.>. CNNs have been shown to work well on n-gram representations of data <cit.>. For example, a convolution layer can perform feature extraction from various n values of a n-gram model and perform personality detection from documents <cit.>. Dynamic k-Max pooling can operate as a global operation over linear sequences. Such dynamic CNNs can perform semantic modelling of sentences <cit.>. The model receives variable dimension size input sentences and induces a feature graph over the sentence that is capable of explicitly capturing short and long-range relations <cit.>.

CNN models are also utilized for learning representations, opinion sentiment understanding, and analysis of products. For example, a CNN with a single output softmax layer can classify a vector representation of text with high accuracy <cit.>. The design of output layers in CNNs varies depending on the application of the model. A collaboration of RNNs and CNNs for feature extraction of sentences has been examined, where a CNN learns features from an input sentence and then a gated RNN model discourses the information <cit.>. In such system, a bi-directional long short term memory (Bi-LSTM) RNNs can sequentially learn words from question-answer sentences. The trained network can select an answer sentence for a question and present the corresponding likelihood for correctness <cit.>. 





§ A BRIEF REVIEW ON CONVOLUTIONAL NEURAL NETWORKS


As NLP for radiology reports is an interdisciplinary research effort, a brief review on CNN is provided in this section. A CNN is a machine learning model inspired by the visual cortex of cats <cit.>. A CNN has at least one layer of convolution and sub-sampling, in which the number of layers can increase in depth, generally followed by a fully connected multi-layer perceptron (MLP) network. A consecutive arrangement of convolutional layers followed by sub-sampling build a pyramid-shape model, where the number of feature maps increases as the spatial resolution decreases. Instead of hand-designing feature extractors, the convolutional layers extract features from raw data and the MLP network classifies the features <cit.>. 
A CNN typically has three main pieces which are local receptive fields, shared weights, and spatial and/or temporal sub-sampling.



 §.§ Local Receptive Fields

Local receptive fields are made from artificial neurons, which observe and extract features from data such as edges in images. Let us consider an image as input to a CNN such as in Figure <ref>. A rectangular kernel with size M× N scans the input matrix at every single element (i.e., a pixel) and performs convolution such as 

    h_u,v = σ(∑_m=0^M-1∑_n=0^N-1(I_u+m,v+n· w_m,n)+b_u,v)

where the output is the state of the neurons at element I_u,v, σ(·) is a non-linear function, W is the shared weights matrix, and b is the bias vector.



 §.§ Shared Weights

The term “shared weights" refers to the weight matrix W which appears repeatedly in the convolution operation for every element of I. Weight sharing reduces the number of free parameters of the model and hence improves generalization of the model  <cit.>.

For a general machine vision task, such as classification of natural images, the convolution kernel uses the correlation among spatial or temporal elements of the image to extract local features. Since the image is stationary (i.e., statistics of one part of the image are similar to any other part), the learned shared weights using one sub-region of the image can also be used at another sub-region to extract different features.



 §.§ Sub-sampling

Passing the whole bag of extracted features after the convolution operation to a classifier is computationally expensive <cit.>. The feature pooling task (i.e. sub-sampling) generalizes the network by reducing the resolution of the dimensionality of intermediate representations (i.e. feature maps) as well as the sensitivity of the output to shifts and distortions <cit.>. The two most popular subsampling methods are mean-pooling and max-pooling. By dividing the feature map into a number of non-overlapping rectangle-shape sub-regions, the mean-pooling computes the average of features sitting inside a sub-region as the pooled feature of that sub-region. Max-pooling performs the same task as mean-pooling but with a maximum operator. A study on these two operations is provided in <cit.>.









§ THE PROPOSED METHOD


An auditing system for radiology is presented in Figure <ref>. The main components of the system are a machine vision model to interpret the input image and a NLP component to interpret the corresponding radiology report. The system notifies the radiologist if there is a discrepancy between the two components of the system. In this paper, our focus is on the design and evaluation of the NLP component.





 §.§ Preprocessing

In general, radiology reports include major sections such as “Indication", “Findings", “Impression", and  the name of the reporting radiologist. However, there are variations in report formatting due to “radiologist style" or institutional requirements. Despite brute-force search approaches, the deep learning models have the advantage of requiring the least amount of data cleaning and preprocessing due to their natural adaptation to the data and non-linear feature extraction ability.

In the preprocessing step, the model extracts the “Findings" and “Impression" sections of the report and performs tokenization and string cleaning. Some of the major tasks include converting upper-case characters to lower-case, removing unnecessary punctuation and symbols, and separating the remainder from the attached string. We add every unique word to a vocabulary dictionary. Each vocabulary has a unique associated index which represents it in the sentence vector. For example, given the sentence “The breasts show scattered fibroglandular tissue.", the entire vocabulary dictionary would consist of 6 words (breasts, show, scattered, fibroglandular, tissue, the), which is represented as a list of integers such as [1,...,6]. A vector to the length of the longest report in terms of words count in the dataset represents the report. Padding fills up the gap for shorter reports. 

In the character embedding step, each integer is mapped to a high dimensional (i.e., D) vector with a uniform random distribution. 
In such a high dimensional space, due to the “curse of dimensionality",  the vectors are considered independent <cit.>, <cit.>. Character embedding vector generation for medical vocabularies based on the correlation among the vectors (i.e., similar to Word2Vec) is an interesting topic for further investigation. 
 
 


 §.§ Bi-CNN Model Architecture

An input report with N words can be represented as a sequence
X= [x_1,... ,x_N], where each word is a vector x_n∈ℝ^D, <cit.>. As Figure <ref> shows, the proposed architecture has two input channels followed by two independent convolution layers. Both channels have an identical design, except the order of non-padded input to one channel is the reverse of the non-padded input to the other channel. In the example in Figure <ref>, the dependencies between feature vectors (x_3,x_4,x_5) are visible to the kernels in both channels. However, the dependencies between feature vectors (x_1,x_0,0) are only visible to the kernels in Channel 2, while these dependencies are not visible to the kernels in Channel 1. Other examples are the feature vectors (x_0,0,0) in Channel 2 and feature vectors (x_4,x_5,0) and (x_5,0,0) in Channel 1, which are not visible in the corresponding other channel.

Each channel in the proposed model in Figure <ref> has three filters with varying window sizes K∈{3,4,5} that slide across the input layer <cit.>, <cit.>. The filters extract features from the input layer to construct feature maps of size (N-K+1)× F, where F is the number of feature maps for each filter. Each feature map h_(N-K+1) has its own shared weight W_K× D and bias b_ N-K+1. The value of a hidden neuron m is


    h_m=σ(∑_k=1^K∑_d=1^D x_m+k-1,d· w_k,d +b_m).

The window word is x_m:m+K-1 and σ(·) is a rectified linear unit (ReLU) activation function defined as 

    σ(z)=max(0,z)
 
where z∈ℝ. The max pooling <cit.> extracts the most significant feature from the feature map of a filter as

    ĥ_l=max({h_1,..,h_N-K+1}).

The output of the max-pooling layer contains the max-pooled features ĥ=[ĥ_1,...,ĥ_L] where the length of the feature vectors is L=C· F and C is the number of classes. For example, for the breast density classification with five classes we have L=5F (see Figure <ref>).
The features from the input channels are concatenated and passed to a fully connected perceptron network. The output of the softmax layer is the probability distribution over all the labels P (e.g., for mammograms the breast density classes). The value of output unit p is 


    y_p = ϕ(∑_l=1^L (ĥ_1,l· r_1,l)· w_l,p^ho + b_p^o)

where w_l,p^ho is the weight of connection from the hidden unit l in the hidden layer h to the output unit p in the output layer o,  b_p^o is the bias of the output unit p, and ϕ(z) is the softmax activation function defined as

    ϕ(z_p)=e^z_p/∑_j=1^Pe^z_j  for   p=[1,...,P].





 §.§ Training

Adam optimizer is a first-order gradient-based optimization method with integrated momentum functionality <cit.>. During training, the momentum helps to diminish the fluctuations in weight changes over consecutive iterations. The drop-out regularization method randomly drops units along with their connections from the 
concatenated layer (i.e. the input layer to classifier) to the output layer using a binary mask vector r_1× L with Bernoulli random distribution <cit.>.



 §.§ Evaluation Scheme

Since we are dealing with discrete categories, we use cross-entropy between a predicted value y^(i) from the network and the real label t^(i) to measure the loss of networks such as

    ℒ̂(Y, T) = -1/R∑_i=1^Rt^(i)ln(y^(i))+(1-t^(i))ln(1-y^(i))

where R is the number of reports, Y is the set of network predictions, and T is the set of targets to predict. Adding the weight decay term (i.e., L_2 regularization) to the loss function helps the network to avoid over-fitting while training such as

    ℒ(Y,  T) =  ℒ̂(Y, T) + η(𝐖^ho_2)

where 𝐖^ho is the weight matrix of connections between the hidden layer h and the output layer o and η is the regularization control parameter.


 




















  
  


§ THE DATASETS


Our institutional review board approved this single-center retrospective study with a waiver for informed consent. The Research Ethics Boards (REB) number is 17-167. A search of our Radiology Information System (RIS) (Syngo; Siemens Medical Solutions USA Inc, Malvern, PA) was preformed for mammography and chest radiograph reports using Montage Search and Analytics (Montage Healthcare Solutions, Philadelphia, PA). This search identified 4,080 mammography reports and 1,030 chest radiographic reports (see Table <ref>). These reports were exported from the RIS and removed of any patient identifying information.



 §.§ Mammogram Reports Dataset







The mammography reports dataset (MRD) contained 4,080 reports. Breast density was classified in each report according to the American College of Radiology Breast Imaging Reporting and Data System (BI-RADS) classification system <cit.> 
(Figure <ref>). Breast density reflects the relative composition of fat and fibroglandular tissue. Almost entirely fatty refers to breasts with less than 25% areas of fibroglandular tissue. Scattered ares of fibroglandular density is used for breasts composed of 25-50% fibroglandular tissue. Heterogeneously dense breasts are composed of 50-75% fibroglandular tissue and the extremely dense breasts have greater than 75% of fibroglandular tissue density. Descriptive statistics of this dataset are presented in Table <ref> and sample reports are presented in Table <ref>. 





 §.§ Chest Radiographs Reports Dataset

The chest radiograph dataset (CRRD) consisted of 1,030 reports. These reports were classified into 11 categories which includes normal examinations and ten pathologic states (Figure <ref>). Descriptive statistics of this dataset are presented in Table <ref> and sample reports are presented in Table <ref>. This dataset has more categories, fewer reports per category, and is less imbalanced when compared with the MRD. 











§ EXPERIMENTS AND RESULTS ANALYSIS



In this section, we compare performance of the proposed Bi-CNN model with the random forest (RF), SVM, and CNN models. The experiments were conducted on both the mammogram and chest radiograph datasets. 

The CNN and Bi-CNN models are implemented in TensorFlow <cit.> and the RF and SVM models are implemented using the classifiers in scikit-learn <cit.>. The experiments are conducted on a DevBox with an Intel Core i7-5930K 6 Core 3.5GHz desktop processor, 64 GB DDR4 RAM, and two TITAN X GPUs with 12GB of memory per GPU.



 §.§ Parameters Setting

Unless stated, the CNN and Bi-CNN models are trained with a mini-batch size of 64, drop-out probability of 0.5, filter sizes {3, 4, 5} and 120 feature maps per filter size. The number of training iterations is set to 50 and the initial learning rate for Adam optimizer is set to 0.001 <cit.>. 
An exponential decay adaptive learning rate is applied. The weights at output layers are initialized 
using the Xavier method <cit.>, the weights in the convolutional layer are selected based on normal distribution with standard deviation of 0.1, and biases are set to 0.1. The activation function before the max-pooling layer is ReLU <cit.>. The L_2 regularization is set to 1.0 × 10^-4 and early-stopping is applied.

For all the experiments, 70%, 15%, and 15% of data is allocated for training, validation, and test, respectively. The data is shuffled before splitting. In each experiment, the model is cross-validated over 30 independent experiments and the results after statistical testing are reported. 



 §.§ Performance Evaluation and Analysis

We experimented with various configurations of the RF <cit.>, SVM <cit.>, and CNN <cit.> models.

  

  * RF: Instead of using a bag-of-the-words technique, which uses the frequency of the words in a query as the features, a n-gram model with a lower boundary of 1 and upper boundary n∈{1,2,3} performs the feature extraction of reports. We use an implementation of RF <cit.> which combines classifiers by averaging their probabilistic predictions. The number of estimators is set to 10. 


  * SVM: Similar to RF, the SVM model uses n-gram feature extraction. The SVM model is deployed for two different kernels, a “sigmoid" and a “polynomial" with degree three.  



  * CNN: The CNN implementation proposed in <cit.> for text classification and later used in <cit.> for the classification of radiology head CT reports is used. This model has a single input channel and the order of input words is similar to the order in the report. The model has a convolution layer followed by a max-pooling layer and a “softmax" classifier. The studies are for kernel sizes k∈{1,2,3}.


  * Bi-CNN: The proposed Bi-CNN with two input channels. The settings are similar to CNN.


The experiments are for breast density classification and chest radiograph classifications based on chest pathology using the MRD and CRRD, respectively. 


The results of performance comparisons between the RF, SVM, CNN, and Bi-CNN models are presented in Table <ref>. As n increases in n-gram for RF, it considers the dependency between a greater number of words (i.e. n). The RF with 3-gram model has better performance than the 2-gram and 1-gram RF models. The SVM with polynomial of degree three (i.e., “poly") and “sigmoid" kernels almost have the same performance. The number of kernels in the CNN models have a minor impact on accuracy. However, the CNN models with 3-kernel have better performance than models with 1-kernel and 2-kernel. The proposed Bi-CNN with 3-kernel has the best performance compared to the other models.










The convergence behaviour of CNN and Bi-CNN over training iterations for the validation dataset is presented in Figure <ref>. For a large learning rate value (i.e., λ=0.1), the models converge rapidly to a local solution. However, for smaller learning rates, the models converge more slowly but with greater stability towards a better solution. As the results in Table <ref> show, the CNN model converges with a lower accuracy for three different learning rates λ∈{0.1,0.01,0.001}. For λ=0.001, the best performance is achieved by Bi-CNN for both MRD and CRRD. One of the advantages of Bi-CNN is adding diversity into the model through the integration of two different representations of feature vectors into the model. This diversity helps the CNN avoid convergence to local solutions with low accuracy.




§ CONCLUSION


Radiology reports contain a radiologist's interpretation of a medical imaging examination. Vast numbers of such reports are digitally stored in health care facilities. Text mining and knowledge extraction from such data may potentially be used for double checking radiologist interpretations as part of an audit system. Furthermore, these methods can be used to audit utilization of limited health care resources and to enhance the study of patient treatment plans over time.

In this paper, we have proposed a Bi-CNN for the interpretation of radiology reports. We have collected and anonymized two datasets for our experiments: a mammogram reports dataset and a chest radiograph reports dataset. The Bi-CNN has two input channels where one channel represents the features of the report (including zero-padding) and the other channel represents the reverse non-padded features of the report. The combination of forward and reverse order of feature vectors represents more information about dependencies between feature vectors. Our comparative studies demonstrate that the Bi-CNN model outperforms the CNN, RF, and SVM methods.










§ ACKNOWLEDGEMENT

The authors thank the support of NVIDIA Corporation with the
donation of the Titan X GPUs used for this project.

IEEEtran














































