This paper is submitted for peer-review.

Interpretation of Mammogram and Chest
Radiograph Reports Using Deep Neural Networks

arXiv:1708.09254v3 [cs.CV] 12 Sep 2017

Hojjat Salehinejad, Shahrokh Valaee, Senior Member, IEEE, Aren Mnatzakanian, Tim Dowdell,
Joseph Barfett, and Errol Colak
Abstract—Radiology reports are an important means of communication between radiologists and other physicians. These
reports express a radiologist’s interpretation of a medical imaging
examination and are critical in establishing a diagnosis and
formulating a treatment plan. In this paper, we propose a Bidirectional convolutional neural network (Bi-CNN) model for the
interpretation and classification of mammograms based on breast
density and chest radiographic radiology reports based on the
basis of chest pathology. The proposed approach is a part of
an auditing system that evaluates radiology reports to decrease
incorrect diagnoses. Our study revealed that the proposed BiCNN outperforms convolutional neural network, random forest
and support vector machine methods.
Index Terms—Breast density, chest radiograph, convolutional
neural networks, mammography, radiology reports classification.

I. I NTRODUCTION
Radiology reports are an important means of communication between radiologists and other physicians [1]. These
reports express a radiologist’s interpretation of a medical imaging examination and are critical in establishing a diagnosis and
formulating a treatment plan.
Radiology is among the medical specialties with the highest
rate of malpractice claims [2]. These claims can arise from
a failure to communicate important findings, a failure of
perception, lack of knowledge, and misjudgment. A failure
to detect an abnormality on a medical imaging examination
can lead to significant medical consequences for patients such
as a delayed diagnosis. With a delayed or incorrect diagnosis,
patients can present later with worsened symptoms and more
advanced disease that may require more aggressive treatment
or may be untreatable.
In this paper, we propose an auditing system for radiologists
that has two main components: a natural language processing
(NLP) model to process and interpret a radiology report and
a machine vision model that interprets the medical imaging
examination. This auditing system reviews the radiologist’s
report and compares it with the interpretation of a machine
vision model. The proposed system would notify the radiologist if there is a discrepancy between the two interpretations.
Many investigators have aimed to develop machine vision
H. Salehinejad is with the Edward S. Rogers Sr. Department of Electrical
& Computer Engineering, University of Toronto, Toronto, Canada, and Department of Medical Imaging, St. Michael’s Hospital, University of Toronto,
Toronto, Canada, e-mail: salehinejadh@smh.ca.
S. Valaee is with the Edward S. Rogers Sr. Department of Electrical
& Computer Engineering, University of Toronto, Toronto, Canada, e-mail:
valaee@ece.utoronto.ca.
A. Mnatzakanian, T. Dowdell, J. Barfett, and E. Colak are with the Department of Medical Imaging, St. Michael’s Hospital, University of Toronto,
Toronto, Canada, e-mail: {mnatzakaniana,dowdellt,barfettj,colake}@smh.ca.

models for the interpretation of medical imaging examinations,
such as chest radiographs [3], mammograms [4], and head
computerized tomography (CT) scans [5]. However, fewer
attempts have been made for the NLP of radiology reports [6]–
[8]. The focus of this paper is on the design and performance
evaluation of the NLP component.
We propose a bi-directional convolutional neural network
(Bi-CNN) for the NLP of radiology reports. The Bi-CNN will
be trained independently for two distinct report data types:
breast mammograms and chest radiographs. In particular,
this model will classify the degree of breast density and
the type of chest pathology based on the radiology report
content. Our proposed NLP model differs from a keyword
search algorithm. It is capable of interpreting and classifying
a radiology report. The proposed Bi-CNN has two independent
input channels, where the order of non-padded input to one
channel is the reverse of the non-padded input to the other
channel. Performance of the proposed model is compared with
a single channel convolutional neural network (CNN), random
forest (RF), and support vector machine (SVM) models and a
comparative study is conducted.
II. R ELATED W ORK
The NLP models used for the classification and interpretation of radiology reports can be categorized into rule-based and
machine learning models. Rule-based systems are usually built
upon a series of “if-then” rules, which require an exhaustive
search through documents. Such rules are typically designed
by human experts. By contrast machine learning models do
not require explicit rules. Instead, they try to train themselves
iteratively and extract features from data.
A. Rule-based Approaches
A radiology report mining system generally has three main
components. These include a medical finding extractor, a
report and image retriever, and a text-assisted image feature
extractor [9]. For example, a set of hand-crafted semantic rules
are presented in [9] for mining brain CT radiology reports.
A natural language parser for medical reports is proposed in
[10] that uses four-gram and higher order systems to assign a
stability metric of a reference word within a given sentence.
This model is inspired from Field theory and uses a dependency tree that represents the global minimum energy state of
the system of words for a given sentence in radiology reports
[10]. A classifier using extracted keywords from clinical chest
radiograph reports is developed in [11]. In [12], a method is
developed to identify patients with mediastinal findings related
to inhalational anthrax. These methods are applied on different

2

2x2 kernel

Feature maps

Convolution
Subsampling

2D Input Data

14x14x64

32x32
28x28x64

Fig. 1: Convolution and sub-sampling in convolutional neural networks.

case studies with different sets of rules. Such methods are not
flexible enough to be applied to case studies for which they
were not designed, and hence are not transferable, and their
performance comparison requires specific implementation and
fine tuning of rules.
B. Machine Learning Approaches
A NLP study for a large database of radiology reports was
conducted in [13] based on two classes: positive and negative radiology findings. These classes have different patient
attributes such as age groups, gender, and clinical indications
[13]. A collection of 99 musculoskeletal radiology examinations were studied using a machine learning model versus a
naive Bayes classification algorithm, followed by a SVM to
detect limb fractures from radiology reports [14]. A CNN was
used in [15] as a supervised tool to map from images to label
spaces. This system has an interactive component between
supervised and unsupervised machine learning models to extract and mine the semantic interactions of radiology images
and reports. Recurrent neural networks (RNN) along with the
CNN have been used to interpret chest x-ray reports for a
limited number of classes [16]. This CNN model learns from
the annotated sequences produced by the RNN model. The
RNN model with long short term memory (LSTM) is capable
of learning long term dependencies between the annotations
[17]. The model proposed in [18] uses a recurrent design to
extract features from word representations. Compared with
window-based neural networks, which uses a rolling window
to extract features, this model generates less noise and is more
stable. The max-pooling method distinguishes the words that
have a major impact in the report. Machine learning models
have also been developed for understanding medical reports in
other languages, Korean being one example, using tools such
as Naive Bayes classifiers, maximum entropy, and SVMs [19].
C. Learning Vector Representations of Words
Word2vec is a well-known model used for learning vector
space representations of words. It produces a vector space
from a large corpus of text with reduced dimensionality. This
model assigns a word vector to each unique word in the corpus.
Vectors with closer contexts are positioned in closer proximity
in this space [20]. GloVe approach provides a global vector
space representation of words. It uses global matrix factorization and local context windowing methods to construct a
global log-bilinear regression model [21]. The word-word cooccurrence matrix of the words is a sparse matrix by nature.
However, GloVe only uses the non-zero elements for training
and does not consider individual context windows in a large

corpus [21]. CNNs have been shown to work well on ngram representations of data [22]. For example, a convolution
layer can perform feature extraction from various n values
of a n-gram model and perform personality detection from
documents [22]. Dynamic k-Max pooling can operate as a
global operation over linear sequences. Such dynamic CNNs
can perform semantic modelling of sentences [23]. The model
receives variable dimension size input sentences and induces
a feature graph over the sentence that is capable of explicitly
capturing short and long-range relations [23].
CNN models are also utilized for learning representations,
opinion sentiment understanding, and analysis of products.
For example, a CNN with a single output softmax layer can
classify a vector representation of text with high accuracy [24].
The design of output layers in CNNs varies depending on the
application of the model. A collaboration of RNNs and CNNs
for feature extraction of sentences has been examined, where
a CNN learns features from an input sentence and then a gated
RNN model discourses the information [25]. In such system, a
bi-directional long short term memory (Bi-LSTM) RNNs can
sequentially learn words from question-answer sentences. The
trained network can select an answer sentence for a question
and present the corresponding likelihood for correctness [26].
III. A B RIEF R EVIEW ON C ONVOLUTIONAL N EURAL
N ETWORKS
As NLP for radiology reports is an interdisciplinary research
effort, a brief review on CNN is provided in this section. A
CNN is a machine learning model inspired by the visual cortex
of cats [27]. A CNN has at least one layer of convolution
and sub-sampling, in which the number of layers can increase
in depth, generally followed by a fully connected multilayer perceptron (MLP) network. A consecutive arrangement
of convolutional layers followed by sub-sampling build a
pyramid-shape model, where the number of feature maps
increases as the spatial resolution decreases. Instead of handdesigning feature extractors, the convolutional layers extract
features from raw data and the MLP network classifies the
features [28]. A CNN typically has three main pieces which
are local receptive fields, shared weights, and spatial and/or
temporal sub-sampling.
A. Local Receptive Fields
Local receptive fields are made from artificial neurons,
which observe and extract features from data such as edges in
images. Let us consider an image as input to a CNN such as
in Figure 1. A rectangular kernel with size M × N scans the

3

Feedback to Radiologist

Radiologist

Findings:
The breast parenchyma
is heterogeneously dense.
No suspicious masses
calcifications or architectural
distortion seen on either side.

NLP Machine

heterogeneously dense
NLP Machine Interpretation
From Radiologist Report

Radiologist Interpretation
Input Image for
Interpretation

Comparision

Machine Vision Model Interpretation
Machine Vision

heterogeneously dense

Fig. 2: Block-diagram of the proposed auditing system.

input matrix at every single element (i.e., a pixel) and performs
convolution such as
M
−1 N
−1
X
X
hu,v = σ(
(Iu+m,v+n · wm,n ) + bu,v )
(1)
m=0 n=0

where the output is the state of the neurons at element Iu,v ,
σ(·) is a non-linear function, W is the shared weights matrix,
and b is the bias vector.
B. Shared Weights
The term “shared weights” refers to the weight matrix W
which appears repeatedly in the convolution operation for
every element of I. Weight sharing reduces the number of free
parameters of the model and hence improves generalization of
the model [28].
For a general machine vision task, such as classification
of natural images, the convolution kernel uses the correlation
among spatial or temporal elements of the image to extract
local features. Since the image is stationary (i.e., statistics of
one part of the image are similar to any other part), the learned
shared weights using one sub-region of the image can also be
used at another sub-region to extract different features.
C. Sub-sampling
Passing the whole bag of extracted features after the convolution operation to a classifier is computationally expensive
[29]. The feature pooling task (i.e. sub-sampling) generalizes
the network by reducing the resolution of the dimensionality
of intermediate representations (i.e. feature maps) as well as
the sensitivity of the output to shifts and distortions [28].
The two most popular subsampling methods are mean-pooling
and max-pooling. By dividing the feature map into a number
of non-overlapping rectangle-shape sub-regions, the meanpooling computes the average of features sitting inside a subregion as the pooled feature of that sub-region. Max-pooling
performs the same task as mean-pooling but with a maximum
operator. A study on these two operations is provided in [29].
IV. T HE P ROPOSED M ETHOD
An auditing system for radiology is presented in Figure 2.
The main components of the system are a machine vision
model to interpret the input image and a NLP component
to interpret the corresponding radiology report. The system
notifies the radiologist if there is a discrepancy between the
two components of the system. In this paper, our focus is on
the design and evaluation of the NLP component.

A. Preprocessing
In general, radiology reports include major sections such
as “Indication”, “Findings”, “Impression”, and the name of
the reporting radiologist. However, there are variations in
report formatting due to “radiologist style” or institutional
requirements. Despite brute-force search approaches, the deep
learning models have the advantage of requiring the least
amount of data cleaning and preprocessing due to their natural
adaptation to the data and non-linear feature extraction ability.
In the preprocessing step, the model extracts the “Findings” and “Impression” sections of the report and performs
tokenization and string cleaning. Some of the major tasks
include converting upper-case characters to lower-case, removing unnecessary punctuation and symbols, and separating
the remainder from the attached string. We add every unique
word to a vocabulary dictionary. Each vocabulary has a unique
associated index which represents it in the sentence vector.
For example, given the sentence “The breasts show scattered
fibroglandular tissue.”, the entire vocabulary dictionary would
consist of 6 words (breasts, show, scattered, fibroglandular,
tissue, the), which is represented as a list of integers such as
[1, ..., 6]. A vector to the length of the longest report in terms
of words count in the dataset represents the report. Padding
fills up the gap for shorter reports.
In the character embedding step, each integer is mapped to
a high dimensional (i.e., D) vector with a uniform random
distribution. In such a high dimensional space, due to the
“curse of dimensionality”, the vectors are considered independent [30], [31]. Character embedding vector generation
for medical vocabularies based on the correlation among the
vectors (i.e., similar to Word2Vec) is an interesting topic for
further investigation.
B. Bi-CNN Model Architecture
An input report with N words can be represented as a
sequence X = [x1 , ..., xN ], where each word is a vector
xn ∈ RD , [32]. As Figure 3 shows, the proposed architecture has two input channels followed by two independent
convolution layers. Both channels have an identical design,
except the order of non-padded input to one channel is the
reverse of the non-padded input to the other channel. In
the example in Figure 4, the dependencies between feature
vectors (x3 , x4 , x5 ) are visible to the kernels in both channels.
However, the dependencies between feature vectors (x1 , x0 , 0)
are only visible to the kernels in Channel 2, while these
dependencies are not visible to the kernels in Channel 1. Other

Padding Non-padding

4

x0

{

K1

{

xn
xn+1

almost entirely fatty

F
K2

2x(N-K1+1)xF

K3

scattered fibroglandular dense

F

heterogeneously dense

xN

F

NxDe

P

Padding Non-padding

{
{

extremely dense
ambiguous

F
xn
K1
x0
xn+1

F

2x(N-K2+1)xF

F

K2
K3

xN

Fully Connected Layer with Softmax

NxDe

Input Report Vector Representation
Convolution with ReLU

2x(N-K3+1)xF Max Pooling + Concatenation

Fig. 3: Proposed bi-directional convolutional neural network for the interpretation and classification of mammograms based on breast density.
For the chest reports case, the labels of the output layer change accordingly but the model architecture is identical.
Channel 1

The
breast
parenchyma
is
heterogeneously
dense

x0
x1
x2
x3
x4
x5
0
0
0

Channel 2

dense
heterogeneously
is
parenchyma
breast
The

the input channels are concatenated and passed to a fully
connected perceptron network. The output of the softmax layer
is the probability distribution over all the labels P (e.g., for
mammograms the breast density classes). The value of output
unit p is

x5
x4
x3
x2
x1
x0
0
0
0

yp = φ(

Fig. 4: The order of feature vectors and padding for the two input
channels of bi-directional convolutional neural network (Bi-CNN).
Each channel has two kernels (green and purple frames) of size three.
The order of feature vectors in Channel 2 is the reverse order of
feature vectors in Channel 1.

examples are the feature vectors (x0 , 0, 0) in Channel 2 and
feature vectors (x4 , x5 , 0) and (x5 , 0, 0) in Channel 1, which
are not visible in the corresponding other channel.
Each channel in the proposed model in Figure 3 has three
filters with varying window sizes K ∈ {3, 4, 5} that slide
across the input layer [24], [32]. The filters extract features
from the input layer to construct feature maps of size (N −
K + 1) × F , where F is the number of feature maps for each
filter. Each feature map h(N −K+1) has its own shared weight
WK×D and bias bN −K+1 . The value of a hidden neuron m
is
hm = σ(

K X
D
X

xm+k−1,d · wk,d + bm ).

(2)

L
X
ho
(ĥ1,l · r1,l ) · wl,p
+ bop )

(5)

l=1
ho
where wl,p
is the weight of connection from the hidden unit l
in the hidden layer h to the output unit p in the output layer
o, bop is the bias of the output unit p, and φ(z) is the softmax
activation function defined as
ezp
φ(zp ) = PP
for p = [1, ..., P ].
(6)
zj
j=1 e

C. Training
Adam optimizer is a first-order gradient-based optimization
method with integrated momentum functionality [34]. During
training, the momentum helps to diminish the fluctuations
in weight changes over consecutive iterations. The drop-out
regularization method randomly drops units along with their
connections from the concatenated layer (i.e. the input layer
to classifier) to the output layer using a binary mask vector
r1×L with Bernoulli random distribution [35].

k=1 d=1

The window word is xm:m+K−1 and σ(·) is a rectified linear
unit (ReLU) activation function defined as
σ(z) = max(0, z)

(3)

where z ∈ R. The max pooling [33] extracts the most
significant feature from the feature map of a filter as
ĥl = max({h1 , .., hN −K+1 }).

(4)

The output of the max-pooling layer contains the max-pooled
features ĥ = [ĥ1 , ..., ĥL ] where the length of the feature
vectors is L = C · F and C is the number of classes.
For example, for the breast density classification with five
classes we have L = 5F (see Figure 3). The features from

D. Evaluation Scheme
Since we are dealing with discrete categories, we use crossentropy between a predicted value y (i) from the network and
the real label t(i) to measure the loss of networks such as
R

−1 X (i)
t ln(y (i) ) + (1 − t(i) )ln(1 − y (i) ) (7)
L̂(Y, T ) =
R i=1
where R is the number of reports, Y is the set of network
predictions, and T is the set of targets to predict. Adding the
weight decay term (i.e., L2 regularization) to the loss function
helps the network to avoid over-fitting while training such as
L(Y, T ) = L̂(Y, T ) + η( Who

2

)

(8)

5

where Who is the weight matrix of connections between the
hidden layer h and the output layer o and η is the regularization
control parameter.
TABLE I: Summary of statistics for the mammogram reports dataset
(MRD) and chest radiograph reports dataset (CRRD) for the “findings” and/or “impression” sections. NR: number of reports; VS:
vocabulary size; ANS: average number of sentences per report; ANW:
average number of words per report; ASL: average sentence length
per report; Mean: sample mean; Med: sample median; Std: sample
standard deviation.
ANS
ANW
ASL
Dataset NR VS
Mean Median Std Mean Median Std Mean Median Std
MRD 4,080 1,691 3.21 3.00 1.27 30.87 27.00 10.44 9.61 9.00 8.22
CRRD 1,030 772 2.01 2.00 1.07 21.46 19.00 9.21 8.21 8.00 7.67

(a)

(b)

(c)

(d)

Fig. 5: Four classes of breast density. a) Almost entirely fatty;
b) Scattered areas of fibroglandular density; c) Heterogeneously
dense; d) Extremely dense.

(a)

(b)

(e)

(c)

(f)

(i)

(d)

(g)

(j)

(h)

Medical Solutions USA Inc, Malvern, PA) was preformed
for mammography and chest radiograph reports using Montage Search and Analytics (Montage Healthcare Solutions,
Philadelphia, PA). This search identified 4,080 mammography
reports and 1,030 chest radiographic reports (see Table I).
These reports were exported from the RIS and removed of
any patient identifying information.
A. Mammogram Reports Dataset
The mammography reports dataset (MRD) contained 4,080
reports. Breast density was classified in each report according to the American College of Radiology Breast Imaging
Reporting and Data System (BI-RADS) classification system
[36] (Figure 5). Breast density reflects the relative composition
of fat and fibroglandular tissue. Almost entirely fatty refers
to breasts with less than 25% areas of fibroglandular tissue.
Scattered ares of fibroglandular density is used for breasts
composed of 25 − 50% fibroglandular tissue. Heterogeneously
dense breasts are composed of 50 − 75% fibroglandular tissue
and the extremely dense breasts have greater than 75% of fibroglandular tissue density. Descriptive statistics of this dataset
are presented in Table II and sample reports are presented in
Table III.
B. Chest Radiographs Reports Dataset
The chest radiograph dataset (CRRD) consisted of 1,030
reports. These reports were classified into 11 categories which
includes normal examinations and ten pathologic states (Figure 6). Descriptive statistics of this dataset are presented in
Table IV and sample reports are presented in Table V. This
dataset has more categories, fewer reports per category, and is
less imbalanced when compared with the MRD.
VI. E XPERIMENTS AND R ESULTS A NALYSIS
In this section, we compare performance of the proposed
Bi-CNN model with the random forest (RF), SVM, and
CNN models. The experiments were conducted on both the
mammogram and chest radiograph datasets.
The CNN and Bi-CNN models are implemented in TensorFlow [37] and the RF and SVM models are implemented
using the classifiers in scikit-learn [38]. The experiments are
conducted on a DevBox with an Intel Core i7-5930K 6 Core
3.5GHz desktop processor, 64 GB DDR4 RAM, and two
TITAN X GPUs with 12GB of memory per GPU.

(k)

Fig. 6: Example chest radiographs for each categories. a) Normal;
b) Cardiomegaly; c) Consolidation; d) Pulmonary Edema; e) Lung
Nodules; f) Lung Mass; g) Pleural Effusion; h) Widened Mediastinum; i) Vertebral Fractures; j) Clavicular Fracture; k) Pneumothorax.

V. T HE DATASETS
Our institutional review board approved this single-center
retrospective study with a waiver for informed consent. The
Research Ethics Boards (REB) number is 17-167. A search
of our Radiology Information System (RIS) (Syngo; Siemens

A. Parameters Setting
Unless stated, the CNN and Bi-CNN models are trained
with a mini-batch size of 64, drop-out probability of 0.5, filter
sizes {3, 4, 5} and 120 feature maps per filter size. The number
of training iterations is set to 50 and the initial learning rate
for Adam optimizer is set to 0.001 [34]. An exponential decay
adaptive learning rate is applied. The weights at output layers
are initialized using the Xavier method [39], the weights in the
convolutional layer are selected based on normal distribution
with standard deviation of 0.1, and biases are set to 0.1. The
activation function before the max-pooling layer is ReLU [40].

6

TABLE II: A general consensus for the breast density descriptors for five categories of breast density.
Category
Almost entirely fatty
Scattered areas of fibroglandular density
Heterogeneously dense
Extremely dense
Ambiguous

Label
0
1
2
3
4

Count
342
1546
1510
436
246

Breast Density
< 25%
25 − 50%
51 − 75%
> 75%
-

Example Breast Density Descriptors
fat; fatty; includes variations of mainly fatty; predominantly fatty; predominantly fat
scattered
heterogenous; heterogeneously
dense; extremely dense; very dense
mild dense; mildly dense; overlap in describing categories with label 1 and 2

TABLE III: A sample of radiology reports with associated categories for mammogram reports dataset (MRD).
Category
Almost entirely fatty
Scattered areas of fibroglandular density
Heterogeneously dense
Extremely dense

The
The
The
The

Example from MRD
breasts are almost entirely fatty bilaterally. No suspicious calcification, dominant mass or architectural distortion. No interval change.
breasts show scattered fibroglandular densities. There are no dominant nodules or suspicious calcification seen in either breast.
breast parenchyma is heterogeneously dense. No suspicious masses, calcification or architectural distortion seen on either side.
breast parenchyma is extremely dense. No suspicious masses, calcifications or architectural distortion seen on either side.

TABLE IV: A general consensus for the chest radiographs descriptors for 11 categories.
Category
Normal
Cardiomegaly
Consolidation
Pulmonary Edema
Lung Nodules
Lung Mass
Pleural Effusion
Widened Mediastinum
Vertebral Fractures
Clavicular Fracture
Pneumothorax

Label
0
1
2
3
4
5
6
7
8
9
10

Count
116
106
81
104
118
89
144
50
67
73
82

Example Chest Radiograph Descriptors
No acute findings; unremarkable study; the cardiopericardial silhouette and hilar anatomy is within normal limits
Mild cardiomegaly/pericardial effusion; enlarged cardiac silhouette; cardiopericardial silhouette is enlarged
Resolving consolidation in the lower lungs bilaterally, with overall improved aeration in the lower lungs
Mild to moderate interstitial pulmonary edema; airspace edema; right pulmonary edema
Calcified granulomas present; nodules have developed; multiple faint bilateral pulmonary nodules
Bilateral pulmonary masses; mass in the right upper lobe; middle mediastinal mass
Persistent bilateral pleural effusions; bilateral pleural effusions; persistent loculated right pleural effusion
Widening of the superior mediastinum; mediastinum appears widened; mediastinum is slightly widened
Vertebral fracture; several old vertebral compression injuries; thoracolumbar vertebral compression injuries
Fracture deformity of the right lateral clavicle; right lateral clavicle fracture
Partial right upper lobe collapse; chronic collapse of the right middle lobe; right apical pneumothorax

TABLE V: A sample of radiology reports with associated categories for chest radiographs dataset (CRRD).
Category
Normal
Cardiomegaly
Consolidation
Pulmonary Edema
Lung Nodules
Lung Mass
Pleural Effusion
Widened Mediastinum
Vertebral Fractures
Clavicular Fracture
Pneumothorax

Example CRRD
Cardiomediastinal contours are within normal limits. The hila are unremarkable. The lungs are clear.
The cardiopericardial silhouette is enlarged with LV prominence. Aortic valve prosthesis in situ. There is unfolding of the aorta with calcification.
The lungs remain overinflated and show mild chronic parenchymal changes. Inhomogeneous airspace consolidation has developed
in the basal segments of the left lower lobe.
Significant fluid has developed in the right minor fissure. Pulmonary venous markings are severely elevated.
The left pulmonary nodular density superimposed on the posterior seventh rib is smaller.
There is mild stable bilateral upper lobe pleuropulmonary scarring and a stable right upper lobe nodule.
Left upper lobe perihilar mass. Prominence of the left hila as well as the left superior mediastinal margins.
Midline sternotomy wires appear stable in position. Small bilateral pleural effusions unchanged from previous exam.
Superior mediastinal widening and vascular engorgement relate to patient positioning. The lungs are clear.
The cardiac silhouette is normal in size and shape with aortic unfolding. The lungs are overinflated but clear.
The pleural spaces, mediastinum and diaphragm appear normal.
Degenerative changes in the spine and an old right mid clavicular fracture.
Right-sided pleural drain in situ. There is very subtle residual right pneumothorax with lateral pleural edge displaced.

(a) λ = 0.1.

(b) λ = 0.01.

(c) λ = 0.001.

Fig. 7: Accuracy (in %) and convergence behaviour of CNN and Bi-CNN models for the MRD and CRRD for different values of learning rate λ. The x-axis
represents the training iteration. The plots are from cross-validation over 30 independent experiments.

7

The L2 regularization is set to 1.0 × 10−4 and early-stopping
is applied.
For all the experiments, 70%, 15%, and 15% of data is
allocated for training, validation, and test, respectively. The
data is shuffled before splitting. In each experiment, the model
is cross-validated over 30 independent experiments and the
results after statistical testing are reported.

TABLE VI: Classification accuracy (in %) for MRD and CRRD using RF,
SVM, CNN, Bi-CNN. The best results are in boldface.
Model
RF (1-gram)
RF (2-gram)
RF (3-gram)
SVM (sigmoid)
SVM (poly)
CNN (1-kernel)
CNN (2-kernel)
CNN (3-kernel)
Bi-CNN (1-kernel)
Bi-CNN (2-kernel)
Bi-CNN (3-kernel)

B. Performance Evaluation and Analysis
We experimented with various configurations of the RF [41],
SVM [42], and CNN [24] models.
• RF: Instead of using a bag-of-the-words technique, which
uses the frequency of the words in a query as the features,
a n-gram model with a lower boundary of 1 and upper
boundary n ∈ {1, 2, 3} performs the feature extraction
of reports. We use an implementation of RF [41] which
combines classifiers by averaging their probabilistic predictions. The number of estimators is set to 10.
• SVM: Similar to RF, the SVM model uses n-gram feature
extraction. The SVM model is deployed for two different
kernels, a “sigmoid” and a “polynomial” with degree
three.
• CNN: The CNN implementation proposed in [24] for text
classification and later used in [8] for the classification
of radiology head CT reports is used. This model has
a single input channel and the order of input words
is similar to the order in the report. The model has a
convolution layer followed by a max-pooling layer and
a “softmax” classifier. The studies are for kernel sizes
k ∈ {1, 2, 3}.
• Bi-CNN: The proposed Bi-CNN with two input channels.
The settings are similar to CNN.
The experiments are for breast density classification and
chest radiograph classifications based on chest pathology using
the MRD and CRRD, respectively. The results of performance comparisons between the RF, SVM, CNN, and BiCNN models are presented in Table VI. As n increases in
n-gram for RF, it considers the dependency between a greater
number of words (i.e. n). The RF with 3-gram model has
better performance than the 2-gram and 1-gram RF models.
The SVM with polynomial of degree three (i.e., “poly”) and
“sigmoid” kernels almost have the same performance. The
number of kernels in the CNN models have a minor impact
on accuracy. However, the CNN models with 3-kernel have
better performance than models with 1-kernel and 2-kernel.
The proposed Bi-CNN with 3-kernel has the best performance
compared to the other models.
The convergence behaviour of CNN and Bi-CNN over
training iterations for the validation dataset is presented in
Figure 7. For a large learning rate value (i.e., λ = 0.1),
the models converge rapidly to a local solution. However, for
smaller learning rates, the models converge more slowly but
with greater stability towards a better solution. As the results
in Table VII show, the CNN model converges with a lower accuracy for three different learning rates λ ∈ {0.1, 0.01, 0.001}.
For λ = 0.001, the best performance is achieved by Bi-CNN
for both MRD and CRRD. One of the advantages of Bi-CNN is

MRD
69.73
73.76
80.53
82.74
82.97
85.72
86.52
86.91
89.60
90.88
92.94

CRRD
67.32
69.27
77.59
80.24
81.73
85.27
86.01
86.05
89.16
89.91
91.34

TABLE VII: Accuracy (Acc. in %) and convergence iteration (C) of CNN
and Bi-CNN models for learning rate λ on MRD and CRRD datasets. Best
results are in boldface.
Model
λ
0.1
0.01
0.001

CNN
MRD
CRRD
Acc.
C
Acc.
C
82.54
36
76.42
43
83.27
100
82.78
100
86.91
100
86.05
100

Bi-CNN
MRD
CRRD
Acc.
C
Acc.
C
90.03
23
84.04
26
89.98
100
87.85
100
92.94
100
91.34
100

adding diversity into the model through the integration of two
different representations of feature vectors into the model. This
diversity helps the CNN avoid convergence to local solutions
with low accuracy.
VII. C ONCLUSION
Radiology reports contain a radiologist’s interpretation of a
medical imaging examination. Vast numbers of such reports
are digitally stored in health care facilities. Text mining and
knowledge extraction from such data may potentially be used
for double checking radiologist interpretations as part of an
audit system. Furthermore, these methods can be used to audit
utilization of limited health care resources and to enhance the
study of patient treatment plans over time.
In this paper, we have proposed a Bi-CNN for the interpretation of radiology reports. We have collected and anonymized
two datasets for our experiments: a mammogram reports
dataset and a chest radiograph reports dataset. The Bi-CNN
has two input channels where one channel represents the
features of the report (including zero-padding) and the other
channel represents the reverse non-padded features of the
report. The combination of forward and reverse order of feature vectors represents more information about dependencies
between feature vectors. Our comparative studies demonstrate
that the Bi-CNN model outperforms the CNN, RF, and SVM
methods.
ACKNOWLEDGEMENT
The authors thank the support of NVIDIA Corporation with
the donation of the Titan X GPUs used for this project.
R EFERENCES
[1] S. S. Naik, A. Hanbidge, and S. R. Wilson, “Radiology reports: examining radiologist and clinician preferences regarding style and content,”
American Journal of Roentgenology, vol. 176, no. 3, pp. 591–598, 2001.
[2] A. Pinto and L. Brunese, “Spectrum of diagnostic errors in radiology,”
World Journal of Radiology, vol. 2, no. 10, pp. 377–383, 2010.

8

[3] M. Cicero, A. Bilbily, E. Colak, T. Dowdell, B. Gray, K. Perampaladas,
and J. Barfett, “Training and validating a deep convolutional neural
network for computer-aided detection and classification of abnormalities
on frontal chest radiographs,” Investigative radiology, vol. 52, no. 5, pp.
281–287, 2017.
[4] J. Wang, H. Ding, F. Azamian, B. Zhou, C. Iribarren, S. Molloi, and
P. Baldi, “Detecting cardiovascular disease from mammograms with
deep learning,” IEEE transactions on medical imaging, 2017.
[5] M. Havaei, A. Davy, D. Warde-Farley, A. Biard, A. Courville, Y. Bengio,
C. Pal, P.-M. Jodoin, and H. Larochelle, “Brain tumor segmentation with
deep neural networks,” Medical image analysis, vol. 35, pp. 18–31, 2017.
[6] D. B. Aronow, F. Fangfang, and W. B. Croft, “Ad hoc classification
of radiology reports,” Journal of the American Medical Informatics
Association, vol. 6, no. 5, pp. 393–411, 1999.
[7] A. Wilcox, “Automated classification of medical text reports,” Ph.D.
dissertation, Columbia University New York, 2000.
[8] B. Shin, F. H. Chokshi, T. Lee, and J. D. Choi, “Classification of
radiology reports using neural attention models,” in Neural Networks
(IJCNN), 2017 International Joint Conference on. IEEE, 2017, pp.
4363–4370.
[9] T. Gong, C. L. Tan, T. Y. Leong, C. K. Lee, B. C. Pang, C. T. Lim,
Q. Tian, S. Tang, and Z. Zhang, “Text mining in radiology reports,” in
Data Mining, 2008. ICDM’08. Eighth IEEE International Conference
on. IEEE, 2008, pp. 815–820.
[10] R. K. Taira, V. Bashyam, and H. Kangarloo, “A field theoretical
approach to medical natural language processing,” IEEE Transactions
on Information Technology in Biomedicine, vol. 11, no. 4, pp. 364–375,
2007.
[11] G. F. Cooper, B. G. Buchanan, M. Kayaalp, M. Saul, and J. K. Vries,
“Using computer modeling to help identify patient subgroups in clinical
data repositories.” in Proceedings of the AMIA Symposium. American
Medical Informatics Association, 1998, p. 180.
[12] W. W. Chapman, G. F. Cooper, P. Hanbury, B. E. Chapman, L. H.
Harrison, and M. M. Wagner, “Creating a text classifier to detect
radiology reports describing mediastinal findings associated with inhalational anthrax and other disorders,” Journal of the American Medical
Informatics Association, vol. 10, no. 5, pp. 494–503, 2003.
[13] P. A. Dang, M. K. Kalra, M. A. Blake, T. J. Schultz, M. Stout, E. F.
Halpern, and K. J. Dreyer, “Use of radcube for extraction of finding
trends in a large radiology practice,” Journal of digital imaging, vol. 22,
no. 6, p. 629, 2009.
[14] G. Zuccon, A. S. Wagholikar, A. N. Nguyen, L. Butt, K. Chu, S. Martin, and J. Greenslade, “Automatic classification of free-text radiology
reports to identify limb fractures using machine learning and the snomed
ct ontology,” AMIA Summits on Translational Science Proceedings, vol.
2013, p. 300, 2013.
[15] H.-C. Shin, L. Lu, L. Kim, A. Seff, J. Yao, and R. M. Summers,
“Interleaved text/image deep mining on a very large-scale radiology
database,” in Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, 2015, pp. 1090–1099.
[16] H.-C. Shin, K. Roberts, L. Lu, D. Demner-Fushman, J. Yao, and R. M.
Summers, “Learning to read chest x-rays: recurrent neural cascade model
for automated image annotation,” in Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, 2016, pp. 2497–2506.
[17] H. Salehinejad, “Learning over long time lags,” arXiv preprint
arXiv:1602.04335, 2016.
[18] S. Lai, L. Xu, K. Liu, and J. Zhao, “Recurrent convolutional neural
networks for text classification.” in AAAI, vol. 333, 2015, pp. 2267–
2273.
[19] H.-S. Oh, J.-B. Kim, and S.-H. Myaeng, “Extracting targets and attributes of medical findings from radiology reports in a mixture of languages,” in Proceedings of the 2nd ACM Conference on Bioinformatics,
Computational Biology and Biomedicine. ACM, 2011, pp. 550–552.
[20] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean,
“Distributed representations of words and phrases and their compositionality,” in Advances in neural information processing systems, 2013,
pp. 3111–3119.
[21] J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors
for word representation.” in EMNLP, vol. 14, 2014, pp. 1532–1543.
[22] N. Majumder, S. Poria, A. Gelbukh, and E. Cambria, “Deep learningbased document modeling for personality detection from text,” IEEE
Intelligent Systems, vol. 32, no. 2, pp. 74–79, 2017.
[23] N. Kalchbrenner, E. Grefenstette, and P. Blunsom, “A convolutional neural network for modelling sentences,” arXiv preprint arXiv:1404.2188,
2014.
[24] Y. Kim, “Convolutional neural networks for sentence classification,”
arXiv preprint arXiv:1408.5882, 2014.

[25] Y. Ren and D. Ji, “Neural networks for deceptive opinion spam detection:
An empirical study,” Information Sciences, vol. 385, pp. 213–224, 2017.
[26] D. Wang and E. Nyberg, “A long short-term memory model for answer
sentence selection in question answering,” in ACL, 2015.
[27] D. H. Hubel and T. N. Wiesel, “Receptive fields and functional architecture of monkey striate cortex,” The Journal of physiology, vol. 195,
no. 1, pp. 215–243, 1968.
[28] Y. LeCun, Y. Bengio et al., “Convolutional networks for images, speech,
and time series,” The handbook of brain theory and neural networks,
vol. 3361, no. 10, p. 1995, 1995.
[29] Y.-L. Boureau, J. Ponce, and Y. LeCun, “A theoretical analysis of feature
pooling in visual recognition,” in Proceedings of the 27th international
conference on machine learning (ICML-10), 2010, pp. 111–118.
[30] R. Weber, H.-J. Schek, and S. Blott, “A quantitative analysis and
performance study for similarity-search methods in high-dimensional
spaces,” in VLDB, vol. 98, 1998, pp. 194–205.
[31] P. Domingos, “A few useful things to know about machine learning,”
Communications of the ACM, vol. 55, no. 10, pp. 78–87, 2012.
[32] H. Salehinejad, J. Barfett, P. Aarabi, S. Valaee, E. Colak, B. Gray, and
T. Dowdell, “A convolutional neural network for search term detection-a
draft,” arXiv preprint arXiv:1708.02238, 2017.
[33] R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and
P. Kuksa, “Natural language processing (almost) from scratch,” Journal
of Machine Learning Research, vol. 12, no. Aug, pp. 2493–2537, 2011.
[34] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
arXiv preprint arXiv:1412.6980, 2014.
[35] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and
R. Salakhutdinov, “Dropout: a simple way to prevent neural networks
from overfitting.” Journal of Machine Learning Research, vol. 15, no. 1,
pp. 1929–1958, 2014.
[36] R. TACo, “American college of radiology breast imaging reporting and
data system bi-rads,” Reston, VA, 1998.
[37] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S.
Corrado, A. Davis, J. Dean, M. Devin et al., “Tensorflow: Large-scale
machine learning on heterogeneous distributed systems,” arXiv preprint
arXiv:1603.04467, 2016.
[38] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion,
O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg et al.,
“Scikit-learn: Machine learning in python,” Journal of Machine Learning
Research, vol. 12, no. Oct, pp. 2825–2830, 2011.
[39] X. Glorot and Y. Bengio, “Understanding the difficulty of training deep
feedforward neural networks.” in Aistats, vol. 9, 2010, pp. 249–256.
[40] V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in Proceedings of the 27th international conference on
machine learning (ICML-10), 2010, pp. 807–814.
[41] L. Breiman, “Random forests,” Machine learning, vol. 45, no. 1, pp.
5–32, 2001.
[42] M. A. Hearst, S. T. Dumais, E. Osuna, J. Platt, and B. Scholkopf, “Support vector machines,” IEEE Intelligent Systems and their applications,
vol. 13, no. 4, pp. 18–28, 1998.

