Detecting and classifying lesions in mammograms with Deep Learning
Dezső Ribli Anna Horváth Zsuzsa Unger Péter Pollner István Csabai
cs.CV


Gaussian Processes for Individualized Continuous Treatment Rule Estimation
    Pavel Shvechikov 

 and Evgeniy Riabenko The reported study was funded by RFBR, according to the research project No. 16-07-01192 А.

 Department of Computer Science, Higher School of Economics
    December 30, 2023
=====================================================================================================================================================================================================



empty



§ INTRODUCTION






 §.§ Screening mammography



Breast cancer is the most common cancer in women and it is the main cause of death from cancer among women in the world <cit.>.
Screening mammography was shown to reduce breast cancer mortality by 38–48% among participants <cit.>.
In the EU 25 of the 28 member states are planning, piloting or implementing screening programs to diagnose and treat breast cancer in an early stage <cit.>.
During a standard mammographic screening examination, X-ray images are captured from 2 angles of each breast.
These images are inspected for malignant lesions by one or two experienced radiologists.
Suspicious cases are called back for further diagnostic evaluation.


Screening mammograms are evaluated by human readers.
The reading process is monotonous, tiring, lengthy, costly and most importantly prone to errors.

Multiple studies have shown that 20-30% of the diagnosed cancers could be found retrospectively on the previous negative screening exam by blinded reviewers
<cit.>.
The problem of missed cancers still persists despite modern full field digital mammography (FFDM)
<cit.>.

The sensitivity  and specificity of screening  mammography is reported to be between 77-87% and 89-97% respectively.
These metrics describe the average performance of readers, and there is substantial variance in the performance of individual physicians, with reported false positive rates between 1-29%, and sensitivities between 29-97%
<cit.>.

Double reading was found to improve the performance of mammographic evaluation and it had been implemented in many countries <cit.>.
Multiple reading can further improve diagnostic performance up to more than 10 readers, proving that there is room for improvement in mammogram evaluation beyond double reading
<cit.>.





 §.§ Computer-aided detection in mammographic screening



Computer-aided detection (CAD) solutions were developed to help radiologists in reading mammograms.
These programs usually analyze a mammogram and mark the suspicious regions, which should be reviewed by the radiologist <cit.>.
The technology was approved by FDA and had spread quickly. 
By 2008, in the US, 74% of all screening mammograms in the Medicare population were interpreted with CAD, the cost of CAD usage is over $400 million a year.
<cit.>.


The benefits of using CAD are controversial.
Initially several studies have shown promising results with CAD
<cit.>.
A large clinical trial in the United Kingdom has shown that single reading with CAD assistance has similar performance to double reading.
<cit.>
However, in the last decade multiple studies concluded that currently used CAD technologies do not improve the performance of radiologists in everyday practice in the United States.
<cit.>.
These controversial results indicate that CAD systems need to be improved before radiologists can ultimately benefit from using the technology in everyday practice.


Currently used CAD approaches are based on describing the X-ray image with meticulously designed hand crafted features, and machine learning for classification on top of these features
<cit.> .
In the field of computer vision, since 2012, deep convolutional neural networks (CNN) have significantly outperformed these traditional methods<cit.>.
Deep CNN-s have reached or even surpassed human performance in image classification and object detection.
<cit.>.
These models have tremendous potential in medical image analysis.

Several studies have attempted to apply Deep Learning to analyze mammograms <cit.>, but the problem is still far from being solved.





 §.§ The Digital Mammography DREAM Challenge


The Digital Mammography DREAM Challenge (DM challenge) <cit.> asked participants to write algorithms which can predict whether a breast in a screening mammography exam will be diagnosed with cancer.
The dataset consisted of 86000 exams, with no pixel level annotation, only a binary label indicating if breast cancer was diagnosed in the next 12 months after the exam.
Each side of the breasts were treated as a separate case that we will call breast-level prediction in this paper.
The participants had to upload their programs to a secure cloud platform, and they were not able to download or view the images, neither interact with their program during training or testing.
The DM challenge provided an excellent opportunity to compare the performance of competing methods in a controlled and fair way instead of self-reported evaluations on different or proprietary datasets.




§ MATERIAL AND METHODS




 §.§ Data



Mammograms with pixel level annotations were needed to train a lesion detector and test the classification and localization performance.
We have trained the model on the public Digital Database for Screening Mammography  (DDSM) <cit.> and a dataset from the Semmelweis University in Budapest, and tested it on the public INbreast <cit.> dataset.

The images used for training contain either histologically proven cancers or benign lesions which were recalled for further examinations, but later turned out to be nonmalignant.
We expect that training with both kinds of lesions helps our model to find more lesions of interest, and differentiate between malignant and benign examples.


The DDSM  dataset contains 2620 digitized film-screen screening mammography exams, with pixel-level ground truth annotation of lesions. 
Cancerous lesions have histological proof.
We have only used the DDSM database for training our model and not evaluating it.
The quality of digitized film-screen mammograms is not as good as full field digital mammograms therefore, evaluation on these cases is not relevant.
We have converted the lossless jpeg images to png format, mapped the pixel values to optical density using the calibration functions from the DDSM website, and rescaled the pixel values to the 0-255 range.


The dataset from the Department of Radiology at the Semmelweis University in Budapest, Hungary contains 847 FFDM images of 214 exams from 174 patients, recorded with a Hologic LORAD Selenia device.
Institutional board approval was obtained for the dataset.
This dataset was not available for the full period of the DM challenge, it is used only for improvement in the second stage of the DM challenge, after the pixel level annotation by the authors.


The INbreast dataset contains 115 FFDM cases with pixel-level ground truth annotations, and histological proof for cancers
<cit.>.
We have adapted the INbreast pixel level annotations to suit our testing scenario. 
We have ignored all benign annotations, and converted the malignant lesion annotations to bounding boxes.
We have excluded 8 exams which had other findings, artifacts, previous surgeries or ambiguous pathological outcome.
The images have low contrast therefore, we have adjusted the window of the pixel levels.
The pixel values were clipped to be minimum 500 pixel lower and maximum 800 pixels higher than the mode of the pixel value distribution (excluding the background) and were rescaled to the 0-255 range.



 §.§ Data Availability


The DDSM dataset is available online at <http://marathon.csee.usf.edu/Mammography/Database.html>.

The INBreast dataset can be requested online at <http://medicalresearch.inescporto.pt/breastresearch/index.php/Get_INbreast_Database>.

For the dataset from the Semmelweis university (http://semmelweis.hu/radiologia/) restrictions apply to the availability of these data, which were used under special licence from Semmelweis University, and so are not publicly available. Data are however available from the authors upon reasonable request and permission of the Semmelweis University.



 §.§ Methods



The heart of our model is a state of the art object detection framework, Faster R-CNN  <cit.>.
Faster R-CNN is based on a convolutional neural network with additional components for detecting, localizing and classifying objects in an image.

Faster R-CNN has a branch of convolutional layers, called Region Proposal Network (RPN), after the last convolutional layer of the original network, which is trained to detect and localize objects on the image, regardless of the class of the object. 
There are default detection boxes with different sizes and aspect ratios in order to find objects with varying sizes and shapes.
The highest scoring default boxes are called the region proposals for the other branch of the network.

The other branch of the neural network evaluates the signal coming from each proposed region of the last convolutional layer, resampled to a fix size.

Both branches try to solve a classification task to detect the presence of objects and a bounding-box regression task in order to refine the boundaries of the object present in the region.

From the detected overlapping objects, the best predictions are selected using non-maximum suppression.
Further details about Faster R-CNN can be found in the original article <cit.>.
An outline of the model can be seen in Fig.  <ref>.





The base CNN used in our model was a VGG16 network, which is 16 layer deep CNN <cit.>.

The final layer can detect 2 kinds of objects in the images, benign lesions or malignant lesions.
The model outputs a bounding box for each detected lesion, and a score, which reflects the confidence in the class of the lesion.

To describe an image with one score, we take the maximum of the scores of all malignant lesions detected in the image.
For multiple images of the same breast, we take the average of the scores of individual images.

For the DM challenge we have trained 2 models using shuffled training datasets.
When ensembling these models, the score of an image was the average score of the individual models.
This approach has been motivated by a previous study on independent human readers, and it has proven reasonably effective, while simple and flexible <cit.>.



We have used the framework developed by the authors of Faster R-CNN <cit.>, which was built in the Caffe framework for deep learning <cit.>.
During training we optimized the object detection part and the classifier part of the model in the same time, this is called the joint optimization <cit.>.
We used backpropagation and stochastic gradient descent with weight decay.
The initial model used for training has been pretrained on 1.2 million images from the ImageNet dataset <cit.>.


We have found that higher resolution yields better results, therefore the mammograms were rescaled isotropically to longer side smaller than 2100 pixels or shorter side smaller than 1700 pixels.
This resolution is close to the maximum size which fits in the memory of the graphics card used.
The aspect ratio was selected to fit the regular aspect ratio of Hologic images.

We applied vertical and horizontal flipping to augment the training dataset.

Mammograms contain fewer object than ordinary images, therefore negative proposals dominate minibatches.
The Intersection over Union (IoU) threshold for foreground objects in the region proposal network was relaxed from 0.7 to 0.5 to allow more positive samples in each minibatch.
Relaxation of positive examples is also supported by the fact that lesions on a mammogram have much less well defined boundaries than a car or a dog on a tradiotional image.

The IoU threshold of the final non maximum suppression (nms) was set to 0.1, because mammograms represent a smaller and compressed 3D space compared to ordinary images, therefore overlapping detections are expected to happen less often than in usual object detection. 
The model was trained for 40k iterations, this number was previously found to be close to optimal by testing multiple models on the DM challenge training data.
The model was trained and evaluated on an Nvidia GTX 1080Ti graphics card.

Our final entry in the DM challenge was an ensemble of 2 models.



§ RESULTS





 §.§ Cancer classification



We also evaluated the model's performance on the public INbreast dataset with the receiver operating characteristics (ROC) metric, Fig. <ref>.
The INbreast dataset has many exams with only one laterality, therefore we have evaluated predictions for each breast.
The system achieved AUC =0.95, (95 percentile interval: 0.91 to 0.98, estimated from 10000 bootstrap samples). 
To our best knowledge this is the highest AUC score reported on the INbreast dataset with a fully automated system based on a single model.







 §.§ FROC analysis



In order to test the model's ability to detect and accurately localize malignant lesions, we evaluated the predictions on the INbreast dataset using the Free-response ROC (FROC) curve <cit.>.
The FROC curve shows the sensitivity (fraction of correctly localized lesions) as a function of the number of false positive marks put on an image Fig. <ref>.


A detection was considered correct if the center of the proposed lesion fell inside a ground truth box.
The same criteria is generally used when measuring the performance of currently used CAD products <cit.>.
The DM challenge dataset has no lesion annotation, therefore we can not use it for an FROC analysis.







 §.§ Examples



To demonstrate the characteristics and errors of the detector, we have created a collection of correctly classified, false positive and missed malignant lesions of the INbreast dataset, see in Fig. <ref>.
The score threshold for the examples was defined at sensitivity =0.9 and 0.3 false positive marks per image.



After inspecting the false positive detections, we have found that most were benign masses or calcifications.
Some of these benign lesions were biopsy tested according to the case descriptions of the INbreast dataset.

While 10% of the ground truth malignant lesions were missed at this detection threshold, these were not completely overlooked by the model.
With a score threshold which corresponds to 3 false positive marks per image, all the lesions were correctly detected (see Fig. <ref>).
Note that the exact number of false positive and true positive detections slightly varies with with different samples of images, indicated by the area in
Fig. <ref>.







§ DISCUSSION



We have proposed a Faster R-CNN based CAD approach, which achieved the 2nd position in the Digital Mammography DREAM Challenge with an AUC = 0.85 score on the final validation dataset.
The competition results have proved that the method described in this article is one of the best approach for cancer classification in mammograms.

Our method was the only one of the top contenders in the DM challenge which is based on the detection of malignant lesions, and whole image classification is just a trivial step from the detection task.
We think that a lesion detector is clinically much more useful than a simple classifier.
A classifier only gives a single score for a case or breast, but it is not able to locate the cancer which is essential for further diagnostic tests or treatment.


We have evaluated the model on the publicly available INbreast dataset.
The system is able to detect 90% of the malignant lesions in the INbreast dataset with only 0.3 false positive marks per image.
It also sets the state of the art performance in cancer classification on the publicly available INbreast dataset.
The system uses the mammograms as the only input without any annotation or user interaction.


An object detection framework developed to detect objects in ordinary images shows excellent performance.
This result indicates that lesion detection on mammograms is not very different from the regular object detection task. 
Therefore the expensive, traditional CAD solutions, which have controversial efficiency, could be replaced with the recently developed, potentially more accurate, deep learning based, open source object detection methods in the near future.
The FROC analysis results suggest that the proposed model could be applied as a perception enhancer tool, which could help radiologists to detect more cancers.


A limitation of our study comes from the small size of the publicly available pixel-level annotated dataset.
While the classification performance of the model has been evaluated on a large screening dataset, the detection  performance could only be evaluated on the small INbreast dataset.






§ AUTHOR CONTRIBUTIONS STATEMENT


D.R., I.C. and P.P. contributed to the conception and design of the study.
A.H. and Z.U. contributed to the acquisition, analysis and interpretation of data.
All authors reviewed the manuscript. 




§ ADDITIONAL INFORMATION



 §.§ Competing financial interests

The authors declare no competing financial interests.



§ ACKNOWLEDGEMENTS


This work was supported by the Novo Nordisk Foundation Interdisciplinary Synergy Programme [Grant NNF15OC0016584]; and  National Research, Development and Innovation Fund of Hungary, [Project no. FIEK_16-1-2016-0005].
The funding sources cover computational and publishing costs.
