DeepPap: Deep Convolutional Networks for Cervical Cell Classification
Ling Zhang Le Lu Isabella Nogues Ronald M. Summers Shaoxiong Liu Jianhua Yao
cs.CV















DeepPap: Deep Convolutional Networks for Cervical Cell Classification
    Ling Zhang,
        Le Lu, Senior Member, IEEE,
				Isabella Nogues,
				Ronald M. Summers,
				Shaoxiong Liu,
        and Jianhua Yao
L. Zhang, L. Lu, R. Summers, and J. Yao are with the Imaging Biomarkers and Computer-Aided Diagnosis Laboratory and also with the Clinical Image Processing Service, Radiology and Imaging Sciences Department, National Institutes of Health Clinical Center, Bethesda, MD 20892 USA e-mail: (ling.zhang3@nih.gov; jyao@nih.gov).
I. Nogues is with the Imaging Biomarkers and Computer-Aided Diagnosis Laboratory, Radiology and Imaging Sciences Department, National Institutes of Health Clinical Center, Bethesda, MD 20892 USA.
S. Liu is with the Department of Pathology, People’s Hospital of Nanshan District, Shenzhen 518052 China.


    
=============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================






Automation-assisted cervical screening via Pap smear or liquid-based cytology (LBC) is a highly effective cell imaging based cancer detection tool, where cells are partitioned into ”abnormal” and ”normal” categories. However, the success of most traditional classification methods relies on the presence of accurate cell segmentations. Despite sixty years of research in this field, accurate segmentation remains a challenge in the presence of cell clusters and pathologies. Moreover, previous classification methods are only built upon the extraction of hand-crafted features, such as morphology and texture. This paper addresses these limitations by proposing a method to directly classify cervical cells – without prior segmentation – based on deep features, using convolutional neural networks (ConvNets). First, the ConvNet is pre-trained on a natural image dataset. It is subsequently fine-tuned on a cervical cell dataset consisting of adaptively re-sampled image patches coarsely centered on the nuclei. In the testing phase, aggregation is used to average the prediction scores of a similar set of image patches. The proposed method is evaluated on both Pap smear and LBC datasets. Results show that our method outperforms previous algorithms in classification accuracy (98.3%), area under the curve (AUC) (0.99) values, and especially specificity (98.3%), when applied to the Herlev benchmark Pap smear dataset and evaluated using five-fold cross-validation. Similar superior performances are also achieved on the HEMLBC (H&E stained manual LBC) dataset. Our method is promising for the development of automation-assisted reading systems in primary cervical screening.




Cell classification, Deep learning, Neural networks, Pap smear, Cervical cytology.









§ INTRODUCTION


Cervical cytology (conventional Pap smear or liquid-based cytology) <cit.>, the most popular screening test for prevention and early detection of cervical cancer, has been widely used in developed countries and has significantly reduced its incidence and number of deaths <cit.>. However, population-wide screening is still unavailable in underdeveloped countries <cit.>, partly due to the complexity and tedious nature of manually screening abnormal cells from a cervical cytology specimen <cit.>. While automation-assisted reading techniques can boost efficiency, their current performance is not adequate for inclusion in primary cervical screening <cit.>. 

During the past few decades, extensive research has been devoted to the creation of  computer-assisted reading systems based on automated image analysis methods <cit.>. Such systems automatically select potentially abnormal cells in a given cervical cytology specimen, from which the cytoscreener/cytopathologist completes the classification. This task comprises three steps: cell (cytoplasm and nuclei) segmentation, feature extraction/selection, and cell classification.

Accurate cell segmentation is crucial to the success of a reading system. However, despite recent significant progress in this area <cit.>, the presence of cell clusters (which is even more problematic in Pap smear than in liquid-based cytology), as well as the large shape and appearance variations between abnormal and normal nuclei, remains a major obstacle to the accurate segmentation of individual cytoplasms and nuclei. On the Herlev benchmark dataset <cit.>, the attained nucleus segmentation accuracy ranging between 0.85 <cit.> and 0.92 <cit.>. 
On an overlapping cervical cell dataset <cit.>, the cytoplasm segmentation accuracy ranges from 0.87 to 0.89 <cit.>. On the other hand, most cell classification studies assume that accurate segmentations of individual cytoplasms and nuclei are already available <cit.>. By optimizing features derived from the segmented cytoplasm and nucleus, high classification accuracies (e.g., 96.8%) are achieved on the Herlev dataset, using 5-fold cross validation (CV)  <cit.>. However, these high values would decrease, once the automated segmentation error (deriving mainly from the error-prone abnormal nucleus segmentation <cit.>), were taken into account. 

Several strategies to remove this dependence on segmentation have been investigated. Classification based only on nucleus features (excluding cytoplasm features) is proposed <cit.>. Comparable results on the Herlev dataset is obtained by using a non-linear dimensionality reduction algorithm and supervised learning <cit.>. Another idea is to classify image patches containing full cervical cells <cit.>. However, extraction of such patches still requires automated cell detection and segmentation. To avoid the pre-segmentation step, pixel-level classification method is designed to directly screen abnormal nuclei with no prior cytoplasm and nucleus segmentation <cit.>, but reports limited validation results. Alternatively, a technique which classifies the cropped blocks from cell images is proposed <cit.>. However, arbitrary cropping could potentially separate a full cell into distinct patches.

Current cervical screening systems are hindered by limitations in the feature design and selection components. At present, extracted features fall under the following categories: handcrafted features describing morphology and chromatin characteristics <cit.> in accordance with “The Bethesda System (TBS)" rules <cit.>, engineered features representing texture distribution <cit.> according to previous computer-aided-diagnosis experiences, or both combined <cit.>. The resulting features are then organized, using a feature selection or dimensionality reduction algorithm, for classification. Handcrafted features are compromised by the current limited understanding of cervical cytology. Engineered features are obtained in an unsupervised manner, and thus often encode redundant information. The feature selection process potentially ignores significant clues and removes complementary information. Moreover, considering that the detection of some abnormal cervical cells is challenging even for human experts <cit.>, the hand-crafted features used in previous studies may not be able to represent complex discriminative information.  In fact, information describing cell abnormality may potentially lie in latent higher level features of cervical cell images, but this has not yet been investigated.

Representation learning refers to a set of methods designed to automatically learn and discover intricate discriminative information from raw data <cit.>. Recently, representation learning has been popularized by deep learning methods <cit.>. In particular, deep convolutional neural networks (ConvNets) <cit.> have achieved unprecedented results in the 2012 ImageNet Large Scale Visual Recognition Challenge, which consisted in classifying natural images in the ImageNet dataset into 1000 fine-grained categories <cit.>. They have also significantly improved performance in a variety of medical imaging applications <cit.>, such as classification of lung diseases and lymph nodes in CT images <cit.>, segmentation (pixel classification) of brain tissues <cit.> in MRI, vessel segmentation <cit.> in fundus images, and detecting cervical intraepithelial neoplasia (CIN, particularly CIN2+) at patient level based on Cervigram images <cit.> or multimodal data <cit.>. Additionally, ConvNets have demonstrated superior performance in the classification of cell images, such as pleural cancer <cit.> and human epithelial-2 cell images <cit.>.


    


Large datasets are crucial to the high performance of Conv-Nets. However, there exists a very limited amount of labeled data for cervical cells, as high expertise is required for quality annotation. For instance, the Herlev benchmark dataset <cit.> only contains 917 cells (675 abnormal and 242 normal). Transfer learning <cit.> is an effective method to overcome this problem. Since the features in the first few ConvNet layers are more generic, they can be appended to various sets of subsequent layers specific to different tasks <cit.>. For instance, ConvNets trained on large-scale natural image datasets (e.g., ImageNet <cit.>) can be transferred to various medical imaging datasets, such as CT <cit.>, ultrasound <cit.> and X-ray <cit.> datasets, and can subsequently reduce overfitting on small datasets while boosting performance through fine-tuning.

In this paper, we apply ConvNets to the classification of cervical cells in cytology images. Our approach directly operates on raw RGB channels sampled from a set of square image patches coarsely centered on each nucleus. A ConvNet pre-trained on ImageNet is fine-tuned to discriminate between patches containing abnormal and normal cells based on deep hierarchical features. For an unseen cell, a set of image patches coarsely centered on the nucleus are classified by the fine-tuned ConvNet. Its classification results are then aggregated to generate the final cell category. Our approach is tested on two cervical cell image datasets: the Herlev dataset consisting of Pap smear images <cit.>; the HEMLBC (H&E stained manual liquid-based cytology) dataset being used to develop automation-assisted cervical screening system <cit.>. In our experiments (conducted using five-fold cross-validation(CV)), the fine-tuned ConvNet obtains classification accuracies of 98.3% on Herlev dataset and 98.6% on HEMLBC dataset, surpass the previous best accuracies of 96.8% and 94.3% on the two datasets, respectively. 

Our contributions are summarized as follows. 1) To the best of our knowledge, this is the first application of deep learning and transfer learning methods to cervical cell classification. 2) Unlike the previous methods, which rely on cytoplasm/nucleus segmentation and hand-crafted features, our method automatically extracts deep hierarchical features embedded in the cell image patch for classification, as long as a coarse nucleus center is provided. As a result, the classification does not suffer from any accuracy loss caused by inaccurate segmentation, and does not explicitly utilize prior medical knowledge of cervical cytology. 3) Our method generates the highest performances on both the Herlev Pap smear and the HEMLBC datasets, and has the potential to improve the performance of automation-assisted cervical cancer screening systems.



    





§ METHODS


The proposed method includes a training and a testing stage, as shown in Fig. <ref>. In the training stage, a ConvNet is first pre-trained on the ImageNet dataset, and data preprocessing is applied on the cervical cell dataset. Next, transfer learning is applied, whereby the pre-trained network parameters are used to initialize a new ConvNet. This ConvNet is then fine-tuned on the preprocessed training samples. In the testing stage, the preprocessed testing images are fed into the fine-tuned ConvNet. The abnormality score is obtained by aggregating the ConvNet's output values. Further details are described below.

 



 §.§ Data Preprocessing




  §.§.§ Patch extraction

Unlike previous patch based cell classification methods <cit.>, our method does not directly operate on images containing full cells (like the images in the Herlev dataset), for practical reasons. In particular, obtaining an individual cell requires cell pre-segmentation (at least cytoplasm segmentation), which remains an unsolved, challenging problem <cit.>. As mentioned in the TBS rules <cit.>, different cervical cytology abnormalities are associated with different nucleus abnormalities. Hence, nucleus features in themselves already include substantial discriminative information. We thus extract image patches of size m × m centered on the nucleus centroid. This strategy allows for embedding not only the nucleus scale/size information (an important discriminative feature between abnormal and normal cells), but also the contextual clues (e.g., the cytoplasm appearance) in the extracted patches. We acknowledge that automated methods for extracting a nucleus patch, e.g., Laplacian-of-Gaussian (LoG) <cit.>, selective search <cit.>, or ConvNets <cit.> exist. However, in this paper, we choose to focus on the classification task. We adopt a simple method of directly translating the centroid of the ground-truth nucleus mask to extract a set of image patches as described below. 



  §.§.§ Data Augmentation


Data augmentation improves the accuracy of ConvNets and reduces overfitting <cit.>. Since cervical cells are rotationally invariant, we perform N_r rotations (with a step size of θ degrees) on each cell image, and thus increase our number of image samples. N_r patches (one per rotated image) of size m × m centered at the rotated nucleus centers are extracted as the training samples, as shown in the middle (blue) panel in Fig. <ref>. Note that rotating a cell image may slightly degrade its high frequency contents (could be considered as a lower imaging quality), but should not change its abnormality/normality for most cells. Actually the augmentation step based on image rotation is crucial to the success of the ConvNet <cit.>, and has been demonstrated to be important for improving accuracy of ConvNet-based cell image classification problem <cit.>, given the limited number of images in the Herlev and HEMLBC dataset. Zero padding is used to void regions that lie outside of the image boundary.

Considering that the detected nucleus center may be inaccurate in practice, we randomly translate (by up to d pixels) each nucleus centroid N_t times to obtain N_t points as the coarse nucleus centers. Accordingly, N_t patches of size m × m centered at these locations are extracted as training samples, as shown in the right (green) panel in Fig. <ref>. These patches not only simulate inaccurate nucleus center detection, but also increase the amount of training samples for ConvNets. Other data augmentation approaches such as scale and color transformations are not used, as both the size and intensity of the nucleus are essential features to distinguish abnormal cervical cells from normal ones.

There are ∼ 3 times more abnormal cell images than normal cell images in the Herlev dataset. Classifiers tend to exhibit bias towards the majority class (abnormal cells). Although achieving a high sensitivity rate (correct classification of abnormal cells) is ideal from a medical point of view <cit.>, a high false positive rate (mis-classification of normal cells as abnormal) is not desirable from a practical standpoint <cit.>. A common solution to this dilemma is to balance the proportions of positive and negative training samples <cit.>. Doing so also improves the accuracy and convergence rate of ConvNets in training <cit.>. Hence, we create a balanced training set by sampling a higher proportion of normal than abnormal patches. 




 §.§ Convolutional Neural Networks

A convolutional neural network (ConvNet) <cit.> is a deep learning model comprising multiple consecutive stages, namely convolutional (conv), non-linearity and pooling (pool) layers, followed by more conv and fully connected (fc) layers. The input of the ConvNet is the raw pixel intensity image (in our case, the image obtained by subtracting the mean image over the training set from the original image <cit.>). The output layer is composed of several neurons each corresponding to one class. The weights (W) in the ConvNet are optimized by minimizing the classification error on the training set using the backpropagation algorithm <cit.>. Fig. <ref> shows two ConvNets. The upper network is trained on the ImageNet dataset, and the lower network is trained on the cervical cell dataset.


    




  §.§.§ Convolutional layer

The conv layer takes local rectangular patches across (with offset by stride and with/without spatial preservation by padding) the input image (for the first layer) or feature maps (for the subsequent layers) as input, on which 2D convolution with a filter is performed. The sum x of the resulting convolutions is fed into a non-linearity function, specifically a rectified linear unit (ReLU) f(x) = max(0,x) <cit.>, in order to increased the speed of training. In a given layer, the same filter is shared in a feature map, while different filters are used for different feature maps. This property of filter sharing in conv layer allows for detecting the same pattern in different locations of the feature map.



  §.§.§ Pooling layer

The pooling operation down-samples the feature map by summarizing feature responses in each non-overlapping local patch, often by computing the maximum activations (max-pooling). This yields features invariant to minor translations in the data. 



  §.§.§ Fully connected layer

conv and pool generate feature maps of smaller dimensions than the input image, which are then passed through several fc layers. The first few fc layers fuse these feature maps into a feature vector. The last fc layer contains two neurons which compute the classification probability for each class using softmax regression. To reduce overfitting, “dropout" <cit.> is used to constrain the fully-connected layers.



  §.§.§ Network training

The weights W in ConvNets are initialized with values from the Gaussian distribution. During training, these weights are iteratively updated with the gradients of the loss function, computed via stochastic gradient descent (SGD) over a mini-batch (size of 256) of training samples. The initial learning rate is decreased after certain epochs. As in Ref. <cit.>, momentum and weight decay are used to speed up the learning and reduce overfitting. The training process is terminated after a pre-determined number of epochs. The model with the lowest validation loss value is selected as the final network.




 §.§ Transfer Learning

Transfer learning refers to the fine-tuning of deep learning models that are pre-trained on other large-scale image datasets. In this study, the first few conv and pool layers of a ConvNet pre-trained on the ImageNet classification dataset (ILSVRC2012) (purple region in the upper part in Fig. <ref>) are used as the base of our network, on top of which several task-specific fc layers with random initialized weights are attached. In order to facilitate the transfer of features, the same network layers (conv and pool) with the BVLC CaffeNet <cit.> are transferred to the same locations in our model (purple regions in Fig. <ref>). Like our network, the CaffeNet also takes RGB channels as input. All of these layers are jointly trained (fine-tuned) on our cervical cell dataset, for which a learning rate 10 times smaller than the original CaffeNet value is used to fine-tune the transferred layers, and the original learning rate is used to train the fc layers from scratch. 




 §.§ Testing


To classify an unseen image, we combine random-view aggregation <cit.> and multiple crop testing <cit.> to produce the final prediction score. In particular, our data augmentation method generates N_v image patches (rotations and translations about the nucleus centroid). From each of these patches, N_c sub-patches are cropped (including its corner, center and mirrored patches). Hence, for each test cervical cell image, N_v× N_c sub-patches are fed into the ConvNet. The final prediction score is obtained by averaging the scores of these N_v× N_c predictions.





§ EXPERIMENTAL METHODS





 §.§ Data set

The cell data used to train and test the ConvNets comes from two datasets with two types of cervical cytology images, which were acquired by different slide preparation, staining methods, and imaging conditions.



  §.§.§ Herlev Dataset

The first one is from a publicly available dataset (http://mde-lab.aegean.gr/downloads) collected at the Herlev University Hospital by a digital camera and microscope <cit.>. The image resolution is 0.201 μ m per pixel <cit.>. The specimens are prepared via conventional Pap smear and Pap staining. The Herlev dataset consists of 917 images – each containing one cervical cell – with ground truth segmentation and classification. There are a total of seven different classes – diagnosed by two cyto-technicians and a doctor, in order to maximize certainty of the diagnosis. These seven classes belong to two categories: class 1-3 are normal, and class 4-7 are abnormal, as shown in Table <ref>. Examples of some cells are provided in Fig. <ref>(a). As can be seen, most abnormal cells have larger nucleus size than normal cells. However, the normal columnar nucleus may have similar size (also maybe similar chromatin distribution) as severe and/or carcinoma nuclei, which makes the classification challenging.




    


For each abnormal cell image in the Herlev dataset, N_r = 10 rotations (θ = 36^∘) and N_t = 10 translations (up to 10 pixels) are performed. For each normal cell, we use N_r = 20 (θ = 18^∘) and N_t = 14, resulting in 100 and 280 image patches for each abnormal and normal cell image, respectively. This yields a relatively balanced data distribution. Note that such different steps of rotation/translation for abnormal and normal cells are only for training not testing set. The image patch size is set to m = 128 pixels to cover some cytoplasm region for most cells, and to contain most of the full nucleus region for the largest one. These image patches are then up-sampled to a size of 256 × 256 × 3 pixels via bi-linear interpolation, in order to facilitate the transfer of pre-trained ConvNet model <cit.>.




  §.§.§ HEMLBC Dataset

The second one is from our own dataset collected at the People's Hospital of Nanshan District by using our previously developed autofocusing system (Olympus BX41 microscope with 20× objective, Jenoptik ProgRes CF Color 1.4 Megapixel Camera, and MS300 motorized stage) <cit.>. Each pixel has a size of 0.208 μ m^2. The specimens are prepared by manual liquid-based cytology with H&E staining. The dataset used in this paper is a subset used to train the abnormal/normal nucleus classifier for our automation-assisted cervical screening system <cit.>. There are totally 989 abnormal cells from 8 biopsy-confirmed CIN slides and 1381 normal cells from another 8 NILM (negative for intraepithelial lesion and malignancy) slide available. To create a balanced data distribution, 989 normal cells are randomly selected. The abnormal cells are diagnosed by two experienced pathologists. Most of them are segmented by an automated algorithm <cit.> and the ill-segmented ones are manually segmented by a pathologist. The normal cells are formed by two subsets: the first subset is collected by a pathologist with automated segmentation; the second subset is some false positive cells (e.g., cells with large nuclei, atrophic cells, etc.) collected during bootstrap process from validation images with manual segmentation for the ill-segmented ones by an engineer. More details are described in <cit.>. Examples of some cells are shown in Fig. <ref>(b).



For both abnormal and normal cell image in the HEMLBC dataset, N_r = 10 rotations (θ = 36^∘) and N_t = 10 translations (up to 10 pixels) are performed, resulting in 100 image patches for each cell image. The image patch size is also set to m = 128 pixels and then up-sampled to a size of 256 × 256 × 3 pixels as in Herlev dataset.





 §.§ Network Architectures and Implementation

Fig. <ref> illustrates our network architecture. The base ConvNet (denote as ConvNet-B) is pre-trained on the ImageNet classification dataset. ConvNet-B contains five conv layers (conv1 - conv5), three pool layers (pool1, pool2, pool5), and three fc layers (fc6 - fc8). Layers from conv1 to pool5 are transferred to the same locations in our model (denote as ConvNet-T). In other word, the first 5 weight layers (conv1 to pool5) of ConvNet-T are copied from the pre-trained ConvNet-B, and fc6 - fc8 layers of ConvNet-T are initialized with random Gaussian distributions. The detailed configurations of our ConvNet-T are listed in Table. <ref>. Local response normalization is used for conv1 and conv2 layers using the same setting as <cit.>, and all hidden layers are equipped with the ReLU activation function. Note that the ConvNet-B and ConvNet-T share the same network structure from conv1 to pool5, but the number of neurons of the three fc layers in ConvNet-B and ConvNet-T are 4096-4096-1000 and 1024-256-2, respectively. The 1024 and 256 are set based on our empirical evaluation, and 2 is to accommodate the new object categories in our 2-class (abnormal/normal) classification problem. Actually, setting the number of neurons of fc6 and fc7 layers in the range of 1024∼256 will result in similar accuracy, while more number of neurons (e.g., 4096-4096) tend to have slightly lower accuracy (1%-2% lower) on our data, which is more compact and specific compared to ImageNet. ConvNet-T is run on the Caffe platform <cit.>, using a  Nvidia GeForce GTX TITAN Z GPU with 12 GB of memory. 






 §.§ Training and Testing Protocols

From each 256 × 256 training image patch or its mirrored version, a 227 × 227 sub-patch is randomly cropped, from which the mean image over the training set is then subtracted. 
Stochastic Gradient Descent (SGD), with a mini-batch size of 256, is used to train the ConvNet-T for 30 epochs. The learning rates of layers conv1 - pool5 and layers fc6 - fc8 start from 0.001 and 0.01, respectively, and are decreased by a factor of 10 at every tenth epoch. Weight decay and momentum are set to 0.0005 and 0.9. A dropout ratio of 0.5 is used to constrain the fc6 and fc7 layers.

In testing, we obtain the final score by averaging the scores of the fc8 output on 1000 patches (N_v = 100  augmentations each with N_c = 10 sub-crops).




 §.§ Evaluation Methods

We evaluate the cervical cell classification using five-fold CV on both Herlev and HEMLBC datasets, to facilitate comparison with most previously reported results. In each of the 5 iterations of the ConvNet, 4 of 5 folds are used as training data, and the remaining one as validation. 
It's worth mentioning that data augmentation is after the training/validation splitting of cell population.
We obtain the model's final performance values by averaging results from the 5 validation sets. The performance evaluation metrics include sensitivity (Sens), specificity (Spec), accuracy (Acc), harmonic mean (H-mean), F-measure, and area under the ROC curve (AUC),


    Sens = TP/TP+FN  , 
    
    Spec = TN/TN+FP  , 
    
    Acc = TP+TN/TP+TN+FP+FN  , 
    
    H-mean = 2 ×Sens× Spec/Sens+Spec


where Sens measures the proportion of correctly identified abnormal cells, and Spec the proportion of correctly identified normal cells; Acc is the global percentage of correctly classified cells; H-mean = 2 ×Sens× Spec/Sens+Spec, used in <cit.>, takes into account the imbalanced data distribution; F-measure, the harmonic mean of precision and recall, is used in <cit.>. The ROC curve is computed by varying thresholds on the final classification scores (each final score is the average score of 1000 predictions).
 To test the robustness of our method against localization error of nucleus center, we randomly translate the ground truth centers of the test cells up to 5 or 10 pixels in both x and y directions, and the resulting performances on Herlev dataset are reported. In addition, the numbers of correct classification (normal vs. abnormal) and the distribution (shown by box plots) of the predicted abnormal scores of all cells for each of the seven cell classes (listed in Table <ref>) are reported.
Finally, we further consider the 7-class classification problem by simply modifying the number of neurons in the last fc layer from 2 to 7, and report the overall error (OE)% as in <cit.>.




§ RESULTS



 §.§ ConvNet Learning Results

Fig. <ref> illustrates a fine-tuning process of ConvNet-T during 30 training epochs on the Herlev dataset. As shown in the figure, after 6 epochs, the validation loss reaches its minimum value (0.119), with a corresponding validation accuracy of 0.972. Fig. <ref> shows the learned filters of the first convolutional layer of ConvNet-T trained on the Herlev Pap smear dataset. These automatically learned filters mainly consist of gradients of different frequencies and orientations and blobs of color, which are necessary for the cervical cell classification task. Along with these learned filters, the activations (feature maps) of an example cell at different pooling layers (pool1, pool2, and pool5) are provided in Fig. <ref>. One can observe that the pooling operation summarizes the input cell image or previous feature maps by highlighting the activated spatial locations, and that the features become increasingly abstract in deeper layers of the ConvNet. 


    



    



    





 §.§ Qualitative Results

Fig. <ref> and Fig. <ref> contain examples of correctly classified abnormal and normal cell patches from the validation Herlev dataset, respectively. Fig. <ref> provides examples of misclassified cervical cells from both Herlev and HEMLBC datasets, including both false negatives and false positives. The first two false negatives are instances of carcinoma, and the third one is an example of severe dysplasia. All false positives are columnar epithelial cells. 


    



    



    





 §.§ Quantitative Results on Herlev Dataset

Table <ref> shows the classification performance (Sens, Spec, Acc, H-mean, F-measure, and AUC) of our method in comparison with previous methods <cit.> on the Herlev dataset. 

The mean values of Sens, Spec, Acc, H-mean, F-measure, and AUC from our method (ConvNet-T) are 98.2%, 98.3%, 98.3%, 98.3%, 98.8%, and 0.998, respectively. We thus outperform previous methods in all metrics but Sens, which is slightly below others. Among these metrics, our Spec (98.3%) substantially surpasses the previous highest result (92.2%). Also note that certain degree of localization error (up to 10 pixels) of nucleus center only results in a small reduction of performances of our method (e.g., Acc from 98.3% to 97.8%).

Table <ref> provides the numbers (and corresponding percentages) of correct classification for each of the seven cell classes. Our method shows perfect performance on two types of normal cell (superficial and intermediate squamous epithelial), as well as one type of abnormal cell (mild dysplasia). While the performances are relatively lower for columnar epithelial and severe squamous non-keratinizing dysplasia (both are 95.9%). The distribution of the abnormal scores of all cells for the seven cell classes are shown as box plots in Fig. <ref>. The proposed method returns abnormality-scores close to 0 or 1 for most normal and abnormal cells, respectively. The few misclassifications mainly occur to normal columnar and severe squamous cells, given the probability threshold at 0.5.

Finally, for the 7-class problem, an overall error (OE) of 1.6% is achieved on average. Such an error is lower than errors of previous methods, such as 7.9% <cit.> and 3.9% <cit.>.






    





 §.§ Quantitative Results on HEMLBC Dataset

Table <ref> compares the classification performance between the proposed deep ConvNet-based method and our previous MLP (multilayer perceptron)-based method <cit.> on the HEMLBC dataset. Although the dataset used in this paper is a subset slightly smaller than that used in <cit.>, an obvious trend of performance improvement can be observed.







 §.§ Computational Speed

The average training time of a ConvNet-T running over up to 30 epochs is about 4 hours. Using the N_v× N_c = 1000 classification strategy, the testing time for one cervical cell is 3.5 seconds on average. 





§ DISCUSSION




 §.§ Comparison With Previous Methods


The methods in <cit.> follow the traditional cell classification pipeline – with features derived from manually segmented cytoplasms/nuclei. The techniques presented in  <cit.> perform direct texture classification of the input image. In contrast, our method automatically learns from the input image patch, and thus is not limited by the shortcomings of cell segmentation or feature design. The Sens values of previous methods are slightly higher than those from our method (99.0% vs. 98.2% under 5-fold CV). Such high Sens results mainly from the imbalanced data distribution – number of abnormal cells ∼ 3 X higher than the number of normal cells – which induces the classifier to predict more cells as abnormal. High Sens even at the expense of fairly low Spec is required for specimen level diagnosis, as all positives will be reexamined by human experts.
However, considering the abundance of normal cells (up to 300,000) in a Pap smear slide, the resulting lower Spec will generate many false positives in clinical practice. For example, a 92% specificity <cit.> will result in about 24,000 false positive cells. As a result, extensive and tedious targeted reading from a human observer will be necessary to refine the accuracy of the screening. Our approach substantially decreases the number of false positives. Although there are still about 1.7% false positives, they only come from columnar epithelial cells (Table <ref>). Actually, the differentiation between some columnar epithelial cells and (severe) abnormal cells are also challenging for experienced pathologists. Our method perfectly eliminates the majority types of normal cells (superficial and intermediate epithelial) in a specimen, and thus alleviates the labor burden of targeted reading and potentially reduces screening errors. Compared to our previous MLP method <cit.> on HEMLBC dataset, the deep ConvNet method achieves both higher Sens and Spec at cell level. Actually, the automation-assisted screening system <cit.> built upon the MLP method has a satisfyingly high Sens=88% and a perfect Spec=100% at slide level by pathologist's targeted reading. Therefore, our new method has a high potential to further improve the Sens of screening system while reducing the labor burden of targeted reading.




 §.§ Advantages of the Proposed Method

1) The proposed method is designed for robust automated screening applications, since it only requires a coarse nucleus centroid as input (no cytoplasm/nucleus segmentation is needed). Our experiments indicate that the proposed image patch based cell classification is robust to inaccurate detection of nucleus centroid (refer to ConvNet-T^Δ 5 and ^Δ 10 in Table <ref>). Some examples can be seen in Fig. <ref> and Fig. <ref>, where most of the image patches are not centered on the exact nucleus centroids, but are still assigned reasonable abnormal scores by our method. 
2) Moreover, our method is able to distinguish the abnormal and normal cells even for some “difficult" cases. For example, the two columnar epithelial cells in the third column in Fig. <ref> appear to exhibit a far greater level of abnormality than a severe dysplasia cell (lower one in the third column in Fig. <ref>), as these columnar cells have much larger nuclei or nonuniform chromatin distributions. Unlike traditional morphology/texture based classification methods, which simply classify both cells as abnormal, our method provides a much higher abnormal score (0.89) for the severe dysplasia cell than the two columnar cells (0.01 and 0.21). This indicates that the ConvNet-T captures some latent but essential features embedded in the cell images. 
3) Finally, the deep learning based method has high Sens and especially high Spec, and produces the highest performances on a Pap-stained Pap smear (Herlev) and a H&E stained liquid-based cytology (HEMLBC) datasets. Such a strong performance has the potential to boost the development of automation-assisted reading systems in primary cervical cancer screening. 



 §.§ Limitations

Despite its high performance, our method demonstrates a few limitations hindering its inclusion in existing cervical screening systems. 
1) Classification of a single patch requires 3.5 seconds, which is far too slow in a clinical setting. One could address this issue by eliminating the image patch augmentation step (100 variants per patch) from the testing phase, thus reducing speed to 0.035 seconds while compromising accuracy by only ∼ 1%. 
2) Despite high classification accuracy on the Herlev dataset, our method misclassify a few severe dysplasia (4.1%) and carcinoma (2%) cells as normal (Table <ref> and Fig. <ref>). As shown in Fig. <ref>(a), two dark stained carcinoma nuclei and a very large severe dysplasia nucleus are incorrectly classified as normal. An ideal screening system should not miss such severe abnormalities. To better handle such mis-classifications, cytoplasm/nucleus segmentation based features could be integrated into the system, either via deep learning or via “TBS" rules. Furthermore, both Herlev and HEMLBC are mainly consisted of expert-selected “typical" cells. The real life situation is more complex so more investigations are needed before transferring the results of this study to practice. For example, refer to the last two false positive cells in Fig. <ref>(b), they are from NILM slides, but it's hard to tell whether: the first one is an abnormal or normal cell due to poor imaging quality, and the second one is a normal atrophic cell or an abnormal cell. 
3) A nucleus center is pre-required for applying our method, and is obtained from the ground truth segmentation in this paper. However, screening of abnormal cells within a given field-of-view requires automated detection of nucleus centers. Our ongoing study shows that this may easily be achieved using the fully convolutional networks (FCN) <cit.> for semantic segmentation of cervical cells. And we already show that our method is robust to certain amount of inaccurate nucleus center detection.
4) The current experiments are conducted on a majority of images with individual cells. The effect of overlapping nuclei, cell clumps and artifacts on classification accuracy needs to be analyzed more extensively  in the future investigation, since a screening system is expected to able to avoid misclassifying such objects as abnormal cells. Task-specific classifiers (mostly likely relying on deep learning) may be needed to handle these problems <cit.>. 





§ CONCLUSION

This paper proposes a convolutional neural network-based method to classify cervical cells. Unlike previous methods, which rely on cytoplasm/nucleus segmentation and hand-crafted features, our method automatically extracts deep features embedded in the cell image patch for classification. It consists in extracting image patches coarsely centered on the nucleus as network input, transferring features from another pre-trained model into a new ConvNet for fine-tuning on the cell image patches, and aggregating multiple predictions to form the final network output. The proposed method yields the highest performance on both the Herlev Pap smear and the HEMLBC liquid-based cytology datasets, compared to previous methods. We anticipate that a segmentation-free, highly accurate cervical cell classification system of this type is promising for the development of automation-assisted reading systems for primary cervical screening.






§ ACKNOWLEDGMENTS

This work was supported in part by the Intramural Research Program at the NIH Clinical Center, and the National Natural Science Foundation of China (81501545). The authors thank Nvidia for the TITAN Z GPU donation.








   
IEEEtran   


