HATNet: An End-to-End Holistic Attention Network for Diagnosis of Breast Biopsy Images

Training end-to-end networks for classifying gigapixel size histopathological images is computationally intractable. Most approaches are patch-based and first learn local representations (patch-wise) before combining these local representations to produce image-level decisions. However, dividing large tissue structures into patches limits the context available to these networks, which may reduce their ability to learn representations from clinically relevant structures. In this paper, we introduce a novel attention-based network, the Holistic ATtention Network (HATNet) to classify breast biopsy images. We streamline the histopathological image classification pipeline and show how to learn representations from gigapixel size images end-to-end. HATNet extends the bag-of-words approach and uses self-attention to encode global information, allowing it to learn representations from clinically relevant tissue structures without any explicit supervision. It outperforms the previous best network Y-Net, which uses supervision in the form of tissue-level segmentation masks, by 8%. Importantly, our analysis reveals that HATNet learns representations from clinically relevant structures, and it matches the classification accuracy of human pathologists for this challenging test set. Our source code is available at \url{https://github.com/sacmehta/HATNet}


I. INTRODUCTION
B REAST cancer is the most common cancer in women accounting for approximately 25% of all cancer instances worldwide [1], [2]. Diagnostic classification errors among pathologists can have significant adverse consequences for patients. The "gold standard" for diagnosis of breast biopsy specimens relies on a pathologists visual assessment of tissue sections and cognitive processing of learned cytologic and morphological criteria, including architectural and cellular changes in the tissue, alterations of the tumor micro-environment, and immune-mediated host response. Assessment of these morphological criteria is subjective and pathologists, even expert pathologists, cannot always reach consensus on diagnostically challenging cases. Diagnostic disagreement occurs throughout the spectrum of benign to malignant lesions [3]. Diagnostic variability is a serious problem as misclassifying breast cancer as benign may lead to fatal progression, and diagnosing a benign lesion as malignant may result in significant morbidity This work was supported in part by the National Cancer Institute awards (R01 CA172343, R01 CA140560, and RO1 CA200690), NSF (IIS-1616112, IIS1252835), an Allen Distinguished Investigator award, and gifts from the Allen Institute for AI, Google, and Amazon.
S. Mehta and X. Lu contributed equally.
including overtreatment, unnecessary emotional strain, anxiety, and cost. Additionally, misdiagnosis of breast cancer has been a leading cause for malpractice claims for decades, and misclassification undoubtedly undermines research quality and hinders medical progress [4], [5]. A computer-aided diagnostic system that could reduce classification uncertainties would have immediate positive clinical impact. This paper introduces a self-attention-based network called Holistic ATtention Network (HATNet) for classifying breast biopsy images in an end-to-end manner. HATNet extends the self-attention network of Vaswani et al. [6]. The core principle is to factorize the input biopsy image into words (or patches) using a bag-of-words approach and then encode inter-word and inter-bag relationships in a hierarchical manner using self-attention. Self-attention enables interaction between inputs (bags or words), allowing the encoding of global information in an end-to-end fashion. This helps the network learn clinically relevant tissue structures without any supervision.
HATNet significantly outperforms state-of-the-art methods; it is 8% more accurate and about 2× faster than the previous best network, Y-Net [7], and also matches the classification performance of participant pathologists on the test set. Importantly, this network pays attention to ductal regions and stromal tissues, important bio-markers in breast cancer diagnosis, suggesting that there is clinical relevance in our method. To the best of our knowledge, this is the first work that (1) uses transformers to classify histopathological images in an end-to-end fashion and (2) correlates model decisions with clinically relevant structures. Our source code is available at https://github.com/sacmehta/HATNet.

II. RELATED WORK
Histopathological image classification: Convolutional neural networks (CNNs) are state-of-the-art networks for image classification (e.g., ResNet [8]), including histopathological image analysis [7], [9]- [13]. These methods follow a bag-ofwords model for learning representations, wherein an image is a treated as a bag while image patches are treated as words (or instances). The first line of research focuses on extracting word-level representations using CNNs, which are then aggregated to produce image-level decisions. Feature selectionbased aggregation methods (e.g., [10], [11], [14]) allows identification of relevant features in these word representations.
However, such methods fail to capture the heterogeneity of diagnosis categories [12]. To address the limitations of these methods, multi-instance learning based methods have been proposed [12], [13], [15], [16]. These methods first identify salient instances (or words), which are then combined using different methods (e.g., thresholding, majority voting, and learned fusion) to produce image-level decisions.
The second line of research considers tissue type, size, and distribution to produce image-level decisions [17]- [19]. Instead of identifying salient words, these approaches produce word-level segmentation masks, which are then combined to produce image-level segmentation masks. Tissue-level structural information (e.g., size and distribution) extracted from these masks is then used to produce diagnosis categories.
The third line of research integrates both saliency-based and segmentation-based approaches [7], [20]- [22]. These approaches simultaneously produce saliency maps and segmentation masks, which are then combined to extract structural information about tissues and to produce image-level decisions.
Though these methods are effective in classifying histopathological images, the context-capturing ability of saliency-based methods is limited to words and is not able to encode spatial relationships between words. Also, some of these methods (e.g., [12], [13]) require manual threshold selection to identify salient regions. The latter segmentationbased methods address these limitations; however, acquiring tissue-level segmentation labels at a large scale is difficult, because experts are required for annotating images. In this work, we introduce a transformer-based method, HATNet, to address the limitations of existing methods. Similar to previous work, HATNet is based on the bag-of-words model. However, unlike existing methods, it hierarchically aggregates information at different levels of the model using self-attention, which allows learning of spatial relationships between words and bags. HATNet outperforms existing methods (saliency-based or segmentation-based or their combination) by a significant margin (Section V-E). Moreover, this network pays attention to clinically relevant structures (Section VI).
Spatial attention: The most widely studied attention mechanism in visual recognition tasks (image classification, segmentation and object detection) is the spatial attention mechanism [23], [24], which weighs the activation maps (or spatial planes) to identify regions of interest. Initially introduced to provide explanations for CNN outputs, variants of this mechanism (supervised [25], [26] and unsupervised [27], [28]) have been incorporated in CNNs to improve the performance across different visual recognition tasks (e.g., [29], [30]), including medical imaging (e.g., [31]- [35]). In general, these networks introduce a spatial attention module (e.g., Attention U-Net [31]) within a CNN. Identifying salient regions in histopathological images using spatial attention is difficult because of their large size (usually of the order of gigapixels). This paper introduces an end-to-end transformer-based network for classifying histopathological images.
Transformers: Recent work has extended transformers [6] (described in Section III) for natural images (e.g., classification [36], segmentation [37], [38] and object detection [39]). Though these approaches are effective in learning local and global representations, extending these approaches to histopathological images is challenging primarily because of their large size (e.g., images in our dataset are 45× larger than the ImageNet dataset [40]). In this work, we extend transformers using bag-of-words models to classify breast biopsy images in an end-to-end fashion. Specifically, we introduce a bottom-up decoding method that allows us to hierarchically refine the information from words to bags to image and produce diagnostic categories. This hierarchical refinement allows clinically relevant tissue structures to be identified without any explicit supervision. To the best of our knowledge, this is the first work that 1) uses transformers for histopathological image analysis and 2) provides explanations for diagnostic decisions.

III. BACKGROUND: TRANSFORMERS
Transformer-based attention networks [6] allow inputs to interact with each other, so that the model can automatically find important inputs on which to focus. The transformer module, shown in Figure 1, consists of two parts: (1) multihead attention that models relationships between inputs, and (2) a feed forward network that learns wider representations.
The multi-head attention module has three projection branches that maps the input X ∈ R N ×d to query (Q), key (K), and value (V), where N is the number of inputs (words and bags in this case) and d is the input dimensionality. In particular, each branch consists of H linear layers (also called heads) that projects X from dto d h -dimensional space, allowing us to learn multiple views of the input, where d h = d/H. To allow the module to attend over different input positions, it computes H dot-products between keys (K) and queries (Q) and produces H attention weight matrices, each of size N × N . These resultant H attention weight matrices X Linear Linear Linear

Multi-head Attention
Multi-head attention Bag-to-bag attention Fig. 2: (a) HATNet: Our end-to-end holistic attention network for classifying breast biopsy whole slide images models the relationships between bags and words in a hierarchical manner using self attention. (b-d) Word-to-word, word-to-bag, and bagto-bag attention modules are visualized that allows learning relationships between bags and words using a bottom-up method, respectively. Note that word-to-bag attention module for processing B cnn and bag-to-image attention module for processing B b2b are similar to (c) and therefore, we do not visualize here.
encode the relationships between N inputs and are combined with the values (V) after a softmax using another dot-product to produce weighted sum outputs. The independent H outputs are then concatenated and fused using another linear layer β mha ∈ R d×d to produce the output X mha ∈ R N ×d . The multi-head attention operation is defined as: The multi-head attention module learns narrower representations in d h -dimensional space. To enable the transformer unit to learn wider representations, the output of multi-head attention X mha is fed to the feed forward network (FFN). The FFN is a stack of two linear layers. The first linear layer with weights β E ∈ R d×4d expands the input from dto 4ddimensional space, while the second linear layer with weights β R ∈ R 4d×d projects back from 4dto d-dimensional space. Mathematically, we can define this operation as: This work extends the transformer unit using the bag-of-word model to encode inter-word and inter-bag relationships. This increases the context-capturing ability of the network and improves performance.

IV. HATNET: HOLISTIC ATTENTION NETWORK
State-of-the-art CNN-based classification networks (e.g., ResNet [8]) stack convolutional layers and down-sampling layers to learn representations at multiple scales. These networks are difficult to apply to histopathological images, primarily because the resolution of these medical images (e.g., 10K × 10K) are much larger than images used in standard image classification tasks (e.g., 224×224). To address this resolution challenge, a standard approach is to learn patch-wise (or word-wise) representations using a sliding window method (e.g., [7], [12], [13]). Though these approaches have shown to be effective for histopathological image classification, the context-capturing ability of such approaches is still limited to patch-level. Also, such approaches are difficult to train in an end-to-end manner.
This paper introduces an end-to-end approach that allows diagnostic class prediction using the entire histopathological image at once. Our method extends the transformer architecture [6] using a bag-of-words approach and is shown in Figure 2. We call our model a Holistic ATtention Network (HATNet) because of its ability to learn inter-word and interbag representations in an end-to-end fashion. With attention, we emphasize the progressive hierarchical refining from words to bags to image to produce the classification output.
Briefly, HATNet first encodes inter-word representations using self-attention (Section IV-A). These representations are then combined to produce bag-level representations (Section IV-B). We then encode inter-bag representations (Section IV-C), which are then combined to produce image-level representations (Section IV-D). These representations are classified to produce the diagnosis category (Section IV-E). Because of the bottom-up decoding (words → bags → image), representations learned using HATNet are expressive and allow the identification of important words and bags in an image. We believe that this will help us build tools to annotate clinically important words and explain diagnosis decisions.

A. Word-to-word attention
The word-to-word attention module (see Figure 2(b)) is comprised of a transformer unit (Section III) with multi-head attention and a feed-forward network, allowing us to model the interactions between words and identify important words in the whole slide image. The input image I ∈ R w×h with width w and height h is first divided into n non-overlapping bags where W i j represents the jth word in the i-th bag. Similar to previous works (e.g. [7], [12]), we feed words W i j inside each bag B i to a CNN to produce word-level representations for each bag: The representations from the CNN does not encode inter-word relationships. We encode interword relationships in each bag B i cnn using the transformer unit (Section III) to produce B i w2w ∈ R m×d as: The key-query-value decomposition in multi-head attention allows us to encode inter-word relationships, and the FFN allows us to learn wider representations.

B. Word-to-bag attention
The word-to-word attention identifies relevant words in each bag. We aggregate these word-level representations to produce bag-level representations (see Figure 2(c)). To do so, we linearly combine the words inside each bag B i w2w . In particular, we first map each word in B i w2w from R d to R 1 using a projection function Ψ. Since each bag has m words, this projection function Ψ produces a vector of length m. We then apply a linear transformation β w2b ∈ R m×m and softmax to produce m coefficients, which are then used to linearly combine words in B i w2w to produce bag-level representation B i w2b ∈ R d as: Similarly, we can combine word-level representations obtained from the CNN for each bag B i cnn using Ψ and linear transformation β w2b ∈ R m×m to produce bag-level represen-

C. Bag-to-bag attention
The bag-level representations B w2b and B w2b do not encode information about surrounding bags. To encode inter-bag relationships, we apply bag-to-bag attention (see Figure 2(d)). The bag-to-bag attention module is similar to the word-toword attention (Section IV-A), except that we use B w2b (Eq. 5) as context to B w2b (Eq. 4). This also mimics the typical skip-connection mechanism in neural networks [8], [41].
We first apply multi-head attention to B w2b to encode interbag representations and produce B b2b ∈ R n×d as: To allow every bag obtained from CNN B b2b to attend over every bag B w2b obtained after word-level self-attention, we apply another multi-head attention in which B b2b serves as a query and B w2b serves as keys and values, allowing us to encode rich inter-bag representations and produce B b2b ∈ R n×d . Mathematically, we can define the bag-to-bag attention operation as: D. Bag-to-image attention The inter-bag representations are encoded in B b2b ∈ R n×d . We aggregate these representations to produce image-level representations. Similar to word-to-bag attention (Section IV-B), we combine these bag-level representations using a function Ψ and linear transformation β b2i ∈ R n×n to produce image-level representations I b2i ∈ R d .
Because of the bottom-up decoding (words to bags to image), these representations are expressive and allows us to identify important words and bags in an image (Figure 3).

E. Classification and Loss
We classify I b2i ∈ R d into C-diagnosis classes using a linear classifier with weights β cls ∈ R d×C as: We minimize the cross-entropy loss L between the ground truth y and predictionŷ to train HATNet. During evaluation, we choose the index that has the highest confidence score in y as our predicted class label.

A. Dataset and evaluation
The breast biopsy dataset [3] consists of 240 whole slide images with haematoxylin and eosin (H&E) staining. The image dataset was designed to include a higher prevalence of cases from diagnostic categories that have lower prevalence in the general population. This provides a robust and more challenging image dataset. A total of 87 pathologists participated in a previous study and interpreted these cases. Each pathologist classified a random subset of 60 slides into four diagnostic categories (benign, atypia, ductal carcinoma in-situ (DCIS), and invasive breast cancer). This resulted in 22 diagnostic labels (on average) per slide. A group of three expert pathologists then interpreted these cases and provided a consensus label per slide. We treat these consensus labels as ground truth diagnostic labels. The pathologists also marked 420 regions of interest (ROIs) that best supported the diagnoses. Following previous studies on this dataset [7], [13], [15], [17], [18], we use these ROIs to train and evaluate our method using the following metrics: (1) classification (or top-1) accuracy, (2) F1-score, (3) sensitivity, (4) specificity, and (5) area under receiver operating characteristic curve (ROC-AUC). In particular, the dataset consists of 164 training, 42 validation, and 216 test ROIs (see Table I).

B. Architecture
The size of the ROIs is variable. To use computational resources efficiently, each ROI is resized to spatial dimensions of about 12544 × 12544. The resized ROIs are then split into n = 49 non-overlapping bags, each bag with a spatial dimension of 1792 × 1792. Each bag is further split into m = 49 non-overlapping words, resulting in words with a spatial dimension of 256 × 256. Overall, each ROI has 2401 words. These words are fed to off-the-shelf CNNs to extract word-level representations. In our experiments, three state-of-the-art light-weight CNNs pretrained on the ImageNet dataset [42] are studied: (1) ESPNetv2 [43], (2) MobileNetv2 [44], and (3) MNASNet [45]. ESPNetv2 follows an Inceptionstyle design [46] and uses four simultaneous 3 × 3 depthwise convolutions with different dilation rates. MobileNetv2 follows ResNet-style design [8]. To improve the computational efficiency, MobileNetv2 uses 3 × 3 depth-wise convolutions instead of 3 × 3 standard convolutions. MNASNet uses the same basic building block as MobileNetv2; however, it uses neural architecture search [47] to identify the optimal model configuration, which provides best trade-off between different parameters. The proposed network is generic and any off-theshelf heavy-weight CNNs (e.g., ResNet [8]) can be used to extract word-level representations. Heavy-weight networks are not explored because of computational constraints. The dimensionality of word-level representations varies from CNN to CNN. Therefore, the output of a CNN is linearly projected to a 256 dimensional space (d = 256). To encode the inter-word and inter-bag representations, 4 heads (H = 4) are used in multi-head attention, resulting in a head dimension of d h = 64. We use the function Ψ to aggregate word-level representations into bag-level representations (Section IV-B) and bag-level representations into image-level representations (Section IV-D). In our experiments, we study three different functions: (1) Euclidean distance (or L2 norm), (2) Manhattan distance (or L1 norm), and (3) mean of a vector.

C. Training
Our models are trained end-to-end using the ADAM [48] optimizer with a learning rate warm-up strategy. We first warm-up the learning rate from 10 −7 to 10 −4 in 600 iterations and then train the model for the next 50 epochs with a learning rate of 10 −4 . After that, learning rate is decayed by half and then train for another 50 epochs. Our model takes about 36 hours for training on two NVIDIA GeForce GTX 1080 GPUs, each with a memory of 8 GB. We accumulate gradients for 8 iterations before updating the weights, yielding an effective batch size of 8 ROIs per update. Training data is augmented by randomly resizing (192×192, 224×224, 256×256, 288×288, 320 × 320), flipping, and rotating (angle: −10 • to 10 • ) the words. For evaluation, a single model is obtained by averaging the best 5 validation checkpoints. Compared to the best model, averaged models delivered 1-1.5 points higher accuracy.

D. Baseline networks
We compare our method with the following methods:  1) Bag-of-words model with hand-crafted features [13]: An input image (bag) is split into words. Following [49], LAB histogram and LBP histogram features are extracted from these words. These word-level features are concatenated and then classified using logistic regression into diagnostic categories with and without saliency (see R2 and R3 in Table II). Similar to a standard practice in saliency-based approaches (e.g., [7], [12], [50]), the class with majority voting in saliency maps is selected as a diagnostic category. 2) Bag-of-words model with deep features [13]: Instead of using hand-crafted features, this approach uses a deep convolutional neural network, FCN [51], to obtain wordlevel representations. These representations are used to identify discriminative or salient regions. In addition to majority voting-based method, a learned fusion method [12] is also tried to model the relationships between words (see R4-R7 in Table II). 3) Multi-resolution segmentation network (MRSegNet) [17]: MRSegNet has two stages: (1) tissue-level segmentation and (2) diagnostic classification. The first stage is a multi-resolution encoder-decoder network which combines the outputs of many words (or patches) at different resolutions to reduce segmentation errors. In the second stage, histogram and co-occurrence features are extracted from tissue-level segmentation masks, which are then classified using a multi-layer perceptron into different diagnostic classes (see R8 in Table II). 4) Structural features [18]: This method extracts structural features from tissue-level segmentation masks produced using MRSegNet. These features allows capturing structural changes in ductal regions, an important biomarker for breast cancer diagnosis [52]- [55], in the breast (see R9 in Table II). 5) Y-Net [7]: Y-Net adds a classification branch to U-Net [41] and allows to jointly predict the tissue-level segmentation mask and the saliecy map. The saliency  map is then combined with the tissue-level segmentation mask to produce a discriminative segmentation mask. Similar to MRSegNet, Y-Net extracts histogram and cooccurrence features, which are then used for classify diagnostic classes (see R10 in Table II). Table II shows that HATNet outperforms state-of-the-art methods significantly and delivers performance comparable to pathologists. For example, HATNet (R13) improves the performance of the best saliency-based models (R5, R7) by about 15%. When compared to approaches that uses tissuelevel segmentation masks (R8-R10) to capture the structural changes in biopsies, HATNet delivers better performance. In particular, HATNet improves the F1-score of the previous best segmentation-based approach (R10) by 8%. Overall, these results shows that HATNet is effective. We note that ensembling these three models (R14) further improves the accuracy and senstivity by 1%.

E. Main Results
Furthermore, Table III shows that HATNet is fast. HAT-Net with MNASNet is about 1.8× faster than the previously best reported network, i.e., Y-Net.

F. Model Ablations
Effect of function Ψ: Table IV      Effect of bag and word sizes: Table V compares the performance of our model with three different bag-word size configurations using ESPNetv2 as a base feature extractor. The bag size of 1792 and word size of 256 delivered slightly better performance than the others. In the rest of the experiments, we use this bag-word size configuration.
Effect of different base feature extractors: Figure 4 compares the class-wise performance of HATNet with three differ-ent base feature extractors. HATNet with MNASNet delivers similar or better class-wise F1-score, sensitivity, and specificity values, except for the invasive case where MobileNetv2 has a higher sensitivity value. Figure 5 plots the overall and class-wise receiver operating characteristics of our model with different base feature extractors. We observe that MNASNet delivers the best performance (higher ROC-AUC) compared to the other two networks. Similarly, in Table II (R11-R13), our model delivers the best overall performance with MNASNet across different evaluation metrics. For example, our model with MNASNet has 6% and 5% higher F1-score than with ESPNetv2 and MobileNetv2, respectively.

VI. ANALYSIS
Several clinical studies have shown that ductal regions and stromal tissue are important bio-markers for diagnosing breast cancer (e.g., [2], [52]- [57], [57]- [59]). Briefly, ducts are thin tubes in the breast and are responsible for carrying milk from lobules (milk glands) to the nipples. These regions are useful in identifying cancers that began in milk ducts (e.g., DCIS [53]- [55]) . On the other hand, the stroma is part of the breast tissue with a structural and developmental role and may be involved in tumor promotion and progression. Many clinical studies have underlined the importance of stroma in tumor progression In (a-f), each sub-figure is organized from left to right as: breast biopsy image, stroma tissue labeled by pathologists, and the top-50% words (words that belong to stroma tissue are shown in pink while the remaining words are shown in blue) identified using our model. The remaining 50% words are shown in white. In (g), we plot the dice score between stromal tissue and top-k word predictions (k varies from 10% to 60%) for different diagnostic classes.
along with its contribution to risk factors that determines tumor formation [56], [57]. It was shown above that our model learns better representations, resulting in significant performance gains compared to existing methods (Table II). A closer analysis (see Figure 6 and 7) reveals that our model pays attention to these important bio-markers, which helps it to achieve these gains.
Ductal regions: To evaluate if our model pays attention to ductal regions or not, we compute the overlap between ductal regions (marked by experts) and top-k bag predictions of our model using dice score 1 . Results are shown in Figure 6g. When 1 We are interested in evaluating if our model pays attention to ductal regions or stroma region. Therefore, we only use top-k bags or words inside these regions while computing the dice score.
considering top-50% bag predictions, HATNet achieves a dice score of 0.68. This shows that HATNet identifies ductal regions as an important structure. Furthermore, Figure 6a-6f shows that HATNet is able to differentiate between ducts of variable size and texture. This shows that HATNet is effective in modeling inter-bag and inter-word relationships, which helps it improve performance.
Stromal tissue: We compute the overlap between stromal tissue (pixel-level annotations by pathologists) and top-k word predicted by our model to determine whether our model pays attention to stromal tissue. We use dice score to measure the overlap and vary k from 10% to 60%. Figure 7g shows that HATNet achieves a dice score of about 0.75 when top-50% word predictions are considered. This indicates that HATNet also identifies stroma as an important tissue. This is further strengthened by visualizations in Figures 7a-7f, which shows that majority of top-50% words lie in stromal tissue.

VII. CONCLUSION
This paper introduces an end-to-end attention-based network for classifying breast biopsy images. HATNet extends bag-of-words models using transformers [6] to learn global representations. Our approach effectively aggregates interword and inter-bag representations, which helps the model to learn representations from clinically relevant tissue structures. HATNet improves the state-of-the-art significantly and matches the classification performance of pathologists on the test set. Furthermore, HATNet learns representations from clinically relevant structures. We believe that top words and bags identified using our method will help us build tools to annotate cell-level structures (e.g., mitoses), which would help in providing detail explanation of diagnosis decisions. In the future, we plan to build such tools and also apply HATNet to other histopathological images, including melanoma.