\section{Introduction}
\label{sec:intro}
\input{fig/teaser}
Tissue phenotyping is a fundamental problem in computational pathology (CPATH) that aims at characterizing objective, histopathologic features within gigapixel whole-slide images (WSIs) for cancer diagnosis, prognosis, and the estimation of response-to-treatment in patients~\cite{ludwig2005biomarkers, kather2016multi, javed2020cellular}. Unlike natural images, whole-slide imaging is a challenging computer vision domain in which image resolutions can be as large as $150000 \times 150000$ pixels\blfootnote{$^{*}$ Contributed Equally.}, with many methods using the following three-stage, weakly-supervised framework based on multiple instance learning (MIL): 1) tissue patching at a single magnification objective (``zoom"), 2) patch-level feature extraction to construct a sequence of embedding instances, and 3) global pooling of instances to construct a slide-level representation for weak-supervision using slide-level labels (\textit{e.g.} - subtype, grade, stage, survival, origin)~\cite{hou2016patch, courtiol2019deep, van2019no, tellez2019neural, ilse2018attention, campanella2019clinical, lu2020data, zhu2017wsisa, lu2021ai}.

Though achieving ``clinical-grade" performance on many cancer subtyping and grading tasks, this three-stage process has a few important design limitations. First, patching and feature extraction are generally fixed to $[256 \times 256]$ context regions. Though able to discern fine-grained morphological features such as nuclear atypia or tumor presence, depending on the cancer type, $[256 \times 256]$ windows have limited context in capturing coarser-grained features such as tumor invasion, tumor size, lymphocytic infiltrates, and the broader spatial organization of these phenotypes in the tissue microenvironment, as depicted in Figure \ref{fig:hier_wsi}~\cite{beck2011systematic, Diao2021, chan2019histosegnet}. Second, in contrast with other image-based sequence modeling approaches such as Vision Transformers (ViTs), MIL uses only global pooling operators due to the large sequence lengths of WSIs~\cite{ilse2018attention}. As a result, this limitation precludes the application of Transformer attention for learning long-range dependencies between phenotypes such as tumor-immune localization, an important prognostic feature in survival prediction~\cite{saltz2018spatial, le2020utilizing, abduljabbar2020geospatial}. Lastly, though recent MIL approaches have adopted self-supervised learning as a strategy for patch-level feature extraction (called tokenization in ViT literature), parameters in the aggregation layers still require training~\cite{ciga2021self, saillard2021self, dehaene2020self, koohbanani2021self, boyd2021self, li2021dual, chen2022self}. In viewing patch-based sequence modeling of WSIs in relation to ViTs, we note that the architectural design choice of using Transformer attention enables pretraining of both the tokenization and aggregation layers in ViT models, which is important in preventing MIL models from over- or under-fitting in low-data regimes~\cite{dosovitskiy2021image, li2022efficient, caron2021emerging, bao2022beit, he2021masked}.

To address these issues, we explore the challenge of developing a Vision Transformer for slide-level representation learning in WSIs. In comparison to natural images which are actively explored by ViTs, we note {\textit{a key difference in modeling WSIs is that visual tokens would always be at a fixed scale for a given magnification objective}. For instance, scanning WSIs at a $20\times$ objective results in a fixed scale of approximately $0.5 \mu m$ per pixel, allowing for consistent comparison of visual elements that may elucidate important histomorphological features beyond their normal reference ranges. Moreover, WSIs also exhibit a hierarchical structure of visual tokens at varying image resolutions at $20\times$ magnification: the $16 \times 16$ images encompass the bounding box of cells and other fine-grained features (stroma, tumor cells, lymphocytes)~\cite{Diao2021, hou2019sparse}, $256 \times 256$ images capture local clusters of cell-to-cell interactions (tumor cellularity)~\cite{graham2019hover, bejnordi2017diagnostic, pati2020hact, abousamra2021multi}, $1024 \times 1024$-$4096 \times 4096$ images further characterize macro-scale interactions between clusters of cells and their organization in tissue (the extent of tumor-immune localization in describing tumor-infiltrating versus tumor-distal lymphocytes)~\cite{abduljabbar2020geospatial, brancati2021bracs}, and finally the overall intra-tumoral heterogeneity of the tissue microenvironment depicted at the slide-level of the WSI~\cite{javed2020cellular,saltz2018spatial, hosseini2019atlas, balkwill2012tumor, marusyk2012intra}. The hypothesis that this work tests is that the judicious use of this hierarchy in self-supervised learning results in better slide-level representations.

We introduce a Transformer-based architecture for hierarchical aggregation of visual tokens and pretraining in gigapixel pathology images, called Hierarchical Image Pyramid Transformer (HIPT). We approach the task of slide-level representation learning in a manner similar to learning long document representations in language modeling, in which we develop a three-stage hierarchical architecture that performs bottom-up aggregation from [$16 \times 16$] visual tokens in their respective $256 \times 256$ and $4096 \times 4096$ windows to eventually form the slide-level representation, as demonstrated in Figure \ref{fig:overview}~\cite{yang2016hierarchical, zhang-etal-2019-hibert}. Our work pushes the boundaries of both Vision Transformers and self-supervised learning in two important ways. By modeling WSIs as a disjoint set of nested sequences, within HIPT: 1) we decompose the problem of learning a good representation of a WSI into hierarchically-related representations each of which can be learned via self-supervised learning, and 2) we use student-teacher knowledge distillation (DINO ~\cite{caron2021emerging}) to pretrain each aggregation layers with self-supervised learning on regions as large as $4096 \times 4096$.

We apply HIPT to the task of learning representations of gigapixel histopathological images extracted at $20\times$ resolution. We show that our method achieves superior performance to conventional MIL approaches. The difference is pronounced in context-aware tasks such as survival prediction in which larger context is appreciated in characterizing broader prognostic features in the tissue microenvironment~\cite{pati2022hierarchical, saltz2018spatial, abduljabbar2020geospatial, chen2021multimodal}. Using K-Nearest Neighbors on the $4096 \times 4096$ representations of our model, we outperform several weakly-supervised architectures in slide-level classification -- an important step forward in achieving self-supervised slide-level representations. Finally, akin to self-supervised ViTs on natural images that can perform semantic segmentation of the scene layout, we find that the multi-head self-attention in self-supervised ViTs learn visual concepts in histopathology tissue (from fine-grained visual concepts such as cell locations in the $\operatorname{ViT_{256}\textrm{-}16}$ to coarse-grained visual concepts such as broader tumor cellularity in the $\operatorname{ViT_{4096}\textrm{-}256}$), as demonstrated in Figure \ref{fig:attention_column}, \ref{fig:hierarchical_attention}. We make code available at \href{https://github.com/mahmoodlab/HIPT}{https://github.com/mahmoodlab/HIPT}.

\section{Related Work}
\label{sec:related}

\vspace{-3mm}
\paragraph{Multiple Instance Learning in WSIs} In general set-based deep learning, Edwards \& Storkey and Zaheer \etal proposed the first network architectures operating on set-based data structures, with Brendel \etal demonstrating ``bag-of-features" able to reach high accuracy on ImageNet~\cite{DBLP:conf/iclr/EdwardsS17, zaheer2017deep, DBLP:conf/iclr/BrendelB19}. Concurrently in pathology, Ilse \textit{et al.} extended set-based network architectures as an approach for multiple instance learning in histology region-of-interests, with Campanella \textit{et al.} later extending end-to-end weak-supervision on gigapixel WSIs~\cite{ilse2018attention, campanella2019clinical}. Lu \textit{et al.} demonstrated that by using a pretrained ResNet-50 encoder on ImageNet for instance-level feature extraction, only a global pooling operator needs to be trained for weakly-supervised slide-level tasks~\cite{lu2020data}. Following Lu \textit{et al.}, there have been many variations of MIL that have adapted image pretraining techniques such as VAE-GANs, SimCLR, and MOCO as instance-level feature extraction~\cite{zhao2020predicting, li2021dual, saillard2021self}. Recent variations of MIL have also evolved to extend the aggregation layers and scoring functions~\cite{zhu2017wsisa, xu2019camel, tellez2019neural, shao2021transmil, yao2019deep, yao2020whole, chen2021multimodal}. Li \textit{et al.} proposed a multi-scale MIL approach that performs patching and self-supervised instance learning at $20\times$ and $5\times$ resolution, followed by spatially-resolved alignment of patches\cite{li2021dual}. The integration of magnification objectives within WSIs has been followed in other works as well~\cite{hashimoto2020multi, marini2021multi, meng2021hierarchical, gao2016multi}, however, we note that combining visual tokens across objectives would not share the same scale. In this work, patching is done at a single magnification objective, with larger patch sizes used to capture macro-scale morphological features, which we hope will contribute towards a shift in rethinking context modeling of WSIs.

\paragraph{Vision Transformers and Image Pyramids} The seminal work of Vaswani \etal has led to remarkable developments in not only language modeling, but also image representation learning via Vision Transformers (ViTs), in which $256 \times 256$ images are formulated as an image patch sequence of $[16 \times 16]$ visual tokens~\cite{vaswani2017attention, dosovitskiy2021image, touvron2021training}. Motivated by multiscale, pyramid-based image processing \cite{rosenfeld1971edge, burt1987laplacian, koenderink1984structure}, recent progress in ViT architecture development has focused on efficiency and integration of multiscale information (\textit{e.g.} - Swin, ViL, TNT, PVT, MViT) in addressing the varying scale / aspect ratios of visual tokens~\cite{han2021transformer, zhang2021multi, liu2021swin, wang2021pyramid, fan2021multiscale}. In contrast with pathology, we highlight that learning scale invariance may not be necessary if the image scale is fixed at a given magnification. Similar to our work is NesT and Hierarchical Perciever, which similarly partitions and then aggregates features from non-overlapping image regions via Transformer blocks\cite{zhang2022nested, carreira2022hierarchical}. A key difference is that we show ViT blocks at each stage can be separately pretrained for high-resolution encoding (up to $4096 \times 4096)$.
\section{Method}
\subsection{Problem Formulation}
\noindent \textbf{Patch Size and Visual Token Notation:}  We use the following notation to distinguish between the sizes of ``images" and ``tokens" that correspond to that image. For an image $\textbf{x}$ with resolution $L \times L$ (or $\textbf{x}_L$), we refer to sequence of extracted visual tokens from non-overlapping patches (of size $[l \times l]$) within $\textbf{x}_L$ as $\{\textbf{x}_{l}^{(i)}\}_{i=1}^{M} \in \mathbb{R}^{M \times d_{l}}$, where $M$ is the sequence length and $d$ is the embedding dimension extracted for $l$-sized tokens. In working with multiple image resolutions (and their respective tokens) in a WSI, we additionally denote the shape of visual tokens (and the patching parameter) within $\textbf{x}_L$ image as $[l \times l]$ (using brackets). For natural images with size $\textbf{x}_{256}$, ViTs generally use $l = L^{1/2} =16$ which results in a sequence length of $M=256$. Additionally, we denote a ViT working on a $L$-sized image resolution with $[l \times l]$ tokens as $\operatorname{ViT}_L-l$. For $\textbf{x}_{\text{WSI}}$ (referring to the slide-level resolution of the WSI), MIL approaches choose $l = 256$ which fits the input shape of CNN encoders that can be pretrained and using for tokenization, resulting in $M > 10,000$ (variable due to the total area of segmented tissue content).

\noindent \textbf{Slide-Level Weak Supervision:} For a WSI $\textbf{x}_{\text{WSI}}$ with outcome $y$, the goal is to solve the slide-level classification task $P(y|\textbf{x}_{\text{WSI}})$. Conventional approaches for solving this task use a three-stage MIL framework which performs: 1) $[256 \times 256]$-patching, 2) tokenization, and 3) global attention pooling.  $\textbf{x}_{\text{WSI}}$ is formulated as the sequence $\{\textbf{x}_{256}^{(i)}\}_{i=1}^{M} \in \mathbb{R}^{M \times 1024}$ which results from using a ResNet-50 encoder pretrained on ImageNet (truncated after the 3rd residual block). Due to the large sequence lengths with $l=256$, neural network architectures in this task are limited to per-patch and global pooling operators in extracting a slide-level embedding for downstream tasks.

\iffalse
with the general form $\mathcal{F}$:
\begin{equation}
\mathcal{F}\left(X\right)= \zeta \big(\rho ( \{ \phi ( \mathbf{x}^{(i)}_{256} ): \mathbf{x}^{(i)}_{256} \in \textbf{x}_{\text{WSI}} \} ) \big)
\end{equation}
in which $\phi: \mathbb{R}^{M \times 3 \times 256 \times 256} \rightarrow \mathbb{R}^{m \times d_{256}}$ is an encoder applied instance-level to $[256 \times 256]$ tokens, $\rho: \mathbb{R}^{m \times d} \rightarrow \mathbb{R}^{1 \times d}$ is an aggregation function that pools the embeddings in the bag, and $\zeta: \mathbb{R}^{1 \times d} \rightarrow \mathbb{R}^{\text{\# class}}$ is a bag-level classifier that further processes the bag-level features for downstream slide-level tasks.
\fi

\subsection{Hierarchical Image Pyramid Transformer (HIPT) Architecture}
In adapting ViTs for slide-level representation learning, we reiterate two important challenges distinct from computer vision in natural images: 1) the fixed scale of visual tokens and their hierarchical relationships across image resolutions, and 2) the large sequence lengths of unrolled WSIs. As mentioned, visual tokens in histopathology are generally object-centric (and vary in granularity) across image resolutions, and also have important contextual dependencies such as tumor-immune (inferring favorable prognosis) or tumor-stroma interactions (inferring invasion). Patching with small visual tokens at high objectives ($\textbf{x}_{256}$ at $20\times$) results in large sequence lengths that make self-attention intractable, whereas patching with large visual tokens at low objectives results in loss-of-detail of fine-grained morphological structures ($\textbf{x}_{256}$ at $5\times$) that still requires $[256 \times 256]$ patching at $20\times$.

To capture this hierarchical structure and the important dependencies that may exist at each image resolution, we approach WSIs similar to long documents as a nested aggregation of visual tokens that recursively break down into smaller tokens until the cell-level (Figure \ref{fig:overview}), written as:
\begin{align*}
    & \operatorname{HIPT}(\mathbf{x}_{\text{WSI}}) = \operatorname{ViT_{WSI}\textrm{-}4096}\big(\big\{ \operatorname{CLS}_{4096}^{(k)} \big\}_{k=1}^{M}\big) \\
    &\rightarrow \operatorname{CLS}_{4096}^{(k)} = \operatorname{ViT_{4096}\textrm{-}256} \big( \{ \operatorname{CLS}_{256}^{(j)} \}_{i=1}^{256} \big) \\
    &\rightarrow \operatorname{CLS}_{256}^{(j)} = \operatorname{ViT_{256}\textrm{-}16} \big( \{ \textbf{x}_{16}^{(i)} \}_{i=1}^{256} \big)
\end{align*}

\noindent where 256 is the sequence length of $[16 \times 16]$- and $[256 \times 256]$-patching in $\textbf{x}_{256}$ and $\textbf{x}_{4096}$ images respectively, and $M$ is the total number of $\textbf{x}_{4096}$ images in $\textbf{x}_{\text{WSI}}$. For ease of notation, we refer to $\textbf{x}_{16}$ images as being at the cell-level, $\textbf{x}_{256}$ images as being at the patch-level\footnote{``Patch" is most often used to describe $256 \times 256$ images in pathology, though we note ``patching" an image into smaller images can refer to any resolution.}, $\textbf{x}_{4096}$ images as being at the region-level, with the overall WSI being the slide-level. In choosing these image sizes, the input sequence length of tokens is always $M=256$ in the forward passes for the $\operatorname{ViT_{256}\textrm{-}16}$ and $\operatorname{ViT_{4096}\textrm{-}256}$ (cell- and patch-level aggregation), and usually $M < 256$ in the forward pass for the $\operatorname{ViT_\text{WSI}\textrm{-}4096}$ (slide-level aggregation). The [CLS] tokens from $\operatorname{ViT_{256}\textrm{-}16}$ (the output of the model) are used as the input sequence for $\operatorname{ViT_{4096}\textrm{-}256}$, with the [CLS] tokens from $\operatorname{ViT_{4096}\textrm{-}256}$ subsequently used as the input sequence for $\operatorname{ViT_\text{WSI}\textrm{-}4096}$, with the number of total visual tokens at each stage decreasing geometrically by a factor of 256. In choosing small ViT backbones for each stage, HIPT has less than 10M parameters and is easy-to-implement and train on commercial workstations. We describe each stage below.

\paragraph{ViT$_{\textbf{256}}$-16 for Cell-Level Aggregation} The computation of $\mathbf{x}_{16}$ cell-level token aggregation within $\mathbf{x}_{256}$ windows follows implementing the vanilla ViT in natural images\cite{dosovitskiy2021image}. Given a $\mathbf{x}_{256}$ patch, the ViT unrolls this image as a sequence of non-overlapping $[16 \times 16]$ tokens followed by a linear embedding layer with added position embeddings to produce a set of 384-dim embeddings $\{\textbf{x}_{16}^{(i)}\}_{i=1}^{256} \in \mathbb{R}^{256 \times 384}$, with a learnable [CLS] token added to aggregate cell embeddings across the sequence. We choose $l = 16$ in this setting to not only follow conventional ViT architectures, but also model important inductive biases in histopathology as at this resolution, a  $[16 \times 16]$ bounding box at $20\times \approx 8 \mu m^2$ area encodes visual concepts that are object-centric in featurizing single cells (\textit{e.g.} - cell identity, shape, roundness).

\iffalse
\begin{align}
\begin{split}
\operatorname{MHSA}(Q, K, V)&=\operatorname{Concat}\left(h_{1}, \ldots, h_{h}\right) W^{o} \\
h_{i}&=SA\left(Q W_{i}^{Q}, K W_{i}^{K}, V W_{i}^{V}\right) \\
\end{split}
\end{align}
\fi

\paragraph{ViT$_{\textbf{4096}}$-256 for Patch-Level Aggregation} To represent $\mathbf{x}_{4096}$ regions, despite the image resolution being much larger than conventional natural images, the number of tokens remains the same since the patch size scales with the image resolution. From the previous stage, we use $\operatorname{ViT_{256}\textrm{-}16}$ to tokenize non-overlapping $\mathbf{x}_{256}$ patches within each $\mathbf{x}_{4096}$ region, forming the sequence $\{\operatorname{[CLS]}_{256}^{(j)}\}_{j=1}^{256}$ that can be plugged into a ViT block to model larger image contexts. We use a $\operatorname{ViT_{4096}\textrm{-}256(n=4, h=3, d=192)}$ with output $\operatorname{[CLS]}_{4096}$.

\paragraph{ViT$_{\textbf{WSI}}$-4096 for Region-Level Aggregation} In computing the slide-level representation for $\textbf{x}_{\text{WSI}}$, we use a $\operatorname{ViT_{\text{WSI}}\textrm{-}4096(n=2, h=3, d=192)}$ in aggregating the $\operatorname{[CLS]}_{4096}$ tokens. $M$ ranges from $1 - 256$ in our observations depending on size of the WSI. Due to potential tissue segmentation irregularities in patching at $[4096 \times 4096]$, we ignore positional embeddings at this stage.

\input{tab/class}

\subsection{Hierarchical Pretraining}
\input{fig/attention_column}
In building a MIL framework using only Transformer blocks, we additionally explore and pose a new challenge referred to as slide-level self-supervised learning - which aims at extracting slide-level feature representations in gigapixel images for downstream diagnostic and prognostic tasks. This is an important problem as current slide-level training datasets in CPATH typically have between 100 to 10,000 data points, which may cause MIL methods to overfit due to over-parameterization and lack of labels.\footnote{For rare disease subtypes and clinical trials that study disease progression over the time-course of years, the collection of large patient datasets is difficult to scale for machine learning application.} To address this problem, we hypothesize that the recursive nature of HIPT in using Transformer blocks for image representation learning can enable conventional ViT pretraining techniques (such as DINO \cite{caron2021emerging}) to generalize across stages (of similar subproblems) for high-resolution images. To pretrain HIPT, first, we leverage DINO to pretrain $\operatorname{ViT_{256}\textrm{-}16}$. Then, keeping fixed the weights of $\operatorname{ViT_{256}\textrm{-}16}$, we re-use $\operatorname{ViT_{256}\textrm{-}16}$ as the embedding layer for $\operatorname{ViT_{4096}\textrm{-}256}$ in a second stage of DINO. We refer to this procedure as hierarchical pretraining, which is similarly performed in the context of learning deep belief networks \cite{erhan2010does} and hierarchical transformers for long documents\cite{zhang-etal-2019-hibert}. Though hierarchical pretraining does not reach the slide-level, we show that: 1) pretrained $\textbf{x}_{4096}$ representations in self-supervised evaluation are competitive with supervised methods for slide-level subtyping, and that 2) HIPT with two-stage hiearchical pretraining can reach state-of-the-art performance.

\paragraph{Stage 1: 256 $\times$ 256 Patch-Level Pretraining} To pretrain $\operatorname{ViT_{256}\textrm{-}16}$, we use the the DINO framework for pretraining of $\mathbf{x}_{256}$ patches, in which a student network $\phi_{s_{256}}$ is trained to match the probability distribution of a siamese teacher network $\phi_{t_{256}}$ using a cross-entropy loss $-p_{t_{256}}(\cdot) \operatorname{log} p_{s_{256}}(\cdot)$ with momentum encoding, with $p_{t_{256}}, p_{s_{256}}$ denoting the outputs of $\phi_{t_{256}}(\cdot), \phi_{s_{256}}(\cdot)$ respectively for $\mathbf{\mathbf{x}}_{256}$. As data augmentation for each $\mathbf{x}_{256}$ patch, DINO constructs a set of $M_l = 8$ local views ($\mathbf{x}_{96}$ crops, passed through $\phi_{s_{256}}$) and $M_g = 2$ global views ($\mathbf{x}_{224}$ crops, passed through $\phi_{t_{256}}$) to encourage local-to-global correspondences between the student and teacher, minimizing the function:
$$
\min _{\theta_{s_{256}}} \sum_{\left\{\mathbf{x}_{224}^{(i)}\right\}_{i=1}^{M_g}}^{M_g = 2} \sum_{\left\{\mathbf{x}_{96}^{(j)}\right\}_{j=1}^{M_l}}^{M_l = 8} H \Big( p_{t_{256}}(\mathbf{x}_{224}^{(i)}), p_{s_{256}}\left(\mathbf{x}_{96}^{(j)}\right) \Big)
$$

An intriguing property that makes this data augmentation suitable for histology data is again the natural part-whole hierarchy of cells in a tissue patch. In comparison to natural images in which $[96 \times 96]$ crops may capture only colors and textures without any semantic information, at $20\times$, local $[96 \times 96]$ crops would capture the context of multiple cells and their surrounding extracellular matrices, which has shared mutual information with the broader cellular communities. Similar to the original DINO implementation, we use horizontal flips and color jittering for all views, with solarizing performed on one of the global views.

\paragraph{Stage 2: 4096 $\times$ 4096 Region-Level Pretraining} With the sequence lengths and computational complexity in tokenizing $\mathbf{x}_{4096}$ regions similar to that of $\mathbf{x}_{256}$ patches, we can also borrow an almost identical DINO recipe in also pretraining $\operatorname{ViT_{4096}\textrm{-}256}$ and defining student-teacher networks $\phi_{s_{4096}}(\cdot), \phi_{t_{4096}}(\cdot)$ at this stage. Following extracting $\operatorname{[CLS]}_{256}$ tokens from $\operatorname{ViT_{256}\textrm{-}16}$ as input for $\operatorname{ViT_{4096}\textrm{-}256}$ input, we rearrange $\{\operatorname{[CLS]}_{256}^{(j)}\}_{j=1}^{M=256}$  as a $16 \times 16 \times 384$ 2D feature grid for data augmentations, performing $[6 \times 6], [14 \times 14]$ local-global crops in matching the scale of $[96 \times 96], [224 \times 224]$ crops for $256 \times 256$ inputs. As additional data augmentation, We apply standard dropout ($p=0.10$) to all views following work in Gao \etal\cite{gao-etal-2021-simcse}.

\section{Experiments}
\vspace{-3mm}
\paragraph{Pretraining:} We pretrain $\operatorname{ViT_{256}\textrm{-}16}$ and $\operatorname{ViT_{4096}\textrm{-}256}$ in different stages, using 10,678 FFPE (formalin-fixed, paraffin-embedded) H\&E-stained diagnostic slides from 33 cancer types in the The Genome Cancer Atlas (TCGA), and extracted 408,218 $\textbf{x}_{4096}$ regions at an $20\times$ objective ($M \approx 38$ regions per slide) for pretraining $\operatorname{ViT_{4096}\textrm{-}256}$, with a total of 104M $\textbf{x}_{256}$ patches for pretraining $\operatorname{ViT_{256}\textrm{-}16}$~\cite{liu2018integrated}. For $\operatorname{ViT_{256}\textrm{-}16}$, we trained for 400,000 iterations using the AdamW optimizer with a batch size of $256$, base learning rate of 0.0005, with the first 10 epochs used to warm up to the base learning rate followed by decay using a cosine schedule. A similar implementation was used for $\operatorname{ViT_{4096}\textrm{-}256}$, with the model trained for 200,000 iterations using the pre-extracted $\operatorname{[CLS}]$ tokens from $\operatorname{ViT_{256}\textrm{-}16}$.

\paragraph{Fine-tuning:} Following hierarchical pretraining, we use the pretrained weights to initialize (and freeze) the $\operatorname{ViT_{256}\textrm{-}16}$ and $\operatorname{ViT_{4096}\textrm{-}256}$ subnetworks, with only a lightweight $\operatorname{ViT_\text{WSI}\textrm{-}4096}$ finetuned. Our work can be viewed as a formulation of MIL that pretrains not only the $[256 \times 256]$ instance-level feature extraction step, but also the downstream aggregation layers which extract coarse-grained morphological features. We finetuned HIPT (and its comparisons) for 20 epochs using the Adam optimizer, batch size of $1$ with $32$ gradient accumulation steps, and a learning rate of $0.01$. For survival prediction, we used the survival cross-entropy loss by Zadeh \& Schmidt\cite{zadeh2020bias}.

\paragraph{Tasks \& Comparisons:} We experiment on several slide-level classification and survival outcome prediction tasks across different organ types in the TCGA~\cite{liu2018integrated}. In comparisons with state-of-the-art weakly-supervised architectures, we tested Attention-Based MIL (ABMIL), and it's variants that use clustering losses (CLAM-SB), clustering prototypes (DeepAttnMISL), modified scoring \& pooling functions (DS-MIL), and graph message passing (GCN-MIL), which used the same hyperparameters as HIPT. Since these methods are agnostic of input features, all comparisons used the pretrained $\operatorname{ViT_{256}\textrm{-}16}$ as instance-level feature extraction. In addition, we also compared variations of HIPT without pretraining and self-attention. Finally, we qualitatively study the attention maps that hierarchical self-supervised ViTs learn in computational histopathology.

\input{tab/surv}

\subsection{Slide-Level Classification}
\vspace{-3mm}
\paragraph{Dataset Description} We follow the study design in ~\cite{lu2020data}; we examined the following tasks evaluated using a 10-fold cross-validated AUC: 1) Invasive Ductal (IDC) versus Invasive Lobular Carcinoma (ILC) in Invasive Breast Carcinoma (BRCA) subtyping, 2) Lung Adenocarcinoma (LUAD) versus Lung Squamous Cell Carcinoma (LUSC) in Non-Small Cell Lung Carcinoma (NSCLC) subtyping, and 3) Clear Cell, Papillary, and Chromophobe Renal Cell Carcinoma (CCRCC vs. PRCC vs. CHRCC) subtyping, with all methods finetuned (for 20 epochs) with varying percentange folds of training data (100\% / 25\%) as data efficiency experiments. Despite RCC subtyping being a relative easy slide-level task due to having distinct subtypes, we ultimately include this task as a benchmark for self-supervised comparisons.

\paragraph{Weakly-Supervised Comparison} Classification results are summarized in Table \ref{tab:class}. Overall, across all tasks and different percentage folds, HIPT consistently achieves the highest macro-averaged AUC performance across all tasks. In comparison with the best performing baseline, CLAM-SB, HIPT achieves a performance increase of 1.86\%, 2.59\%, 0.72\% on BRCA, NSCLC and RCC subtyping respectively using 100\% of training data, with the margin in performance increase widening to 3.14\%, 8.33\%, 1.78\% respectively using 25\% of training data. Similar performance increases are demonstrated on other tasks. HIPT demonstrates the most robust performance when limiting training data, with AUC decreasing slightly from $0.980$ to $0.974$.

\paragraph{K-Nearest Neighbor (KNN)} We take the mean embedding of the pre-extracted embeddings, followed by a KNN evaluation for the above tasks. As a baseline, we use a ResNet-50 pretrained on ImageNet to extract patch-level embeddings. We compare with pre-extracted $\operatorname{ViT_{256}\textrm{-}16}$ patch embeddings from DINO pretraining, and pre-extracted $\operatorname{ViT_{4096}\textrm{-}256}$ region-level embeddings from hierarchical pretraining, with results summarized also in Table \ref{tab:class}. In using the average embedding of each WSI as the ``slide-level representation", we find that $\operatorname{ViT_{4096}\textrm{-}256}$ region-level embeddings in HIPT outperform patch-level embeddings across all tasks, which can be attributed to the broader image contexts used in the WSI for pretraining, and can be intuitively viewed as a closer proxy to the slide-level view than small patches. $\operatorname{ViT_{4096}\textrm{-}256}$ region-level embeddings surpass the AUC performance of weakly-supervised approaches in BRCA and RCC subtyping using 100\% of training data.

\subsection{Survival Prediction}
\vspace{-3mm}
\paragraph{Dataset Description} For survival outcome prediction, we validated on the IDC, CCRCC, PRCC, and LUAD cancer types which have relatively large sample sizes in the TCGA, in addition to Colon \& Rectal (CRC) and Stomach Adenocarcinoma (STAD) which have been frequently evaluated in real-world clinical studies due to their substantial human intra-observer variability~\cite{echle2020bjc, wulczyn2021npj, skrede2020deep}. All tasks were evaluated using cross-validated concordance index (c-Index).

\paragraph{Weakly-Supervised Comparison} For the following survival prediction tasks in which learning context-aware relationships are important, we observe much larger increases in performance, summarized in Table \ref{tab:surv}. Overall, HIPT achieves the best c-Index performance in the IDC, COADREAD, CCRCC, and STAD cancer types, with the largest improvement demonstrated in IDC (0.634) and COADREAD (0.608) in comparison to other methods. Though other methods such as GCN-MIL use message passing for learning context-aware features, we note that the number of layers needed to achieve similar image receptive fields may cause the number of neighbors to grow exponentially~\cite{li2019deepgcns}. In modeling important long-range dependencies between instances using self-attention across various stages of the hierarchy, the Transformer attention in HIPT is able to capture regional perturbations that have been well characterized as portending worse outcome across different cancer types, as further visualized in Figure ~\ref{fig:attention_column}, \ref{fig:hierarchical_attention}~\cite{abduljabbar2020geospatial, wulczyn2020deep, wulczyn2021npj, skrede2020deep}.

\subsection{Self-Supervised ViTs Find Unique Morphological Phenotypes}

\input{fig/hierarchical_attention}

\noindent\textbf{ViT$_{\textbf{256}}$-16 Attention Maps.} For $\mathbf{x}_{256}$ patches, we visualize the different attention heads in MHSA and reveal that ViTs in pathology are able to isolate distinct morphological features. From visual assessment by a board-certified pathologist across several different cancer types, we observe that MHSA in $\operatorname{ViT_{256}\textrm{-}16(n=8, h=6, d=384)}$ captures three distinct fine-grained morphological phenotypes as illustrated in Figure \ref{fig:attention_column}, with general stroma tissue and red blood cells attended in $h=1,2$, cells (normal, atypical, lymphocyte) attended in $h=3,4$, and ``white spaces" (luminal spaces, fat regions, air pockets) attended in $h=5,6$. This observation is in line with current studies that have introspected  self-supervised ViT models, in which the attention heads can be used as a method for object localization or discovery~\cite{caron2021emerging, simeoni2021localizing}. In the application to histopathology tissue, our introspection reveals that the visual tokens at the $[16 \times 16]$ cell-level directly corroborate with semantic, object-centric structures at the $20\times$ objective.

\noindent\textbf{ViT$_{\textbf{4096}}$-256 Attention Maps.} For $\mathbf{x}_{4096}$ regions, we further visualize the attention heads in MHSA from our pretrained $\operatorname{ViT_{4096}\textrm{-}256(n=4, h=6, d=192)}$ model, capturing two distinct coarse-grained phenotypes: tumor-stroma interface attended in $h=1,2,3$, and nested tumor cells and other high tumor cellularity regions in $h=4,5,6$. In comparison with the $\operatorname{ViT_{256}\textrm{-}16}$ attention maps which may capture only nuclear features (\textit{e.g.} - nuclear atypia, shape and size of cells), $\operatorname{ViT_{4096}\textrm{-}256}$ attention maps are able to model the patterns of nested tumor growth, tumor invasion into fat and stroma regions, and other tissue-to-tissue relationships (Figure ~\ref{fig:attention_column}). In factorizing the attention distribution of $[16 \times 16$] cells from $\operatorname{ViT_{256}\textrm{-}16}$ onto highly-attended $[256 \times 256]$ patches from $\operatorname{ViT_{4096}\textrm{-}256}$, we can create a hierarchical attention map, which is able to distinguish tumor cells in stroma tissue from tumor cells in dense tumor cellularity regions (Figure ~\ref{fig:hierarchical_attention}). Overall, these captured coarse- and fine-grained morphological features corroborate with the observed performance increases in both finetuning HIPT in weakly-supervised learning and using averaged HIPT features in KNN evaluation. Additional visualizations are found in the \textbf{Supplement}.
\subsection{Further Ablation Experiments}
Additional experiments are included in the \textbf{Supplementary Materials}, with main findings highlighted below:\\
\noindent\textbf{The role of pretraining.} Hierarchical pretraining of $\operatorname{ViT_{4096}\textrm{-}256}$ is an important component in our method, as HIPT variants without pretraining overfit in MIL tasks.\\
\noindent\textbf{Comparing patch-level representations.} We assessed quality of other embedding types, and found that $\operatorname{ViT_{256}\textrm{-}16}$ achieves strong representation quality of image patches.\\
\noindent\textbf{Organ-specific versus pan-cancer pretraining}. We additionally assessed the performance of $\operatorname{ViT_{256}\textrm{-}16}$ pretraining on different data distributions, with improved performance in cell localization with pan-cancer pretraining.
\section{Conclusion}
We believe our work is an important step towards self-supervised slide-level representation learning, demonstrating pretrained and finetuned HIPT features achieve superior performance on weakly-supervised and KNN evaluation respectively. Though DINO was used for hierarchical pretraining with conventional ViT blocks, we hope to explore other pretraining methods such as mask patch prediction\cite{dosovitskiy2021image, bao2022beit} and efficient ViT architectures\cite{zhang2021multi, liu2021swin, wang2021pyramid, li2022efficient}.

\noindent\textbf{Limitations:} A limitation of HIPT is the difficulty in pretraining the last aggregation layer due to the small number of WSI data points. In addition, end-to-end hierarchical pretraining of HIPT is computationally intractable on commercial workstations, with pretraining needed to be performed in stages. Lastly, the study design of this work has several constraints, such as: 1) excluded slides in each TCGA cohort due to limited tissue content and difficulty patching at $[4096 \times 4096]$, 2) $\operatorname{ViT_{256}\textrm{-}16}$ pretraining performed on almost all of TCGA and evaluation lacking independent test cohorts, 3), analysis limited to TCGA, which overrepresents patients with European ancestry and not representative of the rich genetic diversity in the world\cite{spratt2016racial}.

\noindent\textbf{Broader Impacts:} Many problems in biology and medicine have hierarchical-like relationships\cite{ma2018using, li2021representation, hinton2021represent}. For instances, DNA motifs within exon sequences which contributes towards protein structure, gene expression, and genetic traits\cite{avsec2021effective, jumper2021highly, demetci2021multi}. Our idea of pretraining neural networks based on hierarchical relationships in large, heterogeneous data modalities to derive a patient- or population-level representation can be extended to other domains.

\section{Acknowledgements}
We thank Felix Yu, Ming Y. Lu, Chunyuan Li, and the BioML group at Microsoft Research New England for their feedback. This work was supported in part by the BWH president's fund, BWH \& MGH Pathology, Google Cloud Research Award, and NIGMS R35GM138216 (F.M.). R.J.C. was also supported by the NSF Graduate Fellowship. T.Y.C. was also supported by the NIH T32CA251062. R.G.K. gratefully acknowledges funding from CIFAR.\\

\section{Introduction}
\label{sec:intro}

Please follow the steps outlined below when submitting your manuscript to the IEEE Computer Society Press.
This style guide now has several important modifications (for example, you are no longer warned against the use of sticky tape to attach your artwork to the paper), so all authors should read this new version.

\subsection{Language}

All manuscripts must be in English.

\subsection{Dual submission}

Please refer to the author guidelines on the CVPR\ 2022\ web page for a
discussion of the policy on dual submissions.

\subsection{Paper length}
Papers, excluding the references section, must be no longer than eight pages in length.
The references section will not be included in the page count, and there is no limit on the length of the references section.
For example, a paper of eight pages with two pages of references would have a total length of 10 pages.
{\bf There will be no extra page charges for CVPR\ 2022.}

Overlength papers will simply not be reviewed.
This includes papers where the margins and formatting are deemed to have been significantly altered from those laid down by this style guide.
Note that this \LaTeX\ guide already sets figure captions and references in a smaller font.
The reason such papers will not be reviewed is that there is no provision for supervised revisions of manuscripts.
The reviewing process cannot determine the suitability of the paper for presentation in eight pages if it is reviewed in eleven.

\subsection{The ruler}
The \LaTeX\ style defines a printed ruler which should be present in the version submitted for review.
The ruler is provided in order that reviewers may comment on particular lines in the paper without circumlocution.
If you are preparing a document using a non-\LaTeX\ document preparation system, please arrange for an equivalent ruler to appear on the final output pages.
The presence or absence of the ruler should not change the appearance of any other content on the page.
The camera-ready copy should not contain a ruler.
(\LaTeX\ users may use options of cvpr.sty to switch between different versions.)

Reviewers:
note that the ruler measurements do not align well with lines in the paper --- this turns out to be very difficult to do well when the paper contains many figures and equations, and, when done, looks ugly.
Just use fractional references (\eg, this line is $087.5$), although in most cases one would expect that the approximate location will be adequate.

\subsection{Paper ID}
Make sure that the Paper ID from the submission system is visible in the version submitted for review (replacing the ``*****'' you see in this document).
If you are using the \LaTeX\ template, \textbf{make sure to update paper ID in the appropriate place in the tex file}.

\subsection{Mathematics}

Please number all of your sections and displayed equations as in these examples:
\begin{equation}
  E = m\cdot c^2
  \label{eq:important}
\end{equation}
and
\begin{equation}
  v = a\cdot t.
  \label{eq:also-important}
\end{equation}
It is important for readers to be able to refer to any particular equation.
Just because you did not refer to it in the text does not mean some future reader might not need to refer to it.
It is cumbersome to have to use circumlocutions like ``the equation second from the top of page 3 column 1''.
(Note that the ruler will not be present in the final copy, so is not an alternative to equation numbers).
All authors will benefit from reading Mermin's description of how to write mathematics:
\url{http://www.pamitc.org/documents/mermin.pdf}.

\subsection{Blind review}

Many authors misunderstand the concept of anonymizing for blind review.
Blind review does not mean that one must remove citations to one's own work---in fact it is often impossible to review a paper unless the previous citations are known and available.

Blind review means that you do not use the words ``my'' or ``our'' when citing previous work.
That is all.
(But see below for tech reports.)

Saying ``this builds on the work of Lucy Smith [1]'' does not say that you are Lucy Smith;
it says that you are building on her work.
If you are Smith and Jones, do not say ``as we show in [7]'', say ``as Smith and Jones show in [7]'' and at the end of the paper, include reference 7 as you would any other cited work.

An example of a bad paper just asking to be rejected:
\begin{quote}
\begin{center}
    An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of our previous paper [1], and show it to be inferior to all previously known methods.
   Why the previous paper was accepted without this analysis is beyond me.

   [1] Removed for blind review
\end{quote}

An example of an acceptable paper:
\begin{quote}
\begin{center}
     An analysis of the frobnicatable foo filter.
\end{center}

   In this paper we present a performance analysis of the  paper of Smith \etal [1], and show it to be inferior to all previously known methods.
   Why the previous paper was accepted without this analysis is beyond me.

   [1] Smith, L and Jones, C. ``The frobnicatable foo filter, a fundamental contribution to human knowledge''. Nature 381(12), 1-213.
\end{quote}

If you are making a submission to another conference at the same time, which covers similar or overlapping material, you may need to refer to that submission in order to explain the differences, just as you would if you had previously published related work.
In such cases, include the anonymized parallel submission~\cite{Authors14} as supplemental material and cite it as
\begin{quote}
[1] Authors. ``The frobnicatable foo filter'', F\&G 2014 Submission ID 324, Supplied as supplemental material {\tt fg324.pdf}.
\end{quote}

Finally, you may feel you need to tell the reader that more details can be found elsewhere, and refer them to a technical report.
For conference submissions, the paper must stand on its own, and not {\em require} the reviewer to go to a tech report for further details.
Thus, you may say in the body of the paper ``further details may be found in~\cite{Authors14b}''.
Then submit the tech report as supplemental material.
Again, you may not assume the reviewers will read this material.

Sometimes your paper is about a problem which you tested using a tool that is widely known to be restricted to a single institution.
For example, let's say it's 1969, you have solved a key problem on the Apollo lander, and you believe that the CVPR70 audience would like to hear about your
solution.
The work is a development of your celebrated 1968 paper entitled ``Zero-g frobnication: How being the only people in the world with access to the Apollo lander source code makes us a wow at parties'', by Zeus \etal.

You can handle this paper like any other.
Do not write ``We show how to improve our previous work [Anonymous, 1968].
This time we tested the algorithm on a lunar lander [name of lander removed for blind review]''.
That would be silly, and would immediately identify the authors.
Instead write the following:
\begin{quotation}
\noindent
   We describe a system for zero-g frobnication.
   This system is new because it handles the following cases:
   A, B.  Previous systems [Zeus et al. 1968] did not  handle case B properly.
   Ours handles it by including a foo term in the bar integral.

   ...

   The proposed system was integrated with the Apollo lunar lander, and went all the way to the moon, don't you know.
   It displayed the following behaviours, which show how well we solved cases A and B: ...
\end{quotation}
As you can see, the above text follows standard scientific convention, reads better than the first version, and does not explicitly name you as the authors.
A reviewer might think it likely that the new paper was written by Zeus \etal, but cannot make any decision based on that guess.
He or she would have to be sure that no other authors could have been contracted to solve problem B.
\medskip

\noindent
FAQ\medskip\\
{\bf Q:} Are acknowledgements OK?\\
{\bf A:} No.  Leave them for the final copy.\medskip\\
{\bf Q:} How do I cite my results reported in open challenges?
{\bf A:} To conform with the double-blind review policy, you can report results of other challenge participants together with your results in your paper.
For your results, however, you should not identify yourself and should not mention your participation in the challenge.
Instead present your results referring to the method proposed in your paper and draw conclusions based on the experimental comparison to other results.\medskip\\

\begin{figure}[t]
  \centering
  \fbox{\rule{0pt}{2in} \rule{0.9\linewidth}{0pt}}

   \caption{Example of caption.
   It is set in Roman so that mathematics (always set in Roman: $B \sin A = A \sin B$) may be included without an ugly clash.}
   \label{fig:onecol}
\end{figure}

\subsection{Miscellaneous}

\noindent
Compare the following:\\
\begin{tabular}{ll}
 \verb'$conf_a$' &  $conf_a$ \\
 \verb'$\mathit{conf}_a$' & $\mathit{conf}_a$
\end{tabular}\\
See The \TeX book, p165.

The space after \eg, meaning ``for example'', should not be a sentence-ending space.
So \eg is correct, {\em e.g.} is not.
The provided \verb'\eg' macro takes care of this.

When citing a multi-author paper, you may save space by using ``et alia'', shortened to ``\etal'' (not ``{\em et.\ al.}'' as ``{\em et}'' is a complete word).
If you use the \verb'\etal' macro provided, then you need not worry about double periods when used at the end of a sentence as in Alpher \etal.
However, use it only when there are three or more authors.
Thus, the following is correct:
   ``Frobnication has been trendy lately.
   It was introduced by Alpher~\cite{Alpher02}, and subsequently developed by
   Alpher and Fotheringham-Smythe~\cite{Alpher03}, and Alpher \etal~\cite{Alpher04}.''

This is incorrect: ``... subsequently developed by Alpher \etal~\cite{Alpher03} ...'' because reference~\cite{Alpher03} has just two authors.

\begin{figure*}
  \centering
  \begin{subfigure}{0.68\linewidth}
    \fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
    \caption{An example of a subfigure.}
    \label{fig:short-a}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.28\linewidth}
    \fbox{\rule{0pt}{2in} \rule{.9\linewidth}{0pt}}
    \caption{Another example of a subfigure.}
    \label{fig:short-b}
  \end{subfigure}
  \caption{Example of a short caption, which should be centered.}
  \label{fig:short}
\end{figure*}

\section{Formatting your paper}
\label{sec:formatting}

All text must be in a two-column format.
The total allowable size of the text area is $6\frac78$ inches (17.46 cm) wide by $8\frac78$ inches (22.54 cm) high.
Columns are to be $3\frac14$ inches (8.25 cm) wide, with a $\frac{5}{16}$ inch (0.8 cm) space between them.
The main title (on the first page) should begin 1 inch (2.54 cm) from the top edge of the page.
The second and following pages should begin 1 inch (2.54 cm) from the top edge.
On all pages, the bottom margin should be $1\frac{1}{8}$ inches (2.86 cm) from the bottom edge of the page for $8.5 \times 11$-inch paper;
for A4 paper, approximately $1\frac{5}{8}$ inches (4.13 cm) from the bottom edge of the
page.

\subsection{Margins and page numbering}

All printed material, including text, illustrations, and charts, must be kept
within a print area $6\frac{7}{8}$ inches (17.46 cm) wide by $8\frac{7}{8}$ inches (22.54 cm)
high.
Page numbers should be in the footer, centered and $\frac{3}{4}$ inches from the bottom of the page.
The review version should have page numbers, yet the final version submitted as camera ready should not show any page numbers.
The \LaTeX\ template takes care of this when used properly.

\subsection{Type style and fonts}

Wherever Times is specified, Times Roman may also be used.
If neither is available on your word processor, please use the font closest in
appearance to Times to which you have access.

MAIN TITLE.
Center the title $1\frac{3}{8}$ inches (3.49 cm) from the top edge of the first page.
The title should be in Times 14-point, boldface type.
Capitalize the first letter of nouns, pronouns, verbs, adjectives, and adverbs;
do not capitalize articles, coordinate conjunctions, or prepositions (unless the title begins with such a word).
Leave two blank lines after the title.

AUTHOR NAME(s) and AFFILIATION(s) are to be centered beneath the title
and printed in Times 12-point, non-boldface type.
This information is to be followed by two blank lines.

The ABSTRACT and MAIN TEXT are to be in a two-column format.

MAIN TEXT.
Type main text in 10-point Times, single-spaced.
Do NOT use double-spacing.
All paragraphs should be indented 1 pica (approx.~$\frac{1}{6}$ inch or 0.422 cm).
Make sure your text is fully justified---that is, flush left and flush right.
Please do not place any additional blank lines between paragraphs.

Figure and table captions should be 9-point Roman type as in \cref{fig:onecol,fig:short}.
Short captions should be centred.

\noindent Callouts should be 9-point Helvetica, non-boldface type.
Initially capitalize only the first word of section titles and first-, second-, and third-order headings.

FIRST-ORDER HEADINGS.
(For example, {\large \bf 1. Introduction}) should be Times 12-point boldface, initially capitalized, flush left, with one blank line before, and one blank line after.

SECOND-ORDER HEADINGS.
(For example, { \bf 1.1. Database elements}) should be Times 11-point boldface, initially capitalized, flush left, with one blank line before, and one after.
If you require a third-order heading (we discourage it), use 10-point Times, boldface, initially capitalized, flush left, preceded by one blank line, followed by a period and your text on the same line.

\subsection{Footnotes}

Please use footnotes\footnote{This is what a footnote looks like.
It often distracts the reader from the main flow of the argument.} sparingly.
Indeed, try to avoid footnotes altogether and include necessary peripheral observations in the text (within parentheses, if you prefer, as in this sentence).
If you wish to use a footnote, place it at the bottom of the column on the page on which it is referenced.
Use Times 8-point type, single-spaced.

\subsection{Cross-references}

For the benefit of author(s) and readers, please use the
{\small\begin{verbatim}
  \cref{...}
\end{verbatim}}  command for cross-referencing to figures, tables, equations, or sections.
This will automatically insert the appropriate label alongside the cross-reference as in this example:
\begin{quotation}
  To see how our method outperforms previous work, please see \cref{fig:onecol} and \cref{tab:example}.
  It is also possible to refer to multiple targets as once, \eg~to \cref{fig:onecol,fig:short-a}.
  You may also return to \cref{sec:formatting} or look at \cref{eq:also-important}.
\end{quotation}
If you do not wish to abbreviate the label, for example at the beginning of the sentence, you can use the
{\small\begin{verbatim}
  \Cref{...}
\end{verbatim}}
command. Here is an example:
\begin{quotation}
  \Cref{fig:onecol} is also quite important.
\end{quotation}

\subsection{References}

List and number all bibliographical references in 9-point Times, single-spaced, at the end of your paper.
When referenced in the text, enclose the citation number in square brackets, for
example~\cite{Authors14}.
Where appropriate, include page numbers and the name(s) of editors of referenced books.
When you cite multiple papers at once, please make sure that you cite them in numerical order like this \cite{Alpher02,Alpher03,Alpher05,Authors14b,Authors14}.
If you use the template as advised, this will be taken care of automatically.

\begin{table}
  \centering
  \begin{tabular}{@{}lc@{}}
    \toprule
    Method & Frobnability \\
    \midrule
    Theirs & Frumpy \\
    Yours & Frobbly \\
    Ours & Makes one's heart Frob\\
    \bottomrule
  \end{tabular}
  \caption{Results.   Ours is better.}
  \label{tab:example}
\end{table}

\subsection{Illustrations, graphs, and photographs}

All graphics should be centered.
In \LaTeX, avoid using the \texttt{center} environment for this purpose, as this adds potentially unwanted whitespace.
Instead use
{\small\begin{verbatim}
  \centering
\end{verbatim}}
at the beginning of your figure.
Please ensure that any point you wish to make is resolvable in a printed copy of the paper.
Resize fonts in figures to match the font in the body text, and choose line widths that render effectively in print.
Readers (and reviewers), even of an electronic copy, may choose to print your paper in order to read it.
You cannot insist that they do otherwise, and therefore must not assume that they can zoom in to see tiny details on a graphic.

When placing figures in \LaTeX, it's almost always best to use \verb+\includegraphics+, and to specify the figure width as a multiple of the line width as in the example below
{\small\begin{verbatim}
   \usepackage{graphicx} ...
   \includegraphics[width=0.8\linewidth]
                   {myfile.pdf}
\end{verbatim}
}

\subsection{Color}

Please refer to the author guidelines on the CVPR\ 2022\ web page for a discussion of the use of color in your document.

If you use color in your plots, please keep in mind that a significant subset of reviewers and readers may have a color vision deficiency; red-green blindness is the most frequent kind.
Hence avoid relying only on color as the discriminative feature in plots (such as red \vs green lines), but add a second discriminative feature to ease disambiguation.

\section{Final copy}

You must include your signed IEEE copyright release form when you submit your finished paper.
We MUST have this form before your paper can be published in the proceedings.

Please direct any questions to the production editor in charge of these proceedings at the IEEE Computer Society Press:
\url{https://www.computer.org/about/contact}.
\section{Multiple Instance Learning}

\section{Multi-Head Self-Attention}

We briefly introduce self-attention then extend it to the multi-head version. Recall that $ \{\textbf{x}_{l}^{(i)}\}_{i=1}^{M} \in \mathbb{R}^{M \times d_{l}}$ denotes the sequence of visual token extracted from a $L \times L$ patch, where $M$ is the sequence length and $d_{l}$ is the embedding dimension extracted for $l$-sized tokens. For simplicity, we ignore the the subscript $l$ and lower the superscript $i$ to the subscript. The sequence of visual token is re-written as $S = \{\textbf{x}_{i}\}_{i=1}^{M} \in \mathbb{R}^{M \times d_{l}} $

To perform self-attention,  we add three matrices $\boldsymbol{W}_{q}$,$\boldsymbol{W}_{k}$ and $\boldsymbol{W}_{v}$ $\in \mathbb{R}^{d_{l} \times d_{l}}$ used to compute the relationship between the token itself $\textbf{x}_{i}$ and other tokens $\textbf{x}_{j}$ in $S$. Specifically,
\begin{align*}
    \boldsymbol{q}_{i} = \boldsymbol{W}_{q}\textbf{x}_{i},    \boldsymbol{k}_{i} &= \boldsymbol{W}_{k}\textbf{x}_{i},
    \boldsymbol{v}_{i} = \boldsymbol{W}_{v}\textbf{x}_{i}\\
    w_{ij} &= \textrm{softmax}( \frac{\boldsymbol{q}_{i}^{\intercal}\boldsymbol{k}_{j}}{\sqrt{d}_{l}})\\
    \boldsymbol{o}_{i} &= \sum_{j}w_{ij}\boldsymbol{v}_{j},
\end{align*}
where $\boldsymbol{q}_{i}$, $\boldsymbol{k}_{i}$ and $\boldsymbol{v}_{i}$ are query, key and value vector of input token $\textbf{x}_{i}$. Intuitively, each visual token $\textbf{x}_{i}$ computes a similarity score with other tokens and use the normalized similarity score to perform weighted-sum of value vectors of other tokens. The query, key and value vector space are captured by $\boldsymbol{W}_{q}$,$\boldsymbol{W}_{k}$ and $\boldsymbol{W}_{v}$ learned from data. To increase the feature expressiveness, we increase the number of matrices from 1 to $h$ which leads to a set of matrices $\{\boldsymbol{W}^{h^{\prime}}_{q}, \boldsymbol{W}^{h^{\prime}}_{k}, \boldsymbol{W}^{h^{\prime}}_{v} | h^{\prime} = 1,2,3,\dots,h\}$ for the input sequence. The self-attention with $h > 1$ is called multi-head self-attention (MHSA), which we use not only as the building block in permutation-equivariant aggregation of visual tokens, but also demonstrate to learn representations of morphological phenotypes, as seen in Figure~\ref{fig:dino}.

\input{fig/dino}

\section{Additional Implementation Details}
\paragraph{Efficient Inference and Batching in HIPT} At inference time, we feed in $\textbf{x}_{\text{WSI}} \in \mathbb{R}^{M \times 256 \times 256 \times [3 \times 16 \times 16]}$ with a slide-level batch size $B_{\text{WSI}}=1$ into HIPT, in which the respective dimensions correspond to: length of $\mathbf{x}_{4096}$ regions in $\textbf{x}_{\text{WSI}}$, length of $\mathbf{x}_{256}$ patches in $\mathbf{x}_{4096}$,  length of $\mathbf{x}_{16}$ cells in $\mathbf{x}_{256}$, and the remaining dimensions being the $\mathbf{x}_{16}$ shape. Without pre-extracting any tokens, inference can be performed in first defining a $\operatorname{DataLoader}$ class over a $M \times 3 \times 4096 \times 4096$ view of $\textbf{x}_{\text{WSI}}$ with $B_{4096}=1$ to iterate over single $\mathbf{x}_{4096}$ regions, followed by using $\operatorname{Einsum}$ operations to unroll each region via $\operatorname{Einsum}: \mathbb{R}^{1 \times 3 \times 4096 \times 4096} \rightarrow \mathbb{R}^{256 \times 3 \times 256 \times 256}$. Lastly, another $\operatorname{DataLoader}$ is defined over this view using the first dimension as $B_{256}=256$, which then leads to the bottom-up aggregation strategy starting with the $\operatorname{ViT_{256}\textrm{-}16}$ forward pass.

\paragraph{Pretraining Dataset Curation:} As noted in the main paper, we pretrain $\operatorname{ViT_{256}\textrm{-}16}$ and $\operatorname{ViT_{4096}\textrm{-}256}$ in different stages using 10,687 FFPE (formalin-fixed, paraffin-embedded) H\&E-stained diagnostic slides from 33 cancer types in the The Genome Cancer Atlas (TCGA), and extracted 408,218 $\textbf{x}_{4096}$ regions at an $20\times$ objective ($M \approx 38$ regions per slide) for pretraining $\operatorname{ViT_{4096}\textrm{-}256}$, with a total of 104 Million $\textbf{x}_{256}$ patches for pretraining $\operatorname{ViT_{256}\textrm{-}16}$~\cite{liu2018integrated}. To curate these regions, we used the Tissue Image Analysis (TIA) Toolbox to patch slides into non-overlapping, tissue-containing regions, with additional quality control performed to limit regions that contained predominantly background slide information (\textit{e.g.} - white space). A limitation of this study is that since our method requires $[4096 \times 4096]$ patching, not all slides from the TCGA were used for pretraining and weakly-supervised evaluation. A full table showing number of WSIs and regions used is shown on a per-cancer basis in Table~\ref{tab:tcga}.

\paragraph{Attention Visualization:} To create attention heatmaps, we follow the work of Caron \etal using the attention map for each head at the last ViT stage, and linearly interpolate the attention map such that the attention score each token is $[16 \times 16]$ for $\operatorname{ViT_{256}\textrm{-}16}$ and $[256 \times 256]$ for $\operatorname{ViT_{4096}\textrm{-}256}$. No Gaussian blurring or other smoothing operations were performed. To obtain more granular maps for $\operatorname{ViT_{4096}\textrm{-}256}$, we computed attention scores for each patch using an overlapping stride length of $64$. To create hierarchical attention maps, we factorized the attention weight distribution within each weighted $\textbf{x}_{256}$ patch from $\operatorname{ViT_{4096}\textrm{-}256}$ with the attention weight distribution of $\operatorname{ViT_{256}\textrm{-}16}$. These hierarchical attention maps can be interpreted as: \textit{For weighted $\textbf{x}_{256}$ patches localized by $\operatorname{ViT_{4096}\textrm{-}256}$ by head X, what are the important $\textbf{x}_{16}$ localized within that patch by head Y in $\operatorname{ViT_{256}\textrm{-}16}$?} This allows the creation of certain attention maps that localize: 1) $[16\times 16]$ tumor cells within $256 \times 256$ stromal regions, or 2) $[16\times 16]$ tumor cells within $256 \times 256$ poorly-differentiated glands / larger tumor structures. With $h=6$ for both $\operatorname{ViT_{256}\textrm{-}16}$ and $\operatorname{ViT_{4096}\textrm{-}256}$, a total of 36 hierarchical attention maps can be created. From pathologist inspection, 2-3 heads in each VIT model localized unique morphological phenotypes.

\input{tab/tcga}

\section{Additional Quantitative Experiments}

\input{tab/supp_class}
\input{tab/supp_kather}

\subsection{Variations in HIPT Architecture}
\paragraph{Model Descriptions} We performed ablation experiments assessing the most impactful components of the HIPT architecture, as demonstrated in the slide-level classification results in Table \ref{tab:supp_class}. Specifically, we inspected variations in the HIPT architecture:

\begin{itemize}
    \item $\operatorname{\mathbf{ViT_{256}\textrm{-}16}}_{\textbf{PF}}$, \textbf{AP-256, AP-4096}: This variation uses no ViT components as permutation-equivariant aggregation hidden layers at the patch- and region-level. Features are still pre-extracted using a $\operatorname{ViT_{256}\textrm{-}16}$ (denoted with ``P" for pretrained, ``F" for frozen), however, only attention pooling (AP) is performed across the different stages.
    \item $\operatorname{\mathbf{ViT_{256}\textrm{-}16}_{\textbf{PF}}}$, $\operatorname{\mathbf{ViT_{4096}\textrm{-}256}}$, $\operatorname{\mathbf{ViT_\text{WSI}\textrm{-}4096}}$: This variation uses $\operatorname{ViT_{4096}\textrm{-}256}$ and $\operatorname{ViT_\text{WSI}\textrm{-}4096}$ for patch- and region-level aggregation, however, these aggregation layers are trained from scratch.
    \item $\operatorname{\mathbf{ViT_{256}\textrm{-}16}}_{\textbf{PF}}$, $\operatorname{\mathbf{ViT_{4096}\textrm{-}256}}_{\textbf{P}}$, $\operatorname{\mathbf{ViT_\text{WSI}\textrm{-}4096}}$: This variation uses $\operatorname{ViT_{4096}\textrm{-}256}$ and $\operatorname{ViT_\text{WSI}\textrm{-}4096}$ for patch- and region-level aggregation, with$\operatorname{ViT_{4096}\textrm{-}256}$ additionally pretrained using Stage 2 Hierarchical Pretraining.
     \item $\operatorname{ViT_{256}\textrm{-}16}_{\text{PF}}$, $\operatorname{ViT_{4096}\textrm{-}256}_{\text{PF}}$, $\operatorname{ViT_\text{WSI}\textrm{-}4096}$: This variation uses $\operatorname{ViT_{4096}\textrm{-}256}$ and $\operatorname{ViT_\text{WSI}\textrm{-}4096}$ for patch- and region-level aggregation, with $\operatorname{ViT_{4096}\textrm{-}256}$ additionally pretrained and frozen. Only parameters for $\operatorname{ViT_\text{WSI}\textrm{-}4096}$ are finetuned.
\end{itemize}

\paragraph{Pretraining and Freezing Prevents Overfitting} Across slide-level classification tasks, we observe that pretraining and freezing $\operatorname{ViT_{4096}\textrm{-}256}$ in HIPT is an important component an achieving strong performance. Without freezing, training the parameters for $\operatorname{ViT_{4096}\textrm{-}256}$ results in the total number of trainable parameters to be 3388996. Though small for a Transformer model, training and finetuning on WSI datasets with less than 1000 data points may easily result in overfitting, as demonstrated in Table~\ref{tab:supp_class}. In particular, we observe that without freezing, performance drops from 0.923 to 0.652 and 0.952 to 0.820 on NSCLC subtyping with 25\% and 100\% training data finetuning respectively. We also compare HIPT to a variation that does not use any Transformer attention in patch- and region-level aggregation, which performed better than HIPT variations with $\operatorname{ViT_{4096}\textrm{-}256}$ finetuned, but still worse than HIPT with $\operatorname{ViT_{4096}\textrm{-}256}$ frozen.

\subsection{Assessing Quality of $\textbf{x}_{256}$ Representations} Though the primary focus of our paper is in hierarchical pretraining using the HIPT architecture, we also make publicly available the pretrained weights of $\operatorname{ViT_{256}\textrm{-}16}$, which can be used as a general feature extractor for $256 \times 256$ histology patches. Accordingly, we perform a model audit that assesses the quality of $\textbf{x}_{256}$ representations for our pan-cancer $\operatorname{ViT_{256}\textrm{-}16}$ model (denoted as $\operatorname{ViT_{256}\textrm{-}16}_{\text{PF, PANC, S1}}$), and compare with three other embedding types: 1) Pretrained ImageNet (IN) features from a truncated ResNet-50 (after the 3rd residual block, or $\text{ResNet-50}_{\text{B3, IN}}$), 2) $\operatorname{ViT_{256}\textrm{-}16}_{\text{PF, BRCA, S1}}$ features trained on only data from TCGA-BRCA (as an organ-specific comparison using the same hyper-parameters and \# of iteration), and 3) $\operatorname{ViT_{256}\textrm{-}16}_{\text{PF, PANC, S4}}$ features trained on pan-cancer data from the TCGA (but with features concatenated across the last four hidden layers, denoted as "S4", versus just the last stage, "S1"). We assess these representations quantitatively using KNN evaluation for slide-level tasks (Table~\ref{tab:supp_class}) and most patch-level tasks (using global mean pooling (GMP), Table~\ref{tab:supp_kather}), as well as qualitatively using UMAP scatter-plots. Description of patch-level datasets are found below:

\input{fig/umap}

\begin{itemize}
    \item \textbf{CRC-100K:} CRC-100K is a dataset of 100,000 histological images of human colorectal cancer and healthy tissue, extracted as $224 \times 224$ patches at $20\times$ magnification (available with and without Macenko normalization), and is annotated with the following non-overlapping tissue classes: adipose (Adi), background (Back), debris (Deb), lymphocytes (Lym), mucus (Muc), smooth muscle (Mus), normal colon mucosa (Norm), cancer-associated stroma (Str), colorectal adenocarcinoma epithelium (Tum)~\cite{kather2016multi}. We experiment on CRC-100K with and without Macenko Normalization (denoted with ``-N" and ``-R" respectively). We report multiclass AUC performance.
    \item \textbf{BCSS}: The Breast Cancer Semantic Segmentation (BCSS) Dataset is a dataset that contains over 20,000 segmentation annotations from the TCGA-BRCA cohort, from which we mined $256 \times 256$ patches at $20\times$ magnification for the following overlapping cell types: background tissue, tumor cells, stroma cells, and lymphocyte infiltrates~\cite{amgad2019structured}. Unlike CRC-100K, BCSS overlaps in label categories as tissue patches can have multiple labels of each cell type. As a result, we used the majority cell phenotype as the patch-level label during supervision. We report multiclass AUC performance.
    \item \textbf{BreastPathQ:} BreastPathQ is a challenge dataset from the TCGA-BRCA cohort that measures tumor cellularity, which measures the fractional occupancy of tumor cell presence in the image patch~\cite{petrick2021spie}. We evaluated on the public train/validation split of the challenge, which provides 2579 and 187 patches respectively at $20\times$, and report mean-squared error (MSE) using linear regression.
\end{itemize}

\paragraph{Comparison with ImageNet Features} In comparing $\operatorname{ViT_{256}\textrm{-}16}_{\text{PF, PANC, S1}}$ with $\text{ResNet-50}_{\text{B3, IN}}$, on both patch- and slide-level evaluation using KNN, we observe that $\operatorname{ViT_{256}\textrm{-}16}_{\text{PF, PANC, S1}}$ features are generally more robust. On patch-level classification, $\text{ResNet-50}_{\text{B3, IN}}$ and $\operatorname{ViT_{256}\textrm{-}16}_{\text{PF, PANC, S1}}$ do equally well on CRC-100K-N (MNIST equivalent of patch-level tasks in CPATH), with $\operatorname{ViT_{256}\textrm{-}16}_{\text{PF, PANC, S1}}$ performing better on BCSS and BreastPathQ (more challenging datasets with noisy and fine-grained labels respectively). $\operatorname{ViT_{256}\textrm{-}16}_{\text{PF, PANC, S1}}$ also performed better on CRC-100K-R, in which images are not stain normalized and thus have more variation due to institution-specific staining protocols. One hypothesis for the surprisingly robust $\text{ResNet-50}_{\text{B3, IN}}$ features is that the feature maps before the last residual block are low-level feature descriptors, and are able to distinguish between distinct morphologies such as tumor versus stroma, or tumor versus adipose tissue. In visualizing UMAP scatter plots of pre-extracted $\text{ResNet-50}_{\text{B3, IN}}$ features, despite the high AUC performance on CRC-100K-R and CRC-100K-N, the representation quality is poor as global structures within the same class types are not preserved (Figure \ref{fig:umap}). On the other hand, global structures for classes such as stroma, tumor, normal, and mucous tissue are well-preserved for all ViT models.

In slide-level KNN evaluation, interestingly, we see that mean $\text{ResNet-50}_{\text{B3, IN}}$ features out-perform mean $\operatorname{ViT_{256}\textrm{-}16}_{\text{PF, PANC, S1}}$ features on NSCLC and RCC subtyping, with $\operatorname{ViT_{256}\textrm{-}16}_{\text{PF, PANC, S1}}$ performing better on BRCA subtyping. This result can be attributed to NSCLC and RCC subtyping being generally easier tasks in which subtypes can be more readily distinguished, whereas BRCA subtyping being more challenging due to the phenotypic similarity of IDC to ILC that typically requires stroma context.

\input{fig/cellseg}

\paragraph{Organ-Specific versus Pan-Cancer Image Pretraining} In comparing $\operatorname{ViT_{256}\textrm{-}16}_{\text{PF, PANC, S1}}$ with $\operatorname{ViT_{256}\textrm{-}16}_{\text{PF, BRCA, S1}}$, briefly, we note that both models have similar performance across most patch-level tasks, with $\operatorname{ViT_{256}\textrm{-}16}_{\text{PF, PANC, S1}}$ performing slightly better on BCSS and BreastPathQ evaluation. In examining global structure of morphological subtypes in CRC-100K, despite not being pretrained on CRC data, $\operatorname{ViT_{256}\textrm{-}16}_{\text{PF, BRCA, S1}}$ is able to preserve global structure better in stroma and mucous subtypes. In Figure~\ref{fig:cellseg}, we additionally visualize the cell localization results of each ViT model, with $\operatorname{ViT_{256}\textrm{-}16}_{\text{PF, PANC, S1}}$ performing overall better in localizing tumor cells and lymphocytes, with $\operatorname{ViT_{256}\textrm{-}16}_{\text{PF, BRCA, S1}}$ attending more to blood cells.

\paragraph{Feature Concatenation across $\operatorname{\mathbf{ViT_{256}\textrm{-}16}}$ Hidden Layers} Following Caron \etal, we also concatenated the $\operatorname{[CLS]}$ tokens from the last four stages of $\operatorname{ViT_{256}\textrm{-}16}$, resulting in a 1536-dim embedding~\cite{caron2021emerging}. We observe no improvement in performing feature concatenation.

\paragraph{Concluding Remarks on $\textbf{x}_{256}$ Representations} In addition to having strong representation quality and interpretability mechanisms in finding unique morphological features, a key detail about $\operatorname{ViT_{256}\textrm{-}16}$ is that the embedding dimension is relatively small (with a length of 384), which allows Stage 2 Hierarchical Pretraining and finetuning of $\operatorname{ViT_{4096}\textrm{-}256}$ to be tractable on commercial workstations. Though longer embedding dimensions may lead to better performance on some patch-level tasks, our ultimate goal is in building hierarchical models for slide-level representation learning, which relies on: 1) shorter embedding dimensions, and 2) $[16 \times 16]$ cell-level interpretability.

\section{Additional Visualizations}

Additional visualizations for hierarchical attention maps are shown in Figure~\ref{fig:brca},~\ref{fig:crc}. Attached in is also a hierarchical attention map for Figure 3 in its native $4096 \times 4096$ resolution. Due to space constraints, we created a repository to visualize $\operatorname{ViT_{256}\textrm{-}16}$, $\operatorname{ViT_{4096}\textrm{-}256}$, and hierarchical attention heatmaps at the following link address: \href{https://bit.ly/HIPT-Supplement}{https://bit.ly/HIPT-Supplement}.

\input{fig/brca}

\input{fig/crc}
\section{Introduction}
\label{sec:intro}
\input{fig/teaser}
Tissue phenotyping is a important problem in computational pathology that aims at characterizing objective, histopathologic biomarkers within the tumor-immune microenvironment for cancer diagnosis, prognosis, and the estimation of response-to-treatment in patients \cite{ludwig2005biomarkers, kather2016multi, javed2020cellular}.
The current clinical paradigm is the manual and subjective interpretation of histology whole-slide images (WSI) in determining clinical endpoints such as cancer subtype, grade, and stage~\cite{amin2017eighth}.

The successes of deep learning have found their way to computational histopathology, in which frameworks such as multiple instance learning (MIL) have been used to build predictive models of clinical outcomes using WSI~\cite{hou2016patch, courtiol2019deep, wulczyn2020deep, van2019no, tellez2019neural, ilse2018attention, campanella2019clinical, lu2020data}. Such frameworks follow three steps. First, instance-level patch representations are extracted from non-overlapping tissue patches in the WSI (typically using variations of ResNet models. Second, global pooling is applied over the bag of patches to construct a WSI-level representation which forms the input to different classifiers~\cite{krizhevsky2012imagenet, ilse2018attention}. Finally, a classifier is trained on the WSI-level representation to predict outcomes of interest.

This three stage process has a few important limitations. First, a pre-trained ResNet model may not capture the right representation of the tumor microenvironment. Second, although clinician annotations are the gold standard in taxonomic classifications and staging systems for many cancer types, they have been demonstrated to suffer from large inter- and intra-observer variability~\cite{nicholson2001reproducibility, rabe2019interobserver, echle2020bjc}. e.g. the substantial variability in discerning well-to-poorly differentiated glands in the Gleason grading system for prostate adenocarcinoma, or the evaluation of nuclear size, shape, and nucleolar prominence in the Furhman grading system for renal cell carcinoma~\cite{carlson1998accuracy, novara2007grading}. Recent work has shown that self-supervised learning can serve as an effective strategy to improve upon the representations learned from a ResNet encoder \emph{without} the use of labels\cite{ciga2021self, saillard2021self, dehaene2020self, koohbanani2021self, boyd2021self, li2021dual}.
In this work, we push the limits of self-supervised learning by leveraging structure inherent to histopathological tissue samples to develop new models and algorithms for self-supervised learning closer to the whole-slide level.

A pathologist scanning a WSI will scan the sample at multiple different resolutions with each resolution presenting the clinician with information about cellular abnormalities, spatial organization of cells and tissue level phenotypes.
Figure \ref{fig:hier_wsi} depicts the organization of information at various levels of granularity within a WSI.
WSIs scanned at $20\times$ objective typically use a fixed scale of $0.5 \mu m$ per pixel. This has an important implication: across different WSIs, the fixed scale enables consistent comparison of visual elements that may elucidate important histomorphological features. WSIs also exhibit a hierarchical structure of visual tokens at varying image resolutions at $20\times$ magnification: the $16 \times 16$ images encompass the bounding box of cells and other fine-grained features (stroma, tumor cells, lymphocytes), $256 \times 256$ images capture local clusters of cell-to-cell interactions (tumor cellularity), $1024 \times 1024$-$4096 \times 4096$ images further characterize macro-scale interactions between clusters of cells and their organization in tissue (the extend of tumor-immune localization in describing tumor-infiltrating versus tumor-distal lymphocytes), and finally the overall intra-tumoral heterogeneity of the tissue microenvironment depicted at the slide-level of the WSI~\cite{beck2011systematic, sertel2009histopathological, javed2020cellular, zhu2017wsisa, hou2019sparse, Diao2021, saltz2018spatial}. The hypothesis that this work tests is that the judicious use of this hierarchy in self-supervised learning results in better whole slide level representations.

To operationalize our insight, we introduce a Transformer-based architecture for hierarchical aggregation of visual tokens and pretraining in gigapixel pathology images, called Hierarchical Image Pyramid Transformer (HIPT). We approach the task of slide level representation learning in a manner similar to the construction of representations of documents in language modeling~\cite{yang2016hierarchical, zhang2019hibert}. Specifically, similar to how hierarchical attention networks for long document classification aggregate word embeddings within sentences to form sentence-level embeddings and subsequently into document-level embeddings, we develop a three-stage hierarchical architecture that performs similar bottom-up aggregation from [$16 \times 16$] visual tokens in their respective $256 \times 256$ and $4096 \times 4096$ windows to eventually form the slide-level representation, as demonstrated in Figure \ref{fig:overview}. While patching in WSIs can result in variable sequence lengths, we choose a large image resolution to patch at first ($\textit{e.g.} - [4096 \times 4096$]) so that further patching within these image regions into $[256 \times 256]$ tokens (and subsequently into $[16 \times
16]$ tokens) would have constant sequence lengths.
In summary, the key contributions of our work are as follows.
\begin{enumerate}
    \item We propose the Hierarchical Image Pyramid Transformer (HIPT); a model capable of learning meaningful representations of high-resolution images.
    \item We apply HIPT to the task of learning representations of gigapixel histopathological images extracted at $20\times$ resolution. We show that our method achieves superior performance to conventional MIL approaches. The difference is pronounced in context-aware tasks such as survival prediction in which larger context is appreciated in characterizing broader prognostic features in the tumor microenvironment~\cite{thorsson2018immune, saltz2018spatial, abduljabbar2020geospatial, chen2021multimodal}. Using K-Nearest Neighbors on the $4096 \times 4096$ representations of our model, we outperform several weakly-supervised architectures in slide-level classification -- an important step forward in achieving self-supervised slide-level representations.
    \item Akin to self-supervised ViTs on natural images that encode semantic segmentation of the scene layout, we find that the multi-head self-attention in self-supervised ViTs learns \emph{disentagled} representations of histology images as unique visual concepts (from fine-grained visual concepts such as cell locations in the $\operatorname{ViT-16}$ to coarse-grained visual concepts such as broader tumor cellularity in the $\operatorname{ViT-256}$), as demonstrated in Figure \ref{fig:attention1}.
    \item We make all code and pretrained weights publicly available at \href{https://github.com/707884/707884}{https://github.com/707884/707884}.
\end{enumerate}


\section{Introduction}

Deep neural network (DNN) inference is increasingly being executed on mobile and embedded platforms due to low latency and better privacy. However, efficient deployment on these platforms is challenging due to the intensive computation and memory access.

Static model pruning is an effective method to reduce the model parameters (i.e. computation demand) at the cost of accuracy loss. Platform-aware model pruning \cite{yang2018netadapt} gradually reduces the number of parameters while measuring the latency, and stops pruning once the target latency is achieved in order to keep the accuracy as high as possible. However, the latency measurement is based on a fixed hardware configuration (e.g. CPU/GPU at maximum clock frequency), which is often unavailable at runtime due to thermal throttling or other concurrently executing applications sharing the hardware. The target latency could also change during different execution phases of the same application \cite{xun2020optimising}. Although a conservative hardware configuration can guarantee hardware availability at runtime, more parameters need to be pruned to achieve the same latency target, leading to a fixed model with lower accuracy not fully utilising available hardware.

Unlike static model pruning, which can only fit into a fixed latency target and hardware configuration, dynamic DNNs are trained to have sub-networks with different latency-accuracy trade-offs through either channel scaling \cite{yu2018slimmable, xun2019incremental} or layer scaling \cite{laskaridis2020hapi}. For example, smaller sub-networks are faster but less accurate than larger ones. At runtime, dynamic DNNs can change the architecture among their sub-networks to comply with new latency targets and dynamically available hardware resources. However, existing works treat dynamic DNNs as an algorithm-only approach, without considering hardware trade-off opportunities. This limits the trade-off range and granularity in latency, power and energy. Furthermore, standalone channel/layer scaling cannot fully utilise heterogeneous computing resources on modern mobile/embedded platforms, and their application is limited to ConvNet \cite{lou2021dynamic}.

We propose a holistic system design for DNN performance and energy optimisation, combining the trade-off opportunities in both algorithms and hardware. As shown in Fig \ref{fig1}, a system can be viewed as three abstract layers: the device layer contains heterogeneous computing resources; the application layer has multiple concurrent workloads; and the runtime resource management layer monitors the dynamically changing algorithms' performance targets as well as hardware resources and constraints, and tries to meet them by tuning the algorithm and hardware at the same time. Moreover, We illustrate the runtime approach through a dynamic version of `once-for-all network \cite{cai2019once}' (namely Dynamic-OFA), which can scale the ConvNet architecture to fit heterogeneous computing resources efficiently \cite{lou2021dynamic} and has good generalisation for different model architectures such as Transformer \cite{parry2021dynamic}. Compared to the state-of-the-art Dynamic DNNs, our experimental results using ImageNet on a Jetson Xavier NX show that the Dynamic-OFA is up to 3.5x (CPU), 2.4x (GPU) faster for similar ImageNet Top-1 accuracy, or 3.8\% (CPU), 5.1\% (GPU) higher accuracy at similar latency. Furthermore, compared with Linux governor (e.g. performance, schedutil), our runtime approach reduces the energy consumption by 16.5\% at similar latency.

\begin{figure}[tbp]
\centerline{\includegraphics[width=\columnwidth]{RTM.pdf}}
\caption{A holistic system design for DNN performance and energy trade-offs. Algorithm knobs (e.g. dynamic DNNs) and hardware knobs (e.g. task mapping, dynamic voltage and frequency scaling (DVFS)) are combined to meet algorithms' dynamically changing performance targets (e.g. accuracy, latency) as well as hardware resources (e.g. available cores, clock frequency level) and constraints (e.g. power, temperature).}
\label{fig1}
\end{figure}