{'fraction_non_alphanumeric': 0.03581733784970518, 'fraction_numerical': 0.012043658261196839, 'mean_word_length': 5.0071590052750565, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Bilateral-ViT for Robust Fovea Localization

The fovea is an important anatomical landmark of the retina. Detecting the location of the fovea is essential for the analysis of many retinal diseases. However, robust fovea localization remains a challenging problem, as the fovea region often appears fuzzy, and retina diseases may further obscure its appearance. This paper proposes a novel Vision Transformer (ViT) approach that integrates information both inside and outside the fovea region to achieve robust fovea localization. Our proposed network, named Bilateral-Vision-Transformer (Bilateral-ViT), consists of two network branches: a transformer-based main network branch for integrating global context across the entire fundus image and a vessel branch for explicitly incorporating the structure of blood vessels. The encoded features from both network branches are subsequently merged with a customized Multi-scale Feature Fusion (MFF) module. Our comprehensive experiments demonstrate that the proposed approach is significantly more robust for diseased images and establishes the new state of the arts using the Messidor and PALM datasets.


I. INTRODUCTION
The macula is the central region of the retina. The fovea is an important anatomical landmark located in the center of the macula, responsible for the most crucial part of a person's vision [1]. The severity of vision loss due to retinal diseases is usually related to the distance between the associated lesions and the fovea. Therefore, detecting the location of the fovea is essential for the analysis of many retinal diseases.
Despite its importance, robust fovea localization remains a challenging problem. The color contrast between the fovea region and its surrounding tissue is poor, leading to a fuzzy appearance. Furthermore, the fovea appearance may be obscured by lesions in the diseased retina; for example, geographic atrophy and hemorrhages significantly alter the fovea appearance. Such issues make it more difficult to perform localization based on the fovea appearance alone. Fortunately, anatomical structures outside the fovea region, such as blood vessels, are also helpful for localization [2], [3]. For this reason, we propose a novel Vision Transformer (ViT) approach that integrates information both inside and outside the fovea region to achieve robust fovea localization.
Our proposed network, named Bilateral-Vision-Transformer (Bilateral-ViT), consists of two network branches. We adopt a transformer-based U-net architecture [4] as the main branch for effectively integrating global context across the entire fundus image. In addition, we design a vessel branch that takes in a blood vessel segmentation map for explicitly incorporating the structure of blood vessels. Finally, the encoded features from both network branches are merged with a customized Multi-scale Feature Fusion (MFF) module, leading to significantly improved performance. Thus, our key contributions are as follows: • We propose a novel vision-transformer-based network architecture, that explicitly incorporates global image context and structure of blood vessels, for robust foveal localization. • We demonstrate that the proposed approach is significantly more robust for challenging settings such as fovea localization in diseased retinas (over 9% improvements for specific evaluations). It also has a better generalization capability compared to the baseline methods, as shown in cross-dataset experiments. • We establish the new state of the arts on both the Messidor and PALM datasets.

II. RELATED WORK
Earlier work usually utilize hand-craft features to encode anatomical relationships among optic discs (OD), blood vessels, and fovea regions for fovea localization. Deka et al. [5] and Medhi et al. [6] generate the region of interest (ROI) using processed blood vessels for macula estimation. Certain methods utilize OD in the prediction of ROI and fovea center by selecting specific OD diameters [7], estimating OD orientations and minimum intensity values [8], [9]. Other applications use combined OD and blood vessels features to improve the performance of fovea localization [2], [3]. These methods generally perform less competitively than more recent deep-learning-based approaches.
Many deep learning-based methods formulate the fovea localization as a regression task [10]- [13]. Some methods utilize retinal structures, such as OD and blood vessels, as constraints for inferring the location of the fovea. For example, Meyer et al. [11] adopt a pixel-wise distance regression approach for joint OD and fovea localization. Besides the regression-based approaches, Sedai et al. [14] propose a two-stage image segmentation framework for segmenting the image region around the fovea. Our work also belongs to the image segmentation paradigm [4], [14]- [17]. Unlike all previous works, we customize the recent transformer-based segmentation network [4] to incorporate blood vessel information and demonstrate its superior performance compared to the existing approaches.

A. Network Architecture
The overall architecture of Bilateral-ViT is illustrated in Fig. 1. The proposed Bilateral-ViT is based on a U-shape architecture with a vision transformer-based encoder (the main branch) for exploiting long-range contexts. In addition, we design a vessel branch to encode structure information from blood vessel segmentation maps. Finally, Multi-scale Feature Fusion (MFF) blocks are designed to effectively fuse data from the main and vessel branches.
Main Branch. We adopt the TransUNet [4] as the main branch due to its superior performance on other medical image segmentation tasks. In the main branch, we utilize a CNN-Transformer hybrid structure as the encoder. The CNN part is used as the initial feature extractor. It provides features at different scales for the skip connections to compensate for the information loss in the downsampling operation. The extracted features are then processed by 12 consecutive transformer blocks at the bottleneck of the UNet architecture. The transformer encodes the long-range dependencies of the input fundus image due to the multi-head self-attention structure. The output features of the last transformer block are then resized for later decoding operations. Vessel Branch. In the vessel branch, we aim to exploit the structure information from the blood vessels. Unlike the main branch, where the input is a fundus image, we put in a vessel segmentation map generated by a pre-trained model. The pretrained vessel segmentation model is built on the DRIVE dataset [18] with the TransUNet [4] architecture. Four identical Spatial Information Guidance (SIG) blocks are utilized in the vessel branch to extract multi-scale vessel-based features. The rescaled vessel segmentation maps are fed into the SIG blocks, the details of which are illustrated in Fig. 2-a. The design of the SIG blocks makes extensive use of customized ReSidual U-blocks (RSU). Qin et al. [16] indicate that the RSU block is superior in performance to other embedded structures (e.g., plain convolution, residual-like, inception-like, and dense-like blocks), due to the enlarged receptive fields of the embedded U-shape architecture.
Multi-scale Feature Fusion (MFF) blocks. In contrast to the plain convolutional decoder blocks of the basic TransUNet, we use three Multi-scale Feature Fusion (MFF) blocks as the decoders for effective multi-scale feature fusion. The input to each MFF block is the concatenation of three types of features: (1) the multi-scale skip-connection features from the main branch, (ii) the hidden feature encoded by the last transformer block or the previous MFF block, (iii) the multi-scale SIG features from the vessel branch. The architecture of the MFF blocks is illustrated in Fig. 2-b, which is similar to one of the SIG blocks. From MFF block 1 to MFF block 3, we gradually increase the number of network layers in each MFF block. In this way, the later MFF blocks can capture more spatial context corresponding to larger feature maps. In the end, the concatenated feature maps of MFF block 3 and SIG block 4 are passed to two convolutional layers for outputting the fovea region score maps.

B. Implementation Details
We first remove the uninformative black background from the original fundus image, then pad and resize the cropped image region to a spatial resolution of 512 × 512. We perform intensity normalization and data augmentation on the input images of the main branch and the vessel branch. To train our Bilateral-ViT network, we generate circular fovea segmentation masks from the ground-truth fovea coordinates. During the testing phase, we apply the sigmoid function to network prediction for the probabilistic map. We then collect all pixels with significant probabilistic scores and calculate their median coordinates as the final fovea location coordinates. All experiments were coded using PyTorch and conducted on one NVIDIA GeForce RTX TITAN GPU. The weights of convolutional and linear layers were initialized by Kaiming initialization protocol [19]. The initial learning rate was 1e −3 which gradually decays to 1e −7 over 200 epochs using the Cosine Annealing LR strategy. The optimizer was Adam [20] and the batch size 2. We employed a combination of dice loss and binary cross-entropy as the loss function.

IV. EXPERIMENTS
We performed experiments using the Messidor [21] and PALM [22] datasets. The Messidor dataset is for diabetic retinopathy analysis. It consists of 540 normal and 660 diseased retinas. We utilized 1136 images from this dataset with fovea locations provided by [23]. The PALM dataset was released for the Pathologic Myopia Challenge (PALM) 2019. It consists of 400 images annotated with fovea locations, in which 213 images are pathologic myopia, and the remaining 187 images are normal retinas. For fairness of comparison, we keep our data split identical to [13].
To evaluate the performance of fovea localization, we adopt the following evaluation protocol [23]: the fovea localization is considered successful when the Euclidean distance between the ground-truth and predicted fovea coordinates is no larger than a predefined threshold value, such as the optic disc radius R. For a comprehensive evaluation, accuracy corresponding to different evaluation thresholds (for example, 2R indicating the predefined threshold values are set to twice the optic disc radius R) is usually reported.

A. Fovea Localization on Normal and Diseased Images
In Table I, we evaluate the performance of normal and diseased cases separately. We reimplement several widely used segmentation networks as comparison baselines, such as UNet [15], U2 Net [16], and TransUNet [4]. Bilateral-ViT obtains 100% accuracy from 1/2R to 1R on all the Messidor images, and 100% accuracy from 2/3R to 1R on the normal PALM images. Thus demonstrating that the performance of Bilateral-ViT is highly reliable for normal fundus images. For the diseased cases in the PALM dataset, Bilateral-ViT reaches 92.45% foveal localization accuracy for the threshold of 1R and significantly outperforms the second-best results by a large margin (7.54%). Fig. 3 provides some visual results of fovea localization on diseases images from the PALM dataset. Our Bilateral-ViT generates the most accurate predictions for the severely diseased images with large atrophic regions (see Fig. 3-a and Fig. 3-b), or the heavily blurred image (see Fig. 3-c). In Fig. 3-d where the fovea is close to the image border, the predicted fovea locations from baseline networks (UNet and U2 Net) appear on the wrong side of the optic disc. However, TransUNet [4] and our method still perform well, potentially due to their long-range modeling capability. Such results highlight that our proposed Bilateral-ViT has a significant advantage for diseased cases.

B. Comparison with State-of-the-Art Methods
From Table II, the Bilateral-ViT achieves state-of-the-art performance for all the evaluation settings. In particular, on the Messidor dataset, at 1/8R, our network reaches the best accuracy of 85.65% with a gain of 1.84% compared to the second-best score (83.81%) [13]. It also reaches an accuracy of 100% at evaluation thresholds of 1/2R, 1R, and 2R; in other words, the localization errors are at most 1/2R (approximately 19 pixels for an input image size of 512 × 512). PALM is a considerably more challenging dataset due to fewer images and complex diseased patterns. Our method achieved accuracies of  90% and 96% at 2/3R and 1R, which is 3% and 2% better than the previous work [13], respectively.

C. Ablation Study and Cross-Dataset Experiments
We conducted a comprehensive set of ablation experiments to evaluate the effectiveness of different components (see Table III): • ViT+plain decoder: the TransUNet architecture [4] comprised of a vision transformer-based encoder and a plain decoder used as the comparison baseline. • ViT+VB+plain decoder: we add the vessel branch (vessel segmentation mask as the input) to the baseline network. • ViT+VB+MFF (the proposed Bilateral-ViT): we add the vessel branch (vessel segmentation mask as the input) and MFF blocks to the baseline network. • ViT+VB (fundus as the input)+MFF: we add the vessel branch (fundu image as the input) and MFF blocks to the baseline network. This configuration compares the performance differences between fundus images and vessel segmentation maps as inputs to the vessel branch. The performance of "ViT+plain decoder (TransUNet)" and "ViT+VB+plain decoder" are similar on both datasets; a possible reason is that the plain decoder does not have adequate capacity to fuse features from the vessel branch and transformer blocks. By further adding MFF blocks, the proposed Bilateral-ViT (ViT+VB+MFF) shows superior performance, suggesting the significance of the customized MFF blocks. The performance of "ViT+VB+MFF' is much better than "ViT+VB (fundus as the input)+MFF", demonstrating the usefulness of the vessel segmentation map. On the other hand, we note that "ViT+VB (fundus as the input)+MFF" outperforms all the existing works, implying our network can achieve the state-of-the-art performance even without the input of a vessel segmentation map. We conducted cross-dataset experiments to assess the generalization capability of the proposed Bilateral-ViT. The models were trained on the PALM dataset and tested on the Messidor dataset. From Table IV, the accuracy is 99.38% at 1R, which is a 4.12% improvement over the best-reported result (95.26%). The average localization error for the original image resolution is 8.57 pixels compared to the previous best result of 22.84 pixels. In addition, the proposed Bilateral-ViT outperforms the baselines by a significant margin, especially for 1/8R, demonstrating its robustness for the cross-dataset setting.

V. CONCLUSIONS
This paper proposes a novel Vision Transformer (ViT) approach for robust fovea localization. It consists of a transformer-based main network branch for integrating global context and a vessel branch for explicitly incorporating the structure of blood vessels. The encoded features are subsequently merged with a customized Multi-scale Feature Fusion (MFF) module. Our experiments demonstrate that the proposed approach has a significant advantage in handling diseased images. It also has excellent generalization capability, as shown in the cross-dataset experiments. Thanks to the transformer-based feature encoder, the incorporation of blood vessel structure, and the carefully designed MFF module, our approach establishes the new state of the arts on both Messidor and PALM datasets.

VI. COMPLIANCE WITH ETHICAL STANDARDS
This research study was conducted retrospectively using human subject data available in the open access Messidor and PALM datasets. Ethical approval was not required as confirmed by the license attached with the open access data.