Light-weight Retinal Layer Segmentation with Global Reasoning
Xiang He Weiye Song Yiming Wang Fabio Poiesi Ji Yi Manishi Desai Quanqing Xu Kongzheng Yang Yi Wan




    


Light-weight Retinal Layer Segmentation with Global Reasoning
    Xiang He, Weiye Song, Yiming Wang, Fabio Poiesi, Ji Yi, Manishi Desai, Quanqing Xu, Kongzheng Yang, and Yi Wan
This work was supported in part by the National Natural Science Foundation of China under Grant 51975336, and 62205181, in part by the Natural Science Foundation of Shandong Province under Grant ZR2022QF017, in part by the Natural Science Outstanding Youth Fund of Shandong Province under Grant 2023HWYQ-023, in part by the Taishan Scholar Foundation of Shandong Province under Grant tsqn202211038, in part by the NIH under Grant R01NS108464, in part by the Key Technology Research and Development Program of Shandong Province under Grant 2020JMRH0202 and 2022CXGC020701, in part by the Shandong Province New Old Energy Conversion Major Industrial Tackling Projects under Grant 2021-13, in part by the Key Research and Development Project of Jining City under Grant 2021DZP005, and in part by the Shandong University Education Teaching Reform Research Project under Grant 2022Y133, 2022Y124, and 2022Y312. (Corresponding author: Yi Wan.)
Xiang He is with the School of Mechanical Engineering, and also with the Joint SDU-NTU Centre for Artificial Intelligence Research (C-FAIR), Shandong University, Jinan, Shandong, China (e-mail: he_xiang@mail.sdu.edu.cn).
Weiye Song, Quanqing Xu, Kongzheng Yang, and Yi Wan are with the School of Mechanical Engineering, Shandong University, Jinan 250061, Shandong, China (e-mail:  songweiye@sdu.edu.cn; 202214333@mail.sdu.edu.cn; 202234432@mail.sdu.edu.cn; wanyi@sdu.edu.cn).
Yiming Wang, Fabio Poiesi are with the Fondazione Bruno Kessler, Via Sommarive 18, Povo, TN 38123, Italy (e-mail: ywang@fbk.eu; poiesi@fbk.eu).
Ji Yi is with the Department of Biomedical Engineering and the Department of Ophthalmology, Johns Hopkins University, Baltimore, Maryland 21231, USA (e-mail: jiyi@jhu.edu).
Manishi Desai is with the Department of Ophthalmology, Boston University School of Medicine, Boston Medical Center, Boston 02118, USA (e-mail: madesai@bu.edu).
    April 29, 2024
=====================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================




Automatic retinal layer segmentation with medical images, such as optical coherence tomography (OCT) images, serves as an important tool for diagnosing ophthalmic diseases. However, it is challenging to achieve accurate segmentation due to low contrast and blood flow noises presented in the images. 
In addition, the algorithm should be light-weight to be deployed for practical clinical applications. Therefore, it is desired to design a light-weight network with high performance for retinal layer segmentation. In this paper, we propose  for retinal layer segmentation which can be applied to OCT images. Specifically, our approach follows an encoder-decoder structure, where the encoder part employs multi-scale feature extraction and a Transformer block for fully exploiting the semantic information of feature maps at all scales and making the features have better global reasoning capabilities, while the decoder part, we design a multi-scale asymmetric attention (MAA) module for preserving the semantic information at each encoder scale. The experiments show that our approach achieves a better segmentation performance compared to the current state-of-the-art method TransUnet with 105.7M parameters on both our collected dataset and two other public datasets, with only 3.3M parameters.



retinal layer segmentation, light-weight, multi-scale asymmetric attention, visible-light OCT.




§ INTRODUCTION


With the accelerated pace of lifestyle and the popularity of electronic products, many people lack awareness of eye protection, leading to an increasing incidence of retinal diseases. Many retinal diseases are accompanied by changes in the thickness of the retinal layers. For instance, in patients with glaucoma, the retinal nerve fiber layer is thinner and the optic disc is atrophied compared to the healthy eyes <cit.>. In diabetic retinopathy, the macular region of the retina thickens and becomes edematous <cit.>. 
Current research indicates that the changes in the retinal layer begin to occur in the early stages of the disease<cit.>, therefore, automatic analysis of the retinal layers and monitoring their morphological changes can help to understand the disease progression and provide early treatment.

OCT can be used for non-invasive 3D structural imaging of the retina <cit.>, with which experienced ophthalmologists can manually inspect the retinal layers to identify the disease progression. However, manual inspection is inefficient and tedious. It is also challenging for various levels of ophthalmologists to ensure consistency and objectivity of the inspection. Although imaging techniques with a higher resolution are available now, such as Visible-light OCT images <cit.>, this does not address completely the above status quo. Recent convolutional neural networks (CNNs), as typified by U-net <cit.>, have achieved promising results regarding semantic segmentation in the field of medical image analysis. The seminar work ReLayNet <cit.> has triggered widespread interest in applying CNNs to segment retinal layers. ReLayNet follows a multi-scale encoder-decoder structure, which has inspired many mainstream semantic segmentation frameworks to adopt, such as U-net <cit.> and Attention_Unet <cit.>. 
However, such structure can be limited in two main aspects. Firstly, most U-shape encoder-decoder structures perform the multi-scale feature extraction followed by feature fusion via only residual connections. Thus, it is limited to maintain and fully exploit the semantic information of feature maps at all scales, resulting in false positives in the background region when performing retinal layer segmentation. 
Secondly, many of these methods do not pay attention to the actual application needs of OCT devices, often pursuing accuracy unilaterally, for example, such multi-scale reasoning often comes at the cost of a large number of parameters, which can be computationally demanding for the deployment of real-time clinical applications. 

To overcome the above-mentioned limitations, we propose , a novel multi-scale encoder-decoder network for end-to-end retinal layer segmentation, which is a light-weight segmentation approach tailored to practical OCT device requirements. 
In order to reduce segmentation errors in the background region, we propose to incorporate a Transformer block <cit.> to features with global reasoning and to introduce a multi-scale asymmetric attention (MAA) module to better preserve the semantic information at each encoder scale. 
To achieve computation efficiency, both the backbone and the MAA module adopt light-weight feature extractors, such as depthwise separable convolution and asymmetric convolution <cit.>.  achieves the state-of-the-art segmentation accuracy with only 3.3M parameters, a significantly light model, as shown in Fig. <ref>.





In summary, our contributions are: 

    
  * We propose , a novel encoder-decoder structure for retinal layer segmentation, which exploits global information in a light-weight manner, to improve the segmentation accuracy with reduced computational complexity, it provides valuable experience for algorithms designed for practical applications in medical devices. 
    
  * We propose a novel attention module, the MAA module, to jointly work with Transformer, in order to address erroneous segmentation in the background area by allowing the model to reason in a global manner.
    
  * We perform our method on the visible-light OCT images for the first time and score the best segmentation performance, it provides experience for the algorithm to perform robustly on datasets from different domains.


This paper is organized as follows:
Sec. <ref> describes the research progress in the field of biological tissue segmentation and retinal layer segmentation. 
Sec. <ref> introduces the overall framework, the feature extractor, MAA module and the light-weight designs of the approach.
Sec. <ref> describes the datasets, performance metrics, implementation details, analyzes quantitatively and qualitatively the experimental results, and performs ablation experiments.
Finally, the conclusions of the paper are given in Sec. <ref>. 






§ RELATED WORK


In this section, we first provide an overview of the research progress of  semantic segmentation, especially light-weight semantic segmentation methods, followed by a more specific coverage of deep-learning-based retinal layer segmentation.



 §.§ Medical Image Segmentation


Medical image segmentation has made rapid progress under the promotion of deep learning, especially in the fields of biological tissue recognition and lesion detection, which has sparked a large amount of research on medical image segmentation. 
The morphology of the optic disc and optic cup, as well as the cup-to-disc ratio, can be used to assess glaucoma, Wang et al. proposed an automatic segmentation approach based on CNNs to accurately segment the optic disc and optic cup from fundus images for glaucoma detection<cit.>. 
Precise segmentation of retinal vessels from fundus images is essential for intervention in numerous diseases and high-precision segmentation of retinal vessels still remains a challenging task, Li et al. proposed a dual-path progressive fusion network, named DPF-Net, which achieved better segmentation ability by effectively fusing global and local features<cit.>.
Abdominal multi-organ image segmentation plays a crucial role in the diagnosis and treatment of many diseases. traditional methods of manual depiction are inefficient and subjective, therefore, Chen et al. proposed TransUnet based on CNN and Transformer block for abdominal multi-organ segmentation<cit.>. TransUnet was the first model to introduce the Transformer block into the U-shaped encoder-decoder structure and achieved very high accuracy in segmentation. 
There are also segmentation studies dedicated to lesion regions, such as Astaraki et al. who segmented different types of lung pathological regions to enable screening for lung diseases<cit.>. 
In addition, some lightweight segmentation methods for practical applications have also been proposed, such as SegFormer<cit.>, A-net<cit.>, EMV-Net<cit.>.





 §.§ Retinal Layer Segmentation

The morphology of the retinal layer is associated with the development of many ophthalmic diseases, therefore, there are many studies on retinal layer segmentation based on CNN and OCT images. The success of ReLayNet opened the door to the application of CNNs for retinal layer segmentation<cit.>, a typical U-shaped network based on an encoder-decoder structure, which achieved the best performance for retinal layer segmentation at that time. Since layer segmentation of OCT retinal images is prone to speckle noise, intensity inhomogeneity, Wang et al. proposed a boundary-aware U-Net for retinal layer segmentation by detecting accurate boundaries<cit.>. 
To solve the topological errors caused by not considering the order of retinal layers, Liu et al proposed a novel deep learning-based framework that employed the distance maps of layer surfaces to convert the layer segmentation task into a multitasking problem for classification and regression<cit.>.
To ensure the continuity of the retinal layer boundary and obtain more accurate segmentation results, Hu et al proposed a coarse-to-fine retinal layer boundary segmentation method based on the embedded residual recurrent network and the graph search, it has achieved good performance in both qualitative and quantitative indicators<cit.>.
In addition, considering the impact of different neurological diseases on the retinal layer, Gende et al presented a fully automatic approach for the retinal layer segmentation in multiple neurodegenerative disorder scenarios<cit.>, the results indicate that the model is more robust.
There are also studies aimed at practical application deployment, such as He et al. proposed a light-weight retinal layer segmentation approach that achieved good performance with a lower number of parameters<cit.>.




§ METHODOLOGY





 §.§ Framework Overview

Our proposed  is a U-shape network based on an encoder-decoder structure as shown in Fig. <ref>. 
The network takes a retinal layer OCT image as input. The first N scale feature maps (F_1, ..., F_N-1, F_N) are extracted by the encoder which consists of a multi-scale feature extractor, respectively. In order to further deepen the global reasoning capability in the depth direction, the feature map F_N, which has to pass through a depth feature encoder: Transformer block<cit.>. The feature map F_TN further encoded by the Transformer block has no change in shape and size, but it facilitates long-range global reasoning. To better preserve details at different scales, multi-scale features (F_1, ..., F_N-1, F_N) are then merged with its corresponding layers in the decoder part via our proposed MAA module, which outputs the feature maps (F_M1, ..., F_MN-1, F_MN) with the same shape size as its input. The F_TN output from the Transformer block is merged with the F_MN for feature fusion, and then after 2 fold size up-sampling, it continues to fuse with F_MN-1 for feature fusion and then follows a similar operation until the fusion of the up-sampled restored feature map with F_out is completed, and after channel adjustment, the final retinal layer segmentation image is output.  is designed for real-time clinical operations, where light-weight designs are employed where possible. 

We present in full the multi-scale encoder in Section <ref> followed by the proposed MAA module that serves for multi-scale feature fusion in Section <ref>. Finally, we describe the design details for computation efficiency in Section <ref>.  




 §.§ Multi-scale Encoder


The multi-scale encoder extracts N scale features to fully exploit the semantic information of feature maps at all scales, further, in order to make the features have better global reasoning capabilities, we incorporate a Transformer block<cit.>, which is a module that further optimizes the Nth scale feature map extracted from the deepest scale of the encoder. To ensure the dimensionality is consistent with the Transformer block port, we pre-process the feature map F_N before being processed by the Transformer block.
The feature map F_N is first converted into a 2D patch sequence as


    F_N⇒ F_N^'={.x_p^k∈ R^P^2× c | k=1,...,Z. } , Z=HW/P^2,

where c is the number of channels in the F_N feature map, P is the size of the patch, and H and W are the height and width of the F_N feature map. 

We use a trainable linear projection to map the vector patch x_p to a latent D-dimensional embedding space. To encode the spatial information of the patches, we learn speciﬁc position embeddings that are added to preserve the positional information in the patch embedding as


    z_0=[x_0 ; x_p^1E; x_p^2E; ⋯ ; x_p^NE]+E_pos,

where E ∈ℝ^(P^2· C) × D represents the patch embedding projection, the x_0 is an additional learnable vector
concatenated with the remaining vectors to integrate the information of all remaining vectors, the [·;⋯;·] is the concatenation operator, and E_pos∈ℝ^(Z+1) × D represents the position embedding<cit.>.






Then, inputting z_0 into the Transformer Layer, as shown in Fig. <ref>, first goes through the first Layer Norm (LN) layer and then enters the Multi-head Self-attention (MSA) layer, here MSA(LN(z_0))+z_0 forms the residual structure and get the z_0^'. Then, after passing through the second LN layer, it enters the Multi-Layer Perceptron (MLP) layer, here MSA(LN(z_0^'))+z_0^' once again forms the residual structure and get the z_1.

At this point, the first Transformer layer is finished. We have set three Transformer layers in our approach, so we have to cycle three times.


    z_L=[x_0^' ; x_p^1'; x_p^2'; ⋯ ; x_p^N']⇒[x_p^1'; x_p^2'; ⋯ ; x_p^N']


When the z_L outputs from the Transformer block, we have to remove the x_0^' vector from the z_L, because only then we can reshape it to the same size of F_N.









 §.§ MAA Module


To better preserve the semantic information at each encoder scale, the MAA module is designed in the skip connection part of the encoder and decoder by a large convolution kernel and channel attention mechanism. It's also set with multi-scale feature fusion, and it's based on the ideas of asymmetric convolution<cit.>, multi-scale feature extraction<cit.> and channel attention mechanism<cit.>. 

    F_0^'=Conv_5× 5 ( F_0 )


    F_0i^'=Conv_ki× 1 [Conv_1× ki ( F_0^' ) ],  k=3, 7, 11


    F_0i^”=CA ( F_0i^' ),



The workflow of the MAA module is illustrated in Fig. <ref>, it begins with the input of a feature map F_0 extracted from the feature encoder, assuming a shape size of (C, H_0, W_0). F_0 enters MAA and goes through a 5× 5 convolution kernel to expand the perceptual field, and then has to enter a multi-scale convolution to extract information at different scales (Eq. <ref>, Eq. <ref>). Each scale goes through a channel attention (CA) module to increase the weight of the feature map on the channel dimension (Eq. <ref>). As shown in the channel attention module in Fig. <ref>, the feature F_0i^'∈ℝ^C× H× W reshape to F_0i_cn^'∈ℝ^C× N, where N = H× W, ∈ represents the shape of the feature map. After that we perform a matrix multiplication between F_0i_cn^' and the transpose of it, get the F_0i_cc^'∈ℝ^C× C. Eventually, we apply the softmax layer to get the channel attention map A∈ℝ^C× C 
    F_0i_cc^'=F_0i_cn^'× (F_0i_cn^' )^T


    A=Softmax (F_0i_cc^' )

(Eq. <ref>, Eq. <ref>). Here the matrix A∈ℝ^C× C contains information about the weights between channels, e.g. (A∈ℝ^C× C )_ij represents the impact of i^th channel on j^th channel. In addition, we multiply the matrix between the transpose of A∈ℝ^C× C and

    A_1∈ℝ^C× N=A^T× F_0i_cn^'


    F_0i^”=α A_1∈ℝ^C× Nreshape→C× H× W+F_0i^'

F_0i_cn^', and reshape the result to ℝ^C× H× W. Then we multiply the result by a scale parameter α and perform an element-wise sum operation with F_0i^' to obtain the final output F_0i^” (Eq. <ref> and Eq. <ref>). After the CA module, all feature maps are summed and convolved by 1× 1 kernel to generate a weighted feature map, and we use the weighted feature map to re-weight F_0 by element-wise multiplying (⊗) it with the input F_0 (Eq. <ref> and Eq. <ref>).

    att=Conv_1× 1 [ ∑_i=1^3F_0i^”+CA ( F_0^' ) ]


    F_M0=att⊗ F_0

At last, we obtain the F_M0. According to the above procedure, we can input the extracted four scales feature maps F_1, ..., F_N-1, F_N into MAA module to get F_M1, ..., F_MN-1, F_MN, respectively, as shown in Fig. <ref>. 

Instead, the ASPP employs a series of dilated convolution layers with different rates to obtain larger receptive fields, however, dilated convolutions may lose local detail information. In contrast, the MAA module utilizes asymmetric convolutions with large kernels, which allows capturing multi-scale features without sacrificing too much detail. Furthermore, before the 1x1 convolution compresses the channels in the MAA module, a channel attention mechanism is adopted, enhancing the module's focus on the channel dimension and better preserving important features.







 §.§ Light-weight Designs



The basic principle of our design of  is to improve the overall performance of the model with as few parameters as possible, so we can meet the computational efficiency of deployments for real-time clinical applications. Thus, we make some light-weight designs for part of the structure of the model. For encoder extraction of multi-scale feature maps, we use depthwise separable convolutions (DS-Conv) for our down-sampling<cit.>. DS-Conv considers the channels and spatial regions of the feature map separately, takes different convolution kernels for different channels, and then uses point convolution to aggregate the channel information, achieving a richer feature representation with fewer parameters to learn. We find that using standard convolution for down-sampling is better than pooling, but convolution increases the number of parameters, so we use DS-Conv to ensure the segmentation performance while decreasing the burden of the number of parameters. In addition, as mentioned in Sec. <ref>, multi-scale asymmetric convolution is used in our MAA module. Initially, our idea is to perform multi-scale extraction of feature information like ASPP <cit.>, but we find that using standard convolution would cause a greater burden on the number of parameters, since the MAA module is supposed to perform extraction of the N scale feature maps, which would certainly further increase the number of parameters. In order to reduce the number of parameters in the MAA module, we use asymmetric convolution instead of standard convolution, shown in Fig. <ref>, and the overall performance of  is nearly the same. In particular, the Transformer block has a significant proportion of the number of parameters in . We have optimized the internal parameters of the Transformer block structure, such as the setting of heads=8 for multiple heads in MSA and dim_head=64 for hidden linear layers in MSA<cit.>, etc. All these designs to minimize the number of parameters are made under the precondition of ensuring the performance of .



§ EXPERIMENT


In this section, we compare our  against the state-of-the-art methods for retinal layer segmentation on i) two publicly available OCT datasets with unhealthy eyes, where the Glaucoma dataset is for the Glaucoma disease and the DME dataset is for Diabetic Macular Edema (DME), and ii) a new dataset, named as Vis-105H, that we create on top of raw images collected from <cit.> which correspond to only healthy human eyes with visible light OCT. Moreover, we present a thorough ablation study with our Vis-105H dataset to validate the effectiveness of the proposed MAA and the introduction of the Transformer block. 



 §.§ Datasets





  
Vis-105H dataset
We create the Vis-105H dataset for the method evaluation which contains images captured using a custom visible light OCT device that takes optic disc cubes from each subject <cit.>, as shown in Fig. <ref>(a). The raw images are originally collected and presented in <cit.>, and they correspond to 21 eyes from 14 healthy subjects, all of whom with written informed consent prior to the participation. The inclusion criteria include i) age 40 years, ii) best corrected visual acuity better than 20/40, iii) no previous intraocular surgery of any kind, and iv) no known retinal disease, resulting in 21 scanning cubes with 5376 B-scan images.
Two professional ophthalmologists further select 5 images with high imaging quality from every 256 B-scans, as shown in Fig. <ref>(b), with the selection criteria including i) low noise, ii) clear retinal layer boundaries, and iii) suitability for ophthalmologists to use for diagnosis. 
With the supervision of medical ophthalmology professionals, we annotate these 105 pre-processed retinal Visible-light OCT images of healthy human eyes for semantic segmentation, and complete the Vis-105H dataset for 7-class semantic segmentation including the background, as shown in Fig. <ref>(c). 
Specifically, we exploit the graphics software Inkscape to annotate the six layers, nerve fiber layer (NFL), ganglion cell layer (GCL) and inner plexiform layer (IPL), inner nuclear layer (INL) and out plexiform layer (OPL), outer nuclear layer (ONL), outer segment (OS), and retinal pigment epithelium (RPE), as red, yellow, green, light blue, dark blue, and pink, respectively, and to annotate the remaining regions as background black. Each annotated image is further examined by a professional ophthalmologist and exported in PNG format with a size of 660×300 pixels. 
Fig. <ref>(d) shows the average pixel percentages of all categories on the OCT image except for the background, from which we can observe that the most dominant two classes are the NFL and ONL layers while the OS and GCL layers are the least dominant classes. The Vis-105H dataset is divided into a training set, a validation set, and a test set consisting of 75, 15, and 15 images, respectively. The data for the Vis-105 dataset is conducted at BMC, and both BU and BMC hold ownership rights to the data.




  
Glaucoma dataset The dataset is collected from 61 different subjects <cit.>, where 12 radial OCT B scans of each subject are collected using DRI OCT-1 Atlantis at the Ophthalmology Department of Shanghai General Hospital. All images are scanned at 20.48 mm × 7.94 mm field of view in the optic nerve head region. Under the supervision of a glaucoma specialist, two ophthalmologists manually annotate these images into the optic disc and nine retinal layers. The image size is 1024 × 992 and the dataset is divided into training, validation, and test sets by 148, 48, and 48 images, respectively. 



  
DME dataset The dataset is collected by Chiu et al. using the Duke Enterprise Data Unified Content Explorer search engine to retrospectively identify DME subjects within the Duke Eye Center<cit.>. It consists of 110 OCT B-scans obtained from 10 patients with DME with a size of 496×768 pixels, each B-scan is annotated with 9 layers. The dataset is divided by 88, 11, 11 into training, validation, and test set, respectively. All subjects are scanned with the macula at the center. 










 §.§ Performance Metrics

We evaluate the segmentation performance quantitatively using the Dice similarity coefficient (DSC), mean Intersection over Union (mIoU), pixel accuracy (PA), and mean pixel accuracy (mPA). They can be computed as:

    S_DSC = 2× TP/2× TP+FP+FN,


    S_mIoU = 1/k+1∑_i=0^kTP/FN+FP+TP,


    S_PA = TP+TN/TP+FP+TN+FN

and

    S_mPA = 1/k+1∑_i=0^kTP+TN/TP+FP+TN+FN,

where TP, FP, TN, and FN represent True Positives, False Positives, True Negatives, and False Negatives respectively. k represents the number of categories. In addition, we count the number of parameters for compared approaches to represent their computation complexity.




 §.§ Implementation Details

 is implemented based on the PyTorch and trained with the Adam optimizer with the cross-entropy loss. The initial learning rate is set to 0.001 which is then gradually halved every 40 epochs. Data augmentation is applied on all three datasets, including horizontal flipping with probability P=0.5, random center rotation within plus or minus 20 degrees with probability P=0.5, median and motion blur processing with P=0.5 probability, random Gaussian noise addition, as well as random brightness and contrast with P=0.5 probability. All experiments are conducted on an NVIDIA GeForce RTX 3090 Graphics card. The code of the proposed  could be found at: https://github.com/Medical-Image-Analysis/LightReSeg.




 §.§ Comparison



We compare  with the state-of-the-art approaches including ReLaynet<cit.>, EMV-Net<cit.>, Attention_Unet<cit.>, BiSeNet<cit.>, DFANet<cit.>, OS_MGU<cit.>, U-net <cit.> and TransUnet<cit.>. We report the results on the above-mentioned three datasets.




  §.§.§ Quantitative Analysis


Tab. <ref> shows the comparison of our approach with state-of-the-art methods on the Vis-105H dataset. 
We can see that our approach  scores the best performance in both S_mPA and S_mIoU metrics, with +1% improvement in terms of the S_mIoU compared to the second-best performing method OS_MGU <cit.>. 
In terms of the S_DSC metric, our approach achieves the highest segmentation performance in the GCL layer, INL/OPL layer, ONL layer, and OS layer. Regarding the S_PA metric, our approach achieves the first place in the GCL layer, and the second place in the INL/OPL layer and RPE layer, which also shows the outstanding performance of our proposed approach. 
The segmentation performance of ReLayNet, the lightest algorithm in the field of retinal layer segmentation, is -2.5% lower than our proposed  in terms of the S_mIoU metric. 
We further perform a statistical significance test, using the Wilcoxon rank sum test, to compare the Dice Score performance of different methods on each layer. When comparing our approach with ReLayNet, we observe a P_value of 0.031250 (p<0.05), indicating a statistically significant difference.
The poor performance of ReLayNet might be due to its rather simple feature fusion strategy for combining features from encoder and decoder, i.e. skip connections, leading to limited semantic information at the fusion. Instead,  further introduces the MAA module for the feature fusion at multiple scales. Moreover, we add Transformer layers at the bottom of the encoder for efficient global reasoning which further improves the segmentation accuracy. 
SegFormer is a new light-weight model in the field of semantic segmentation and has a good performance in many tasks <cit.>, but as can be seen from Tab. <ref>, SegFormer-B0 only ranks 7th in terms of the S_mIoU and S_mPA metrics, while in terms of the S_DSC metric, its segmentation performance of all layers is much lower than ours. In terms of the S_mPA metric, only the ONL layer is higher than our by 0.7 percentage points, and the segmentation performance in the rest of the layers is much lower than our approach.
We analyze that the poor segmentation performance of SegFormer-B0 on this task may be mainly due to two factors: first, its excessive pursuit of lightweight results in limited ability to extract detailed features; second, the up-sampling part of SegFormer-B0's decoder is too rough, resulting in the lack of fine detail recovery.



We evaluate our approach on the DME dataset, as shown in Tab. <ref>. We achieve the best performance in terms of both S_mPA and S_mIoU. 
In terms of S_DSC we achieve first place in 4 layers (i.e. GCL/IPL, INL, OPL, and ONL/ISM) and second place in another 2 layers (i.e. NFL and ISE), while in terms of S_mPA, we are on par with the state-of-the-art performance. 
Among other approaches, such as TransUnet, OS_MGU, and ReLayNet, although they are more advanced in the S_mPA metric, they are 0.5, 0.7, and 0.9 percentage points lower than the first place in the S_mIoU metric, respectively, which also indicates that our model also has a strong comprehensive performance on the DME dataset. 
We further perform the statistical significance test by using the Wilcoxon rank sum test. For example, when comparing our method with TransUnet on the same domain, we observe a P_value of 0.017629 (p<0.05), indicating a statistically significant difference. Similar statistically significant differences are observed, with P_values of 0.023437 (p<0.05), 0.039062 (p<0.05), and 0.017755 (p<0.05) respectively when comparing our method with OS_MGU, EMV-Net, and SegFormer-B0.
We further evaluate our approach on the Glaucoma dataset, as shown in Tab. <ref>. Our  achieves the best results on both the overall metrics S_mPA and S_mIoU. Additionally, our approach also scores better on most of the layers in terms of both S_DSC and S_PA. 

Fig. <ref> (a) and (b) show the performance of the ten mainstream approaches on the three datasets measured by the mIoU and mPA metrics, respectively.
Our approach  achieve the best performance on all three datasets in terms of S_mIoU and S_mPA. TransUnet performs is the second best-performing method with a small margin lower than ours on all three datasets, however, its number of parameters is 32 times of the one of our approach. The lightest model ReLayNet is inferior to our method in terms of performance by -2.5%, -0.9%, and -0.8% on the Vis-105H, DME, and Glaucoma dataset, respectively. Therefore, our approach achieves state-of-the-art performance with a relatively light-weight model. 

In addition, we find that regardless of the method, certain layers (NFL, ONL, RPE) have significantly higher accuracy compared to other layers, as shown in Tab. <ref>. We believe that class imbalance is the most likely cause, as the number of samples in a certain class is much larger than in other classes, resulting in the model learning better for that class during training and exhibiting higher accuracy in evaluation. From Fig. <ref>(d), we can see that the average pixel proportion of NFL, ONL, and RPE layers is relatively high, and Tab. <ref> also shows that these three layers have higher accuracy compared to other layers. Also, there are differences in shape, color, texture, and other aspects among different layers, which may make certain categories of targets easier to segment while others are more difficult. In summary, we believe that the best way to solve this problem is to first address class imbalance.








  §.§.§ Qualitative Analysis


Fig. <ref> shows the segmentation prediction maps of several mainstream segmentation methods in a retinal B-scan image that contains blood flow information disturbance. As shown in Fig. <ref>(j), our approach achieves much better segmentation performance. The prediction graphs of all the approaches in Fig. <ref>(c), (d), (f), (h) all showed false positive errors, specifically, the region above the NFL layer or below the RPE layer of the retina is incorrectly identified as the retinal layer and segmented, resulting in the wrong segmentation target. The TransUnet in Fig. <ref>(e) performs well, but two locations on the boundary between the NFL and GCL layers are inaccurately segmented, as marked by the two locations in Fig. <ref>(e). The same occurs in EMV-Net in one place, as shown in Fig. <ref>(g). In comparison, our approach is not only without false positive errors, but also segmentation boundaries are more accurate.


Fig. <ref> shows the comparative prediction plots of several mainstream segmentation approaches in a single B-scan image over the retinal optic nerve head region. As shown in Fig. <ref>(j), our approach shows a better segmentation performance. The black dashed region in Fig. <ref>(d), where the NFL and GCL layer boundaries appear distinctly jagged, and the black dashed region in Fig. <ref>(f), where a distinct honeycomb shape appears, are signs of unsmooth and inaccurate layer boundary prediction, and the same occurs in Fig. <ref>(c), (e), and (h). In addition, the problem of segmentation category error occurs, such as the red dashed area in Fig. <ref>(d), where the segmentation results in incorrectly segmenting part of the OPL layer into ONL layers, and this segmentation category error occurs at least twice in Fig. <ref>, (e), (f), and (g). In contrast, our approach not only has no problem with segmentation category error but also the segmented boundary is smoother than other approaches.




 §.§ Ablation study





  §.§.§ Performance of Different Modules



We conduct ablation experiments on the Vis-105H dataset with the main purpose of verifying the effect of the MAA module and the contribution of the Transformer block on the overall method segmentation performance. The ablated methods are: Base is the U-shaped backbone; Base_MAA and Base_trans3 are MAA and 3 transformer layers added to Base, respectively; Base_MAA_trans3 and Base_MAA_trans6 are based on Base_MAA with 3 and 6 transformer layers respectively. Note that Base_MAA_trans3 represents the final version of our proposed .
As shown in Tab. <ref>, the Base_MAA approach improves over the Base approach, the S_mPA and S_mIoU by +0.4% and +1.9%, respectively. All six layers except for the OS layer are also significantly improved in the S_DSC. We also observe that Base_MAA_trans3 improves over Base_trans3 in terms of both S_mPA and S_mIoU. Fig. <ref>(c) and (d) show that the segmentation performance of the model is greatly improved by adding the MAA module to the model, and the results of mis-segmentation are significantly reduced in the background region below the RPE layer. 

Regarding the Transformer block, as shown in Tab. <ref>, Base_trans3 improves +0.5% and +2.4% over Base in terms of S_mPA and S_mIoU, respectively, and the segmentation ability of each layer is improved in terms of S_DSC. In addition, we also find that increasing the number of Transformer layers can further improve the segmentation performance within a certain range. For example, Base_MAA_trans6 adds 3 more Transformer layers than Base_MAA_trans3, which improves S_mPA and S_mIoU by +0.1% and 0.6%, respectively, this comes at a cost of +40% more of parameters. Fig. <ref>(c) and (e) show qualitatively that adding three sequential Transformer blocks to the model contributes to a big improvement in the segmentation performance of the model, and there is no segmentation error in the background region below the RPE layer. 







To further interpret the role of different modules, we use a modified version of Grad-CAM<cit.> to visualize the feature activations in different layers of the model (see Fig. <ref>). Specifically, in accordance with the model settings in Tab. <ref>, we extract the feature activation heat maps for different retinal layers at different model structures and at different positions of the model, respectively. 
By comparing Fig. <ref>(e) and (i), with the addition of the MAA module, we can see that the heat map of the corresponding NFL layer in Fig. <ref>(i) is further enhanced compared to Fig. <ref>(e). The same trend is also observed in Fig. <ref>(h) and (l). 
It is further found that with the addition of the MAA module to the Base_trans3 model structure, the feature region corresponding to the segmentation category is also further enhanced, as shown in Fig. <ref>(p) and (t). 
Adding the Transformer blocks to the model also causes changes in the heat map, as in Fig. <ref>(f) and (n). The heat map of the corresponding RPE layer region is activated more comprehensively with the addition of the Transformer blocks, and the OS and background layers of the adjacent regions also show different degrees of activation. 



When analyzing the activation maps, we observe that not only the regions corresponding to the predicted layers prominently highlighted, but also the feature weights of the adjacent layers are also significantly amplified. To interpret this behavior, we conduct further analysis and find that the boundaries of the retinal layers exhibit a high degree of correlation with the adjacent layers. As in Fig. <ref>(e), the focus is mainly on the NFL layer, but adjacent layers (e.g., background and GCL) also show various degrees of activation. Similarly, in Fig. <ref>(r), the RPE layer of the retinal layer is the main focus, while information from the OS and background layers at the upper and lower boundaries of the retinal layer is also utilized. This phenomenon suggests that the model uses information from adjacent layers to enhance its predictions. In addition, we also export the feature activation maps for different deep network layers in the prediction of the RPE layer, and we find that the heat map of the deep network focuses on more detailed and accurate regions compared to the heat map of the superficial network. As in Fig. <ref>(o), the regions where the heat map is focused on activation are highly coincident with the RPE layer, while the regions activated by the heat map in Fig. <ref>(n) are much coarser, which is similarly shown in Fig. <ref>(r) and (s). This observation suggests that the deeper extract image features of the model network contain more accurate and insightful semantic information. Overall, the visualization of deep feature activation provides valuable insights into the model's decision process, and the model can utilize multiple layers of information to obtain more robust segmentation results.



  §.§.§ Light-weight Setup



To evaluate the potential impact of DS-Conv on feature extraction capabilities, we replace all DS-Conv in the encoder section with standard convolutions and conduct comparative analyses<cit.>, as shown in Tab. <ref>. The results indicate that using standard convolutions instead of DS-Conv does not enhance feature extraction in our optimized model and leads to an 11% increase in the parameter count. This demonstrates that a simple increase in the model's parameter count does not always result in performance.


For assessing the impact of model light-weight on performance, we modify the number of channels in the encoder's multi-scale features, both halving and doubling them, the specific results are shown in Tab. <ref>. We find that, based on our model, when the channel of the multi-scale features extracted by the encoder is reduced to 0.5 and 0.25 times, the model's parameter count decreased, but its segmentation performance significantly deteriorated. When we increase the channel count to 2 and 4 times, there is no significant change in the model's accuracy, yet the parameter counts are 3 and 11 times the original size, respectively. From this, we can also conclude that our lightweight design maintains accuracy while using the fewest possible parameters.





 §.§ Limitation


Although  boasts only 3.3M trainable parameters and delivers the optimal retinal layer segmentation outcomes, in real-world applications, model parameter size is not the sole determinant of algorithm efficacy. Inference speed emerges as a crucial metric for assessing algorithmic effectiveness. As evidenced by Tab. <ref>, our proposed method exhibits inference speeds of 0.11s, 0.27s, and 0.07s on the three datasets, respectively. While these figures already signify remarkable efficiency, there remains scope for further enhancement. For example, although the segmentation accuracy of ReLayNet is far inferior to our approach, it must be admitted that its inference efficiency is slightly higher than ours.



Moreover, in the context of practical clinical applications, constraints related to dataset limitations preclude the inclusion of all device data types. Imagery obtained through disparate devices may exhibit domain discrepancies, inevitably resulting in false positive errors in the segmentation output for unfamiliar new devices. Consequently, apart from enlarging the dataset, enhancing the model's domain generalization competency is a subject worthy of further investigation. The supplementary segmentation result evaluation system will also augment ophthalmologists' confidence in the segmentation outcomes.




§ CONCLUSION


In this paper, we propose a novel light-weight method  for retinal layer segmentation. Our method introduces a Transformer-based block in the encoder part to enable global reasoning for reducing errors in the background region and an attention mechanism, named MAA module, to best exploit rich semantic information for the multi-scale feature fusion to improve the segmentation accuracy. The extensive evaluation shows that our approach achieves the best segmentation performance on both our collected Vis-105H dataset and two other public ones, i.e. DME and Glaucoma, indicating that our model has good reliability in the face of noise and uncertainty in the data. 
To improve the efficiency, our method also incorporates light-weight designs. The ablation experiments detailed in Section <ref> demonstrate that our lightweight design is precisely adequate, maintaining the highest level of accuracy while significantly reducing the number of parameters compared to other high-performing methods. This further highlights the practicality of our proposed approach, particularly evident in the real-time preview functionality of small-field-of-view OCTA imaging.
In the future, we will collect more clinical data to further improve our approach and continuously contribute to the efficient performance of OCT devices.



IEEEtran 
 

