Features Fusion for Dual-View Mammography Mass Detection
Arina Varlamova Valery Belotsky Grigory Novikov Anton Konushin Evgeny Sidorov



High-fidelity and Fault-tolerant Teleportation of a Logical Qubit using Transversal Gates and Lattice Surgery on a Trapped-ion Quantum Computer
    D. Hayes
    April 29, 2024
===============================================================================================================================================




Detection of malignant lesions on mammography images is extremely important for early breast cancer diagnosis. In clinical practice, images are acquired from two different angles, and radiologists can fully utilize information from both views, simultaneously locating the same lesion. However, for automatic detection approaches such information fusion remains a challenge. In this paper, we propose a new model called MAMM-Net, which allows the processing of both mammography views simultaneously by sharing information not only on an object level, as seen in existing works, but also on a feature level. MAMM-Net's key component is the Fusion Layer, based on deformable attention and designed to increase detection precision while keeping high recall. Our experiments show superior performance on the public DDSM dataset compared to the previous state-of-the-art model, while introducing new helpful features such as lesion annotation on pixel-level and classification of lesions malignancy.



instance segmentation, mammography, lesion detection


  2024 IEEE.  Personal use of this material is permitted.  Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.



§ INTRODUCTION



Breast cancer is one of the most common diseases, ranking first among the causes of cancer mortality among women<cit.>. Approximately 12% of all diagnosed cases of cancer in women are related to breast cancer, making it the dominant cancer-related disease in many countries.

Screening digital mammography is the most widely employed modality for the successful diagnosis of breast cancer<cit.>. Presently, Artificial Intelligence (AI), mostly in the form of computer vision, is extensively utilized for tasks involving the automatic detection of breast cancer features, demonstrating high diagnostic accuracy comparable to or surpassing that of a radiologist<cit.>.

One of the primary tasks in forming radiological descriptions of mammography studies is the identification and comprehensive description of breast tissue abnormalities in both projections <cit.>. The differential diagnosis of lesions involves a comparative analysis of two views of each breast and between the two breasts. However, most current neural network architectures do not take into account the context of both views, limiting model performance.

To the best of our knowledge, existing two-view approaches fuse information  between different angles only on object level, using features obtained independently <cit.>. In our study we propose new approach called MAMM-Net, where network is able to fuse information on features level in addition to object level, imitating radiologist's diagnosis process more naturally. We observe that additional information fusion helps model to effectively filter false positive detections while preserving the high recall, thus allowing to achieve state-of-the-art results.



§ RELATED WORK




 §.§ Dual-view lesion segmentation





The idea of dual-view segmentation was considered in several approaches. Liu et al. <cit.> use a bipartite graph convolutional network to incorporate the intrinsic geometric and semantic relations of ipsilateral views. Ma et al. <cit.> propose a relation module to model correspondence between mass ROIs from different mammography images. In CL-Net <cit.> authors used cross-attention between object queries, generated by Deformable DETR <cit.> and a special module called Lesion Linker to verify object pairs.    



 §.§ Object recognition

Transformer-based architectures gained popularity in tasks related to computer vision, including object detection and instance segmentation. Detection models originate from DETR <cit.>, which was the first to introduce the concept of query-based instance proposal. The idea was further adopted for instance segmentation <cit.>, resulting in SOTA architecture at the time of writing.



 §.§ Multi-view segmentation


3D segmentation in multi-camera space was actively explored in BEV Transformers <cit.>. Feature fusion here was utilized through reprojection into the same 2D space and usage of deforamble attention in the local proximilty to points of interest. However, such approach is tricky in mammography application, since angles between images are in general unknown. In this settings there is a need for more flexible approach for generation of reference points, which can be achieved through deformable attention introduced in <cit.>.



§ METHODS





Our proposed architecture will be explored further in this section, which we will reference as MAMM-Net (Multi-view Attention for Mass Matching). A brief overview of its structure is shown in Fig. <ref>. 

We organize methods as follows: firstly, we briefly introduce key points of Mask2Former <cit.> and CL-Net <cit.> architectures, which our model is based on, and then we explain our proposed Fusion Layer and View-Interactive Transformer Decoder (VITD) in more details.



 §.§ Mask2Former


Our model mainly inherits the Mask2Former structure in its main components: backbone, pixel decoder, and transformer decoder with masked attention. The transformer decoder's key component is the masked attention operator, which allows the usage of spatial features restricted to the foreground area of the predicted masks. We left this part mostly unchanged except for using two object query branches for cranio-caudal (CC) and mediolateral oblique (MLO) mammography views and adding additional cross-attention logic between them.

Our Fusion Pixel Decoder, however, has more significant differences. Instead of generating feature maps independently, we combine them each time resolution increases. To provide intermediate feature maps of different resolutions, at each step we fuse features of CC and MLO projections into each other using our proposed Fusion Layers.  



 §.§ CL-Net


Key components of CL-Net we were interested in are the VILD (View-Interactive Lesion Detector) and LL (Lesion Linker) modules. The main idea behind VILD is to add a cross-view inter-attention step in addition to self- and cross-attention in DETR's transformer decoder in order to help the model capture the relationship between objects in CC and MLO views. LL module at the same time aims to model correspondence between detected objects, outputing classification of the presence of object pair and link embeddings. 



 §.§ Fusion Pixel Decoder




The Fusion Pixel Decoder is designed to provide both high-resolution mask feature maps for CC and MLO views as well as multi-scale fused feature maps for the VITD's masked attention. A brief overview of module structure is shown in Fig. <ref>. 

The module consists of several consequent blocks for combining information between projections. Each block takes two feature maps of different resolutions from the shared backbone, which are passed through the Fusion layer in two branches. Generated feature maps are further concatenated and convolved using 1×1 convolution to maintain the same channel dimension. Such feature maps are generated from low to high resolution and merged by element-wise addition in an FPN <cit.> manner.

A key component of our pixel decoder is the Fusion layer, which is based on deformable attention <cit.>. The Fusion layer uses two feature maps: main to provide queries and reference from another view to provide key and values components. Since classic cross-attention between two feature maps is computationally demanding, it's essential to focus on a limited subset of spatial positions for key-values. In order to achieve that, we use the predefined amount of uniformly distributed points to initialize the start position of candidates. The special network generates their relative offsets, which are used to sample deformed key-values. Produced queries, keys, and values are then processed by a standard multi-head attention block.

It is worth noting that we use the Fusion Layer only in two blocks with feature maps with the lowest resolution. As resolution increases, we replace it with two independent 1 × 1 convolutions. Mask feature maps are generated by separately convolving the fused feature map with the highest resolution.



 §.§ VITD


Following <cit.>, blocks of our VITD consist of masked attention, self-attention, and FFN layers. We added an additional inter-attention layer to share object information between different views, similar to <cit.>. More formally, at i^th decoder's iteration, module uses Q^v_i=f_Q(X^v_i-1) ∈ℝ^N × C and K_i, V_i ∈ℝ^H_iW_i× C which are fused feature maps with applied transformations f_K(·) and f_V(·) respectively. X^v_i refers to query features at layer i^th and specific view (CC or MLO). In these terms, output of the masked attention layer may be defined as follows:

    X_m^v_i = softmax(ℳ^v_i-1 + Q^v_iK_i^T)V_i + X^v_i-1,

where attention mask at spatial location (x, y) ℳ^v_i-1(x, y) = 0 if the binarized output of the resized mask of corresponding view from a previous layer of decoder equals 1, and ∞ otherwise. To utilize both high- and low-level features,  fused feature maps are fed to the decoder in a round-robin fashion.

After applying the self-attention layer to X_m^v_i for both CC and MLO, query vector is passed as Q to inter-attention layer with the same view, and as K and V to inter-attention layer for another projection. Outputs are processed by the feed-forward network, providing X^v_i for the next iteration of the decoder. 

Finally, we apply three feed-forward networks to each X^v_i to provide a classification of objects, their malignancy class, and masks embeddings E^v_i, which are further multiplied to feature maps of high resolution from Fusion Pixel Decoder in order to produce objects masks. 



 §.§ Loss

Our training loss is a combination of detection loss <cit.>, linker loss <cit.> and malignancy loss. Detection and linker losses were implemented as in original publications, and a malignancy loss is a binary cross-entropy loss, where the target class is objects being malignant.



§ EXPERIMENTS





 §.§ Implementation Details

We adopt EfficientNet-b3 <cit.> as a backbone. We set the number of object queries and link queries to 100 and 50 respectively. In our experiments, we used 10 blocks in VITD. A downsample factor of 4 was used to generate reference points in Fusion layers. We set the number of heads of multi-head attention to 8 in both VITD and Fusion Pixel Decoder.

The network was trained with a batch size equal to 5, each item in a batch containing one mammography pair. We train our model with a learning rate set to 1 × 10 ^-4. To avoid overfitting we used L2-regularization with a weight of 1× 10^-5 and a variety of augmentations, including image flip, rotation, brightness and contrast modification, and random scaling.



 §.§ Datasets


We conducted our experiments using the DDSM dataset <cit.>. Originally, the dataset does not provide matching between objects in different views. In order to obtain it, cases were annotated by a skilled radiologist. In such annotations, we didn’t change any contour properties or coordinates, but rather made a classification of objects being pair or not.


Similar to <cit.>, we use recall (R) at t false positives per image (FPI) to compare the performance of our model with other studies (R@t). Unlike in previous methods, we recall an object if it has IoU with ground true more than 0.1 since we have masks instead of bounding boxes and observe a relatively high proportion of true positives in diapason of [0.1, 0.2]. An example of such an object is shown in Fig. <ref>.

Data splits are another matter of discussion. Train splits proposed in <cit.> is ambiguous since Liu et al. <cit.> mention 512 test cases and <cit.> doesn't specify test set size. At the same time, they both refer to <cit.>, which uses 512 images. We decided to follow the test split proposed in <cit.> with some modifications. Originally, images from cancer volumes without any masses were excluded. Since additional images without ground true objects can only worsen R@t, we selected all cases from these volumes for more representative comparison, resulting in 270 test cases and 1080 images.



 §.§ Comparison with other studies




 We show the comparison of our MAMM-Net with other methods in Table <ref>. Result from Table <ref> are reported from <cit.>. We keep the same FPIs as in <cit.> as the previous SOTA (CL-Net). It can be concluded that our model surpasses CL-Net by a large margin at all reported FPIs. It is worth noting, that we use a lighter backbone compared to ResNet-50 used in <cit.>, since we achived similar performance as CL-Net in setting with fusion on object level only. We show more details on that in section <ref>. We believe that the result on R@0.25 is of the most significance and additionally provide binary malignancy metrics (per mammary gland) for this setting, such as ROC-AUC (85.3), sensitivity (80.2) and specificity (76.2). 



 §.§ Ablation Study








We evaluate the performance of the networks that use only one fusion component, either the VITD or the Fusion Pixel Decoder, against the performance of the MAMM-Net, which incorporates both fusion blocks. Table <ref> illustrates the differences in recall for selected FPI thresholds. We further discuss the implementation details of networks with only one fusion component.



  §.§.§ VITD
 


In this setting, we used a pixel decoder from Mask2Former , which outputs two independent sets of multi-scale feature maps for both views. We got similar values to CL-Net (78.2 vs 78.1, 83.3 vs 83.1, 87.3 vs 88.0 at respectively), which is to be expected since both models have similar key components. 



  §.§.§ Fusion Pixel Decoder


In this setup the Fusion Pixel Decoder was left intact, while the VITD was significantly modified. Instead of two branches for different views, we used a single branch for object queries and excluded the inter-attention layer. For the mask in masked attention, we used a logical union of binarized mask outputs for both views. We skipped the LL block, using VITD's output directly in the loss computation and forcing matched objects to be predicted in the same position. Similar to <cit.>, we observe a significant drop in performance compared to the complete model. We support that forced prediction at the same position wasn't the best choice for modeling relationships between different objects. However, it was observed that for lower FPIs Fusion Pixel Decoder performs better than our full model (e.g. 73.3 vs 69.7 R@t0.14). We hypothesize that Fusion Layers help the model to effectively filter candidates that look worthy of attention only from one view.



§ CONCLUSION



The main novelty introduced in our paper is the Fusion Layer, which enables feature-level fusion of two projections, leading to a decrease in false positive predictions without an increase in false negatives. This component is integrated into our newly proposed MAMM-Net architecture, designed for effective object recognition across two projections. Experiments on the DDSM dataset have demonstrated that our architecture outperforms previous state-of-the-art models.



§ ACKNOWLEDGMENTS

















No funding was received for conducting this study.



§ COMPLIANCE WITH ETHICAL STANDARDS



This research study was conducted retrospectively using human subject data sourced from DDSM. Ethical approval was not required as
confirmed by the license attached with the open access data.





IEEEbib



