--- abstract: 'Recently, there has been an increasing concern about the privacy issue raised by using personally identifiable information in machine learning. However, previous portrait matting methods were all based on identifiable portrait images. To fill the gap, we present P3M-10k in this paper, which is the first large-scale anonymized benchmark for Privacy-Preserving Portrait Matting (P3M). P3M-10k consists of 10,000 high resolution face-blurred portrait images along with high-quality alpha mattes. We systematically evaluate both trimap-free and trimap-based matting methods on P3M-10k and find that existing matting methods show different generalization abilities under the privacy preserving training setting, i.e., training only on face-blurred images while testing on arbitrary images. Based on the gained insights, we propose a unified matting model named P3M-Net consisting of three carefully designed integration modules that can perform privacy-insensitive semantic perception and detail-reserved matting simultaneously. We further design multiple variants of P3M-Net with different CNN and transformer backbones and identify the difference of their generalization abilities. To further mitigate this issue, we devise a simple yet effective Copy and Paste strategy (P3M-CP) that can borrow facial information from public celebrity images without privacy concerns and direct the network to reacquire the face context at both data and feature level. P3M-CP only brings a few additional computations during training, while enabling the matting model to process both face-blurred and normal images without extra effort during inference. Extensive experiments on P3M-10k demonstrate the superiority of P3M-Net over state-of-the-art methods and the effectiveness of P3M-CP in improving the generalization ability of P3M-Net, implying a great significance of P3M for future research and real-world applications. The dataset, code and models are available here[^1].' author: - 'Sihan Ma$^{1*}$, Jizhizi Li$^{1*}$, Jing Zhang$^{1}$, He Zhang$^{2}$, Dacheng Tao$^{3,1}$' bibliography: - 'matting.bib' date: 'Received: date / Accepted: date' title: ' Rethinking Portrait Matting with Privacy Preserving [^2] ' --- Introduction ============ The success of deep learning in many computer vision and multimedia areas largely relies on large-scale of training data [@zhang2020empowering]. However, for some tasks such as face recognition [@masi2018deep], human activity analysis [@sun2019deep], and portrait animation [@9229197; @Chen_2020_CVPR], privacy concerns about the personally identifiable information in the datasets, e.g., face, gait, and voice, have attracted increasing attention recently. Unfortunately, how to alleviate the privacy concerns in data while not affecting the performance remains challenging and under-explored [@yang2021study]. Specifically, portrait matting, which refers to estimating the accurate foregrounds from portrait images, suffers more from the privacy issue as most of the images contain identifiable faces in the previous matting datasets [@dim; @hatt; @dapm]. Due to the population of the virtual video meeting in post COVID-19 pandemic era, this issue has received more and more concerns since portrait matting is a key technique in this multimedia application for changing virtual background. However, we found that all the previous portrait matting methods paid less attention to the privacy issue and adopted the intact identifiable portrait images for both training and evaluation, leaving privacy-preserving portrait matting (P3M) as an open problem. In this paper, we make the first attempt to address the privacy issue by setting up a new task P3M that requires training only on face-blurred portrait images (i.e., the Privacy-Preserving Training (PPT) setting) while testing on arbitrary images. We present the very first large-scale anonymized portrait matting benchmark P3M-10k consisting of 10,000 high-resolution face-blurred portrait images where we carefully collect and filter from a huge number of images with diverse foregrounds, backgrounds and postures, along with the carefully labeled high quality ground truth alpha mattes. It surpasses existing matting datasets [@dapm; @lf] in terms of diversity, volume and quality. Besides, we choose face obfuscation as the privacy protection technique to remove the identifiable face information while retaining fine details such as hairs. We split out 500 images from P3M-10k to serve as a face-blurred validation set, named P3M-500-P. Some examples are shown in Figure \[fig:introduction\](a). Furthermore, to evaluate the generalization ability of matting models on non-privacy images when training on privacy-preserved images, we construct a validation set with 500 images without privacy concerns, named P3M-500-NP. All the images in P3M-500-NP are either frontal images of celebrities or profile/back images without any identifiable faces. Some examples are shown in Figure \[fig:introduction\](b). It can be observed from Figure \[fig:introduction\] that face obfuscation brings noticeable artefacts to the images which are not observed in normal portrait images. Then, one common and interesting question to explore is that how will the proposed PPT (Privacy-Preserving Training (PPT)) setting impact existing SOTA matting models. We notice that a contemporary work [@yang2021study] has shown empirical evidences that face obfuscation only has minor side impact on object detection and recognition models. However, the impact remains unclear in the context of portrait matting, where the pixel-wise alpha matte (a soft mask) with fine details is expected to be estimated from a high-resolution portrait image. To address the above problem, we systematically evaluate both trimap-based [@dim; @lu2019indices; @li2020natural] and trimap-free matting methods [@shm; @lf; @modnet] on P3M-10k and provide our insights and analyses. Specifically, we found that for trimap-based matting, where the trimap is used as an auxiliary input, face obfuscation shows less impact on the matting models, i.e., a slight performance change of models following the PPT setting. As for trimap-free matting which involves two sub-tasks: foreground segmentation and detail matting, we found that the methods using a multi-task framework that explicitly model and jointly optimize both tasks [@gfm; @hatt] are able to obtain a good generalization ability on both face-blurred images and non-privacy ones at the same time. In contrast, matting methods that solve the problem in a 'segmentation followed by matting' manner [@shm; @dapm] show a significant performance drop under the PPT setting. The main reason lies in the fact that the segmentation errors led by face obfuscation may amplify error of the following matting model. Other methods that involve several stages of networks to progressively refine the alpha mattes from coarse to fine [@shmc] seem to be less affected by face obfuscation but still encounter a performance drop due to the lack of explicit semantic guidance. Meanwhile, these methods require a tedious training process. Based on the above observations, we propose a novel automatic portrait matting model P3M-Net, which is able to serve as a strong trimap-free matting baseline for the P3M task. Technically, we adopt a multi-task framework like proposed in [@gfm; @modnet] as our basic structure, which learns common visual features through a sharing encoder and task-aware features through a segmentation decoder and a matting decoder. To further improve the generalization abilities on P3M, we design a deep Bipartite-Feature Integration (dBFI) module to improve network's robustness ability to privacy-preserving training data by leveraging deep features with high-level semantics, a Shallow Bipartite-Feature Integration (sBFI) module to enhance network's ability to obtain fine details in the portrait images by extracting shallow features in the matting decoder, and a Tripartite-Feature Integration (TFI) module to promote the interaction between two decoders. We further design multiple variants of P3M-Net based on both CNN and vision transformer backbones and identify the difference of their generalization abilities. Extensive experiments on the P3M-10k benchmark provide useful empirical insights on the generalization abilities of different matting models under the PPT setting and demonstrate that P3M-Net along with its variants outperform all the previous trimap-free matting methods by a large margin. Although our P3M-Net model achieves better performance than previous methods, there is still a performance gap when testing on face-blurred images and normal non-privacy portrait images, especially on face regions. How to compensate for the lack of facial details in face-blurred training data to reduce the performance gap remains unresolved. To mitigate this problem, we devise an simple yet effective Copy and Paste strategy (P3M-CP) that can borrow facial information from publicly available celebrity images without privacy concerns and direct the network to reacquire the face context at both data and feature level. P3M-CP only brings a few additional computations during training, while enabling the matting model to process both face-blurred images and the non-privacy ones pretty well without extra effort during inference. To sum up, the contributions of this paper are four-fold. **First**, to the best of our knowledge, we are the first to study the problem of privacy-preserving portrait matting and establish the largest privacy-preserving portrait matting dataset P3M-10k, which can serve as the benchmark for P3M. **Second**, we systematically investigate the impact of PPT setting and provide insights about the evaluation protocol, generalization ability and model design. **Third**, we propose a novel multi-task trimap-free matting model P3M-Net with three carefully designed interaction modules to enable privacy-insensitive semantic perception and details reserved matting. We then further devise multiple P3M-Net variants based on both CNN and vision transformer backbones to investigate their generalization abilities. **Fourth**, we devise a simple yet effective P3M-CP strategy that can improve the generalization ability of matting models for P3M under the PPT setting. Extensive experiments have demonstrated the value of our proposed methods, confirming P3M's ability to facilitate future research. The remainder of the paper is organized as follows. Section \[sec:lr\] introduces the existing works related to matting, privacy issues in visual tasks, and vision transformer. In Section \[sec:benchmark\], we systematically evaluate both trimap-based and trimap-free methods on P3M-10k, and analysize the impact of PPT setting. We introduce our P3M-Net model and its variants in Section \[sec:p3mnet\], and present the copy and paste strategy P3M-CP in Section \[sec:p3mcp\]. Section \[sec:experiments\] manifests the subjective and objective experiment results of both P3M-Net and P3M-CP. Finally, we conclude the paper in Section \[sec:conclusion\]. Related Work {#sec:lr} ============ Image Matting ------------- Image matting is a typical ill-posed problem to estimate the foreground, background, and alpha matte from a single image. Specifically, portrait matting refers to a specific image matting task where the input image is a portrait. From the perspective of input, image matting can be divided into two categories, i.e., trimap-based methods and trimap-free methods. Trimap-based matting methods use a user-defined trimap, i.e., a 3-class segmentation map, as an auxiliary input, which provides explicit guidance on the transition area. Previous methods include affinity-based methods [@levin2007closed; @aksoy2018semantic], sampling-based methods [@he2011global; @shahrian2013improving], and deep learning based methods [@lu2019indices; @hou2019context; @dai2022boosting]. Besides, there are other methods using different auxiliary inputs, e.g., a background image [@backgroundmatting; @backgroundmattingv2], or a coarse map [@yu2021mask]. To enable automatic (portrait) image matting, recent works [@shm; @lf; @hatt; @gfm; @aim] tried to estimate the alpha matte directly from a single image without using any auxiliary input, also known as trimap-free methods. For example, DAPM [@dapm] and SHM [@shm] tackled the task by separating it into two sequential stages, i.e., segmentation and matting. However, the semantic error produced in the first stage will mislead the matting stage and can not be corrected. LF [@lf] and SHMC [@shmc] solved the problem by first generating coarse alpha matte and then refining it. Besides of the tedious training process, these methods suffer from ambiguous boundaries due to the lack of explicit semantic guidance. HATT [@hatt] and GFM [@gfm] proposed to model both the segmentation and matting tasks in a unified multi-task framework, where a sharing encoder was used to learn base visual features and two individual decoders are used to learn task-relevant features. However, HATT [@hatt] lacks explicit supervision on the global guidance while GFM [@gfm] and MODNet [@modnet] lacks modeling the interactions between both tasks. By contrast, we propose a novel model named P3M-Net, which is also based on the multi-task framework but specifically focuses on modeling the interactions between encoders and decoders to better perform privacy-insensitive matting. Besides, several P3M-Net variants with CNN and vision transformer backbones have been designed and analysed. Privacy Issues in Visual Tasks ------------------------------ There are two kinds of privacy issues in visual tasks, i.e., private data protection and private content protection in public academic datasets. For the former, there are concerns of information leak caused by insecure data transferring and membership inference attacks to the trained models [@shokri2017membership; @hisamoto2020membership; @fredrikson2015model; @carlini2019secret]. Privacy-preserving machine learning (PPML) aims to solve these problems based on homonorphic encryption [@erkin2009privacy; @yonetani2017privacy], differential privacy [@NEURIPS2020_fc4ddc15; @pmlr-v161-he21a], and federated learning [@truex2019hybrid]. For public academic datasets, there is no concern for information leak, thus PPML is no longer needed. But, there still exists privacy breach incurred by exposure of personally identifiable information, e.g., faces, addresses. It is a common problem in the benchmark datasets for many vision tasks, e.g., object recognition and semantic segmentation. Recently a contemporary work [@yang2021study] has shown empirical evidences that face obfuscation, as an effective data anonymization technique, only has minor side impact on object detection and recognition. However, since portrait matting requires to estimate a pixel-wise soft mask (alpha matte) for a high-resolution portrait image, the impact and difficulty remain unclear. To explore the privacy-preserving portrait matting, a suitable face anonymization technique is necessary. A common method is to add empirical obfuscations [@uittenbogaard2019privacy; @caesar2020nuscenes; @frome2009large; @yang2021study], such as blurring and mosaicing at certain regions. For the portrait matting task, we make the first attempt to construct a large-scale anonymized dataset named P3M-500-P and adopt face obfuscation as the privacy-preserving strategy. Specifically, we adopt multiple different face obfuscation methods, e.g., Gaussian blurring, mosaicing, zero masking, to construct different versions of P3M-500-P and validate the generalization ability of models trained with blurred images. Matting Datasets ---------------- Existing matting datasets either contain only a small number of high-quality images and annotations, or the images and annotations are in low-quality. For example, the online benchmark alphamatting [@TUW-180666] only provides 27 high-resolution training images and 8 test images. None of them is portrait image. Composition-1k [@dim], the most commonly used dataset, contains 431 foregrounds for training and 20 foregrounds for testing. However, many of them are consecutive video frames, making it less diverse. GFM [@gfm] provides 2,000 high-resolution natural images with alpha mattes, but they are all animal images. With respect to portrait image matting dataset, DAPM [@dapm] provided a large dataset of 2,000 low-resolution portrait images with alpha mattes generated by KNN matting [@chen2013knn] and closed form matting [@levin2007closed], whose quality is limited. Late fusion [@lf] built a human image matting dataset by combining 228 portrait images from Internet and 211 human images in Composition-1k. Distinction-646 [@hatt] is a dataset containing 364 human images but with only foregrounds provided. There are some large-scale portrait datasets, e.g., SHM [@shm], SHMC [@shmc], and background matting [@backgroundmatting], which are unfortunately not public. Most importantly, no privacy preserving method is used to anonymize the images in the aforementioned datasets, making all the frontal faces exposed. By contrast, we establish the very first large-scale matting dataset with 10,000 high-resolution portrait images with high-quality alpha mattes, where all images are anonymized using face obfuscation. Vision Transformer ------------------ Transformer is a new deep neural structure that utilizes the self-attention mechanism to model the long-range dependency. It was first applied in the machine translation tasks in NLP, and shows great potential in vision tasks recently due to its superior representation capacity and flexible structure [@dosovitskiy2020image; @swin; @vitae; @zhang2022vitaev2]. ViT is the very first work [@dosovitskiy2020image] to utilize pure transformer structures in image recognition and shows promising results. @vitae introduced the inductive bias into vision transformers to fully utilize the long-range modelling ability of self-attention and the locality and scale-invariance modelling ability of convolutions. Their ViTAE models achieved promising results in image recognition and many downstream tasks. Transformers also can handle multiple vision tasks at same time. Image Processing Transformer (IPT) [@IPT] is proposed to process multiple low-level image tasks, including denoising, deraining, and super resolution. In matting, very few works have tried to explore and utilize the transformer related structure. In this work, we are the first to incorporate the vision transformers in our P3M-Net model to handle the global segmentation and detail matting tasks at same time, which manifests great generalization abilities. Comparison with the Conference Version -------------------------------------- Compared with our conference version @p3m_mm, we extend the study by introducing three major improvements. **First**, we rethink the problem of privacy-preserving portrait matting, and identify the performance gap of matting models between testing on face-blurred images and generalizing on normal non-privacy images. **Second**, we extend the P3M-Net model by proposing multiple variants based on both CNN and vision transformer backbones, and investigate the difference of their generalization abilities. This is the first time that vision transformers are adopted in trimap-free matting task. Extensive experiments validate that P3M-Net and its variants outperform all previous trimap-free matting methods on both face-blurred images and normal non-privacy images by a large margin, and even achieve comparable results with trimap-based matting methods. **Third**, we introduce a simple yet effective P3M-CP strategy to mitigate the issue of absent face information during training under the PPT setting. It can borrow facial information from public celebrity images without privacy concerns and guide the network to re-acquire the face context at both data and feature level. Extensive experiments validate its effectiveness in improving the generalization ability of different matting models on normal non-privacy images. ![image](dataset_samples.jpg){width="\linewidth"} A Benchmark for P3M and Beyond {#sec:benchmark} ============================== Privacy-preserving portrait matting is an important and meaningful topic due to the increasing privacy concerns. In this section, we first clearly define this new setting, then establish a large-scale anonymized portrait matting dataset P3M-10k to serve as the benchmark for P3M. A systematic evaluation of the existing trimap-based and trimap-free matting methods on P3M-10k is conducted to investigate the impact of the privacy-preserving training setting on different matting models. We then gain some useful insights in terms of the evaluation protocol, generalization ability, and model design. PPT setting ----------- Due to the privacy concern, we propose the **p**rivacy-**p**reserving **t**raining (PPT) setting in portrait matting, i.e., training only on privacy-preserved images (e.g., processed by face obfuscation) and testing on arbitrary images with or without privacy content. As an initial step towards privacy-preserving portrait matting problem, we only define the identifiable faces in frontal and some profile portrait images as the private content in this work. Intuitively, PPT setting is challenging since face obfuscation brings noticeable artefacts to the images which are not observed in normal portrait images, i.e., there is a clear domain gap between face-blurred images and normal images. How to eliminate the side impact of PPT setting and make model generalize well on normal images remains challenging but is of great significance for privacy-sensitive applications. ![image](face-obfuscation.jpg){width="\linewidth"} P3M-10k dataset --------------- To answer the above question and provide a solid testbed for P3M, we establish the very first large-scale privacy-preserving portrait matting benchmark named P3M-10k. It contains 10,000 anonymized high-resolution portrait images by face obfuscation along with high-quality ground truth alpha mattes. Specifically, we carefully collect, filter and annotate about 10,000 high-resolution images from the websites with open licenses[^3]. We ensure all the images are not duplicate and have at least 1080 pixels on both two directions to ensure they are all high-resolution. We also check on each image to make sure it contains a salient and clear person. As for the privacy-preserving method, we propose to use blurring to obfuscate the identifiable faces. Instead of using a face detector to obtain the bounding box of face and blurring it accordingly as in [@yang2021study], we adopt a facial landmark detector to obtain the face mask. It is because different from the classification and detection tasks in [@yang2021study], which may not sensitive to the blurry boundaries, portrait matting requires to estimate the foreground alpha matte with clear boundaries, including the transition areas of face such as cheek and hair. As shown in Figure \[fig:face\_obfuscation\], after obtaining the landmarks, a pixel-level face mask is automatically generated along the cheek and eyebrow landmarks in step (3). Then, we exclude the transition area shown in step (4) and generate an adjusted face mask at step (5). Finally, we use Gaussian blur to obfuscate the identifiable faces in the mask and the final result is shown in step (6). Note that for those images with failure landmark detection, we manually annotate the face mask. Eventually, there are 9,921 images with face obfuscation remained. We then split them as 9,412 training images and 500 test images noted as P3M-500-P to evaluate models' performance on face-blurred images in P3M. In addition, to evaluate models' generalization ability on non-privacy images in P3M, we further collect and annotate another 500 public celebrity images from the Internet without face obfuscation to form P3M-500-NP. Some examples of the training set and two validation sets are shown in Figure \[fig:data\_p3m-10k\]. Our P3M-10k outperforms existing matting datasets in terms of dataset volume, image diversity, privacy preserving, and providing natural images instead of composite ones. The diversity is not only shown in foreground, e.g., half and full body, frontal, profile, and back portrait, different genders, races, and ages, $etc$., but also in background, i.e., images in P3M-10k are captured in different indoor and outdoor environments with various illumination conditions. Some examples are shown in Figure \[fig:data\_p3m-10k\]. In addition, we argue that large volume and high diversity of P3M-10k enable models to train on the natural images without the need of image composition using low-resolution background images, which is a common practice in previous works [@dim; @hatt], where they use composition to increase data diversity due to the small scale dataset volume but will bring in obvious composition artefacts due to the discrepancy of foreground and background images in noise, resolution, and illumination. The composition artefacts may have a side impact on the generalization ability of matting models as shown in [@gfm]. By contrast, the background in P3M-10k are compatible with the foreground since they are captured from the same scene. \[htb\] \[htb\] \[htb\] Benchmark Setup --------------- We evaluate both trimap-based and trimap-free matting methods including traditional and deep learning ones on P3M-10k. The full list of the methods are shown in Table \[tab:benchmark\_trimap\_based\_traditional\], \[tab:benchmark\_trimap\_based\_dl\], \[tab:benchmark\_e2e\]. We use the common metrics including MSE, SAD, and MAD to evaluate their performance. For trimap-based methods, the metrics are only calculated over the transition area, while for trimap-free methods, they are calculated over the whole image. To evaluate methods' generalization ability under PPT setting, we train and evaluate them under three protocols: 1) trained on blurred images, tested on blurred ones (B:B); 2) trained on blurred images and tested on normal ones (B:N); and 3) trained on normal images and tested on normal ones (N:N). All the evaluation are conducted on P3M-500-P validation set for a fair comparison, with or without privacy content preserved. Study on the Impact of PPT -------------------------- ### Impact on Trimap-based Traditional Methods We benchmarked multiple trimap-based traditional methods including Closed [@levin2007closed], IFM [@aksoy2017designing], KNN [@chen2013knn], Compre [@shahrian2013improving], Robust [@wang2007optimized], Learning [@zheng2009learning], Global [@he2011global] and Shared [@gastal2010shared] in Table \[tab:benchmark\_trimap\_based\_traditional\]. As seen from the table, trimap-based traditional methods show neglectable performance variance under different training and evaluation protocols, indicating that the PPT setting brings little impact on these methods. This observation is reasonable, since traditional methods mainly make prediction based on local pixels in the transition area, where no blurring occurs, although a few of sampled neighboring pixels may be blurred. Note that we define the transition area as in previous works but exclude the blurred area. ### Impact on Trimap-based Deep Learning Methods We also benchmarked many trimap-based deep learning methods including DIM [@dim], AlphaGAN [@bmvcLutzAS18], GCA [@li2020natural], IndexNet [@lu2019indices] and FBA [@forte2020fbamatting] in the Table \[tab:benchmark\_trimap\_based\_dl\]. As shown in the table, similar to traditional trimap-based methods, deep learning methods also show very minor changes across different settings. This is because trimap-based deep learning methods use the ground truth trimap as an auxiliary input and focus on estimating the alpha matte of the transition area, probably guiding the model to pay less attention to the blurred areas. In addition, there are also some observations opposed to intuition. When testing on normal images, models trained on the normal training images surprisingly fall behind of those trained on the blurred ones. For instance, the SAD of IndexNet on "N:N" is 0.6 higher than the score on "B:N". Similar results can also be found for AlphaGAN and GCA in Table \[tab:benchmark\_trimap\_based\_dl\]. We suspect that the blurred pixels near the transition area may serve as a random noise during the training process, which makes the model more robust and leads to a better generalization performance. \[t\] ### Impact on Trimap-free Methods Different from trimap-based methods, trimap-free methods show significant performance changes under three protocols. We benchmarked several methods including SHM [@shm], LF [@lf], MODNet [@modnet], HATT [@hatt] and GFM [@gfm] on P3M-10k. We summarize the results in Table \[tab:benchmark\_e2e\] and gain some interesting insights about their generalization abilities under the PPT setting. **First**, we start with evaluating models' generalization abilities and the impact of PPT training by comparing the results on the "B:N" and "N:N" settings. Models trained on normal training images (N:N) usually outperform those using the blurred ones (B:N), from 24.33 to 17.13 in SAD for SHM [@shm]. This observation makes sense since there is a domain gap between blurred images and normal ones due to face obfuscation. By comparison, we found that trimap-free methods show different abilities in dealing with this domain gap. For example, SHM has the largest drop of 7 SAD, while MODNet [@modnet] and GFM [@gfm] only show a drop less than 3 in SAD. We suspect that an end-to-end multi-task framework like GFM and MODNet perform better than others due to their joint optimization nature. By contrast, a two-stage method like SHM [@shm] may produce segmentation errors, which can mislead the following matting stage and can not be corrected. To validate this hypothesis, we devise a baseline model called "BASIC" by adopting a similar multi-task framework like GFM and MODNet but removing the bells and whistles, i.e., only using a sharing encoder and two individual decoders. As shown in Table \[tab:benchmark\_e2e\], the small performance drop (less than 1 in SAD) proves its superiority in overcoming domain gap and supports our hypothesis. **Second**, we further evaluate their performance in the "B:B" setting. The SAD scores vary from 42.95 for LF [@lf] to 11.37 for MODNet [@modnet]. Similarly, the models based on a unified framework like GFM, MODNet and BASIC still perform better than others with sequential matting or two-stage structure, validating the superiority of joint optimization of both sub-tasks. These results suggest that it is better to develop a matting model based on the unified multi-task framework for P3M, which probably has a good performance on face-blurred images as well as generalizes well on normal images. \[sec:ppt\] A Strong Baseline for P3M: P3M-Net {#sec:p3mnet} ================================== P3M-Net Framework ----------------- As discussed in Section \[sec:ppt\], trimap-free matting models benefit from explicitly modeling both semantic segmentation and detail matting tasks and jointly optimizing them in an end-to-end multi-task framework. Therefore, we follow GFM [@gfm] to adopt the multi-task framework, where base visual features are learned from a sharing encoder and task-relevant features are learned from individual decoders, i.e., semantic decoder and matting decoder, respectively. Both decoders have five blocks, each with three convolution layers. Different upsampling operations are used in each task, i.e., bilinear interpolation in semantic decoder for simplicity and max unpooling with indices in matting decoder to preserve fine details. Most of the previous matting methods either model the interaction between encoder and decoder such as the U-Net [@unet] style structure in [@gfm] or model the interaction between two decoders like the attention module in [@hatt]. In this paper, we try to model the comprehensive interactions between the sharing encoder and two decoders through three carefully designed integration modules, i.e., 1) a tripartite-feature integration (TFI) module to enable the interaction between encoder and two decoders; 2) a deep bipartite-feature integration (dBFI) module to enhance the interaction between the encoder and segmentation decoder; and 3) a shallow bipartite-feature integration (sBFI) module to promote the interaction between the encoder and matting decoder. ### TFI: Tripartite-Feature Integration {#sec:tim} Specifically, for each **TFI**, it has three inputs, i.e., the feature map of the previous matting decoder block $\mathbf{F_m^i} \in \mathbb{R}^{C\times H/r \times W/r}$, the feature map from same level semantic decoder block $\mathbf{F_s^i} \in \mathbb{R}^{C\times H/r \times W/r}$, and the feature map from the symmetrical encoder block $\mathbf{F_e^i} \in \mathbb{R}^{C\times H/r \times W/r}$, where $i\in \{1,2,3,4\}$ stands for the block index, $r$ stands for the downsample ratio of the feature map compared to the input size, and $r=2^i$. For each feature map, we use an $1\times1$ convolutional projection layer $\mathcal{P}$ for further embedding and channel reduction. The output of $\mathcal{P}$ for each feature map is $\mathbf{F^i} \in \mathbb{R}^{C/2\times H/r \times W/r}$. We then concatenate the three embedded feature maps and feed them into a convolutional block $\mathcal{C}$ containing a $3\times3$ convolutional layer, a batch normalization layer, and a ReLU layer. As shown in Eq. \[equa:tim\], the output feature is $\mathbf{F_m^i}\in\mathbb{R}^{C\times H/r \times W/r}$: $$\mathbf{F_m^i} = \mathcal{C}(Concat(\mathcal{P}(\mathbf{F^i_m}),\mathcal{P}(\mathbf{F^i_s}),\mathcal{P}(\mathbf{F^i_e}))). \label{equa:tim}$$ \[t\] \[!t\] ### sBFI: Shallow Bipartite-Feature Integration With the assumption that shallow layers in the encoder contain abundant structural detail features and are useful to provide matting decoder with fine foreground details, we propose the shallow bipartite-feature integration (sBFI) module. Specifically, sBFI takes the feature map $\mathbf{E_0}\in \mathbb{R}^{64\times H \times W}$ in the first encoder block as a guidance to refine the feature map $\mathbf{F_m^i}\in\mathbb{R}^{C\times H/r \times W/r}$ from the previous matting decoder block. Here, $i\in \{1,2,3\}$ stands for the layer index, $r$ stands for the downsample ratio of the feature map compared to the input size, and $r=2^i$. Since $\mathbf{E_0}$ and $\mathbf{F_m^i}$ are with different resolution, we first adopt max pooling $MP$ with a ratio $r$ on $\mathbf{E_0}$ to generate a low-resolution feature map $\mathbf{E_0^{'}}\in \mathbb{R}^{64\times H/r \times W/r}$. We then feed both $\mathbf{E_0^{'}}$ and $\mathbf{F_m^i}$ to two projection layers $\mathcal{P}$ implemented by $1\times1$ convolution layers for further embedding and channel reduction, i.e., from $C$ to $C/2$. Finally, the two feature maps are concatenated and and fed into a convolutional block $\mathcal{C}$ containing a $3\times3$ convolutional layer, a bacth normalization layer, and a ReLu layer. As shown in Eq. \[equa:sgr\], we adopt the residual learning idea by adding the output feature map back to the input matting decoder feature map $\mathbf{F_m^i}$. In this way, sBFI helps the matting decoder block to focus on the fine details guided by $\mathbf{E_0}$. $$\mathbf{F_m^i} = \mathcal{C}(Concat(\mathcal{P}(\mathcal{MP}(\mathbf{E_0})),\mathcal{P}(\mathbf{F^i_m})))+\mathbf{F_m^i}. \label{equa:sgr}$$ ### dBFI: Deep Bipartite-Feature Integration Same as sBFI, features in the encoder can also provide valuable guidance to the segmentation decoder. In contrast to sBFI, we chose the feature map $\mathbf{E_4}\in \mathbb{R}^{512\times H/32 \times W/32}$ from the last encoder block, since it encodes abundant global semantics. Specifically, we devise the deep bipartite-feature integration (dBFI) module to fuse it with the feature map $\mathbf{F_s^i}\in\mathbb{R}^{C\times H/r \times W/r}$ from the $i$th segmentation decoder block to improve the feature representation ability for the high-level semantic segmentation task. Here, $i\in \{1,2,3\}$. Note that since $\mathbf{E_4}$ is in low-resolution, we use a upsampling operation $UP$ with a ratio $32/r$ on $\mathbf{E_4}$ to generate $\mathbf{E_4^{'}}\in \mathbb{R}^{512\times H/r \times W/r}$. we then feed both $\mathbf{E_4^{'}}$ and $\mathbf{F_s^i}$ into two projection layers $\mathcal{P}$, concatenated together, and fed into a convolutional block $\mathcal{C}$. We adopt the identical structures for $\mathcal{P}$ and $\mathcal{C}$ as those in sBFI. Similarly, this process can be described as follows. Note that we reuse the symbols of $\mathcal{C}$ and $\mathcal{P}$ in Eq. \[equa:tim\], Eq. \[equa:sgr\], and Eq. \[equa:dgr\] for simplicity, although each of them denotes a specific layer (block) in TFI, sBFI, and dFI, respectively. $$\mathbf{F_s^i} = \mathcal{C}(Concat(\mathcal{P}(\mathcal{UP}(\mathbf{E_4})),\mathcal{P}(\mathbf{F^i_s})))+\mathbf{F_s^i}. \label{equa:dgr}$$ P3M-Net Variants {#sec:variants} ---------------- By integrating the above three modules into the P3M-Net framework, we are able to utilize the visual features learned from the sharing encoder comprehensively and exchange the features at global and local levels promptly. However, how to improve the representation ability of the sharing encoder to make it feasible for portrait matting remains as a challenge. As shown in Figure \[fig:encoder\], we design five blocks $\mathbf{E_0}$ to $\mathbf{E_4}$ in the encoder, adopt maxpooling layers instead of strided convolution in the sharing encoder to reserve the details in the encoder as much as possible. Specifically, in $\mathbf{E_0}$ and $\mathbf{E_1}$, there are several convolution layers and maxpooling layers involved. From $\mathbf{E_2}$ to $\mathbf{E_4}$, each block has a series of P3M Basic Blocks (P3M BB) and one maxpooling layer, while $\mathbf{E_4}$ has an extra series of P3M Basic Blocks. In the following part, we design multiple variants of P3M Basic Blocks based on CNN and vision transformers. We leverage the ability of transformers in modeling long-range dependency to extract more accurate global information and the locality modelling ability to reserve lots of details in the transition areas, and expect they can deliver better performance on both face-blurred and non-privacy images under the PPT setting. The structures of these variants are illustrated in Figure \[fig:encoder\] and the details are presented as follows. ### P3M BB (ResNet-34) Following previous matting methods that adopt CNN basic block in their structures [@aim; @hatt; @shm], we use the residual block in ResNet-34 [@he2016deep] as our basic version of P3M BB. In particular, it consists of two convolution layers followed by a BN and ReLU. The residual structure is used to ease the training process. The number of the basic blocks in the sharing encoder from $\mathbf{N_1}$ to $\mathbf{N_4}$ are 3, 4, 6, 3 by following the original design of ResNet-34. The numbers of output channels from $\mathbf{C_1}$ to $\mathbf{C_4}$ in this case are 256, 128, 64 and 64. Based on the above P3M BB, we obtain the "P3M-Net (ResNet-34)" variant, which is able to extract the contour of the human and achieves good results on most cases, as demonstrated in Figure \[fig:variants\_compare\]. Nevertheless, there are still room for further improvement in regards to both accurate semantic information and precise details when facing the complex case. The shortage of using CNN-only basic block is twofold: 1) it is still sensitive to blurred artefacts, resulting in wrong prediction around face region as shown in the red box in the figure; and 2) the limited ability of CNN in modeling long-range dependency leads to error semantics in the alpha matte prediction as shown in the blue box in the figure. ### P3M BB (Swin-T) Based on the above analysis, it is necessary to model the long-range dependency among pixels to enhance the model's ability in perceiving different semantic content. A reasonable solution is to adopt transformer-based basic block in our sharing encoder. As shown in Figure \[fig:encoder\], we leverage the Swin [@swin] transformer layers as the P3M BB, which consists of a shifted window based multi-head self attention (MSA) module with a 2-layer MLP with GELU non-linearity in between. Compared with the original full attention mechanism, the window based MSA is more computationally efficient while maintaining the ability of modelling long-range dependency. The number of the basic blocks used in the sharing encoder from $\mathbf{N_1}$ to $\mathbf{N_4}$ are 2, 2, 6, 2 by following the original design of Swin-T. The numbers of output channels from $\mathbf{C_1}$ to $\mathbf{C_4}$ in this case are 384, 192, 96 and 96. Based on the Swin-based P3M BB, we obtain the "P3M-Net (Swin-T)" variant. As shown in Figure \[fig:variants\_compare\], it has better ability in understanding the foreground semantics, i.e., 1) becomes more insensitive to blurring artefacts as shown in the red box; and 2) provides correct semantic results compared with its CNN counterpart as shown in the blue box. Nevertheless, the predicted alpha matte of the transition area as enclosed by the green and yellow boxes misses some details. We suspect it is caused by the lack of the locality modelling ability of convolutions. ### P3M BB (ViTAE-S) Based on the above observation and analysis, we find that it is important to take the advantage of transformer-based structure for modelling long-range dependency and CNN-based structure for modelling locality, which coincide with the inherent requirement of the semantic segmentation task and detail matting task in trimap-free matting. To this end, we adopt the NC block proposed in ViATE [@vitae] as our P3M BB. It consists of two parallel branches responsible for modeling long-range dependency and locality respectively, followed by a feed forward network for feature transformation. Specifically, the transformer branch consists of a multi-head self-attention module, while the local branch consists of two groups of convolution, BN, SiLU layers followed by another one convolution layer and SiLU layer. The number of the basic blocks used in the sharing encoder from $\mathbf{N_1}$ to $\mathbf{N_4}$ are 2, 2, 12, 2 as the original design of ViTAE-S. The numbers of output channels from $\mathbf{C_1}$ to $\mathbf{C_4}$ are 256, 128, 64 and 64. Based on the ViTAE-based P3M BB, we obtain the "P3M-Net (ViTAE-S)" variant. As can be seen from Figure \[fig:variants\_compare\], compared with the CNN-only and Transformer-only basic blocks, the ViTAE-based one is able to predict correct foreground as well as fine details in the transition area simultaneously. As shown in the yellow and green boxes, the fine details of small but meticulous objects like the earring and hairpin can be distinguished clearly in the predicted alpha matte. More results of the three variants are shown in Section \[sec:exp\]. A Simple yet Effective Data and Feature Augmentation Strategy for P3M: P3M-CP {#sec:p3mcp} ============================================================================= Overview of P3M-CP ------------------ ![Some failure results of MODNet (Top) and P3M-Net (Swin-T) (Bottom) under the PPT setting on non-privacy images.<span data-label="fig:p3m-cp-motivation"></span>](cp_motivation.jpg){width="\linewidth"} ![image](p3m-cp.png){width="\linewidth"} Although the P3M-Net model variants achieve better performance than previous methods as shown in the bottom rows in Table \[tab:benchmark\_e2e\] and Figure \[fig:variants\_compare\], there is still a performance gap when testing on face-blurred images and generalizing on normal non-privacy images, especially for CNN-based P3M-Net. How to compensate for the lack of facial details in face-blurred training data to reduce the performance gap remains unresolved. Without seeing faces during training, the matting model is less effective in discriminating them from background, thereby affecting its generalization ability on non-privacy images. Some failure examples are shown in Figure \[fig:p3m-cp-motivation\]. To mitigate this problem, we devise a simple yet effective Copy and Paste strategy (P3M-CP) that can borrow facial information from publicly available celebrity images without privacy concerns and guide the network to reacquire the face context at both data and feature level. As shown in Figure \[fig:p3m-cp\], P3M-CP adopts a Copy and Paste Module $\mathcal{CP}$ to process the images (or the features of images generated by part of the matting model, i.e., $\mathcal{G}_1$) from the source domain $\mathbf{S}$ and target domain $\mathbf{T}$ and generate the merged data $\mathbf{D'}$, which is fed into the complete matting model $\mathcal{G}_1$ and $\mathcal{G}_2$ (or the remaining part of the matting model, i.e., $\mathcal{G}_2$). The source domain $\mathbf{S}$ consists of external facial images or portrait images $\mathbf{I_S}$ without privacy concerns, along with their facemasks $\mathbf{M_S}$ generated in advance. The target domain $\mathbf{T}$ contains the face-blurred images $\mathbf{I_T}$ from the training set of P3M-10k, along with their facemasks $\mathbf{M_T}$ that indicate which part is obfuscated to protect privacy. The copy and paste module $\mathcal{CP}$ takes both the source domain data and target domain data as inputs, copy the face region from the source data and then paste it onto the target data. In this module, the source and target data can be either images or the features extracted by part of matting model ($\mathcal{G}_1$), resulting in P3M-ICP at the image level and P3M-FCP at the feature level. **Copy and Paste Module**. As shown in the yellow box in Figure \[fig:p3m-cp\], the copy and paste module $\mathcal{CP}$ consists of two steps, (1) $\mathcal{CA}$: copy and augment, and (2) $\mathcal{AM}$: align and merge. No extra learnable weights are required during this process. First, $\mathcal{CA}$ takes the source data $\mathbf{D_S}$ (images or features) and its facemask $\mathbf{M'_S}$ as input. The original facemask is resized to the same size as the source data, denoted $\mathbf{M'_S}$. Then, the face area $\mathbf{D_S^{face}}$ in the source data $\mathbf{D_S}$ is cut out based on the facemask $\mathbf{M_S^{'}}$ and augmented by random resize and rotation. Similarly, the face-blurred area $\mathbf{D_T^{blur}}$ is cropped out from the target data $\mathbf{D_T}$ based on the resized facemask $\mathbf{M'_T}$ without augmentation. Second, the augmented source face $\mathbf{D_S^{face}}$ and the target face-blurred part $\mathbf{D_T^{blur}}$ are further processed by $\mathcal{AM}$. Specifically, the center points $\mathbf{P_S}$ and $\mathbf{P_T}$ of $\mathbf{D_S^{face}}$ and $\mathbf{D_T^{blur}}$ are calculated based on their facemasks. After aligning $\mathbf{D_S^{face}}$ and $\mathbf{D_T^{blur}}$ according to the center points, we paste the overlap region from $\mathbf{D_S^{face}}$ to $\mathbf{D_T^{blur}}$, we get the merged data $\mathbf{D'}$. This $\mathcal{CP}$ process can be applied on both images and features, depending on the type of inputs, which can be formulated as follows: $$\mathbf{D_S^{face}} = Rot(RS(\mathbf{D_S} \odot \mathbf{M'_S})),\ \mathbf{D_T^{blur}} = \mathbf{D_T} \odot \mathbf{M'_T}, \label{equa:cp-process-1}$$ $$\mathbf{P_S} = center(\mathbf{D_S^{face}}), \ \mathbf{P_T} = center(\mathbf{D_T^{blur}}), \label{equa:cp-process-2}$$ $$\mathbf{D'} = \mathcal{AM}(\mathbf{D_S^{face}}, \mathbf{D_T^{blur}}, \mathbf{P_S}, \mathbf{P_T}, \mathbf{D_T}), \label{equa:cp-process-3}$$ where $Rot$, $RS$, $center$ denote the rotate, resize, and calculate the center operations, respectively. ![Illustration of the process of generating facemask for the source image. **left top**: An image from CelebAMask-HQ [@CelebAMask-HQ]. **right top**: the annotation of facial parts. The red line is placed at right above the brows. **left bottom**: the generated facemask. **right bottom**: the face region.<span data-label="fig:celeba_anno"></span>](celeba_facemask.png){width="0.8\linewidth"} **Source Data and Facemask Annotation.** It is noteworthy that the source data $\mathbf{I_S}$ do not need to have alpha matte labels, which require neglectable effort to collect. Nevertheless, they should contain clear faces, which is quite difficult to collect from ordinary people in the context of privacy preserving. Luckily, we find that public celebrity images can be used for non-commercial purposes by law, which have no privacy issues. Therefore, we adopt the existing celebrity face dataset CelebAMask-HQ [@CelebAMask-HQ] as the source images for P3M-CP, which also provides the annotations of facial parts. We use the annotated masks of 'skin' and 'brow' to extract the accurate face region. The process is illustrated in Figure \[fig:celeba\_anno\]. Specifically, we first determine a line right above the left and right brows, and then take the face skin under the line as the face region. P3M-ICP: Copy and Paste at Image Level -------------------------------------- With source face images and facemasks prepared, the P3M-CP module aim to capture the facial information and use it to guide the learning process of matting models. P3M-ICP accomplishes it at image level, i.e., directly applying "copy and paste" process on images. The process is shown in Figure \[fig:p3m-cp\], where the copy and paste module in the yellow box is applied directly on images in front of matting model $\mathcal{G}_1$. Note that in P3M-ICP, source images $\mathbf{I_S}$ and source data $\mathbf{D_S}$ are equivalent. So are the target images $\mathbf{I_T}$ and target data $\mathbf{D_T}$. When training with P3M-ICP, we randomly select a pair of images from CelebAMask-HQ dataset and the training set, and apply P3M-ICP on them with a probability of 0.5. Some examples are provided in Figure \[fig:data-cp-examples\]. P3M-ICP is an easy-to-implement plug-in module without extra learnable parameters, which can serve as a flexible data augmentation method and is compatible with any matting model. Besides, it only brings a few additional computations during training, while enabling the matting model to process both face-blurred images and the non-privacy ones pretty well without extra effort during inference. Although P3M-ICP shows superior generalization ability on non-privacy images (see Section \[sec:p3mcp-results\]), it can be improved in the future in many aspects. For example, the face-context may be inconsistent between the source and target images, regarding the size, pose, emotion, gender of the face. More research efforts are needed to match those attributes of source and target images. ![P3M-ICP examples. Top: merged image after P3M-ICP, source face-blurred image, and source facemask. Bottom: source image and source facemask.<span data-label="fig:data-cp-examples"></span>](icp_examples.jpg){width="\linewidth"} P3M-FCP: Copy and Paste at Feature Level ---------------------------------------- Directly cropping the face region out from the source image neglects the face context information, which may result in an incomplete face representation. To address this issue, we propose P3M-FCP that captures and merges face and context information at feature level. As shown in Figure \[fig:p3m-cp\], instead of straight removing the context residing in source image and only pasting the face area onto the target image, P3M-FCP first extract features $\mathbf{D_S}$ and $\mathbf{D_T}$ of both source image $\mathbf{I_S}$ and target face-blurred image $\mathbf{I_T}$ using the first few layers in the encoder of the matting model, denoted as $\mathcal{G}_1$. In this way, the face context information in the source image could be propagated to and embedded in the face area. Then, $\mathcal{CP}$ is conducted on the features $\mathbf{D_S}$ and $\mathbf{D_T}$, along with their resized facemasks $\mathbf{M'_S}$ and $\mathbf{M'_T}$, same as that in P3M-ICP. Finally, we get the merged feature $\mathbf{D'}$ from the P3M-FCP module and feed it to the remaining part of the matting model, denoted as $\mathcal{G}_2$. Similar to P3M-ICP, this process can be formulated as follows: $$\mathbf{D_S} = \mathcal{G}_1(\mathbf{I_S}), \mathbf{D_T} = \mathcal{G}_1(\mathbf{I_T}), \label{equa:p3m-fcp-1}$$ $$\mathbf{D'} = \mathcal{CP}(\mathbf{D_S}, \mathbf{M'_S}, \mathbf{D_T}, \mathbf{M'_T}), \label{equa:p3m-fcp-2}$$ where $\mathcal{CP}$ is defined as in Eq. \[equa:cp-process-1\] $\sim$ Eq. \[equa:cp-process-3\]. In implementation, we pass both the source image $\mathbf{I_S}$ and the target training image $\mathbf{I_T}$ through the encoder of the matting model, and only apply the P3M-FCP module on a single selected layer in the encoder with a probability of 0.5. Usually, using P3M-FCP on a shallow layer could deliver better results. After P3M-FCP, the merged features go through the remaining part of the matting model, while the features of the source image are abandoned. Besides, the gradients corresponding to the the source image would not be back-propagated. At each iteration, only one source image is selected for each mini-batch of training images. The batch size is usually set to 8. Compared to P3M-ICP, the P3M-FCP module can extract more face context information residing in source images and merge the source and the target features in a more consistent manner through end-to-end training. Therefore, with less demand on source data, P3M-FCP can achieve better generalization performance on non-privacy images (see Section \[sec:p3mcp-results\]). \[htb\] \[htb\] Experiments {#sec:experiments} =========== Experiment Settings ------------------- To compare the proposed P3M-Net with existing trimap-free methods, such as SHM [@shm], LF [@lf], HATT [@hatt], GFM [@gfm], and MODNet [@modnet], we train them on the P3M-10k face-blurred images and evaluate them on 1) the face-blurred validation set P3M-500-P, 2) the normal validation set P3M-500-NP, following the PPT setting, 3) P3M-500-P, processed by different privacy-preserving methods, e.g., blurring, mosaicing, and masking, and 4) RWP test set @yu2021mask consisting of 636 normal portrait images for testing. Furthermore, we apply P3M-ICP and P3M-FCP on MODNet [@modnet] and the P3M-Net variants during training, and validate them on P3M-500-P and P3M-500-NP. **Implementation Details** For training P3M-Net, we crop a patch from the image with a size randomly chosen from $512\times512$, $768\times768$, $1024\times1024$, and then resize it to $512\times512$. We randomly flip the patches for data augmentation. The learning rate is fixed as $1\times10^{-5}$. We train P3M-Net variants on NVIDIA Tesla V100 GPUs with a batch size of 8 for 150 epochs, which takes about 2 days. It takes 0.132s to test on an $800\times800$ image. For GFM [@gfm], LF [@lf] and MODNet [@modnet], we use the code provided by the authors. For SHM [@shm], HATT [@hatt] and DIM [@dim] which have no official code, we re-implement them. For P3M-ICP or P3M-FCP, we apply them on each image with a probability of 0.5 during training. **Evaluation Metrics** We follow previous works and adopt the evaluation metrics including the sum of absolute differences (SAD), mean squared error (MSE), mean absolute difference (MAD), gradient (Grad.) and Connectivity (Conn.) [@rhemann2009perceptually]. We calculated them over the whole image for trimap-free methods. We also report the SAD-T, MSE-T, MAD-T metrics to calculate the score within the transition area, and SAD-FG, SAD-BG to calculate the score within the foreground and background, respectively. Results and Analysis of P3M-Net Variants ---------------------------------------- ### Objective and Subjective Results {#sec:exp} The objective and subjective results of the proposed P3M-Net variants and some representative methods are shown in Table \[tab:experiment\], Figure \[fig:variants\_compare\_results\_val500p\], and Figure \[fig:variants\_compare\_results\_val500np\], respectively. As can be seen, all variants of P3M-Net outperform the previous trimap-free methods in all metrics and even achieve competitive results with trimap-based method DIM [@dim], which requires the ground truth trimap as an auxiliary input, denoted as DIM$\star$. These results validate the design of the three integration modules which are able to model abundant interactions between encoder and decoder as well as the three P3M BB variants which leverage the advantages of long-range dependency modelling ability and locality modelling ability. As for SHM [@shm], it has worse SAD than all P3M-Net variants on both validation sets, i.e., 21.56 v.s. 6.24 and 20.77 v.s. 7.59, due to its two-stage pipeline, which produces many segmentation errors that cannot be corrected in the following stage. LF [@lf] and HATT [@hatt] have large error in transition areas, e.g., 12.43 and 11.03 SAD v.s. 5.65 SAD of ours, since they lack explicit semantic guidance for the matting task. As in Figure \[fig:variants\_compare\_results\_val500p\], they have ambiguous segmentation results and inaccurate matting details. MODNet [@modnet] and GFM [@gfm] are able to predict more accurate foreground and background owing to the multi-task learning framework. However, they may fail to predict correct context and have worse performance than ours, i.e., 13.32 and 13.20 v.s. 6.24 in SAD, since it lacks of exploring feature interactions between encoder and decoders. DIM [@dim] has lower SAD than ours since it uses ground truth trimap. Nevertheless, all P3M-Net variants still achieve competitive performance in the transition area, e.g., 6.89 v.s. 4.89 SAD. \[hbtp\] \[hbtp\] Although all three P3M-Net variants have already surpasses the SOTA methods, they show different performance due to the use of different backbones. As discussed in Section \[sec:variants\], comparing with P3M-Net (ResNet-34), P3M-Net (Swin-T) reduces the foreground and
{'source': 'arXiv_out/2203'}