Automatically Extract the Semi-transparent Motion-blurred Hand from a Single Image
Xiaomei Zhao Yihong Wu








Automatically Extract the Semi-transparent Motion-blurred Hand from a Single Image
     Xiaomei Zhao, Yihong Wu
This work was supported by the National Natural Science Foundation of China under 61836015 , 61421004 , 61572499.
X. Zhao is with the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China, and also with the University of Chinese Academy of Sciences, Beijing, China.
Y. Wu is with the National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China, and also with the University of Chinese Academy of Sciences, Beijing, China (e-mail: yhwu@nlpr.ia.ac.cn).
    December 30, 2023
=============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================





When we use video chat, video game, or other video applications, motion-blurred hands often appear. Accurately extracting these hands is very useful for video editing and behavior analysis. However, existing motion-blurred object extraction methods either need user interactions, such as user supplied trimaps and scribbles, or need additional information, such as background images. In this paper, a novel method which can automatically extract the semi-transparent motion-blurred hand just according to the original RGB image is proposed. The proposed method separates the extraction task into two subtasks: alpha matte prediction and foreground prediction. These two subtasks are implemented by Xception based encoder-decoder networks. The extracted motion-blurred hand images can be calculated by multiplying the predicted alpha mattes and foreground images. Experiments on synthetic and real datasets show that the proposed method has promising performance.




Motion-blurred hand, Semi-transparent, Alpha matte prediction, Foreground prediction, Automatically









§ INTRODUCTION


Hand language is one of the most important human gesture languages. Poor hand extraction results can greatly reduce the performance of image or video editing, behavior recognition, and behavior analysis. Extracting hands can be implemented by hand segmentation methods [1, 2]. However these methods can’t deal with motion-blurred hands, which are very common in practical applications. Existing traditional methods [3-5], which were designed to predict the alpha mattes or foreground images of motion-blurred objects, generally need user interactions [3, 4] or short-exposure frames [5]. Zhao et. al. [6] proposed a deep learning network to predict the alpha mattes of motion-blurred hands, and then extracted motion-blurred hands by subtracting background components from the original images. A simple flow chart of this method is shown in Fig. 1 (a). An obvious drawback of this method is that it needs background images, which are inconvenient to obtain. In this paper, we propose a method which can automatically extract semi-transparent motion-blurred hands just according to the original RGB images, without requiring any additional information.





An image I, which contains a motion-blurred hand, is made up by combining foreground hand F and background B: I=α· F+(1-α)· B,α∈[0,1], where α is called alpha matte. As shown in this equation, both α and F are related to motion-blurred hands: α demonstrates their transparency; F demonstrates their color. The proposed method separates the task of extracting semi-transparent motion-blurred hand into two subtasks: alpha matte prediction and foreground prediction. The extracted hands can be calculated by multiplying the predicted α and F. A simple flow chart of the proposed method is shown in Fig. 1 (b).

Alpha mattes can be calculated by matting methods [7-11]. However, most of matting methods need additional information, such as user supplied trimaps and scribbles [7-11]. In order to avoid the need for user interactions, several matting methods [12-14] employ CNN networks to predict trimaps explicitly or implicitly. The above matting methods focus on static objects, rather than motion-blurred objects. Zhao et. al. [6] proposed a motion-blurred object matting network which only used RGB images as inputs and directly outputted the predicted alpha mattes. In this paper, our alpha matte prediction network is developed from the matting network in [6] by adding a perceptual loss [15].



Up to now, most of matting methods concentrate on alpha matte prediction. Very few methods can predict foreground images. A recently proposed sampling- and learning-based matting method [11] can estimate the foreground color of unknown regions. However, this method needs user supplied trimaps to annotate the background, foreground, and unknown regions respectively. Besides, this method focuses on static objects, rather than motion-blurred object. In this paper, a network which can automatically predict the foreground images of motion-blurred hands is proposed. The foreground prediction network is a Xception [16] based encoder-decoder network. It only uses the original RGB images as inputs and directly outputs the predicted foreground images. During training, L1-loss and perceptual loss [15] are employed. As shown in Fig. 2, the proposed foreground prediction network shares the same encoder with alpha matte prediction network. But these two prediction networks have different decoders.

Human matting methods [12, 13] extracted human just by α· I, rather than α· F. A human image I can also be formulated as I=α· F+(1-α)· B. Therefore, in semi-transparent areas where 0<α<1, α· I still contains background information. However, in human images, almost all of the pixels, whose alpha values are between 0 and 1, are located at isolated hairs. Isolated hairs are very thin and have low transparency. Thus, in human images without motion blur, the background information contained in α· I is not obvious and α· I ≈α· F. In contrast, in motion-blurred hand images, the areas with high transparency are large. If we extract motion-blurred hands by α· I, the extracted hand images will contain obvious background information, which can greatly reduce the sense of reality when changing background. Thus,  motion-blurred hands should be calculated by α· F, rather than α· I.


In summary, the main contributions in this paper are:

(1)A novel framework which can automatically extract the semi-transparent motion-blurred object from a single image is proposed. This framework consists of two main parts: alpha matte prediction and foreground prediction. The images of extracted objects are calculated by multiplying the predicted alpha mattes and foreground images.

(2)The proposed framework is employed in semi-transparent motion-blurred hand extraction task. Alpha matte prediction and foreground prediction are implemented by Xception-based encoder-decoder networks and trained by L1-loss and perceptual loss.

(3)We enlarge the synthetic motion-blurred hand dataset proposed in [6], and use the enlarged dataset to train the proposed model. Then we use the trained model to process real videos. Experiments demonstrate that the proposed method has promising performance.




§ METHOD


The architecture of the proposed method is shown in Fig. 2. As shown in this figure, the CNN network outputs two kinds of results: predicted alpha mattes and predicted foreground images. The images of extracted hands are calculated by multiplying these two kinds of results.



 §.§ Network architecture


Encoder-decoder networks have demonstrated their great performance on many pixel-to-pixel prediction tasks, such as segmentation[17-19], depth prediction[20], matting[9, 21], and so on. Encoder-decoder networks usually employ pre-trained image recognition networks, such as VGG[22], ResNet[23], and Xception[16], as the backbone of encoder, and employ several upsampling blocks as decoder. Previous article [17] has shown that encoder-decoder networks based on Xception have better performance and faster speed. Therefore, in this paper, we employ Xception based encoder-decoder networks for alpha matte prediction and foreground prediction. As shown in Fig. 2, the proposed alpha matte prediction network and foreground prediction network share the same encoder, which contains 4 downsampling steps. In each step, the downsampling ratio equals to 0.5. These two prediction networks have two independent decoders. Each decoder contains 3 upsampling steps. The upsampling ratios equal to 4, 2, 2 respectively. In each upsampling step, skip connection is used to recover spatial information. The decoders for alpha matte prediction and foreground prediction have similar structure. But the output of alpha prediction decoder has one channel, while the output of foreground prediction decoder has three channels, which are red channel, green channel, and blue channel respectively.



 §.§ Loss function


For most of pixel-to-pixel prediction methods, including matting methods [9-11, 13, 14], pixel-wise losses, such as pixel-wise L1-loss and L2-loss, are generally used. However, pixel-wise loss ignores the correlation among pixels. A solution to this problem is employing Conditional Random Fields (CRF) [24-26]. However CRF runs slowly. Another solution is employing perceptual loss. Perceptual loss has been successfully used for style transfer and super-resolution [15, 27]. It calculates the differences between high-level features extracted from predicted images and groundtruth images, and minimizes these differences by backpropagation. High-level features can be extracted by pre-trained convolutional networks.

In this paper, the overall loss L^o=L^α+L^f, where L^α denotes alpha prediction loss, L^f denotes foreground prediction loss. Both L^α and L^f are made up by combining L1 losses and perceptual losses.



  §.§.§ Alpha prediction loss


The alpha prediction loss used in this paper contains three parts: alpha absolute loss l_ab^α, alpha compositional loss l_c^α, and alpha perceptual loss l_p^α. l_ab^α is the L1 loss between predicted alpha mattes and groundtruth alpha mattes. l_c^α is the L1 loss between predicted compositional images and groundtruth compositional images. Compositional images are generated by I_c=α_⋆· F+(1-α_⋆)· B, where F and B are given foreground images and background images, α_⋆ denotes predicted alpha mattes or groundtruth alpha mattes.


The alpha perceptual loss l_p^α calculates the L2 loss between high-level features extracted from predicted alpha mattes and groundtruth alpha mattes. In our experiments, VGG-16 [22], which is pretrained for image recognition and contains 5 convolutional blocks, is used as the feature extractor. All of the 5 level feature maps extracted by these 5 convolutional blocks are used to calculate perceptual loss. It should be mentioned that VGG-16 is just employed as a feature extractor, which is only used during training.

The overall alpha prediction loss L^α=λ_ab^α l_ab^α+λ_c^α l_c^α+λ_p^α l_p^α, where λ_ab^α, λ_c^α, and λ_p^α are the loss weights. In our experiments, λ_ab^α and λ_c^α are set to 0.5. λ_p^α is set to 0.001.



  §.§.§ Foreground prediction loss


The foreground prediction loss contains two parts: foreground absolute loss l_ab^f  and foreground perceptual loss l_p^f. l_ab^f is the  L1 loss between predicted foreground images and groundtruth foreground images. l_p^f is the L2 loss between high-level features extracted from predicted foreground images and groundtruth foreground images. For l_p^f, VGG-16 is also used as the feature extractor and all of its 5 levels of feature maps are employed. The overall foreground prediction loss L^f=λ_ab^f l_ab^f+λ_p^f l_p^f, where λ_ab^f and λ_p^f are the loss weights. In our experiment, they are set to 1 and 0.001 respectively.



§ EXPERIMENT


In motion-blurred object extraction task, it is almost impossible for human to assign accurate alpha value and foreground color to each image pixel. Therefore, it is very difficult to generate real dataset to train the proposed motion-blurred hand extraction model. In order to solve this problem, synthetic dataset provided by [6] is employed. Besides, in order to increase the diversity of skin colors,  we enlarge this dataset by the same synthetic dataset generation method in  [6], which can generate synthetic motion-blurred hand images, and their corresponding alpha mattes and foreground images. In our experiment, we generate 3 synthetic datasets: training dataset, validation dataset, and testing dataset. These 3 datasets contain 30279, 8283, and 10140 cases respectively. In this paper, all of our models are trained on the synthetic training dataset. The synthetic validation dataset is used to monitor the training processes. The synthetic testing dataset is used to evaluate the performance of our trained models. Then, according to the evaluation results on synthetic testing dataset, the best performing model is chosen to process real videos. In this paper, real videos are captured by a camera on a mobile phone.

Our experiments are implemented under tensorflow with one Nvidia RTX 2080ti GPU and one Intel Core i7 9700k CPU.




 §.§ Evaluation on synthetic testing dataset







In this paper, four models are trained. These four models are called as Model 1, Model 2, Model 3, and Model 4 respectively. Particularly, Model 1 is trained without perceptual loss and its two subtasks do not share the same encoder; Model 2 is trained without perceptual loss and its two subtasks share the same encoder; Model 3 is trained with perceptual loss and its two subtasks do not share the same encoder; Model 4 is trained with perceptual loss and its two subtasks share the same encoder. The alpha matte prediction network of Model 1 is the same matting network proposed in [6]. The evaluation scores on the synthetic testing dataset are shown in Table 1. As shown in this table, Model 4 has the best performance. Therefore, Model 4 is chosen to process real videos in next subsection. In our experiment, during training Model 4, the learning rate was set to 3.5×10^-3; the momentum was set to 0.9; the weight decay was set to 4×10^-5. The training process ended after 80 epoches.


As shown in Table 1, sharing the same encoder can slightly improve the prediction performance, while adding perceptual losses can obviously improve the prediction performance. In order to show the effectiveness of perceptual loss qualitatively, several alpha prediction and foreground prediction examples of Model 2 and Model 4 are shown in Fig. 3. Model 2 and Model 4 have the same network structure  and are trained without and with perceptual loss respectively. Fig. 3 shows that the results predicted by Model 4 have more accurate and reasonable shapes and textures than Model 2.



 §.§ Experiments on real videos


As described in last subsection, the best-performing model, Model 4, is chosen to predict the alpha mattes and foreground images of motion-blurred hands in real videos. The extracted hand images are calculated by multiplying the predicted alpha mattes and foreground images. Because it is very hard to generate groundtruth for real datasets, in this subsection, we just show and compare the motion-blurred hand extraction performances qualitatively.



  §.§.§ Compare with other methods




The proposed motion-blurred hand extraction method is compared with other two previous methods. One method extracts motion-blurred hands simply by multiplying the predicted alpha mattes and original frames. This strategy is popularly used in human extraction methods [12, 13]. The other method extracts motion-blurred hands by subtracting the background component from the original frames [6]. Its flow chart is shown Fig.1 (a). In the following, the above two methods are called as Method 1 and Method 2 for convenience. As shown in Fig. 4, the hand images extracted by Method 1 contains obvious background information, and the hand images extracted by Method 2 contains obvious color distortions. In contract, the hand images generated by our method are smooth and natural. In addition, Method 2 needs background images, which are not convenient to obtain, while our method doesn't need any additional information. Therefore, our method has the best performance and can be applied more widely.



  §.§.§ Modify human segmentation results




In previous work [6], the extracted motion-blurred hand images were used to modify human segmentation results. In this paper, we also do the same job, in order to verify that the proposed motion-blurred hand extraction method is useful in practical applications. The method used in [6] needs background images. Therefore, in [6], videos were captured by static cameras in order to obtain background images from the nearby frames. In contrast, the method proposed in this paper doesn’t need background images and can be used in videos captured by moving cameras. The examples shown in Fig. 5 are captured by a hand-held camera. Particularly, we hold camera by one hand and make hand gestures by the other hand. In Fig. 5, the 4 examples are captured at home, at workplace, in a park, and near a road respectively. These examples demonstrate that state-of-art segmentation method can’t deal with motion-blurred hands, and our motion-blurred hand extraction method has good performance to modify human segmentation results and change background.




§ CONCLUSION AND FUTURE WORK


In this paper, a novel method which can automatically extract the semi-transparent motion-blurred hand from a single image is proposed. This novel method contains two main parts: alpha matte prediction and foreground prediction. The images of extracted hands are calculated by multiplying the predicted alpha mattes and foreground images. In the future, we will try to find an efficient method which can extract complete motion-blurred objects, such as motion-blurred human.

34



S. Bambach, S. Lee, D. J. Crandall, and C. Yu, "Lending a hand: detecting hands and recognizing activities in complex egocentric interactions," in Proc. IEEE International Conference on Computer Vision, Santiago, Chile, 2015, pp. 1949–1957. [Online]. Available: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7410583


A. Urooj and A. Borji, "Analysis of hand segmentation in the wild," in Proc. IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 4710–4719. [Online]. Available: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8578593


H. T. Lin, Y.-W. Tai, and M. S. Brown, "Motion regularization for matting motion blurred objects," IEEE transactions on pattern analysis and machine intelligence, vol. 33, issue 11, pp. 2329–2336, Nov. 2011. doi: 10.1109/TPAMI.2011.93.


R. Kohler, M. Hirsch, B. Scholkopf, and S. Harmeling, "Improving alpha matting and motion blurred foreground estimation," in Proc. IEEE International Conference on Image Processing, Melbourne, VIC, Australia, 2013, pp. 3446–3450. [Online]. Available: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6738711


H. Myeong, S. Lin, and K. M. Lee, "Alpha matting of motion-blurred objects in bracket sequence images," in Proc. European Conference on Computer Vision, Zurich, Switzerland, 2014, pp. 125–139. [Online]. Available: https://link.springer.com/content/pdf/10.1007%2F978-3-319-10578-9_9.pdf


X. Zhao and Y. Wu, "Improving alpha matting and motion blurred foreground estimation,"  IEEE International  Conference on Image Processing, Taipei, Taiwan, 2019. (accepted)


 A. Levin, D. Lischinski, and Y. Weiss, "A closed-form solution to natural image matting," IEEE transactions on pattern analysis and machine intelligence, vol. 30, issue 2, pp. 228–242, Feb. 2008. doi: 10.1109/TPAMI.2007.1177.


 Q. Chen, D. Li, and C.-K. Tang, "KNN matting," IEEE transactions on pattern analysis and machine intelligence, vol. 35, issue 9, pp. 2175–2188, Sept. 2013. doi: 10.1109/TPAMI.2013.18.


N. Xu, B. Price, S. Cohen, and T. Huang, "Deep image matting," in Proc. IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017, pp. 311–320. [Online]. Available: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8099524


D. Cho, Y.-W. Tai, and I. S. Kweon, "Deep convolutional neural network for natural image matting using initial alpha mattes," IEEE Transactions on Image Processing, vol. 28, issue 3, pp. 1054–1067, March 2019. doi: 10.1109/TIP.2018.2872925.


J. Tang, Y. Aksoy, C. Oztireli, M. Gross, and T. Ozan Aydin, "Learning-Based Sampling for Natural Image Matting," in Proc. IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019, pp. 3055-3063. [Online]. Available: http://openaccess.thecvf.com/content_CVPR_2019/papers/Tang_Learning
    -Based_Sampling_for_Natural_Image_Matting_CVPR_2019_paper.pdf


X. Shen, X. Tao, H. Gao, C. Zhou, and J. Jia, "Deep automatic portrait matting," in Proc. European Conference on Computer Vision, Amsterdam, Netherlands, 2016, pp. 92–107. [Online]. Available: https://link.springer.com/content/pdf/10.1007%2F978-3-319-46448-0_6.pdf


Q. Chen, T. Ge, Y. Xu, Z. Zhang, X. Yang, and K. Gai, "Semantic human matting," in Proc. ACM Multimedia Conference on Multimedia Conference, Seoul, Korea, 2018, pp. 618–626. [Online]. Available: http://delivery.acm.org/10.1145/3250000/3240610/p618-chen.pdf?ip=159.226.182.77&id=3240610


Y. Zhang, L. Gong, L. Fan, P. Ren, Q. Huang, H. Bao, et al., "A Late Fusion CNN for Digital Matting," in Proc. IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019, pp. 7469-7478. [Online]. Available: http://openaccess.thecvf.com/content_CVPR_2019/papers/Zhang_
    A_Late_Fusion_CNN_for_Digital_Matting_CVPR_2019_paper.pdf


J. Johnson, A. Alahi, and L. Fei-Fei, "Perceptual losses for real-time style transfer and super-resolution," in Proc. European Conference on Computer Vision, Amsterdam, Netherlands, 2016, pp. 694–711. [Online]. Available: https://link.springer.com/content/pdf/10.1007%2F978-3-319-46475-6_43.pdf



F. Chollet, "Xception: deep learning with depthwise separable convolutions," in Proc. IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017, pp.  1800–1807. [Online]. Available: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8099678


L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, "Encoder-decoder with atrous separable convolution for semantic image segmentation," in Proc. European Conference on Computer Vision, Munich, Germany, 2018, pp. 833–851. [Online]. Available: https://link.springer.com/content/pdf/10.1007%2F978-3-030-01234-2_49.pdf


O. Ronneberger, P. Fischer, and T. Brox, "U-Net: Convolutional Networks for Biomedical Image Segmentation," in Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, 2015, pp. 234–241. [Online]. Available: https://link.springer.com/content/pdf/10.1007%2F978-3-319-24574-4_28.pdf


B. Vijay, K. Alex, and C. Roberto, "SegNet: a deep convolutional encoder-decoder architecture for scene segmentation," IEEE transactions on pattern analysis and machine intelligence, vol. 39, issue 12, pp. 2481–2495, Dec. 1 2017. doi: 10.1109/TPAMI.2016.2644615.


L. He, G. Wang, and Z. Hu, "Learning depth from single images with deep neural network embedding focal length," IEEE Transactions on Image Processing, vol. 27, issue 9, pp. 4676–4689, Sept. 2018. doi: 10.1109/TIP.2018.2832296.


G. Chen, K. Han, and K.-Y. K. Wong, "TOM-Net: learning transparent object matting from a single image," in Proc. IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 9233–9241. [Online]. Available: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8579060


K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," in preprint arXiv, 2014. [Online]. Available: https://arxiv.org/abs/1409.1556


K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proc. IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp. 770–778. [Online]. Available: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7780459


P. Krähenbühl and V. Koltun, "Efficient inference in fully connected crfs with gaussian edge potentials," in Advances in neural information processing systems, Granada, Spain, 2011, pp. 109–117. [Online]. Available: http://papers.nips.cc/paper/4296-efficient-inference-in-fully-connected-crfs-with-gaussian-
    edge-potentials.pdf


S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet, Z. Su, D. Du et al., "Conditional random fields as recurrent neural networks," in Proc. IEEE International Conference on Computer Vision, Santiago, Chile, 2015, pp. 1949–1957. [Online]. Available: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7410536


L. C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, "DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs," IEEE transactions on pattern analysis and machine intelligence, vol. 40, issue 4, pp. 834–848, April 2018. doi: 10.1109/TPAMI.2017.2699184.


C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta et al., "Photo-realistic single image super-resolution using a generative adversarial network," in Proc. IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017, pp. 4681–4690. [Online]. Available: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8099502




