Light Field Image Coding Using Dual Discriminator Generative Adversarial Network and VVC Temporal Scalability
Nader Bakir Wassim Hamidouche Sid Fezza Khouloud Samrouth Olivier Déforges
eess.IV




𝐱
ŁL











Nader Bakir^∗†, Wassim Hamidouche^∗, Sid Ahmed Fezza^, Khouloud Samrouth^† and Olivier Déforges^∗



^∗Univ. Rennes, INSA Rennes, CNRS, IETR - UMR 6164, Rennes, France 

    ^†Lebanese University, Tripoli, Lebanon

		^National Institute of Telecommunications and ICT, Oran, Algeria

whamidou@insa-rennes.fr

Light Field Image Coding using Dual Discriminator Generative Adversarial Network and VVC Temporal Scalability
    [
    December 30, 2023
=============================================================================================================



Light field technology represents a viable path for providing a high-quality VR content. However, such an imaging system generates a high amount of data leading to an urgent need for LF image compression solution. In this paper, we propose an efficient LF image coding scheme based on view synthesis. Instead of transmitting all the LF views, only some of them are coded and transmitted, while the remaining views are dropped. The transmitted views are coded using Versatile Video Coding (VVC) and used as reference views to synthesize the missing views at decoder side. The dropped views are generated using the efficient dual discriminator GAN model. The selection of reference/dropped views is performed using a rate distortion optimization based on the VVC temporal scalability. Experimental results show that the proposed method provides high coding performance and overcomes the state-of-the-art LF image compression solutions.




Light Field, Deep Learning, D2GAN, VVC, Coding Structure, RDO.





§ INTRODUCTION


Light Field (LF) image can be captured and sampled by a plenoptic camera composed of an array of microlens such as Lytro and Raytrix cameras. LF imaging has many advantages over traditional 2D imaging systems, as its allows the user to change various camera settings after capture, thus providing more flexibility. Specifically, LF image captures both spatial and angular information enabling several multimedia services: multi-focus, multi-perspective, viewpoint rendering and even 6-Degree of Freedom (6-DoF) viewing <cit.>. A light field can be described by a 4 dimensional function with 2 parallel plans s,t and u,v denoted by L(u,   v,   s,  t).
There are several ways to represent a LF image, including micro-image, epipolar image and sub-aperture image <cit.>. The latter, illustrated in Fig. <ref>, is the the widely used representation. 

However, in sight of the huge amount of data involved by LF image, its processing, storage and transmission raise a real challenge that received increased research attention. In response, several solutions have been proposed to encode a LF image in sub-aperture representation. The straight forward coding approach organizes the LF views in a pseudo video sequence, which is then encoded with a classical 2D hybrid video encoder <cit.>. Another approach consists in encoding a spare set of views using a video encoder, while the rest of views are synthesized at the decoder side. The latter solution has been followed by several authors <cit.>, for instance, linear approximation has been investigated in <cit.> to estimate the views at the decoder from neighbour views, while a combination of linear approximation and  convolutional neural network has been proposed in <cit.> to synthesize missing views at the decoder side. In the same way, Jia et al. <cit.> proposed to use the generative adversarial network to generate unsampled views. To enhance the coding efficiency, the authors proposed to encode and transmit the residual error between the generated uncoded views and their original versions. Jiang et al. <cit.> proposed a coding method called Homography-based Low Rank Approximation. This method jointly optimizes multiple homographies that align the LF views and low rank approximation matrices. Hou et al. <cit.> proposed a method that exploits the inter- and intra-view correlations effectively by characterizing its particular geometrical structure using both learning and advanced video coding techniques.











In this paper, we propose an efficient approach to encode the LF images, which consists in encoding a sparse set of views, and estimate the rest of views at the decoder side. In particular, the first set of selected reference views are coded with the next generation video coding standard called Versatile Video Coding (VVC). While the second set of views are either synthesized from the first decoded set of views using a Dual Discriminator Generative Adversarial Network (D2GAN) or decoded by a VVC decoder. The D2GAN have been trained with a large set of LF images coded at different distortions. The architecture offered by the D2GAN, composed by a generator and two discriminators, enables better training and thus synthesizes views with high visual quality. In addition, to increase the coding efficiency, a rate distortion optimization (RDO) is adopted to select which views should be encoded and transmitted and which ones should be dropped and synthesized at the decoder side.   



The remainder of this paper is organized as follows. Section <ref> describes the concepts of D2GAN and VVC. Then, in Section <ref>, we describe the proposed LF image compression solution. Section <ref> presents and discusses the experimental results. Finally, Section <ref> concludes this paper. 




§ BACKGROUND


As mentioned in Section <ref>, the proposed coding approach is based on  Dual Discriminator Generative Adversarial Network (D2GAN) and Versatile Video Coding (VVC) standard. In this section, we briefly introduce these two concepts.


 §.§ Dual Discriminator Generative Adversarial Nets


Generative Adversarial Networks (GANs) are deep neural net architectures composed of two consecutive neural network models, namely generator G and discriminator D. GAN enables to simultaneously train the two models: the generative model G that captures the data distribution, and the discriminative model D that estimates the probability that a sample came from the training data rather than from the generator G <cit.>. GAN has recently achieved great success in various fields, especially in fake video generation, super-resolution and objects detection <cit.>.

Dual discriminator generative adversarial network (D2GAN), is a novel framework based on GAN, which uses two discriminators D_1 and D_2, where D_1 tries to assign high scores for real data, and D_2 tries to assign high scores for the fake data. This technique uses the two discriminators to minimize the Kullback-Leibler (KL) divergence and reverse KL between the generated image and the target image <cit.>.

Formally, D_1, D_2 and G now play the following three player minimax optimization game


    min_Gmax_D1,D2 (G,D1,D2) = α 𝔼_x ∽ P_data [logD_1(x)] 
        + 𝔼_z ∽ P_z [-D_1(G(z))] + 𝔼_x ∽ P_data [-D_2(x)] 
        + β 𝔼_z ∽ P_z [logD_2(G(z))],
 
where z is a noise vector, 𝔼 represents expected value, x is the real data, P represents the probability distribution, α and β are two hyper-parameters (0  < α, β≤ 1) to stabilize the learning of the model and control the effect of KL and reverse KL divergences on the optimization problem <cit.>.



More specifically, with a batch of M noise samples z^(1), z^(2), ..., z^(M) given as inputs, the generator generates M artificial samples, and this process is defined as G(z^(i)). While, x^(1), x^(2), ..., x^(M) represents a batch of M real data samples.

Three cost functions defined in (<ref>), (<ref>) and (<ref>) are computed to obtain the error that should be transmitted respectively to D_1, D_2 and G for their backward weights updating, as shown in Fig. <ref> (dash lines).


    ∇_θ_D11M∑_m=1^M  [ α logD_1(x^(m)) -D_1(G(z^(m)))],
 

    ∇_θ_D21M∑_m=1^M  [ β logD_2(G(z^(m))) -D_2(x^(m))],



    ∇_θ_G1M∑_m=1^M  [ β logD_2(G(z^(m))) -D_1(G(z^(m)))].
 

In this work, we use D2GAN to synthesize the dropped LF views, where the generator consists of two conventional neural networks (CNN) <cit.>. The first CNN estimates the disparity and the second one generates the color image.




 §.§ Versatile Video Coding


Based on High Efficiency Video Coding (HEVC), the Joint Video Exploration Team (JVET) is developing a new video coding standard called Versatile Video Coding (VVC) <cit.>. VVC already enables a bitrate saving of 35% to 40% with respect to HEVC for the same visual quality <cit.>. VVC introduces several new coding tools at different levels of the coding chain including frame partitioning, Intra/Inter predictions, transform, quantization, entropy coding and in-loop filters. For more details about the VVC coding tools the reader can refer to <cit.>. VVC supports by design the temporal scalability through the Random Access (RA) coding configuration. This latter, illustrated in Fig. <ref>, enables different temporal layers and each temporal layer uses as reference only frames from lower temporal resolution, i.e., lower layer. Therefore, frames of each temporal layer t_i can be removed without impacting the decoding of frames of lower temporal resolution t_j with t_i>t_j.

In the proposed coding approach, we exploit the concept of temporal resolution to drop views at the encoder without impacting the decoding process and thus performing the best rate distortion performance.        







§ PROPOSED LF IMAGE COMPRESSION METHOD


The idea behind the proposed coding method is, instead of transmitting all the LF views, to drop a sub-set of views at the encoder side and synthesize them at decoder side, thus considerably reducing the required bitrate for LF images. To efficiently achieve that, we exploit the temporal scalability of VVC and use the D2GAN model, all within a rate distortion optimization process (RDO).  

At the encoder side, first, LF sub-aperture views are organized into groups of 16 views that form Groups of Pictures (GOPs), as illustrated in Fig. <ref>. Next, in each GOP, the images of temporal levels 0, 1 and 2 are encoded using the VVC codec, which constitute the reference views used later in the synthesis process at the decoder side. Then, the images at the remaining levels 3 and 4 are either coded using the  VVC codec or dropped. In contrast to fix the number of dropped views, in our approach this is done adaptively on the basis of the proposed RDO process described in the Algorithm 1 and explaining in the following.

As illustrated in Fig. <ref>, we apply RDO process on the 3 consecutive frames, i.e.,  frame i at level 4, frame i+1 at level 3 and frame i+2 at level 4. It should be noted that if one of the views at temporal level 4 (frame i or i+2) must be encoded using VVC, then the frame i+1 at level 3 is also encoded using VVC, because this layer will be used as a reference for the frames at temporal level 4.


Main reasons behind only considering the 2 upper levels exclusively to the RDO block are, firstly, after an extensive study, we found that these levels occupy together around 28% of the total bitrate. Second, the views at the upper levels are not used as references in the VVC coding scheme.

Thus, we proposed a RDO block deciding which views from the upper level can be encoded using VVC or dropped and synthesized using D2GAN. To reach this goal, the encoder computes the rate distortion (RD) cost function J given by (<ref>) for both the VVC decoded view and the one synthesized by the D2GAN.

    𝒥 = D + λ  R

where λ is the Lagrangian multiplier, D is the distortion and R is the rate in bits per pixel (bpp). To set the Lagrangian multiplier (λ), we empirically determine its value by testing a large set of LF images. We found that the value of 0.1 for λ is optimal and for which the Lagrangian optimization is giving the best performance.



At the decoder side, the dropped views are synthesized using D2GAN block. As a reminder, the D2GAN  is composed of a generator G and two discriminators D_1 and D_2. G consists of two CNNs  <cit.>, the first CNN estimates the disparity and the second one generates the color image. A set of features (mean and standard deviation) of a sparse set of views (16 views) are fed to the disparity CNN that estimates the disparity at an intermediate view, and then used it to warp (backward) all the input views to the intermediate view. The second color CNN uses all the warped images, derived from the first CNN, to predict the color and synthesizes the dropped views.



Given that the generator G and discriminators (D1 and D2) are CNN-based blocks, a training phase is required to fix respectively their parameters θ_G, θ_D1 and θ_D2. Unlike GAN, in D2GAN, the scores returned by G are values in ℝ^+ rather than probabilities in [0,1]. The discriminators and generator are alternatively updated using stochastic gradient ascent and descent, respectively. The backward propagation of errors (i.e., cost functions) is applied to update the discriminators and generator with mini-batch size equal to M, as shown in Fig. <ref>.

For the training phase of D2GAN, 3 configurations were considered : 1) training with the original views , 2) training with reconstructed views at multiple distortion levels including the original views and 3) one training for each distortion level separately. We compared the three configurations, and the obtained results are given in Table <ref>. Based on these results, the third configuration, i.e., D2GAN reconstructed separately, outperforms the other configurations and hence we used it for the D2GAN training.






















































§ RESULTS AND DISCUSSIONS




 §.§ Experimental setup

The proposed deep learning-based architecture described in the previous section was trained with 140 LF images, where 70 LF images are from EPFL dataset <cit.>, 50 LF images are from Stanford Lytro LF image dataset <cit.> and 20 LF images are from HCI dataset <cit.>. Each sub-aperture view was split into patches of size 60×60, thus resulting in more than 150,000 patches that were used in the training phase. For the testing phase, 9 LF images are selected, 6  LF images are from EPFL dataset <cit.>, 1 LF image from Stanford Lytro LF dataset <cit.> and 2  LF images from HCI dataset <cit.>. Each of these LF images is composed of 8×8 sub-aperture views. These views are rearranged in a pseudo sequence using spiral order scan and coded using VVC in random access (RA) coding configuration at 4 QP values of 18, 24, 28 and 32.

The training configuration of D2GAN was set as follows: we trained the generator G and two discriminators (D_1 and D_2) with the ADAM optimizer by setting β_1 = 0.9, β_2= 0.999, learning rate = 0.0002, batch-size of 10 and kernel size of convolutional layers as depicted in Figure <ref>. The regularization coefficients of D_1 and D_2 were set as α= 0.2 and β= 0.2, respectively. For the generator, we used input patch of 60×60, stride of 16, and output patch equal to 36×36 (reduced size is due to the convolutions).




 §.§ Evaluations


We compared the proposed scheme with four state-of-the-art methods: 1) VVC-All that encodes all views with the VVC in RA coding configuration, 2) LF-GAN method proposed in <cit.>, where a sub-set of views are coded with HEVC, while the remaining views are generated by GAN and the residual error of views are transmitted to the decoder, 3) the method proposed in <cit.> encodes the views as a pseudo-video sequence using specific order scan, 4) the method of Hou el al. <cit.> that exploits the Inter- and Intra-views correlation to encode the views using HEVC. The latter method is considered as the anchor method.




 §.§ Results


The BD-BR <cit.> is a Peak Signal to Noise Ratio (PSNR) based metric. It is used in this paper to assess the gain of the proposed approach compared to the anchor solution. A negative BD-BR value refers to a bitrate reduction compared to the anchor method, while a positive value expresses a bitrate overhead.

R-D curves based on PSNR for the 9 LF images are provided in Fig. <ref>. We can notice that for all considered images, the proposed coding method provides the highest performance for all bitrates. The previous conclusion is confirmed by Table <ref>, providing the Bjøntegaard results of the four coding solutions compared to the anchor one <cit.>. The proposed method achieved an average BD-BR gain of -24.1% and BD-PSNR of 0.83 dB compared to the anchor method <cit.>.


The complexity of the proposed coding approach is also evaluated and compared to the other methods on both CPU and GPU platforms. The performance has been carried-out on an Intel core i9-7900X CPU running at 3.3GHz PC with 64 GB memory and a TITAN Xp NVDIA GPU. It is important to note that the GPU is only used when the D2GAN block is involved in the coding scheme. 

 Table <ref> gives the encoding and decoding run times in seconds. We can notice that the proposed solution requires almost the same complexity in the encoding for all QP compared to <cit.> and <cit.> methods. The GPU enables to speedup the encoding part related to the D2GAN block. However, the decoder of the proposed solution is more complex than the other solutions due the D2GAN block.


	
		

	
	
		
			

			
			
			
			
			
			
			
			
			
			
			
		
	



















































§ CONCLUSION


In this paper, we have proposed a view synthesis based LF image compression approach. In the proposed coding scheme, a set of views are encoded using VCC, while the remaining views are dropped. The dropped views are synthesized using enhanced GAN-based approach known as D2GAN. The transmitted and dropped views are selected using RDO process. In addition, in order to avoid impacting the decoder with the dropped views, the latter are determined according to the temporal scalability of VCC. All these features allow reducing bitrate required by LF image, while providing views with high visual quality.

The experimental results showed the efficiency of our scheme, which achieved bitrate reduction of -24.1% in terms of BD-BR and increased the visual quality by 0.83 dB in BD-PSNR with respect to the state-of-the-art solution.  






IEEEbib

