Image compression optimized for 3D reconstruction by utilizing deep neural networks
Alex Golts Yoav Y. Schechner
eess.IV cs.CV cs.LG


Alex Golts et al.









1]Alex Goltscor1
[cor1]Corresponding author:
alex@golts.net


2]Yoav Y. Schechner


[1]Rafael Advanced Defense Systems LTD., Israel, e-mail: alex@golts.net
[2]Viterbi Faculty of Electrical Engineering, Technion - Israel Institute of Technology, Haifa, Israel. e-mail: yoav@ee.technion.ac.il










Computer vision tasks are often expected to be executed on compressed images. Classical image compression standards like JPEG 2000 are widely used. However, they do not account for the specific end-task at hand. Motivated by works on recurrent neural network (RNN)-based image compression and three-dimensional (3D) reconstruction, we propose unified network architectures to solve both tasks jointly. These joint models provide image compression tailored for the specific task of 3D reconstruction. Images compressed by our proposed models, yield 3D reconstruction performance superior as compared to using JPEG 2000 compression. Our models significantly extend the range of compression rates for which 3D reconstruction is possible. We also show that this can be done highly efficiently at almost no additional cost to obtain compression on top of the computation already required for performing the 3D reconstruction task.










Image compression3D reconstructionDeep learningRecurrent neural networks









§ INTRODUCTION


Image compression is an essential step in many image processing and computer vision pipelines. At times, the sole goal of compression is to deliver images that a human viewer would perceive as having high quality given some compression ratio.
However, computer vision systems are often autonomous. This means that evaluation criteria and compression goals should be 
tailored to computer vision tasks. This paper focuses on the computer vision task of multi-view three-dimensional (3D) reconstruction. Compression is thus jointly optimized with 3D reconstruction.

Compression in this context is important when multi-view data has to be transmitted using limited resources. Specifically, this applies to imaging using drones (<cit.>), airplanes (<cit.>) and satellites (<cit.>), where power and connectivity are limited. For 3D reconstruction, computation is applied on the transmitted images to extract 3D information in the form of volumetric occupancy grids (<cit.>, <cit.>), point clouds, depth maps or surface mesh models (<cit.>). The images are typically compressed  (<cit.>), however, the compression methods that are mostly used are well known standards tailored to image quality metrics, and not directly to evaluation metrics of 3D reconstruction.     

To solve our 
problem we use a deep neural network (DNN). DNN-based methods are increasingly successful and popular at solving various image processing and computer vision tasks.
In particular, <cit.> proposed a recurrent neural network (RNN)-based method for image compression that outperforms the well-known JPEG standard <cit.>.

More recently, several non-RNN based compression methods were proposed. They outperformed <cit.> achieving compression performance competitive <cit.>, or better than (<cit.>, <cit.>) the JPEG 2000 standard <cit.>. Some <cit.> even exceeding the performance of the more recent BPG codec <cit.>. Neural networks are a natural choice for image compression, as they often compress their input signal, even when compression is not their explicit goal.
For the task of stereoscopic 3D reconstruction, <cit.>  proposed an RNN architecture that learns a mapping from images of objects at different viewpoints, to a 3D occupancy grid corresponding to the object's shape. 

More recently, <cit.> improved 3D reconstruction performance while using a more computationally efficient non RNN-based approach.

When applying image compression to multiple view scenarios where a large overlap exists between adjacent images, some works (<cit.>) leverage the redundancy in the overlapped regions to compress better.  

We postulate that jointly learning these two tasks (image compression and 3D reconstruction) may lead to compression better suited for 3D reconstruction. 
<cit.> showed that learning multiple visual tasks jointly requires less labeled data to achieve the performance obtained by separate learning-based systems. However, this was shown for small auxiliary sub-tasks that aid in a more complex grand task. In our case, both compression and 3D recovery are critical components. Moreover, one can hardly expect compression to actually help the mission. It is a necessity. As compared to stand-alone compression, it can be expected that tailored compression enhances the mission performance, for a given compression rate. 

In the context of image compression, <cit.> showed that image understanding tasks such as classification and semantic segmentation can be performed directly on compressed representations derived by DNNs . This alleviates the need to decompress files prior to image understanding. They also further show that by jointly training image compression and classification networks, synergies are obtained leading to performance gains in both tasks.

In this work we propose methods to compress images so that they can be optimally used for 3D reconstruction. Our method can work with only negligible computations for image compression, on top of a system for 3D reconstruction based on <cit.>. It also exceeds the 3D reconstruction performance obtained from images compressed by JPEG-2000 <cit.> or learned compression, across a wide range of medium to ultra aggressive compression rates that we focused on. 
Our main focus in this work is on the regime of high compression rates. We find that lower compression rates may be better suited when the goal is to obtain visually satisfying decompressed images. When the ultimate task is different, it is possible to compress images further. We study these limits for the task of 3D reconstruction.


Our approach uses RNN-based basic components for image compression <cit.> and 3D reconstruction <cit.>. It does not aim to propose the most current state-of-the-art for these separate tasks. Rather, we wish to illustrate a generic concept involving the multi-task learning of these tasks. We demonstrate the benefits in considering such a framework, while assuming the above RNN-based building blocks. We hope that our findings provides motivation for further research around this subject, which may also be tailored for using more current basic methods.



§ BACKGROUND


Sec. <ref> provides basic background on RNNs and LSTMs. The reader familiar with these topics may skip to Secs. <ref>, <ref>, which provide background on specific works in RNN-based image compression and 3D reconstruction that we build on.


 §.§ Recurrent neural networks




  §.§.§ Motivation

A feedforward neural network's forward pass accepts a single fixed size input vector 𝐱. The network applies a fixed amount of computations, depending on the network depth and architecture. The network then outputs a single fixed size output vector 𝐲. A common example is image classification, where, 𝐱 is an image and 𝐲 is a vector of probabilities that the imaged object belongs to a class. 
Consider for a moment 𝐱= {𝐱_t} to be a movie sequence, where t denotes a time step. The task is to classify events over time (<cit.>). Proper understanding of a moment in the movie should rely on understanding of previous frames. Feedforward neural networks cannot reason about previous events to inform later ones. RNNs address this issue.

Fig. <ref> shows a block diagram of an RNN in its cycled (left), and equivalent unrolled form (right). 

Here f_𝐰 is a neural network block with parameters w, which is a recurrence formula applied at every time step t on the current input 𝐱_t. A loop shown in Fig. <ref> (left) allows information to be passed from one time step of the network to the next. The unrolled form (right) reveals that an RNN can be thought of as simply multiple copies of the same network layer, each passing a message to a successor.
The recurrence formula f_𝐰 in its general form is 

    {𝐡_t, 𝐲_t} =f_𝐰(𝐡_t-1,𝐱_t)

Eq. (<ref>) operates on the current input 𝐱_t and the RNN's hidden state 𝐡_t-1 from the previous time step. The outputs of  Eq. (<ref>) are 𝐡_t and 𝐲_t, the RNN's hidden state and output in the current time step, respectively.
Contrary to feedforward neural networks, RNNs commonly operate over sequences of vectors, rather than an individual vector (<cit.>). 



  §.§.§ Sequential processing of individual vectors

It is sometimes beneficial to apply RNNs even when both the input and output are individual, fixed size vectors (See <cit.>).


For example, in image compression, (Sec. <ref>), the input is a fixed size single image, and the output is either a binary compressed code, or a fixed size image. 

An output compressed code is often of a variable length, depending on the desired compression rate.
While such compression can be performed using a feedforward neural network, it can benefit from the sequential processing power and memory capability of RNNs (<cit.>, <cit.>). 





  §.§.§ Vanilla RNNs

In a standard (vanilla) RNN, the block f_w performs a simple recurring operation

    𝐡_t = tanh(𝐖_h𝐡_t-1+𝐖_x𝐱_t),    𝐲_t = 𝐖_y𝐡_t.

Here the tanh activation function is applied elementwise, and W_h, W_x and W_y are learned parameters (weights) of the RNN. 
Multiple such RNN layers can be concatenated to form deep RNNs.
In practice, such standard RNNs suffer from difficulty in learning long-term dependencies throughout a sequence (<cit.>).
Therefore, a special kind of RNN called Long-Short Term Memory (LSTM) network, was proposed by <cit.>, and adopted widely and successfully.


  §.§.§ LSTM networks
     
LSTMs are explicitly designed to avoid the long-term dependency problem.
When f_w is an LSTM, it has a more complicated structure compared to the vanilla RNN. Its block diagram is shown in Fig. <ref>.

Mathematically, the LSTM recurrence formulae are given by a set of equations as follows. In these equations, ⊙ denotes element-wise multiplication, σ denotes the Sigmoid function σ(u) = [ 1+e^-u]^-1, and 𝐛_f, 𝐛_i, 𝐛_o and 𝐛_c are learned bias vectors.

There are three basic gates involved in LSTM computation: forget, input and output gates, which are respectively:

    𝐟_t = σ(𝐖_fx𝐱_t+𝐖_fh𝐡_t-1+𝐛_f),


    𝐢_t = σ(𝐖_ix𝐱_t+𝐖_ih𝐡_t-1+𝐛_i),


    𝐨_t = σ(𝐖_ox𝐱_t+𝐖_oh𝐡_t-1+𝐛_o).

Another gate, which usually does not have an explicit name in the literature, is

    𝐠_t = tanh(𝐖_cx𝐱_t+𝐖_ch𝐡_t-1+𝐛_c),

The cell state of the LSTM, is defined by 

    𝐜_t = 𝐟_t⊙𝐜_t-1 + 𝐢_t⊙𝐠_t,

The hidden state 𝐡_t depends on the cell state through

    𝐡_t = 𝐨_t⊙tanh( 𝐜_t),

In the case of an LSTM, the hidden state is also the final RNN output (Eqs. (<ref>-<ref>))

    𝐲_t≡𝐡_t.

Intuitively, and following Eq. (<ref>), the forget gate regulates whether to forget the previous cell state 𝐜_t-1 in calculating the current cell state 𝐜_t. The input gate regulates whether to write the new input 𝐱_t and hidden state 𝐡_t to the current cell state. The gate 𝐠_t weights how much of these new vectors to write to the cell state. Finally, the output gate regulates how much to reveal the current cell state for updating the hidden state.


This formulation shows that an LSTM unit explicitly controls the flow from input to output. The gates regulate how much and which information from the previous time steps and current input, proceeds to the next time step. 
It can be shown (<cit.>, <cit.>) that in an LSTM, unlike a vanilla RNN, the gradient of the cell state with respect to itself at some previous time step, does not decay exponentially in the time step difference. This is what allows the LSTM to mitigate the vanishing or exploding gradient issue, which exists in vanilla RNNs and may prevent learning of long term dependencies.  

Multiple slight alternative formulations of the LSTM equations have been proposed. A more conceptual alternative is the Convolutional LSTM proposed by <cit.>, in which the matrix multiplications in Eqs. (<ref>-<ref>) are replaced with convolutional filtering, in analogy to convolutional neural networks (CNNs). This is especially useful for image-related tasks.  


 §.§ RNN-based image compression


We now briefly discuss an RNN-based method for image compression proposed by <cit.>, which we use in our joint compression and 3D reconstruction framework.
The method is a single model architecture capable of producing variable rate compression (see Fig. <ref>).

The encoder E^comp and decoder D^comp are RNN-based, therefore the processing occurs sequentially over time t.  
A single iteration is shown in Fig. <ref>.
In the first iteration, the original image x is encoded by E^comp to a vector of length m in floating point representation, per element. Then, a binarizer module B converts the encoded representation into a binary vector. This binary vector is part of the compressed representation. In each further iteration, a new binary vector of length m is formed and added to the full compressed representation. This way, the number of iterations controls the compression rate. 

Decoding applies D^comp on the binary representation b_t and outputs either (γ=0) a full reconstructed image x̂_t or (γ=1) a residual from the previous reconstruction x̂_t - x̂_t-1. At subsequent iterations, the input to the encoder becomes the residual image 𝐫_t=𝐱-𝐱_t, which is the difference between the original and reconstructed image. During training, the absolute value of this residual is minimized.
The process is defined formally as follows

    𝐱̂_t=D_t^comp(𝐛_t)+γ𝐱̂_t-1,    𝐛_t=B[ E_t^comp(𝐫_t-1)] ,


    𝐫_t=𝐱-𝐱̂_t,    𝐫_0 = 𝐱,     𝐱̂_0=0

Here E_t^comp and D_t^comp represent the RNN-based encoder and decoder with their states at iteration t, respectively; 𝐱 is an original image of size H× W × 3 and 𝐱̂_t is its progressive reconstruction, with γ=0 for “one-shot" reconstruction[Full image reconstruction is computed at each iteration], and γ=1 for additive reconstruction[Only the residual is computed at each iteration. The final reconstructed image is a sum of the outputs of all iterations.]; 𝐫_t is the residual between 𝐱 and the reconstruction 𝐱̂_t; and 𝐛_t∈{ -1,1 }^m is an encoded bit stream produced by a binarizer function B,  where m is the number of bits produced per iteration. It depends on H, W and M, the number of output channels from the final layer of E^comp through m=(H/16)×(W/16) × M [M=32 in <cit.>.]. The number of RNN iterations N controls the overall compression ratio. A compression rate c is defined as the ratio of the number of bits in the raw image to that of the binarized representation. 

In our case we have 8 bit-per-pixel RGB images. Thus, in order to obtain a given compression rate c, the number of RNN iterations needs to be set to

    N = H × W × 3 × 8/c× m.

The encoder consists of a convolutional layer followed by three convolutional RNN layers, i.e. convolutional LSTMs. 

<cit.> evaluated additional options for the recurrence unit, besides LSTM.
The binarizer in <cit.> consists of a convolutional layer, followed by a binarizing operation, such as the one used in <cit.>. In <cit.>, further lossless compression is achieved using entropy coding.

During training, the binarizer in  [<cit.>,  <cit.>] is a stochastic variable. 
Assuming the input x is the output of a tanh layer, the binarizer is defined as






    b(x)= 
    
    1    with probability  1+x/2
    
    -1    with probability  1-x/2.

For back-propagation, the derivative of the expectation is taken. Since, 𝔼 [ b(x)  ]=x   ∀ x∈ [ -1,1  ], the gradients are passed through b(x) unchanged. 

The decoder in <cit.> starts with a convolutional layer, followed by four convolutional RNN layers, with each such layer followed by a depth-to-space[The depth-to-space or Pixel Shuffle layer rearranges elements of a tensor of shape (*,C × k^2,H,W) to a tensor of shape (*,C, H× k,W× k). This, followed by a convolution layer is useful for upsampling sub-pixel convolution with a stride of 1/k ] layer (<cit.>). Each such layer decreases the depth size by a factor of 4, thus increasing spatial resolution by a factor of 2 in both row and column dimensions. Finally, another convolutional layer is applied to produce a H × W ×3 reconstructed image. 

During training, a weighted L_1 loss on the residual 𝐫_t is minimized

    L_comp=β∑_ijt=0^H W N | r_ijt |

where for minibatch size B, β = ( B × H × W × 3 × N ) ^-1 . The sum is over the image spatial dimensions i,j and RNN iterations t.



 §.§ RNN-based 3D reconstruction


We now discuss the second important component for our joint framework for compression designed for 3D reconstruction, namely the 3D reconstruction method of <cit.>. They proposed a network architecture whose input is a single or multiple images of an object, from arbitrary viewpoints. The network output is a reconstruction of the object in the form of a 3D occupancy grid (see Fig. <ref>).

Here, {𝐱_i}_i=1^V are images of an object from V different viewpoints, and 𝐩_V is the 3D reconstructed occupancy grid produced after V viewpoint. It is represented as Softmax probabilities indicating occupancy of each point in the 3D grid. 

The encoder E^3D and decoder D^3D are feedforward CNNs. Two architecture variants are proposed in <cit.>: a shallow and a deeper one with residual blocks (<cit.>).
Both encoder variants consist of convolutional layers followed by LeakyReLU nonlinearities, and six MaxPooling operations, each with a stride of 2, for a total encoder stride of 64. Finally, a fully-connected layer is applied, which returns a vector of size K, an embedding of each viewpoint image. 

The 3D convolutional LSTM module is a core component proposed in <cit.>. It allows the network to retain details it has observed in previous views and update the memory when it receives a new image.
The module consists of a grid of 4 × 4 × 4 3D Convolutional LSTM units, each with a hidden state of size N_h. These are different from the convolutional LSTM units in Sec. <ref> in two regards: First, the input of the LSTM module is a vector which undergoes multiplication as in a standard LSTM definition. Only the hidden state undergoes convolution. Second, with the hidden state being three-dimensional, the convolution operation is now in 3D. Each such unit is responsible for reconstructing a particular part of the 3D voxel space.
In <cit.> the choices for K and N_h are 1024 and 128, respectively.

During training, a 3D voxel-wise Softmax loss over the final viewpoint's output 𝐩_V, is minimized

    L_3D= ∑𝐩̃log𝐩_V+ ( 1-𝐩̃ )log ( 1-𝐩_V )

Here, 𝐩̃∈{ 0,1 } is the 3D ground-truth occupancy, and the sum is over the three voxel dimensions (indices 
omitted for simplicity).


The 3D Convolutional LSTM enables the network to be invariant to the order of viewpoints that are fed to it. Indeed, the V viewpoints consisting each input sequence are selected at random, and there is no single preferred order. 


§ JOINT COMPRESSION AND 3D RECONSTRUCTION


The motivation for joining the tasks of compression and 3D reconstruction into a unified framework can be twofold. A primary motivation is to obtain compressed representations better suited for 3D reconstruction, and thus obtain improved 3D reconstruction performance, as compared to when using known compression standards.
Another motivation can be in reducing 
overall computational cost by providing a unified network architecture that would be more efficient than applying sequentially the compression and 3D reconstruction architectures shown in Figs. <ref>, <ref>.

We now discuss different unified models that achieve the above needs.
In all of the following proposed network architectures, the loss is optimized with respect to all model parameters jointly. No sub-model elements are pre-trained, and no sequential training protocols take place, such as where one sub-model is trained first, then freezed, while another sub-model is trained. The Binarizer in all of the following proposed architectures has no learned parameters, but a gradient is passed through it as explained in Sec. <ref>. 



 §.§ 3D reconstruction from decoded images (Sequential model)


Here we propose a sequential approach, as shown in Fig. <ref>. The compression model (Fig. <ref>) and 3D reconstruction model (Fig. <ref>) are simply concatenated. The 3D reconstruction part of the network receives as input a decompressed image.
During training, we optimize a loss which depends on viewpoint i as follows

    L_total= 
    
    L_comp    i<V
    
    L_comp + L_3D    i=V
     .

For i<V we use D^comp to obtain a decompressed image 𝐱̂_i, and the compression loss L_comp is applied and attempts to make 𝐱̂_i similar to the original image 𝐱_i.  In the last viewpoint V, this is done as well, but now, additionally the 3D reconstruction 𝐩_V is calculated, and the corresponding loss L_3D is also added to the optimization.

For simplicity, we refer to this model from now on as the sequential model.


 §.§ 3D reconstruction from compressed codes (Direct model)


We also propose a more computationally efficient approach shown in Fig. <ref>. Here, the output codes following E^comp and B are used directly to feed the 3D LSTM module. Thus, applying E^3D is not required, and some computation is reduced.
Here we also optimize the loss from Eq. (<ref>). 
 
For simplicity, we refer to this model from now on as the direct model.


 §.§ 3D reconstruction with implicit compression (Implicit model)


Finally, we propose another approach that is significantly more efficient computationally than those in Secs. <ref>-<ref>. 
The idea here is to use the RNN-based 3D reconstruction architecture discussed in Sec. <ref>, and augment it with a binarizer B so that compression is obtained implicitly, without explicitly minimizing an image compression loss L_comp. The loss we minimize here is simply L_3D. This makes sense in a scenario where we wish to solve the 3D reconstruction task by supplying compressed image codes. We do not require that our model reconstruct viewable images. Only that it successfully solves the task at hand, which is to reconstruct a 3D occupancy map. Since we focus on the 3D reconstruction task, we can minimize L_3D only. 

The binarizer module B can be thought of as a form of regularization in training the network on the 3D reconstruction task. It imposes a constraint on the minimization of L_3D, which is to yield a good 3D reconstruction while requiring the encoded representation to take binary form.
This architecture does not use the variable compression rate RNN framework of <cit.>, where multiple RNN iterations controlled the compression rate. Rather, we train a separate slightly modified model for every desired compression rate. The compression rate is now controlled by K, the output vector length of the CNN encoder ℰ_K^3D.

The proposed architecture is shown in Fig. <ref>. 
Here, the encoder ℰ_K^3D differs from E^3D in that the default choice of K, the encoded vector length is now varied with every compression rate. A separate model is trained for different choices of K obtained by modifying the number of channels in the output of the final convolutional layer of the encoder. Here K is also the number of bits of the compressed representation (after the binarizer).

 
For simplicity, we refer to this model from now on as the implicit model. 


Note that the direct model from Sec. <ref> can be seen as a possible extension of the implicit model, in case decompressed viewable images are required. It is similar in structure, except the encoder is ℰ^comp instead of ℰ_K^3D, a decoder is added to produce the decompressed images, and decompression is done recurrently, allowing variable rate compression in a single architecture. 


 §.§ Implementation details


Our unified models were trained using a minibatch size of 6 so that training could be done on a single GTX 1080 Ti GPU. 

In Tab. <ref> we compare our reimplementation of  <cit.> to the original work. Note that the model evaluated here is for 3D reconstruction only, without image compression involved. The evaluation criterion is mean Intersection-over-Union (mIoU) between a 3D voxel reconstruction 𝐩 and its ground truth voxelized model 𝐩̃, over the test set. The IoU criterion is defined [<cit.>] as

    IoU≡1(𝐩 > τ) ∩1(𝐩̃)/1(𝐩 > τ) ∪1(𝐩̃) = ∑[1(𝐩 > τ)1(𝐩̃)]/∑[1(1( 𝐩 > τ) + 1(𝐩̃))] .

Here, τ is a threshold we set to 0.4 as in <cit.>, ∩ and ∪ denote the intersection and union operations, respectively, 1 denotes the Indicator function, and the sums are over the three voxel dimensions.
Out of the different architecture variations that  <cit.> experimented with, we chose one of the simplest, termed 3D-LSTM-3. It consists of a network of moderate depth and uses an LSTM recurrence unit with a 3x3x3 convolution kernel.
<cit.> trained the network for roughly 60 epochs. We see that we can train for only 20 epochs and still obtain good performance, only 3% lower than the optimum. Therefore, for practical reasons, we settle on training all of our models for 20 epochs, from here onward.
We also see that our implementation results in somewhat improved performance compared to the original implementation of  <cit.>. This performance gain was not found to result from the few deviations from  <cit.> that are known to us, i.e, different minibatch size. We discuss the architecture of the compression part in our sequential and direct models (Secs. <ref>-<ref>) in the appendix.


In <cit.>, the compression model was trained on small 32x32 image patches. In our sequential and direct proposed approaches (Secs. <ref>-<ref>), we want to build upon the architecture of  <cit.> to train a unified model for both compression and 3D reconstruction. Therefore, we want the input to our models to be full resolution 128x128 images from the ShapeNet dataset used by <cit.>. Compared to  <cit.> and  <cit.>, we wish to train larger models on larger inputs. We also wish to be able to do this on a single GPU. To handle this challenge, reducing the minibatch size to 6, as mentioned in Sec. <ref> was not enough. We needed to further optimize the compression network architecture. We provide elaborate discussion of our choice of compression network architecture in the appendix.

In this work we focus on the ShapeNet dataset of 128x128 rendered images. In different scenarios one may wish to apply our methods on much larger images. Our architectures are fully convolutional, therefore they can be directly be scaled to larger inputs with linear increase in memory and computation. However, the spatial receptive field would not grow unless further architectural changes are made, such as increasing the depth of the network, or using dilated convolutions (<cit.>).



§ RESULTS




 §.§ 3D reconstruction


In Fig. <ref>, we report our models' mean Intersection-over-Union (mIoU) scores on the ShapeNet test set, using 5 viewpoints for the 3D reconstruction, as in  <cit.>. Tab. <ref> shows a visual comparison of 3D reconstructed occupancy grids produced for 5 random viewpoints of the objects shown in Tabs. <ref>, <ref>.


 

We see that for aggressive compression rates of 128 and above, the sequential model is superior to a 3D reconstruction network trained on images compressed by JPEG 2000 using a random compression rate at each minibatch. Moreover, for an extremely aggressive compression rate of 384, JPEG 2000 typically compresses images to the extent that no details are seen at all. The mIoU of the network trained on JPEG 2000 compressed images at this point is 0.1, which is meaningless. In our sequential and direct models, this compression ratio is obtained by using only one RNN timestep. It still produces images that contribute to 3D reconstruction with mIoU>0.5. We also see that the best results are obtained using the most efficient implicit model. 


 §.§ Statistical significance


The 3D reconstuction IOU metric varies significantly for different test examples, even without compression. This is due to the natural variability between objects. Different objects can be easier or more challenging for accurate 3D reconstruction than others. For the baseline 3D reconstruction model of  <cit.>, the STD of the IOU across the test examples is roughly 0.2 (see Tab. <ref>). We ask ourselves then, how significant are some of the trends depicted in Fig. <ref>? To obtain a better insight for this, we separate the test examples into bins of roughly equal IOU STD of 0.04 on the baseline model. For this value of STD, we obtain 7 such bins which contain (in increasing order of mean IOU) 17, 82, 131, 273, 340, 384 and 233 examples. To illustrate this, we show the mIOU results for bin #5 with error bars, in Fig. <ref>. Here, we see more clearly that the performance trends and differences between models, are of statistical significance.
In the two lowest mIOU data bins, the mIOU for all the different models and most compression rates is below 0.2. For such low IOU examples, the evidence for statistical significance is lacking. It may also be explained by the relatively small number of examples that are contained in these low IOU bins. 
  


 §.§ Lower compression rates


Note that the original network of <cit.> naturally obtains an image compression ratio of 1:12, without any use of a binarizer (the magenta square in Fig. <ref>) or otherwise special design. It can be thought of as a special case of the implicit model, with the binarizer module disabled, and set to K=1024 in ℰ_K^3D. Thus, the magenta square is plotted on top of the implicit model curve.

This work focused on high, limiting compression rates. Still, we wanted to provide a more complete overview of our methods. Therefore, we additionally evaluated all our models for two lower compression rates of 1:12 and 1:6. We can say roughly that all models tend to approach the “ideal" case of the magenta square at these low compression rates. 

If we used our sequential, direct and implicit architectures with binarizer, obtaining such low compression rates would become a memory burden. Therefore, we disabled the binarizer and used a 32 bit floating point encoded vector. We set N the number of RNN iterations, or the number of channels in the last encoder layer K (for the implicit architecture) appropriately in order to achieve these rates. 
For the network trained on JPEG 2000 compressed images, we re-trained it separately for 1:12 and 1:6 compression rates. 


 §.§ Run time


Tab. <ref> shows time of execution for different proposed models and building blocks on a single GTX 1080 Ti GPU, during both train (forward and backward pass) and test time (forward pass only)The backward pass time of ℰ^3𝒟 is approximated. It was not measured directly since it was done through the whole 3D reconstruction module with a single PyTorch .backward() call. It is reasonable to assume that it is identical to the forward pass, as this was the case when measured on the whole 3D reconstruction module.
The timing of the joint 3D reconstruction with implicit compression model (Sec. <ref>) is approximated as identical to just the 3D reconstruction model of <cit.>, since it only adds a single binarizer element on top of it, that is negligible in computational complexity compared to the whole model in both forward and backward passes..
A typical scenario of 5 viewpoints and 4 RNN iterations was assumed when timing 3D reconstruction and compression modules, respectively. 
We see that the using 𝒩^𝓈𝓂𝒶𝓁𝓁 as the compression module in our sequential unified model results in a speed-up of 15% in the forward pass compared to using 𝒩^ℴ𝓇𝒾ℊ𝒾𝓃𝒶𝓁. Using the more efficient direct model further improves the relative speed-up percentage to 26%. 


This improvement is moderate, implying that the most of the computation lies within the recurrent compression module. One way to increase the speed-up could be in eliminating the variable compression rate constraint and train a model per compression rate. This is done in the implicit model, however without reconstructing viewable decompressed images.

The most significant speed-up is not surprisingly obtained by the implicit model which only optimizes L_3D, and simply augments the 3D reconstruction model by <cit.> with a binarizer module B (Sec. <ref>). Here, the relative speed-up percentage in forward pass reaches 75%. Note that we trained all models on the same data and number of epochs, so these relative speed-ups also indicate the difference between whole training procedures required to obtain a deployable model.  





 §.§ Compressed image decoding


Although our goal is 3D reconstruction from compressed image representations, it is useful to also be able to decompress into visually satisfying 2D images. Our sequential and direct models allow this, since they were also optimized for the 2D image decoding task. Here, we evaluate their performance on this task.
Fig. <ref> shows the compression loss magnitude that our models achieve on the ShapeNet test set, and Tab. <ref> shows a visual comparison.
We see that the compressed image decoding performance of the sequential model is virtually indistinguishable from that of 𝒩^𝓈𝓂𝒶𝓁𝓁, a model trained on the image compression task only (See appendix). In contrast, the more computationally efficient direct model is somewhat inferior.
This could be explained by a harder constraint that this model imposes on the compression codes, requiring them to be suitable for directly feeding the 3D reconstruction model, in addition to allowing for a good compressed image decoding.



 


In order to gain more qualitative insight on the decompression capability, we show in Tab. <ref> another visual example, with four other images which were manually selected to contain richer texture content. Here we also show JPEG-2000 decompression results for comparison.
We can see that for high compression rates, the neural network based compression is superior to JPEG-2000, however in the 48:1 rate, JPEG-2000 manages to keep details better, particularly in areas of rich, high spatial frequency texture. This may show that such image details were not essential to learn good 3D reconstruction, in our case where the 3D models ground truth is relatively low resolution (32x32x32 voxel grids).
 


§ DISCUSSION





The main conclusions from the paper can be summarized as follows:



0em
  
  * Three neural network architectures for joint image compression and 3D reconstruction were proposed which trade-off 3D reconstruction quality, image decompression, and run time.
  
  * Near optimal 3D reconstruction is obtained for a wide range of compression rates, including aggressive rates where JPEG-2000 fails completely.
  
  * The implicit architecture reaches the most aggressive compression rates without performance loss, while also being the fastest in run time.



Let us now discuss in more detail. The sequential and direct model architectures include an RNN-based module dedicated for image compression and allows for compressed image decoding. These architectures are trained to optimize two loss functions 
simultaneously, one for 2D image compression, and another for 3D reconstruction. As such, they imply significant additional computations (both in train and test time) compared to the 3D reconstruction module alone. The implicit model architecture is both more computationally efficient and accurate. It is trained only by minimizing the loss function associated with the 3D reconstruction task. The compression is obtained implicitly via a binarizer module which is inserted as part of the 3D reconstruction network architecture. The computational overhead that this variant imposes on top of an existing 3D reconstruction module alone, is negligible.


In future work, techniques for speeding up training time of current models, especially the sequential and direct, could be explored. Training speed-up could be obtained by not training from scratch as we did, and instead using pre-trained weights. Training speed-up could also be obtained by using more efficient base models for compression and 3D reconstruction. Finally, existing numerical techniques such as mixed precision training could be applied.

We showed that all our proposed architectures allow for reasonable 3D reconstruction from images compressed aggressively at a 
compression ratio of 384:1, where JPEG-2000 compression is impractical. Two of our more accurate architectures also outperform RNN-based 3D reconstruction trained on JPEG-2000 compressed images at less 
aggressive compression rates. Our best performing implicit architecture outperforms RNN-based 3D reconstruction trained on JPEG-2000 compressed images for almost the whole range of compression rates, while also reaching acceptable 3D reconstruction performance under even more aggressive compression rates than all other examined variants (up to 480:1). 

The implicit model yields relatively constant and nearly “ideal", non-compressed performance throughout a wide range of compression rates. This motivated us to try apply it using even more aggressive compression rates then the previously chosen 384. And indeed we see that reasonable results are still obtained for up to a compression rate of 480, after which they deteriorate quickly.


The implicit model is not optimized for acceptable image recovery. It only considers 3D reconstruction. This has limitations, in that such a model could only be used in specific applications, where a human observer is not present, or is not interested in the images, but only in the final 3D reconstructed occupancy grids. However, this also means that this model solves a much simpler problem. It does not need to account for two constraints simultaneously. Thus, for suitable applications, it is highly beneficial. It achieves superior 3D reconstruction capability over a very broad range of compression rates. And it does so using much less computations. 

The somewhat noisy, non-monotonic trend of the mIoU vs. compression rate of the implicit model may perhaps be explained by that in contrast to the sequential and direct models, here, a separate model is trained for every compression rate, so it is reasonable that there can be some degree of inconsistency. 

Another reason could be related to our choice of varying the compression rate by modifying the number of output channels from the last layer of ℰ^3D_K. This choice may not necessarily be the best one.
One could modify the architecture in different ways to control the compression rate, and it may be that restricting the modification to just a single layer is somewhat sub-optimal and may result in some overfitting if the number of neurons in a single layer is too large. This could explain the slightly lower performance counter-intuitively obtained for better compression rates. 


Some techniques can be suggested in order to obtain a more smooth performance curve across compression rates.
Increasing regularization through weight decay, dropout or other known techniques, could lead to more stability. Additionally, it's possible to use a common backbone architecture with identical weights across all compression rates, while only retraining the last convolutional layer of the encoder, for each rate. This could also reduce variance across compression rates. 

We also note that for low compression rates, not using the binarizer and setting K accordingly is one possible design choice, and not necessarily the most optimal one. We could, for instance, use a quantizer that outputs 4, 8 or 16 bits, together with a different choice of K. Or modify the encoder architecture in another way. We leave the focus on moderate to low compression rates for future work.

We find it interesting that 3D reconstruction based on neural networks can be made highly robust to image compression, with only slight performance degradation for an extremely wide range of compression rates. We believe that it may be worth exploring neural networks' robustness to significant image compression for other computer vision tasks as well.


Another promising direction for future work may be to further improve the achievable compression rates, by considering the overlap between adjacent image views in the compression module, instead of compressing the images separately before feeding the compressed representations to the 3D reconstruction module.


We note again that our approach is meant to propose a generic concept which offers benefit in jointly framing the tasks of image compression and 3D reconstruction. Both these separate tasks are well researched and the state-of-the-art in each of them has been improved in recent years over the methods we chose to use in our models.





§ APPENDIX - CHOICE OF COMPRESSION ARCHITECTURE


Due to the challenge in single GPU training of network architectures that extend RNN-based compression modules, on full resolution 128x128 images, we propose a smaller compression network architecture 𝒩^𝓈𝓂𝒶𝓁𝓁 relative to the original one 𝒩^ℴ𝓇𝒾ℊ𝒾𝓃𝒶𝓁 proposed in  <cit.>. 
We now elaborate the differences between them. 

In our proposed smaller architecture, the 3rd (last) convolutional RNN layer of the encoder ℰ^𝒸ℴ𝓂𝓅 outputs 16 feature maps instead of 512. Therefore, the encoded representation is already compact and the binarizer does not include a convolutional layer. The binarized representation vector length in 𝒩^𝓈𝓂𝒶𝓁𝓁 is 16, rather than 32 in 𝒩^ℴ𝓇𝒾ℊ𝒾𝓃𝒶𝓁. This allows us to achieve a more aggressive maximal compression ratio of 384 vs. 192 (The maximal compression rate is obtained when just a single compression RNN iteration is used. Lower rates are obtained as desired using more iterations). 

The decoder 𝒟^𝒸ℴ𝓂𝓅 in our proposed 𝒩^𝓈𝓂𝒶𝓁𝓁 consists of only three convolutional RNN layers instead of four, and they are smaller in capacity compared to 𝒩^ℴ𝓇𝒾ℊ𝒾𝓃𝒶𝓁.

The 3D reconstruction model in  <cit.> used a random number of viewpoints during training. This was useful for obtaining a model that at test time, can reconstruct shape from an arbitrary number of viewpoints. In contrast,  <cit.> trained their image compression model using a constant, maximal number of RNN iterations, which corresponded to a compression ratio of 24:1. We propose to train 𝒩^𝓈𝓂𝒶𝓁𝓁 and our models based on it, as described in Sec. <ref>, using a number of RNN iterations (and thus, compression rate), selected at random in each training iteration. This significantly reduces training time, and also facilitates slightly more robust performance across varying compression rates. 

Fig. <ref> compares the loss L_comp over the ShapeNet test set of 𝒩^𝓈𝓂𝒶𝓁𝓁 to that of 𝒩^ℴ𝓇𝒾ℊ𝒾𝓃𝒶𝓁, using both constant and randomly varying number of RNN iterations. We see a significant gap of around a factor of 2 in terms of the loss value in favor of N^original. 
To gain an intuition of the visual effect of such gap, we show in Tab. <ref> a comparison of a selection of four images from the ShapeNet test set, decoded using the different architectures, at different compression rates. We see a noticeable difference in quality in favor of N^original, relative to N^small. 
Despite the difference in visual image quality, we use N^small throughout our experiments. In Sec. <ref>, we show that this choice is still appropriate given that our actual task of interest is that of 3D reconstruction. On this task, the performance that our unified model achieves is only a few percent lower than that achieved using uncompressed images, for a wide range of compression rates.

 
 



§ ACKNOWLEDGMENTS

We would like to thank Christopher Choy for his responsiveness to our questions about his work, as well as Alona Golts for her helpful insights.

Yoav Schechner is the Mark and Diane Seiden Chair in Science at the Technion. He is a Landau Fellow - supported by the Taub Foundation. His work is conducted in the Ollendorff Minerva Center. Minvera is funded through the BMBF. This project is funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (grant agreement No 810370: CloudCT) and the Technion Autonomous Systems Program (TASP).



model1-num-names.bst



