Question: Represent the scientific article to find a short TLDR

Question: Recent advances in deep learning have shown promising in many low-level vision tasks. However, solving the single-image-based view synthesis is still an open problem. In particular, the generation of new images at parallel camera views given a single input image is of great interest, as it enables 3D visualization of the 2D input scenery. We propose a novel network architecture to perform stereoscopic view synthesis at arbitrary camera positions along the X-axis, or Deep 3D Pan, with "t-shaped" adaptive kernels equipped with globally and locally adaptive dilations. Our proposed network architecture, the monster-net, is devised with a novel t-shaped adaptive kernel with globally and locally adaptive dilation, which can efficiently incorporate global camera shift into and handle local 3D geometries of the target image's pixels for the synthesis of naturally looking 3D panned views when a 2-D input image is given. Extensive experiments were performed on the KITTI, CityScapes and our VXXLXX_STEREO indoors dataset to prove the efficacy of our method. Our monster-net significantly outperforms the state-of-the-art method, SOTA, by a large margin in all metrics of RMSE, PSNR, and SSIM. Our proposed monster-net is capable of reconstructing more reliable image structures in synthesized images with coherent geometry. Moreover, the disparity information that can be extracted from the "t-shaped" kernel is much more reliable than that of the SOTA for the unsupervised monocular depth estimation task, confirming the effectiveness of our method. Recent advances in deep learning have pushed forward the state-of-the-art performance for novel view synthesis problems. Novel view synthesis is the task of generating a new view seen from a different camera position, given a single or multiple input images, and finds many applications in robotics, navigation, virtual and augmented reality (VR/AR), cinematography, etc. In particular, the challenging task of generating stereo images given a single input view is of great interest as it enables 3D visualization of the 2D input scene. In addition, the falling price and the increasing availability of the equipment required for VR/AR has fueled the demand for stereoscopic contents. The previous works, such as the Deep3D , have addressed the right-view generation problem in a fully supervised fashion when the input is the left-view to which the output is the synthetic right-view at a fixed camera shift. In contrast, our proposed Deep 3D Pan pipeline enables the generation of new views at arbitrary camera positions along the horizontal X-axis of an input image with far better quality by utilizing adaptive "t-shaped" convolutions with globally and locally adaptive dilations, which takes into account the camera shift amount and the local 3D geometries of the target pixels. Panning at arbitrary camera positions allows our proposed model to adjust the baseline (distance between cameras) for different levels of 3D sensation. Additionally, arbitrary panning unlocks the possibility to adjust for different inter-pupillary distances of various persons. Figure 1 shows some generated left and right view images for a given single image input by our proposed Deep 3D Pan pipeline, which we call it the "monster-net" (monocular to stereo network). In this paper, we define "pan" in the context of 3D modeling, implying that camera movement is in parallel to the center view plane. In the following sections, we review the related works to stereoscopic view synthesis and discuss the differences with our proposed method, followed by the formulation of our Deep 3d Pan pipeline and finally, we present outstanding on various challenging stereo datasets, showing superior performance against the previous state-of-the-art methods. Novel view synthesis is a well-studied problem in deep learning-based computer vision, and has already surpassed the classical techniques for both cases of the multiple-image (; ;) and single-image input . The latter, single-image based novel view synthesis, is known to be a much more complex problem compared to multiple-image based ones. Previous deep learning-based approaches usually tend to utilize one of the two techniques to generate a novel view: (i) optical flow guided image warping, and (ii) a "flavor" of kernel estimation, also known as adaptive convolutions. The first technique, optical flow guided image warping, has been widely used by many researchers to indirectly train convolutional neural networks (CNNs) to estimate optical flow or disparity from single or stereo images in an unsupervised fashion, but its final goal was not to synthesize new views. These works include those of (; ; b; ; ; b; ;). However, not all methods have used flow-guided warping to do unsupervised training or to regularize supervised methods for flow estimation. The work of implements plane sweep at the feature level to generate a cost volume for multi-view stereo depth estimation. Plane sweep can be seen as a type of 1D convolution, similar to the 1D kernel utilized in . On the other hand, the second approach, kernel estimation or adaptive convolutions, has proved to be a superior image synthesis technique and has been executed in different ways by many authors. For example: in their early DeepStereo work formulated a CNN capable of synthesizing a middle view by blending multiple plane-swept lateral view inputs weighted by a "selection volume", which can be considered a 1D (or line-shaped) adaptive convolution; in a similar way, devised the Deep3D, a non fully-convolutional network that estimates a series of "probabilistic disparity maps" that are then used to blend multiple shifted versions of the left-view input image to generate a synthetic right-view; The adaptive separable convolutions (SepConv) in the work of approximated adaptive 2D convolutions by two (vertical and horizontal) 1D kernels that are applied sequentially to the input t 0 and t 1 frames for the video interpolation problem; In the works of , although with additional considerations, their multiplane image representation approach can be loosely un- derstood as a 1D adaptive convolution as the final operation involves the reduction of a plane sweep volume; Geometric-aware networks in the work of indirectly achieved adaptive convolutions by learning a fixed number of affine transformations on an input image, where the ing affine-transformed images are then blended together to generate one output image; and finally, in the work of Gonzalez & Kim (2019a), the authors developed the Deep 3D Zoom Net, which estimates a selection volume for the "blending of multiple upscaled versions of the input image", which can be treated as an special case of a 1D adaptive convolution. The DeepStereo and the multiplane image approaches require two or more images as inputs, thus, greatly reducing the complexity of the synthesis task as most ambiguities are removed by counting on multiple views. In our work, we focus on the single-image based stereoscopic view synthesis task, which is a far more difficult problem as the network needs to understand the 3D geometry in the scene, and to handle complex occlusions, ambiguities and non-Lambertian surfaces. Although the aforementioned methods are distinguished one another, as the different synthesis techniques have their own properties, they can be all interpreted as belonging to a category of adaptive convolutions which are visualized in Figure 2. As observed in Figure 2 -(a), DeepStereo and Deep3D share a same shape of kernels, that is, a 1D horizontal-only kernel that samples the pixels at a fixed interval or dilation along the X-axis for all target output pixels. A 1D horizontal-only constantdilation kernel suffers from three major drawbacks: 1. Inefficient usage of kernel values. When sampling the positions opposite to the camera movement (which are the pixel locations corresponding to a 1 -a 3 in Figure 2 -(a) assuming a rightward camera shift), experiments showed that these kernel values would often be zeros. The same effect repeats when sampling the positions further away from the maximum disparity value of the given scene (which corresponds to the pixel location at a 7, assuming that the maximum disparity is 2 and the dilation is 1) as the network is not able to find valid stereo correspondences for these kernel positions; 2. Right-view synthesis is limited to the trained baseline (distance between stereo cameras), as the models over-fit to a specific training dataset with a fixed baseline; and 3. The 1D line kernel has limited occlusion handling capabilities, as the network will try to fill in the gaps with the information contained only along the horizontal direction, limiting the reconstruction performance of the models on the occluded areas. In contrast, the kernels predicted by the geometric-aware networks have deformable structures adaptive to the given input images. However, only one deformed kernel shape is predicted and shared to synthesize all target output pixels, leading to limited performance. Another drawback of the geometric-aware networks is their complexity as they require three sub-networks and a super-pixel segmentation step as pre-processing, hindering the processing of high-resolution images. For the Deep 3D Zoom Net case, the kernel tends to point to the center of the image, as it performs a blending operation of multiple upscaled versions of the input image. The kernel dilation size of the Deep 3D Zoom Net is adaptive according to the zoom factor applied to the input image, which allows for the generation of arbitrary 3D-zoomed output images. Finally, for the video interpolation case, the SepConv approximates an NxN adaptive kernel via a 1xN and an Nx1 component which are sequentially applied to the input images to generate the output. SepConv has, by design, limited receptive fields, as the dilation size is fixed to 1. Besides, the sequential nature of the kernel forces the vertical component to sample pixels from the output of the horizontal convolution, which could be already degraded due to heavy deformations introduced by the horizontal component. Recent works have also attempted to improve upon the stereoscopic view synthesis by improving the loss functions used to train the CNNs. The work of proposed a multi-scale adversarial correlation matching (MS-ACM) loss that learns to penalize structures and ignore noise and textures by maximizing and minimizing the correlation-l 1 distance in the discriminator's featurespace between the generated right-view and the target-view in an adversarial training setup. Whereas the objective function is a key factor in training any CNN, we believe that, at its current state, the stereoscopic view synthesis problem can benefit more from a better pipeline that can handle the previously mentioned issues and using the widely accepted l 1 and perceptual losses for image reconstruction, rather than a more complex loss function. Our proposed dilation adaptive "t-shaped" convolutions incorporate global (new camera position along the X-axis) and local (3D geometries of specific target pixels) information of the input scene into the synthesis of each output pixel value by not only learning the specific kernel that will generate each output pixel, but also by learning the proper dilation value for each kernel. The "t" shape of the kernel allows the network to account for occlusions by filling-in the gaps (missing information in the output) due to shifted camera positions using not only left-and-right pixels (like DeepStereo and Deep3D), but also up-and-down neighboring pixel information. In addition, the notions of global and local dilations allow our proposed monocular to stereo network, the monster-net, to generate arbitrarily 3D panned versions of the input center view along the X-axis, a useful feature not present in previous works that allows adjusting for eye-to-eye separation and/or level of 3D sensation. In order to effectively synthesize an arbitrary 3D panned image, we propose a global dilation filter as shown in Figure 3. Our proposed cross-shaped global dilation filter where T c (x, y) is the filter parameter value of T d (p) at the center location p. The upper, bottom, left and right wing parameters (T u, T b, T l, T r) of the cross-shaped dilation (d) filter are defined as where n u, n b, n l and n r indicate the numbers of filter parameters in T u, T b, T l, and T r, respectively. For the cross-shaped dilation filter shown in Figure 3, it is more appropriate to have a longer length of Figure 4: Our proposed "t-shaped" kernels are overlaid on top of a center input image. The distance between samples (dilation) is adaptive according to the amount and direction of 3D panning to be applied to the input image and the local 3D geometry of the scene. the right (left) filter wing than the other three wings when the camera panning is rightward (leftward), as it allows capturing more useful information for the synthesis of a right (left) panned image. In this case, n r (n l) is set to be greater than n l (n r), n u and n b, such that the global dilation filter showed in Figure 3 can be elaborated as a "t-shaped" kernel which can then take into account the camera panning direction for synthesis. Figure 4 shows examples of "t-shaped" kernels overlaid on top of an input center image. As shown in Figure 4 -(a), the "t-shaped" kernel has a longer left wing of filter parameters for the synthesis of a leftward camera panning while in Figure 4 -(b) it shows a longer right-wing of filter parameters for the synthesis of a rightward camera panning. Why "t" shape? Experiments with symmetric kernel shapes (e.g., "+" shape) were performed first, but it was noted that most of the elements on the left (right), upper and bottom sides against the centered red dot of the kernel tended to have very small values close to zeros for most target pixels for the rightward (leftward) movement of a camera. Similar to SepConv, the experiments with a horizontal kernel applied first followed by a vertical kernel were performed, yielding poor . It was discovered that the "t" shape is more efficient than the "+" shape as it picks up more effective sampling positions with a fewer parameters than the standard adaptive convolutions such as those in . As depicted in Figure 5, the "t-shaped" kernels can embed useful information like disparity and occlusion from a monocular image into the stereo synthesis process. The longer right (left) wing of the "t-shaped" kernel contains disparity information, as it will try to sample pixels from the right (left) side to the target position when the camera is assumed to move in the rightward (leftward) direction. Figure 5 -(a) depicts a primitive disparity map D p that was constructed by the weighted sum of the kernel values in the longer wing part as described by where T r (x + id, y) is the i-th value of the longer wing T r at pixel location p = (x, y) for the rightward 3D panning of an input center image I c. Note that D p is normalized in the range. Interestingly, as shown in Figure 5 -(a), the generated disparity map looks very natural and appropriate, which implies the effectiveness of our "t-shaped" kernel approach. The short left (right), upper and bottom wings of the "t-shaped" kernel contain occlusion information, as the network will try to fill in the gaps utilizing surrounding information that is not present in the long part of the "t-shaped" kernel. It is also interesting to see the occlusion map in Figure 5 -(b) where a primitive rightward occlusion map O r p was constructed by summing up the "t-shaped" kernel values in the short wing parts according to the following: The bright regions or spots in Figure 5 -(b) indicate the occlusions due to the camera shift along the horizontal axis of the input center image, which are likely to happen for the case of the camera's rightward panning. For both Equations and, the primitive disparity and occlusion maps for the leftward panning case can be obtained by swapping the r and l indices. ) maps generated from the proposed "t-shaped" kernel. In general, the disparity amounts between stereo images are variable at different pixel locations according to the distance between stereo cameras and the local scene geometries. Therefore, it is necessary to take into account the variable disparity in synthesizing a new view in a globally and locally adaptive fashion. For this, a "t-shaped" kernel is introduced with a controllable dilation factor by which both camera shift and local changes in image geometry can be effectively taken into account when synthesizing a new (left or right) view for the input center image. Any kernel with a fixed dilation may cause a limited accuracy in synthesizing a novel view because the disparity amounts vary over the whole image according to the cameras' baseline and the local geometries. So, our "t-shaped" kernel is proposed to make the synthesis of novel views not only globally, but locally adaptive to the camera shift and its local changes in image geometry by controlling its dilation size per-pixel in the output image. Globally, a short dilation value is more appropriate when slightly shifting the camera, while a high dilation value is desirable when largely shifting the camera position. In a local manner, a small dilation value is appropriate for far-away objects from the camera while very close objects to the camera can be better reconstructed with a larger dilation value. We define the global dilation g d as the pixel distance between two consecutive kernel sampling positions, which is given by the pan amount P a to be applied to the input center image I c divided by the total number of filter parameters in the longer "t-shaped" kernel wing (n l or n r). P a has a unit of pixels mapped in the image corresponding to the camera shift into the left or right direction and takes on floating numbers. Therefore, the global dilation g d is given by where P a takes on positive (negative) values for the rightward (leftward) panning scenario. The pan amount needed to generate a left-view or a right-view is determined during training according to the closest possible objects to the camera. The "closest possible objects" vary over different training datasets. For our novel view synthesis task, like in (; b), we assume the KITTI dataset to have a maximum or "closest possible object" disparity of 153 pixels. During training, P a is set to 153 and -153 for the rightward and leftward panning, respectively. While global dilation allows the "t-shaped" kernel to take into account the global camera shift, a locally adaptive mechanism is needed to synthesize new views of locally variable disparity. Such a mechanism is realized by first generating multiple images with the "t-shaped" kernel at N different dilations and blending them per-pixel in a locally adaptive manner. The blending is a weighted sum of filtered images by the "t-shaped" kernel with N different dilations, where the blending weights (w 1, w 2, . . ., w N) control the local dilation per-pixel and are learned via a convolutional neural network (CNN) along with the parameter values of the "t-shaped" kernel. Let |g d | be the maximum dilation value that is a fractional number. filtered images according to the corresponding blending weights (w 1, w 2, . . ., w N). Based on the N different global dilations, the output image value I t o (p) at a pixel location p can be calculated as where indicates a blending weight for the i-th global dilation. We propose an end-to-end trainable CNN, called the "monster-net" (monocular to stereo net). The monster-net is made of two main building blocks, a novel view synthesis network, the "t-net", and a resolution restoration block, the "sr-block". Given an input center image I c and pan amount P a, the final output panned image I o is obtained by sequentially applying the aforementioned modules by where θ t and θ sr parameterize the t-net and the sr-block respectively. {I n cs} is the stack of progressively shifted-downscaled versions of the input center image I c described in the SR-BLOCK section. The t-net. The "t-net" estimates both the "t-shaped" global dilation kernel parameters and the adaptive local dilation weights. The t-net is designed to have large receptive fields to synthesize detailed image structures of a new view image which corresponds to a shifted camera position. This is because such a large receptive field is useful in capturing the global image structure and contextual information for a new view image to be synthesized. For this, an auto-encoder with skip connections (not a U-net structure) is adopted, which allows to effectively have very large receptive fields and to efficiently fuse global and local (fine details) information on the decoder stage. For better feature extraction, we adopt the residual connections in the encoder side as proposed by (b). The t-net estimates all necessary values to perform the operation described by Equation. The t-net, depicted in Figure 6, has two output branches: the first output branch yields 81 channels, where the first 49 are horizontal parameter maps and the following 32 vertical parameter maps; the second output branch generates the 3-channel blending weight maps for the local adaptive dilation. That is, each channel-wise vector at a pixel location for the first output branch corresponds to the t-kernel parameter values [T c, T, and each channel-wise vector for the second output branch corresponds to the blending weights [w 1, w 2, . . ., w N] for local dilations in Equation. As our t-net is devised to generate arbitrarily panned novel views, feeding the pan amount as a 1-channel constant feature map (P a (p) = P a ∀ p) helps the network take into account the varying pan direction and the amount of occlusions on the 3D panned output. The effect of feeding the pan amount is further discussed in appendix A-1. Super resolution (SR) block. As generating a full resolution dilation-adaptive t-kernel would be computationally too expensive, we propose to estimate it at a low resolution (LR) to synthesize a novel view of the half-resolution, and then to apply deep learning based SR techniques to bring the LR novel view to the high (or original) resolution (HR). In comparison, in Deep3D and SepConv, Figure 7: (a) Shifted-LR versions of the center-view contain different information as they are sampled from different groups of pixels via bilinear interpolation depending on the stride (controlled by the maximum disparity). (b) Our light sr-block. All convs have 3x3 kernels otherwise specified. the estimated LR kernel is upscaled with conventional methods to the HR and then applied to the input image(s), which is a costly operation as it is carried out in the HR dimensions and can lead to blurred areas as the kernel is just bilinearly interpolated. In our proposed pipeline, instead of utilizing common single image SR methods like (; ;), we propose to apply a stereo-SR method. The stereo-SR technique in takes a LR stereo pair (left and right views) as input and progressively shifts the right-view producing a stack that is concatenated with the left-view and later processed by a CNN to obtain the super-resolved left-view. This process is made at an arbitrary and fixed stride (e.g. 1 pixel at every step of the stack) and does not take into account the maximum disparity between the input views. For our Deep 3D Pan pipeline, we propose to use the maximum disparity prior that can be obtained from the long wing of the t-kernel to dynamically set the shifting stride. Additionally, instead of interpolating and processing the low resolution panned view I t o (p) on the HR dimensions, we progressively shift and then downscale the high-resolution center view I c by a factor of x2. This allows our sr-block to operate on the LR dimensions without performance degradation, as high frequency information in the horizontal axis is not lost but distributed along the levels of the shifted center view stack as depicted in Figure 7 -(a). Our sr-block, depicted in Figure 7 -(b), is a simple, yet effective module that takes as input the LR I t o view and the shifted-downscaled center view stack I n cs described by where g(I, s) is an s-strided horizontal-shift and 2x down-scaling operator applied on image I. The stride s can take any real number and the ing image is obtained via bilinear interpolation. N s is the depth of the stack, and was set to N s = 32 for all our experiments). The stack is concatenated with the LR I t o and passed trough four Conv-ReLU layers followed by a residual connection as shown in Figure 7 -(b). The final step up-scales the ing features into the target resolution via nearest interpolation followed by a convolutional layer. The last layer reduces the number of channels to three for the final RGB output I o. Nearest upscaling was adopted as it yields no checkerboard artifacts in contrast with transposed or sub-pixel convolutions . Published as a conference paper at ICLR 2020 To demonstrate the effectiveness of our "t-shaped"-dilation-adaptive kernel, we performed several experiments on the challenging KITTI2012 , KITTI2015 , and CityScapes datasets. As these stereo datasets only consist of outdoor scenes, we also performed experiments on our indoors dataset, called the VICLAB STEREO dataset. Surprisingly, to our best knowledge, this is the first stereo dataset available that focuses on the indoor scene, which is planned to be publicly available for research. Additionally, our formulation of global and local adaptive dilations allows our monster-net to be trained on multiple stereo datasets at the same time, even if these have different baselines. Instead of over-fitting on a single camera baseline like the previous methods (Xie et), our monster-net can build knowledge when simultaneously trained on many datasets. To our best knowledge, our Deep 3D Pan pipeline is the first method designed to be trained on multiple baseline datasets concurrently for the stereoscopic view synthesis problem where unsupervised monocular depth estimation is even used particularly. For more details about the datasets and multi-dataset training, please see the appendix A-3. We compare our monster-net against the stereoscopic view synthesis SOTA: Deep3D and a version of SepConv modified for right-view synthesis. Firstly, for a fair comparison, the backbone convolutional auto-encoders for the Deep3D and SepConv were set up to be equivalent to our t-net's, that is, a six-stage encoder-decoder with skip connections and residual blocks in the encoder side. Secondly, we compare our monster-net with Deep3D-B, a "Bigger" version of Deep3D, where, instead of 32 elements in the 1D kernel as in its original work, we use 49 elements to match the number of horizontal kernel values in our t-net. Thirdly, we compare against SepConv-D, a dilated version of the SepConv such that the receptive field of the separable convolutions has a size of 153x153. The Deep3D and the SepConv models are trained without using perceptual loss as in their original works. For a more meaningful comparison, the Deep3D-B and the SepConv-D are trained with a combination of l 1 and perceptual loss l p , and demonstrate that a better loss function than l 1 does not contribute enough to the stereoscopic view synthesis problem. For more implementation details, reefer to the appendix A-4. Additionally, we compare the quality of the embedded disparity in the long wing of the "t-shaped" kernel with those of the state-of-the-art models for the monocular depth estimation task. For that, we first define a disparity refinement sub-network that uses the primitive disparity obtained from the long wing of the "t-shaped" kernel as prior information. Secondly, we define a special postprocessing (spp) step, which, instead of relying on a naive element wise summation as in , takes into account the ambiguities of the first and second forward passes to generate a remarkable sharp and consistent disparity map. For more details on the refinement block and our special post-processing, reefer to the appendix A-2. Table 1 shows the performance comparison for our method and previous works. It is important to mention that our monster-net performs inference on full resolution images, while the previous approaches for single-view novel view synthesis perform estimation on reduced resolution inputs. Our method outperforms the Deep3D baseline by a considerable margin of 0.7dB in PSNR, 2.0 in RMSE, and 0.03 in SSIM. The qualitative are shown in Figure 8. Our method produces superior looking images. In Deep3D and SepConv, many objects appear too blurred such that their boundaries can hardly be recognized in the synthetic images (e.g the motorcycle, persons, traffic signs, etc.). We challenged the models trained on KITTI (K) to perform inference on the CityScapes validation split (CS), and observed that our method generalizes much better than the Deep3D baseline with up to 3dB higher in PSNR. When training the monster-net with K+CS, we get an additional improvement of 4dB PSNR in the validation CS dataset. Incorporating an indoor dataset to our training pipeline is also possible, making our network applicable to a wide variety of scenarios. We added the VI-CLAB STEREO (VL) dataset to the training, that is K+CS+VL, and observed little impact on the K dataset performance as shown in Table 1. We also tested the performance of our monster-net on the validation split of the VL dataset. We observed that our full monster-net trained on K+CS achieved a mean PSNR of 19.92dB, while achieving a mean PSNR of 21.78 dB when trained on K+CS+VL. For a network trained on the outdoors dataset only it is difficult to generalize to the indoors case, as the latter contains mainly homogeneous areas, whereas the outdoors case mainly contains texture rich scenes. Visualizations on CS and VL, and ablation studies that prove the efficacy of each of our design choices can be found in the appendices A-5, A-6 and A-8. With the addition of a relatively shallow disparity refinement sub-network, the monster-net remarkably outperforms all the state-of-the-art models for the unsupervised monocular depth estimation task, as shown in Table 2. Our monster-net with disparity refinement even outperforms the supervised monocular disparity estimation methods such as and multiple view unsupervised methods such as (a;). We presented an adaptive "t-shaped" kernel equipped with globally and locally adaptive dilations for the Deep 3D Pan problem, defined as the task of arbitrarily shifting the camera position along the X-axis for stereoscopic view synthesis. Our proposed monster-net showed superior performance to the SOTA for right-view generation on the KITTI and the CityScapes datasets. Our monsternet also showed very good generalization capabilities with 3dB gain in PSNR against the Deep3D baseline. In addition, our method presents no-discontinuities, consistent geometries, good contrast, and naturally looking left or right synthetic panned images. Our monster-net can be extended for image registration, monocular video to stereo video, and generation of novel views at any camera translation by just allowing pixel-wise rotation of our "t-shaped" kernel. The larger the pan amount, the greater the occlusions to be handled in the synthetic output image. The effect of feeding the pan amount P a as a one-channel constant feature to the t-net can be visualized in Figure 9, where multiple primitive disparity and occlusion maps are depicted for different pan amounts. As shown in Figure 9, the network generates different maps for different magnitudes and directions of P a while keeping the input center image I c unchanged, confirming the effect of the pan amount as prior knowledge to the network. The difference between the disparity maps can be appreciated in the "holes" or "shadows" casted in the objects borders, as they represent the occluded content seen from the output 3D panned image. In the red box in Figure 9 it is observed that the shadows casted by leftward and rightward camera panning appear in opposite sides ob the objects. In the yellow box, it is observed that the larger the P a, the larger the shadows projected, as more occlusions are to be handled. The disparity refinement network architecture, as depicted in Figure 10, has two input branches: one takes a stack of the 2x bilinearly upscaled center image disparity prior (D cp) and the RGB center image (I c); and the other is fed with a stack of the 2x bilinearly upscaled output panned view disparity prior (D op) and the generated panned view (I o). The disparity refinement block is a relatively shallow auto-encoder with skip connections and rectangular convolutions as fusion layers in the decoder stage. This allows to increase the receptive field size in the horizontal axis, thus improving the stereo matching performance, as suggested by (b). We configure the output layer of our refinement network with the last layer of Gonzalez & Kim (2019b)'s architecture, which allows to do ambiguity learning in our refinement block. Ambiguity learning allows the network to unsupervisedly account for occlusions and complex or clutered regions that are difficult to minimize in the photometric reconstruction loss (b). We train the refinement block with the loss functions defined in (b) and a new additional loss towards producing the refined disparity maps similar to the primitive disparity maps D cp and D op. The refinement network is encouraged to produce refined center and panned disparity maps D c and D o similar to their primitive counterparts of half the resolution (as they are estimated from the t-net), by minimizing the following primitive disparity loss where D 1/2 c and D are the bilinearly downscaled and refined center and panned disparity maps by a factor of 1/2, respectively. We give a weight of 0.5 to this new loss term. The disparity refinement block can be trained end-to-end along with the monster-net or from a pre-trained monster-net. Instead of relying on naive post-processing approaches like in , which consist on running the disparity estimation twice with normal and horizontally flipped inputs and then taking the average depth, we define a novel special post-processing step (spp) by taking into account the ambiguities in the first and second forward pass. We noted that the ambiguity learning from (b), which we incorporate in our disparity refinement block, can be used to blend the ing disparities from the first and the second forward pass such that only the best disparity estimation (or ambiguity free) from each forward pass is kept on the final post-processed disparity. Figure 11 depicts our novel post-processing step, which consist on running the forward pass of our monster-net with disparity refinement block with P a = 153 and P a = −153, for the first and the second pass respectively. Then, the generated ambiguity masks of each forward pass are concatenated to create a two-channel tensor and passed through a softmax operation along the channel axis. The ing soft-maxed ambiguities are used to linearly combine the disparity maps of each forward pass. As can be observed in Figure 11, the soft-maxed ambiguity mask effectively segment the best disparity estimation from each forward pass. Figure 12 shows the primitive disparity map, the subsequent refinement step, the naive post-processing (pp) and our novel post-processing (spp). The KITTI dataset is a very large collection of mid-resolution 370x1226 stereo images taken from a driving perspective. We used the KITTI split as suggested in , which consists of 29,000 stereo pairs from 33 different scenes of the KITTI2012 dataset. We set apart the KITTI2015 dataset for validation as it contains 400 images excluded from the KITTI split. Additionally, the KITTI2015 contains sparse disparity ground truths (GTs) which are obtained from LIDAR and then are refined by car CAD models. We use these GTs to evaluate the quality of the estimated disparity that can be extracted from the long wing of the t-kernel. CityScapes is a higher resolution stereo dataset with 24,500 stereo pairs that we extract from the train, test and train extra directories for training. The val directory is left for validation with 500 stereo pairs. We pre-process the CityScapes dataset for faster and more robust training. We first remove the top 25, bottom 200 To our best knowledge, our Deep 3D Pan pipeline is the first method designed to be trained on multiple baseline datasets at the same time for the stereoscopic view synthesis problem and the unsupervised monocular depth estimation task. It should be noted that the work of only handled the supervised monocular depth estimation task for multiple datasets with different camera intrinsics utilizing "CAM-Convs", which is a simpler problem than our unsupervised problem. While they require to know the intrinsic matrix for each dataset, along with added computational complexity to perform the "CAM-Convs", our method only requires to know the dataset baseline (distance between cameras). To train on multiple datasets, the only required step is to multiply the P a by the relative baseline with respect to a reference dataset. For instance, the baseline in the KITTI dataset is about 54cm, and, as mentioned before, we have set this baseline to correspond to P K a = 153. Then for the CityScapes dataset, whose baseline is 22cm, its pan amount will be given by P For the training of all models, we used the Adam optimizer with the recommended β's (0.5 and 0.999) for the regression task with a batch size of 8 for 50 epochs. The initial learning rate was set to 0.0001 and was halved at epochs 30 and 40. The following data augmentations on-the-fly were performed: Random resize with a factor between 0.5 and 1 conditioned by the subsequent 256x512 random crop; random horizontal flip, random gamma, and random color and RGB brightness. It was observed that vertical flip has made the learning more difficult, thus it was avoided. When training our model (the monster-net), the training images were sampled with a 50% chance for rightward (P a > 0) or leftward (P a < 0) panning. Additionally, random resolution degradation was applied to the input only by down-scaling followed by up-scaling back to its original size using a bicubic kernel with a scaling factor between 1 and 1/3 while keeping the target view unchanged. Random resolution degradation has improved our by making the network more sensitive to edges and forcing it to focus more on structures and less on textures. Similar "tricks" have been used in previous works in the form of adding noise to the input of discriminators (Sønderby et al., 2017;) to make them invariant to noise, and more sensible to structures. When training our monster-net, either the left or the right view can be used as the input during training. When the left-view is used as the center input view I c, the pan amount P a is set to 153 and the ground truth I gt is set to be the right image. In the opposite case, when the center view is the right image, P a is set to -153 and the left-view is set as the GT. For our monster-net, the "t-shaped" kernel was set to have short wings of 16 elements and a long wing of 32 elements plus one center element T c. For the Deep3D, the 1D kernel is set to have 33 elements and 49 elements for the Deep3D-B variant. For the SepConv and the SepConv-D cases, we set the horizontal and vertical 1D kernels to have 1x51 and 51x1 shape, respectively, as in . We train our monster-net with a combination of l 1 loss and perceptual loss . The later measures the distance between the generated view (I o or I t o) and the ground truth (I gt) images in the deep feature space of a pre-trained network for image classification. The perceptual loss is especially good to penalize deformations, textures and lack of sharpness. The mean square error of the output of the first three max-pooling layers from the pre-trained V GG19 , denoted by φ l, was utilized as the perceptual loss function. To balance the contributions of the l 1 and perceptual losses, a constant α p = 0.01 was introduced. This combination of loss terms was applied to both the low-resolution panned image I t o and super-resolved panned image I o to yield the total loss function L pan as follows: where I 1/2 gt is the bilinearly downscaled version of the ground truth by a factor of 1/2. Visualizations on the CittyScapes (CS) datasets for our monster-net trained on KITTI (K) and on KITT + CityScapes (K+CS) are depicted in Figure 13. The First row of Figure 13 shows the synthesized images from the Deep3D baseline, it can be noted that it over-fits to the training baseline of the KITTI dataset, performing very poorly on CityScapes. The subsequent rows show the for our monster-net when trained without and with the CityScapes dataset. Our models generate very good structures and sharp panned views as depicted the red highlighted regions on Figure 13 for both cases of with and without CityScapes training. When trained on KITTI-only, our method generalizes very well on the CityScapes dataset, with a performance improvement of 3dB over the Deep3D baseline as shown in Table 1. When trained on K+CS, we obtain an additional improvement of 4dB against the KITTY-only trained monster-net. Additionally, we present for our monster-net trained with and without perceptual loss, (L 1) and (l 1 + l p) respectively, on the CityScapes dataset. Sharper with clear edges and structures are achieved when utilizing the perceptual loss, as depicted in the highlighted areas in Figure 13. A network that is trained on outdoor datasets only is not able to generalize well on highly homogeneous areas that are common in the indoors datasets but rare in the outdoor scenes. Visualization of the synthetic views generated for the VICLAB STEREO (VL) indoors dataset is provided in Figure 14. We compare the of our network trained on K+CS versus those of our monster-net trained on K+CS+VL. The latter achieves better generalization, as depicted in Figure 14, with marginal performance decrease in the KITTI dataset (-0.09dB) and considerable quality improvement on the VICLAB STEREO dataset (+1.86dB), as showed in the last row of Table 1. A.7 3D PANNING BEYOND THE BASELINE As our method allows for arbitrary camera panning, it is possible to perform 3D pan beyond the baseline as depicted in Figure 15, where the pan amount was set to go 30% beyond the baseline for both leftward and rightward camera panning, that is P a = −200 and P a = 200 for the per-scene top and bottom samples respectively, where the input image for both pan amounts was set to be the left-view. It is observed that the monster-net with adaptive dilations generates naturally looking new views with consistent structures and without discontinuities even at beyond training baseline 3D panning. We demonstrate the contribution of our design choices in this section. Our main contribution is the adaptive "t-shaped" kernel equipped with globally and locally adaptive dilations. Figure 16 -(a) shows the effect of adaptive dilations in comparison with the fixed dilation. As can be observed, the ing synthesized image by a fixed local dilation kernel shows unnaturally looking regions with low contrast, discontinuities and blurs. Unlike any previous work, our pipeline can greatly benefit from training on multiple datasets at the same time, as shown in Figure 16 -(b). That is, our method greatly benefits from training on two datasets (KITTI and CityScapes) as it exploits the baseline information via its global adaptive dilation property. Training on both KITTI and CityScapes contributes to improved geometry reconstruction as the network is exposed to a wide variety of objects at many different resolutions where, in addition to applying random resizing during training, the resolutions and baselines of these two datasets are very different. Figure 16 -(c) shows the effect of utilizing our sr-block. Even if the quality of the panned image I t o is good in terms of structure, the sharpness is further improved by the addition of the super resolution block. Finally, we analyze the effect of the perceptual loss. By utilizing the perceptual loss, our monter-net is able to better synthesize rich textures and complex or thin structures, as depicted in Figure 16 -(d), even though the PSNR and SSIM are slightly lower as shown in Table 1. The last is known as the "perception-distortion which suggests that better synthetic looking images not always yield higher PSNR/SSIM. Our monster-net with refinement block beats the current state-of-the-art methods for unsupervised monocular depth estimation in terms of prediction accuracy for the KITTI2015 dataset. As shown in Table 3, the primitive disparity D p, that can be extracted from the longer wing of the "t-shaped" kernel, is already among the best performing unsupervised methods. When we add the refinement block with ambiguity learning, our model outperform those of the state-of-the-art. Furthermore, we get an additional improvement in the a 1 accuracy metric when we add our novel specialpost-processing (spp) step. The qualitative comparison against previous methods and the ground truth disparity is shown in Figure 17. It is noted that our monster-net with disparity refinement and special-post-processing generates very reliable disparity maps even on thin structures and image borders. Additionally, our method benefits from very good detection of far away objects Table 3: Disparity estimation performance on the KITTI2015 metrics from . Models are trained with (V) video, (S) stereo, semi global matching (SMG) or GT depth (Supp), and can take stereo inputs (s), nine consecutive input frames during testing and training (9-view), or one input frame during testing and nine consecutive views as supervision during training (1/9-view). Additionally, (st) stands for student, (t) for teacher, (pp) for post-processing and (spp) for special post-processing. The best performing models in terms of a 1 threshold, which is the percentage of disparity values with a relative error less than 0.25, are highlighted in bold.

Novel architecture for stereoscopic view synthesis at arbitrary camera shifts utilizing adaptive t-shaped kernels with adaptive dilations.

We present and evaluate sampling-based point cloud decoders that outperform the baseline MLP approach by better matching the semantics of point clouds.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Deep Neutral Networks(DNNs) require huge GPU memory when training on modern image/video databases. Unfortunately, the GPU memory as a hardware resource is always finite, which limits the image resolution, batch size, and learning rate that could be used for better DNN performance. In this paper, we propose a novel training approach, called Re-forwarding, that substantially reduces memory usage in training. Our approach automatically finds a subset of vertices in a DNN computation graph, and stores tensors only at these vertices during the first forward. During backward, extra local forwards (called the Re-forwarding process) are conducted to compute the missing tensors between the subset of vertices. The total memory cost becomes the sum of the memory cost at the subset of vertices and the maximum memory cost among local re-forwards. Re-forwarding trades training time overheads for memory and does not compromise any performance in testing. We propose theories and algorithms that achieve the optimal memory solutions for DNNs with either linear or arbitrary computation graphs. Experiments show that Re-forwarding cuts down up-to 80% of training memory on popular DNNs such as Alexnet, VGG, ResNet, Densenet and Inception net. The standard DNN training process consists of two alternated stages: forward and backward. FIG0 (a) illustrates an example of feed-forward neural networks. In the forward stage, the network takes an input tensor, [BatchSize × Channel × W idth × Height], and computes the tensors at each layer until producing the output. In the backward stage, difference between the output and ground truth is passed back along the network to compute the gradients at each layer. The regular training approach saves tensors at all layers during forward, because they are all needed to compute gradients during backward. The total memory cost is the sum of cost over all layers. In popular backbone DNNs for feature extraction of images, such as AlexNet BID13 ), VGG BID22 ) and ResNet BID10 ), the memory cost increases quadratically with the input image resolution and network depth. For example, given an median size input tensor of, ResNet101 requires around 5000 MB. In more challenging tasks, DNNs that detect small objects and large number of object categories require input image resolution of more than 600 × 600 BID18; BID23; BID17 ). The memory issue is worse for video-based DNNs, such as CDC BID21 ), C3D BID12 ) and 3D-ResNet BID9 ). To model complex activities in video, the input tensor may contain 64 frames. Moreover, DNN training takes much more memory than testing. In order to train DNNs with large databases and big learning rate, the batch size can be up to 64. In training DNN compositions, such as Generative adversarial networks (GANs), multiple generator and discriminator networks are simultaneously stored in GPU memory. Existing efforts to address memory issues presented three main approaches: Better single GPUs. Recent GPUs provide larger memory at the expense of exponentially growing price and power consumption. For instance, from TitanXp, Quadro P6000 to Tesla V100, for 1-2.7 times increase in memory, the prices increase 2.8-8.5 times. Parallelization among multiple GPUs BID8; BID20;; BID15 BID16; BID27; BID2; BID1 ), which requires expensive The regular approach saves all tensors during forward, and uses these tensors to compute gradients during backward. (b) Reforwarding (our) saves a subset of tensors during the first forward, and conducts "Re-forward" to compute tensors for gradients during backward.clusters, introduces substantial I/O cost, and does not reduce the total memory cost. Low-level heuristic techniques. Optimization of computation graphs BID3 ), which merges inplace operations into non-inplace operations to cut down memory. Liveness analysis BID3 ), which dynamically recycles garbage tensors in training epochs. These approaches are specific to certain DNN structures, data and tasks. To address above issues, we propose a fundamental approach that explores trade-off between memory and computation power of GPUs. Note that recent affordable GPUs, although limited in memory (12GB), provide exceptional improvement in GPU cores and FLOPS. Trading computational time for memory is a very attractive solution that make it possible to train very heavy DNNs with finite GPU memory. Our approach only saves tensors at a subset of layers during the first forward, and conduct only extra local forwards to compute the missing tensors needed during backward. We call the extra forward process as Re-forwarding. The total memory cost is the sum of the cost at the subset of layers and the maximum memory cost among local re-forwards. Training with Reforwarding, see FIG0 (b), leads to substantial memory reduction. We propose sophisticate theories and efficient algorithms that achieve the optimal memory solution of arbitrary computation graphs. To alleviate the memory pressure from a single GPU processor, many researchers utilized the wellestablished techniques for distributed computation BID8; BID20 BID15 BID16 BID27; BID2; BID1 ). These techniques distribute memory pressure to possibly infinite GPUs or server clusters, but do not reduce the total memory cost of DNNs. Other researchers reduced the memory on finite hardware by optimizing computation graph of DNN and performing liveness analysis. The computation graph of DNNs describes the dependencies of tensors among layers. Liveness analysis recycles garbage to manage memory. These ideas were originated from compiler optimization BID3 ) and has been widely adopted by deep learning frameworks: Theano BID4; BID5 ), MXNet BID6 ), Tensorflow BID0 ) and CNTK BID26 ). Some other techniques efficiently swap data between CPU and GPU BID25; BID19 ). These techniques usually cost extra I/O time and still do not actually reduce the total memory cost. The closest work to our approach, Chen et al. ), uses the gradient checkpoints (similar to the subset of layers in Re-forwarding). However, ) only worked on linear computation graph via a heuristic algorithm. Our approach generates optimal solutions for both linear and arbitrary computation graphs. Our algorithm reduces training memory by manipulating high-level tensors, therefore is generalizable to any DNNs and their compositions. All previous techniques are compatible to our approach and can further improve the memory efficiency of DNN training. Denote a computation graph as G = E, V. E = {e i} and V = {v i} are the edges and vertices in the computation graph, respectively. In deep neural networks, the vertices represent the tensors and the edges represent operations. Denote function l(·) as a measure of memory cost. V R is the subset of vertices saved during the first forward. l(v i) is defined as the memory cost of storing vertex v i. For two adjacent vertices v i and v j in set V R, the memory cost during re-forwarding from v i to v j is defined as l(v i, v j) = j−1 t=i+1 l(v t), which is the sum of cost over all the vertices between v i and v j. Using these notations, the memory cost of training with re-forwarding is formulated as DISPLAYFORM0 where the first term is the sum of the memory cost of all the stored tensors, and the second term is the maximal cost among the re-forwards. For easy illustration, we start by formulating Re-forwarding on Linear Computation Graphs (LCG) (FIG1). For LCGs, Eqn. 1 can be solved in two cases. Case LCG with Non-identical Vertex Cost: When the assumption of identical cost does not hold, the solution to Eqn. 1 does not have an analytic form. Denote the maximal Re-forward cost max j l(v j, v j+1) as a constant C, and the solution to Eqn. 1 is reduced to solving for min DISPLAYFORM1 Set the maximal term as l(v i, v j) Construct Accessibility Graph 4:Find the shortest path in the Accessibility Graph as the solution 5:Compute the actual total cost of the solution 6:Save the solution if it's better. Suppose the actual max term of this solution is B, and l(v i, v j) = C, skip the loops where DISPLAYFORM0 All the Re-forward costs in an optimal solution satisfy the constraint l(v j, v j+1) ≤ C. We solve Eqn. 1 by constructing a new graph, called Accessibility Graph DISPLAYFORM1 DISPLAYFORM2 is equivalent to finding the shortest path from the source vertex and the target vertex in the Accessibility Graph. Notice that in the optimal solution, the max term equal the one maximal term among all l(v i, v i+1) terms. To traverse all possible max terms, we can simply compute the loss of every vertex pair and use it as a possible max term. Given a max term C, suppose the actual max term of the solution under C is B and B < C. It's obvious that for all the max terms B ≤ max < C, the solution would be the same solution. Therefore, these max terms can be skipped. Algorithm 1 summarizes the process for searching an optimal solution for LCG. Suppose there are N vertices in the computation graph, the time complexity of Algorithm 1 is O(N 4) 1. As generalization of DNNs with LCG, we present theory 2 and algorithms for DNNs with Arbitrary Computation Graphs (ACG), in particular the acyclic directed graphs FIG1 ). The optimal solution of Re-forwarding corresponds to an optimal division of ACG, such that memory cost (Eqn. 1) is minimum. We denote that an ACG is divided into end-to-end segments by a set of vertices. These end-to-end segments can have multiple endpoint vertices, for example, multiple source vertices and multiple target vertices. In this paper, as an assumption and also for simplification, these end-to-end segments are narrowed down to those with only one source vertex and one target vertex. Another assumption in the case of ACG is imposed on the operation that has multiple inputs: one can compute the gradients of output with respect to the gradients of inputs without using the current value of inputs. Examples of operations that meet this assumption are: concatenation (the gradient of output is also the concatenation of the gradient of input), add (the gradient of output equals the gradient of input), etc. An example that breaks this assumption is multiplication (the gradient of input depends on the input). Fortunately, most of the popular networks meet this assumption. A simple way to remove this assumption is to store all the input tensors of this multi-input operation. However, this is not modeled by our loss function and may lead to sub-optimal solution. In summary, there are only two assumptions in our approach: the segment in a solution only has two endpoints (source and target). the multi-input operation can compute the gradients of output without using the current value of input. Under these two assumptions, our approach is optimal for ACGs.. Definition 1. Closed Set: A set s containing vertices and edges is a closed set if and only if it satisfies the following three properties: 1. All the vertices of s have a common ancestor v i and a common descendent v j; 2. Denote the vertex subset of s as V, edge subset as E, and the set of edges between two arbitrary vertices of V ∪ {v i, v j} is E, the edge from v i to v j (if exists) as e ij. E must either be E or E − {e ij}; 3. An arbitrary v 1 ∈ V doesn't have edge with another arbitrary v 2 / ∈ V ∪ {v i, v j}. For multiple valid closed sets between v i and v j, we denote the largest one as DISPLAYFORM0 In the definition of Closed Set, property 1 corresponds to the two endpoint assumption in section 4.1 where the two endpoints become v i and v j in the definition. Property 2 confines the edge subsets of 1 More detailed complexity analysis is in the appendix due to space limitation 2 All proofs are in the appendix due to space limitation.s to be one of two cases: E or E − {e ij}. Both cases are valid although they have different edges. Property 3 guarantees the independence of such a set s, meaning that the vertices within s have no connections with other vertices outside s ∪ {v i, v j}. As there might be multiple valid closed sets between v i and v j, which corresponds to the Branched Closed Set in Definition 5, we denote the largest closed set between v i and v j as s ij and denote smaller closed set with an extra superscript, such as s 1 ij. Definition 3. Splitting Vertex: A vertex v t ∈ s ij is a splitting vertex of s ij if and only if s it exists, s tj exists and s ij = s it ∪ s tj ∪ {v t} and s it ∩ s tj = ∅ Definition 4. Splittable Closed Set (Type 1): closed set with at least 1 splitting vertex. The definition of Splitting Vertex is to describe whether a closed set can be divided into two linearly arranged closed set. A closed set is splittable if it has at least 1 splitting vertex and is defined as Closed Set Type 1. Definition 5. Branched Closed Set (Type 2): A closed set is branched if it has 0 splitting vertex and can be divided into branches: s ij = s DISPLAYFORM1 Definition 8. Division of Closed Set: For type 1, its division is the linear segments separated by all its splitting vertices; for type 2, its division is all its branches, any of which cannot be divided into more branches; for type 3, its division is its maximal split. For closed set type 1, it can be divided into linearly arranged segments. For closed set type 2, it can be divided into branches. So here we investigate the division of closed set type 3. As we don't want trivial division, for example, division that is formed by every edge in the closed set, we define Maximal Split to describe the split such that each member of the split is as large as possible. An example of maximal split is shown in FIG4. In the definition of maximal split, the term maximal is implied by saying that any subset of this split cannot be combined into a single closed set. If it can, then the maximal split will be formed by this larger closed set and all the rest of the previous split. For closed set type 3, we use its maximal split as its division. Definition 9. Division Tree: Division tree is a representation of a computation graph, where the root node is the whole computation graph, the leaf nodes are all the single tensors in the computation graph, and for a non-leaf node, its children is the members of its division. With the division of 3 types of closed sets, the computation graph can be reorganized into a division tree (Figure 5) where a non-leaf node would be a closed set and its children would be its corresponding division. The root node is the whole computation graph, the largest closed set, and the leaf nodes would be single tensors in the computation graph. With division tree, we can apply divide-and-conquer to search for optimal solution. Figure 5: In this tree, the root node is the whole computation graph. All the leaf nodes are single tensors. Every other node except root and leaves is a member of the division of its parent. Theorem 1. The division tree of a computation graph is unique and complete. The uniqueness of the division tree indicates that the optimal solution of the division tree would also be the optimal solution of the whole computation graph. The completeness indicates that the division tree has included all the possible members of solution and represents the whole search space for the optimal solution. Theorem 1 is proved in the appendix. We search optimal solutions for ACGs by solving several sub-problems using Algorithm 2-4 respectively. Based on these components, we present our final solver as Algorithm 5.Algorithm 2 judges whether a vertex is a splitting vertex of a closed set. This algorithm mainly follows the Definition 3 and uses vertex set to check the property of a splitting vertex. With this algorithm, we can judge whether a closed set is type 1 and get its division if it is. Suppose there are N vertices in s ij, the time complexity of Algorithm 2 is O(N 2). Algorithm 3 examines whether a closed set is branched. It uses a growing algorithm to check whether an independent subpart of this closed set can form a closed set. If a non-trivial closed set s ij has an edge from v i to v j, then it's branched because this edge itself can be treated as a closed set. Combined with Algorithm 2, we can know the type of a closed set and get its division if it's type 2. Suppose there are N vertices in s ij, the time complexity of Algorithm 3 is O(N 2).Algorithm 4 addresses the problem of finding the maximal split, the division of a closed set type 3 s ij. First get all the possible closed sets within s ij and use a property of maximal split to judge whether this closed set is a member of the maximal split. The property is: there cannot exist another closed set s ab s ij but contains any member of this maximal split. This property is proved in Lemma 6 of the appendix. Suppose there are N vertices in s ij, the time complexity of Algorithm 4 is O(N 4).Algorithm 5 is the solver for ACGs. First, the division tree of the computation graph is built. Similar to the linear solver, a max term list is formed by the cost of all the possible closed sets for traverse. Given a max term, we propose a greedy idea: for a closed set, never expand it unless the its cost exceed the max term. In other word, if the max term doesn't allow a leap over this closed set, we expand it, otherwise, do not expand it. Because once expanded, some cost of other vertices inside this closed set might be introduced, and the cost will never be smaller than unexpanded. If some children of the closed set type 1 are expanded, the rest reforms a few linear segments and still can be solved by the linear solver. If some children of the closed set type 2 or 3 are expanded, the other Initialize a vertex set s = {v k}. v k ∈ s ij is a randomly chosen vertex. while True do 7:For any v t ∈ s ij, v t ∈ s that has connection to any v k ∈ s, add v t to s. if No more vertex can be added to s then 9:Break 10:if s = {v ∈ s ij} then Return false Algorithm 4 Find the maximal split of a non-branched s ij with 0 splitting vertex DISPLAYFORM0 For all the vertices {v} that have paths from v k and have paths to v t. if ∃v 2 ∈ {v} and v 2 = v k, v t, v 2 has connection to a v 1 ∈ {v} then 4:Form a closed set s kt with all these vertices. 5: for each formed closed set s kt do If there doesn't exist a s ab such that s kt s ab s ij, put s kt into the maximal split. For all the children that have cost larger than current max term. Expand them and solve the next level. All the expanded children have separated the current closed set to linear segments. Solve all the linear segments with current max term. For all the children that have cost larger than current max term. Expand them and solve the next level. All the other members remain unexpanded. Summarize the total loss, save the current solution if it's better. We evaluated Re-forwarding on two main groups of neural networks networks with linear structures, such as Alexnet BID13 ) and vgg series BID22 ). networks with non-linear structures, such as Resnet series BID10 ), Densenet series BID11 ) and Inception net BID24 ). For each network in TAB4, an computation graph is built such that every vertex is a Float32 tensor, every edge is an operation, and the memory cost of a vertex is its tensor size (measured in MB). We compared Re-forwarding with Chen ) and the regular training approach. Note that only worked on linear computation graphs. To compare with ) on non-linear networks, we manually re-organized all the non-linear computation graphs into linear computation graphs with their splitting vertices, and fed them to (see TAB4 (MB)"). Our Re-forwarding approach directly works on arbitrary computation graphs. We have also included a customized network ("CustomNet"), on which even the manual version of Chen's approach is not applicable. Our approach directly works on all networks. The computation graph of this network is visualized in the appendix. All experiments were conducted in Pytorch. [BatchSize, 3, 224, 224]. We also measure the training time (time of 1 training iteration) for the regular approach and our approach. Each time is measured as the average of 20 iterations. Our approach has the same training time as Chen's approach and its manual version, see "Space Efficient Training Time" in TAB4. Table. 1 shows that Re-forwarding cuts down huge amount of memory from the regular approach at reasonable time overheads: 26% space off and 40% time overhead for Alexnet, around 40% space off and 40% time overhead for Vgg series. For Resnet series, the deeper network, the more memory was cut down. On the deepest Resnet152, 80% space off was achieved with only 39% time overhead. For Densenet series, more than 80% space off was achieved with around 40% time overhead. Notice that, only works on linear networks. Its on non-linear networks were manually synthesized. Re-forwarding directly works on non-linear networks and constantly outperformed and its "manual" version. This supports our claim that Re-forwarding is optimal. Re-forwarding is a fundamental approach that explores trade-off between memory and computation power of GPUs. By saving tensors at a subset of layers during forward, and conducting extra local forwards for backward, Re-forwarding makes it possible to train very heavy DNNs with finite GPU memory. To our knowledge, our theoretical and algorithmic are the first top-down work that achieve an optimal memory solution for arbitrary computation graphs in DNNs. Re-forwarding can be further embedded and optimized with any low-level techniques such as distributed computing, GPU/CPU swapping, computation graph optimization and liveness analysis. Same on v q, v q must be v j or v t. As s ⊂ [s ij), ∀v 1 ∈ s, v 1 has no edge with v 2 ∈ [s ij). As s kj is close, ∀v 1 ∈ s, v 1 has no edge with v 2 ∈ s kj. ∀v 1 ∈ s, v 1 can only have edge with v 2 ∈ [s]. Thus the independence of s is guaranteed. Therefore, s is closed set, v k is the splitting vertex of s ij. DISPLAYFORM0 Same on v j, v j is the splitting vertex of s kt Lemma 4. If s ij has n splitting vertices {v 1, v 2, ..., v n}, then s ij = s i1 ∪ s 12 ∪... ∪ s nj ∪ {v 1, v 2, ..., v n} Proof. If n = 2, the splitting vertices are DISPLAYFORM1 According to Lemma 3, v 1 is splitting vertex of s i2 and v 2 is splitting vertex of s 1j. Therefore, DISPLAYFORM2 For n > 2, the lemma can be proved by repetitively using the in n = 2. Lemma 6. Any member of a maximal split can not be the subset of another closed set s s ij.Proof. Suppose the source vertex of s is v 1 and target vertex is v 2, a member s xy of the maximal split is inside s. Suppose a member s ab of the maximal split has its source vertex v a inside s and target vertex v b outside s. Then the boundary vertex (the vertex that has edges to the non-overlapping parts of both sets) must be v 2, otherwise the independence of s will be violated. Notice that v 2 is inside s ab and the independence of s ab needs to be guaranteed, for ∀v p ∈ s, v p / ∈ s ∩ s ab, v q ∈ s ∩ s ab, v p has no edge with v q. Therefore, v a is a splitting vertex of s. Similarly, if s ba has its target vertex v a inside s and source vertex v b outside s, the boundary vertex must be v 1 and v a is a splitting vertex of s. For the closed set s, from the discussion above, we know that there are at most 2 members of the maximal split that can overlap with s. Other members must be either completely inside s or completely outside s. Let's discuss the number of members that overlaps with s. If there are 0 member that overlaps with s, s is the union of a subset of members of the maximal split, which violates the definition of maximal split. If there is 1 member that overlaps with s, suppose the corresponding splitting vertex is v b, and the boundary vertex is actually v 2. Then s 1b is a closed set containing s xy and corresponds to the situation of 0 member overlapping. s 1b is the union of a subset of members of the maximal split, and violates the definition of maximal split. If there are 2 members that overlaps with s, suppose they generate two different splitting vertex v a and v b. Then s ab is a closed set containing s xy and corresponds to the situation of 0 member overlapping. s ab is the union of a subset of members of the maximal split, and violates the definition of maximal split. If they generate the same splitting vertex v b, from lemma 5, v b is also the endpoint vertex of at least 1 other member s ab which has to be inside s. Suppose the two overlapping members are s cb that contains v 1, and s bd that contains v 2. As the source vertex of s, v 1 has path to v b and v 1 has path to v a, which implies v b has path to v a. As the target vertex of s, v 2 has path from v b and v 2 has path from v a, which implies v b has path from v a. This conflicts with the fact that s is acyclic. Therefore, this case is not possible. Therefore, this lemma is proved. Lemma 7. If non-branched s ij has at least 1 vertex but has 0 splitting vertex, then its maximal split has length > 2 Proof. As s ij is not branched, the members of its maximal split cannot have the starting vertex as v i and the ending vertex as v j at the same time. If s ij has at least 1 vertex, and its maximal split has length 2, then its maximal split must be {[s ik], [s kj]}, and v k will be the splitting vertex of s ij, which violates that s ij has no splitting vertex. If s ij has at least 1 vertex without splitting vertex, it has at least 2 edges and cannot have a trivial length 1 maximal split. Therefore, its maximal split has length > 2 To prove this uniqueness, we simply discuss the division uniqueness of closed set type 1, 2 and 3. Proof. By the definition of this division and Lemma 4, the uniqueness of the division is equivalent to the uniqueness of the splitting vertex set of a closed set type 1. The splitting vertex set is obviously unique. Proof. If there exists another division, there must be a branch member s Proof. As the closed set in the division tree has at least 1 vertex, with Lemma 7, we know that the division, i.e. maximal split of a closed set type 3 s ij within the division tree will have length > 2. Denote this maximal split as {[s pq]}, we only need to prove this maximal split is unique. In all the cases, there cannot exist another different maximal split. Therefore, the maximal split is unique. Similar with the uniqueness, the completeness of division tree is equivalent to the completeness of the division of a closed set. To prove this completeness, we simply discuss the division completeness of closed set type 1, 2 and 3.An equivalent statement of the division completeness is: there doesn't exist a closed set whose head is in one member of the division and whose tail is in another member of the division. Proof. Suppose there exists a closed set s whose head v p is in one member s 1 and whose tail v q is in another member s 2.If v p is not an endpoint of s 1, then according to Lemma 3, v p is also a splitting vertex in s 1 and can break s 1 into smaller segments, which makes v p also the splitting vertex of the whole closed set. However, v p is not the splitting vertex of the whole closed set s ij. This also applies to v q. Therefore, the division of closed set type 1 is complete. Proof. Suppose there exists a closed set s whose head v p is in one member s 1 and whose tail v q is in another member s 2. Same with closed set type 2, the boundary vertex v has to be the endpoint vertex of s 1 or the independence of s 1 will be violated. According to Lemma 5, v is the endpoint vertex of at least 3 members, meaning that v will at least have 1 connection with another closed set s 3. To maintain the independence of s, s has to include s 3 as well. However, s 3 also has its endpoints. This will propagate until s becomes the whole closed set. Therefore, there cannot exist such a closed set s. The division of closed set type 3 is complete. DISPLAYFORM0. These steps will cost O(N 2). Then we traverse each (v i, v j) pair to form the edges of the accessibility graph, which also cost O(N 2). Solving the shortest path problem in accessibility graph will also cost O(N 2) as the accessibility graph has N vertices. Therefore, the overall time complexity of Algorithm 1 would be O(N 4).The space complexity would be O(N 2) for the table of l(v i, v j) and the accessibility graph itself. Suppose there are N vertices in the closed set s ij. In step 1, getting {v in} and {v out} will cost O(N) time for traversing the ancestors and descendents of v t. In our implementation, an array a of length N is used to represent {v in} and {v out}: a i = 1 indicates v i ∈ {v in}, a i = 2 indicates v i ∈ {v out} and a i = 0 indicates v i / ∈ {v in} ∪ {v out}. Then the union check and intersection check in step 2 can be done in O(N). The connection check in step 2 traverses the edges and costs O(N 2). Other steps are O. Therefore, the overall time complexity of Algorithm 2 would be O(N 2).The space complexity would be O(N) for the array to represent {v in} and {v out}. Suppose there are N vertices in the closed set s ij. The most time consuming part will be from step 5 to step 13. Other steps are O. In step 5 to step 13, every edge between two vertices in s ij is at most visited once and there are O(N 2) edges. Therefore, the overall time complexity of Algorithm 3 would be O(N 2).In our implementation, an array of length N is used to represent the vertex set s = {v k}. Therefore, the space complexity would be O(N). Suppose there are N vertices in the closed set s ij and there are O(N 2) vertex pairs. For each vertex pair, the connection check in step 2-4 will cost O(N 2), similar to the connection check in Algorithm 2. Thus step 1-4 will cost O(N 4). In our implementation, for each vertex in the closed set s ij, we select the largest formed closed set s kt that contains this vertex. The closed set number is then reduced to O(N) and step 5-6 can be done in O(N 3). Therefore, the overall time complexity of Algorithm 4 would be O(N 4)As O(N 2) closed sets can be formed in step 1-4 and each closed set is a smaller DAG with O(N) vertices and cost O(N 2) space, the space complexity would be O(N 4) for all these closed sets. B.5 ALGORITHM 5Step 1 is similar to step 1-4 in Algorithm 4 with s ij being the whole computation graph. Therefore, the overall time complexity for step 1 is O(N 4).In step 2, the complexity of building division tree is related to the complexity of getting the division of a closed set. For closed set type 1, Algorithm 2 is called for each vertex to get all splitting vertices. Thus getting the division of closed set type 1 cost O(N 3) time. For closed set type 2, Algorithm 3 is used to solve for its division and costs O(N 2) time. For type 3, Algorithm 4 is called to solve for its division. Notice that we have already stored all possible closed sets in step 1, step 1-4 in Algorithm 4 can be skipped and thus the time complexity of getting the division of closed set type 3 is reduced to O(N 3). Therefore, getting the division of an arbitrary closed set costs O(N 3) time. In depth i of the division tree, suppose there are k closed sets, and the number of vertices of jth closed sets is a j. To build depth i + 1 of the division tree, we need to get the division of all these closed sets, which will cost j O(a DISPLAYFORM0 As the depth of division tree is at most N, the overall time complexity of step 2 would be O(N 4).For step 3-10, if the computation graph is linear, the ACG solver will reduce to LCG solver and has complexity O(N 4). If the computation graph is non-linear, the length of {m} would be O(N 2) for there are O(N 2) vertex pair. For a max term m, from step 4-10, the actual time costing part will be step 6 which calls the LCG solver, and other steps would be O. Suppose the LCG solver is called k times, solving problems of a 1, a 2,..., a k vertices. The total complexity of this would be O(a Step 1 would cost O(N 4) space to store all the possible closed sets. Step 2 would cost O(N 2) space for the division tree. Step 3-10 would cost O(N 2) space for calling LCG solver. In , the overall time complexity of Algorithm 5 is O(N 4) and the overall space complexity of Algorithm 5 is O(N 4). C RUNTIME AND THEORETICAL ANALYSISThe number of vertices in the computation graph and the runtime of ACG Solver (Algorithm 5) for each network are listed in TAB10. All the runtimes were measured on a single core of CPU i7-8700.Notice that the runtime is measured on only 1 cpu core, it can be massively reduced by parallelization on multiple cpu cores. The runtime can also be further reduced through a better implementation as our implementation is a prototype. Although it might be concerning that the runtime is too much for some deep networks, it is still relatively small compared to training processes which might cost days or even weeks. More importantly, solving the optimal solution for a network is an one-time effort. The optimal solutions for all popular networks will be released online for people to use without taking the time to run ACG solver. To see how well the reality matches with the theory, we also compare the measured memory cut off and theoretical memory cut off (given by Algorithm 5) in TAB10. Observe that all the measured memory cut off are slightly lower than theoretical memory cut off. This is because, in implementation, we assume that the whole input tensors of each operation are always stored for backward. In reality, some operations only need to store small tensors for backward. For example, batchnormalization only needs a few statistics for backward and doesn't need the whole input tensor. We visualize the computation graph of Alexnet, vgg11, vgg13, vgg16, vgg19 and CustomNet and the solution of our approach (in green) and the solution of Chen's approach (in red). In the computation graphs, the cost of each vertex and the actual operation of each edge are also marked. The cost of each vertex is the size of this tensor during forward given the input as ( for inception v3). For example, in Alexnet, the input is and thus the source vertex has the cost 150528 = 1 × 3 × 224 × 224. After 2D convolution and relu, the tensor becomes and thus the second vertex has the cost 193600 = 1×64×55×55.

represent the tldr for finding the corresponding scientific article

This paper proposes fundamental theory and optimal algorithms for DNN training, which reduce up to 80% of training memory for popular DNNs.

Represent the short summary for finding the corresponding scientific article (long)

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.