Monocular Depth Parameterizing Networks

Monocular depth estimation is a highly challenging problem that is often addressed with deep neural networks. While these are able to use recognition of image features to predict reasonably looking depth maps the result often has low metric accuracy. In contrast traditional stereo methods using multiple cameras provide highly accurate estimation when pixel matching is possible. In this work we propose to combine the two approaches leveraging their respective strengths. For this purpose we propose a network structure that given an image provides a parameterization of a set of depth maps with feasible shapes. Optimizing over the parameterization then allows us to search the shapes for a photo consistent solution with respect to other images. This allows us to enforce geometric properties that are difficult to observe in single image as well as relaxes the learning problem allowing us to use relatively small networks. Our experimental evaluation shows that our method generates more accurate depth maps and generalizes better than competing state-of-the-art approaches.


Introduction
Dense depth or disparity estimation is a classical problem in computer vision [1,33,37,2]. Traditional methods use stereo (or multi-camera) setups and attempt to match every pixel in the reference image to a corresponding pixel in a neighbouring image using appearance cues. While the accuracy of the recovered depth is often very high for correctly matched pixels ambiguous texture can degrade the matching and often leads to a noisy depth map.
To stabilize the result a popular approach is to add geometric regularization terms such as derivative [4,22,13] or curvature [26,40,31] penalties. These can be realized as low order potentials in a conditional random field and efficient inference can be performed with move-making Image Ours DeepFactors [7] MegaDepth [27] Figure 1: Our monocular depth parameterizing network produces high quality and detailed reconstructions. We achieve realistic-looking depth maps and reconstructions for a large variety of scenes. [4,25,38] or message passing algorithms [21,13]. While this kind of prior can drastically improve the estimation in ambiguous image regions, they lack any ability to recognize complex geometries and are basically limited to encouraging piecewise planar or smooth surfaces. A more recent approach is to use neural networks, trained on examples, to directly infer depth or dense matching [29,43,39,5,36,44,5,19,41]. These models can potentially work better in low textured regions since they can learn high level recognition of geometric structures on a larger scale than what traditional methods work on. On the other hand they can generalize poorly and require lots of training data [39,43,42]. An extreme case of this is monocular depth estimation where a neural network is used to estimate depth from a single image [28,9,6,24,9]. These networks typically require a huge amount of training data and may generalize poorly [14]. In addition, while these achieve meaningful results with plausible object shapes, the resulting depth maps are often inaccurate because of the ambiguous nature of the problem [30,18], see Figure 1.
In this context it is interesting to think about what variations in the depth map can be explained by the image features. As a simplified example consider an orthographic camera viewing a known object. It is well known that the absolute distance between the object and the camera cannot be determined from image data [17]. In a sense there is a loss of information due to the projection from 3D to 2D not being injective (one-to-one). Therefore we cannot hope to invert it without supplying additional information such as an extra view. Hence a network that uses ground truth data where the absolute distance varies will not be able to predict such variations regardless of the amount of training data it is being fed. There may be other factors than the image formation model that limits what information can be extracted from a single image. In practice the size and architecture of the network determines what features can be used for prediction. We refer to information about the depth map that cannot be extracted from the features as the knowledge-gap.
In this paper we aim to resolve ambiguities of the monocular depth map estimation by combining learning models with traditional geometric formulations. Our goal is to use a neural network to extract a low dimensional shape parameterization from a single image that is flexible enough to allow depth fitting using traditional stereo ques such as photo consistency. To achieve this we add an additional input, which models the knowledge-gap, to a U-net architecture designed to complement the model with the information that is not directly observed in the image. The result is a network that takes as input an image and gives a class of possible depth-maps parametrized by the hidden variables. We emphasize that in contrast to current monocular depth approaches [28,9,6,24] we are not aiming for a network that predicts absolute pixel depths but rather one that extracts reliable shape information that reduces the dimensionality of a traditional stereo formulation. This allows us to use smaller networks, lessening the need for training data.
Our network can be trained without requiring any knowledge of the shape variables, using either images with accompanying depth maps or self-supervised through geometric and photo-metric consistency losses. We show in our experimental evaluation that our approach gives more accurate solutions than what is normally achieved with learning based depth prediction methods.
In summary our main contributions are: • We present a new network for extracting low dimensional shape priors from monocular images.
• We show that explicitly modelling the knowledge gap enables training of a flexible model that can be combined with traditional stereo methods.
• We show that the model can be trained in a selfsupervised end-to-end fashion.
• Our empirical evaluation demonstrates that our approach achieves much more accurate depth estimates than stat-of-the-art learning based depth prediction.

Related Articles
Until recently, the best practice has been to train monocular depth prediction networks in a supervised manner [9,7,3]. However, these require large amounts of training data and for this reason the use of multiple view geometry to aid weakly or self-supervised training, has become popular [14,28,15,16]. For example, [28] produces high quality depth maps from Internet Photos using SfM and multiview stereo (MVS) methods to achieve strong generalization. A loss function measuring photo-consistency of the source image warped to a neighboring image according to its predicting depth map and the camera geometry is used in [14]. Similarly [15] uses epipolar geometry to define a suitable loss function. In [35] the training of a monocular depth network is coupled with a SLAM system to form a self-improving loop. While these methods use geometry to simplify/improve training, the end goal is still a monocular network able to predict depth from the image alone. Such approaches do not help in disambiguating the problem when there is a loss of information due to projection.
To be able to disambiguate 3D structure from a single image, a so called depth attention volume is used in [18] to modify the bottleneck features of an encoder-decoder type architecture to favour planar regions. Another approach is presented in [30], where a histogram of pixel depths is supplied to the network in addition to the image.
The work that is perhaps most similar to our is [7,3] where an image dependent latent scene representation is leaned in a supervised manor, using a depth-map autoencoder coupled with a U-Net for convolutional image feature extraction. At testing, the depth map encoder is removed resulting in a network that takes a code and outputs a depth map conditioned on the image. They observe that this allows the code to represent local geometry that cannot be directly observed in the image. A search over both code and camera pose is performed during testing. The presented network is however complex and it is trained with ground truth RGB-D images. The depth decoder is linear suggesting that the prediction might be highly dependent on image features (see Figure 8 of [3]). Our latent variable model is not trained to recreate a depth map through an autoencoder but to complement image features with information needed to predict the depth map. We will show that in our approach we can achieve better depth accuracy using a much simpler network, training setup and using only self-supervision, see Section 5.

Parameterizing Feasible Shapes
While it may be hard for a network to learn exact monocular depth prediction we argue that it is of interest to be able to extract what the network is able to learn, even if this is not sufficient to pinpoint a particular depth map with certainty from a single image. Instead we design our network so that it can be complemented with additional information to resolve ambiguities.
For this purpose we use a network reminiscent of a U-net [32], with a contracting and an expanding part connected with skip connections for accurate localization, but add an additional input that provides the expansion part with additional information that allows it disambiguate. This essentially results in a low dimensional shape model which, at testing time, we search for a photo-consistent solution. Before we present our network we illustrate the basic principle using a simplified example.

Overall Principle
We consider a matrix formulation with a contractive part, represented by a matrix F , which takes an input in the form a row vector x and outputs a feature vector xF . Here xF represents features extracted from the image. To this we concatenate a vector z of latent variables that complements the feature information into a larger feature vector xF z and feed the result to an expanding map G giving the output Here G x and G z are the rows of G corresponding to the x and z variables respectively. Given ground truth data x i and y i a feasible loss function would be where X, Y and Z contain the row stacked vectors x i , y i and z i respectively. Note that the network is allowed to freely determine Z during training. First consider the case when Z = 0 and suppose that Y cannot be fully recreated using the term XF G x . Since G x can be selected freely during training this occurs when the feature matrix XF has a lower rank than Y , which is either due to an information loss between Y and X or F being to small to extract all relevant information. For simplicity we restrict ourselves to the former situation, the latter is similar. When dividing the matrix Y into two components, PY which is the projection of the columns of Y onto the column space of X and P ⊥ Y = Y − PY , the training objective reduces to The first term can be made to vanish by any choice of F and G such that F G x = X † PY , where X † is the pseudoinverse of X. The second term is not affected by the choice of F and G x and represents the variations in the y i variables that cannot be explained by the inputs x i .
For comparison consider what happens when we allow usage of Z. If the columns of Z are chosen to be perpendicular to the column space of X the objective (2) becomes Hence after minimization Z will contain the information missing in the feature space needed to explain the ground truth variations and G z will have learned how to decode this information.
We give more explanations and details on this matrix model and study its properties in the orthographic projection example in the appendix.

Network Structure
The network we propose takes an image I, a set of normalized pixel coordinates Q and a vector z and outputs a transformed depth map ρ. The basic construction, which is shown in Figure 2 consists of a U-net type architecture with the addition of the extra inputs z. The contracting component computes features at different resolutions using convolutional layers. They are supplied to the expanding part at corresponding levels. The additional input z takes a vector of 192 values (corresponding to a grid of size 12 × 16) and complements the features with information needed for prediction. The input image x and the output map ρ are both of size 192 × 256. Hence given an input image our network provides a map from a 192-dimensional parameter space into a 49152-dimensional space of depth maps. Compared to a traditional stereo matching formulation optimizing over all pixels we thus get a significant reduction of the search space. We also remark that due to its simple structure, the number of parameters in our network is relatively small, all in all roughly 400000.
The network can be thought of as a functional ρ = N (I, Q, z|W ) which takes an image I, normalized pixel coordinates Q, the latent variable z and outputs a transformed depth map ρ with values in [0, 1]. We include the normalized pixel coordinates as input to allow the network to account for different camera calibrations. Similar to [3], the output ρ(p) at a pixel p from the network relates to depth D(p) as where α is the mean depth. This will map the network output from [0, 1] to [0, ∞]. The depth map D(p) can then be recovered using By d we denote the functional that takes as input ρ = N (I, Z, z|W ) and α and gives the depth map D as output, that is, D = d(ρ, α). In this paper we train the network using two different approaches. In the first we use image and ground-truth depth pairs to do supervised learning, see Section 3. In the second we assume that no ground-truth depth is available and instead train using only photometric and geometric errors between co-visible images, see Section 4. The latter results in self-supervised learning where we learn depth indirectly from the images. This has the benefit that any partially overlapping collection of images can be used to train the network. The training therefore becomes more scalable since no hard to obtain depth ground truth is needed. In addition it makes it easy to calibrate for a new camera since we only need images from this camera to update the network.

Testing and Optimization
As previously discussed the output of our network can be seen as a family of feasible depth maps parameterized by the latent variables z. In the testing phase we therefore need to perform an optimization over z to find the depth map with the best photometric fit. For this purpose we define a set of relative transformations T ij that transforms 3D points in camera coordinate system i to j. Given an image I i and associated depth map D i and normalized image coordinates Q i a pixel p in image i is the projection of the 3D point This point is transformed to pixel q ij (p) = π (T ij X(p)) , in image j, where π is the projection mapping (division with the 3rd coordinate). For a photoconsistent solution we therefore want to be small. Here M ij is a masking matrix that removes pixels that are not visible (we discuss the generation of this mask in detail below), and · δ is the Huber norm calculated pixel-wise and λ p = 100 is a scalar weight factor. Since we are dealing with discrete images the composition I j • q ij involves re-sampling image j at the points q ij (p) using bilinear interpolation. Note also that the above loss depends on z i through q ij . We also want the depth maps from different cameras to give consistent 3D point clouds. The depth D ij (p) of the point X i (p) in camera j is given by the third coordinate of T ij X i (p). We therefore add the loss to encourage consistent depth maps. Here λ d = 10 is a weight factor and α j is the mean depth of D j . Division with α j makes the above term scale invariant. We conclude this section by discussing the masking operators M ij . When transforming one view into another there are inevitably going to be some pixels that cannot be transferred due to limited field of view, chirality and occlusion. Pixels that end up outside the image bounds or have negative depth are easily removed based on their location in the camera coordinate system. However to determine occlusion we also need to consider the current estimate of the depth map. For this purpose we compare the predicted depth D j with the projected depth D ij . If there is no occlusion we assume that the error ∆ ij (p) = D j •q ij (p)−D ij (p) is approximately Gaussian distributed (i.i.d over the pixels). We use median absolute deviation to get a robust estimate of the mean and standard deviation of the depth error distribution. We classify pixel p as occluded if the corresponding depth error ∆ ij (p) satisfies where τ is 4.44 and the median and median absolute deviation is taken over all pixels in the image. In addition to invalid and occluded points, we also mask out points with a shallow viewing angle. These occur on surfaces that are very slanted compared to the viewing direction and on the boundary of depth discontinuities. In both situations the pixel and depth values at these points may not be informative or may even be detrimental to the solution. They are therefor masked out. These are found by estimating the normal of the surface point and calculating the angle between normal and viewing direction. If the angle is above a threshold of 85 • , the point is masked out.
For optimizing the above model we use an AdaMax optimizer [20] to minimize the loss. This is iterated until the relative difference between the current and previous loss falls below a threshold. 1

Mono
Photometric Input   Table 2: Results comparing ours (Photometric with z variables) using 2 co-visible cameras with stereo method in [31].

Evaluation with Supervised Learning
In this section we will evaluate the properties of our network architecture. For simplicity and for the purpose of fair comparisons we will limit ourselves to a relatively standard supervised learning approach (our self supervised approach is presented in Section 4). Given pairs of images I i and ground truth depths D gt i , we form the loss function Here and α i is calculated as the mean of the ground truth depth. The L 2 loss over z and W acts as a regularizer.
The loss is minimised batch-wise with respect to the network weights W and z variables, using an AdaMax optimizer with a learning rate of 10 −3 . We train on a subset of the Scannet training datasets [8]. The subset was drawn randomly from every second available scene/folder. In order not to obtain prohibitive training times we drew in total 150000 samples from these scenes/folders.

Ablation Studies
Our approach for depth map estimation relies on both the network identifying a set of feasible depth maps from the image and the use of traditional stereo to pinpoint the desired solution. In the following two experiments we compare out joint model with the performance of these two components separately. The tests are run on Scannet scenes [8] and the abbreviated TUM scenes seq1=freiburg3 nostructure texture near withloop, seq2=freiburg3 structure texture far validation and seq3= freiburg3 long office household [34].
Effects of Latent Variables. To investigate the impact of the additional latent variables we train two networks on the same Scannet dataset scenes. The networks have the structure as described in Figure 2, with the exception that in the second network, the z input has been removed which turns it into a regular monocular depth predictor. These networks were then evaluated on a subset of the test datasets of Scannet. For the depth parameterizing network, co-visible images were extracted (see appendix for a description), and the process described in Section 2.3 to estimate the depth maps. Table 1 shows metrics for the two networks. The shape parameterizing network is referred to as photometric and the monocluar depth predictor network is referred to as monodepth. Further, we show some qualitative results of the two model performances in Figure 3. In all tested cases there is a substantial improvement when using photo- metric information. In addition we note that the monocular network performs worse on the TUM sequences than the Scannet sequences. This is an effect of training on Scannet (albeit on different scenes) and poor generalization properties. It is clear from the results that the ability to use photometric information makes our approach generalize much better to the TUM sequences. Figure 4 displays the impact of the components in z on the estimated depth. From the figure we can see that the components have a local impact, with a structure and size that depends on the image. See appendix for more examples. Stereo Comparison. To evaluate the contribution of the learning component we next compare our approach with a publicly available stereo method [31] relying on photometric information. This method uses rectified cameras and therefore we limit our method to use only pairs of images for a fair comparison. Due to the arbitrary camera motion of the dataset, only a subset of the the pairs are suitable for rectification. We therefore filter out unsuitable pairs that have few remaining valid pixel after rectification. The results are shown in Table 2. Note that our formulation generally achieves better results than the stereo method despite only optimizing over 192 variables. This suggests that the shape parameterization that is provided by our network is able to reduce the dimensionallity of the search-space without any significant loss of accuracy. Further, the stereo method tend to produce unrealistic representations of shape due to the fitted tangent planes, see appendix for more details.

Multiple View Stereo for Self-Supervision
In this section we present the self-supervised training. Using this approach we can train the network end-to-end without ground truth depth maps, in exactly the way it is expected to be used later. Our approach trains the network to learn the mapping ρ = N (I, Q, z) that maximizes Here D i = d(N (I i , Q i , z i |W), α i ), I −i is the set of all images co-visible with I i except I i itself and D −i defined similarly. Notice here that we do not assume that we know the ground truth value of D −i . Rather these are current estimates of the depth in the co-visible views. When we update the weights W and the latent variables z i , we always update co-visible images jointly. We assume that the photometric and depth variables are conditionally independent and can be factorized as Under the assumption that the distributions belong to the exponential family, we can define the negative log-likelihood of the photometric and depth distribution as in Equations (8) and (9). The normalizing constants do not depend on the network weights W and z i and are therefor neglected. To this we add the additional loss ρi(p) which only holds if L αi = 0. The loss will penalize solutions where α i is not the mean of the current depth map. The total loss becomes We minimize this loss batch-wise w.r.t. W, α i and z i , for i = 1, ..., M using an AdaMax optimizer with learning rate of 10 −3 .
When experimentally comparing training with ground truth depth maps from Section 3 with the self supervised approach (using the same number of images from Scannet training dataset) we observe that both methods yield similar results (see appendix for an evaluation). This shows that the latter option is viable and furthermore we could expect to see an improvement if more training data was used as indicated in [14,28,15,16].

Experimental Comparisons
We compare our method with depth predictions from DeepFactors [7], MegaDepth [28] and PSMNet [5]. For each method the predictions are scaled with the median of the fractions between predicted and ground truth depth, more specifically s = Median(D gt /D pred ), for a fair comparison. A similar scale is used in e.g. [45] and [7].
The presented metrics are computed on different subsets of the dataset for the different methods. Scannet scenes are evaluated as follows; we run DeepFactors on the full dataset and calculate the metrics on the keyframes it is preconfigured to generate (to not affect the performance). MegaDepth and our method consider a larger set of the dataset. How this is selected from the testset is described in  Table 3: Quantitative results. Evaluation of our system on validation scenes from Scannet [8] and TUM RGB-D [34]. We compare our system against DeepFactors [7], MegaDepth [27] and PSMNet [5]. All metrics are presented as averages of all samples.
the appendix. PSMNet requires rectified images, and thus it is evaluated on the set of meaningful rectifications of the dataset.
On the TUM scenes our, MegaDepth and DeepFactors are all evaluated on the keyframes selected by DeepFactors for a fair comparison. For PSMNet we evaluate on the subset of images with meaningful rectifications. We trained our network on a subset of Scannet training data (v2) along with some scenes from [12] , using the loss function given in (15). Full description of training data is given in appendix. The methods are subsequently evaluated on Scannet testset (v1 and v2) and TUM RGB-D datasets. The TUM RGB-D depths and images are associated as in [34]. We evaluate the predictions on several metrics from previous work, as RMSE, Absolute Relative difference, Squared relative difference, scale-invariant RMSE and Accuracy under a threshold [23] as in [10,11]. The results shown in Table 3 are means over the number of samples. Figure 5 shows a subset of the estimated depth maps. In Figure 6 we also plot these in 3D.  Figure 6: Qualitative results. Examples of reconstructed scenes from datasets TUM/seq2, Scannet scene0707 00, Scannet scene0799 00 and Scannet scene0565 00.

Conclusions
In this work we presented a learning approach for monocular depth estimation that takes ambiguities into account by providing a low dimensional parameterization of a family of feasible depth maps. We have shown that optimizing over this representation using photo-consistency losses yields accurate and realistic geometries. Our experimental results indicate, both qualitative and quantitative, that our approach generalizes better than competing state-of-the-art methods.

A. Matrix Model Continued
In this section we further discuss the matrix model given in Section 2.1 and give a concrete example with orthogonal projection of a scene consisting of three rigidly moving objects. Figure 7 shows some scene examples and the images resulting from the projection.
Suppose that S is a 3 × p matrix containing column vectors representing the coordinates of the p points of one of the scene objects. We rigidly move the object in the xycoordinate plane by computing W = RS + t1, where and 1 is a 1 × p vector of all ones. Note that W is of the same size as S. The row vector y i which represents scene will contain the column-stacked rows w 1 , w 2 , w 3 of W . If s 1 ,s 2 and s 3 are the rows of S we define the block matrix where 0 is a 1 × p row vector of all zeros. We can now write where v contains the transformation parameters Now for multiple transformations we collect their parameters in row vectors v j as in (19), and column-stack them into a matrix V . The object pose in a particular scene is then represented by the corresponding row of the matrix VS. It is clear that the matrixS has rank 7 unless the s 1 ,s 2 ,1 are linearly dependent. On the other hand it is easy to see that the row space of V is spanned by a 5-dimensional basis. Therefore the rank of VS is 5.
For each of the three objects in the scene we now create structure-S 1 ,S 2 ,S 3 and parameter-matrices V 1 , V 2 , V 3 . The block matrix Y = V 1S1 V 2S2 V 3S3 which contains the coordinates from all three objects can then be written The matrix containing the structure is clearly of full rank as long asS 1 ,S 2 andS 3 all have full rank. However V contains three columns with all ones and therefore the rank is reduced by when concatenating V 1 ,V 2 and V 3 . This gives rank(Y ) = 3 · 5 − 2 = 13.
The image matrix X is generated by applying a matrix Π which simply discards columns of Y that correspond to depth (all y-coordinates in our application).
For each object this corresponds to applying a matrix Π such that We can therefore write Note that the columns of V i that corresponds to t 2 completely disappear from the product V iSi Π these can therefore be assumed to be 0. This explains why an objects distance to the camera cannot be recovered from X. With this element set to zero the rank of V i is now 4. The concatenation V 1 V 2 V 3 still contains three columns with all ones. We therefore get rank(V ) = 3 · 4 − 2 = 10, which similar to above can easily be seen to be the rank of X as well.
The final step of generating the images is to add noise to the image coordinates. This results in a full rank matrix where N is a matrix that contains Gaussian elements. In the following sections we will consider the effects of trying to learn this model with and without auxiliary variables.

A.1. Learning Without Z-variabels
Recall the network construction from (1). In this section we consider the case where Z = 0 and apply it to the orthographic projection problem described above. Figure 8 illustrates the mappings and spaces involved in the problem. Each scene is represented by a row vector with 78   elements, corresponding to the coordinates of all the scene points. The values of these 78 variable are however not arbitrary but belongs to 13-dimensional linear subspace that contains all the scenes we can expect encounter. Thus our scene space is a subspace of R 78 . Similarly the images are represented by row vectors of size 52 and the feasible images belong to the 10-dimensional image space. The forward projectionΠ : R 78 → R 52 maps scenes to images. From elementary linear algebra it is clear that it cannot be bijective due to the sizes of the subspaces. (As we saw in the previous section the missing information required to invert the mapping is one absolute depth for each object.) Applying F : R 52 → R 10 to an image vector x gives a representation xF in the feature space. To get back to the scene space we apply G x : R 10 → R 72 to xF giving the point xF G x in the scene space. Note that since the feature space has dimension 10 there is no way we can recover the whole scene space, but we have to settle for a 10 dimensional subspace.
To train the model we generated 50 ground truth scenes Y train and corresponding images X train as described in the previous section. Since the training data has noise we use the loss function The additional matrix X contains a clean estimate of the image subspace spanned by the rows of X test . Note that the first term XF G x − Y train 2 F only depends on the rank of X and not which particular subspace it spans. Therefore the optimal X will be given by where [X] 10 is obtained by thresh holding the singular values of X train . The above loss is therefore equivalent to In Figure 10 we compare a few of the scenes predicted from the test data to the true scene. While the rotational pose of the objects is well predicted their depths are as expected not correct. Since X train is corrupted by noise it is in fact of full rank. Therefore it is possible to try the naive approach of making the loss completely vanish by increasing the size of the feature space to 13. This will however result in overfitting since the 3extra dimensions will depend purely on noise. This is confirmed by the results in presented in Table 4.    Table 4: Sum-of-squares distances between predictions to true scene points, for both training and test data.

A.2. Learning With Z-variables
We now add the shape parameterization to the problem. Rather than predicting a single point in the scene space from an image we are now interested in finding the whole subspace of scenes, parametrized by z, that project to the viewed image. Figure 9 illustrates our approach. In contrast to the previous section we add the latent variabels z to the feature space increasing its dimensionality to 13 making it possible to pinpoint a specify unique point in the scene space that projects to the viewed image.
We consider the objective where rank(X) = 10. Similar to above the first term only depends on the rank of X and not which particular subspace it spans. Therefore optimization over X reduces the problem to We now divide Y train into two components; PY train which is the projection of Y train onto the columnspace of [X] 10 , and P ⊥ Y train which is perpendicular to it. Letting Z have columns that are perpendicular to [X] 10 gives Since [X] 10 spans the columns of PY train it is clear that the first term can be made to vanish by selecting F G x = [X] † 10 PY train . The second term is minimized when ZG z is the best rank 3 approximation of P ⊥ Y train , which can be computed from the SVD of P ⊥ Y train . Figure 11 shows the result of applying the above training approach to the orthographic projection problem. Here we plot the best fit to the true 3D scene that was used to generate the test image. Note that given the image we get different scene predictions when varying z. Here we have optimized over the z to find the scene that is as close to true scene as possible. In the last row of Table 4 we also display the sum-of-squares distance between the predictions and the scenes. It is clear that the introduction of the z-variables gives a shape model that captures all the variations in the training data even if the images themselves are not enough to predict them. In a situation where the true scene shape is unknown the free parameters could for example be fixed using a second camera in a stereo setup.

B. Selecting Co-Visible Images
To be able to train the network or estimate depth maps using self supervision, we need to have sets of visually overlapping images, called co-visible images. These images must have certain characteristics to be informative, such as sufficient overlap, parallax and low redundancy. Redundancy here means images that are captured from approximately the same view points. When using images from structure from motion solutions, as in [12], we assume that all images have been captured with sufficient relative parallax and low redundancy. We therefore simply group images by selecting a reference image and then selecting N − 1 other images that observe the most 3d points in common with the reference image.
The Scannet dataset [8] comprises of sequences of images accompanied with ground truth depth maps and camera poses. This means that we do not have direct knowledge of visual overlap, and since they are captured close in time, the most visually overlapping view will be very redundant, with very low parallax.
To select co-visible images from this dataset, we first find which images have visual overlap. This is done by creating a voxel map, were a camera id is added to a voxel if its un-projected depth map intersects with the voxel. This is done in one pass for all cameras. Next a second pass is done, where each camera fetches the set of cameras that intersected with the same voxels it does. This set becomes the set of overlapping cameras. This allows us to find the overlapping cameras in linear time, rather than cubic time.
In the next step we create the co-visible set for each camera. The set initially contains only the camera itself. We then iteratively add new cameras to the set from the set of overlapping cameras. In each iteration we select the camera that has the largest parallax to all cameras already in the set and with sufficiently high overlap to all of the cameras.

B.1. Evaluation of Training Approach
While self supervision gives access to more training data, the quality of that data may not be as high as ground truth depth maps. In this section we therefore compare our two supervision approaches to ensure that we can achieve similar results.
Using the same datasets as in Section 3, we extract sets of five co-visible images as described in B. The amount of sets is then sub-sampled to 30000 by drawing them at random, yielding in total 150000 images. We do this to have the same amount of training data in both cases. Table 5 shows that with this data, supervised and selfsupervised training yield comparable results, with the supervised training giving slightly better metrics. Still, we are able to achieve very good performance with a the self supervised approach which is scalable and can benefit from the large amount of available structure from motion datasets.

C. Description of training data
Here we present the training data used to train the network for the experimental comparison in Section 5. We used two sources for our training dataset. Firstly, the Scannet training datasets (v2) [8] where every dataset with even number was used, only considering the sequence ending with 00 therein. Secondly, a subset of the structure lower is better higher is better Dataset Method absrel log RMSE sc. inv.
δ < 1.  Table 5: Evaluation of training approach, comparing networks that have been trained supervised and selfsupervised.
from motion datasets presented in [12], with the following names; •

D. Stereo comparison
In this section we present depth maps of the two models compared in Section 3.1 Ablation studies, Stereo comparison. In Figure 12 we show examples of how our model captures the geometrical shapes of the objects more correctly than the stereo method. The stereo method from [31] tend to put the objects into planar shapes.

E. Effects of Latent Variables
In Figure 13 more examples of the impact of the individual z-components are displayed. As stated before, the z-components have a local impact on the depth map, where the shape and size depend on the image.