Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Determinantal Point Processes (DPPs) provide an elegant and versatile way to sample sets of items that balance the point-wise quality with the set-wise diversity of selected items. For this reason, they have gained prominence in many machine learning applications that rely on subset selection. However, sampling from a DPP over a ground set of size N is a costly operation, requiring in general an O(N^3) preprocessing cost and an O(Nk^3) sampling cost for subsets of size k. We approach this problem by introducing DppNets: generative deep models that produce DPP-like samples for arbitrary ground sets. We develop an inhibitive attention mechanism based on transformer networks that captures a notion of dissimilarity between feature vectors. We show theoretically that such an approximation is sensible as it maintains the guarantees of inhibition or dissimilarity that makes DPP so powerful and unique. Empirically, we demonstrate that samples from our model receive high likelihood under the more expensive DPP alternative. Selecting a representative sample of data from a large pool of available candidates is an essential step of a large class of machine learning problems: noteworthy examples include automatic summarization, matrix approximation, and minibatch selection. Such problems require sampling schemes that calibrate the tradeoff between the point-wise quality -e.g. the relevance of a sentence to a document summary -of selected elements and the set-wise diversity 1 of the sampled set as a whole. Determinantal Point Processes (DPPs) are probabilistic models over subsets of a ground set that elegantly model the tradeoff between these often competing notions of quality and diversity. Given a ground set of size N, DPPs allow for O(N 3) sampling over all 2 N possible subsets of elements, assigning to any subset S of a ground set Y of elements the probability DISPLAYFORM0 where L ∈ R N ×N is the DPP kernel and L S = [L ij] i,j∈S denotes the principal submatrix of L indexed by items in S. Intuitively, DPPs measure the volume spanned by the feature embedding of the items in feature space (Figure 1). BID31 to model the distribution of possible states of fermions obeying the Pauli exclusion principle, the properties of DPPs have since then been studied in depth BID19 BID6, see e.g.). As DPPs capture repulsive forces between similar elements, they arise in many natural processes, such as the distribution of non-intersecting random walks BID22, spectra of random matrix ensembles BID37 BID13, and zerocrossings of polynomials with Gaussian coefficients BID20 ). More recently, DPPs have become a prominent tool in machine learning due to their elegance and tractability: recent applications include video recommendation BID10, minibatch selection BID46, and kernel approximation BID28 BID35.However, the O(N 3) sampling cost makes the practical application of DPPs intractable for large datasets, requiring additional work such as subsampling from Y, structured kernels (Gartrell et al., (a) (b) (c) φ i φ j Figure 1: Geometric intuition for DPPs: let φ i, φ j be two feature vectors of Φ such that the DPP kernel verifies L = ΦΦ T; then P L ({i, j}) ∝ Vol(φ i, φ j). Increasing the norm of a vector (quality) or increasing the angle between the vectors (diversity) increases the volume spanned by the vectors BID25, Section 2.2.1).2017; BID34, or approximate sampling methods BID2 BID27 BID0. Nonetheless, even such methods require significant pre-processing time, and scale poorly with the size of the dataset. Furthermore, when dealing with ground sets with variable components, pre-processing costs cannot be amortized, significantly impeding the application of DPPs in practice. These setbacks motivate us to investigate the use of more scalable models to generate high-quality, diverse samples from datasets to obtain highly-scalable methods with flexibility to adapt to constantly changing datasets. Specifically, we use generative deep models to approximate the DPP distribution over a ground set of items with both fixed and variable feature representations. We show that a simple, carefully constructed neural network, DPPNET, can generate DPP-like samples with very little overhead, while maintaining fundamental theoretical properties of DPP measures. Furthermore, we show that DPPNETs can be trivially employed to sample from a conditional DPP (i.e. sampling S such that A ⊆ S is predefined) and for greedy mode approximation. • We introduce DPPNET, a deep network trained to generate DPP-like samples based on static and variable ground sets of possible items.• We derive theoretical conditions under which the DPPNETs inherit the DPP's log-submodularity.• We show empirically that DPPNETs provide an accurate approximation to DPPs and drastically speed up DPP sampling. DPPs belong to the class of Strongly Rayleigh (SR) measures; these measures benefit from the strongest characterization of negative association between similar items; as such, SR measures have benefited from significant interest in the mathematics community BID39 BID5 BID4 BID32 and more recently in machine learning BID3 BID27 BID35. This, combined with their tractability, makes DPPs a particularly attractive tool for subset selection in machine learning, and is one of the key motivations for our work. The application of DPPs to machine learning problems spans fields from document and video summarization BID14 BID9, recommender systems BID47 BID10 and object retrieval BID1 to kernel approximation BID28, neural network pruning BID33, and minibatch selection BID46. BID48 developed DPP priors for encouraging diversity in generative models and BID40 showed that DPPs accurately model inhibition in neural spiking data. In the general case, sampling exactly from a DPP requires an initial eigendecomposition of the kernel matrix L, incurring a O(N 3) cost. In order to avoid this time-consuming step, several approximate sampling methods have been derived; BID0 approximate the DPP kernel during sampling; more recently, by BID2 followed by BID27 showed that DPPs are amenable to efficient MCMC-based sampling methods. return S Exact methods that significantly speed up sampling by leveraging specific structure in the DPP kernel have also been developed BID34 BID12. Of particular interest is the dual sampling method introduced in BID25: if the DPP kernel can be composed as an inner product over a finite basis, i.e. there exists a feature matrix Φ ∈ R N ×D such that the DPP kernel is given by L = ΦΦ, exact sampling can be done in DISPLAYFORM0 However, MCMC sampling requires variable amounts of sampling rounds, which is unfavorable for parallelization; dual DPP sampling requires an explicit feature matrix Φ. Motivated by recent work on modeling set functions with neural networks BID45 BID11, we propose instead to generate approximate samples via a generative network; this allows for simple parallelization while simultaneously benefiting from recent improvements in specialized architectures for neural network models (e.g. parallelized matrix multiplications). We furthermore show that, extending the abilities of dual DPP sampling, neural networks may take as input variable feature matrices Φ and sample from non-linear kernels L. In this section, we build up a framework that allows the O(N 3) computational cost associated with DPP sampling to be addressed via approximate sampling with a neural network. Given a positive semi-definite matrix L ∈ R N ×N, we take P L to represent the distribution modeled by a DPP with kernel L over the power set of DISPLAYFORM0 Although the elegant quality/diversity tradeoff modeled by DPPs is a key reason for their recent success in many different applications, they benefit from other properties that make them particularly well-suited to machine learning problems. We now focus on how these properties can be maintained with a deep generative model with the right architecture. BID7. Although conditioning comes at the cost of an expensive matrix inversion, this property make DPPs well-suited to applications requiring diversity in conditioned sets, such as basket completion for recommender systems. Standard deep generative models such as (Variational) Auto-Encoders BID23 (VAEs) and Generative Adversarial Networks BID15 ) (GANs) would not enable simple conditioning operations during sampling. Instead, we develop a model that given an input set S, returns a prediction vector v ∈ R N such that DISPLAYFORM0 where Y ∼ P L: in other words, v i is the marginal probability of item i being included in the final set, given that S is a subset of the final set. Mathematically, we can compute v i as DISPLAYFORM1 for i ∈ S BID25; for i ∈ S, we simply set v i = 0.With this architecture, we sample a set via Algorithm 1, which allows for trivial basket-completion type conditioning operations. Furthermore, Algorithm 1 can be modified to implement a greedy sampling algorithm without any additional cost. Log-submodularity. As mentioned above, DPPs are included in the larger class of Strongly Rayleigh (SR) measures over subsets. Although being SR is a delicate property, which is maintained by only few operations BID5 ), log-submodularity 3 (which is implied by SR-ness) is more robust, as well as a fundamental property in discrete optimization BID44 BID8 BID16 and machine learning BID9. Crucially, we show in the following that (log)-submodularity can be inherited by a generative model trained on a log-submodular distribution: THEOREM 1. Let P be a strictly submodular function over subsets of Y, and Q be a function over the same space such that DISPLAYFORM2 where D TV indicates the total variational distance. Then Q is also submodular. COROLLARY 1.1. Let P L be a strictly log-submodular DPP over Y and DPPNET be a network trained on the DPP probabilities p(S), with a loss function of the form p − q where · is a norm and p ∈ R 2 N (resp. q) is the probability vector assigned by the DPP (resp. the DPPNET) to each subset of Y. Let α = max x ∞=1 1 x. If DPPNET converges to a loss smaller than DISPLAYFORM3 its generative distribution is log-submodular. The follows directly from Thm. 3 and the equivalence of norms in finite dimensional spaces. REMARK 1. Cor. 1.1 is generalizable to the KL divergence loss D KL (P Q) via Pinsker's inequality. For this reason, we train our models by minimizing the distance between the predicted and target probabilities, rather than optimizing the log-likelihood of generative samples under the true DPP.Leveraging the sampling path. When drawing samples from a DPP, the standard DPP sampling algorithm (, Alg. 1) generates the sample as a sequence, adding items one after the other until reaching a pre-determined size 4, similarly to Alg. 1. We take advantage of this by recording all intermediary subsets generated by the DPP when sampling training data: in practice, instead of training on n subsets of size k, we train on kn subsets of size 0,..., k − 1. Thus, our model is very much like an unrolled recurrent neural network. In the simplest setting, we may wish to draw many samples over a ground set with a fixed feature embedding. In this case, we wish to model a DPP with a fixed kernel via a generative neural network. Specifically, we consider a fixed DPP with kernel L and wish to obtain a generative model such that DISPLAYFORM0 More generally, in many cases we may care about sampling from a DPP over a ground set of items that varies: this may be the case for example with a pool of products that are available for sale at a given time, or social media posts with a relevance that varies based on context. To leverage the speed-up provided by dual DPP sampling, we can only sample from the DPP with kernel given by L = ΦΦ; for more complex kernels, we once again incur the O(N 3) cost. Furthermore, training a static neural network for each new feature embedding may be too costly. Instead, we augment the static DPPNET to include the feature matrix Φ representing the ground set of all items as input to the network. Specifically, we draw inspiration for the dot-product attention introduced in BID41. In the original paper, the attention mechanism takes as input 3 matrices: the keys K, the values V, and the query Q. Attention is computed as DISPLAYFORM1 where d is the dimension of each query/key: the inner product acts as a proxy to the similarity between each query and each key. Finally, the reweighted value matrix AV is fed to the trainable neural network. DISPLAYFORM2 Figure 2: Transformer network architecture for sampling from a variable ground set. Here, the feature representation of items in the input set S acts as the query Q ∈ R k×d; the feature representation Φ ∈ R N ×d of our ground set is both the keys and the values. In order for the attention mechanism to make sense in the framework of DPP modeling, we make two modifications to the attention in BID41 ):• We want our network to attend to items that are dissimilar to the query (input subset): for each item i in the input subset S, we compute its pairwise dissimilarity to each item in Y as the vector DISPLAYFORM3 • Instead of returning this k × N matrix D of dissimilarities d i, we return a vector a ∈ R N in the probability simplex such that a j ∝ i∈S D ij. This allows us to have a fixed-size input to the neural network, and simultaneously enforces the desirable property that similarity to a single item is enough to disqualify an element from the ground set. Note that we could also return D in the form of a N × N matrix, but this would be counterproductive to speeding up DPP sampling. Putting everything together, our attention vector a is computed via the inhibitive attention mechanism DISPLAYFORM4 where represents the row-wise multiplication operator; this vector can be computed in O(kDN) time. The attention component of the neural network finally feeds the element-wise multiplication of each row of V with a to the feed-forward component. Given Φ and a subset S, the network is trained as in the static case to learn the marginal probabilities of adding any item in Y to S under a DPP with a kernel L dependent on Φ. In practice, we set L to be an exponentiated quadratic kernel L ij = exp(−β φ i − φ j 2) constructed with the features φ i.REMARK 2. Dual sampling for DPPs as introduced in BID25 ) is efficient only when sampling from a DPP with kernel L = ΦΦ; for non-linear kernels, a low-rank decomposition of L(Φ) must first be obtained, which in the worst case requires O(N 3) operations. In comparison, the dynamic DPPNET can be trained on any DPP kernel, while only requiring Φ as input. To evaluate DPPNET, we look at its performance both as a proxy for a static DPP (Section 4.1) and as a tool for generating diverse subsets of varying ground sets (Section 4.2). Our models are trained with TensorFlow, using the Adam optimizer. Hyperparameters are tuned to maximize the normalized log-likelihood of generated subsets. We compare DPPNET to DPP performance as well as two additional baselines:• UNIF: Uniform sampling over the ground set.• k-MEDOIDS: The k-medoids clustering algorithm (, 14.3.10), applied to items in the ground set, with distance between points computed as the same distance metric used by the DPP. Conversely to k-means, k-MED uses data points as centers for each cluster. We use the negative log-likelihood (NLL) of a subset under a DPP constructed over the ground set to evaluate the subsets obtained by all methods. This choice is motivated by the following considerations: DPPs have become a standard way of measuring and enforcing diversity over subsets of data in machine learning, and b) to the extent of our knowledge, there is no other standard method to benchmark the diversity of a selected subset that depends on specific dataset encodings. We begin by analyzing the performance of a DPPNET trained on a DPP with fixed kernel over the unit square. This is motivated by the need for diverse sampling methods on the unit hypercube, motivated by e.g. quasi-Monte Carlo methods, latin hypercube sampling BID36 and low discrepancy sequences. The ground set consists of the 100 points lying at the intersections of the 10 × 10 grid on the unit square. The DPP is defined by setting its kernel L to L ij = exp(− x i − x j 2 2 /2). As the DPP kernel is fixed, these experiments exclude the inhibitive attention mechanism. We report the performance of the different sampling methods in FIG2. Visually FIG2 ) and quantitively FIG2 ), DPPNET improves significantly over all other baselines. The NLL of DPPNET samples is almost identical to that of true DPP samples. Furthermore, greedily sampling the mode from the DPPNET achieves a better NLL than DPP samples themselves. Numerical are reported in TAB0. We evaluate the performance of DPPNETs on varying ground set sizes through the MNIST , CelebA BID30, and MovieLens BID17 datasets. For MNIST and CelebA, we generate feature representations of length 32 by training a Variational Auto-Encoder BID23 on the dataset 5; for MovieLens, we obtain a feature vector for each movie by applying nonnegative matrix factorization the rating matrix, obtaining features of length 10. Experimental presented below were obtained using feature representations obtained via the test instances of each dataset. The DPPNET is trained based on samples from DPPs with a linear kernel for MovieLens and with an exponentiated quadratic kernel for the image datasets. Bandwidths were set to β = 0.0025 for MNIST and β = 0.1 for CelebA in order to obtain a DPP average sample size ≈ 20: recall that for a DPP with kernel L, the expected sample size is given by the formula For MNIST, FIG3 shows images selected by the baselines and the DPPNET, chosen among 100 digits with either random labels or all identical labels; visually, DPPNET and DPP samples provide a wider coverage of writing styles. However, the NLL of samples from DPPNET decay significantly, whereas the DPPNET mode continues to maintain competitive performance with DPP samples. DISPLAYFORM0 Numerical for MNIST are reported in Table 2; additionally to the previous baselines, we also consider two further ways of generating subsets. INHIBATTN samples items from the multinomial distribution generated by the inhibitive attention mechanism only (without the subsequent neural network). NOATTN is a pure feed-forward neural network without attention; after hyper-parameter tuning, we found that the best architecture for this model consisted in 6 layers of 585 neurons each. Table 2 reveals that both the attention mechanism and the subsequent neural network are crucial to modeling DPP samples. Strikingly, DPPNET performs significantly better than other baselines even on feature matrices drawn from a single class of digits (Table 2), despite the training distribution over feature matrices being much less specialized. This implies that DPPNET sampling for dataset summarization may be leveraged to focus on sub-areas of datasets that are identified as areas of interest. Numerical for CelebA and MovieLens are reported in TAB2, confirming the modeling ability of DPPNETs. Finally, we verify that DPPNET allows for significantly faster sampling by running DPP and DPPNET sampling for subsets of size 20 drawn from a ground set of size 100 with both a standard DPP Table 2: NLL (mean ± standard error) under the true DPP of samples drawn uniformly, according to the mode of the DPPNET, and from the DPP itself. We sample subsets of size 20; for each class of digits we build 25 feature matrices Φ from encodings of those digits, and for each feature matrix we draw 25 different samples. Bolded numbers indicate the best-performing (non-DPP) sampling method. TAB0 49.2 ± 0.1 52.2 ± 0.1 60.5 ± 0.1 49.8 ± 0.0 50.7 ± 0.1 51.0 ± 0.1 50.4 ± 0.1 51.6 ± 0.1 51.5 ± 0.1 50.9 ± 0.1 52.7 ± 0.1 UNIF 51.6 ± 0.1 54.9 ± 0.1 65.1 ± 0.1 51.5 ± 0.1 52.9 ± 0.1 53.3 ± 0.1 52.4 ± 0.1 54.6 ± 0.1 55.1 ± 0.1 53.3 ± 0.1 56.2 ± 0.1 MEDOIDS 51.0 ± 0.1 55.1 ± 0.1 65.0 ± 0.1 51.5 ± 0.0 52.9 ± 0.1 53.1 ± 0.1 52.4 ± 0.0 54.4 ± 0.1 55.1 ± 0.1 53.2 ± 0.1 56.1 ± 0.1 INHIBATTN 51.3 ± 0.1 54.7 ± 0.1 65.0 ± 0.1 51.4 ± 0.1 52.8 ± 0.1 53.0 ± 0.1 52.1 ± 0.1 54.5 ± 0.1 54.9 ± 0.1 53.2 ± 0.1 55.9 ± 0.1 NOATTN 51.4 ± 0.1 54.9 ± 0.1 65.4 ± 0.1 51.5 ± 0.1 52.9 ± 0.1 53.3 ± 0.1 52.2 ± 0.1 54.6 ± 0.1 55.2 ± 0.1 53.3 ± 0.1 56.1 ± 0.1 DPPNET MODE 48.6 ± 0.2 53.6 ± 0.3 63.6 ± 0.4 50.8 ± 0.2 51.4 ± 0.3 51.6 ± 0.4 51.8 ± 0.3 52.8 ± 0.3 52.7 ± 0.4 50.9 ± 0.3 55.0 ± 0.4 BID29 according to root mean squared error (RMSE) and wallclock time. We observe that subsets selected by DPPNET achieve comparable and lower RMSE than a DPP and the MCMC method respectively while being significantly faster.and DPPNET (using the MNIST architecture). Both methods were implemented in graph-mode TensorFlow. Sampling batches of size 32, standard DPP sampling costs 2.74 ± 0.02 seconds; DPPNET sampling takes 0.10 ± 0.001 seconds, amounting to an almost 30-fold speed improvement. As a final experiment, we evaluate DPPNET's performance on a downstream task for which DPPs have been shown to be useful: kernel reconstruction using the Nyström method BID38 BID43. Given a positive semidefinite matrix K ∈ R N ×N, the Nyström method DISPLAYFORM0 where K † denotes the pseudoinverse of K and K ·,S (resp. K S,·) is the submatrix of K formed by its rows (resp. columns) indexed by S. The Nyström method is a popular method to scale up kernel methods and has found many applications in machine learning (see e.g. (Bac; She; Fow; Tal)). Importantly, the approximation quality directly depends on the choice of subset S. Recently, DPPs have been shown to be a competitive approach for selecting S BID35 BID28. Following the approach of BID28, we evaluate the quality of the kernel reconstruction by learning a regression kernel K on a training set, and reporting the prediction error on the test set using the Nyström reconstructed kernelK. Additionally to the full DPP, we also compare DPPNET to the MCMC sampling method with quadrature acceleration BID28 c) FIG4 reports our on the Ailerons dataset 6 also used in BID28. We start with a ground set size of 1000 and compute the ing root mean squared error (RMSE) of the regression using various sized subsets selected by sampling from a DPP, the MCMC method of BID29, using the full ground set and DPPNET. FIG4 reports the runtimes for each method. We note that while all methods were run on CPU, DPPNet is more amenable to acceleration using GPUs. We introduced DPPNETs, generative networks trained on DPPs over static and varying ground sets which enable fast and modular sampling in a wide variety of scenarios. We showed experimentally on several datasets and standard DPP applications that DPPNETs obtain competitive performance as evaluated in terms of NLLs, while being amenable to the extensive recent advances in speeding up computation for neural network architectures. Although we trained our models on DPPs on exponentiated quadratic and linear kernels; we can train on any kernel type built from a feature representations of the dataset. This is not the case for dual DPP exact sampling, which requires that the DPP kernel be L = ΦΦ for faster sampling. DPPNETs are not exchangeable: that is, two sequences i 1,..., i k and σ(i 1),..., σ(i k) where σ is a permutation of [k], which represent the same set of items, will not in general have the same probability under a DPPNET. Exchangeability can be enforced by leveraging previous work BID45; however, non-exchangeability can be an asset when sampling a ranking of items. Our models are trained to take as input a fixed-size subset representation; we aim to investigate the ability to take a variable-length encoding as input as future work. The scaling of the DPPNET's complexity with the ground set size also remains an open question. However, standard tricks to enforce fixed-size ground sets such as sub-sampling from the dataset may be applied to DPPNETs. Similarly, if further speedups are necessary, sub-sampling from the ground set -a standard approach for DPP sampling over very large set sizes -can be combined with DPPNET sampling. In light of our on dataset sampling, the question of whether encoders can be trained to produce encodings conducive to dataset summarization via DPPNETs seems of particular interest. Assuming knowledge of the (encoding-independent) relative diversity of a large quantity of subsets, an end-to-end training of the encoder and the DPPNET simultaneously may yield interesting . Finally, although Corollary 1.1 shows the log-submodularity of the DPP can be transferred to a generative model, understanding which additional properties of training distributions may be conserved through careful training remains an open question which we believe to be of high significance to the machine learning community in general. A MAINTAINING LOG-SUBMODULARITY IN THE GENERATIVE MODEL THEOREM 2. Let p be a strictly submodular distribution over subsets of a ground set Y, and q be a distribution over the same space such that DISPLAYFORM0 Then q is also submodular. Proof. In all the following, we assume that S, T are subsets of a ground set Y such that S = T and S, T ∈ {∅, Y} (the inequalities being immediate in these corner cases). For the MNIST encodings, the VAE encoder consists of a 2d-convolutional layer with 64 filters of height and width 4 and strides of 2, followed by a 2d convolution layer with 128 filters (same height, width and strides), then by a dense layer of 1024 neurons. The encodings are of length 32. CelebA encodings were generated by a VAE using a Wide Residual Network BID44 ) encoder with 10 layers and filter-multiplier k = 4, a latent space of 32 full-covariance Gaussians, and a deconvolutional decoder trained end-to-end using an ELBO loss. In detail, the decoder architecture consists of a 16K dense layer followed by a sequence of 4 × 4 convolutions with filters interleaved with 2× upsampling layers and a final 6 × 6 convolution with 3 output channels for each of 5 components in a mixture of quantized logistic distributions representing the decoded image.

We approximate Determinantal Point Processes with neural nets; we justify our model theoretically and empirically.

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the scientific article to find a short TLDR

Question: This paper introduces a network architecture to solve the structure-from-motion (SfM) problem via feature-metric bundle adjustment (BA), which explicitly enforces multi-view geometry constraints in the form of feature-metric error. The whole pipeline is differentiable, so that the network can learn suitable features that make the BA problem more tractable. Furthermore, this work introduces a novel depth parameterization to recover dense per-pixel depth. The network first generates several basis depth maps according to the input image, and optimizes the final depth as a linear combination of these basis depth maps via feature-metric BA. The basis depth maps generator is also learned via end-to-end training. The whole system nicely combines domain knowledge (i.e. hard-coded multi-view geometry constraints) and deep learning (i.e. feature learning and basis depth maps learning) to address the challenging dense SfM problem. Experiments on large scale real data prove the success of the proposed method. The Structure-from-Motion (SfM) problem has been extensively studied in the past a few decades. Almost all conventional SfM algorithms BID46 BID39 BID16 BID13 jointly optimize scene structures and camera motion via the Bundle-Adjustment (BA) algorithm BID43 BID1, which minimizes the geometric BID46 BID39 or photometric BID17 BID13 error through the Levenberg-Marquardt (LM) algorithm BID35. Some recent works BID44 attempt to solve SfM using deep learning techniques, but most of them do not enforce the geometric constraints between 3D structures and camera motion in their networks. For example, in the recent work DeMoN BID44, the scene depths and the camera motion are estimated by two individual sub-network branches. This paper formulates BA as a differentiable layer, the BA-Layer, to bridge the gap between classic methods and recent deep learning based approaches. To this end, we learn a feed-forward multilayer perceptron (MLP) to predict the damping factor in the LM algorithm, which makes all involved computation differentiable. Furthermore, unlike conventional BA that minimizes geometric or photometric error, our BA-layer minimizes the distance between aligned CNN feature maps. Our novel feature-metric BA takes CNN features of multiple images as inputs and optimizes for the scene structures and camera motion. This feature-metric BA is desirable, because it has been observed by BID17 that the geometric BA does not exploit all image information, while the photometric BA is sensitive to moving objects, exposure or white balance changes, etc. Most importantly, our BA-Layer can back-propagate loss from scene structures and camera motion to learn appropriate features that are most suitable for structure-from-motion and bundle adjustment. In this way, our network hard-codes the multi-view geometry constraints in the BA-Layer and learns suitable feature representations from training data. We strive to estimate a dense per-pixel depth, because dense depth is critical for many tasks such as object detection and robot navigation. A major challenge in solving dense per-pixel depth is to find a compact parameterization. Direct per-pixel depth is computational expensive, which makes the network training intractable. So we train a network to generate a set of basis depth maps for an arbitrary input image and represent the depth map as a linear combination of these basis 2 RELATED WORK Monocular Depth Estimation Networks Estimating depth from a monocular image is an ill-posed problem because an infinite number of possible scenes may have produced the same image. Before the raise of deep learning based methods, some works predict depth from a single image based on MRF BID37 BID36, semantic segmentation BID29, or manually designed features BID27. BID15 propose a multi-scale approach for depth prediction with two CNNs, where a coarse-scale network first predicts the scene depth at the global level and then a fine-scale network will refine the local regions. This approach was extended in BID14 to handle semantic segmentation and surface normal estimation as well. Recently, BID30 propose to use ResNet BID24 based structure to predict depth, and BID47 construct multi-scale CRFs for depth prediction. In comparison, we exploit monocular image depth estimation network for depth parameterization, which only produces a set of basis depth maps and the final will be further improved through optimization. Structure-from-Motion Networks Recently, some works exploit CNNs to resolve the SfM problem. BID22 solve the camera motion by a network from a pair of images with known depth. employ two CNNs for depth and camera motion estimation respectively, where both CNNs are trained jointly by minimizing the photometric loss in an unsupervised manner. implement the direct method BID40 as a differentiable component to compute camera motion after scene depth is estimated by the method in. In BID44, the scene depth and the camera motion are predicted from optical flow features, which help to make it generalizing better to unseen data. However, the scene depth and the camera motion are solved by two separate network branches, multi-view geometry constraints between depth and motion are not enforced. Recently, propose to solve nonlinear least squares in two-view SfM using a LSTM-RNN BID26 as the optimizer. Our method belongs to this category. Unlike all previous works, we propose the BA-Layer to simultaneously predict the scene depth and the camera motion from CNN features, which explicitly enforces multi-view geometry constraints. The hard-coded multi-view geometry constraints enable our method to reconstruct more than two images, while most deep learning methods can only handle two images. Furthermore, we propose to minimize a feature-metric error instead of the photometric error in to enhance robustness. Before introducing our BA-Net architecture, we revisit the classic BA to have a better understanding about where the difficulties are and why feature-metric BA and feature learning are desirable. We only introduce the most relevant content and refer the readers to BID43 and BID1 for a comprehensive introduction. Given images I = {I i |i = 1 · · · N i}, the geometric BA BID43 BID1 jointly optimizes camera poses T = {T i |i = 1 · · · N i} and 3D scene point coordinates P = {p j |j = 1 · · · N j} by minimizing the re-projection error: DISPLAYFORM0 where the geometric distance e g i,j (X) = π(T i, p j) − q i,j measures the difference between a projected scene point and its corresponding feature point. The function π projects scene points to image space, q i,j = [x i,j, y i,j, 1] is the normalized homogeneous pixel coordinate, and DISPLAYFORM1 contains all the points' and the cameras' parameters. The general strategy to minimize Equation is the Levenberg-Marquardt (LM) BID35 BID32 algorithm. At each iteration, the LM algorithm solves for an optimal update ∆X * to the solution by minimizing: DISPLAYFORM2 Here, DISPLAYFORM3 Ni,Nj (X)], and J(X) is the Jacobian matrix of E(X) respect to X, D(X) is a non-negative diagonal matrix, typically the square root of the diagonal of the approximated Hessian J(X) J(X). The non-negative value λ controls the regularization strength. The special structure of J(X) J(X) motivates the use of Schur-Complement BID6.This geometric BA with re-projection error is the golden standard for structure-from-motion in the last two decades, but with two main drawbacks:• Only image information conforming to the respective feature types, typically image corners, blobs, or line segments, is utilized.• Features have to be matched to each other, which often in a lot of outliers. Outlier rejection like RANSAC is necessary, which still cannot guarantee correct . These two difficulties motivate the recent development of direct methods BID17 BID13 which propose the photometric BA algorithm to eliminate feature matching and directly minimizes the photometric error (pixel intensity difference) of aligned pixels. The photometric error is defined as: DISPLAYFORM4 where d j ∈ D = {d j |j = 1 · · · N j} is the depth of a pixel q j at the image I 1, and d j · q j upgrade the pixel q j to its 3D coordinate. Thus, the optimization parameter is DISPLAYFORM5. The direct methods have the advantages of using all pixels with sufficient gradient magnitude. They have demonstrated superior performance, especially at less textured scenes. However, these methods also have some drawbacks:• They are sensitive to initialization as demonstrated in BID33 and BID41 because the photometric error increases the non-convexity BID16 ).• They are sensitive to camera exposure and white balance changes. An automatic photometric calibration is required BID16 ).• They are more sensitive to outliers such as moving objects. To deal with the above challenges, we propose a feature-metric BA algorithm which estimates the same scene depth and camera motion parameters X as in photometric BA, but minimizes the feature-metric difference of aligned pixels: DISPLAYFORM0 where BID49 as the backbone network, a Basis Depth Maps Generator that generates a set of basis depth maps, a Feature Pyramid Constructor that constructs multi-scale feature maps, and a BA-Layer that optimizes both the depth map and the camera poses through a novel differentiable LM algorithm. DISPLAYFORM1 We learn features suitable for SfM via back-propagation, instead of using pre-trained CNN features for image classification BID10. Therefore, it is crucial to design a differentiable optimization layer, our BA-Layer, to solve the optimization problem, so that the loss information can be back-propagated. The BA-Layer predicts the camera poses T and the dense depth map D during forward pass and back-propagates the loss from T and D to the feature pyramids F for training. As illustrated in FIG0, our BA-Net receives multiple images and then feed them to the backbone DRN-54. We use DRN-54 BID49 because it replaces max-pooling with convolution layers and generates smoother feature maps, which is desirable for BA optimization. Note the original DRN is memory inefficient due to the high resolution feature maps after dilation convolutions. We replace the dilation convolution with ordinary convolution with strides to address this issue. After DRN-54, a feature pyramid is then constructed for each input image, which are the inputs for the BA-Layer. At the same time, the basis depth maps generator generates multiple basis depth maps for the image I 1, and the final depth map is represented as a linear combination of these basis depth maps. Finally, the BA-Layer optimizes for the camera poses and the dense depth map jointly by minimizing the feature-metric error defined in Equation FORMULA6, which makes the whole pipeline end-to-end trainable. The feature pyramid learns suitable features for the BA-Layer. Similar to the feature pyramid networks (FPN) for object detection BID31, we exploit the inherent multi-scale hierarchy of deep convolutional networks to construct feature pyramids. A top-down architecture with lateral connections is applied to propagate richer context information from coarser scales to finer scales. Thus, our feature-metric BA will have a larger convergence radius. As shown in Figure 2 (a), we construct a feature pyramid from the backbone DRN-54. We denote the last residual blocks of conv1, conv2, conv3, conv4 in DRN-54 as {C 1, C 2, C 3, C 4}, with strides {1, 2, 4, 8} respectively. We upsample a feature map C k+1 by a factor of 2 with bilinear interpolation and concatenate the upsampled feature map with C k in the next level. This procedure is iterated until the finest level. Finally, we apply a 3 × 3 convolution on the concatenated feature maps to reduce its dimensionality to 128 to balance the expressiveness and computational complexity, which leads to the final feature pyramid DISPLAYFORM0 We visualize some typical channels from the raw image I (i.e. the RGB channels), the pre-trained DRN-54 C 3 and our learned F 3 in Figure 2 (b). It is evident that, after training with our BA-Layer, the feature pyramid becomes smoother and each channel correspondences to different regions in the image. Note that our feature pyramids have higher resolution than FPN to facilitate precise alignment. To have a better intuition about how much the BA optimization benefits from our learned features, we visualize different distances in Figure 3. We evaluate the distance between a pixel marked by a yellow cross in the top image in Figure 3 (a) and all pixels in a neighbourhood of its corresponding point in the bottom image of Figure 3 (a). The distances evaluated from raw RGB values, pretrained feature C 3, and our learned feature F 3 are visualized in (b), (c), and (d) respectively. All distances are normalized to and visualized as heat maps. The x-axis and y-axis are the offsets to the ground-truth corresponding point. The RGB distance in (b) (i.e. e p in Equation FORMULA4) has no clear global minimum, which makes the photometric BA sensitive to initialization BID17. The distance measured by the pretrained feature C 3 has both global and local minimums. Finally, the distance measured by our learned feature F 3 has a clear global minimum and smooth basin, which is helpful in gradient based optimization such as the LM algorithm. After building feature pyramids for all images, we optimize camera poses and a dense depth map by minimizing the feature-metric error in Equation. Following the conventional Bundle Adjustment principle, we optimize Equation using the Levenberg-Marquardt (LM) algorithm. However, the original LM algorithm is non-differentiable because of two difficulties:• The iterative computation terminates when a specified convergence threshold is reached. This if-else based termination strategy makes the output solution X non-differentiable with respect to the input F .• In each iteration, it updates the damping factor λ based on the current value of the objective function. It raises λ if a step fails to reduce the objective; otherwise it reduces λ. This if-else decision also makes X non-differentiable with respect to F.When the solution X is non-differentiable with respect to F, feature learning by back-propagation becomes impossible. The first difficulty has been studied in and the author proposes to fix the number of iterations, which is refered as'incomplete optimization'. Besides making the optimization differentiable, this'incomplete optimization' technique also reduces memory consumption because the number of iterations is usually fixed at a small value. The second difficulty has never been studied. Previous works mainly focus on gradient descent or quadratic minimization BID3 BID38. In this section, we propose a simple yet effective approach to soften the if-else decision and yields a differentiable LM algorithm. We send the current objective value to a MLP network to predict λ. This technique not only makes the optimization differentiable, but also learns to predict a better damping factor λ, which helps the optimization to reach a better solution within limited iterations. To start with, we illustrate a single iteration of the LM optimization as a diagram in Figure 4 by interpreting intermediate variables as network nodes. During the forward pass, we compute the solution update ∆X from feature pyramids F and current solution X as the following steps:• We compute the feature-metric error • We then compute the Jacobian matrix J(X), the Hessian matrix J(X) J(X) and its diagonal matrix D(X); • To predict the damping factor λ, we use global average pooling to aggregate the aboslute value of E(X) over all pixels for each feature channel, and get a 128D feature vector. We then send it to a MLP sub-network to predict λ; • Finally, the update ∆X to the current solution is computed as a standard LM step: DISPLAYFORM0 DISPLAYFORM1 In this way, we can consider λ as an intermediate variable and denote each LM step as a function g about features pyramids F and the solution X from the previous iteration. In other words, ∆X = g(X ; F). Therefore, the solution after the k-th iteration is: DISPLAYFORM2 Here, • denotes parameters updating, which is addition for depth and SE exponential mapping for camera poses. Equation is differentiable with respect to the feature pyramids F, which makes back-propagation possible through the whole pipeline for feature learning. The MLP that predicts λ is also shown in Figure 4. We stack four fully-connected layers to predict λ from the input 128D vector. We use ReLU as the activation function to guarantee λ is non-negative. Following the photometric BA BID17, we solve our feature-metric BA using a coarse-to-fine strategy with feature map warping at each iteration. We apply the differentiable LM algorithm for 5 iterations at each pyramid level, leading to 15 iterations in total. All the camera poses are initialized with identity rotation and zero translation, and the initialization of depth map will be introduced in Section 4.4. Parameterizing a dense depth map by a per-pixel depth value is impractical under our formulation. Firstly, it introduces too many parameters for optimization. For example, an image of 320 × 240 pixels in 76.8k parameters. Secondly, in the beginning of training, many pixels will become invisible in the other views because of the poorly predicted depth or motion. So little information can be back-propagated to improve the network, which makes training difficult. To deal with these problems, we use the convolutional network for monocular image depth estimation as a compact parameterization, rather than using it as an initialization as in BID42 and BID48. We use a standard encoder-decoder architecture for monocular depth learning as in BID30. We use DRN-54 as the encoder to share the same backbone features with our feature pyramids. For the decoder, we modify the last convolutional feature maps of BID30 to 128 channels and use these feature maps as the basis depth maps for optimization. The final depth map is generated as the linear combination of these basis depth maps, which is: DISPLAYFORM0 Here, D is the h · w depth map that contains depth values for all pixels, B is a 128 × h · w matrix, representing 128 basis depth maps generated from network, w is the linear combination weights of these basis depth maps. The w will be optimized in our BA-Layer. The ReLU activation function guarantees the final depth is non-negative. Once B is generated from the network, we fix B and use w as a compact depth parameterization in BA optimization, and the feature-metric distance becomes: DISPLAYFORM1 where B[j] is the j-th column of B, and ReLU(w B [j] ) is the corresponding depth of q j. To further speedup convergence, we learn the initial weight w 0 as a 1D convolution filter for an arbitrary image, i.e. D 0 = ReLU(w 0 B). The B of various images are visualized in the appendix. The BA-Net learns the feature pyramid, the damping factor predictor, and the basis depth maps generator in a supervised manner. We apply the following commonly used loss for training, though more sophisticated ones might be designed. The camera rotation loss is the distance between rotation quaternion vectors L rotation = q − q *. Similarly, translation loss is the Euclidean distance between prediction and groundtruth in metric scale, L translation = t − t *. For each dense depth map we applies the berHu Loss BID51 as in BID30.We initialize the back-bone network from DRN-54 BID49, and the other components are trained with ADAM from scratch with initial learning rate 0.001, and the learning rate is divided by two when we observe plateaus from the Tensorboard interface. ScanNet ScanNet BID11 ) is a large-scale indoor dataset with 1,513 sequences in 706 different scenes. Camera poses and depth maps are not perfect, because they are estimated via BundleFusion BID12. The metric scale is known in all data from ScanNet, because the data are recorded with a depth camera which returns absolute depth values. To sample image pairs for training, we apply a simple filtering process. We first filter out pairs with a large photo-consistency error, to avoid image pairs with large pose or depth error. We also filter out image pairs, if less than 50% of the pixels from one image are visible in the other image. In addition, we also discard a pair if their roundness score BID4 ) is less than 0.001, which avoids pairs with too narrow baselines. We split the whole dataset into the training and the testing sets. The training set contains the first 1,413 sequences and the testing set contains the rest 100 sequences. We sample 547,991 training pairs and 2,000 testing pairs from the training and testing sequences respectively. KITTI KITTI BID20 ) is a widely used benchmark dataset collected by car-mounted cameras and a LIDAR sensor on streets. It contains 61 scenes belonging to the "city", "residential", or "road" categories. BID15 select 28 scenes for testing and 28 scenes from the remaining for training. We use the same data split, to make a fair comparison with previous methods. Since ground truth pose is unavailable from the raw KITTI dataset, we compute camera poses by LibVISO2 BID19 and take them as ground truth after discarding poses with large errors. ScanNet To evaluate the ' quality, we use the depth error metrics suggested in BID14, where RMSE (linear, log, and log, scale inv.) measure the RMSE of the raw, the logarithmical, and aligned logarithmical depth values, while the other two metrics measure the mean of the ratios that divide the absolute and square error by groundtruth depth.. The errors in camera Table 1: Quantitative comparisons with DeMoN and classic BA. The superindex * denotes that the model is trained on the trainning set described in BID44. poses are measured by the rotation error (the angle between the ground truth and the estimated camera rotations), the translation direction error (the angle between the ground truth and estimated camera translation directions) and the absolute position error (the distance between the ground truth and the estimated camera translation vectors).In Table 1, we compare our method with DeMoN BID44 and the conventional photometric and geometric BA. Note that we cannot get DeMoN trained on the ScanNet. For fair comparison, we train our network on the same training data as DeMoN and test both networks on our testing data 1. We also show the of our network trained on ScanNet. Our BA-Net consistently performs better than DeMoN no matter which training data is used. Since DeMoN does not recover the absolute scale, we align its depth map with the groundtruth to recover its metric scale for evaluation. We further compare with conventional geometric BID34 and photometric BID17 BA. Again, our method produces better . The geometric BA works poorly here, because feature matching is difficult in indoor scenes. Even the RANSAC process cannot get rid of all outliers. While for photometirc BA, the highly non-convex objective function is difficult to optimize as described in Section 3.KITTI We use the same metrics as the comparisons on ScanNet for depth evaluation. To evaluate the camera poses, we follow to use the Absolute Trajectory Error (ATE), which measures the Euclidean differences between two trajectories BID40, on the 9th and 10th sequences from the KITTI odometry data. In this experiment, we create short sequences of 5 frames by first computing 5 two-view reconstructions from our BA-Net and then align the two-view reconstructions in the coordinate system anchored at the first frame. minimize the photometric error. BID21 BID15 Table 2: Quantitative comparisons on KITTI with supervised BID15 and unsupervised BID21 methods. Table 2 summarizes our on KITTI. Our method outperforms the supervised methods BID15 as well as recent unsupervised methods BID21. Our method also achieves more accurate camera trajectories than and. We believe this is due to our feature-metric BA with features learned specifically for SfM problem, which makes the objective function closer to convex and easier to optimize as discussed in Section 4.2. In comparison, and minimize the photometric error. More comparison with DeMoN, ablation studies, and multi-view SfM (up to 5 views) are reported in the appendix due to page limit. This paper presents the BA-Net, a network that explicitly enforces multi-view geometry constraints in terms of feature-metric error. It optimizes scene depths and camera motion jointly via feature-metric bundle adjustment. The whole pipeline is differentiable and thus end-to-end trainable, such that the features are learned from data to facilitate structure-from-motion. The dense depth is parameterized as a linear combination of several basis depth maps generated from the network. Our BA-Net nicely combines domain knowledge (hard-coded multi-view geometry constraint) with deep learning (learned feature representation and basis depth maps generator). It outperforms conventional BA and recent deep learning based methods. DISPLAYFORM0 (a) Architecture of the DRN-54 backbone FIG4 illustrates the detailed network architectures for the backbone DRN-54 and the depth basis generator. The architecture of the feature pyramid has been provided in Figure 2(a). We modify the dilated convolution of the original DRN-54 to convolution with strides and discard the conv7 and conv8 as shown in FIG4 (a). C 1 to C 6 are layers with {1,2,4,8,16,32} strides and {16,32,256,512,1024,2048} channels, where C 1 and C 2 are basic convolution layers, while C 3 to C 6 are standerd bottleneck blocks as in ResNet BID24. Figure 5(b) visualizes our depth basis generator which adopts the up-projection structure proposed in BID30. The depth basis generator is a stander decoder that takes the output of C 6 as input and stacks five up-projection blocks to generate 128 basis depth maps, and each of the basis depth maps is half the resolution of the input image. The up-projection block is shown on the right of FIG4 (b) which upsample the input by 2× and then apply convolutions with projection connection. Evaluation Time To evaluate the running time of our method, we use the Tensorflow profiler tool to retrieve the time in ms for all network nodes and then summarize the corresponding to each component in our pipeline. As shown in TAB5, our method takes 95.21 ms to reconstruct two 320 × 240 images, which is slightly faster than DeMoN that takes 110 ms for two 256 × 192 images. The current computation bottleneck is the BA-Layer which contains a large amount of matrix operations and can be further speeded up by direct CUDA implementation. Since we explicitly hard-code the multi-view geometry constraints in the BA-Layer, it is possible to share the backbone DRN-54 with other high-level vision tasks, such as semantic segmentation and object detection, to maximize reuse of network structures and minimize extra computation cost. TAB6, the pre-trained features (i.e. w/o Feature Learning) produce larger error. This proves the discussion in Section 4.2.Bundle Adjustment Optimization vs SE Pose Estimation Our BA-Layer optimizes depth and camera poses jointly. We compare it to the SE camera pose estimation with fixed depth map (e.g. the initialized depth D 0 in Section 4.4), and similar strategy is adopted in. To make a fair comparison, we also use our learned feature pyramids for the SE camera pose estimation. As shown in TAB6, without BA optimization (i.e. w/o Joint Optimization), both the depth maps and camera poses are worse, because the errors in the depth estimation will degrades the camera pose estimation. Differentiable Levenberg-Marquardt vs Gauss-Newton To make the whole pipeline end-to-end trainable, we makes the Levenberg-Marquardt algorithm differentiable by learning the damping factor from the network. We first compare our method against vanilla Gauss-Newton without damping factor λ (i.e. λ = 0). Since the objective function of feature-metric BA is non-convex, the Hessian matrix J(X) J(X) might not be positive definite, which makes the matrix inversion by Cholesky decomposition fail. To deal with this problem, we use QR decomposition instead for training with Gauss-Newton. As shown in TAB6, the Gauss-Newton algorithm (i.e. w/o λ) generates much larger error, because the BA optimization is non-convex and the Gauss-Newton algorithm has no guaranteed convergence unless the initial solution is sufficiently close to the optimal BID35. This comparison reveals that, similar to conventional BA, our differnetiable Levenberg-Marquardt algorithm is superior than the Gauss-Newton algorithm for feature-metric BA. Predicted vs Constant λ Another way to make the Levenberg-Marquardt algorithm differentiable is to fix the λ during the iterations. We compare with this strategy. As shown in FIG6 (a), increasing λ makes the both rotation and translation error decreases, until λ = 0.5, and then increases. The reason is that a small λ makes the algorithm close to the Gauss-Newton algorithm, which has convergence issues. A large λ leads to a small update at each iteration, which makes it difficult to reach a good solution within limited iterations. While in FIG6 (b), increasing λ always makes depth errors decrease, probably because a larger λ leads to a small update and makes the final depth close to the initialed depth, which is better than the optimized one with small constant λ. Using constant λ value consistently generates worse than using a predicted λ from the MLP network, because there is no optimal λ for all data and it should be adapted to different data and different iterations. We draw the errors of our method in FIG6 (a) and FIG6 (b) as the flat dash lines for a reference. APPENDIX C: EVALUATION ON DEMON DATASET Table 5 summarizes our on the DeMoN dataset. For a comparison, we also cite the from DeMoN BID44 and the most recent work LS-Net. We further cite the from some conventional approaches as reported in DeMoN, indicated as Oracle, SIFT, FF, and Matlab respectively. Here, Oracle uses ground truth camera poses to solve the multi-view stereo by SGM BID25, while SIFT, FF, and Matlab further use sparse features, optical flow, and KLT tracking respectively for feature correspondence to solve camera poses by the 8-pt algorithm BID23 Table 5: Quantitative comparisons on the DeMoN dataset. Our method consistently outperforms DeMoN BID44 at both camera motion and scene depth, except on the'Scenes11' data, because we enforce multi-view geometry constraint in the BA-Layer. Our are poorer on the'Scene11' dataset, because the images there are synthesized with random objects from the ShapeNet BID8 without physically correct scale. This setting is inconsistent with real data and makes it harder for our method to learn the basis depth map generator. When compared with LS-Net, our method achieves similar accuracy on camera poses but better scene depth. It proves our feature-metric BA with learned feature is superior than the photometric BA in the LS-Net. Our method can be easily extended to reconstruct multiple images. We evaluate our method in the multi-view setting on the ScanNet BID11 dataset. To sample multi-view images for training, we randomly select two-view image pairs that shares a common image to construct N -view sequences. Due to the limited GPU memory (12G), we limit N to 5.As shown in the Table 6, the accuracy is consistently improved when more views are included, which demonstrates the strength of the multi-view geometry constraints. Instead, most existing deep learning approaches can only handle two views at a time, which is sub-optimal as known in structure-from-motion literature. Table 6: Quantitative comparisons on multi-view reconstruction on ScanNet. We compare our method with CodeSLAM which adopts similar idea for depth parameterization. But the difference is that CodeSLAM learns the conditioned depth auto-encoder separately and uses the depth codes in a standalone photometric BA component, while our method learns the feature pyramid and basis depth maps generator through feature-metric BA end-to-end. Since there is no public code for CodeSLAM, we directly cite the from their paper. 2 To get the trajectory on the EuroC MH02 sequence of our method, we select one frame every four frames and concatenate the reconstructed groups that contains every five selected frames. Then we use the same evaluation metrics as in CodeSLAM, which measures the translation errors correspond to different traveled distances. As shown in FIG7, our method outperforms CodeSLAM. Our median error is less than the half of CodeSLAM's error, i.e. CodeSLAM exhibits an error of roughly 1 m for a traveled distance of 9 m, while our method's error is about 0.4 m. This comparison demonstrates the superiority of end-to-end learning with feature pyramid and feature-metric BA over learning depth parameterization only. In FIG8, we visualize four typical basis depth maps as heat maps for each of the four images. An interesting observation is that one basis depth map has higher responses on close objects while another oppositely has higher responses to the far . Some other basis depth maps have smoothly varying responses and correspond to the layouts of scenes. This observation reveals that our learned basis depth maps have captured the latent structures of scenes. Finally, we show some qualitative comparison with the previous methods. Figure 9 shows the recovered depth map by our method and DeMoN BID44 on the ScanNet data. As we can see from the regions highlighted with a red circle, our method recovers more shape details. This is consistent with the quantitative in Table 1. FIG0 shows the recovered depth maps by our method,, and BID21 respectively. Similarly, we observe more shape details in our , as reflected in the quantitative in Table 2. FORMULA0 and BID21.

Represent the TLDR for finding the corresponding scientific article

This paper introduces a network architecture to solve the structure-from-motion (SfM) problem via feature bundle adjustment (BA)

Represent the short summary for finding the corresponding scientific article (long)