Question: Represent the scientific article to find a short TLDR

Question: We present a 3D capsule architecture for processing of point clouds that is equivariant with respect to the SO rotation group, translation and permutation of the unordered input sets. The network operates on a sparse set of local reference frames, computed from an input point cloud and establishes end-to-end equivariance through a novel 3D quaternion group capsule layer, including an equivariant dynamic routing procedure. The capsule layer enables us to disentangle geometry from pose, paving the way for more informative descriptions and a structured latent space. In the process, we theoretically connect the process of dynamic routing between capsules to the well-known Weiszfeld algorithm, a scheme for solving iterative re-weighted least squares (IRLS) problems with provable convergence properties, enabling robust pose estimation between capsule layers. Due to the sparse equivariant quaternion capsules, our architecture allows joint object classification and orientation estimation, which we validate empirically on common benchmark datasets. It is now well understood that in order to learn a compact representation of the input data, one needs to respect the symmetries in the problem domain (; a). Arguably, one of the primary reasons of the success of 2D convolutional neural networks (CNN) is the translation-invariance of the 2D convolution acting on the image grid (; . Recent trends aim to transfer this success into the 3D domain in order to support many applications such as shape retrieval, shape manipulation, pose estimation, 3D object modeling and detection. There, the data is naturally represented as sets of 3D points or a point cloud (a; b). Unfortunately, extension of CNN architectures to 3D point clouds is non-trivial due to two reasons: 1) point clouds are irregular and unstructured, 2) the group of transformations that we are interested in is more complex as 3D data is often observed under arbitrary non-commutative SO rotations. As a , achieving appropriate embeddings requires 3D networks that work on points to be equivariant to these transformations, while also being invariant to the permutations of the point set. In order to fill this important gap, we propose the quaternion equivariant point capsule network or QE-Network that is suited to process point clouds and is equivariant to SO rotations compactly parameterized by quaternions (Fig. 2), in addition to preserved translation and permutation equivariance. Inspired by the local group equivariance ), we efficiently cover SO by restricting ourselves to the sparse set of local reference frames (LRF) that collectively characterize the object orientation. The proposed capsule layers deduces equivariant latent representations by robustly combining those local LRFs using the proposed Weiszfeld dynamic routing. Hence, our latent features specify to local orientations disentangling the pose from object existence. Such explicit storage is unique to our work and allows us to perform rotation estimation jointly with object classification. Our final architecture is a hierarchy of QE-networks, where we use classification error as the only training cue and adapt a Siamese version when the relative rotation is to be regressed. We neither explicitly supervise the network with pose annotations nor train by augmenting rotations. Overall, our contributions are: 1. We propose a novel, fully SO-equivariant capsule architecture that is tailored for simultaneous classification and pose estimation of 3D point clouds. This network produces in- shows the LRFs randomly sampled from (a) and these are inputs to the first layer of our network. Subsequently, we obtain a multi-channel LRF that is a set of reference frames per pooling center (d). Holistically, our network aggregates the LRFs to arrive at rotation equivariant capsules. variant latent representations while explicitly decoupling the orientation into capsules, thus attaining equivariance. Note that equivariance have not been previously achieved regarding the quaternion parameterization of the 3D special orthogonal group. 2. By utilizing LRFs on points, we reduce the space of orientations that we consider and hence can work sparsely on a subset of the group elements. 3. We theoretically prove the equivariance properties of our 3D network regarding the quaternion group. Moreover, to the best of our knowledge, we for the first time establish a connection between the dynamic routing of and Generalized Weiszfeld iterations . By that, we theoretically argue for the convergence of the employed dynamic routing. 4. We experimentally demonstrate the capabilities of our network on classification and orientation estimation of 3D shapes. In this paper we will speak of the equivariance of point clouds under the actions of quaternions. We now provide the necessary required for the grasp of this content. Definition 1 (Equivariant Map). For a G-space acting on X, the map Φ: G × X → X is said to be equivariant if its domain and co-domain are acted on by the same symmetry group (; a): where g 1 ∈ G and g 2 ∈ G. Equivalently Φ(T (g 1) x) = T (g 2) Φ(x), where T (·) is a linear representation of the group G. Note that T (·) does not have to commute. It suffices for T (·) to be a homomorphism: In this paper we use a stricter form of equivariance and consider g 2 = g 1. Definition 2 (Equivariant Network). An architecture or network is said to be equivariant if all of its layers are equivariant maps. Due to the transitivity of the equivariance, stacking up equivariant layers will in globally equivariant networks e.g., rotating the input will produce output vectors which are transformed by the same rotation. The choice of 4-vector quaternions has multiple motivations: 1. All 3-vector formulations suffer from infinitely many singularities as angle goes to 0, whereas quaternions avoid those. 2. 3-vectors also suffer from infinitely many redundancies (the norm can grow indefinitely). Quaternions have a single redundancy: q = −q, a condition that is in practice easy to enforce. 3. Computing the actual'manifold mean' on the Lie algebra requires iterative techniques with subsequent updates on the tangent space. Such iterations are computationally harmful for a differentiable GPU implementation. Definition 3 (Quaternion). A quaternion q is an element of Hamilton algebra H 1, extending the complex numbers with three imaginary units i, j, k in the form: q = q 1 1 + q 2 i + q 3 j + q 4 k = Algorithm 1: Quaternion Equivariant Dynamic Routing. q 1 ∈ R denotes the scalar part and v = (q 2, q 3, q 4) T ∈ R 3, the vector part. The conjugateq of the quaternion q is given byq:= q 1 − q 2 i − q 3 j − q 4 k. A unit quaternion q ∈ H 1 with 1! = q:= q ·q and q −1 =q, gives a compact and numerically stable parametrization to represent orientation of objects on the unit sphere S 3, avoiding gimbal lock and singularities . Identifying antipodal points q and −q with the same element, the unit quaternions form a double covering group of SO. H 1 is closed under the non-commutative multiplication or the Hamilton product: Definition 4 (Linear Representation of H 1). We follow and use the To be concise we will use capital letters to refer to the matrix representation of quaternions e.g. Q ≡ T (q), G ≡ T (g). Note that T (·), the injective homomorphism to the orthonormal matrix ring, by construction satisfies the condition in Dfn. 1 : det(Q) = 1, Q = Q −1, Q = Q i,: = Q:,i = 1 and Q − q 1 I is skew symmetric: Q + Q = 2q 1 I. It is easy to verify these properties. T linearizes the Hamilton product or the group composition: g • q T (g)q Gq. Definition 5 (Point Cloud). We define a 3D surface to be a differentiable 2-manifold embedded in the ambient 3D Euclidean space: M 2 ∈ R 3 and a point cloud to be a discrete subset sampled on Definition 6 (Local Geometry). For a smooth point cloud {x i} ∈ M 2 ⊂ R N ×3, a local reference frame (LRF) is defined as an ordered basis of the tangent space at x, T x M, consisting of orthonormal vectors: Usually the first component is defined to be the surface normal ∂ 1 n ∈ S 2: n = 1 and the second one is picked according to a modality dependent heuristic. Note that recent trends such as acknowledge the ambiguity and either employ a gauge (tangent frame) equivariant design or propagate the determination of a certain direction until the last layer . Here, we will assume that ∂ 2 can be uniquely and igure 2: Our quaternion equivariant (QE) network for processing local patches: Our input is a 3D point set X on which we query local neighborhoods {x i} with precomputed LRFs {q i}. Essentially, we learn the parameters of a fully connected network that continuously maps the canonicalized local point set to transformations t i, which are used to compute hypotheses (votes) from input poses. By a special dynamic routing procedure that uses the activations determined in a previous layer, we arrive at latent capsules that are composed of a set of orientationsq i and new activationsα i. Thanks to the decoupling of local reference frames,α i is invariant and orientationsq i are equivariant to input rotations. All the operations and hence the entire QE-network are equivariant achieving a guaranteed disentanglement of the rotation parameters. Hat symbol (q) refers to'estimated'. repeatably computed, a reasonable assumption for the point sets we consider . For the cases where this does not hold, we will rely on the network's robustness. We will explain our method of choice in Sec. 4 and visualize LRFs of an airplane object in Fig. 1. Disentangling orientation from representations requires guaranteed equivariances and invariances. Yet, the original capsule networks of cannot achieve equivariance to general groups. To this end, proposed to use a manifold-mean and a special aggregation that makes sure that the trainable transformations get pose-aligned points as input. We will extend this idea to the non-abelian SO and design capsule networks sparsely operating on a set of LRFs computed on local neighborhoods of points, parameterized by quaternions. In the following, we first explain our novel capusle layers, the main building block of our architecture. We then show how to stack those layers via a simple aggregation ing in an SO-equivariant 3D capsule network that yields invariant representations (or activations) as well as equivariant rotations (latent capsules). To construct equivariant layers on the group of rotations, we are required to define a left-equivariant averaging operator A that is invariant under permutations of the group elements, as well as a distance metric δ that remains unchanged under the action of the group. For these, we make the following choices: Definition 7 (Geodesic Distance). The Riemannian (geodesic) distance in the manifold of rotations lead to the following geodesic distance δ(·) ≡ d quat (·): Definition 8 (Quaternion Mean µ(·)). For a set of Q rotations S = {q i} and associated weights w = {w i}, the weighted mean operator A(S, w): H 1 n × R n → H 1 n is defined through the following maximization procedure : where M ∈ R 4×4 is defined as: The average quaternion is the eigenvector of M corresponding to the maximum eigenvalue. This operation lends itself to both analytic and automatic differentiation . Theorem 1. Quaternions, the employed mean A(S, w) and geodesic distance δ(·) enjoy the following properties: Intermediate Capsules Rotation Estimation Figure 3: Our entire capsule architecture. We hierarchically send all the local patches to our Qnetwork as shown in Fig. 2. At each level the points are pooled in order to increase the receptive field, gradually reducing the LRFs into a single capsule per class. We use classification and pose estimation (in the siamese case) as supervision cues to train the point-to-transform maps. A 3. The transformations g ∈ H 1 preserve the geodesic distance δ(·) given in Dfn. 7. Proof. The proofs are given in the supplementary material. We also note that the above mean is closed form, differentiable and can be implemented batchwise. We are now ready to construct the group dynamic routing (DR) by agreement that is equivariant thanks to Thm. 1. The core idea is to route from or assign the primary capsules that constitute the input LRF set, to the latent capsules by an iterative clustering which respects the group structure. At each step, we assign the weighted group mean to each output capsule. The weights w ← σ(x, y) are inversely propotional to the distance between the vote quaternion and the new quaternion (cluster center). See Alg. 1 for details. In the following, we analyze our variant of routing as an interesting case of the affine, Riemannian Weiszfeld algorithm (; 2014). q−2 the equivariant routing procedure given in Alg. 1 is a variant of the affine subspace Wieszfeld algorithm (; 2014) that is a robust algorithm for computing the L q geometric median. Proof Sketch. The proof follows from the definition of Weiszfeld iteration and the mean and distance operators defined in Sec. 3.1. We first show that computing the weighted mean is equivalent to solving the normal equations in the iteratively reweighted least squares (IRLS) scheme . Then, the inner-most loop correspond to the IRLS or Weiszfeld iterations. We provide the detailed proof in supplementary material. Note that, in practice one is quite free to choose the weighting function σ(·) as long as it is inversely proportional to the geodesic distance and concave . We leave the analyses of the variants of these algorithms as a future work. The original dynamic routing can also be formulated as a clustering procedure with a KL divergence regularization. This holistic view paves the way to better routing algorithms. Our perspective is akin yet more geometric due to the group structure of the parameter space. Thanks to the connection to Weiszfeld algorithm, the convergence behavior of our dynamic routing can be directly analyzed within the theoretical framework presented by Aftab et al. (2014; . Theorem 2. Under mild assumptions provided in the appendix, the sequence of the DR-iterates generated by the inner-most loop almost surely converges to a critical point. Proof Sketch. Proof, given in the appendix, is a direct consequence of Lemma 1 and directly exploits the connection to the Weiszfeld algorithm. The essential ingredient of our architecture, QE-Network, is shown in Fig. 2 . We also provide a corresponding pseudocode in Alg. 3 of suppl. material. The input of the QE-Network are a local patch of points with coordinates x i ⊂ R K×3, rotations parametrized as quaternions and activations . q i also represents input primary capsules and local reference frames. N c is the number of input capsule channels per point and it is equal to the number of output capsules (M) from the last layer. In the initial layer, q i represents the pre-computed LRFs and N c is equal to 1. Given points x i and rotations q i, we compute the quaternion average µ i in channel-wise as the initial pose candidates:. These candidates are used to bring the receptive field in multiple canonical orientations by rotating the points: Since the points in the local receptive field lie in continuous R 3, training a discrete set of pose transformations t i,j based on local coordinates is not possible. Instead, we employ a point-to-transform network t(·): c ×4 that maps the point in multiple canonical orientations to transformations. The network is shared over all points to compute the transformations, which are used to calculate the votes for dynamic routing as v i,j = q i • t i,j. The network t(·) consists of fully-connected layers that regresses the transformations, similar to common operators for continuous convolutions (Schütt et al., 2017; . It is the continuous alternative to directly optimizing transformations lying in a grid kernel, as it is done in the original dynamic routing by . Note that t(·) predicts quaternions by unitnormalizing the regressed output:. Although Riemannian layers of Bécigneul & or spherical predictions of can improve the performance, the simple strategy works reasonably for our case. After computing the votes, we utilize the input activation α i to iteratively refine the output capsules (weighted average of votes)q i and activationsα i by routing by agreement as shown in Alg. 1. In order to gradually increase the receptive field, we stack QE-networks creating a deep hierarchy, pooling the points and the LRFs before each layer. Note that we are allowed to do so thanks to the properties of equivariance. In particular, we input N = 64 patches to our architecture that is composed of two QE-networks. We call the centers of these patches pooling centers. In the first layer, each of those centers is linked to their immediate vicinity leading to K = 9-star local connectivity from which we compute the 64 × 64 × 4 intermediary capsules. The input LRFs of the first layer are sampled from pre-calculated LRF-set and the input activation is set to 1. The LRFs in the second layer l = 2 are the output capsules of the first layer, l = 1 and are routed to the output capsules that are as many as the number of classes C, M 2 = C. The activation of the second layer is updated by the output of the first layer as well. This construction is shown in Fig. 3. Specifically, for l = 1, we use K = 9, N l c = 1, M l = 64 and for l = 2, K = 64, N l c = 64, M l = C = 40. This way, in this last layer all the pooling centers act as a single patch (K = 64). A single QE-network acts on this patch to create the final C × 4 capsules and C activations. More details are reported in Alg. 3 of the appendix. Implementation Details We implement our network in PyTorch and use the ADAM optimizer with a learning rate of 0.001. The point-transformation mapping network is implemented by two FC-layers composed of 64 hidden units. We set the initial activation of the input LRF to 1.0. In each layer, we use 3 iterations of DR. For classification we use the spread loss and the rotation loss is identical to δ(·). Surface normals are computed by local plane fits . We compute the second axis of the LRF, ∂ 2, by FLARE , that uses the normalized projection of the point within the periphery of the support showing the largest distance, onto the tangent plane of the center: ∂ 2 = pmax−p pmax−p. Note that using other LRFs such as SHOT or the more modern GFrames of is possible. We found FLARE to be sufficient for our experiments. Prior to all operations, we flip all the LRF quaternions such that they lie on the northern hemisphere: {q i ∈ S 3 : q w i > 0}. (a), Point2Seq (b), Spherical CNNs , PRIN and PPF-FoldNet (PPF) (a). We also present a version of our algorithm (Var) that avoids the canonicalization within the QEnetwork. This is a non-equivariant network that we still train without data augmentation. While this version gets comparable to the state of the art for the NR/NR case, it cannot handle random SO variations (AR). Note that PPF uses the point-pair-feature encoding and hence creates invariant input representations. For the scenario of NR/AR, our equivariant version outperforms all the other methods, including equivariant spherical CNNs by a significant gap of at least 5% even when uses the mesh. The object rotational symmetries in this dataset are responsible for a significant portion of the errors we make. It is worth mentioning that we also trained TFNs for that task, but their memory demand made it infeasible to scale to this application. Number of Parameters. Use of LRFs helps us to restrict the rotation group to certain elements and thus we can use networks with significantly less parameters (as low as 0.44M) compared to others as shown in Tab. 1. Number of parameters in our network depends upon the number of classes, e.g. for ModelNet10 we have 0.047M parameters. Rotation estimation in 3D point clouds. Our network can estimate both the canonical and relative object rotations without pose-supervision. To evaluate this desired property, we used the well classified shapes on ModelNet10 dataset, a sub-dataset of Modelnet40 . We generate multiple instances per shape by transforming the instance with five arbitrary SO rotations. As we are also affected by the sampling of the point cloud, we resample the mesh five times and generate different pooling graphs across all the instances of the same shape. Our QE-architecture can estimate the pose in two ways: 1) by directly using the output capsule with the highest activation, 2) by a siamese architecture that computes the relative quaternion between the capsules that are maximally activated as shown in Fig. 4. Both modes of operation are free of the data augmentation and we give further schematics of the latter in our appendix Fig. 5. Our against the baselines including a naive averaging of the LRFs (Mean LRF) and principal axis alignment (PCA) are reported in Tab. 2 as the relative angular error (RAE). We further include of and IT-Net , two state of the art 3D networks that iteratively aligns two point sets. These methods are in nature similar to iterative closest point (ICP) algorithm but 1) do not require an initialization (first iteration estimates the pose), 2) learn data driven updates. Methods that use mesh inputs such as Spherical CNNs cannot be included here as the random sampling of the same surface would not affect those. We also avoid methods that are just invariant to rotations (and hence cannot estimate the pose) such as Tensorfield Networks . Finally, note that, IT-net and PointLK need to train a lot of epoches (e.g. 500) with random SO rotation augmentation in order to get the models that cover the full SO, whereas we train only for ∼ 100 epochs. We include more details about the baselines in the appendix under Fig. 8. RAE between the ground truth and the prediction is computed as the relative angle in degrees: d(q 1, q 2)/π. Note that resampling and random rotations render the job of all methods difficult. However, both our version that tries to find a canonical alignment and the siamese variant which seeks a relative rotation are better than the baselines. As pose estimation of objects with rotational symmetry is a challenging task we also report on the non-symmetric subset (No Sym). Robustness against point and LRF resampling. Density changes in the local neighborhoods of the shape are an important cause of error for our network. Hence, we ablate by applying random resamplings (patch-wise dropout) to the objects in ModelNet10 dataset and repeating the pose estimation and classification as described above. The first part (LRF-10K) of Tab. 3 shows our findings against gradual increases of the number of patches. Here, we sample 2K LRFs from the 10K LRFs computed on an input point 10K. 100% dropout corresponds to 2K points in all columns. On second ablation, we reduce the amount of points on which we compute the LRFs, to 2K and 1K respectively. As we can see from the table, our network is robust towards the changes in the LRFs as well as the density of the points. Deep learning on point sets. The capability to process raw, unordered point clouds within a neural network is introduced by the prosperous PointNet (a) thanks to the point-wise convolutions and the permutation invariant pooling functions. Many works have extended PointNet primarily to increase the local receptive field size (b; b; ;). Point-clouds are generally thought of as sets. This makes any permutationinvariant network that can operate on sets an amenable choice for processing points . Unfortunately, common neural network operators in this category are solely equivariant to permutations and translations but to no other groups. Equivariance in Neural Networks. The early attempts to achieve invariant data representations usually involved data augmentation techniques to accomplish tolerance to input transformations (; ; 2017a). Motivated by the difficulty associated with augmentation efforts and acknowledging the importance of theoretically equivariant or invariant representations, the recent years have witnessed a leap in theory and practice of equivariant neural networks (; . While laying out the fundamentals of the group convolution, G-CNNs guaranteed equivariance with respect to finite symmetry groups. Similarly, Steerable CNNs and its extension to 3D voxels considered discrete symmetries only. Other works opted for designing filters as a linear combination of harmonic basis functions, leading to frequency domain filters (; b). Apart from suffering from the dense coverage of the group using group convolution, filters living in the frequency space are less interpretable and less expressive than their spatial counterparts, as the basis does not span the full space of spatial filters. Achieving equivariance in 3D is possible by simply generalizing the ideas of the 2D domain to 3D by voxelizing 3D data. However, methods using dense grids suffer from increased storage costs, eventually rendering the implementations infeasible. An extensive line of work generalizes the harmonic basis filters to SO by using e.g., a spherical harmonic basis instead of circular harmonics (b; ;). In addition to the same downsides as their 2D, these approaches have in common that they require their input to be projected to the unit sphere , which poses additional problems for unstructured point clouds. A related line of research are methods which define a regular structure on the sphere to propose equivariant convolution operators (a;) To learn a rotation equivariant representation of a 3D shape, one can either act on the input data or on the network. In the former case, one either presents augmented data to the network (a;) or ensures rotation-invariance in the input (a; b;). In the latter case one can enforce equivariance in the bottleneck so as to achieve an invariant latent representation of the input (; ; . Further, equivariant networks for discrete sets of views (b) and crossdomain views (a) have been proposed. Here, we aim for a different way of embedding equivariance in the network by means of an explicit latent rotation parametrization in addition to the invariant feature. Vector Field Networks, which was followed by the 3D Tensor Field Networks (TFN) that are closest to our work. Based upon a geometric algebra framework, the authors did achieve localized filters that are equivariant to rotations, translations and permutations. Moreover, they are able to cover the continuous groups. However, TFN are designed for physics applications, is memory consuming and a typical implementation is neither likely to handle the datasets we consider nor can provide orientations in an explicit manner. Capsule Networks. The idea of capsule networks was first mentioned by , before proposed the dynamic routing by agreement, which started the recent line of work investigating the topic. Since then, routing by agreement has been connected to several well-known concepts, e.g. the EM algorithm , clustering with KL divergence regularization and equivariance. They have been extended to autoencoders and. Further, capsule networks have been applied for specific kinds of input data, e.g. graphs , 3D point clouds or medical images . In this work, we have presented a new framework for achieving permutation invariant and SO equivariant representations on 3D point clouds. Proposing a variant of the capsule networks, we operate on a sparse set of rotations specified by the input LRFs thereby circumventing the effort to cover the entire SO. Our network natively consumes a compact representation of the group of 3D rotations -quaternions, and we have theoretically shown its equivariance. We have also established convergence for our Weiszfeld dynamic routing by making connections to the literature of robust optimization. Our network is among the few for having an explicit group-valued latent space and thus naturally estimates the orientation of the input shape, even without a supervision signal. Limitations. In the current form our performance is severely affected by the shape symmetries. The length of the activation vector depends on the number of classes and for achieving sufficiently descriptive latent vectors we need to have a significant number of classes. On the other side, this allows us to perform with merit on problems where the number of classes are large. Although, we have reported robustness to those, the computation of LRFs are still sensitive to the point density changes and resampling. LRFs themselves are also ambiguous and sometimes non-unique. Future work. Inspired by and our feature work will involve establishing invariance to the direction in the tangent plane. We also plan to apply our network in the broader context of 3D object detection under arbitrary rotations and look for equivariances among point resampling. A PROOF OF PROPOSITION 1 Before presenting the proof we recall the three individual statements contained in Prop. 1: Operator A is invariant under permutations: A({q σ,..., q σ(Q) }, w σ ) = A({q 1, . . ., q Q}, w). 3. The transformations g ∈ H 1 preserve the geodesic distance δ(·). Proof. We will prove the propositions in order. 1. We start by transforming each element and replace q i by (g • q i) of the cost in Eq: where M i = w i q i q i and p = G q. From orthogonallity of G it follows p = G −1 q =⇒ g • p = q and hence g • A(S, w) = A(g • S, w). 2. The proof follows trivially from the permutation invariance of the symmetric summation operator over the outer products in Eq. 3. It is sufficient to show that |q 1 q 2 | = |(g • q 1) (g • q 2)| for any g ∈ H 1: where g • q ≡ Gq. The is a direct consequence of the orthonormality of G. We will begin by recalling some preliminary definitions and that aid us to construct the connection between the dynamic routing and the Weiszfeld algorithm. Definition 9 (Affine Subspace). A d-dimensional affine subspace of R N is obtained by a translation of a d-dimensional linear subspace V ⊂ R N such that the origin is included in S: Simplest choices for S involve points, lines and planes of the Euclidean space. Definition 10 (Orthogonal Projection onto an Affine Subspace). An orthogonal projection of a point x ∈ R N onto an affine subspace explained by the pair (A, c) is defined as: c denotes the translation to make origin inclusive and A is a projection matrix typically defined via the orthonormal bases of the subspace. Definition 11 (Distance to Affine Subspaces). Distance from a given point x to a set of affine subspaces {S 1, S 2 . . . S k} can be written as: Lemma 2. Given that all the antipodal counterparts are mapped to the northern hemisphere, we will now think of the unit quaternion or versor as the unit normal of a four dimensional hyperplane h, passing through the origin: q d is an added term to compensate for the shift. When q d = 0 the origin is incident to the hyperplane. With this perspective, quaternion q i forms an affine subspace with d = 4, for which the projection operator takes the form: Proof. We consider Eq for the case where c = 0 and A = (I − qq). The former follows from the fact that our subspaces by construction pass through the origin. Thus, we only need to show that the matrix A = I−qq is an orthogonal projection matrix onto the affine subspace spanned by q. To this end, it is sufficient to validate that A is symmetric and idempotent: A A = AA = A 2 = A. Note that by construction q q is a symmetric matrix and hence A itself. Using this property and the unit-ness of the quaternion, we arrive at the proof: = I − 2qq + qq qq It is easy to verify that the projections are orthogonal to the quaternion that defines the subspace by showing proj S (q) q = 0: Also note that this choice corresponds to tr(qq) = d+1 i=1 α i = 1. Lemma 3. The quaternion mean we suggest to use in the main paper is equivalent to the Euclidean Weiszfeld mean on the affine quaternion subspaces. Proof. We now recall and summarize the L q -Weiszfeld Algorithm on affine subspaces , which minimizes a q-norm variant of the cost defined in Eq: Defining M i = I − A i, Alg. 2 summarizes the iterative procedure. Note that when q = 2, the algorithm reduces to the computation of a non-weighted mean (w i = 1 ∀i), and a closed form solution exists for Eq and is given by the normal equations: Compute the weights w t = {w t i}: Solve: For the case of our quaternionic subspaces c = 0 and we seek the solution that satisfies: It is well known that the solution to this equation under the constraint x = 1 lies in nullspace M i and can be obtained by taking the singular vector of M that corresponds to the largest singular value. Since M i is idempotent, the same can also be obtained through the eigendecomposition: which gives us the unweighted Quaternion mean. Once the Lemma 1 is proven, we only need to apply the direct convergence from the literature. Consider a set of points Y = {y 1 . . . y K} where K > 2 and y i ∈ H 1. Due to the compactness, we can speak of a ball B(o, ρ) encapsulating all y i. We also define the D = {x ∈ H 1 | C q (x) < C q (o)}, the region where the loss decreases. We first state the assumptions that permit our theoretical . Note that H5 is not a strict requirement as there are multiple ways to circumvent (simplest being a re-initialization). Under these assumptions, the sequence produced by Eq will converge to a critical point unless x t = y i for any t and i. For q = 1, this critical point is on one of the subspaces specified in Eq and thus is a geometric median. Note that due to the assumption H2, we cannot converge from any given point. For randomly initialized networks this is indeed a problem and does not guarantee practical convergence. Yet, in our experiments we have not observed any issue with the convergence of our dynamic routing. As our is one of the few ones related to the analysis of DR, we still find this to be an important first step. For different choices of q: 1 ≤ q ≤ 2, the weights take different forms. In fact, this IRLS type of algorithm is shown to converge for a larger class of weighting choices as long as the aforementioned conditions are met. That is why in practice we use a simple sigmoid function. For estimation of the relative pose with supervision, we benefit from a Siamese variation of our network. In this case, latent capsule representations of two point sets X and Y jointly contribute to the pose regression as shown in Fig. 5. Figure 5: Our siamese architecture used in the estimation of relative poses. We use a shared network to process two distinct point clouds (X, Y) to arrive at the latent representations (C X, α X) and (C Y, α Y) respectively. We then look for the highest activated capsules in both point sets and compute the rotation from the corresponding capsules. Thanks to the rotations disentangled into capsules, this final step simplifies to a relative quaternion calculation. We show additional from the computation of local reference frames and the multi-channel capsules deduced from our network in Fig. 6. Finally, the overall algorithm of our network is summarized under Alg. 3. Details on the evaluation protocol. For Modelnet40 dataset used in Tab. 1, we used the official split with 9,843 shapes for training and 2,468 different shapes for testing. For rotation estimation in Tab. 2, we used the official Modelenet10 dataset split with 3991 for training and 908 shapes for testing. 3D point clouds (10K points) are randomly sampled from the mesh surfaces of each shape Qi et al. (2017a; b). The objects in training and testing dataset are different, but they are from the same categories so that they can be oriented meaningfully. During training, we did not augment the dataset with random rotations. All the shapes are trained with single orientation (well-aligned). We call this trained with NR. During testing, we randomly generate multiple arbitrary SO rotations for each shape and evaluate the average performance for all the rotations. This is called test with AR. This protocol is used in both our algorithms and the baselines. We now report the confusion matrix in the task of classification on the all the objects of ModelNet10. The classification and rotation estimation affects one another. As we can see from Fig. 7, the first five categories that exhibit less rotational symmetry has the higher classification accuracy than their rotationally symmetric counterparts. Distribution of errors reported in Tab. 2. We now provide more details on the errors attained by our algorithm as well as the state of the art. To this end, we report, in Fig. 8 the histogram of errors that fall within quantized ranges of orientation errors. It is noticeable that our Siamese architecture behaves best in terms of estimating the objects rotation. For completeness, we also included the of the variants presented in our ablation studies: Ours-2kLRF, Ours-1kLRF. They evaluate the model on the re-calculated LRFs in order to show the robustness towards to various point densities. We have also modified IT-Net and PointNetLK only to predict rotation because the original works predict both rotations and translations. Finally, note here that we do not use data augmentation for training our networks (see AR), while both for PointNetLK and for IT-Net we do use augmentation. • the worse the performance of the method.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Deep architectures for 3D point clouds that are equivariant to SO(3) rotations, as well as translations and permutations.

Represent the TLDR for finding the corresponding scientific article:

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: represent the scientific article to find a short tldr

Question: Vector semantics, especially sentence vectors, have recently been used successfully in many areas of natural language processing. However, relatively little work has explored the internal structure and properties of spaces of sentence vectors. In this paper, we will explore the properties of sentence vectors by studying a particular real-world application: Automatic Summarization. In particular, we show that cosine similarity between sentence vectors and document vectors is strongly correlated with sentence importance and that vector semantics can identify and correct gaps between the sentences chosen so far and the document. In addition, we identify specific dimensions which are linked to effective summaries. To our knowledge, this is the first time specific dimensions of sentence embeddings have been connected to sentence properties. We also compare the features of different methods of sentence embeddings. Many of these insights have applications in uses of sentence embeddings far beyond summarization. Vector semantics have been growing in popularity for many other natural language processing applications. Vector semantics attempt to represent words as vectors in a high-dimensional space, where vectors which are close to each other have similar meanings. Various models of vector semantics have been proposed, such as LSA BID10, word2vec BID14, and GLOVE BID17, and these models have proved to be successful in other natural language processing applications. While these models work well for individual words, producing equivalent vectors for sentences or documents has proven to be more difficult. In recent years, a number of techniques for sentence embeddings have emerged. One promising method is paragraph vectors (Also known as Doc2Vec), described by BID12. The model behind paragraph vectors resembles that behind word2vec, except that a classifier uses an additional'paragraph vector' to predict words in a Skip-Gram model. Another model, skip-thoughts, attempts to extend the word2vec model in a different way BID9. The center of the skip-thought model is an encoder-decoder neural network. The , skip-thought vectors, achieve good performance on a wide variety of natural language tasks. Simpler approaches based on linear combinations of the word vectors have managed to achieve state-of-the-art for non-domain-specific tasks BID20. Arora et al. BID1 offer one particularly promising such approach, which was found to achieve equal or greater performance in some tasks than more complicated supervised learning methods. The large volume of news articles published every day motivates effective forms of automatic summarization. The most basic and most well-studied form of automatic summarization is sentence extraction, in which entire unmodified sentences are selected from the original document. These selected sentences are concatenated to form a short summary, which ideally contains the most important information from the original document while avoiding redundancy. Any practical summarization system would likely include other steps after sentence selection, such as sentence reordering or text-simplification. However, effective algorithms for these tasks already exist, and we are primarily concerned with sentence selection in this paper. We are primarily concerned with multi-document summarization algorithms. The nature of summarization requires some specific properties from sentence vectors, making it a practical real world example for which current sentence embedding methods can be easily used. In particular, news summarization may not require some of the nuances in meaning to be represented, as we are primarily concerned with the topic, not the meaning, of a sentence. In particular, news articles will rarely contain sentences expressing contradictory views on the same subject. For instance, our algorithm will rarely need to differentiate between the sentence embeddings for sentences such as "John loves Mary" and "Mary loves John", which have divergent meanings but the same words. This is in sharp contrast to typical testing cases for sentence embeddings, such as the detection of paraphrased sentences. Then, summarization gives us an opportunity to compare the effectiveness of different sentence embeddings in practice. State-of-the-art extractive summarization techniques have been achieved with a wide variety of methods. Here, we provide a short overview of some recent successful techniques techniques. Cao et al. BID2 used a recursive neural network, which operates on a parsing tree of a sentence, to rank sentences for summarization. BID3 successfully used a neural-network-based sentence extractor, which considered the document encodings and the previously selected sentences, as well as the current sentence in making its decisions. Parveen et al. BID16 used a graph-based approach, modeling a document as "a bipartite graph consisting of sentence and entity nodes".Ren et al. Ren et al. achieved state-of-the-art through a regression-based approach. A variety of engineered features are used as inputs into a regression model. The single highest rated sentence is selected, and the process is repeated. To our knowledge, no one has explored the use of modern sentence embedding methods, such as Paragraph Vectors or Skip-Thought vectors, in summarization. However, some work has been done on summarization using word2vec representations. BID6 presented a version of text summarization based on vector semantics. However, instead of using the embeddings of a word as determined by a larger corpus, they attempted to calculate the embedding of a word based off of analysis only on the document in question. In addition they used LSA instead of newer techniques such as word2vec. Kageback et al. BID8 used cosine similarity between the sum of word2vec embeddings, as well as a recursive auto-encoder, to modify another standard summarization algorithm (sub-modular optimization) Ren et al. Ren et al. used "Average Word Embedding" as one of many independent variables in a regression model, but the degree to which word embeddings effected the final model was not clear. BID3 used word embeddings as the input to a neural network as part of summarization, but they did not directly compare embeddings. They used a single-layer convolutional neural network to encode sentences, and a LSTM neural network to encode documents. BID15 recently achieved state-of-the-art using a modified version of LexRank, using a combination of cosine similarity of weighted averages of vectors and named entity overlap. To explore potential sentence embeddings, we implement the four sentence embeddings above as vector functions, which convert sentences or documents to vectors. • SIF Average: The most basic sentence embedding is simply the weighted average of word vectors from Arora et al. BID1, without the common component removal. We use the Brown corpus (Francis & BID5 for word frequency information.• Arora: This method is simply the method described in Arora et al. It is equivalent to the one above, except with common component removal added. We use the Brown corpus both to compute the common component vector, and for word frequency information.• Paragraph Vectors: The paragraph vector approach described above. We used the 300-dimensional DBOW model pretrained by Lau et al. BID11 on the wikipedia corpus.• Skip-Thought Vectors The skip-thought vector approach described above. We used the 4800-dimensional combined-skip model BID9 BID0 .All sentence embeddings are normalized to produce unit vectors To explore the design space, we consider combinations of vector functions and selector functions, functions which, given vector representations for sentences, extracts a summary. We present a large variety of example selector functions in order to allow us to explore interaction effects between selector functions and vector functions.• Near: The most basic selector function, Near, selects the sentences whose sentence vectors have the highest cosine similarity with the document vector• Near Nonredundant: An attempt at balancing redundancy with salience, Near Nonredundant down-weights the cosine similarity scores by their average cosine similarity the sentences selected so far. Because this redundancy measure is strongly (quadratically) correlated with cosine similarity to the document, we fit a regression for redundancy for each vector function, and use the residual on this regression for the final algorithm.• LexRank: Our next selector is based off of the classical LexRank algorithm BID4. We use a modified version of this algorithm, where the weights of edges are determined by the cosine similarity between sentence embeddings.• Cluster: We use an Agglomerative Clustering algorithm (using cosine similarity as its distance metric) to find clusters in the set of sentence embeddings. We then find the sentence closest to the average of each cluster and add it to the summary. To ensure we find summaries which meet the word-length requirement, we increase the number of clusters we search for until we have selected sentences totaling 100 words.• Greedy: The greedy selector, at each step, selects the sentence such that the cosine similarity of the new summary (including previously selected sentences) is maximized. This is subtly different than the Near selector for average-based vector functions, but significantly different for Paragraph Vectors.• Brute Force: Another attempt at optimizing the cosine similarity between the summary and the document, this selector creates a pool of the 20 sentences with the highest cosine similarity. From this pool, every combination of sentences (with an appropriate word count) is tried, and the combination with the highest cosine similarity is selected as the summary.• Max Similarity: A proof-of-concept selector which computes for both the Greedy and Brute Force selectors and then selects the with the highest cosine similarity to the document vector.• Near-then-Redundancy: Starting with the pool from the Brute Force algorithm, this algorithm optimizes via brute force to minimize redundancy (defined as the average cosine similarity between pairs of sentences). Note that the size of the sentence pool, which is essentially a computational shortcut in the Brute Force selector, is now a performance-critical hyper-parameter.• PCA: This selector performs Principal Component Analysis (PCA) on the set of sentence vectors in a document. Then, the algorithm selects the one sentence closest to the first component, one sentence closest to the second component, and so on, until the length capacity is met.• Random: This selector simply selects sentences at random, until the word limit is reached. This provides a lower-bound on the performance of an effective algorithm, and is used for baseline comparisons Because evaluation of summarization is fundamentally a subjective task, human evaluations are ideal. However, human evaluations are often expensive and time-consuming to obtain. Luckily, some metrics of automatically evaluating summaries, by comparison to a human-written summary, have been developed. Traditionally, various forms of the ROUGE metric, which compare shared n-grams, have been used. BID13. ROUGE has been shown to correlate strongly with human judgments BID18, and is our primary metric for evaluating summaries 1. We report ROUGE-1 and ROUGE-2 statistics, which correspond to unigrams and bigrams, respectively. We split the document clusters in the DUC 2004 dataset into a testing set and a validation set of approximately equal sizes. The pre-defined training set of the DUC 2001 dataset was used as a training set for some of the graphs and data analysis presented here. We present for Multi-Document Summarization on the DUC 2004 dataset TAB1. A few notes on the :• The best performing selector, Greedy, is both very simple and based on fundamental principles of vector semantics. • Paragraph Vectors work much worse with the Clustering and Greedy algorithms, and work much better with Near and SVMs.• Many combinations of selector function and vector function do not work above the level of random chance.• In general, despite their sophistication, Paragraph Vectors and Skip-Thought vectors perform worse than much more basic approaches. Despite the poor performance of our models compared to the baselines, analyses of the underlying data provide many useful insights into the behavior of vector semantics in real-world tasks. The cosine scores between all sentence vectors and the corresponding document vectors follow a normal distribution for all vector functions FIG0 ), but this effect is most pronounced for paragraph vectors (r 2 = .996). In addition, the sentence embeddings for paragraph vectors and skip-thought vectors are far closer to the document embedding than would be expected from a random distribution, with mean cosine similarities of.65 and.84, respectively (Unsurprisingly, this also holds for Average and Arora vectors). By identifying the sentences present in an optimal summarization, we show that optimal sentences have higher cosine scores, and that this effect is increased after adjusting cosine scores for word length (Fig. 2). However, there is a lot of overlap, implying that, although this method has some power to discern good summaries from bad summaries, the power of this method alone is not high enough to product good summaries. We calculated the isolated ROUGE score of each individual sentence in the training set, and the sentence embeddings for these sentences on all four vector functions. To partially eliminate the effects of the sentence's context, we subtract the corresponding document vector from all sentence vectors before regression. Due to the large number of predictor variables, we use a Bonferroni correction, considering values significant only if they have p-values of α n, which, for α =.05, corresponds approximately to p <.00001 for the skip-thought vectors, and p <.00016 for all other vectors. Three dimensions are significant at this level for SIF Average vectors. No dimensions are significant at this level for Arora vectors, though the three values significant for SIF Average achieve p values of.0021,.0485 and.0006. 29 dimensions are significant for Paragraph Vectors. 5 dimensions are significant, at the much higher threshold, for Skip-Thought Vectors. It appears that these specific dimensions correspond to aspects of a sentence that make it somehow more suited for a summary. Despite the theoretical implications of this , the regression models do not have enough predictive power to create good summaries by themselves. The Greedy algorithm is the most successful algorithm among those we have explored. As its name implies, the Greedy algorithm appears to be simply an attempt at maximizing the following objective function:f cos (summary) = vector(summary) · vector(document)Of course, any simple objective function can only be an approximation to the informally-defined criteria for good summaries. Even so, TAB1 suggests that the performance of the greedy algorithm is not based on the accuracy of the corresponding objective function. In particular, the two other strategies which try to maximize the same objective functio (Brute force, and Maximum Similarity) consistently and significantly create summaries with higher cosine similarity to the document, outperforming the Greedy selector on its objective function, but both of these algorithms perform much worse than the Greedy algorithm. Deeper analysis into the decisions of the Greedy algorithm reveals some reasons for this discrepancy. It appears that the good performance of the Greedy algorithm not from the associated objective function, but by the way in which it maximizes this objective function. In particular, the Greedy algorithm selects sentences with low cosine similarity scores in a vacuum, but which increase the cosine similarity of the overall sentence. To understand why this is true, it we consider the step-by-step behavior of the Greedy algorithm, which can be understood as maximizing the following equation (derived in Appendix A): DISPLAYFORM0 wheres p = i j=0s j i j=0s jNote that this equation consists of three parts: id ·s 1 (a constant wrt.s),d ·s, which is simply the salience of a sentence measured by cosine similarity, and the denominator, which is essentially a measure of redundancy. Not only does this simple metric lead to a'natural' penalty for redundancy, it performs better than our handcrafted redundancy penalties. The way this expression scales when picking the i th sentence is particularity noteworthy: the behavior of this function changes as i increases. In particular, the function becomes more sensitive to redundancy, and less sensitive to salience, as the algorithm selects more sentences. In other words, the algorithm will first try to select important sentences, and then select sentences to fill in the gaps. This , and the success of the ing algorithm, has implications for balancing salience and redundancy in future summarization algorithms. In general, there are two ways to compute a document vector. The most obvious is to pass the entire text of the document into the vector function. This has two theoretical problems. The first is that the'documents' in our algorithms are really clusters of documents, and are therefore non-coherent. The second is that Skip-thought vectors are not designed to handle text longer than a sentence. However, an alternative document vector, docvec-avg, is defined as the mean of the (normalized) sentence vectors. This corresponds to treating the document as a collection of sentences, instead of a collection of words. We compare the two here, and present full in Appendix B.As expected, Skipthought vectors, which are not designed for text larger than a sentence, perform significantly better with the docvec-avg strategy. More notable is the poor performance of the docvec-avg strategy with Paragraph Vectors. The size of the performance gap here implies that Paragraph Vectors can combine information from multiple sentences in a manner more sophisticated than simple averaging. More interesting is the performance for SIF Average and Arora vectors. For these vector functions, which are based on taking the average of words, docvec-avg very closely resembles the simple strategy. And yet there is a small but significant performance gap. The difference between the two document vectors is the weighting. Docvec-avg, which normalizes vectors before adding them together, removes some weighting information present in the simple strategy. In particular, the simple strategy assigns more weight to sentences with a lot of highly-weighted words. Presumably, docvec-avg, by ignoring this weighting, leaves out useful information. This hypothesis is supported by the greater performance gap for Arora vectors, which effectively downweights certain common words and therefore could be expected to carry more information in word weightings. Similar, but much smaller, gaps exist when computing the vectors for summaries at each step in the greedy algorithm. We present a broad comparison of the properties of different sentence embedding schemes.6.6.1 SIF AVERAGE/ARORA VECTORS Three selector functions perform better with both SIF Average and Arora vectors: Near Nonredundant, Greedy, and PCA. These functions seem to be unified by their comparisons between the vectors of sentence embeddings (implicitly, in the case of the greedy algorithm). These selector functions correspond to the most basic test for sentence embeddings: Judging the similarity of two sentences. The exact difference the common component removal makes is less clear. Arora vectors hold a slight performance edge for all selectors except for Near and Greedy (the Greedy algorithm loses a full 2 points). Two selector functions perform better with Paragraph Vectors: Near and Brute Force. Both of these are very similar: They require comparing sentence vectors to the document vector. The poor performance on algorithms such as Near Nonredundant suggests that Paragraph Vectors are especially poor at comparing sentence vectors to each other. These suggest that Paragraph Vectors are especially good at computing document vectors, a hypothesis also implied by the of Section 6.5. The other distinguishing property of Paragraph Vectors is their very high correlation when regressing on the individual features. It is hard to disentangle the properties of Skipthought vectors from the high dimensionality of the pretrained vectors we used. In general, Skipthought vectors performed poorly. They only performed better than other vector functions with one selector, Clustering, although their performance with this selector was significant. We have identified differences in different forms of sentence vectors when applied to real-world tasks. In particular, each sentence vector form seems to be more successful when used in a particular way. Roughly speaking, Arora's vectors excel at judging the similarity of two sentences while Paragraph Vectors excel at representing document vectors, and at representing features as dimensions of vectors. While we do not have enough data to pinpoint the strengths of Skipthought vectors, they seem to work well in specific contexts that our work did not fully explore. These differences are extremely significant, and will likely make or break real-world applications. Therefore, special care should be taken when selecting the sentence vector method for a real-world task. This material is based upon work supported by the National Science Foundation under Grant No. 1659788 and 1359275 A APPENDIX A The first choice of the greedy algorithm is simple: it chooses the sentence with maximum cosine similarity to the document vector:s 1 = argmax s∈Ss ·d (Recall that all vectors have unit-length, so cosine similarity is equivalent to the dot product).To select the second vector, the greedy algorithm is maximizing the following equation: DISPLAYFORM0

Represent the TLDR for finding the corresponding scientific article

A comparison and detailed analysis of various sentence embedding models through the real-world task of automatic summarization.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the scientific article to find a short TLDR:

Question: We present Value Propagation (VProp), a parameter-efficient differentiable planning module built on Value Iteration which can successfully be trained in a reinforcement learning fashion to solve unseen tasks, has the capability to generalize to larger map sizes, and can learn to navigate in dynamic environments. We evaluate on configurations of MazeBase grid-worlds, with randomly generated environments of several different sizes. Furthermore, we show that the module enables to learn to plan when the environment also includes stochastic elements, providing a cost-efficient learning system to build low-level size-invariant planners for a variety of interactive navigation problems. Planning is a key component for artificial agents in a variety of domains. However, a limit of classical planning algorithms is that one needs to know how to search for an optimal (or good) solution, for each type of plan. When the complexity of the planning environment and the diversity of tasks increase, this makes writing planners difficult, cumbersome, or entirely infeasible. "Learning to plan" has been an active research area to address this shortcoming BID13 BID5. To be useful in practice, we propose that methods for learning to plan should have at least two properties: they should be traces free, i.e. not require traces from an optimal planner, and they should generalize, i.e. learn planners that generalize to plans of the same type but of unseen instance and/or planning horizons. In a Reinforcement Learning (RL) setting, learning to plan can be framed as the problem of finding a policy that maximises the expected reward, where such policy is a greedy function that selects actions that will visit states with a higher value for the agent. In such cases, Value Iteration (VI) is a algorithm that is naturally used to learn to estimate the value of states, by propagating the rewards and values until a fixed point is reached. When the environment can be represented as an occupancy map (a 2D grid), it is possible to approximate this learning algorithm using a deep convolutional neural network (CNN) to propagate the value on the grid cells. This enables one to differentiate directly through the planner steps and perform end-to-end learning. One way to train such models is with a supervised loss on the trace from a search/planning algorithm, e.g. as seen in the supervised learning section of Value Iteration Networks (VIN) BID17, in which the model is tasked with reproducing the function to iteratively build values aimed at solving the shortest path task. However, this baseline violates our wished trace free property because of the required target values, and it doesn't fully demonstrate the capabilities to deal with interactive and generalized settings. That is what we set out to extend and further study. In this work we extend the formalization used in VIN to more accurately represent the structure of gridworld-like scenarios, enabling Value Iteration modules to be naturally used within the reinforcement learning framework, while also removing some of the limitations and underlying assumptions of the model. Furthermore we propose hierarchical extensions of such a model that allow agents to do multi-step planning, effectively learning models with the capacity to provide useful path-finding and planning capabilities in relatively complex tasks and comparably large scenarios. We show that our models can not only learn to plan and navigate in complex and dynamic environments, but that their hierarchical structure provides a way to generalize to navigation tasks where the required planning and the size of the map are much larger than the ones seen at training time. Our main contributions include: introducing VProp, a network module which successfully learns to solve pathfinding via reinforcement learning, demonstrating the ability to generalize, leading our models to solve large unseen maps by training exclusively on much smaller ones, and showing that our modules can learn to navigate environments with more complex dynamics than a static grid-world. We consider the control of an agent in a "grid world" environment, in which entities can interact with each other. The entities have some set of attributes, including a uniquely defined type, which describes how they interact with each other, the immediate rewards of such interactions, and how such interactions affect the next state of the world. The goal is to learn to plan through reinforcement learning, that is learning a policy trained on various configurations of the environment that can generalize to arbitrary other configurations of the environment, including larger environments, and ones with a larger number of entities. In the case of a standard navigation task, this boils down to learning a policy which, given an observation of the world, will output actions that take the agent to the goal as quickly as possible. An agent may observe such environments as 2D images of size d x × d y, with d pix input panes, which are then potentially passed through a local embedding function Φ: R dpix×dx×dy → R drew×dx×dy. The function could for instance be implemented as a 2D convolutional network that extracts the type and position of entities. This formalisation of the navigation problem is often employed in robotics to represent 3D and 2D surfaces, finding its use in, but not limited to, frameworks such as Simultaneous Localization and Mapping (SLAM), physics simulators, and more generic planning environments. This makes it appropriate for developing models and evaluating their impact on real-world planning tasks BID18. The problem of reinforcement learning is typically formulated in terms of the computation of optimal policies for a Markov Decision Problem (MDP) BID16 ). An MDP is defined by the tuple (S, A, T, R, γ), where S is a finite set of states, A is the set of actions a that the agent can take, T: s → a → s is a function describing the state-transition matrix, R is a reward function, and γ is a discount factor. In this setting, an optimal policy π * is a distribution over the state-action space that maximises in expectation the discounted sum of rewards k γ k r k, where r k is the single-step reward. A standard method (such as the one used in Q-Learning BID20) to find the optimal policy π: s → a is to iteratively compute the value function, Q π (s, a), updating it based on rewards received from the environment. Using this framework, we can view learning to plan as a structured prediction problem with the planning algorithm Value Iteration as inference procedure. When doing so, the problem generalises to learning a model of the environment that is approximating the state-transition probabilities and the immediate reward function. Beside value-based algorithms, there exist other types which are able to find optimal policies, such as policy gradient methods, which directly regress to the policy function π, instead of approximating the value function. Finally, a third type is represented by the actor-critic algorithms family, which combine the policy gradient methods' advantage of being able to compute the policy directly, with the low-variance performance estimation of value-based RL used as a more accurate feedback signal to the policy estimator BID7. Let us denote by o the current observation of the environment, and q 0 the zero tensor of dimensions (A, d x, d y). Then, the Value Iteration (VI) module proposed by BID17 is a recurrent neural network where each layer first performs max-pooling over the first dimension followed by a layer h: R drew×dx×dy × R dx×dy → R A×dx×dy, giving rise to the following recurrence, for k ≥ 1: DISPLAYFORM0 Given the agent's position (x 0, y 0) and the current observation of the environment o, the control policy π is then defined by π(o, (x 0, y 0)) = argmax a=1.. A q ax0y0.Since the state space can be identified with the coordinates in a 2-dimensional environment of size d x × d y then a discounted MDP on that state is defined by the discount factor γ together with:• a transition probability function DISPLAYFORM1 such that the matrix obtained fixing the last three coordinates sum to 1.• an immediate reward function R:= A × d x × d y, where R a,i,j represents the reward obtained by performing action a in state i, j. Given a starting Q-function Q 0 (with same size as R), value iteration defines a sequence Q k, V k of (state, action)-and state-value functions through: DISPLAYFORM2 It follows that when d rew:= A and Φ(x):= R, h is a linear layer of both of its inputs with T as parameters for the v k input: DISPLAYFORM3 Thus, a linear VI module with K iteration has the capacity to represent the application of K iterations of value iteration in an MDP where the state space is the set of 2D coordinates, requiring in the worst case a number of steps equal to the number of states -d x d y in grid worlds -to evaluate an entire plan. In practice however, much fewer iterations are needed in the considered tasks: for instance path finding in a grid-world requires planning for only the length of the shortest path, which is much smaller unless the configuration corresponds to a very complicated (and unusual) maze. We can also see that corresponds to only a very special case of linear VI modules. For instance, if the recurrence uses a fully connected layer with weights W ∈ R A×dx×dy×(drew+1)×dx×dy and biases b ∈ R A×dx×dy so that DISPLAYFORM4 then not only corresponds to a special case with extremely sparse weights, but also exhibits a specific structure where the dot product is non-trivial only on the recurrent part. This relationship ultimately forms the motivation for the development of VI modules BID17, Section 3.1). Model-based planning with end-to-end architectures has recently shown promising on a variety of tasks and environments, often using Deep Reinforcement Learning as the algorithmic framework BID14 BID11 BID21 BID3. 3D navigation tasks have also been tackled within the RL framework BID9, with methods in some cases building and conditioning on 2D occupancy maps to aid the process of localization and feature grounding BID1.Other work has furthermore explored the usage of VIN-like architectures for navigation problems: BID10 present a generalization of VIN able to learn modules on more generic graph structures by employing a graph convolutional operator to convolve through each node of the graph. BID12 demonstrate a method for multi-agent planning in a cooperative setting by training multiple VI modules and composing them into one network, while also adding an orientation state channel to simulate non-holonomic constraints often found in mobile robotics. BID4 and BID6 propose to tackle partially observable settings by constructing hierarchical planners that use VI modules in a multi-scale fashion to generate plans and condition the model's belief state. It is clear that all such work can indeed greatly benefit from employing a powerful planner with capable of learning in a larger combination of tasks and environments, motivating us to develop one which can directly substitute VIN. In this section we formalize two novel alternatives to the Value Iteration module, both of which can function as drop-in replacements for VIN and be used as low-level planners in graph-based tasks. A problematic and key limitation of the VI module is that the transition probabilities are encoded as the weights W of the convolution, which are naturally translation-invariant. This means that the transition probabilities at one state, a cell in this case, do not depend on its content or the surrounding ones'. This restriction affects the capability of the model to learn any complex dynamics, while also artificially constraining the design of the architecture and other experimental parameters. For instance, non-achievable state must be implemented so that they can be represented by a sufficiently low cost to compensate for any surrounding positive cost. We can obtain a minimal parametrization of a VI module that represents reasonable prior knowledge on the model by simply observing that the transition probabilities must depend on the state we are in, and that the rewards associated to transitions from state s to s with action a takes an additive form r in s − r out s, where r in and r out depend on the state at hand. We can now therefore consider a variant of the VI module that puts emphasis on conditioning on a different aspect of the grid's environment dynamics: state reachability. Our new module, which we call Value Propagation module (VProp), takes as input three embeddings: a value r in (o), a cost r out (o), and a propagation p(o), all of which are scalar maps of size R dx×dy. Using these functions as input, the ing output is then computed as DISPLAYFORM0, max DISPLAYFORM1 where v (k), r in, r out and p depend on the observation o, and N (i, j) represents the coordinates of the cells adjacent to the cell (i, j). Note that here we made the convolution and the maximum explicit, since they can be done in a single step. VProp's model corresponds to a deterministic model in which the reward propagates from adjacent states to current states, capturing the prior that the reachability of a state does not depend on the adjacent cell the agent is in, but by the underlying -and less constraining -observed transition dynamics. As such, instead of setting the probability to get to a unachievable state to 0 from its adjacent cells, the model represents unachievable states by setting their propagation to 0 and r in to some negative value.2 Goal states, which are also absorbing for the underlying MDP, are represented with a propagation close 0, and a positive r in, whereas other types of cell are to have high propagation value bounded by the discount factor, while the cost of their traversal is represented by either a negative r in or a positive r out. Finally, given the agent's coordinates s, the agent policy becomes a function that takes as input the vector DISPLAYFORM2, and either outputs an action or a distribution over actions. Since both VI module and VProp module described above are purely convolutional, they should be able to exploit the lack of size constraints in the input to generalize to larger environments than seen during training time by simply increasing the recursion depth. However, despite the convolutions effectively hard-coding the local planning algorithm, these models cannot be proven to generalize properly across the entire distribution of environment configurations. To see this, let us consider the simple problem of pathfinding. Even in a simple grid-world with blocks and a single goal, there are many sets of parameters of the VI / VProp module that are equally good for pathfinding with a fixed environment size. For instance, in an environment of size 20 × 20 in which paths are of length 40 with very high probability (over the randomized configurations of the environment), we can trivially check that a VProp module with a r in value of 1 for the goal entity and 0 otherwise, r out to −0.021 Note that while we use a minus sign, we do not impose specific sign constraints on r in and r out, since natural environments can easily be represented with both values as positive.2 Absorbing states can be represented in the same way.for empty cells, −1 for blocks and −1 for the goal state, and with propagation set to 1, everywhere, will effectively create a value map v (K) which is affinely related to the shortest path to the goal on empty cells. Thus, as far as the training distribution of environments is concerned, this would be an acceptable set of parameters. However, if we increase the size of the environment such that paths are strictly longer than 50, the goal value would become 1 − 51 × 0.02, which is less than 0, hence stopping the propagation of the path after the 51st step. The model is therefore bound to the capacity of the additive rollout in the value map, which is responsible for both identifying the goal and computing the shortest path. To solve this problem, we further introduce the Max-Value Propagation module (MVProp), which constrains the network to represent goals with a high base reward, propagates them multiplicatively through cells with lower values but high propagation, and learns reward and propagation maps depending on the content of the individual cells in the same fashion as VProp. More precisely, denoting again the initial reward map by r(o) ∈ dx×dy and the propagation map by p(o) ∈ dx×dy, MVProp iteratively computes value maps v (k) (o) as follows: DISPLAYFORM0 The input to the policy, given the agent position s = (i 0, j 0), is then the 3 × 3 map of (v DISPLAYFORM1, padded with 0 at the boundaries. This propagation system guarantees that values are propagated in the direction of lower-value cells at all times (even at the start of training), and that costs of individual cells are dealt with the 1 − p map. In other words, the path length is propagated multiplicatively, whereas the reward map is used to distinguish between goal cells and other cells. Given this setup, the optimal policy should therefore be able to locally follow the direction of maximum value. we focus on evaluating our our modules strictly using Reinforcement Learning, since we are interested in moving towards environments that better simulate tasks requiring interaction with the environment, hence limiting the chance to gather representative datasets. For training, we use an actor-critic architecture with experience replay. We collect transition traces of the form (s t, a t, r t, p t, s t+1), where s t is the state at time step t, a t is the action that was chosen, p t is the vector of probabilities of actions as given by the policy, and r t is the immediate reward. The state is represented by the coordinates of the agent and 2D environment observation, excluding any agent channel. Terminal states are instead represented by some value ∅ when sampled as s t+1. The architecture contains the policy π θ described in the previous sections, together with a value function V w, which takes the same input as the softmax layer of the policy, concatenated with the 3x3 neighborhood of the agent. Note that w and θ share all their weights until the end of the convolutional recurrence. At training time, given the stochastic policy at time step t denoted by π θ t, we sample a minibatch ofB transitions, denoted B, uniformly at random from the last L transitions, and perform gradient ascent over importance-weighted rewards (more details on the constants η, η, λ, C below): DISPLAYFORM0 where 1 {s =∅} is 1 if s is terminal and 0 otherwise. The capped importance weights min DISPLAYFORM1 p(a), C are standard in off-policy policy gradient (see e.g., BID19 and references therein). The capping constant (C = 10 in our experiments) allows control of the variance of the gradients at the expense of some bias. The second term of the update acts as a regularizer and forces the current predictions to be close enough the the ones that were made by the older model. It is supposed to play the role of TRPO-like regularizations as implemented in BID19, where they use probabilities of an average model instead of the previous probabilities. We observed that by keeping only the last 50000 transitions (to avoid trying to fit predictions of a bad model) worked well on our task. The update rule for the parameters of the value function also follows the standard rule for off-policy actor-critic. The learning rates η, λ and η also control the relative weighting of the different objectives when the weights are shared. In practice, we use RMSProp rather than plain SGD, with relative weights λ = η = 100.0η. Comparison between a random map of the VIN dataset, and a few random configuration of our training environment. In our custom grid-worlds, the number of blocks increases with size, but their percentage over the total available space is kept fixed. Agent and goal are shown as circles for better visualization, however they still occupy a single cell. Our grid-world experimental setting is mostly identical to the ones used in previous work on the same topic: an agent and a goal are placed at random positions in a 2d world of fixed dimensions. The agent is allowed to move in all 8 directions at each step, and a terminal state is reached when the agent either reaches the goal or hits one of the walls randomly placed when the task is sampled from the task distribution. The ratio of unwalkable blocks over total space is fixed to 30%, and the blocks are sampled uniformly within the space, unless specified otherwise (Figure 1). This setting provides environments with a good distribution of configurations, with chunks of obstacles as well as narrow and dangerous paths. Unless specified otherwise, attempting to walk into walls yields a reward of −1, valid movements give a reward of −0.01 × f (a t, s t), where f (a, s) is the cost of moving in the direction specified by action a in state s, and reaching the goal provides a positive reward of 1 to the agent. In our experiments we define f as the L2 distance between the agent position at state s t, and the one at s t+1. The agent is tasked to navigate the maze as fast as possible, as total cost increases with time since noop actions are not allowed. Episodes terminate whenever the agent takes any illegal action such as hitting a wall, or when the maximum number of steps is reached (which is set to 50). We use MazeBase BID15 to generate the configurations of our world and the agent interface for both training and testing phases. Additionally we also evaluate our trained agents on maps uniformly sampled from the 16 × 16 dataset originally used by BID17, so as to get a direct comparison with the previous work, and to confirm the quality of our baseline. We tested all the models on the other available datasets (8 × 8 and 28 × 28) too, without seeing significant changes in relative performance, so they are omitted from our evaluation. Similar to BID17, we employ a curriculum where the map size doesn't change at training time, but where the average length of the optimal path from the starting agent position is bounded by some value which gradually increases after a few training episodes. This allows the agent to more easily encounter the goal at initial stages of training, allowing for easier conditioning over the goal feature. Across all our tests, both VProp and MVProp greatly outperformed our implementation of VIN (15.6% ± 5.3% 68.8% ± 27.2% 100% 0.8 ± 0.3 0.4 ± 0.3 0.0 ± .0 64×64 4.0% ± 4.1% 53.2% ± 31.8% 100% 1.5 ± 0.4 0.5 ± 0.4 0.0 ± .0 TAB0 : Average performance at the end of training of all tested models on the static grid-worlds with 90% confidence value, across 5 different training runs (with random seeding). v16x16 correspond to the maps sampled from VIN's 16x16 grid test dataset, while the rest of the maps are sampled uniformly from our generator using the same parameters employed at training time. The distance to the optimal path is averaged only for successful episodes.randomly. The original VIN architecture was mostly tested in a fully supervised setting (via imitation learning), where the best possible route was given to the network as target. In the appendix, however, BID17 show that VIN can perform in a RL setting, obtaining an 82.5% success rate, versus the 99.3% success rate of the supervised setting on a map of 16 × 16. The authors do not show for the larger 28 × 28 map dataset, nor do they provide learning curves and variance, however overall these are consistent with the best performance we obtained from testing our implementation. That said, on average the model didn't perform as well as expected. Figure 2: Average reward of all the models as they train on our curriculum. Note again that in both cases the map size is 32 × 32. a and c demonstrate performances respectively on the VIN dataset and our generated 64x64 maps. b shows performance on evaluation maps constrained by the curriculum settings (segmented line), and without (continuous line).The final average performances of each model against the static-world experiments TAB0 clearly demonstrate the strength of VProp and MVProp. In particular, MVProp very quickly learns the correctly approximate transition dynamics, ing into strong generalization right from the first few hundreds episodes, hence obtaining near-optimal policies during the first thousand training episodes. Finally, we propose a set of experiments in which we allow our environment to spawn dynamic adversarial entities. Such entities at each step query a custom policy, which is executed in parallel to the agent's. Examples of these policies might include a -noop strategy, which makes the entity move in random direction with probability or do nothing, or a -direction policy, which makes the entity move to a specific direction with probability or do nothing. We use the first category of policies to augment our standard path-planning experiments, generating enemies_only environments where 20% of the space is occupied by agents with = 0.5, and mixed environments with the same amount of entities, half consisting of fixed walls, and the remaining of agents with = 0.2 The latter type of policies is instead used to generate a deterministic but continuously changing environment which we call avalanche, in which the agent is tasked to reach the goal as quickly as possible while avoiding "falling" entities (which also are also uniformly sampled up to 20% of the walkable area).To deal with these new environments, the agent needs to re-plan at each step because the agent does not have any prior knowledge about the entities policies, which forces us to train on 8x8 maps to reduce the time spent rolling-out the convolutional modules. This however allows us to train without curriculum, as the agent is more likely to successfully hit the goal in a smaller area. FIG1 shows that VProp can successfully learn to handle the added stochasticity in the environment dynamics, and its generalization capabilities allow it to also successfully learn to tackle the avalanche maps on larger sizes FIG2 ). VIN agents however while managing to learn policies able to tackle small sizes, gradually lose performance on larger sizes, which become significantly harder to deal with unless the dynamics are modelled correctly. Architectures that try to solve the large but structured space of navigation tasks have much to benefit from employing planners that can be learnt from data, however these need to quickly adapt to local environment dynamics so that they can provide a flexible planning horizon without the need to collect new data and training again. Value Propagation modules' performances show that, if the problem is carefully formalized, such planners can be successfully learnt via Reinforcement Learning, and that great generalization capabilities can be expected when these models are built on convnets and are correctly applied to 2D path-planning tasks. Furthermore, we have demonstrated that our methods can even generalize when the environments are dynamics, enabling them to be employed in complex, interactive tasks. In future we expect to test our methods on a variety of tasks that can be embedded as graph-like structures (and for which we have the relevant convolutional operators). We also plan to evaluate the effects of plugging VProp into architectures that are employing VI modules (see Section 3), since most of these models could make use of the ability to propagate multiple channels to tackle more complex interactive environments. Finally, VProp architectures could be applied to algorithms used in mobile robotics and visual tracking BID2, as they can learn to propagate arbitrary value functions and model a wide range of potential functions.

We propose Value Propagation, a novel end-to-end planner which can learn to solve 2D navigation tasks via Reinforcement Learning, and that generalizes to larger and dynamic environments.

We view exploration in RL as a problem of matching a marginal distribution over states.