Question: Represent the scientific article to find a short TLDR

Question: In this paper, we propose a novel approach to improve a given surface mapping through local refinement. The approach receives an established mapping between two surfaces and follows four phases: (i) inspection of the mapping and creation of a sparse set of landmarks in mismatching regions; (ii) segmentation with a low-distortion region-growing process based on flattening the segmented parts; (iii) optimization of the deformation of segmented parts to align the landmarks in the planar parameterization domain; and (iv) aggregation of the mappings from segments to update the surface mapping. In addition, we propose a new method to deform the mesh in order to meet constraints (in our case, the landmark alignment of phase (iii)). We incrementally adjust the cotangent weights for the constraints and apply the deformation in a fashion that guarantees that the deformed mesh will be free of flipped faces and will have low conformal distortion. Our new deformation approach, Iterative Least Squares Conformal Mapping (ILSCM), outperforms other low-distortion deformation methods. The approach is general, and we tested it by improving the mappings from different existing surface mapping methods. We also tested its effectiveness by editing the mappings for a variety of 3D objects. C OMPUTING a cross-surface mapping between two surfaces (cross-parameterization) is a fundamental problem in digital geometric processing. A wide range of methods have been developed to find such mappings,,, but no single method in a perfect mapping in every case. Quite often, the mapping may be good overall, but some specific, sometimes subtle, semantic features, such as articulations and facial features, may remain misaligned, as illustrated in Figure 1. These imperfections of the final are often unacceptable in a production setting where the artist needs a high degree of control over the final , and will often sacrifice automation of a method for higher control. Typically, improving using surface mapping methods requires the user to iteratively insert some landmarks and solve for the mapping globally. However, since the imperfections are typically localized to a specific region, a local solution that does not change the mapping globally would be preferred in order to ensure that the method does not introduce artifacts elsewhere on the map. This paper proposes a surface mapping editing approach providing local and precise control over the map adjustments. The process begins with the inspection of an existing vertex-topoint surface mapping between two meshes. In regions where the mapping exhibits some discrepancy, the user sets landmarks positioned at corresponding locations on both meshes. For each such region, we extract a patch on both meshes in order to localize the changes in the mapping, and we flatten them on a common planar domain. The mapping is improved based on a 2D deformation optimization that steers the landmarks toward correspondence while limiting distortion and having theoretical guarantees to maintain the local injectivity of the map. We developed a new 2D deformation approach denoted Iterative Least Squares Conformal Maps (ILSCM), which iteratively minimizes a conformal energy, each iteration ensuring that flips do not occur, and in practice, ensuring progress toward satisfying the constraints. We chose to work with a conformal energy as we want to be able to improve mappings where the deformation between the pair of meshes is not isometric. Our editing approach can successfully align the mapping around landmarks without any degradation of the overall mapping. The local surface maps are extracted from their respective deformed segments and parameterization domains, and are then combined to form an improved global surface mapping. Our approach solves an important practical problem and offers three novel scientific contributions. The first is a practical approach for local surface map editing, which we show, using both qualitative and quantitative metrics, provides better than other stateof-the-art methods. The second involves a compact segmentation which in a compromise between a low-distortion flattening and a low-distortion deformation when aligning the landmarks. The third is a new deformation approach, ILSCM, which preserves conformal energy better than other state-of-the-art methods, and that has theoretical guarantees preventing the introduction of foldovers. While a lot of research has been done on creating correspondences between 3D objects, comparatively fewer methods have been proposed on correspondence editing. Nguyen et al. measure and optimize the consistency of sets of maps between pairs belonging to collections of surfaces. They compute a score for the map, and then apply an optimization to iteratively improve the consistency. The limitation of their method lies in its requirement of having multiple maps instead of a single map for a pair of surfaces. Ovsjanikov et al. propose the functional maps representation to establish correspondences between surfaces based on Laplacian eigenfunctions rather than points. Working in that smooth basis function space makes it easy and efficient to generate smooth mappings, but significant modifications to the underlying method would be required to allow local adjustments of the mapping guided by the user. Another limitation is that their method is limited to near-isometric surfaces since non-isometric deformation overrides the assumption that the change of basis matrix is sparse. Ezuz and Ben-Chen remove the isometric restriction in their proposed The initial mapping between mesh A and B is globally good, but is locally misaligned in the head region. Using our approach, the mapping is locally improved (c). method for the deblurring and denoising of functional maps. They smooth a reconstructed map by mapping the eigenfunctions of the Laplacian of the target surface in the span of the source eigenfunctions. Their technique can be incorporated into existing functional mapping methods, but selecting the right number of eigenfunctions to perform the denoising is difficult. Compared with a ground truth mapping, increasing the number of eigenfunctions decreases the error until a minimum is reached, but adding more eigenfunctions beyond this point increases the error. While this can be observed on a ground truth mapping, there are no methods to achieve the minimum error for an arbitrary mapping. Gehre et al. incorporate curve constraints into the functional map optimization to update the mapping between non-isometric surfaces. They provide an interactive process by proposing a numerical method which optimizes the map with an immediate feedback. While their method is not limited to the isometric surfaces, it does however need several curve constraints to obtain a meaningful functional map. Vestner et al. improve dense mappings even in the case of non-isometric deformations. Their method is an iterative filtering scheme based on the use of geodesic Gaussian kernels. An important restriction of their method is that it requires both surfaces to be discretized with the same number of vertices and vertex densities. Panozzoet al. propose the weighted averages (WA) on surfaces framework. They use WA with landmarks in order to define mappings and then the user can improve the mapping by adjusting the landmarks. Although WA generates good mapping , its improved mapping application cannot use a mapping as input. Since most state-of-the-art methods improve the mapping globally, this makes it hard for the user to fine-tune the mapping without risking modifying areas that should not be affected. Furthermore, some methods face significant limitations such as being constrained to isometric deformations and requiring compatible meshes on both surfaces. An alternative way to frame the correspondence editing problem is as a deformation method in a planar parameterization space. Least Squares Conformal Maps (LSCM) apply a deformation energy which contains a term for preserving surface features and a term for position constraints. In contrast, Jacobson et al. provide a smooth and shape-preserving deformation using biharmonic weights. The main drawback of the LSCM and biharmonic weights methods is that they can introduce fold-overs while deforming the mesh. Injectivity is a key property we want to achieve in our mapping editing, but extracting a mapping from meshes with fold-overs breaks this property. The method of Chen and Weber avoids fold-overs by applying inequality constraints solely to the boundaries. Locally Injective Mappings (LIM) and Scalable Locally Injective Mappings (SLIM) propose a strategy where the energy of a triangle tends to infinity as the triangles become degenerate, and thus, any locally minimal solution will be by construction exempt of fold-overs. While this is an elegant approach, the problem is that in some cases where, due to the user constraints, some triangles will come close to becoming degenerate, they carry a disproportionately high share of the total energy as compared to the rest of the triangles. For our mapping editing application, such cases occur frequently, and we observed that in their presence, LIM and SLIM often produce an inferior both qualitatively and quantitatively. Gollaet al. outperform LIM and SLIM by modifying the Newton iteration for the optimization of nonlinear energies on triangle meshes. They analytically project the per-element Hessians to positive semidefinite matrices for efficient Newton iteration and apply global scaling to the initialization. In this work, we propose to edit a surface mapping by locally adjusting the mapping in a bid to align landmarks set by the user. To move the landmarks toward their expected positions, we deform a local segmented patch of the mesh. We found that current deformation methods had drawbacks (flipped triangles and high distortion) forbidding their use in our mapping editing framework. We thus derived a new deformation approach that iteratively minimizes a conformal energy, making sure that in each iteration we have no flipped triangles. More specifically, our ILSCM approach optimizes the quadratic LSCM energy, but it relaxes the user constraints to avoid flips. Therefore, after each iteration, the user constraints may not be satisfied, but by repeating the process, we reach a configuration that has low conformal energy (lower than LIM or SLIM), and the user constraints are guaranteed to be better satisfied than initially. In practice, the user constraints are always satisfied up to a user-provided epsilon. Our approach opens the door to a new family of deformation methods that optimize the deformation by effectively finding an optimal flow of the vertices based on conformal energy minimization. As explained in the introduction, mappings computed by state-ofthe-art (automatic) methods are often globally good, but locally wrong in a few areas. We provide an approach to locally improve the surface mapping. The user typically inspects the mapping visually through texture transfer. In local regions where the mapping should be improved, the user sets landmarks at locations that should correspond on the pair of surfaces (Fig. 2a). We edit the mapping by deforming parts of the meshes with respect to each other (Fig. 2c) to improve the alignment of the user-provided landmarks, and then we rebuild the mapping from the deformed parts (Fig. 2d). Our main goal is to obtain a low distortion of the meshes at each phase of our approach. To deform the mapping with a good control on distortion, we conduct a planar parameterization of the meshes. Since the planar parameterization of smaller segments of a mesh leads to less distortion versus when computed for the entire mesh, we do a segmentation based on user-identified local regions where the mapping needs to be updated. Afterwards, we deform one segment with respect to the other in the planar parameterization space. The deformation is aided by our new ILSCM, ensuring that the deformation causes limited distortion on the meshes. Finally, the mapping is extracted based on how segments overlap each other. Our approach has four key phases: Our method works with input meshes A and B, along with a vertexto-point surface mapping between the meshes. The mapping links each vertex of a mesh to a barycentric coordinate on the other mesh. It should be noted that our approach works regardless of the initial method used to establish the surface mapping, as long as a dense mapping is provided. Given a mapping, the user will visualize it using texture maps to identify mismatching regions. These correspond to isolated zones of the meshes where the mapping is incorrect. For each region, the user sets corresponding landmarks on both meshes at locations that should match each other. The landmarks for each region i, are expressed as barycentric coordinates on A and B, respectively. The user provides hints where the mapping needs to be modified by setting pairs of landmarks on both meshes. In order to keep the map editing local, a segment is identified on both meshes where the map editing will be performed. Computing such a segment is not trivial as there are a number of requirements: the segment should be a single connected component with disk topology, should be compact, and should contain all the landmarks of the region i. The size of the segment is also important. If the segment is too large we may lose locality, but if it is too small, we may introduce further distortion if the vertices need to move over a long distance. We assume that outside these segments, the mapping is satisfactory, and it can be used to set boundary conditions when deforming a segment with respect to the other to align the landmarks. Our segmentation method has three steps. In the first step (Fig. 3a), we grow an initial patch on the 3D surface from the landmarks, ensuring that it is one connected component, that it encloses all of the landmarks, as well as the positions corresponding to the landmarks from the other mesh. We flatten this patch in 2D (Fig. 3b), where we have more tools available to control the size and shape of the patch. In the second step, we compute a compact 2D patch from the convex hull of the landmarks in the 2D space (Fig. 3c), and ensure that we fill any artificial internal boundaries. In the third step, we grow the compact patch from the previous step to allow enough room between the boundary and the landmarks (Fig. 3d), preparing the ground for a low-distortion deformation phase (Sec. 3.3). This segmentation is applied to each region of meshes A and B independently. We now explain in more detail the process as applied to one region i of mesh A, but the same is also conducted for mesh B and other regions. Based on the mapping from mesh B to A, corresponding landmark positions are calculated on A for each landmark of L B (i), The goal of the first step is to extract an initial patch and to flatten it. To meet the two conditions of having a single connected component and containing all of the landmarks from L A (i) and CP B→A (i), we will compute the union of face groups computed by identifying faces around each landmark l A (i) j and around each corresponding position cp B→A (i) j. For each, we iteratively add rings of faces until the group of faces contains at least half plus one of the landmarks from L A (i) and CP B→A (i). The requirement to include half plus one of the landmarks ensures that when we combine the groups of faces, this initial patch meets the two conditions. This procedure in a "disk with holes" topology, which is sufficient to flatten the patch using ABF++. One disadvantage of the initial patch is that it can contain concavities, and even internal "holes" with polygons from the full mesh missing. From the initial patch of step one, the second step extracts a compact patch that surrounds the landmarks. To this end, we identify the convex hull of the landmarks in the 2D parameterization space. Then, we only consider the faces which have at least one of their vertices within the convex hull (faces identified in black in Fig. 3c). The use of the convex hull in a patch exempt of large concavities in its boundary. Nevertheless, depending on the meshes and the arrangement of landmarks, some of the initial patches have "holes" with polygons from the full mesh missing, creating artificial internal boundaries in the patch (Fig. 4). We add the missing faces by analyzing the inner boundaries (holes). Filling the whole by adding the missing faces from the full mesh has the advantage of preventing unwanted deformation that would from such artificial boundaries, and it ensures that there are no internal areas within the region where the mapping would not be adjusted. The third step tries to balance the conflicting goals of having a small versus a large patch. As can be seen in Fig. 5, the larger the patch, the greater the distortion between the patch in 3D and after flattening to 2D. The distortion would be even higher if flattening the whole mesh (Fig. 6). Conversely, a smaller patch means that the landmarks are closer to the boundary, and the deformation that aligns the landmarks will induce more distortion to the triangles between the boundary and the landmarks. We thus want to grow the compact patch to ensure that there is enough room around each landmark and corresponding landmark position pairs to diffuse the distortion from the deformation phase. We also know that as we get closer to the landmarks, we are getting closer to the areas where the mapping is wrong, and as such, extending outwards is necessary in order to have a good boundary condition for the deformation. Regarding how far away the patch should be extended, we use a distance proportional to the geodesic distance (on the 3D mesh) between the landmark and its corresponding landmark position, adding faces that are within that distance from each landmark of the pair. We compared the distortion between the 3D patch and the 2D patch for 1 to 10 times the geodesic distance. Fig. 7 shows a pattern where very small patches do not have enough room to move the landmarks without high distortion. At the same time, patches that are too large also exhibit large distortion because of the flattening from 3D to 2D. A good compromise between the two sources of distortion is around two times the geodesic distance. It is necessary to apply steps one and two, because step three alone could lead to disconnected components or artificial internal boundaries that do not exist in the mesh, but that exist in the patch because of the patch growth process (Fig. 4). The sequence of steps one to three provides the final segments from A, {A 1, A 2, . . ., A n}, and the same for mesh B, yielding {B 1, B 2, . . ., B n}. These final segments are flattened through ABF++ and we refer to them as A 1, A 2,..., A n and B 1, B 2,..., B n. We selected ABF++ to flatten our segments as we can assume it will yield a parameterization exempt of flipped triangles (injective). Our main observation is that the initial surface mapping is globally adequate, but wrong in localized regions. With this assumption, we line up the boundary of the regions by relying on the surface mapping. We will then deform the interior to align the landmarks while keeping a low distortion. As we have two segmented meshes, there are two ways to align the landmarks: deform A i on B i or B i on A i. We select the one with the lower L 2 distortion (between the segment in 3D and 2D), keeping it fixed and deforming the other. Here, we will explain the deformation of B i with respect to A i, but the deformation of A i with respect to B i proceeds in the same way. We deform B i in order to implicitly adjust the mapping by applying an energy minimization. This is achieved by using positional user constraints (E L -aligning the landmarks, E B -aligning the boundary) coupled with a distortion preventing regularization (E D -globally deforming B i with low distortion), leading to the following equation: The user constraints are by soft constraints as follows: where k(i) are the landmarks of the segment i; l a j are the landmarks on A i; vertices v (j,1), v (j,2), and v (j,3) correspond to the three vertices of the triangle on B i containing the related landmark l b j; and β (j,1), β (j,2), and β (j,3) are the barycentric coordinates. We use Ω(B i) to denote the set of vertices on the boundary of B i, and map(v j) to denote the corresponding position of v j on A i based on the mapping. The energy E B pulls the vertices of the boundary of B i to the positions on A i where they correspond given the mapping. When λ is small, the map will be injective, but the constraints are generally not satisfied (ultimately, if λ = 0, B i stays the same). Conversely, when λ is large, the user constraints are satisfied, but flips may be introduced (λ = ∞ corresponds to using hard constraints). An ideal deformation energy E D (V) must meet three criteria: preserving the shape, maintaining the injectivity of the mapping (i.e., no flipped triangles), and satisfying the user constraints. This is an example where we want to adjust the mapping by moving the red landmarks to the positions of the corresponding green landmarks. All deformations were done with the same patch boundary (the one from the ABF parameterization). as much as possible. For shape preservation, we experimented with several E D: LSCM, LIM, SLIM, and KPNewton. Each of the energies has a number of pros and cons. LSCM preserves the shape the best, but tends to introduce flips, as illustrated in Fig. 8 (c). LIM, SLIM, and KP-Newton on the other hand, guarantee injectivity (no flips), but introduce more distortion (between B i before and after deformation) than LSCM. The graph in Fig. 9 illustrates these observations: LSCM has the least distortion, but flipped triangles would destroy the injectivity of the mapping. LIM, SLIM, and KP-Newton have no flips, but overall, they have more distortion as compared to LSCM. LIM minimizes a joint energy where one term optimizes the distortion and the second term optimizes the flips. In such joint optimization frameworks, no one term may be close to a minimum of its own, as shown in Fig. 9, where the from LIM are worse in terms of the distortion energy than LSCM. Fig. 10. This graph compares in terms of the residual error of LSCM energy for fixed values of λ. The horizontal axis labels present λ and the number of iterations (λ, #iter). The black bars highlight with flipped triangles. Since all these methods have shortcomings, we propose an approach that bridges the gap between the shape preservation of the original LSCM formulation with the injectivity preservation of LIM, SLIM, and KP-Newton. Our approach, Iterative LSCM (ILSCM), is a different approach where we iteratively optimize to decrease E(V), while preventing flips from occurring. ILSCM performs iterative LSCM steps. The first iteration uses the cotangent weights from segment B i. The deformed segment from the first iteration is then used to set the weights for the second iteration, and so on. At each iteration, if a triangle flip is detected, we decrease the value of λ and redo the same iteration. This way, we are guaranteed to eventually find a λ that prevents flips from occurring. We will now explain how we adaptively adjust λ to guarantee that we have no flips, while making as much progress as possible toward achieving the user constraints. In order to measure if the constraints are satisfied, we consider the initial maximal distance between any landmark and corresponding landmark position pair dist 0 = max j l A (i) j − cp B→A (i) j, and iterate until the current maximal distance is below the threshold ε = dist 0 /250. The Appendix A demonstrates that since the progression of landmarks is continuous with respect to λ, the approach will always find a λ that prevents having any flips and that enables progress toward the user constraints. The progress could asymptotically stop, but in all cases, we are guaranteed to prevent triangles from flipping and we limit the mesh distortion. For the mapping adjustment application, all of the examples we tested converged to meet the user constraints. A larger λ will converge faster, but increases the likelihood of flipped triangles (Fig. 10, black bars). A small λ decreases the probability of flipped triangles, but increases the number of iterations needed to satisfy the user constraints. As can be seen in Fig. 10, whether using a small or large λ, the conformal residual is almost the same and it plateaus for smaller values of λ. Consequently, even in the theoretical case where our approach would take very small incremental steps, the solution remains valid in the sense that it meets the constraints and the conformal residual remains close to the solution with larger steps. In our experiments, we start with λ = 1000. After each iteration, we automatically detect if there are flipped triangles, and if so, we redo the deformation of the iteration with λ = λ /2. Fig. 11a demonstrates that the movement of a landmark is continuous with respect to different values of λ. Fig. 11b further shows that even with small values of λ, we make progress toward satisfying the constraints. We see that we make progress toward satisfying the user constraints even for small values of λ. We also see that it is to our advantage to begin with a large value of λ to reduce the number of iterations before convergence. As the last phase of our approach, we update the surface mapping between A and B from the planar parameterizations. We first extract the mappings from each pair (B i, A i). Then, we aggregate and transfer them to A and B. With the mapping being expressed as barycentric coordinates on the other mesh, we can update it by simply getting the barycentric coordinates of vertices from B i to faces of A i and vice-versa. We validate our approach with various cases of faces, as well as with a wider range of objects with different morphologies from different data sets and artist contributions (see Table 1). Experiments are presented based on different initial mapping methods: orbifold tutte embeddings, Elastiface, deformation transfer, functional mapping method, WA method, and joint planar method. The number of landmarks and segments is proportional to the quality of the initial surface mapping and the complexity of the objects (see Table 1). We evaluate the capabilities of our approach based on a qualitative evaluation by visual inspection and a quantitative evaluation based on geodesic distance. This method prevents flipped triangles, and, essentially, it preserves the shape, while satisfying the user constraints. Furthermore, it distributes the deformation error more uniformly across the mesh surface. As can be seen in Fig. 9, our distortion energy is lower than SLIM, is often lower than LIM, and is only slightly greater than LSCM. We believe that this optimization strategy is more suited to this type of problem than a joint optimization strategy. Fig. 12 compares LSCM, LIM, SLIM, and KP-Newton to our ILSCM (iterated 257 times and final λ was 31.25) for the example from Fig. 8. ILSCM distributed errors more uniformly over the whole deformed mesh, as compared to LIM, SLIM, and KP-Newton. The accompanying video shows how our iterative approach progressively conducts the deformation, in comparison to LSCM, LIM, SLIM, and KP-Newton. The meshes we deform in the video are the same as some of the examples from the LIM paper. For a fair comparison, we perform SLIM, LIM, and KP-Newton, all using the LSCM energy. For LIM, we apply a 1E−12 barrier weight, which is sufficient to prevent flips. We experimented with barrier weights of LIM ranging from 1E−4 to 1E−20. Barrier weights smaller than 1E−12 had an imperceptible impact, while those equal to or lower than 1E−20 did not converge. For each deformation energy, we experimented with two different initial states: weights from the 3D triangles and weights from the flattened B i. The distortion between B i before and after deformation was lowest when deforming using the weights from the flattened B i. We thus used the weights from the flattened B i. Visual inspection of is a common form of validation for mapping problems. We use a visualization method based on texture transfer. We copy texture coordinates from one mesh to the other using mapping, setting the uv coordinates of a vertex to the barycentric interpolation of the uv coordinates of the other mesh. For this visualization, we used two different types of textures. The first type was a grid texture. Figs. 1, 20, and 13 qualitatively show that we obtain considerably better mappings using our editing approach. An important assumption of our approach is that we can edit the mapping locally. This implies that it is important to have a smooth transition at the boundary of the regions where we conduct local editing. Fig. 14 shows a typical example of the smoothness of our edited mapping across the boundary of the segmented region. The accompanying video also compares the transition of the mapping across the boundary by transferring texture from mesh A to mesh B using both initial mapping and edited mapping for the test cases of Fig. 13, Fig. 20 (top row), and Fig. 21. For the specific case of faces, we use realistic facial textures, making it easier to highlight important semantic facial features. These features are derived from three important considerations: modeling, texturing, and animation. A realistic facial texture is often enough to highlight modeling and texturing issues. Problems around the nose (Figs. 15 and 23), lips (Figs. 15 and 23), and These examples show cases that are ideal for our approach: the initial mappings are globally good, with few local misalignments. Instead of solving for the mapping globally, our approach provides a local solution for these specific semantic regions. For facial animation, other features need to be identified in the textures. Accordingly, some of our texture visualizations use a set of curves that are positioned relative to the areas that deform during animation, based on the facial anatomy. Fig. 15 illustrates the improvement in the correspondence of these animation-related features as compared against the initial surface mapping. Our approach assumes that the segments can be flattened to 2D without any flipped triangles. While the hypothesis is essential to get injective mappings, our approach is still robust to cases where the flattened segments would contain flipped faces. Meshes used in the industry often exhibit small discrepancies such as cracks, holes, and handles. Fig. 16 presents such a case where one of the meshes is of a different genera (contains two handles in the ear region). Although it is not possible to get injective mappings when dealing with different genera, our approach behaves robustly: it can improve the mapping in the region with different genera and it does not degrade the mapping in the edited region nor in its vicinity. Furthermore, even if it is not possible to achieve injective mappings in such cases, our edited mappings have reasonable properties: the mapping from the lower genera (A → B) is injective, and the mapping from the higher genera (B → A) is surjective. While the qualitative evaluations of Sec. 4.1 demonstrate that our approach in clear improvements, we also quantitatively measure how our approach improves the mappings. We first use the same process as in the paper of Kim et al. in order to measure the accuracy of the surface mapping. Their method transfers vertex positions to the other mesh using the mapping under evaluation and a ground truth mapping. It then computes the geodesic distances from the corresponding positions. Fig. 17 shows the error of the initial mapping after our editing approach. The comparative evaluation shown here relies on the ground truth mapping from SCAPE (Fig. 17 (a) ) and TOSCA (Fig. 17 (b) ) data sets. We can see that applying our approach improves the mapping in the related regions without causing a degradation of the overall mapping. Another way to measure the quality of a mapping is to morph a mesh into the shape of the other using the mapping. Then, we evaluate the mapping by computing L 2 and L in f between the mesh and the morphed mesh to estimate the distortion which occurs in the mapping-based morphing process. Fig. 18 shows the morphing of mesh A into mesh B using both the initial and new mappings. With our updated mapping (Fig. 18 (d) ), the vertices of the head A are pulled back to the correct place. This has the advantage of mapping the right density of vertices where needed, which is very important for morphing and in any transfer related to animation attributes (e.g., bones, vertex weights, and blend shapes). Table 2 illustrates an evaluation of the quality of the edited mapping in comparison to the initial mapping between 3D shapes. It shows that our edited mapping is as good as or better than the initial mapping when considering the distortion of the morphed mesh. We can see that there is a single case where this measurement of distortion is slightly higher after the map is edited. Even in this case, while the distortion is slightly higher, the edited mapping is clearly superior, as can be seen in Fig. 13. We performed a qualitative comparison of the mapping editing versus the methods of Ezuz and Ben-Chen. We also did comparisons using LIM, SLIM, KM-Newton, and ILSCM to conduct the mapping editing. We finally compared local editing to global editing using the method of Panozzo et al. and the joint planar method. For the comparison to the method of Ezuz and Ben-Chen, we established an initial mapping using a state-of-the-art functional mapping method. Note that we use the raw functional map, without the high-dimensional iterative closest point post-process refinement. Fig. 19 compares the mappings improved using our approach and the method of Ezuz and Ben-Chen (which improves the mapping without any landmark). Note how the added control of the landmarks provides a significantly improved mapping, exactly where intended. Fig. 21 presents when LIM, SLIM, KP-Newton, and ILSCM are used to conduct the mapping editing. The comparison through texture transfer visualization shows that ILSCM is superior in adjusting the mapping as compared to LIM, SLIM, and KPNewton. The accompanying video also compares LIM, SLIM, KP-Newton, and ILSCM in editing the mapping for the test case of Fig. 13. Adjusting the mapping globally requires having the initial constraints, the initial parameters of the method, and the method itself, which is constraining. In addition, some mapping methods, such as that of Nogneng and Ovsjanikov, do not let the user guide the process with landmarks, while others, such as OBTE, only support a fixed number of landmarks (three or four landmarks for OBTE ), which will be insufficient in many cases. Furthermore, we believe that it is advantageous to ensure that changes occur locally, avoiding unexpected changes elsewhere in the mapping. Fig. 20c (top row) shows that solving for the mapping globally is sometimes as effective as solving it locally. Conversely, Fig. 20c (bottom row) shows that improving the mapping globally introduced artifacts on the fish head as compared to our local refinement (Fig. 20d bottom row), which is exempt of such artifacts away from the edited region. Fig. 22 compares the mappings improved using our approach as compared to solving globally using the WA method of Panozzo et al.. We established an initial mapping using the WA method. Afterwards, with the WA method, we added two additional landmarks to improve the initial mapping. For our method, we only consider the two new landmarks in improving the mapping. It can be seen in Fig. 22 that editing the mapping locally was beneficial for this test case as well. Several applications rely on a mapping between surfaces: texture transfer, animation setup transfer, and deformation transfer. We use the methods of Sumner et al. and Avril et al. to illustrate how the proposed approach can significantly improve the of techniques relying on a mapping. Fig. 23 shows a facial transfer before and after editing. Results demonstrate several issues and unpleasant deformations for fine features, such as strange deformations on the corners of the mouth. With the corrected mapping, these problems disappear. Fig. 24 shows a skeleton transfer before and after the mapping is edited. Results demonstrate that the joint that was erroneously positioned outside of the thumb moves to the right place when improving the surface mapping locally in the thumb region instead of improving the mapping globally over the mesh. Our approach works even for surfaces with boundaries inside the segments. Such boundaries are commonly encountered with the ears, eyes, nostrils, and mouths of characters. While we constrain the segment boundaries to prevent them from moving, an initial mesh boundary lying inside a segment will be free to move. Leaving these inner boundaries completely free has a negative impact on the deformation. Fig. 25 shows the deformation of the mouth without (c) and with (d) inner boundary fillings. Note here the improvement of the mouth deformation when filling the inner boundary. Our approach carves a new path in between the more classical shape-preserving methods, which often lose local injectivity, and the more current methods, which formulate the injectivity constraint as part of the optimization. These latter approaches typically do not have a bound on the shape-preserving error. In our approach, we are minimizing only the shape-preserving term (i.e., LSCM energy) and iteratively improving the user constraints while maintaining a locally injective map in each iteration. We achieve this by carefully controlling the λ parameter in Eq. 1. At one extreme, if λ is very large (i.e., infinity), the formulation is equivalent to the LSCM formulation. If λ is very small, it takes many iterations for the user constraints to be satisfied, or in some cases, the user constraints may ultimately not be satisfied. Our iterative scheme relies on two important observations. If λ is 0, the solution is the same as the initial configuration. Therefore, if we start in a locally injective configuration, the final will be a locally injective configuration. If the initial configuration is locally injective, there always exists a λ (however small) that will in a locally injective configuration, where the user constraints are closer to the target. This scheme will converge to a locally injective configuration. Consequently, we iteratively repeat the optimization to fight against flipped faces, but convergence cannot be guaranteed. It is always possible to design a landmark configuration in which the constraints cannot be met without flipped faces. This is true for the other deformation methods as well. Appendix B demonstrates different failure cases using different deformation methods. In our experiments, the constraints are satisfied (up to numerical precision), even for extreme deformations. In our , we improved mappings which were initially computed from a variety of methods,,,,,. Even if these initial mappings minimize different deformation energies, the fact that we rely on the LSCM conformal energy to edit them did not prevent our approach to improve the mappings. One must keep in mind that the goal of the editing is not to strictly minimize a deformation energy, but to align important semantic features of the objects and maintain injectivity. We analyzed our to verify the degree to which the deformation deteriorates the shape of the triangles. We checked 13 of the found in this paper, and we considered that a detrimental deformation is one in which the angle becomes more than 20 times narrower after deformation. Eleven cases had no such triangles, while the two other cases had two and three, respectively. The worst triangle in our 13 test cases was 24 times narrower than before deformation. Any deformation method is prone to in thin triangles, so we compared our approach to LIM, SLIM, and KP-Newton for six examples. When looking at the worst triangle found in the deformed meshes, ILSCM performed best for four of the test cases, while KP-Newton performed best for two of the test cases. SLIM and LIM were systematically in third and fourth place behind ILSCM and KP-Newton. Furthermore, our were better than LIM, SLIM, and KP-Newton in terms of shape preservation and final triangulation, as can be seen in Fig. 12 and in the video. We ran our experiments on a 3.40 GHz Intel Core-i7-4770 CPU with 12 GB of memory. The presented approach was implemented with MATLAB, taking advantage of its sparse matrices and linear solvers. Table 1 shows computation times for the segmentation and the deformation (including mapping extraction) phases. Since our deformation phase is an iterative method, the time to edit a mapping depends on the size of the mismatching regions and iterations. We have presented a novel approach for improving surface mappings locally. Our approach is based on a low-distortion region-growing segmentation followed by an independent planar parameterization of each segment. The mapping is then optimized based on an alignment of the user-prescribed landmarks in the parameterization space of each segment. Our joint planar parameterization deformation for the segments is robust, and in low distortion. Our new iterative LSCM approach can be reused in several contexts where a deformation with low distortion is required. From a practical perspective, our approach has several (a) Mesh A (b) Mesh B, initial mapping (c) Mesh B, WA (d) Mesh B, our edited mapping (e) Mesh A (f) Mesh B, initial mapping (g) Mesh B, WA (h) Mesh B, our edited mapping advantages. It can be used to improve the mapping ing from (a) Mesh A skeleton (b) Mesh B, initial skeleton (c) Mesh B, edited skeleton Fig. 24. When using the mapping to retarget attributes, in this case the skeleton, an incorrect mapping will lead to problems, here putting the thumb joint outside of the mesh. By locally editing the mapping, it is easy to fix such issues. any surface mapping method. It also provides a great deal of control, allowing the user to restrict editing to a specific region and to add as few or as many landmarks as necessary to achieve a desired . Our local editing leads to interesting questions which open many avenues for future work. One such prospective area is higherlevel landmarks such as lines. This will lead to challenges in terms of easing the interactive placement of these lines on both meshes, but will provide a better set of constraints for the deformation. Another avenue would be to extend the scope to editing deformation transfer. This will combine deformation with editing and enable the user to control animation retargeting. To ensure that there is always a single solution, even if λ is arbitrarily small, we add a new term E B to Eq. 1: where v old j denotes the position of vertex v j at the previous iteration. The energy E B pulls the vertices of the boundary to where they correspond given the mapping. The term E B pulls the vertices on the boundary of B i to their position at the previous iteration. Eq. 3b is weighted by a small constant ξ = 0.001 such that in practice the vertices will converge to map(v j). The previous position v old j is initialized with the position on the boundary of the ABF of B i. Our deformation method proceeds iteratively by finding a sequence of 2D embeddings V i of a given patch. We show that if the initial embedding of the mesh V 0 has no fold-overs, then the ing embedding at every iteration V i also has no fold-overs. We prove this by induction. The base case for i = 0 is given by the hypothesis, and thus, we are showing that if V i has no foldovers, our procedure will yield a configuration V i+1 that also has no fold-overs. At every iteration, the new set of vertex positions V i+1 is obtained by solving Eq. 1: arg min, where V i is the embedding in the current iteration, V i+1 is the new embedding we are computing, andλ > 0 is a parameter of the algorithm, constant w.r.t. this minimization. We selectλ as follows: we create a monotonically decreasing positive sequence λ j > 0 such that lim j→∞ (λ j) = 0; we solve the optimization problem for the λ j in the sequence and stop at the first element in the sequenceλ = λ k that yields a fold-over free configuration, and we now show that such a λ k always exists. Let B(x) ∈ R n×n, x ∈ R and B i j (x) is continuous in x ∀i, j. Lemma.1. det(B(x)) is a continuous function in x. Proof: We prove by induction on n. If n = 1, B(x) is a continuous real function. det(B(x)) = B 1 1 (x) is also continuous. We assume that the statement is true for n − 1 and we prove for n. We write det(B(x)) using the Laplace formula: where M i j is the minor of the entry (i, j) defined as the determinant of the sub-matrix obtained by removing row i and column j from B. As each element of this matrix is also continuous in x and this reduced matrix is n − 1 × n − 1, it follows from the inductive hypothesis that M i j is also continuous in x. As det(B(x)) is obtained by using addition and multiplication of continuous functions, it follows that det(B(x)) is continuous in x. Proof: If det(B(x)) = 0 ∀x then B(x) is invertible ∀x the inverse of a matrix has the following analytic expression: where C is the matrix of co-factors: C i j = (−1) i+ j · M i j and M i j is the minor of the entry (i, j). M i j (x) is continuous in x from Lemma.1. It follows trivially that C i j is continuous in x. Since det(B(x)) = 0 ∀x it follows from Corollary 1 that is continuous in x. Since B −1 (x) is obtained by multiplying a scalar function continuous in x by a matrix whose entries are all continuous in x, it follows that B −1 i j (x) is continuous in x ∀i, j. V i+1 (λ) is the minimizer of a quadratic energy function, and therefore, it has the standard least squares analytical solution: where the matrix A and vector b are computed from Eq. 1 in a standard way for a least squares solution. The matrix A(λ) has the following structure: where: 1) A 1 corresponds to E L and is a k × n matrix that encodes the landmark constraints of the patch. These constraints are weighted by λ (Eq. 2a). 2) A 2 corresponds to E B and is a b × n matrix that encodes the boundary constraints of the patch. These constraints are weighted by λ (Eq. 2b). 3) A 3 corresponds to E B, and is a 2×n matrix that constrains two boundary vertices to their positions in the previous iteration. These constraints are weighted by a small constant ξ independent of λ (Eq. 2c). 4) A 4 corresponds to E D and is the m × n matrix from the original LSCM formulation, where n is the number of vertices in the patch and m > n. Lemma Proof: The LSCM paper shows that if we constrain exactly 2 vertices, we obtain a unique solution, which means that: has rank n. Since the matrix A 3 4 does not depend on λ, it follows that rank(A 3 4) = n ∀λ. Since the rank of A 3 4 cannot be larger than n, and the rank of the ing matrix does not decrease by adding rows to the matrix, it follows that when stacking A 1 and A 2 to A 3 4 to form the final matrix A, rank(A(λ)) = n ∀λ. Since the rank of the Gramm matrix is the same as that of the matrix, it follows that rank(A t (λ) · A(λ)) = n ∀λ. Since A t (λ) · A(λ) is a n × n matrix, this means that A has full rank, therefore det(A(λ)) = 0 ∀λ. Lemma.5. V i+1 (λ) is continuous in λ. Proof: λ = 0 reduces the linear system to only A 3 and A 4. Theorem.7. ∃k > 0 s.t. V i+1 (λ k) has no fold-overs. Proof: From Lemma.6, if λ = 0, then V i+1 has no fold-overs. Since V i+1 is continuous in λ (Lemma .5), it follows that for all vertex positions ∃λ > 0 s.t. ∀λ, 0 < λ <λ,V i+1 (λ) has no foldovers. Since the sequence λ j is monotonically decreasing and lim j→∞ (λ j) = 0, it follows that ∃k s.t.0 < λ k <λ. It follows that V i+1 (λ k) has no fold-overs. By proving Theorem.7, we show that at every iteration, our embedding V i+1 has no fold-overs and thus yields an injective map. Our deformation method guarantees progress toward meeting the landmarks, constraints free of flipped faces, but it cannot guarantee that the user constraints will be satisfied. In fact, there are cases where it is impossible to meet these constraints, such as the example in Fig. 26. There are also "hard" cases (Fig. 27) where, while it might be possible to find a deformation that meets the constraints, deformation methods, such as LIM, SLIM, KP-Newton and our approach, are not able to find it.

Represent the TLDR for finding the corresponding scientific article

We propose a novel approach to improve a given cross-surface mapping through local refinement with a new iterative method to deform the mesh in order to meet user constraints.

We analyze and develop a computationally efficient implementation of Jacobian regularization that increases the classification margins of neural networks.


Question: Represent the scientific article to find a short TLDR

Question: Understanding the groundbreaking performance of Deep Neural Networks is one of the greatest challenges to the scientific community today. In this work, we introduce an information theoretic viewpoint on the behavior of deep networks optimization processes and their generalization abilities. By studying the Information Plane, the plane of the mutual information between the input variable and the desired label, for each hidden layer. Specifically, we show that the training of the network is characterized by a rapid increase in the mutual information (MI) between the layers and the target label, followed by a longer decrease in the MI between the layers and the input variable. Further, we explicitly show that these two fundamental information-theoretic quantities correspond to the generalization error of the network, as a of introducing a new generalization bound that is exponential in the representation compression. The analysis focuses on typical patterns of large-scale problems. For this purpose, we introduce a novel analytic bound on the mutual information between consecutive layers in the network. An important consequence of our analysis is a super-linear boost in training time with the number of non-degenerate hidden layers, demonstrating the computational benefit of the hidden layers. Deep Neural Networks (DNNs) heralded a new era in predictive modeling and machine learning. Their ability to learn and generalize has set a new bar on performance, compared to state-of-the-art methods. This improvement is evident across almost every application domain, and especially in areas that involve complicated dependencies between the input variable and the target label BID19. However, despite their great empirical success, there is still no comprehensive understanding of their optimization process and its relationship to their (remarkable) generalization abilities. This work examines DNNs from an information-theoretic viewpoint. For this purpose we utilize the Information Bottleneck principle BID37. The Information Bottleneck (IB) is a computational framework for extracting the most compact, yet informative, representation of the input variable (X), with respect to a target label variable (Y). The IB bound defines the optimal trade-off between representation complexity and its predictive power. Specifically, it is achieved by minimizing the mutual information (MI) between the representation and the input, subject to the level of MI between the representation and the target label. Recent BID35, demonstrated that the layers of DNNs tend to converge to the IB optimal bound. The pointed to a distinction between two phases of the training process. The first phase is characterized by an increase in the MI with the label (i.e. fitting the training data), whereas in the second and most important phase, the training error was slowly reduced with a decrease in mutual information between the layers and the input (i.e. representation compression). These two phases appear to correspond to fast convergence to a flat minimum (drift) following a random walk, or diffusion, in the vicinity of the training error's flat minimum, as reported in other studies (e.g. BID39).These observations raised several interesting questions: (a) which properties of the SGD optimization cause these two training phases? (b) how can the diffusion phase improve generalization perfor-mance? (c) can the representation compression explain the convergence of the layers to the optimal IB bound? (d) can this diffusion phase explain the benefit of many hidden layers?In this work we attempt to answer these questions. Specifically, we draw important connections between recent inspired by statistical mechanics and information-theoretic principles. We show that the layers of a DNN indeed follow the behavior described by BID35. We claim that the reason may be found in the Stochastic Gradient Decent (SGD) optimization mechanism. We show that the first phase of the SGD is characterized by a rapid decrease in the training error, which corresponds to an increase in the MI with the labels. Then, the SGD behaves like non-homogeneous Brownian motion in the weights space, in the proximity of a flat error minimum. This non-homogeneous diffusion corresponds to a decrease in MI between the layers and the input variable, in "directions" that are irrelevant to the target label. One of the main challenges in applying information theoretic measures to real-world data is a reasonable estimation of high dimensional joint distributions. This problem has been extensively studied over the years (e.g. BID28), and has led the that there is no "efficient" solution when the dimension of the problem is large. Recently, a number of studies have focused on calculating the MI in DNNs using Statistical Mechanics. These methods have generated promising in a variety of special cases BID8, which support many of the observations made by BID35.In this work we provide an analytic bound on the MI between consecutive layers, which is valid for any non-linearity of the units, and directly demonstrates the compression of the representation during the diffusion phase. Specifically, we derive a Gaussian bound that only depends on the linear part of the layers. This bound gives a super linear dependence of the convergence time of the layers, which in turn enables us to prove the super-linear computational benefit of the hidden layers. Further, the Gaussian bound allows us to study mutual information values in DNNs in real-world data without estimating them directly. Let X ∈ X and Y ∈ Y be a pair of random variables of the input patterns and their target label (respectively). Throughout this work, we consider the practical setting where X and Y are continuous random variables that are represented in a finite precision machine. This means that both X and Y are practically binned (quantized) into a finite number of discrete values. Alternatively, X, Y may be considered as continuous random variables that are measured in the presence of small independent additive (Gaussian) noise, corresponding to their numerical precision. We use these two interpretations interchangeably, at the limit of infinite precision, where the limit is applied at the final stage of our analysis. We denote the joint probability of X and Y as p(x, y), whereas their corresponding MI is defined as DISPLAYFORM0. We use the standard notation D[p||q] for the Kullback-Liebler (KL) divergence between the probability distributions p and q. Let f W K (x) denote a DNN, with K hidden layers, where each layer consists of d k neurons, each with some activation function σ k (x), for k = 1,..., K. We denote the values of the k th layer by the ranom vector T k. The DNN mapping between two consecutive layers is defined as DISPLAYFORM1 Note that we consider both the weights, W k and the layer representations, T k, as stochastic entities, because they depend on the stochastic training rule of the network and the random input pattern (as described in Section 2.1). However, when the network weights are given, the weights are fixed realizations of the random training process (i.e. they are "quenched"). Note that given the weights, the layers form a Markov chain of successive internal representations of the input variable X: Y → X → T 1 →... → T K, and their MI values obey a chain of Data Processing Inequalities (DPI), as discussed by BID35.We denote the set of all K layers weight matrices as W K = {W 1, . . ., W K}. Let the training sample, S n = {(x 1, y 1),..., (x n, y n)} be a collection of n independent samples from p(x, y). Let W K (x i, y i) be a (differentiable) loss function that measures the discrepancy between a prediction of the network f W K (x i) and the corresponding true target value y i, for a given set of weights W K. Then, the empirical error is defined as DISPLAYFORM2 The corresponding error gradients (with respect to the weights) are denoted as DISPLAYFORM3 2.1 TRAINING THE NETWORK -THE SGD ALGORITHM Training a DNN corresponds to the process of setting the values of weights W K from a given set of samples S n. This is typically done by minimizing the empirical error, which approximates the expected loss. The SGD algorithm is a common optimization method for this purpose BID30. Let S (m) be a random set of m samples drawn (uniformly, with replacement) from S n, where m < n. We refer to S (m) as a mini-batch of S n. Define the corresponding empirical error and gradient of the mini-batch as DISPLAYFORM0 respectively. Then, the SGD algorithm is defined by the update rule: DISPLAYFORM1, where W K (l) are the weights after l iterations of the SGD algorithm and η ∈ R + is the learning rate. The SGD algorithm plays a key role in the astonishing performance of DNNs. As a , it has been extensively studied in recent years, especially in the context of flexibility and generalization BID4. Here, we examine the SGD as a stochastic process, that can be decomposed into two separate phases. This idea has been studied in several works BID22 BID16 BID9. Murata argued that stochastic iterative procedures are initiated at some starting state and then move through a fast transient phase towards a stationary phase, where the distribution of the weights becomes time-independent. However, this may not be the case when the SGD induces non-isotropic state dependent noise, as argued, for example, by BID2.In contrast, BID35 described the transient phase of the SGD as having two very distinct dynamic phases. The first is a drift phase, where the means of the error gradients in every layer are large compared to their batch-to-batch fluctuations. This behaviour is indicative of small variations in the gradient directions, or high-SNR gradients. In the second part of the transient phase, which they refer to as diffusion, the gradient means become significantly smaller than their batch-to-batch fluctuations, or low-SNR gradients. The transition between the two phases occurs when the training error saturates and weights growth is dominated by the gradient batch-to-batch fluctuations. Typically, most SGD updates are expended in the diffusion phase before reaching Murata's stationary phase. In this work we rigorously argue that this diffusion phase causes the representation compression; the observed reduction in I(T k ; X), for most hidden layers. It is well known that the discrete time SGD (2.1) can be considered as an approximation of a continuous time stochastic gradient flow if the discrete-time iteration parameter l is replaced by a continuous parameter τ. BID20 recently showed that when the mini-batch gradients are unbiased with bounded variance, the discrete-time SGD is an approximation of a continuous-time Langevin dynamics, DISPLAYFORM0 where C W K (τ) is the sample covariance matrix of the weights, B(τ) is a standard Brownian motion (Wiener process) and β is the Langevin temperature constant. The first term in is called the gradient flow or drift component, and the second term corresponds to random diffusion. Although, this stochastic dynamics hold for the entire SGD training process, the first term dominates the process during the high SNR gradient phase, while the second term becomes dominant when the gradients are small, due to saturation of the training error in the low SNR gradient phase. Hence, these two SGD phases are referred to as drift and diffusion. The mean L 2 displacement (MSD) measures the Euclidean distance from a reference position over time, which is used to characterize a diffusion process. Normal diffusion processes are known to exhibit a power-law MSD in time, DISPLAYFORM1 γ is related to the diffusion coefficient, and 0 < α ≤ 0.5 is the diffusion exponent. For a standard flat space diffusion, the MSD increases as a square root of time (α = 0.5). BID14 showed (empirically) that the weights' MSD, in a DNNs trained with SGD, indeed behaves (asymptotically) like a normal diffusion, where the diffusion coefficient γ depends on the batch size and learning rate. In contrast, BID13 showed that the weights' MSD demonstrates a much slower logarithmic increase. This type of dynamics is also called "ultra-slow" diffusion. Following BID36 and BID35, we study the layer representation dynamics in the two-dimensional (I(X; T k), I(T k ; Y)) plane. Specifically, for any input and target variables, X, Y, let T T (X) denote a representation, or an encoding (not necessarily deterministic), of X. Clearly, T is fully characterized by its encoder, the conditional distribution p(t|x). Similarly, let p(y|t) denote any (possibly stochastic) decoder of Y from T. Given a joint probability function p(x, y), the Information Plane is defined the set of all possible pairs I(X; T) and I(T ; Y) for any possible representation, p(T |X).It is evident that not all points on the plane are feasible (achievable), as there is clearly a tradeoff between these quantities; the more we compress X (reduce I(X; T)), the less information can be maintained about the target, I(T ; Y).Our analysis is based on the fundamental role of these two MI quantities. We argue that for large scale (high dimensional X) learning, for almost all (typical) input patterns, with mild assumptions (ergodic Markovian input patterns): (i) the MI values concentrate with the input dimension; (ii) the minimal sample complexity for a given generalization gap is controlled by I(X; T); and (iii) the accuracy -the generalization error -is governed by I(T ; Y), with the Bayes optimal decoder representation. Here, we argue that the sample-size -accuracy trade-off, of all large scale representation learning, is characterized by these two MI quantities. For DNNs, this amounts to a dramatic reduction in the complexity of the analysis of the problem. We discuss these ideas in the following sections and prove the connection between the input representation compression I(T ; X), the generalization gap (the difference between training and generalization errors), and the minimal sample complexity (Theorem 1 below). Optimizing mutual information quantities is by no means new in either supervised and unsupervised learning BID7 BID21 BID26. This is not surprising, as it can be shown that I(T ; Y) corresponds to the irreducible error when minimizing the logarithmic loss BID25 BID10. Here, we emphasize that I(T ; Y), for the optimal decoder of the representation T, governs all reasonable generalization errors (under the mild assumption that label y is not completely deterministic; p(y|x) is in the interior of the simplex, ∆(Y), for all typical x ∈ X). First, note that with the Markov chain DISPLAYFORM0. By using the Pinsker inequality BID6 ) the variation distance between the optimal and the representation decoders can be bound by their KL divergence, DISPLAYFORM1 Hence, by maximizing I(T ; Y) we minimize the expected variation risk between the representation decoder p(y|t) and p(y|x). For more similar bounds on the error measures see BID24. The Minimum Description Length (MDL) principle BID29 suggests that the best representation for a given set of data is the one that leads to the minimal code-length needed to represent of the data. This idea has inspired the use of I(X; T) as a regularization term in many learning problems (e.g. BID5 ; BID27). Here, we argue that I(X; T) plays a much more fundamental role; we show that for large scale (high dimensional X) learning and for typical input patterns, I(X; T) controls the sample complexity of the problem, given a generalization error gap. Theorem 1 (Input Compression bound). Let X be a d-dimensional random variable that obeys an ergodic Markov random field probability distribution, asymptotically in d. Let T T (X) be a representation of X and denote by T m = {(t 1, y 1),..., (t m, y m)} an m-sample vector of T and Y, generated with m independent samples of x i, with p(y|x i) and p(t|x i). Assume that p(x, y) is bounded away from 0 and 1 (strictly inside the simplex interior). Then, for large enough d, with probability 1 − δ, the typical expected squared generalization gap satisfies DISPLAYFORM0 where the typicality follows the standard Asympthotic Equipartition Property (AEP) BID6.A proof of this Theorem is given in Appendix A. This Theorem is also related to the bound proved by BID33, with the typical representation cardinality, |T (X)| ≈ 2 I(T ;X). The ergodic Markovian assumption is common in many large scale learning problems. It means that p(DISPLAYFORM1 The consequences of this input-compression bound are quite striking: the generalization error decreases exponentially with I(X; T), once I(T ; X) becomes smaller than log 2m -the query samplecomplexity. Moreover, it means that M bits of representation compression, beyond log 2m, are equivalent to a factor of 2 M training examples. The tightest bound on the generalization bound is obtained for the most compressed representation, or the last hidden layer of the DNN. The inputcompression bound can yield a tighter and more realistic sample complexity than any of the worstcase PAC bounds with any reasonable estimate of the DNN class dimensionality, as typically the final hidden layers are compressed to a few bits. Nevertheless, two important caveats are in order. First, the layer representation in Deep Learning are learned from the training data; hence, the encoder, the partition of the typical patterns X, and the effective "hypothesis class", depend on the training data. This can lead to considerable over-fitting. Training with SGD avoids this potential over-fitting because of the way the diffusion phase works. Second, for low I(T ; Y) there are exponentially (in d) many random encoders (or soft partitions of X) with the same value of I(T ; X). This seems to suggest that there is a missing exponential factor in our estimate of the hypothesis class cardinality. However, note that the vast majority (almost all) of these possible encoders are never encountered during typical SGD optimization. In other words, they act like a "dark hypothesis space" which is never observed and does not affect the generalization bound. Moreover, as I(T ; Y) increases, the number of such random encoders rapidly collapses all the way to O when I(T ; Y) approaches the optimal IB limit, as we show next. As presented above, we are interested in the boundary of the achievable region in the information plane, or in encoder-decoder pairs that minimize the sample complexity (minimize I(X; T)) and generalize well (maximize I(T ; Y)).These optimal encoder-decoder pairs are given precisely by the Information Bottleneck framework BID37, which is formulated by the following optimization problem: min p(t|x) I (X; T) − βI (T ; Y), over all possible encoders-decoders pairs that satisfy the Markov condition Y − X − T. Here β is a positive Lagrange multiplier associated with the decoder information on I(T ; Y), which also determines the complexity of the representation. The Information Bottleneck limit defines the set of optimal encoder-decoder pairs, for the joint distribution p(x, y). Furthermore, it characterizes the achievable region in the Information Plane, similar to Shannon's Rate Distortion Theory (RDT) BID6. By our previous analysis it also determines the optimal tradeoff between sample complexity and generalization error. The IB can only be solved analytically in very special cases (e.g., jointly Gaussian X, Y BID3). In general, a (locally optimal) solution can be found by iterating the self-consistent equations, similar to the Arimoto-Blahut algorithm in RDT BID37. For general distributions, no efficient algorithm for solving the IB is known, though there are several approximation schemes BID1 BID23. The self-consistent equations are exactly satisfied along the IB limit, aka the Information Curve. By applying the DPI to the Markov chain of the DNN layers we obtain the following chains: I(X; T 1) ≥ I(X; T 2) ≥ · · · ≥ I(X; T k) ≥ I(X;Ŷ) and I(X; Y) ≥ I(T 1 ; Y) ≥ · · · ≥ I(T k ; Y) ≥ I(Ŷ ; Y) whereŶ is the output of the network. The pairs (I(X; T k), I(T k, Y)), for each SGD update, form a unique concentrated Information Path for each layer of a DNN, as demonstrated by BID35.For any fixed realization of the weights, the network is, in principle, a deterministic map. This does not imply that information is not lost between the layers; the inherent finite precision of the layers, with possible saturation of the nonlinear activation functions σ k, can in non-invariable mapping between the layers. Moreover, we argue below that for large networks this mapping becomes effectively stochastic due to the diffusion phase of the SGD.On the other hand, the Information Plane layer paths are invariant to invertible transformations of the representations T k. Thus the same paths are shared by very different weights and architectures, and possibly different encoder-decoder pairs. This freedom is drastically reduced when the target information, I(T k, Y), increases and the layers approach the IB limit. Minimizing the training error (ERM), together with standard uniform convergence arguments clearly increase I(T ; Y), but what in the SGD dynamics can lead to the observed representation compression which further improves generalization? Moreover, can the SGD dynamics push the layer representations all the way to the IB limit, as claimed in?We provide affirmative answers to both questions, using the properties of the drift and diffusion phases of the SGD dynamics. In this section we quantify the roles of the drift and diffusion SGD phases and their influence on the MI between consecutive layers. Specifically, we show that the drift phase corresponds to an increase in information with the target label I(T k ; Y), whereas the diffusion phase corresponds to representation compression, or reduction of the I(X; T k). The representation compression is accompanied by further improvement in the generalization. The general idea is as follows: the drift phase increases I(T k ; Y) as it reduces the cross-entropy empirical error. On the other hand, the diffusion phase in high dimensional weight space effectively adds an independent non-uniform random component to the weights, mostly in the directions that do not influence the loss -i.e, irrelevant directions. This in a reduction of the SNR of the irrelevant features of the patterns, which leads to a reduction in I(X; T k), or representation compression. We further argue that different layers filter out different irrelevant features, ing in their convergence to different locations on the Information Plane. First, we notice that the DPI implies that I(X; T k) ≤ I(T k−1 ; T k). We focus on the second term during the diffusion phase and prove an asymptotic upper bound for I(T k−1 ; T k), which reduces sub-linearly with the number of SGD updates. For clarity, we describe the case where T k ∈ R d k is a vector and T k+1 ∈ R is a scalar. The generalization to higher d k+1 is straightforward. We examine the network during the diffusion phase, after τ iterations of the SGD beyond the drift-diffusion transition. For each layer, k, the weights matrix, W k (τ) can be decomposed as follows, DISPLAYFORM0 The first term, W k, denotes the weights at the end of the drift phase (τ 0 = 0) and remains constant with increasing τ. As we assume that the weights converge to a (local, flat) optimum during the drift phase, W k is close to the weights at this local optimum. The second term, δW k (τ), is the accumulated Brownian motion in τ steps due to the batch-to-batch fluctuations of the gradients near the optimum. For large τ we know that δW DISPLAYFORM1 ) where τ 0 is the time that the diffusion phase began. Note that for any given τ, we can treat the weights as a fixed (quenched) realization, w k (τ), of the random Brownian process W k (τ). We can now model the mapping between the layers T k and T k+1 at that time as DISPLAYFORM2 where w * ∈ R d k is the SGD's empirical minimizer, and δw ∈ R d k is a realization from a Gaussian vector δw ∼ N (0, C δw), of the Brownian process discussed in Section 2.3. In addition, we consider Z ∼ N (0, σ 2 z) to be the small Gaussian measurement noise, or quantization, independent of δw k and T k. This standard additive noise allows us to treat all the random variables as continuous. For simplicity we first assume that the d k components of T k have zero mean and are asymptotically independent for d k → ∞.Proposition 2. Under mild technical conditions which are met with probability 1 in standard deep learning (see Appendix B), we have that DISPLAYFORM3 almost surely, where σ 2 T k is the variance of the components of T k.A proof for this CLT proposition is given in Appendix B.Proposition 2 shows that under standard conditions, w * T T k and δw T T k are asymptotically jointly Gaussian and independent, almost surely. We stress that the components of T k do not have to be identically distributed to satisfy this property; Proposition 2 may be adjusted for this case with different normalization factors. Similarly, the independence assumption on T k can easily be relaxed to Markovian ergodic, as we assume the input patterns. Finally, it is easy to verify that Proposition 2 can be extended to the general case where w *, δw ∈ R d k ×d k+1, under similar technical conditions. We can now bound the mutual information between T k+1 and the linear projection of the previous layer W * T k, during the diffusion phase, for sufficiently high dimensions d k, d k+1, under the conditions above. Note that in this case, behaves like an additive Gaussian channel where w * T T k is the signal and δw T T k + Z is an independent additive Gaussian noise (i.e., independent of signal and normally distributed). Hence, for sufficiently large d k and d k+1, we have DISPLAYFORM4 almost surely, where the first inequality is due to DPI for the Markov chain T k − w * T T k − T k+1. Finally, we apply an orthogonal eigenvalue decomposition to this multivariate Gaussian channel. Let δw T δw = QΛQ T where QQ T = I and Λ is a diagonal matrix whose diagonal elements are the corresponding eigenvalues, λ i, of δw T δw. Then, we have that DISPLAYFORM5 The last inequality is due to the Hadamard inequality. Plugging into yields that for sufficiently large d k and d k+1, DISPLAYFORM6 As previously established, δw is a Brownian motion along the SGD iterations during the diffusion phase. This process is characterized by a low (and fixed) variance of the informative gradients (relevant dimensions), whereas the remaining irrelevant directions suffer from increasing variances as the diffusion proceeds (see, for example, BID31 ; BID15). In other words, we expect the "informative" λ i to remain fixed, while the irrelevant consistently grow, sub-linearly with time. Denote the set of "informative/relevant" directions as Λ * and the set of "non-informative" as Λ N I. Then our final limit, as the number of SGD steps grow, is DISPLAYFORM7. Note that the directions that are compressed and the ones that are preserved depend on the required compression level. This is the reason that different layers converge to different values of I(T k ; X). The analysis above suggests that the SGD compresses during the diffusion phase in many directions of the gradients. We argue that these directions are the ones in which the variance of the gradients is increasing (non-informative) whereas the information is preserved in the directions where the variance of the gradients remain small. This statement is consistent with recent (independent) work on the statistical properties of gradients and generalization. BID31;; BID40 showed that typically, the covariance matrix of the gradients is highly non-isotropic and that this is crucial for generalization by SGD. They suggested that the reason lies in the proximity of the gradients' covariance matrix to the Hessian of the loss approximation. Furthermore, it was argued by BID40; BID17 BID15 that SGD tends to converge to flat minima. These flat minima often correspond to a better generalization. BID40 emphasized that SGD converges to flat minima values characterized by high entropy due to the non-isotropic nature of the gradients' covariance and its alignment with the error Hessian at the minima. In other words, all of the finding above suggest that good generalization performance is typically characterized by non-isotropic gradients and Hessian, that are in orthogonal directions to the flat minimum of the training error objective. Our Gaussian bound on the representation compression allows us to relate the convergence time of the layer representation information, I(T k ; X), to the diffusion exponent α, defined in section 2.3.Denote the representation information at the diffusion time τ as I(X; T k)(τ). It follows from that I(X; T k)(τ) ≤ C + 1 2 DISPLAYFORM0 where C depends on the informative information for this layer, but not on τ.Notice that λ i (τ) are the singular values of the weights of a diffusion process, which grow as τ α where α is the diffusion exponent. Hence, DISPLAYFORM1 The change of weights, the SNR of the gradients, the MI and the Gaussian bound during the training for one layer. In log-log scale. Note that R depends solely on the problem, f (x) or p(y, x), and not on the architecture. The idea behind this argument is as follows -one can expand the function in any orthogonal basis (e.g. Fourier transform). The expansion coefficients determine both the dimensionality of the relevant/informative dimensions and the total trace of the irrelevant directions. Since these traces are invariant to the specific function basis, these traces remain the same when expanding the function in the network functions using the weights. Now, with K hidden layers, where each layer only needs to compress from the previous (compressed) layer, by ∆I k and the total compression is ∆I X = k ∆I k. Under these assumptions, even if the layers compress one after the other, the total compression time breaks down into K smaller steps, as at DISPLAYFORM2 if the ∆I k are similar, we obtain a super-linear boost in the computational time by a factor K 1 α. Since α ≤ 0.5 this is at least a quadratic boost in K. For ultra-slow diffusion we obtain an exponential boost (in K) in the convergence time to a good generalization. This is consistent with the observations reported by BID35. We now illustrate our in a series of experiments. We examine several different setups. MNIST dataset-In the first experiment, we evaluate the MNIST handwritten digit recognition task BID18. For this data set, we use a fully-connected network with 5 hidden layers of width 500 − 250 − 100 − 50 − 20, with an hyperbolic tangent (tanh) activation function. The relative low dimension of the network and the bounded activation function allow us to empirically measure the MI in the network. The MI is estimated by binning the neurons' output into the interval [−1, 1].The discretized values are then used to estimate the joint distributions and the corresponding MI, as described by BID35. Figure 1a depicts the norms of the weights, the signal-to-noise ratio (the ratio between the means of the gradients and their standard deviations), the compression rate I(X; T) and the Gaussian upper bound on I(X; T), as defined in. As expected, the two distinct phases correspond to the drift and diffusion phases. Further, these two phases are evident by independently observing the SNR, the change of the weights ||W (l) − W||, the MI and the upper bound. In the first phase, the weights grow almost linearly with the iterations, the SNR of the gradients is high, and there is almost no change in the MI. Then, after the transition point (that accrues almost at the same iteration for all (a) CIFAR-10 (b) CIFAR-100Figure 2: Change in the SNR of the gradients and the Gaussian bound on the MI during the training of the network for one layer on ResNet-32, in log-log scale.the measures above), the weights behave as a diffusion process, and the SNR and the MI decrease remarkably. In this phase, there is also a clear-cut reduction of the bound. CIFAR-10 and CIFAR-100 -Next, we validate our theory on large-scale modern networks. In the second experiment we consider two large-scale data sets, CIFAR-10 and CIFAR-100. Here, we train a ResNet-32 network, using a standard architecture (including ReLU activation functions as described in BID11 . In this experiment we do not estimate the MI directly, due to the large scale of the problem. Figure 2 shows the SNR of the gradients and the Gaussian bound for one layer in CIFAR-10 and CIFAR-100 on the ResNet-32 network, averaged over 50 runs. Here, we observed similar behavior as reported in the MNIST experiment. Specifically, there is a clear distinction between the two phases and a reduction of the MI bound along the diffusion phase. Note that the same behavior was observed in most of the 32 layers in the network. Recently there have been several attempts to characterize the correspondence between the diffusion rate of the SGD and the size of the mini-batch BID14 BID13). In these articles, the authors claimed that a larger mini-batch size corresponds to a lower diffusion rate. Here, we examined the effect of the mini-batch size on the transition phase in the Information Plane. For each mini-batch size, we found both the starting point of the information compression and the gradient phase transition (the iteration where the derivative of the SNR is maximal). Figure 1b illustrates the . The X-axis is the iteration where the compression started, and the Y -axis is the iteration where the phase transition in the gradients accrued for different mini-batch sizes. There is a clear linear trend between the two. This further justifies our suggested model, since that the two measures are strongly related. Next, we validate our on the computational benefit of the layers. We train networks with a different number of layers (1-5 layers) and examine the iteration for which the network converge. Then, we find the α which fits the best trend K 1 α where K is the number of layers. FIG1 shows the for two data-sets -MNIST and the symmetric dataset from BID35. As our theory suggest, as we increase the number of layers, the convergence time decreases with a factor of k 1 α for different values of α. In this work we study DNNs using information-theoretic principles. We describe the training process of the network as two separate phases, as has been previously done by others. In the first phase (drift) we show that I(T k ; Y) increases, corresponding to improved generalization with ERM. In the second phase (diffusion), the representation information, I(X; T k) slowly decreases, while I(T K ; Y) continues to increase. We rigorously prove that the representation compression is a direct consequence of the diffusion phase, independent of the non-linearity of the activation function. We provide a new Gaussian bound on the representation compression and then relate the diffusion exponent to the compression time. One key outcome of this analysis is a novel proof of the computational benefit of the hidden layers, where we show that they boost the overall convergence time of the network by at least a factor of K 2, where K is the number of non-degenerate hidden layers. This boost can be exponential in the number of hidden layers if the diffusion is "ultra slow", as recently reported.1 m m i=1 h (x i, y i) be the empirical error. Hoeffding's inequality BID12 shows that for every h ∈ H, DISPLAYFORM0 Then, we can apply the union bound and conclude that DISPLAYFORM1 We want to control the above probability with a confidence level of δ. Therefore, we ask that 2 H exp −2 2 m ≤ δ. This leads to a PAC bound, which states that for a fixed m and for every h ∈ H, we have with probability 1 − δ that DISPLAYFORM2 Note that under the definitions stated in Section 1.1, we have that |H| ≤ 2 X. However, the PAC bound above also holds for a infinite hypotheses class, where log |H| is replaced with the VC dimension of the problem, with several additional constants BID38 BID34 BID32.Let us now assume that X is a d-dimensional random vector which follows a Markov random field structure. As stated above, this means that p(x i) = i p(x i |P a(x i)) where P a(X i) is a set of components in the vector X that are adjacent to X i. Assuming that the Markov random field is ergodic, we can define a typical set of realizations from X as a set that satisfies the Asymptotic Equipartition Property (AEP) BID6. Therefore, for every > 0, the probability of a sequence drawn from X to be in the typical set A is greater than 1 − and |A | ≤ 2 H(X)+. Hence, if we only consider a typical realization of X (as opposed to every possible realization), we have that asymptotically H ≤ 2 H(X). Finally, let T be a mapping of X. Then, 2 H(X|T) is the number of typical realizations of X that are mapped to T. This means that the size of the typical set of T is bounded from above by 2 H(X) 2 H(X|T) = 2 I(X;T). Plugging this into the PAC bound above yields that with probability 1 − δ, the typical squared generalization error of T,

represent the short summary for finding the corresponding scientific article (long)

Introduce an information theoretic viewpoint on the behavior of deep networks optimization processes and their generalization abilities

Represent the TLDR for finding the corresponding scientific article

We propose a hypothesis for why gradient descent generalizes based on how per-example gradients interact with each other.