Question: Represent the scientific article to find a short TLDR!

Question: Inspired by neurophysiological discoveries of navigation cells in the mammalian brain, we introduce the first deep neural network architecture for modeling Egocentric Spatial Memory (ESM). It learns to estimate the pose of the agent and progressively construct top-down 2D global maps from egocentric views in a spatially extended environment. During the exploration, our proposed ESM network model updates belief of the global map based on local observations using a recurrent neural network. It also augments the local mapping with a novel external memory to encode and store latent representations of the visited places based on their corresponding locations in the egocentric coordinate. This enables the agents to perform loop closure and mapping correction. This work contributes in the following aspects: first, our proposed ESM network provides an accurate mapping ability which is vitally important for embodied agents to navigate to goal locations. In the experiments, we demonstrate the functionalities of the ESM network in random walks in complicated 3D mazes by comparing with several competitive baselines and state-of-the-art Simultaneous Localization and Mapping (SLAM) algorithms. Secondly, we faithfully hypothesize the functionality and the working mechanism of navigation cells in the brain. Comprehensive analysis of our model suggests the essential role of individual modules in our proposed architecture and demonstrates efficiency of communications among these modules. We hope this work would advance research in the collaboration and communications over both fields of computer science and computational neuroscience. Egocentric spatial memory (ESM) refers to a memory system that encodes, stores, recognizes and recalls the spatial information about the environment from an egocentric perspective BID24. Such information is vitally important for embodied agents to construct spatial maps and reach goal locations in navigation tasks. For the past decades, a wealth of neurophysiological have shed lights on the underlying neural mechanisms of ESM in mammalian brains. Mostly through single-cell electrophysiological recordings studies in mammals BID23, there are four types of cells identified as specialized for processing spatial information: head-direction cells (HDC), border and boundary vector cells (BVC), place cells (PC) and grid cells (GC). Their functionalities are: According to BID38, HDC, together with view cells BID5, fires whenever the mammal's head orients in certain directions. The firing behavior of BVC depends on the proximity to environmental boundaries BID22 and directions relative to the mammals' heads BID1. PC resides in hippocampus and increases firing rates when the animal is in specific locations independent of head orientations BID1. GC, as a metric of space BID35, are regularly distributed in a grid across the environment BID11. They are updated based on animal's speed and orientation BID1. The corporation of these cell types enables mammals to navigate and reach goal locations in complex environments; hence, we are motivated to endow artificial agents with the similar memory capability but a computational architecture for such ESM is still absent. Inspired by neurophysiological discoveries, we propose the first computational architecture, named as the Egocentric Spatial Memory Network (ESMN), for modeling ESM using a deep neural network. ESMN unifies functionalities of different navigation cells within one end-to-end trainable framework and accurately constructs top-down 2D global maps from egocentric views. To our best knowledge, we are the first to encapsulate the four cell types respectively with functionally similar neural networkbased modules within one integrated architecture. In navigation tasks, the agent with the ESMN takes one egomotion from a discrete set of macro-actions. ESMN fuses the observations from the agent over time and produces a top-down 2D local map using a recurrent neural network. In order to align the spatial information at the current step with all the past predicted local maps, ESMN estimates the agent's egomotion and transforms all the past information using a spatial transformer neural network. ESMN also augments the local mapping module with a novel spatial memory capable of integrating local maps into global maps and storing the discriminative representations of the visited places. The loop closure component will then detect whether the current place was visited by comparing its observation with the representations in the external memory which subsequently contributes to global map correction. Neuroscience-inspired AI is an emerging research field BID12. Our novel deep learning architecture to model ESMN in the mammalian navigation system attempts to narrow the gap between computer science (CS) and computational neuroscience (CN) and bring interests to both communities. On one hand, our novel ESMN outperforms several competitive baselines and the state-of-the-art monocular visual SLAMs. Our outstanding performance in map construction brings great advancements in robotics and CS. It could also have many potential engineering applications, such as path planning for robots. In CN, the neuroplausible navigation system with four types of cells integrated is still under development. In our work, we put forward bold hypothesis about how these navigation cells may cooperate and perform integrated navigation functions. We also faithfully propose several possible communication links among them in the form of deep architectures. We evaluate ESMN in eight 3D maze environments where they feature complex geometry, varieties of textures, and variant lighting conditions. In the experiments, we demonstrate the acquired skills of ESMN in terms of positional inference, free space prediction, loop closure classification and map correction which play important roles in navigation. We provide detailed analysis of each module in ESMN as well as their functional mappings with the four cell types. Lastly, we conduct ablation studies, compare with state-of-the-art Simultaneous Localization And Mapping (SLAM) algorithms and show the efficacy of our integrated framework on unifying the four modules. There is a rich literature on computational models of egocentric spatial memory (ESM) primarily in cognitive science and AI. For brevity, we focus on the related works in machine learning and robotics. Reward-based learning is frequently observed in spatial memory experiments BID14. In machine learning, reinforcement learning is commonly used in attempts to mathematically formalize the reward-based learning. Deep Q-networks, one of the reinforcement learning frameworks, have been employed in the navigation tasks where the agent aims to maximize the rewards while navigating to the goal locations BID26; BID28. The representation of spatial memory is expressed implicitly using long short term memory (LSTM) BID13. In one of the most relevant works BID10, Gupta et al.introduced a mapper-planner pipeline where the agent is trained in a supervised way to produce a top-down belief map of the world and thus plans path; however, the authors assume that the agents perform mapping tasks in an ideal scenario where the agent knows which macro-action to take and takes the macro-action without control errors at each time step. Different from their work, our ESM network takes into account the action uncertainty and predicts the agent's pose based on egocentric views. Moreover, we propose the mechanism of loop closure and map correction for long-term exploration. Apart from reinforcement learning, there are works on navigation in the domain of robotics where the spatial memory is often explicitly represented in the form of grid maps or topological structuresElfes; BID33; BID20. SLAM tackles the problem of positional inference and mapping BID36 BID4; BID3; BID39. While classical SLAM achieves good performances with the aid of multimodal sensors BID21, SLAM using monocular cameras still has limitations where the feature matching process relies heavily on hand-crafted features extracted from the visual scenes BID40; BID34. As great strides have been made using deep learning in computer vision tasks which in a significant performance boost, several works endeavor to replace parts of the classical SLAM workflows with deep learning based modules BID17; BID0; BID16. However, there is no existing end-to-end neural network for visual SLAM to our best knowledge. Inspired by the neurophysiological , we propose the first unified ESM model using deep neural networks. This model simultaneously solves the problems of positional inference, intrinsic representation of places and space geometry during the process of 2D map construction. Before we elaborate on our proposed architecture, we introduce the Egocentric Spatial Memory Network (ESMN) modeling problem and relevant notations. ESM involves an object-to-self representational system which constantly requires encoding, transforming and integrating spatial information from the first-person view into a global map. At the current time step t, the agent, equipped with one RGB camera, takes the camera view I t as the current visual input. Without the aid of any other sensors, it predicts the egomotion from the pair of camera views (I t−1, I t). At time step t, the agent is allowed to take only one egomotion A θ,d,l,t out of a discrete set of macro-actions which include rotating left/right by θ degrees and moving in the directions l: forward/backward/left/right by the distance of d relative to the agent's current pose. We assume the agent moves in a 2D grid world and the camera coordinate is fixed with respect to the agent's body. The starting location p 0 of the agent is always where the triplet denotes the positions along the x and y-axis and the orientation in the world coordinate. The problem of modeling ESM is to learn a global map in the egocentric coordinate based on the visual input pairs (I t−1, I t) for t = 1,..., T. We define the global map as a top-view 2D probabilistic grid map where the probability infers the agent's belief of free space. For precise controls of the experimental variables, we consider the egocentric spatial memory (ESM) modeling problem in artificial 3D maze environments. In order to tackle this problem, we propose a unified neural network architecture named as ESMN. The architecture of ESMN is illustrated in FIG0 and is elaborated in details in Figure 2. Inspired by the navigation cells mentioned in the introduction, our proposed ESMN comprises four modules: Head Direction Unit (HDU), Boundary Vector Unit (BVU), Place Unit (PU), and Grid Unit (GU). This is to incorporate multiple objectives for modeling ESM. spatial information to the current egocentric coordinate via a 2D-CNN. PU learns to encode the latent representation of visited places in a 2D-CNN pre-trained using a triplet loss. GU, as a memory, integrates the predicted local maps from BVU over time and keeps track of all the visited places. In GU, we add a gating mechanism to constantly detect the loop closure and thus eliminate the accumulated errors during long-term mapping. In this paper, we adopt a "divide and conquer" approach by composing different navigation modules within one framework systematically. The division of these modules are inspired by scientific supports which have shown the advantages in biological systems BID31. Leveraging on rich features extracted from deep networks, the learnt features are suitable for recognizing visual scenes and hence, boost up the map construction and correction performances. The efficiency and robustness of our algorithm are demonstrated by comparing with other spatial navigation methods replied on visual inputs. At each time step, the RGB camera image (or frame) is the only input to the agent. In order to make spatial reasoning about the topological structure of the spatially extended environment, the agent has to learn to take actions to explore its surroundings and predict their own poses by integrating the egomotion over time. The HDU module in our proposed ESMN model employs a 2D-CNN to predict the macro-action the agent has taken at the time step t based on the inputs of two camera views {I t−1, I t}. Though there are two possible implementations for the loss in training the HDU module, i.e. regression and classification, we consider the latter in order to reduce the dimensionality of action space. The macro-actions A θ,d,l are discretized into 2 classes for rotation and 4 classes for translation as explained at the beginning of Section 3. Though we solve the problem of positional inference using a feed-forward neural network based on two most recent observations alone, the extensions to use recurrent neural network would be interesting to explore in the future. We explain how ESMN integrates egocentric views into a top-down 2D representation of the environment using a recurrent neural network. Similar to BID10, the BVU in ESMN serves as a local mapper and maintains the accumulative free space representations in the egocentric coordinate for a short-term period. Given the current observation I t, function g first encodes its geometric representation about space g(I t) via a 2D-CNN and then transform g(I t) into egocentric top-down view m t via de-convolutional layers. Together with the accumulative space representation m t−1 at the previous step and the estimated egomotion A θ,d,l,t from t − 1 to t, BVU estimates the current local space representation m t using the following update rule: DISPLAYFORM0 where W is a function that transforms the previous accumulative space representation m t−1 to the current egocentric coordinate based on the estimated egomotion A θ,d,l,t. We parameterize W by using a spatial transformer network BID15 composing of two key elements: it generates the sampling grid which maps the input coordinates of m t−1 to the corresponding output locations after egomotion A θ,d,l,t transformation; the sampling kernel then takes the bilinear interpolation of the values on m t−1 and outputs the transformed m t−1 in the current egocentric coordinate. U is a function which merges the free space prediction m t from the current observation with the accumulative free space representation at the previous step. Specifically, we simplify merging function U as a weighted summation parameterized by λ followed by hyperbolic tangent operation: DISPLAYFORM1 Loop closure is valuable for the agents in the navigation tasks as it encourages efficient exploration and spatial reasoning. In order to detect loop closure during an episode, given the current observation I t, ESMN learns to encode the discriminative representation h(I t) of specific places independent of scaling and orientations via an embedding function F. Based on the similarity of all the past observations Ω t = {I 1, I 2, ..., I t} at corresponding locations P t = {p 1, p 2, ..., p t}, we create training targets by making an analogy to the image retrieval tasks BID8 and define the triplet (I t, I +, I −) as anchor sample (current camera view), positive and negative samples drawn from Ω t respectively. PU tries to minimize the triplet loss: DISPLAYFORM0 where we parameterize F using a three-stream 2D-CNN where the weights are shared across streams. D is a distance measure between pairs of embedding. Here, mean squared error is used. A loop closure label equals 1 if the mean squared error is below the threshold α which is set empirically. Apart from the similarity comparison of observations, we consider two extra criteria for determining whether the place was visited: the current position of the agent p t is near to the positions visited at the earlier times. We empirically set a threshold distance between the current and the recently visited locations based on the loop closure accuracy during training. We implemented it via a binary mask in the center of the egocentric global map where the binary states denote the accepted "closeness". the agent only compares those positions which are far from the most recent visited positions to avoid trivial loop closure detection at consecutive time steps. It is implemented via another binary mask which tracks these recently visited places. These criterion largely reduce the false alarm rate and improves the searching speed during loop closure classification. While BVU provides accumulative local free space representations in high resolution for a short-term period, we augment the local mapping framework with memory for long-term integration of the local maps and storage of location representations. Different from Neural Turing Machine BID9 where the memory slots are arranged sequentially, our addressable memory, of size F × H × W, is indexed by spatial coordinates {(i, j): i ∈ {1, 2, ..., H}, j ∈ {1, 2, ..., W}} with memory vector M (i, j) of size F at location (i, j). Because ESM is often expressed in the coordinate frame of the agent itself, we use location-based addressing mechanism and the locations of reading or writing heads are fixed to be always in the center of the memory. Same as BVU, all the past spatial information in the memory is transformed based on the estimated egomotion A θ,d,l,t. Mathematically, we formulate the returned reading vector r h,w as DISPLAYFORM0 where the memory patch covers the area of memory vectors with the width w and height h. We simplify the writing mechanism for GU and use Equation 4 for writing the vector r h,w. Figure 3: Overview of maze layouts, with differing geometries, textures and lighting conditions. Each maze is stimulated with normal, weak and strong lighting conditions. Maze 5_S and 5_W refers to Maze 5 with strong and weak lighting conditions respectively. Maze 1 is adopted from. Maze 8 is inspired by radial arm maze used to test spatial memory in rats BID32. The digits are pasted on walls along specific pathways for loop closure classification tasks. In our case, two writing heads and two reading heads are necessary to achieve the following objectives: one reading head returns the memory vectors m t−1 in order for BVU to predict m t using Equation 1; GU performs updates by writing the predicted local accumulative space representation m t back into the memory to construct the global map in the egocentric coordinate; GU keeps track of the visited places by writing the discriminative representation h(I t) at the center of the egocentric global map denoted as (DISPLAYFORM1 W 2); GU returns the memory vectors near to the current location for loop closure classification where the size of the searching area is parameterized by w and h. The choice of w and h are application-based. We simplify the interaction between local space representations m t and m t−1 with GU and set w and h to be the same size as m t and m t−1.If the loop closure classifies the current observation as "visited", GU eliminates the discrepancies on the global map by merging the two places together. The corrected map has to preserve the topological structure in the discovered areas and ensure the connectivity of the different parts on the global map is maintained. To realize this, we take three inputs in a stack of 3D convolution and de-convolution layers for map correction. The inputs are: the local map predicted at the anchor place; the local map predicted at the recalled place in the current egocentric coordinate; all the past integrated global maps. To make the training targets, we perturb the sequence of ground truth egomotions with random ones and generate synthetic integrated global maps with rotation and scaling augmentations. We minimize regression loss between the predicted maps and the ground truth. We train ESMN end-to-end by stochastic gradient descent with learning rate 0.002 and momentum 0.5. Adam Optimizer BID18 is used. At each time step, the ground truths are provided: local map, egomotion and loop closure classification label. For faster training convergence, we first train each module separately and then load these pre-trained networks into ESMN for fine-tuning. The input frame size is 3 × 64 × 64. We normalize all the input RGB images to be within the range [−1, 1]. The batch size is 10. The discriminative representation h(I t) is of dimension 128. The size of the local space representation m t is h × w = 32 × 32 covering the area 7.68 meters × 7.68 meters in the physical world whereas the global map (the addressable memory) is of size H × W = 500 × 500. We set λ = 0.5 in BVU and α = 0.01 in PU. The memory vector in GU is of size F = 128. Refer to Supplementary Material for detailed architecture. We conduct experiments on eight 3D mazes in Gazebo for robotic simulation BID19. We will make our simulation source code public once our paper is accepted. Figure 3 shows an overview of eight mazes. They feature complex geometry and rich varieties of textures. To prevent over-fitting in a single-modality environment, we augment these observations with different illumination conditions classified into three categories: strong, normal and weak, and train our ESMN in these mixed lighting conditions. Based on the in Section 4.1, our ESMN performs equally well in the test set across permutations of these conditions which shows its capacity of generalizing in various environments. For loop closure classification tasks, we create digits on walls as the unique features of specific pathways. The agent randomly walks through the mazes with obstacle avoidance. The ground truths for 2D top-view of mazes are obtained by using one virtual 2D laser scanner attached to the agent. We use the data collected in maze 1 to 5 for training and validation and the data in maze 6 to 8 for testing. The macro-actions for egomotions are divided into 2 classes for rotation by θ = 10 degrees and 4 classes for translation with d = 0.1 meters relative to the agent's current pose in the physical world. The simulation environment enables us to conduct exhaustive evaluation experiments and train individual module separately using fully supervised signals for better functionality evaluation. The integrated framework is then trained jointly after loading the pre-trained weights from individual modules. Head Direction Unit (HDU): Positional Inference. Head Direction Cells fire whenever the animal's head orients in certain directions. The study BID25 suggests that the rodent's orientation system is as a of neural integration of head angular velocities derived from vestibular system. Thus, after jointly training ESMN, we decode HDU for predicting head direction in the world coordinate by accumulating the estimated egomotions over time. FIG2 shows the head direction prediction in the world coordinate compared with the agent's ground truth global poses. We observe that the head directions integrated from the estimated egomotions are very similar to the ground truths. Moreover, we compute the egomotion classification errors from 2 classes of rotation and 4 classes of translation under three illumination conditions, see Table 1. There is a slight degradation of rotation + translation compared with rotation alone. One possible reason is that the change of egocentric camera views after 10 degrees of rotation is more observable than the one induced from translation by 0.1 meters. We also provide quantitative analysis of our predicted local maps across the first 32 time steps in FIG4 using Mean Squared Error (MSE), Correlation (Cor), Mutual Information (MI) which are standard image similarity metrics. At each time step, the predicted local maps are compared with the ground truth maps at t = 32. As the agent continues to explore in the environment, the area of the predicted free space expands leading to the decrease of MSE and the increase of correlation and MI in our test set. This validates that ESMN is able to accurately estimate the proximity to the physical obstacles relative to the agent itself and continuously accumulate the belief of free space representations based on the egomotions. Place Unit (PU): Loop Closure Classification. One of the critical aspects of ESM is the ability to recall and recognize a previously visited place independent of head orientation. Place cell (PC) resides in hippocampus and increases firing patterns when the animal is in specific locationsBurgess. Recent anatomical connectivity research BID37 implies that grid cells (GC) are the principle cortical inputs to PC. In our proposed ESMN, we design a novel memory unit (GU) for storing the discriminative representations of visited places based on their relative locations with respect to the agent itself in the egocentric coordinate. GU interacts constantly with PU by reading and writing the memory vectors. We evaluate the learnt discriminative representations of visited Table 2: Ablation study on the global map performance in Maze 6 at t = 448 and t = 1580 using metrics in Section 4.1. From top to bottom, the models are: 3D-CNN baseline, LSTM baseline, our ablated model with PU and GU removed, our proposed architecture. The best values are highlighted in bold.places by comparing the current observation (anchor input) with all the past visited places. FIG5 presents example pairs of observations when the loop closure is detected. Qualitative imply that our ESMN can accurately detect loop closure when the agent is at the previously visited places irrespective of large differences between these camera views. Grid Unit: Mapping Integration and Correction. After the loop closure is detected, ESMN performs map correction to eliminate the discrepancies during long-term mapping. FIG6 presents the example of the predicted global maps after map correction. It is shown that the map gets corrected at t = 448 (Column 1). Thus, the predicted global map (Row 3) is structurally more accurate compared with the one without loop closure (Row 2). Our ablation analysis is as follows: FORMULA0 To study the necessity of egomotion estimation from HDU, we take a sequence of camera views at all the previous time steps as inputs to predict the global map directly. We implement this by using a feed-forward 3D convolution neural network (3D-CNN). Practically, since it is hard to take all the past camera views across very long time period, we choose the input sequence with one representative frame every 15 time steps. As ESM requires sustainable mapping over long durations, we create one more baseline by taking the sequence of camera views as inputs and using Long Short Term Memory architecture to predict global maps directly (LSTM Direct). To maintain the same model complexity, we attach the same 2D-CNN in our BVU module before LSTM and fully connected layers after LSTM. To explore the effect of loop closure and thus map correction, we create one ablated model with PU and GU removed (HDU + BVU). FORMULA3 We present the of our integrated architecture with loop closure classification and map correction enabled (HDU + BVU + PU + GU). We report the evaluation in Table 2 using the metrics MSE, correlation and MI as introduced in Section 4.1.We observe that our proposed architecture surpasses the competitive baselines and the ablated models. At t = 448, compared with the first baseline (3D-CNN), there is decrease of 0.03 in MSE and increase of 0.17 in correlation and 0.09 in MI. The significant improvement infers that it is necessary to estimate egomotion for better map construction. Additionally, the integration of local maps based on the egomotion makes the computation more flexible and efficient by feeding back the accumulative maps to the system for future time steps. In the second baseline (LSTM Direct), we observe that the performance drops significantly when it constructs global maps for longer durations. As GU serves as an external memory to integrate local maps, the baseline confirms GU has advantages over LSTMDirect in terms of long-lasting memory. To explore the effect of loop closure and thus map correction, we have the ablated model with PU and GU removed (HDU + BVU Table 3 : Comparison of our method with state-of-the-art SLAM algorithms. The global map after the agent completes one loop closure in Maze 6, 7, and 8 are reported using metrics introduced in Section 4.1. The numbers highlighted in bold are the best.our proposed architecture with all four modules enabled at t = 448 and t = 1580, the decreased performance validates that these steps are necessary to eliminate the errors during long-term mapping. We evaluate our ESMN by comparing with state-of-the-art monocular visual SLAM methods, , and . We used the codes provided by the authors and replaced the intrinsic camera parameters with ours. In order to construct maps in the world scale, we provide explicit scaling factor for both methods. Results show that our ESMN significantly outperforms the rest in terms of MSE, Cor and MI as shown in Table 3 . In the experiments, we also observe that ORBslam has high false positives in loop closure detection which leads to incorrect map correction. It may be caused by the limitations of local feature descriptors. Different from them, our ESMN can robustly detect loop closure and correct maps over time. Furthermore, we notice that both SLAMs tend to fail to track feature descriptors between frames especially during rotations. The parameters of searching windows in order to match features have to be manually set and they cannot be adapted in large motion cases. The in Maze 8 using EKFslam are not provided in Table 3 as the original algorithm requires the dense local feature matching and it is not applicable in very low texture environments. We get inspirations from neurophysiological discoveries and propose the first deep neural network architecture for modeling ESM which unifies the functionalities of the four navigation cell types: head-direction cells, border cells and boundary vector cells, place cells and grid cells. Our learnt model demonstrates the capacity of estimating the pose of the agent and constructing a top-down 2D spatial representations of the physical environments in the egocentric coordinate which could have many potential applications, such as path planning for robot agents. Our ESMN accumulates the belief about the free space by integrating egocentric views. To eliminate errors during mapping, ESMN also augments the local mapping module with an external spatial memory to keep track of the discriminative representations of the visited places for loop closure detection. We conduct exhaustive evaluation experiments by comparing our model with some competitive baselines and state-of-the-art SLAM algorithms. The experimental demonstrate that our model surpasses all these methods. The comprehensive ablation study suggests the essential role of individual modules in our proposed architecture and the efficiency of communications among these modules. We introduce the anatomy of our model for reproducibility in this section. We follow exactly the same convention as Torch. Inspired by neurophysilogical discoveries, our framework consists of four units: Head Direction Unit (HDU), Boundary Vector Unit (BVU), Place Unit (PU), and Grid Unit (GU). Refer to Table 7 for HDU, Table 6 for BVU, Table 8 for PU, Table 9 for global map correction. In GU, as explained in Section 3.5 in the main manuscript, egocentric spatial memory is often expressed in the coordinate frame of the agent itself, we use location-based addressing mechanism and the locations of reading or writing heads are fixed to be always in the center of the external memory. As the receptive field parameterized by w and h for reading and writing are fixed. We designed a 2D binary masking map where 1 denotes the place Table 5: Reading Head 2 and Writing Head 2 in Grid Unit (GU). It either reads memory vectors or writes the discriminative spatial representation h(I t) of size 1 × 128 from or into the global map of size 128 × 500 × 500 in GU.to read and write and vice versa. We use element-wise multiplication between each feature map of the external memory and the binary masking map for reading and writing operations. Refer to TAB5 for reading and writing local maps in GU, and Table 5 for reading and writing spatial representations of visited places in GU.Figure 9: Example of predicted local maps over first 32 time steps in Maze 6. Frames #1, 5, 9, 13, 17, 21, 25, 29, 32 are shown (left to right columns). The topmost row shows the camera view. Row 2 shows the ground truth with red arrows denoting the agent's position and orientation from the top view. The white region denotes free space while the black denotes unknown areas. Row 3 shows the corresponding top-view accumulative belief of the predicted local maps where the red color denotes higher belief of the free space. Best viewed in color. current camera view estimated egomotion predicted local map at previous time step nn. Sequential nn. ParallelTable nn. SpatialConvolution nn. AffineGridGeneratorBHWD(A, A) nn. Reshape(batchSize, 1, 32, 32) nn. ReLU(true) nn. Transpose({3,4},{2,4}) nn. SpatialConvolution nn. BilinearSamplerBHWD nn. SpatialBatchNormalization (256,1e-3) nn. Transpose({2,4},{3,4}) nn. ReLU(true) nn. Reshape(batchSize, 32, 32) nn. SpatialConvolution nn. ReLU nn. SpatialBatchNormalization (512,1e-3) nn. ReLU(true) nn. SpatialConvolution nn. SpatialBatchNormalization(1024,1e-3) nn. ReLU(true) nn. SpatialFullConvolution nn. SpatialBatchNormalization(512,1e-3) nn. ReLU(true) nn. SpatialFullConvolution nn. SpatialBatchNormalization(256,1e-3) nn. ReLU(true) nn. SpatialFullConvolution nn. SpatialBatchNormalization(128,1e-3) nn. ReLU(true) nn. SpatialFullConvolution nn. SpatialBatchNormalization(1,1e-3) nn. ReLU(true) nn. Squeeze nn. View (batchSize, 32, 32) nn. CAddTable nn. HardTanh Table 6: Architecture of Boundary Vector Unit (BVU) FIG0: Example of predicted local maps over first 32 time steps in Maze 7. Frames #1, 5, 9, 13, 17, 21, 25, 29, 32 are shown (left to right columns). The topmost row shows the camera view. Row 2 shows the ground truth with red arrows denoting the agent's position and orientation from the top view. The white region denotes free space while the black denotes unknown areas. Row 3 shows the corresponding top-view accumulative belief of the predicted local maps where the red color denotes higher belief of the free space. Best viewed in color. 5, 9, 13, 17, 21, 25, 29, 32 are shown (left to right columns). The topmost row shows the camera view. Row 2 shows the ground truth with red arrows denoting the agent's position and orientation from the top view. The white region denotes free space while the black denotes unknown areas. Row 3 shows the corresponding top-view accumulative belief of the predicted local maps where the red color denotes higher belief of the free space. Best viewed in color.contacted two RGB camera views at t and t − 1 nn. Sequential nn. SpatialConvolution nn. ReLU(true) nn. SpatialConvolution nn. SpatialBatchNormalization(256,1e-3) nn. ReLU(true) nn. SpatialConvolution nn. SpatialBatchNormalization(512,1e-3) nn. ReLU(true) nn. SpatialConvolution nn. SpatialBatchNormalization(1024,1e-3) nn. ReLU(true) nn. SpatialConvolution nn. SpatialBatchNormalization(1024,1e-3) nn. ReLU(true) nn. Reshape nn. Linear nn. ReLU nn. Linear nn. ReLU nn. Linear nn. CrossEntropyCriterion Table 7: Architecture of Head Direction Unit (HDU)

first deep neural network for modeling Egocentric Spatial Memory inspired by neurophysiological discoveries of navigation cells in mammalian brain

We propose a novel Intrinsically Motivated Goal Exploration architecture with unsupervised learning of goal space representations, and evaluate how various implementations enable the discovery of a diversity of policies.


Question: Represent the scientific article to find a short TLDR!

Question: We show that if the usual training loss is augmented by a Lipschitz regularization term, then the networks generalize. We prove generalization by first establishing a stronger convergence , along with a rate of convergence. A second resolves a question posed in: how can a model distinguish between the case of clean labels, and randomized labels? Our answer is that Lipschitz regularization using the Lipschitz constant of the clean data makes this distinction. In this case, the model learns a different function which we hypothesize correctly fails to learn the dirty labels. While deep neural networks networks (DNNs) give more accurate predictions than other machine learning methods BID30, they lack some of the performance guarantees of these other methods. One step towards performance guarantees for DNNs is a proof of generalization with a rate. In this paper, we present such a , for Lipschitz regularized DNNs. In fact, we prove a stronger convergence from which generalization follows. We also consider the following problem, inspired by . Problem 1.1. [Learning from dirty data] Suppose we are given a labelled data set, which has Lipschitz constant Lip(D) = O (see below). Consider making copies of 10 percent of the data, adding a vector of norm to the perturbed data points, and changing the label of the perturbed points. Call the new, dirty, data setD. The dirty data has Lip(D) = O(1/). However, if we compute the histogram of the pairwise Lipschitz constants, the distribution of the values on the right hand side of, are mostly below Lip(D) with a small fraction of the values being O(1/), since the duplicated images are apart but with different labels. Thus we can solve with L 0 estimate using the prevalent smaller values, which is an accurate estimate of the clean data Lipschitz constant. The solution of using such a value is illustrated on the right of Figure 1. Compare to the Tychonoff regularized solution on the right of Figure 2. We hypothesis that on dirty data the solution of replaces the thin tall spikes with short fat spikes leading to better approximation of the original clean data. In Figure 1 we illustrate the solution of (with L 0 = 0), using synthetic one dimensional data. In this case, the labels {−1, 0, 1} are embedded naturally into Y = R, and λ = 0.1. Notice that the solution matches the labels exactly on a subset of the data. In the second part of the figure, we show a solution with dirty labels which introduce a large Lipschitz constant, in this case, the solution reduces the Lipschitz constant, thereby correcting the errors. Learning from dirty labels is studied in §2.4. We show that the model learns a different function than the dirty label function. We conjecture, based on synthetic examples, that it learns a better approximation to the clean labels. We begin by establishing notation. Consider the classification problem to fix ideas, although our restuls apply to other problems as well. Definition 1.2. Let D n = x 1,..., x n be a sequence of i.i.d. random variables sampled from the probability distribution ρ. The data x i are in X = d. Consider the classification problem with D labels, and represent the labels by vertices of the probability simplex, Y ⊂ R D. Write y i = u 0 (x i) for the map from data to labels. Write u(x; w) for the map from the input to data to the last layer of the network.1 Augment the training loss with Lipschitz regularization DISPLAYFORM0 The first term in is the usual average training loss. The second term in the Lipschitz regularization term: the excess Lipschitz constant of the map u, compared to the constant L 0.In order to apply the generalization theorem, we need to take L 0 ≥ Lip(u 0), the Lipschitz constant of the data on the whole data manifold. In practice, Lip(u 0) can be estimated by the Lipschitz constant of the empirical data. The definition of the Lipschitz constants for functions and data, as well as the implementation details are presented in §1.3 below. Figure 1: Synthetic labelled data and Lipschitz regularized solution u. Left: The solution value matches the labels exactly on a large portion of the data set. Right: dirtly labels: 10% of the data is incorrect; the regularized solution corrects the errors. Our analysis will apply to the problem which is convex in u, and does not depend explicitly on the weights, w. Of course, once u is restricted to a fixed neural network architecture, the corresponding minimization problem becomes non-convex in the weights. Our analysis can avoid the dependence on the weights because we make the assumption that there are enough parameters so that u can exactly fit the training data. The assumption is justified by. As we send n → ∞ for convergence, we require that the network also grow, in order to continue to satisfy this assumption. Our apply to other non-parametric methods in this regime. Generalization bounds have been obtained previously via VC dimension analysis of neural networks . The generalization rates have factors of the form A k for a k-layer neural network with bounds w i ≤ A for all weight vectors w i in the network. Such bounds are only applicable for low-complexity networks. Other works have considered connections between generalization and stability BID8 ). More recently, BID5 proposed the Lipschitz constant of the network as a candidate measure for the Rademacher complexity, which is a measure of generalization (, Chapter 26). Also, BID12 showed that Lipschitz regularization can be viewed as a special case of distributional robustness. Unlike other recent contributions such as BID26, our analysis does not depend on the training method. In fact, our analysis has more in common with inverse problems in image processing, such as Total Variation denoising and inpainting BID6 BID36. For further discussion, see Appendix C.The estimate of Lip(u; X) provided by can be quite different from the the Tychonoff gradient regularization BID14, DISPLAYFORM0 since corresponds to a maximum of the values of the norms, and the previous equation corresponds to the mean-squared values. In fact, recent work on semi-supervised learning suggests that higher p-norms of the gradient are needed for generalization when the data manifold is not well approximated by the data BID15 BID10 BID29 BID39. In Figure 2 we compare to the problems in Figure 1 using Tychonoff regularization. The Tychonoff regularization is less effective at correcting errors. The effect is more pronounced in higher dimensions. Figure 2: Synthetic labelled data and Tychonoff regularized solution u. Left: The solution value matches the labels exactly on a large portion of the data set. Right dirty labels: 10% of the data is incorrect; the regularized solution is not as effective at correcting errors. The effect is more pronounced in higher dimensions. An upper bound for the Lipschitz constant of the model is given by the norm of the product of the weight matrices (, Section 4.3). Let w = (w 1, . . ., w J) be the weight matrices for each layer. Then DISPLAYFORM0 Regularization of the network using methods based on has been implemented recently in BID23 and . Because the upper bound in does not take into account the coefficients in weight matrices which are zero due to the activation functions, the gap in the inequality can be off by factors of many orders of magnitude for deep networks BID19.Implementing can be accomplished using backpropagation in the x variable on each label, which can become costly for D large. Special architectures could also be used to implement Lipschitz regularization, for example, on a restricted architecture, BID31 renormalized the weight matrices of each layer to be norm 1.Lipschitz regularization may help with adversarial examples BID40 BID22 which poses a problem for model reliability BID21. Since the Lipschitz constant L ℓ of the loss, ℓ, controls the norm of a perturbation DISPLAYFORM1 maps with smaller Lipschitz constants may be more robust to adversarial examples. BID19 implemented Lipschitz regularization of the loss, and achieved better robustness against adversarial examples, compared to adversarial training BID22 alone. Lipschitz regularization may also improve stability of GANs. 1-Lipschitz networks are also important for Wasserstein-GANs ) BID0. In the gradient penalty away from norm 1 is implemented, augmented by a penalty around perturbed points, with the goal of improved stability. Spectral regularization for GANs was implemented in BID33 ). Definition 1.3 (Lipschitz constants of functions and data). Choose norms · Y, and · X on X and Y, respectively. The Lipschitz constant (in these norms) of a function u: DISPLAYFORM0 When X 0 is all of X, we write Lip(u; X) = Lip(u). The Lipschitz constant of the data is given by DISPLAYFORM1 Finlay & Oberman FORMULA0 implement Lipschitz regularization as follows. The basis for the implementation of the Lipschitz constant is Rademacher's Theorem BID18, §3.1), which states that if a function g(x) is Lipschitz continuous then it is differentiable almost everywhere and DISPLAYFORM2 Restricting to a mini-batch, we obtain the following method for estimating the Lipschitz constant. Let u(x; w) be a Lipschitz continuous function. Then max DISPLAYFORM3 For vector valued functions, the appropriate matrix norm must be used, see §B. The variational problem admits Lipschitz continuous minimizers, but in general the minimizers are not unique. When L 0 = Lip(u 0), it is clear that u 0, is a solution of FORMULA0: both the loss term and the regularization term are zero when applied to u 0. In addition, any L 0 -Lipschitz extension of u 0 | Dn is also a minimizer of, so solutions are not unique. Let u n be any solution of the Lipschitz regularized variational problem. We study the limit of u n as n → ∞. Since the empirical probability measures ρ n converge to the data distribution ρ, the continuum variational problem corresponding to is min DISPLAYFORM0 where in FORMULA8 we have introduced the following notation. Definition 2.1. Given the loss function, ℓ, a map u: X → Y, and a probability measure, µ, supported on X, define DISPLAYFORM1 to be the expectation of the loss with respect to the measure. In particular, the generalization loss of the map u: DISPLAYFORM2 for the average loss on the data set D n, where ρ n:= 1 n δ xi is the empirical measure corresponding to D n. Remark 2.2. Generalization is defined in (, Section 5.2) as the expected value of the loss function on a new input sampled from the data distribution. As defined, the full generalization error includes the training data, but it is of measure zero, so removing it does not change the value. We introduce the following assumption on the loss function. Assumption 2.3 (Loss function). The function ℓ: Y × Y → R is a loss function if it satisfies (i) ℓ ≥ 0, (ii) ℓ(y 1, y 2) = 0 if and only if y 1 = y 2, and (iii) ℓ is strictly convex in y 1.Example 2.4 (R D with L 2 loss). Set Y = R D, and let each label be a basis vector. Set ℓ(y 1, y 2) = y 1 − y 2 2 2 to be the L 2 loss. Example 2.5 (Classification). In classification, the output of the network is a probability vector on the labels. Thus Y = ∆ D, the D-dimensional probability simplex, and each label is mapped to a basis vector. The cross-entropy loss ℓ DISPLAYFORM0 Example 2.6 (Regularized cross-entropy). In the classification setting, it is often the case that the softmax function DISPLAYFORM1 is combined with the cross-entropy loss. In this paper, we regard softmax as the last layer of the DNN, so we assume the output u(x) of the network lies in the probability simplex. If the output, z, of the second to last layer of the DNN, which is the input to softmax in, lies in a compact set, i.e., |z j | ≤ C for all i and some C > 0, then softmax(z) j ≥ e −2C, and so the range of softmax lies in the set DISPLAYFORM2 which is strictly interior to the probability simplex. Restricted to A, the cross-entropy loss ℓ KL is strongly convex and Lipschitz continuous, which is required in Theorems 2.12 and 2.11 below. In our analysis, it is slightly more convenient to define the regularized cross entropy loss with parameter > 0 DISPLAYFORM3 For classification problems, where z = e k, we have ℓ KL (y, e k) = −(1 +) log((y k +)/(1 +)), which is Lipschitz and strongly convex for any 0 ≤ y i ≤ 1 within the probability simplex. Thus, the regularized cross entropy ℓ KL satisfies the strong convexity and Lipschitz regularity required by Theorems 2.12 and 2.11 on the whole probability simplex. Here, we show that solutions of the random variational problem converge to solutions of. We make the standard manifold assumption BID11, and assume the data distribution ρ is a probability density supported on a compact, smooth, DISPLAYFORM0 We denote the probability density again by ρ: M → [0, ∞). Hence, the data D n is a sequence x 1,..., x n of i.i.d. random variables on M with probability density ρ. Associated with the random sample we have the closet point projection map σ n: X → {x 1, . . ., x n} ⊂ X that satisfies DISPLAYFORM1 for all x ∈ X. We recall that W 1,∞ (X; Y) is the space of Lipschitz mappings from X to Y. Throughout this section, C, c > 0 denote positive constants depending only on M, and we assume C ≥ 1 and 0 < c < 1. We follow the analysis tradition of allowing the particular values of C and c to change from line to line. We establish that that minimizers of are unique on M in Theorem A.1, which follows from the strict convexity of the loss restricted to the data manifold M. See also FIG0 which shows how the solutions need not be unique off the data manifold. Our first is in the case where Lip[u 0] ≤ L 0, and so the Lipschitz regularizer is not fully active. This corresponds to the case of clean labels. We state our in generality, for approximate minimizers of, and specialize to the case DISPLAYFORM2 Theorem 2.7 (Convergence ). Assume inf x∈M ρ(x) > 0. For any t > 0, with probability at least 1 − Ct DISPLAYFORM3 is any sequence of minimizers of FORMULA0 and DISPLAYFORM4 and Theorem 2.7 applies to the sequence u n, yielding DISPLAYFORM5 It is important to note that Theorem 2.7 does not requires u n to be minimizers of-we just require zero empirical loss, which is often achieved in practice . This allows for approximation errors in solving on the whole domain X, due to the restriction that u must be expressed via a Deep Neural Network. As an immediate corollary, we can prove that the generalization loss converges to zero, and so we obtain generalization. Corollary 2.9. Assume that for some q ≥ 1 the loss ℓ satisfies DISPLAYFORM6 Then under the assumptions of Theorem 2.7 DISPLAYFORM7 holds with probability at least 1 − Ct −1 n −(ct−1).Proof. By FORMULA21, we can bound the generalization loss as follows DISPLAYFORM8 The proof is completed by invoking Theorem 2.7.We now turn to the proof of Theorem 2.7, which requires a bound on the distance between the closest point projection σ n and the identity. The is standard in probability, and we include it for completeness in Lemma 2.10 proved in §A.1. We refer the interested reader to BID34 for more details. Lemma 2.10. Suppose that inf M ρ > 0. Then for any t > 0 DISPLAYFORM9 with probability at least 1 − Ct −1 n −(ct−1).We now give the proof of Theorem 2.7.Proof of Theorem 2.7. Since L[u n, ρ n] = 0 we have u 0 (x i) = u n (x i) for all 1 ≤ i ≤ n. Thus for any x ∈ X we have DISPLAYFORM10 Therefore, we deduce DISPLAYFORM11 The proof is completed by invoking Lemma 2.10. We now consider the setting of Problem 1.1, illustrated in Figure 1 right. We assume that we only have access to a "dirty" label function, which corresponds to an additive error of the form DISPLAYFORM0 where u clean is the label function, and u e: X → Y is some error function, which is assumed to be zero with high probability. Assume that the error vector e has a much larger Lipschitz constant than the labels, so that Lip(u 0) ≫ Lip(u clean).We wish to fit the clean labels, while not fitting the errors, having access only to u 0. The labels correspond to the subset of the data which generate the low Lipschitz constant L clean, while the errors correspond to pairs of labels that generate a high Lipschitz constant. Thus L clean can easily be estimated from the distribution of the pairwise Lipschitz constants of the data. With the goal in mind, we set L 0 = L clean in. The Lipschitz regularizer is active in, which can lead to the solution succeeding in avoiding the dirty labels, as in Figure 1 right. Our main (Theorems 2.12 and 2.11) show that minimizers of J n converge to minimizers of J almost surely as the number of training points n tends to ∞. It is beyond the scope of this work to estimate to what extent the errors are corrected, however we do know that the solution cannot fit u 0 due to the value of the Lipschitz constant, which is already an improvement over the case λ = 0.The proofs for this section can be found in Section A.2. Theorem 2.11. Suppose that ℓ: Y × Y → R is Lipschitz and strongly convex and let L = Lip(u 0). Then for any t > 0, with probability at least 1 − 2t − m m+2 n −(ct−1) all minimizing sequences u n of and all minimizers u * of satisfy DISPLAYFORM1 The next drops the assumption of strong convexity of the loss. Theorem 2.12. Suppose that inf M ρ > 0, ℓ: Y × Y → R is Lipschitz, and let u * ∈ W 1,∞ (X; Y) be any minimizer of. Then with probability one DISPLAYFORM2 where u n is any sequence of minimizers of. Furthermore, every uniformly convergent subsequence of u n converges on X to a minimizer of. Remark 2.13. In Theorem 2.12 and Theorem 2.11, the sequence u n does not, in general, converge on the whole domain X. The important point is that the sequence converges on the data manifold M, and solves the variational problem off of the manifold, which ensures that the output of the DNN is stable with respect to the input. See FIG0. In this section we provide the proof of stated in §2.3. Theorem A.1. Suppose the loss function satisfies Assumption 2.3. If u, v ∈ W 1,∞ (X; Y) are two minimizers of and DISPLAYFORM0 Therefore, w is a minimizer of J and so we have equality above, which yields DISPLAYFORM1 Since ℓ is strictly convex in its first argument, it follows that u = v on M.Proof of Lemma 2.10 of §2.3. There exists M such that for any 0 < ≤ M, we can cover M with N geodesic balls B 1, B 2,..., B N of radius, where N ≤ C −m and C depends only on M BID25. Let Z i denote the number of random variables x 1,..., x n falling in B i. Then DISPLAYFORM2 m. Let A n denote the event that at least one B i is empty (i.e., Z i = 0 for some i). Then by the union bound we deduce DISPLAYFORM3 In the event that A n does not occur, then each B i has at least one point, and so |x DISPLAYFORM4 The proof of Theorem 2.12 requires a preliminary Lemma. Let DISPLAYFORM5 holds with probability at least 1 − 2t DISPLAYFORM6 The estimate FORMULA0 is called a discrepancy BID41 BID25, and is a uniform version of concentration inequalities. A key tool in the proof of Lemma A.5 is Bernstein's inequality BID7 ), which we recall now for the reader's convenience. For X 1,..., X n i.i.d. with variance DISPLAYFORM7 surely for all i then Bernstein's inequality states that for any > 0 DISPLAYFORM8 Proof of Lemma A.5. We note that it is sufficient to prove the for w ∈ H L (X; Y) with M wρ dV ol(x) = 0. In this case, we have w(x) = 0 for some x ∈ M, and so w L ∞ (X;Y) ≤ CL.We first give the proof for M = X = m. We partition X into hypercubes B 1,..., B N of side length h > 0, where N = h −m. Let Z j denote the number of x 1,..., x n falling in B j. Then Z j is a Binomial random variable with parameters n and p j = Bj ρ dx ≥ ch m. By the Bernstein inequality we have for each j that DISPLAYFORM9 provided 0 < ≤ h m. Therefore, we deduce DISPLAYFORM10 holds with probability at least 1−2h DISPLAYFORM11 holds with probability at least 1 − 2t DISPLAYFORM12 trivially holds, and hence we can allow t > n/ log(n) as well. We sketch here how to prove the on the manifold M. We cover M with k geodesic balls of radius > 0, denoted B M (x 1,),..., B M (x k,), and let ϕ 1,..., ϕ k be a partition of unity subordinate to this open covering of M. For > 0 sufficiently small, the Riemannian exponential map exp x: B(0,) ⊂ T x M → M is a diffeomorphism between the ball B(0, r) ⊂ T x M and the geodesic ball B M (x,) ⊂ M, where T x M ∼ = R m. Furthermore, the Jacobian of exp x at v ∈ B(0, r) ⊂ T x M, denoted by J x (v), satisfies (by the Rauch Comparison Theorem) DISPLAYFORM13 Therefore, we can run the argument above on the ball B(0, r) ⊂ R m in the tangent space, lift the to the geodesic ball B M (x i,) via the Riemannian exponential map exp x, and apply the bound DISPLAYFORM14 to complete the proof. Remark A.6. The exponent 1/(m + 2) is not optimal, but affords a very simple proof. It is possible to prove a similar with the optimal exponent 1/m in dimension m ≥ 3, but the proof is significantly more involved. We refer the reader to BID41 for details. Remark A.7. The proof of Theorem 2.12 shows that Γ-converges to almost surely as n → ∞ in the L ∞ (X; Y) topology. Γ-convergence is a notion of convergence for functionals that ensures minimizers along a sequence of functionals converge to a minimizer of the Γ-limit. While we do not use the language of Γ-convergence here, the ideas are present in the proof of Theorem 2.12. We refer to BID9 for details on Γ-convergence. Proof of Theorem 2.12. By Lemma A.5 the event that lim DISPLAYFORM15 for all Lipschitz constants L > 0 has probability one. For the rest of the proof we restrict ourselves to this event. Let u n ∈ W 1,∞ (X; Y) be a sequence of minimizers of, and let u * ∈ W 1,∞ (X; Y) be any minimizer of. Then since DISPLAYFORM16 we have Lip(u n) ≤ Lip(u 0) =: L for all n. By the Arzelà-Ascoli Theorem BID37 there exists a subsequence u nj and a function u ∈ W 1,∞ (X; Y) such that u nj → u uniformly as n j → ∞. Note we also have Lip(u) ≤ lim inf j→∞ Lip(u nj). Since DISPLAYFORM17 Therefore, u is a minimizer of J. By Theorem A.1, u = u * on M, and so u nj → u * uniformly on M as j → ∞. Now, suppose that does not hold. Then there exists a subsequence u nj and δ > 0 such that DISPLAYFORM18 for all j ≥ 1. However, we can apply the argument above to extract a further subsequence of u nj that converges uniformly on M to u *, which is a contradiction. This completes the proof. Proof of Theorem 2.11. Let L = Lip(u 0). By Lemma A.5 DISPLAYFORM19 (FORMULA0 holds with probability at least 1 − 2t − m m+2 n −(ct−1) for any t > 0. Let us assume for the rest of the proof that holds. (LE) is not practical for large scale problems. There has be extensive work on the Lipschitz Extension problem, see, BID28, for example. More recently, optimal Lipschitz extensions have been studied, with connections to Partial Differential Equations, see BID2. We can interpret as a relaxed version of (LE), where λ −1 is a parameter which replaces the unknown Lagrange multiplier for the constraint. Variational problems are fundamental tools in mathematical approaches to image processing BID3 Lipschitz regularization in not nearly as common. It appears in image processing in (, §4.4) BID16 and BID24 ). Variational problems of the form can be studied by the direct method in the calculus of variations BID13. The problem can be discretized to obtain a finite dimensional convex convex optimization problem. The variational problem can also be studied by finding the first variation, which is a Partial Differential Equation BID17, which can then be solved numerically. Both approaches are discussed in BID3. In FIG1 we compare different regularization terms, in one dimension. The difference between the regularizers is more extreme in higher dimensions.

represent the tldr for finding the corresponding scientific article

We prove generalization of DNNs by adding a Lipschitz regularization term to the training loss. We resolve a question posed in Zhang et al. (2016).

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.