Question: Represent the text to find a very short summary (1-2 sentences max).

Question: The vertebrate visual system is hierarchically organized to process visual information in successive stages. Neural representations vary drastically across the first stages of visual processing: at the output of the retina, ganglion cell receptive fields (RFs) exhibit a clear antagonistic center-surround structure, whereas in the primary visual cortex (V1), typical RFs are sharply tuned to a precise orientation. There is currently no unified theory explaining these differences in representations across layers. Here, using a deep convolutional neural network trained on image recognition as a model of the visual system, we show that such differences in representation can emerge as a direct consequence of different neural resource constraints on the retinal and cortical networks, and for the first time we find a single model from which both geometries spontaneously emerge at the appropriate stages of visual processing. The key constraint is a reduced number of neurons at the retinal output, consistent with the anatomy of the optic nerve as a stringent bottleneck. Second, we find that, for simple downstream cortical networks, visual representations at the retinal output emerge as nonlinear and lossy feature detectors, whereas they emerge as linear and faithful encoders of the visual scene for more complex cortical networks. This predicts that the retinas of small vertebrates (e.g. salamander, frog) should perform sophisticated nonlinear computations, extracting features directly relevant to behavior, whereas retinas of large animals such as primates should mostly encode the visual scene linearly and respond to a much broader range of stimuli. These predictions could reconcile the two seemingly incompatible views of the retina as either performing feature extraction or efficient coding of natural scenes, by suggesting that all vertebrates lie on a spectrum between these two objectives, depending on the degree of neural resources allocated to their visual system. Why did natural selection shape our visual representations to be the way they are? Traditionally, the properties of the early visual system have been explained with theories of efficient coding, which are based on the premise that the neural representations are optimal at preserving information about the visual scene, under a set of metabolic constraints such as total firing rate or total number of synapses. These theories can successfully account for the antagonistic center-surround structure of receptive fields (RFs) found in the retina BID0 BID18 BID35 BID21 BID13, as well as for the oriented structure of RFs found in the primary visual cortex V1 BID30 BID3.However, a number of properties of the early visual system remain unexplained. First, it is unclear why RF geometries would be so different in the retina and V1. A study BID36 has proposed that both representations are optimal at preserving visual information under different metabolic constraints: a constraint on total number of synapses for the retina, and one on total firing rate in V1. However, it is unclear why the two systems would be optimized for these two different objectives. Second, there is a great diversity of ganglion cell types at the output the retina BID17, with each cell type tiling the entire visual field and performing a specific computation. Interestingly, some of these types perform a highly nonlinear computation, extracting specific, behaviorally-relevant cues from the visual scene (e.g. direction-selective cells, objectmotion-selective cells), whereas other types are better approximated by a quasi-linear model, and respond to a broad range of stimuli (e.g. midget cells in the primate BID32 and quasi-linear pixel-encoders in the mouse BID20). Intriguingly, although quasi-linear and more nonlinear types exist in species of all sizes (e.g. primate parasol cells are nonlinear BID10), the proportion of cells performing a rather linear encoding versus a nonlinear feature detection seems to vary across species. For example, the most common ganglion cell type in the primate retina is fairly well approximated by a quasi-linear pixel-encoder (midget cells, 50% of all cells and >95% in the central retina BID32 BID11), whereas the most common cell type in mouse acts as a specific feature detector, thought to serve as an alarm system for overhead predators (W3 cells, 13% of all ganglion cells BID38). Again, theories of efficient coding have not been able to account for this diversity of computations found across cell types and across species. The limitations of current efficient coding theories might reside in the simplistic assumption that the objective is to simply relay indiscriminately all visual information to the next stages of processing. Indeed, the ultimate goal of the visual system is to extract meaningful features from the visual scene in order to produce an adequate behavioral response, not necessarily to faithfully encode it. A recent line of work has proposed using the information bottleneck framework as a way to move beyond the simplistic objective of information preservation towards more realistic objectives BID5. Another study has shown that by changing the objective from efficiently encoding the present to efficiently encoding the future (predictive coding), one could better account for the spatio-temporal RFs of V1 cells BID34. Although promising, these approaches were limited to the study of a single layer of neurons, and they did not answer the aforementioned questions about cross-layer or cross-species differences. On the other hand, deep convolutional networks have proven to be accurate models of the visual system, whether they are trained directly on reproducing neural activity BID28 BID4, or on a behaviorally relevant task BID37 BID14 BID4 ), but they have not yet been used to study the visual system through the lens of efficient coding theories. In this study, we trained deep convolutional neural networks on image recognition (CIFAR-10, BID23) and varied their architectures to explore the sets of constraints that could have shaped vertebrates' early visual representations through natural selection. We modeled the visual system with a series of two convolutional networks, one corresponding to the retina and one downstream network corresponding to the ventral visual system in the brain. By varying the architecture of these networks, we first found that a reduction in the number of neurons at the retinal output -corresponding to a realistic physical constraint on the number of fibers in the optic nerve -accounted simultaneously for the emergence of center-surround RFs in our model of the retina, and for the emergence of oriented receptive fields in the primary visual relay of the brain. Second, we found that the degree of neural resources allocated to visual cortices in our model drastically reshaped retinal representations. Given a deep visual cortex, the retinal processing emerged as quasi-linear and retained substantial information about the visual scene. In contrast, for a shallow cortex, the retinal processing emerged as nonlinear and more information-lossy, but was better at extracting features relevant to the object classification task. These observations make testable predictions on the qualitative differences that should be found in retinal representations across species, and could reconcile the seemingly incompatible theories of retinal processing as either performing efficient encoding or feature detection. The retinal architecture is strongly conserved across species BID27, and consists of three layers of feed-forward convolutional neurons (photoreceptors, bipolar cells, ganglion cells) and two layers of inhibitory interneurons (horizontal, amacrine cells). However, we chose to model the retina as a convolutional neural network BID24 with only two layers (fig. 1A). Indeed the retinal response of many species to complex stimuli has been modeled successfully with only one or two-layer models BID12 BID26 BID17, with some rare exceptions of models requiring more layers BID28. We refer to this network as the retina-net. In our simulations, we varied the number of neurons in the second layer of the retina-net, which is the output of the retina, corresponding to the physical bottleneck of the optic nerve conveying all the visual information to the brain (fig. 1B).We modeled the ventral visual system -the system associated with object recognition in the brain BID19 -as a convolutional neural network taking its inputs from the retina-net (fig. 1A). We varied the neural resources allocated to the ventral visual system network (VVS-net) by changing the number of layers it is composed of (fig. 1B).We trained the neural network composed of the retina-net and VVS-net end-to-end on an object classification task (CIFAR-10, fig. 1A -B-C). Even though the visual system does much more than just classify objects in natural images, this objective is already much more complex and biologically realistic than the one used in previous studies of efficient coding, namely preserving all information about the visual scene. Moreover, we are encouraged by the fact that previous studies using this objective have found a good agreement between neural activity in artificial and biological visual networks BID37 BID4.More specifically, we trained a convolutional neural network on a grayscale version of the standard CIFAR-10 dataset for image classification. The retina-net consisted of two convolutional layers with 32 channels and N BN channels respectively, and with ReLU nonlinearities at each layer. The VVSnet consisted of a varying number D V V S of convolutional layers with 32 channels followed by two fully connected layers (the first one with 1024 neurons and the second one with 10 neurons mapping to the 10 object categories), with ReLU nonlinearities at each layer and a softmax nonlinearity at the last layer. The full system encompassing the retina-net and VVS-net thus had 32 → N BN → 32 → 32 →... channels respectively, where we varied the retinal bottleneck width, N BN, as well as the number D V V S of convolutional brain layers (not counting the fully connected layers). In each convolutional layer, we used 9x9 convolutional filters with a stride of 1 at each step. The large filter size was chosen to give the network flexibility in determining the optimal filter arrangement. We trained our network with the RMSProp optimizer for 20 epochs on the training set with batches of size 32. All optimizations were performed using Keras and TensorFlow. For all presented, we Figure 1: Illustration of the framework we used to model early visual representations. A: We trained convolutional neural networks on an image recognition task (CIFAR-10). The networks were composed of two parts, a retina-net and a ventral-visual-system-net (VVS-net), which receives input from the retina-net. B: We varied the number of layers in the VVS-net (white boxes) and the number of channels at the output of the retina-net (blue box). C: Key : A bottleneck at the output of the retina yielded center-surround retinal RFs. A shallow VVS-net yielded more nonlinear retinal responses (linearity is schematized by the red arrow), which better disentangled image classes (represented as bent manifolds). D: Test-set accuracy of all model architectures on CIFAR-10, averaged over ten networks with random initial weights for each architecture. Performance increases with VVS-net depth and retinal channel, indicating that both factors are meaningful constraints on the network in the regime tested.tested statistical significance by training 10 identical networks with different random initializations of weights and biases taken from a Glorot-uniform distribution BID16.After training, we determined the linear approximation of RFs of each convolutional channel of the network in each layer. This was achieved by computing the gradient of the activation of that channel with respect to a blank image. This gradient map gives a first-order approximation of the image pattern that maximally activates the cells in the channel of interest. In the limit of small noise variance, this computation is mathematically equivalent to measuring the cell's spike-triggered average in response to a perturbative white-noise stimulus BID22 BID33, a commonly used method for determining receptive fields in experimental biology BID7. This equivalence allowed us to compare directly the geometries of RFs experimentally measured in biological networks with the ones found in our models. The test accuracy of our neural network model of the visual system at the recognition task increased both with the number of channels in the retinal bottleneck, and with the number of layers in the VVS-net (fig. 1D), confirming that we were in a regime where the restrictions on neural resources in the VVS-net and at the output of the retina were critical to the ability of the network to perform the task. Here we investigate the effects of a dimensionality bottleneck at the retinal output on early visual representations in our model of the visual system. When reducing the number of neurons at the output of the retina we found that RFs with antagonistic center and surround emerged. For N BN = 32, our control setting with no bottleneck at the retinal output, we observed mostly oriented receptive fields in the second layer of the network (FIG1). For N BN = 4, 2, and 1, we observed center-surround receptive fields in the second layer of the network and mostly oriented receptive fields in the third layer, which is the first layer of the ventral visual system in our model (FIG1). We quantified these in App. A. The RF geometries did not depend qualitatively on the VVS-net depth D V V S ( shown for D V V S = 2), except for the shallowest VVS-net tested (D V V S = 0, no convolutional layer, and thus no dimensionality expansion), for which the shape of emergent retinal RFs were variable across trials and difficult to interpret. These are in good agreement with the organization of the biological visual system, where retinal RFs are center-surround and most downstream RFs in primary visual cortex (V1) are sharply oriented BID19, suggesting that the dimensionality bottleneck at the output of the retina is sufficient to explain these differences in representations. It is worth noting that for both conditions (bottleneck and no bottleneck), the RFs of downstream layers in the VVS-net after the first layer exhibited complex shapes that were neither clearly oriented, nor circular, and the RFs in the first layer of the retina did not appear to have any well-defined structure (data not shown).We then tested in our model the hypothesis of Hubel and Wiesel concerning how center-surround cells are pooled to give rise to oriented RFs in V1 BID19. We found that orientation-selective neurons in the VVS-net typically draw primarily from center-surround neurons in the retina-net that are aligned with the direction of the edge, with positive or negative weights corresponding to whether the polarity (light-selective / dark-selective) of the two neurons are consistent or inconsistent (fig FIG1, and App. A for a quantification). These qualitative are in good agreement with Hubel and Wiesel's hypothesis. Of course, this hypothesis remains to be tested in the real brain, since there is no evidence that the micro-circuitry of the brain matches that of our simulation. In the visual system of mammals, the main relay of visual information taking its input from the retina is the LGN (thalamus), which has center-surround RFs and a similar total number of neurons as the retinal output BID19. We created a network reflecting this architecture by having two lowdimensionality layers in a row instead of just one (FIG1). After training, we found center-surround RFs in the two layers with a bottleneck (retinal output and LGN), and oriented RFs in the next layer, corresponding to the primary visual cortex (V1). These suggest that center-surround representations remain advantageous as long as the dimensionality of the representation remains low, and hence dimensionality expansion seems to be the crucial factor explaining the qualitative change of RFs found between LGN and V1.It is an interesting question to ask whether neurons in our model of the VVS are more similar to simple or complex cells BID19. To test this, we performed a one-step gradient ascent on the neural activity of VVS neurons with respect to the image, starting from several random initial images (App. B). If the neurons were acting as simple cells (i.e. are approximately linear in the stimulus), we would expect all optimized stimuli to converge to the same preferred stimulus. On the other hand, if the cells were complex (i.e. OR function between several preferred stimuli), we would expect the emergent preferred stimuli to depend on the exact initialization. Interestingly, we found that most neurons in the first layer of the VVS-net behaved as simple cells, whereas most neurons in the second layer of the VVS-net behaved as complex cells. Note that in biology, both simple and complex cells are found in V1. These expose the fact that anatomical regions of visual cortex involve multiple nonlinearities and hence may map onto more than one layer of our simple model. Indeed, V1 itself is a multilayered cortical column, with LGN inputs coming in to layer 4, and layer 4 projecting to layers 2 and 3 BID19. Simple cells are predominantly found in layer 4 and complex cells are predominantly found in layers 2 and 3. These observations bolster the interpretation that biological V1 may correspond to multiple layers in our model. Local divisive normalization (i.e. local gain control) is an ubiquitous source of nonlinearity in the visual system BID15 BID18 BID12. We thus tested the robustness of our main to a more realistic model of the visual system with local normalization, by adding it at every layer of the network (App. C). We found that receptive fields still emerged as center-surround in the retina-net, and as oriented in our model of V1. We note that the local normalization slightly degraded the performance of the network on the task for all parameter settings we tried. We then verified that the emergence of center-surround RFs in the retina-net is a consequence of reducing the number of neurons at the retinal output, not of reducing the number of channels, our model's equivalent of biological retinal cell types. In the retina, there exist 20-30 types of ganglion cells BID32, each with a different and stereotyped receptive field, density, polarity (i.e. ON or OFF), and nonlinearities. Cells of each type tile the entire visual field like a convolutional channel in our model, so there is a direct analogy between channels in our model and ganglion cell types in the retina. In order to test whether the emergence of center-surround RFs depends on the number of types that we allow, or just on the number of neurons that we allow at the output of the retina (i.e. dimensionality bottleneck), we employed locally connected layers -equivalent to convolutional layers, but without parameter-tying between artificial neurons within a channel at different spatial locations. In this manner, we can limit the number of neurons at the retinal output without imposing a constraint on the number of cell types. Such a network contains too many parameters to be trained from scratch by gradient descent; to work around this, we trained the model stage-wise by first training our convolutional control network (N BN = 32 with parameter tying) and then we trained a three-layers untied network (with bottleneck dimension N BN = 4 in the second layer) to reproduce the edge-like activations of the second layer of the control network. Even in the untied retina-net, in which each neuron is effectively its own channel, we found that centersurround RFs emerged (FIG1), indicating that center-surround RFs are the network's preferred strategy for passing information through a dimensionality bottleneck even when no constraint on the number of cell types is imposed. We then found that the cells cluster in two distinct populations. To demonstrate this, we measured their activations in response to 10000 natural images, computed the first 20 principal components of this 10000-dimensional space, and ran t-SNE to visualize the clustering of neuron types. We found that two distinct clusters emerged, that corresponded visually to ON and OFF center-surround RFs (FIG1). We thus observe in our model the emergence of one of the most prominent axes of dichotomy of biological ganglion cell types, namely the classification of cells in ON and OFF populations with RFs of opposite polarity. To what extent are retinal representations in our model shaped by the degree of neural resources allocated to downstream processing? To investigate this question, we studied the effects of varying the degree of neural resources in the VVS-net, on emergent visual representations in the retina-net. As we increased the number of layers in the VVS-net, the retinal computation became more linear (FIG3, as measured by the ability of the raw image to linearly map onto the neural representation at the retinal output (see methods, and App. F for a visualization of retinal representation as VVS-net depth increases). This observation is consistent with the current state of knowledge of the differences found in retinal representations across vertebrate species with different brain sizes. The linearization of the retinal response with increased brain complexity was true for different values of bottleneck N BN. However, when we did not use any bottleneck (N BN = 32), the trend became non-monotonic, with a peak in linearity of the response when the VVS-net had 1 conv layer (data not shown). Another interesting phenomenon to note is that linearity of the retinal response decreased as we increased the number of channels in the bottleneck, at any fixed brain depth (FIG3 .The two main sources of nonlinearity in the retina are thought to be the inner retinal rectifications (bipolar and amacrine cells, corresponding to the first rectified layer in our model) and the ganglion cell rectification (corresponding to the second rectified layer in our model). As we decreased VVSnet depth, we observed that the retinal response became more nonlinear. Is this increase in response nonlinearity due to the first or second stage of nonlinearity in our retina-net? To test this, we plotted the real response against the response predicted by a purely linear model for the most shallow and for the deepest VVS-nets tested (FIG3 . If the linear prediction were inaccurate because of the first stage of nonlinear processing in the retina-net, we would expect the points on the scatter plot to be scattered around the unit line. If the prediction error were due to the second-stage of nonlinearity, we would expect the linear approximation to make incorrect negative predictions for inactive neurons. In practice, we found that the prediction error of the linear model was partly explained by both stages of nonlinearity in the retina-net model, predicting that both inner retinal nonlinear processing and ganglion cell rectifications should be more pronounced in animals with fewer neural resources in their visual cortices. Why would retinal representations be more linear when the subsequent ventral visual stream has more resources? One hypothesis is that with a restricted number of neurons, the retina must trade-off between the two incentives of compressing visual information in order to transmit it to downstream layers and extracting nonlinear features from the scene to start disentangling the manifolds corresponding to different classes of objects BID8. According to this hypothesis, when the VVS is shallow, the priority of the retina should be to work toward extracting relevant features. When the VVS is deep, the priority of the retina should be to transmit as much visual information as possible for downstream processing. We validated this hypothesis in two ways in our model. Responses of example retina-net output cell to natural images, vs. best linear fit prediction from raw image, for most (top) and least (bottom) deep VVS-nets. Nonlinearity arises from two sources: rectification within the retina-net (corresponds to the spread of the bulk of the point cloud) and rectification at the retina-net output (corresponds to inactive neurons being incorrectly predicted to have negative activations). C: Quality of image reconstruction from the retinal representation as a function of VVS-net depth. The retinal representation retains more information about the raw image for deep VVS-nets. D: Linear separability of classes of objects at the retinal output, as a function of VVS-net depth. Dashed line indicates separability of classes of images from the raw image pixels. Classes are less separable at the retinal output for deeper VVS-nets. E: Performance on CIFAR-10 for a two-layer densely connected network taking its input from the retina-net or from a raw image. Class information is more accessible from retinal representation. F: Class separability at all layers of network for a deep VVS-net (D V V S = 4) with and without bottleneck (N BN = 1 and N BN = 32). Retinal representation of bottleneck network has low separability. However, the first layer of the VVS-net has high separability (see text).First we showed that the retinal representation retained more information about the image as VVSnet complexity increased (FIG3). To estimate information retention, we trained a linear decoder (see methods) from the output of the retina to reconstruct the image and we measured the reconstruction error. The reconstruction error provided a lower bound on the information that the retina retained about the stimulus (note that more information for reconstruction might be accessible by a nonlinear decoder). This corroborated our hypothesis that, as the VVS-net becomes more complex, the retinal representation gets better at retaining visual information for further processing by the VVS-net. Second, we found that different classes of objects of CIFAR-10 (e.g. trucks, frogs) were more linearly separable from the retina-net representation when the VVS-net was shallow than when it was deep (FIG3). To measure linear separability of manifolds, we trained a linear SVM decoder to separate all pairs of classes and evaluated the performance of the SVM classifier on held-out images (see methods). Moreover, we showed that a VVS-net consisting of two fully connected layers only (no convolutional layers) equipped and trained end-to-end with a retina with a tight bottleneck N BN = 1 (dimensionality of retinal output matches dimensionality of the input image) performed better at image recognition than the same VVS-net trained without a retina-net, taking raw images as input (FIG3). Both these corroborate our hypothesis that retinas followed by a simple cortex performs meaningful feature extraction, whereas retinas followed by more complex visual cortices prioritize non-lossy encoding, postponing feature extraction to downstream layers that are better equipped to do it. Next, we show that within a single network, each retinal channel is trading-off between linearly transmitting visual information to the brain, and extracting relevant features for the object classification task. For 10 instantiations of a network with a retinal bottleneck containing 4 channels, we represented the linearity of each of these 4 channels against the linear separability of object categories obtained from each of these representations. We found, across all networks, a systematic negative correlation between linearity and linear separability across all 4 channels (App. D). Again, this strongly suggests that extracting features and transmitting visual information are indeed two competing goals shaping representations in our model of the retina. In the case of the deepest VVS-nets tested, the retinal processing was quasi-linear for the tightest bottleneck (var.expl. = 0.9, N BN = 1, FIG3). One might take this to suggest that the retinanet in such models does little more than copy image information. However the very first layer of the VVS-net after the retina disentangled classes (as measured by linear separability) almost as well as the second layer of a VVS-net without a retina (FIG3, suggesting that the retinal representation, while only moderately linearly separable itself, is especially transformable into a representation with a high linear separability. This suggests that even when the retina-net is quasi-linear, it can still participate in extracting relevant features for downstream processing by the brain. The increased separability allowed by the retinal pre-processing for this deep VVS-net could be due to the linear processing or the slightly nonlinear part of the retinal processing a combination of both linear and nonlinear processing. To distinguish between these hypotheses, we replaced the true retinal processing by its best linear approximation, retrained the VVS-net on the output of this linearized retina, and tested whether separability was as high as with the true retinal processing (App. E). We found that the first layer trained on the output of the linearized retinal representation was indeed much more separable than the first layer of the control network (trained directly on natural images) at separating classes of objects, suggesting that the linear operation done by the retina does indeed play a crucial role in making the representation especially separable for subsequent layers. To estimate the linearity of the response of retinal neurons, we fit a linear model to predict the neural response from the image on 8,000 images. In order to prevent overfitting, we regularized the linear weights with an L2 penalty and optimized the weights using ridge regression. The value of the penalty term was chosen by 10-fold cross-validation on the training set. We then measured the Pearson correlation between the linearized responses and original model responses on a testing set of 2,000 images. To estimate the information about the input image retained by the retinal output representation, we fit a linear model to reconstruct the image from the (fixed) outputs of the trained retina-net of interest. All numerical figures given are variance-explained on the held-out test set. To estimate the linear separability of classes of objects from the neural representation, we trained an SVM classifier between all pairs of classes on half of the testing set of CIFAR-10 (1,000 images that were not used to train the network), and we tested the performance of the SVM classifier on 1,000 held-out images from the testing set, as measured by the percentage of images classified correctly. We then averaged the performance of the SVM across all pairs of classes to obtain the linear separability score. A unified theoretical account for the structural differences between the receptive field shapes of retinal neurons and V1 neurons has until now been beyond the reach of efficient coding theories. BID21 found that efficient encoding of images with added noise and a cost on firing rate produce center-surround RFs, whereas the same task without noise produces edge detectors. However, this observation (as they note) does not explain the discrepancy between retinal and cortical representations. BID36 propose a different set of constraints for the retina and V1, in which the retina optimizes for a metabolic constraint on total number of synapses, whereas V1 optimizes for a constraint on total firing rate. It is not clear why each of these constraints would predominate in each respective system. Here we show that these two representations can emerge from the requirement to perform a biologically relevant task (extracting object identity from an image) with a bottleneck constraint on the dimensionality of the retinal output. Interestingly, this constraint differs from the ones used previously to account for center-surround RFs (number of synapses or total firing rate). It is worth noting that we unsuccessfully tried to reproduce the of BID21 in our network, by adding noise to the image and applying an L1 regularization to the retina-net activations. In our framework (different than the one of BID21 in many ways), the receptive fields of the retina-net without bottleneck remained oriented across the full range of orders of magnitude of noise and L1 regularization that permitted successful task performance. There is a long-standing debate on whether the role of the retina is to extract relevant features from the environment BID25 BID17 BID32, or to efficiently encode all visual information indistinctly BID2 BID0 BID18. In this work, we show that our model of the visual system, trained on the same task and with the same input statistics, can exhibit different retinal representations depending on the degree of neural resources allocated to downstream processing by the ventral visual stream. These suggest the hypothesis that, despite its conserved structure across evolution, the retina could prioritize different computations in different species. In species with fewer brain resources devoted to visual processing, the retina should nonlinearly extract relevant features from the environment for object recognition, and in species with a more complex ventral visual stream, the retina should prioritize a linear and efficient transmission of visual information for further processing by the brain. Although all species contain a mix of quasi-linear and nonlinear cell types, the proportion of quasi-linear cells seems to vary across species. In the mouse, the most numerous cell type is a two-stage nonlinear feature detector, thought to detect overhead predators BID38. In contrast, the most common ganglion cell type in the primate retina is fairly well approximated by a linear filter (midget cells, 50% of all cells and >95% in the central retina BID32 BID11). Note however that two-stage nonlinear models are also present in larger species, such as cat Y-type cells and primate parasol cells BID10, making it difficult to make definitive statements about inter-species differences in retinal coding. To gain a better understanding of these differences, it would be useful to collect a dataset consisting of recordings of complete populations of ganglion cells of different species in response to a common bank of natural scenes. A related question is the role of the parcellation of visual information in many ganglion cell types at the retinal output. A recent theory of efficient coding has shown that properties of midget and parasol cells in the primate retina can emerge from the objective of faithfully encoding natural movies with a cost on the total firing rate traversing the optic nerve BID29. On the other hand, many cell types seem exquisitely sensitive to behaviorally relevant features, such as potential prey or predators BID17. For example, some cell types in the frog are tuned to detect moving flies or looming predators BID25. It is an intriguing possibility that different cell types could subserve different functions within a single species, namely efficient coding of natural scenes for some types and extraction of behaviorally-relevant features for others. In this study we allowed only a limited number of cell types (i.e. convolutional channels) at the retinal output (1 to 4), in order to have a dimensionality expansion between the retinal representation and the representation in the ventral visual stream (32 channels), an important condition to see the retinal center-surround representation emerge. By using larger networks with more channels in the retina-net and the VVS-net, we could study the emergence of a greater diversity of neuron types in our retina-net and compare their properties to real retinal cell types. It would also be interesting to extend our model to natural movies. Indeed, most feature detectors identified to date seem to process some form of image motion: wide-field, local or differential BID32. Adding a temporal dimension to the model would be necessary to study their emergence. In , by studying emergent representations learned by a deep network trained on a biologically relevant task, we found that striking differences in retinal and cortical representations of visual information could be a consequence of the anatomical constraint of transmitting visual information through a low-dimensional communication channel, the optic nerve. Moreover, our computational explorations suggest that the rich diversity of retinal representations found across species could have adaptively co-evolved with the varying sophistication of subsequent processing performed by the ventral visual stream. These insights illustrate how deep neural networks, whose creation was once inspired by the visual system, can now be used to shed light on the constraints and objectives that have driven the evolution of our visual system. The following analysis corroborates our qualitative observation that a dimensionality bottleneck in the retina-net yields center-surround retinal receptive fields and oriented, edge-detecting receptive fields in the first layer of the VVS-net (V1). For a given receptive field, we quantified its orientedness as follows: we displayed rectangular bar stimuli of all possible combinations of width, orientations and spatial translations that fit in the input image window. Among all these combinations, we selected the bar stimulus width, orientation, and translation that yielded the strongest response from the RF. Bars with the same width as the best stimuli were presented at all orientations and translations, and for each orientation, we select the strongest response it produced (across all translations). In this manner we obtained a measure of the strength of a receptive field's preference for all orientations. We measured the strength of each RF preference (maximum strength of response) for its preferred orientation and for the orthogonal orientation, and computed the ratio of these strengths. Completely isotropic filters would be expected to give a ratio of 1, while oriented filters should give higher ratios. Note however that some deviation from 1 may indicate noise in the filter rather than true orientedness. For each network layer, we averaged this ratio across filters (for convolutional layers with multiple layers) and trials (re-training of the same neural network architecture with different random initializations). We found that the average ratios were 1.56(±0.22) for the retinal output, 3.05(±0.30) for the first VVS-net layer, and 2.57(±0.27) for the second VVS-net layer, where error margins given are 95% confidence intervals. To help assess whether retinal RFs were more isotropic than expected by chance, we compared them to receptive fields composed of random Gaussian noise as a baseline. These give an average ratio (as computed above) of 1.97(±0.08), significantly higher than that for retinal RFs. Furthermore, the standard deviation of RF preference across orientations was significantly lower for the retinal RFs (0.118 ± 0.036) than for random RFs (0.177 ± 0.007), also indicating that retinal RFs were more isotropic than expected by chance. We also plot the average RF preference for different orientations at each layer to more comprehensively assess the isotropy of RFs at each network layer. To aggregate across multiple trials and filters, we rotated the coordinates of each receptive field such that its preferred orientation was vertical, and averaged our across filters and trials. (See FIG4 .The confirm our qualitative observations that RFs in the second layer of a vanilla network (N BN = 32) are highly oriented (FIG4) RFs in the second layer (retina output) of a bottleneck network (N BN = 1) are much more isotropic, consistent with center-surround RFs (FIG4 top), and RFs in the layer immediately following the retina-net in the bottleneck network are oriented (FIG4 .We also quantitatively corroborate our observation that oriented receptive fields in the V1 layer pool input from oriented arrays of center-surround filters in the retina-net output layer. We apply our method of isotropy quantification described above to the weight matrix for each input-output filter combination in the V1 convolutional layer. We find that this weight matrix itself exhibits orientedness across filters and trials, confirming our observation ( FIG4). To investigate whether neurons in our model's early layers more closely resembled simple or complex cells, we performed the following analysis. As before, we obtained local linear approximations of receptive fields by computing the gradient in input space with respect to the response of a given neuron. Rather than beginning with a blank input, we ran multiple trials with different randomly initialized inputs. A purely linear cell would give the same no matter the initialization; a somewhat nonlinear but still "simple" cell is expected to give similar across initializations. A "complex" cell is expected to give different RF visualizations for different random inputs, reflecting multiple peaks in its response as a function of input. In FIG5 we show examples of receptive fields at different layers of our retina-net + VVS-net model (with N BN = 1, D V V S = 2) for different random intializations of the image (uniform random in). The retina-net output and first VVS-net layer exhibit "simple" behavior, but the second VVS-net layer exhibits observably "complex" behavior. To quantify this effect, we measure the average (across filters within each layer and re-trainings of the same network architecture) standard deviation of computed RFs (normalized to the range) for each network layer. We found that the average standard deviations were 7.9(±1.1) × 10 −3, 15.4(±0.8) × 10 −3, and 35.9(±0.8) × 10 −3 for the retina-net output, first VVS-net layer, and second VVS-net layer, respectively, where the margins of error given are 95% confidence intervals. These corroborate the observation of significantly more complex behavior in the second VVS-net layer, mirroring the biological phenomenon in which complex cells pool from simple cells in V1. We tested the robustness of our first main finding -that a bottlenecked retina-net + VVS-net model yields center-surround receptive fields in the retina and oriented receptive felds in V1 -to the use of biologically realistic local response normalization at every layer of the network. In particular, we normalized the output x of each channel (row r, column c) of each layer as follows (during training and testing): Retinal representation of bottleneck network has low separability. However, the first layer of the VVS-net has high separability. We additionally plot the separability of the linearized bottleneck (NBN = 1) network (see test) as a function of layer. That the jump in linear separability between layers 2,3 survives linearization suggests that the main effect of retinal processing in this network is whitening rather than non-linear processing. Figure 8: Class separability at all layers of network for a deep VVS-net (D V V S = 4) with and without bottleneck (N BN = 1 and N BN = 32). Retinal representation of bottleneck network has low separability. However, the first layer of the VVS-net has high separability. We additionally plot the separability of the linearized bottleneck (N BN = 1) network (see test) as a function of layer. That the jump in linear separability between layers 2,3 survives linearization suggests that the main effect of retinal processing in this network is whitening (see Fig. 9) rather than nonlinear processing. In the case of the deepest VVS-nets tested, the retinal processing was quasi-linear for the tightest bottleneck (var.expl. = 0.9, N BN = 1, FIG3). However the very first layer of the VVS-net after the retina disentangled classes (as measured by linear separability) almost as well as the second layer of a VVS-net without retina (FIG3), suggesting that the retinal representation, while only moderately linearly separable itself, is especially transformable into a representation with a high linear separability. To determine to what degree this increased separability was due to the linear processing or the slightly nonlinear part of the retinal processing, we performed an ablation experiment to eliminate factor. We first replaced the true retinal processing by its best approximation by a onelayer linear convolution (of sufficient filter width to correspond to two convolutional layers with 9 by 9 filters). After this linearization process, we retrained the VVS-net using the linearized retinal representation as input, keeping the linearized retina weights frozen. We found that the first layer trained on the output of the linearized retinal representation was indeed much better than the first layer of the control network (trained directly on natural images) at separating classes of objects (Figure 9: Visualization of the output of the retina-net (one-channel-bottleneck, i.e. N BN =1) for different images from the testing set (x-axis) as a function of VVS-net depth (y-axis). Each pixel intensity of the retinal image is proportional to the activation of the corresponding neuron of the retina, where light shades indicate high activities and dark shades low activities. While retinas for every VVS-net depth appear to whiten the input, we can see that the retinal image is more and more processed and less and less recognizable as VVS-net depth decreases.

represent the short summary for finding the corresponding scientific article (long)

We reproduced neural representations found in biological visual systems by simulating their neural resource constraints in a deep convolutional model.

represent this text to find a much much longer article about it with say hundreds-thousands of words.

A theoretical analysis of a new class of RNNs, trained on neuroscience tasks, allows us to identify the role of dynamical dimensionality and cell classes in neural computations.


Question: Represent the scientific article to find a short TLDR

Question: While it has not yet been proven, empirical evidence suggests that model generalization is related to local properties of the optima which can be described via the Hessian. We connect model generalization with the local property of a solution under the PAC-Bayes paradigm. In particular, we prove that model generalization ability is related to the Hessian, the higher-order "smoothness" terms characterized by the Lipschitz constant of the Hessian, and the scales of the parameters. Guided by the proof, we propose a metric to score the generalization capability of the model, as well as an algorithm that optimizes the perturbed model accordingly. Deep models have proven to work well in applications such as computer vision BID18 BID8 BID14, speech recognition, and natural language processing BID35 BID6 BID25. Many deep models have millions of parameters, which is more than the number of training samples, but the models still generalize well BID11.On the other hand, classical learning theory suggests the model generalization capability is closely related to the "complexity" of the hypothesis space, usually measured in terms of number of parameters, Rademacher complexity or VC-dimension. This seems to be a contradiction to the empirical observations that over-parameterized models generalize well on the test data 1. Indeed, even if the hypothesis space is complex, the final solution learned from a given training set may still be simple. This suggests the generalization capability of the model is also related to the property of the solution. BID15 and BID1 empirically observe that the generalization ability of a model is related to the spectrum of the Hessian matrix ∇ 2 L(w *) evaluated at the solution, and large eigenvalues of the ∇ 2 L(w *) often leads to poor model generalization. Also, BID15, BID1 and BID31 introduce several different metrics to measure the "sharpness" of the solution, and demonstrate the connection between the sharpness metric and the generalization empirically. BID2 later points out that most of the Hessian-based sharpness measures are problematic and cannot be applied directly to explain generalization. In particular, they show that the geometry of the parameters in RELU-MLP can be modified drastically by re-parameterization. Another line of work originates from Bayesian analysis. first introduced Taylor expansion to approximate the (log) posterior, and considered the second-order term, characterized by the Hessian of the loss function, as a way of evaluating the model simplicity, or "Occam factor". Recently BID34 use this factor to penalize sharp minima, and determine the optimal batch size. BID4 connect the PAC-Bayes bound and the Bayesian marginal likelihood when the loss is (bounded) negative log-likelihood, which leads to an alternative perspective on Occam's razor. BID19, and more recently, BID7 BID28 BID29 use PAC-Bayes bound to analyze the generalization behavior of the deep models. Since the PAC-Bayes bound holds uniformly for all "posteriors", it also holds for some particular "posterior", for example, the solution parameter perturbed with noise. This provides a natural The sharp minimum, even though it approximates the true label better, has some complex structures in its predicted labels, while the flat minimum seems to produce a simpler classification boundary. way to incorporate the local property of the solution into the generalization analysis. In particular, BID28 suggests to use the difference between the perturbed loss and the empirical loss as the sharpness metric. BID3 tries to optimize the PAC-Bayes bound instead for a better model generalization. Still some fundamental questions remain unanswered. In particular we are interested in the following question:How is model generalization related to local "smoothness" of a solution?In this paper we try to answer the question from the PAC-Bayes perspective. Under mild assumptions on the Hessian of the loss function, we prove the generalization error of the model is related to this Hessian, the Lipschitz constant of the Hessian, the scales of the parameters, as well as the number of training samples. The analysis also gives rise to a new metric for generalization. Based on this, we can approximately select an optimal perturbation level to aid generalization which interestingly turns out to be related to Hessian as well. Inspired by this observation, we propose a perturbation based algorithm that makes use of the estimation of the Hessian to improve model generalization. We consider the supervised learning in PAC-Bayes scenario BID24 BID22 BID23 BID20 ). Suppose we have a labeled data set S = DISPLAYFORM0 The PAC-Bayes paradigm assumes probability measures over the function class F: X → Y. In particular, it assumes a "posterior" distribution D f as well as a "prior" distribution π f over the function class F. We are interested in minimizing the expected loss, in terms of both the random draw of samples as well as the random draw of functions: DISPLAYFORM1 Correspondingly, the empirical loss in the PAC-Bayes paradigm is the expected loss over the draw of functions from the posterior: DISPLAYFORM2 PAC-Bayes theory suggests the gap between the expected loss and the empirical loss is bounded by a term that is related to the KL divergence between D f and π f BID20. In particular, if the function f is parameterized as f (w) with w ∈ W, when D w is perturbed around any w, we have the following PAC-Bayes bound BID32 ) BID33 BID28 BID29: DISPLAYFORM3, and π be any fixed distribution over the parameters W. For any δ > 0 and η > 0, with probability at least 1 − δ over the draw of n samples, for any w and any random perturbation u, DISPLAYFORM4 One may further optimize η to get a bound that scales approximately as BID33. 3 A nice property of the perturbation bound FORMULA4 is it connects the generalization with the local properties around the solution w through some perturbation u around w. In particular, supposeL(w *) is a local optimum, when the perturbation level of u is small, E u [L(w * + u)] tends to be small, but KL(w * + u||π) may be large since the posterior is too "focused" on a small neighboring area around w *, and vice versa. As a consequence, we may need to search for an "optimal" perturbation level for u so that the bound is minimized. DISPLAYFORM5 While some researchers have already discovered empirically the generalization ability of the models is related to the second order information around the local optima, to the best of our knowledge there is no work on how to connect the Hessian matrix ∇ 2L (w) with the model generalization rigorously. In this section we introduce the local smoothness assumption, as well as our main theorem. It may be unrealistic to assume global smoothness properties for the deep models. Usually the assumptions only hold in a small local neighborhood N eigh(w *) around a reference point w *. In this paper we define the neighborhood set as DISPLAYFORM0 + is the "radius" of the i-th coordinate. In our draft we focus on a particular type of radius κ i (w *) = γ|w * i | +, but our argument holds for other types of radius, too. In order to get a control of the deviation of the optimal solution we need to assume in N eigh γ, (w *), the empirical loss functionL in is Hessian Lipschitz, which is defined as: Definition 1 (Hessian Lipschitz). A twice differentiable function f (·) is ρ-Hessian Lipschitz if: DISPLAYFORM1 where · is the operator norm. The Hessian Lipschitz condition has been used in the numeric optimization community to model the smoothness of the second-order gradients BID27 BID0 BID13. In the rest of the draft we always assume the following: FORMULA2 is convex, and ρ-Hessian Lipschitz. DISPLAYFORM2 For the uniform perturbation, the following theorem holds: Theorem 2. Suppose the loss function l(f, x, y) ∈, and model weights are bounded |w i | + κ i (w) ≤ τ i ∀i. With probability at least 1 − δ over the draw of n samples, for anyw ∈ R m such that assumption 1 holds DISPLAYFORM3 Theorem 2 says if we choose the perturbation levels carefully, the expected loss of a uniformly perturbed model is controlled. The bound is related to the diagonal element of Hessian (logarithmic), the Lipschitz constant ρ of the Hessian (logarithmic), the neighborhood scales characterized by κ (logarithmic), the number of parameters m, and the number of samples n. Also roughly the perturbation level is inversely related to ∇ 2 i,iL, suggesting the model be perturbed more along the coordinates that are "flat". 4 Similar argument can be made on the truncated Gaussian perturbation, which is presented in Appendix B. In the next section we walk through some intuitions of our arguments. Suppose the empirical loss functionL(w) satisfies the local Hessian Lipschitz condition, then by Lemma 1 in BID27, the perturbation of the function around a fixed point can be bounded by terms up to the third-order, DISPLAYFORM0 For perturbations with zero expectation, i.e., E[u] = 0, the linear term in, E u [∇L(w) T u] = 0. Because the perturbation u i for different parameters are independent, the second order term can also be simplified, since FORMULA10 and assumption 1, it is straightforward to see the bound below holds with probability at least DISPLAYFORM1 DISPLAYFORM2 That is, the "posterior" distribution of the model parameters are uniform distribution, and the distribution supports vary for different parameters. We also assume the perturbed parameters are bounded, i.e., |w i | + κ i (w) ≤ τ i ∀i.5 If we choose the prior π to be u i ∼ U (−τ i, τ i), and then KL(w + u||π) = i log(τ i /σ i).The third order term in is bounded by DISPLAYFORM3 Unfortunately the bound in theorem 2 does not explain the over-parameterization phenomenon since when m n the right hand side explodes. 5 One may also assume the same τ for all parameters for a simpler argument. The proof procedure goes through in a similar way. DISPLAYFORM4 Solve for σ that minimizes the right hand side, and we have the following lemma: Lemma 3. Suppose the loss function l(f, x, y) ∈, and model weights are bounded |w i | + κ i (w) ≤ τ i ∀i. Given any δ > 0 and η > 0, with probability at least 1 − δ over the draw of n samples, for any w * ∈ R m such that assumption 1 holds, DISPLAYFORM5 where DISPLAYFORM6. uniformly perturbed random variables, and DISPLAYFORM7 In our experiment, we simply treat η as a hyper-parameter. Other other hand, one may further build a weighted grid over η and optimize for the best η BID33. That leads to Theorem 2. Details of the proof are presented in the Appendix C and D.5 ON THE RE-PARAMETERIZATION OF RELU-MLP BID2 points out the spectrum of ∇ 2L itself is not enough to determine the generalization power. In particular, for a multi-layer perceptron with RELU as the activation function, one may re-parameterize the model and scale the Hessian spectrum arbitrarily without affecting the model prediction and generalization when cross entropy (negative log likelihood) is used as the loss and w * is the "true" parameter of the sample distribution. In general our bound does not assume the loss to be the cross entropy. Also we do not assume the model is RELU-MLP. As a we would not expect our bound stays exactly the same during the re-parameterization. On the other hand, the optimal perturbation levels in our bound scales inversely when the parameters scale, so the bound only changes approximately with a speed of logarithmic factor. According to Lemma, if we use the optimal σ * on the right hand side of the bound, ∇ 2L (w), ρ, and w * are all behind the logarithmic function. As a consequence, for RELU-MLP, if we do the re-parameterization trick, the change of the bound is small. In the next two sections we introduce some heuristic-based approximations enlightened by the bound, as well as some interesting empirical observations. 6 AN APPROXIMATE GENERALIZATION METRIC AssumingL(w) is locally convex around w *, so that ∇ 2 i,iL (w *) ≥ 0 for all i. If we look at Lemma 3, for fixed m and n, the only relevant term is i log τi σ * i. Replacing the optimal σ *, and using |w i | + κ i (w) to approximate τ i, we come up with PAC-Bayes based Generalization metric, called pacGen, DISPLAYFORM8.6 Even though we assume the local convexity in our metric, in application we may calculate the metric on every points. When ∇ A self-explained toy example is displayed in FIG0. To calculate the metric on real-world data we need to estimate the diagonal elements of the Hessian ∇ 2L as well as the Lipschitz constant ρ of the Hessian. For efficiency concern we follow Adam and approximate ∇ To estimate ρ, we first estimate the Hessian of a randomly perturbed model ∇ (w + u), and then DISPLAYFORM0. For the neighborhood radius κ we use γ = 0.1 and = 0.1 for all the experiments in this section. We used the same model without dropout from the PyTorch example 7. Fixing the learning rate as 0.1, we vary the batch size for training. The gap between the test loss and the training loss, and the metric Ψ κ (L, w *) are plotted in Figure 2. We had the same observation as in BID15 ) that as the batch size grows, the gap between the test loss and the training loss tends to get larger. Our proposed metric Ψ κ (L, w *) also shows the exact same trend. Note we do not use LR annealing heuristics as in BID5 which enables large batch training. Similarly we also carry out experiment by fixing the training batch size as 256, and varying the learning rate. Figure 4 shows generalization gap and Ψ κ (L, w *) as a function of epochs. It is observed that as the learning rate decreases, the gap between the test loss and the training loss increases. And the proposed metric Ψ κ (L, w *) shows similar trend compared to the actual generalization gap. Similar trends can be observed if we run the same model on CIFAR-10 as shown in Figure 3 and Figure 5. Adding noise to the model for better generalization has proven successful both empirically and theoretically BID38 BID10 BID12 BID3 BID30. Instead of only minimizing the empirical loss, (Langford & Caruana, The right hand side of has E u [L(w + u)]. This suggests rather than minimizing the empirical lossL(w), we should optimize the perturbed empirical loss E u [L(w + u)] instead for a better model generalization power. We introduce a systematic way to perturb the model weights based on the PAC-Bayes bound. Again we use the same exponential smoothing technique as in Adam to estimate the Hessian ∇. The details of the algorithm is presented in Algorithm 1, where we treat η as a hyper-parameter. Even though in theoretical analysis E u [∇L · u] = 0, in applications, ∇L · u won't be zero especially when we only implement 1 trial of perturbation. On the other hand, if the gradient ∇L is close to zero, then the first order term can be ignored. As a consequence, in Algorithm 1 we only perturb the parameters that have small gradients whose absolute value is below β 2. For efficiency issues we used a per-parameter ρ i capturing the variation of the diagonal element of Hessian. Also we decrease the perturbation level with a log factor as the epoch increases. We compare the perturbed algorithm against the original optimization method on CIFAR-10, CIFAR-100 BID17, and Tiny ImageNet 8. The are shown in FIG6. We use the Wide-ResNet BID36 as the prediction model. 9 The depth of the chosen model is 58, and the widen-factor is set as 3. The dropout layers are turned off. For CIFAR-10 and CIFAR-100, we use Adam with a learning rate of 10 −4, and the batch size is 128. For the perturbation parameters we use η = 0.01, γ = 10, and =1e-5. For Tiny ImageNet, we use SGD with learning rate 10 −2, and the batch size is 200. For the perturbed SGD we set η = 100, γ = 1, Require: η, γ = 0.1, β 1 = 0.999, β 2 = 0.1, =1e-5. 1: Initialization: DISPLAYFORM0 for minibatch in one epoch do for all i do if t > 0 then 6: DISPLAYFORM0 g t+1 ← ∇ wLt (w t + u t) (get stochastic gradients w.r.t. perturbed loss) 11: DISPLAYFORM1 w t+1 ← OPT(w t) (update w using off-the-shell algorithms) 13: ImageNet. For CIFAR, Adam is used as the optimizer, and the learning rate is set as 10 −4. For the Tiny ImageNet, SGD is used as the optimizer, and the learning rate is set as 10 −2. The dropout method in the comparison uses 0.1 as the dropout rate. Details can be found in Appendix G. and =1e-5. Also we use the validation set as the test set for the Tiny ImageNet. We observe the effect with perturbation appears similar to regularization. With the perturbation, the accuracy on the training set tends to decrease, but the test on the validation set increases. The perturbedOPT also works better than dropout possibly due to the fact that the it puts different levels of perturbation on different parameters according to the local smoothness structures, while only one dropout rate is set for the all the parameters across the model for the dropout method. DISPLAYFORM2 We connect the smoothness of the solution with the model generalization in the PAC-Bayes framework. We prove that the generalization power of a model is related to the Hessian and the smoothness of the solution, the scales of the parameters, as well as the number of training samples. In particular, we prove that the best perturbation level scales roughly as the inverse of the square root of the Hessian, which mostly cancels out scaling effect in the re-parameterization suggested by BID2. To the best of our knowledge, this is the first work that integrate Hessian in the model generalization bound rigorously. It also roughly explains the effect of re-parameterization over the generalization. Based on our generalization bound, we propose a new metric to test the model generalization and a new perturbation algorithm that adjusts the perturbation levels according to the Hessian. Finally, we empirically demonstrate the effect of our algorithm is similar to a regularizer in its ability to attain better performance on unseen data. This section discusses the details of the toy example shown in FIG0. We construct a small 2-dimensional sample set from a mixture of 3 Gaussians, and then binarize the labels by thresholding them from the median value. The sample distribution is shown in FIG0. For the model we use a 5-layer MLP with sigmoid as the activation and cross entropy as the loss. There are no bias terms in the linear layers, and the weights are shared. For the shared 2-by-2 linear coefficient matrix, we treat two entries as constants and optimize the other 2 entries. In this way the whole model has only two free parameters w 1 and w 2. The model is trained using 100 samples. Fixing the samples, we plot the loss function with respect to the model variablesL(w 1, w 2), as shown in FIG0. Many local optima are observed even in this simple two-dimensional toy example. In particular: a sharp one, marked by the vertical green line, and a flat one, marked by the vertical red line. The colors on the loss surface display the values of the generalization metric scores (pacGen) defined in Section 6. Smaller metric value indicates better generalization power. As displayed in the figure, the metric score around the global optimum, indicated by the vertical green bar, is high, suggesting possible poor generalization capability as compared to the local optimum indicated by the red bar. We also plot a plane on the bottom of the figure. The color projected on the bottom plane indicates an approximated generalization bound, which considers both the loss and the generalization metric.10 The local optimum indicated by the red bar, though has a slightly higher loss, has a similar overall bound compared to the "sharp" global optimum. On the other hand, fixing the parameter w 1 and w 2, we may also plot the labels predicted by the model given the samples. Here we plot the prediction from both the sharp minimum FIG0 ) and the flat minimum FIG0. The sharp minimum, even though it approximates the true label better, has some complex structures in its predicted labels, while the flat minimum seems to produce a simpler classification boundary. Because the Gaussian distribution is not bounded but the inequality requires bounded perturbation, we first truncate the distribution. The procedure of truncation is similar to the proof in BID29 and BID24. DISPLAYFORM0 Now let's look at the event DISPLAYFORM1, by union bound P(E) ≥ 1/2. Here erf −1 is the inverse Gaussian error function defined as erf(x) = Suppose the coefficients are bounded such that i w 2 i ≤ τ, where τ is a constant. Choose the prior π as N (0, τ I), and we have DISPLAYFORM2 10 the bound was approximated with η = 39 using inequality Notice that after the truncation the variance only becomes smaller, so the bound of for the truncated Gaussian becomes DISPLAYFORM3 Again whenL(w) is convex around w * such that ∇ (w *) ≥ 0, solve for the best σ i and we get the following lemma: Lemma 4. Suppose the loss function l(f, x, y) ∈, and model weights are bounded i w 2 i ≤ τ. For any δ > 0 and η, with probability at least 1 − δ over the draw of n samples, for any w * ∈ R m such that assumption 1 holds, DISPLAYFORM0 DISPLAYFORM1. random variables distributed as truncated Gaussian, DISPLAYFORM2 and σ * 2 i is the i-th diagonal element in Σ *.Again we have an extra term η, which may be further optimized over a grid to get a tighter bound. In our algorithm we treat η as a hyper-parameter instead. C PROOF OF LEMMA 3Proof. We rewrite the inequality below DISPLAYFORM3 The terms related to σ i on the right hand side of are DISPLAYFORM4 Since the assumption is DISPLAYFORM5 Solving for σ that minimizes the right hand side of FORMULA2, and we have DISPLAYFORM6 The term DISPLAYFORM7 i on the right hand side of FORMULA14 is monotonically increasing w.r.t. σ 2, so DISPLAYFORM8 Combine the inequality, and the equation FORMULA2 with FORMULA2, and we complete the proof. Proof. Combining and FORMULA14, we get DISPLAYFORM0 The following proof is similar to the proof of Theorem 6 in BID33. Note the η in Lemma cannot depend on the data. In order to optimize η we need to build a grid of the form DISPLAYFORM1 For a given value of i log τǐ σi, we pick η j, such that j = 1 2 log i log F A LEMMA ABOUT EIGENVALUES OF HESSIAN AND GENERALIZATION By extrema of the Rayleigh quotient, the quadratic term on the right hand side of inequality FORMULA10 is further bounded by DISPLAYFORM2 This is consistent with the empirical observations of BID15 that the generalization ability of the model is related to the eigenvalues of ∇ 2L (w). The inequality still holds even if the perturbations u i and u j are correlated. We add another lemma about correlated perturbations below. Lemma 5. Suppose the loss function l(f, x, y) ∈. Let π be any distribution on the parameters that is independent from the data. Given δ > 0 η > 0, with probability at least 1 − δ over the draw of n samples, for any local optimal w * such that ∇L(w *) = 0,L(w) satisfies the local ρ-Hessian Lipschitz condition in N eigh κ (w *), and any random perturbation u, s.t., DISPLAYFORM3 Proof. The proof of the Lemma 5 is straightforward. Since ∇L(w *) = 0, the first order term is zero at the local optimal point even if E[u] = 0. By extrema of the Rayleigh quotient, the quadratic term on the right hand side of inequality FORMULA10 is further bounded by DISPLAYFORM4 Due to the linearity of the expected value, DISPLAYFORM5 which does not assume independence among the perturbations u i and u j for i = j. This section contains several figures comparing dropout and the proposed perturbation algorithm. Dropout can be viewed as multiplicative perturbation using Bernoulli distribution. It has already been widely used in almost every deep models. For comparison we present using the exact same wide resnet architectures except the dropout layers are turned on or off. We report the accuracy with dropout rate of 0.0, 0.1, 0.3, and 0.5 on CIFAR-10 and CIFAR-100. For Tiny ImageNet we report the with dropout rate being 0.0, 0.1, and 0.3. Again for the pertOPT algorithm all the dropout layers are turned off. The depth of the chosen wide resnet model BID36 is 58, and the widenfactor is set as 3. For CIFAR-10 and CIFAR-100, we use Adam with a learning rate of 10 −4, and the batch size is 128. For the perturbation parameters we use η = 0.01, γ = 10, and =1e-5. For Tiny ImageNet, we use SGD with learning rate 10 −2, and the batch size is 200. For the perturbed SGD we set η = 100, γ = 1, and =1e-5. Also we use the validation set as the test set for the Tiny ImageNet. Figure FORMULA14,, and show the accuracy versus epochs for training and validation in CIFAR-10, CIFAR-100, and Tiny ImageNet respectively. It is pretty clear that with added dropout the validation/test accuracy got boosted compared to the original method. For CIFAR-10, dropout rate 0.3 seems to work best compared to all the other dropout configurations. For CIFAR-100 and Tiny ImageNet, dropout 0.1 seems to work better. This may be due to the fact that CIFAR-10 has less training samples so more regularization is needed to prevent overfit. Although both perturbedOPT and dropout can be viewed as certain kind of regularization, in all experiments the perturbed algorithm shows better performance on the validation/test data sets compared to the dropout methods. One possible explanation is maybe the perturbed algorithm puts different levels of perturbation on different parameters according to the local smoothness structures, while only one dropout rate is set for all the parameters across the model.

Represent the TLDR for finding the corresponding scientific article!

a theory connecting Hessian of the solution and the generalization power of the model

Represent this text to find a much much longer article about it with say hundreds-thousands of words

General analysis of sign-based methods (e.g. signSGD) for non-convex optimization, built on intuitive bounds on success probabilities.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Unsupervised learning is about capturing dependencies between variables and is driven by the contrast between the probable vs improbable configurations of these variables, often either via a generative model which only samples probable ones or with an energy function (unnormalized log-density) which is low for probable ones and high for improbable ones. Here we consider learning both an energy function and an efficient approximate sampling mechanism for the corresponding distribution. Whereas the critic (or discriminator) in generative adversarial networks (GANs) learns to separate data and generator samples, introducing an entropy maximization regularizer on the generator can turn the interpretation of the critic into an energy function, which separates the training distribution from everything else, and thus can be used for tasks like anomaly or novelty detection. This paper is motivated by the older idea of sampling in latent space rather than data space because running a Monte-Carlo Markov Chain (MCMC) in latent space has been found to be easier and more efficient, and because a GAN-like generator can convert latent space samples to data space samples. For this purpose, we show how a Markov chain can be run in latent space whose samples can be mapped to data space, producing better samples. These samples are also used for the negative phase gradient required to estimate the log-likelihood gradient of the data space energy function. To maximize entropy at the output of the generator, we take advantage of recently introduced neural estimators of mutual information. We find that in addition to producing a useful scoring function for anomaly detection, the ing approach produces sharp samples (like GANs) while covering the modes well, leading to high Inception and Fréchet scores. The early work on deep learning relied on unsupervised learning BID13 BID2 BID17 ) to train energy-based models BID18, in particular Restricted Boltzmann Machines, or RBMs. However, it turned out that training energy-based models without an analytic form for the normalization constant is very difficult, because of the challenge of estimating the gradient of the partition function, also known as the negative phase part of the log-likelihood gradient (described in more details below, Sec. 2). Several algorithms were proposed for this purpose, such as Contrastive Divergence BID12 and Stochastic Maximum Likelihood BID28 BID26, relying on Monte-Carlo Markov Chains (MCMC) to iteratively sample from the energy-based model. However, because they appear to suffer from either high bias or high variance (due to long mixing times), training of RBMs and other Boltzmann machines has not remained competitive after the introduction of variational auto-encoders BID16 ) and generative adversarial networks or GANs.In this paper, we revisit the question of training energy-based models, taking advantage of recent advances in GAN-related research, and propose a novel approach to training energy functions and sampling from them, called EnGAN. The main inspiration for the proposed solution is the earlier observation BID4 made on stacks of auto-encoders that sampling in latent space (and then applying a decoder to map back to data space) led to faster mixing and more efficient sampling. The authors observed that whereas the data manifold is generally very complex and curved, the corresponding distribution in latent space tends to be much simpler and flatter. This was verified visually by interpolating in latent space and projecting back to data space through the decoder, observing that the ing samples look like data samples (i.e., the latent space manifold is approximately convex, with most points interpolated between examples encoded in latent space also having high probability). We propose a related approach, EnGAN, which also provides two energy functions, one in data space and one in latent space. A key ingredient of the proposed approach is the need to regularize the generator (playing the role of the decoder in auto-encoders, but with no need for an encoder) so as to increase its entropy. This is needed to make sure to produce negative examples that can kill off spurious minima of the energy function. This need was first identified by BID15, who showed that in order for an approximate sampler to match the density associated with an energy function, a compromise must be reached between sampling low energy configurations and obtaining a high-entropy distribution. However, estimating and maximizing the entropy of a complex high-dimensional distribution is not trivial, and we take advantage for this purpose of very recently proposed GAN-based approaches for maximizing mutual information BID1 BID24, since the mutual information between the input and the output of the generator is equal to the entropy at the output of the generator. In this context, the main contributions of this paper are the following:• proposing EnGAN, a general architecture, sampling and training framework for energy functions, taking advantage of an estimator of mutual information between latent variables and generator output and approximating the negative phase samples with MCMC in latent space, • showing that the ing energy function can be successfully used for anomaly detection, improving on recently published with energy-based models, • showing that EnGAN produces sharp images -with competitive Inception and Frechet scores -and which also better cover modes than standard GANs and WGAN-GPs, while not suffering from the common blurriness issue of many maximum likelihood generative models. Let x denote a sample in the data space X and E θ: X → R an energy function corresponding to minus the logarithm of an unnormalized density density function DISPLAYFORM0 where Z θ:= e −E θ (x) dx is the partition function or normalizing constant of the density sample in the latent space. Let p D be the training distribution, from which the training set is drawn. Towards optimizing the parameters θ of the energy function, the maximum likelihood parameter gradient is DISPLAYFORM1 where the second term is the gradient of log Z θ, and the sum of the two expectations is zero when training has converged, with expected energy gradients in the positive phase (under the data p D) matching those under the negative phase (under p θ (x)). Training thus consists in matching the shape of two distributions: the positive phase distribution (associated with the data) and the negative phase distribution (where the model is free-running and generating configurations by itself). This observation has motivated the pre-GAN idea presented by BID3 that "model samples are negative examples" and a classifier could be used to learn an energy function if it separated the data distribution from the model's own samples. Shortly after introducing also made a similar connection, related to noise-contrastive estimation BID11. One should also recognize the similarity between Eq. 2 and the objective function for Wasserstein GANs or WGAN. In the next section, we examine a way to train what appears to be a particular form of WGAN that makes the discriminator compute an energy function. The main challenge in Eq. 2 is to obtain samples from the distribution p θ associated with the energy function E θ. Although having an energy function is convenient to obtain a score allowing to compare the relative probability of different x's, it is difficult to convert an energy function into a generative function. The commonly studied approaches for this are based on Monte-Carlo Markov chains, in which one iteratively updates a candidate configuration, until these configurations converge in distribution to the desired distribution p θ. For the RBM, the most commonly used algorithms have been Contrastive Divergence BID12 and Stochastic Maximum Likelihood BID28 BID26, relying on the particular structure of the RBM to perform Gibbs sampling. Although these MCMC-based methods are appealing, RBMs (and their deeper form, the deep Boltzmann machine) have not been competitive in recent years compared to autoregressive models BID27 ), variational auto-encoders and generative adversarial networks or GANs.What has been hypothesized as a reason for the poorer obtained with energy-based models trained with an MCMC estimator for the negative phase gradient is that running a Markov chain in data space is fundamentally difficult when the distribution is concentrated (e.g, near manifolds) and has many modes separated by vast areas of low probability. This mixing challenge is discussed by BID4 who argue that a Markov chain is very likely to produce only sequences of highly probable configurations. If two modes are far from each other and only local moves are possible (which is typically the case with MCMCs), it becomes exponentially unlikely to traverse the'desert' of low probability which can separate two modes. This makes mixing between modes difficult in high-dimensional spaces with strong concentration of probability mass in some places (e.g. corresponding to different categories) and very low probability elsewhere. In the same papers, the authors propose a heuristic method for jumping between modes, based on performing the random walk not in data space but in the latent space of an auto-encoder. Data samples can then be obtained by mapping the latent samples to data space via the decoder. They argue that auto-encoders tend to flatten the data distribution and bring the different modes closer to each other. The EnGAN sampling method proposed here is highly similar but leads to learning both an energy function in data space and one in latent space, from which we find that better samples are obtain. The energy function can be used to perform the appropriate Metropolis-Hastings rejection. Having an efficient way to approximately sample from the energy function also opens to the door to estimating the log-likelihood gradient with respect to the energy function according to Eq. 2, as outlined below. Turning a GAN discriminator into an energy function has been studied in the past BID15 BID31 BID6 but in order to turn a GAN discriminator into an energy function, a crucial and difficult requirement is the maximization of entropy at the output of the generator. Let's see why. In Eq. 2, we can replace the difficult to sample p θ by another generative process, say p G, such as the generative distribution associated with a GAN generator: DISPLAYFORM0 where Ω is a regularizer which we found necessary to avoid numerical problems in the scale (temperature) of the energy. In this paper we use a gradient norm regularizer BID10 ) DISPLAYFORM1 2 for this purpose. This is similar to the training objective of a WGAN as to Eq. 2, but this interpretation allows to train the energy function only to the extent that p G is sufficiently similar to p θ. To make them match, consider optimizing G to minimize the KL divergence KL(p G ||p θ), which can be rewritten in terms of minimizing the energy of the samples from the generator while maximizing the entropy at the output of the generator: DISPLAYFORM2 as already shown by BID15. When taking the gradient of KL(p G ||p θ) with respect to the parameters w of the generator, the partition function of p G disappears and we equivalently can optimize w to minimize DISPLAYFORM3 where p z is the prior distribution of the latent variable of the generator. In order to maximize the entropy at the output of the generator, we propose to exploit another GANderived framework in order to estimate and maximize mutual information between the input and output of the generator network. The entropy at the output of a deterministic function (the generator in our case) can be computed using an estimator of mutual information between the input and output of that function, since the conditional entropy term is 0 because the function is deterministic. With x = G(z) the function of interest: DISPLAYFORM4 Hence, any neural mutual information maximization method such as MINE BID1, noise constrastive estimation BID24 and DeepINFOMAX can be applied to estimate and maximize the entropy of the generator. All these estimators are based on training a discriminator which separates the joint distribution p(X, Z) from the product of the corresponding marginals p(X)p(Z). As proposed by BID5 in the context of using a discriminator to minimize statistical dependencies between the outputs of an encoder, the samples from the marginals can be obtained by creating negative examples pairing an X and a Z from different samples of the joint, e.g., by independently shuffling each column of a matrix holding a minibatch with one row per example. The training objective for the discriminator can be chosen in different ways. In this paper, we used the Deep INFOMAX (DIM) estimator, which is based on maximizing the Jensen-Shannon divergence between the joint and the marginal (see Nowozin et al. for the original F-GAN formulation). DISPLAYFORM5 where s+(a) = log(1+e a) is the softplus function. The discriminator T used to increase entropy at the output of the generator is trained by maximizing I JSD (X, Z) with respect to the parameters of T. With X = G(Z) the output of the generator, IJSD (G(Z), Z) is one of the terms to be minimized the objective function for training G, with the effect of maximizing the generator's output entropy H(G(Z)). The overall training objective for G is DISPLAYFORM6 where Z ∼ p z, the latent prior (typically a N (0, I) Gaussian). Figure 1: EnGAN model overview where G ω is the Generator network, T φ is the Statistics network used for MI estimation and E θ is the energy network One option to generate samples is simply to use the usual GAN approach of sampling a z ∼ p z from the latent prior and then output x = G(z), i.e., obtain a sample x ∼ p G. Since we have an energy function, another option is to run an MCMC in data space, and we have tried this with both Metropolis-Hastings (with a Gaussian proposal) and adjusted Langevin (detailed below, which does a gradient step down the energy and adds noise, then rejects high-energy samples). However, we have interestingly obtained the best samples by considering E θ • G as an energy function in latent space and running an adjusted Langevin in that space (compare Fig. 4 with Fig. 7.1). Then, in order to produce a data space sample, we apply G. For performing the MCMC sampling, we use the Metropolis-adjusted Langevin algorithm (MALA), with Langevin dynamics producing a proposal distribution in the latent space as follows: DISPLAYFORM0 Next, the proposedz t+1 is accepted or rejected using the Metropolis Hastings algorithm, by computing the acceptance ratio DISPLAYFORM1 and accepting (setting z t+1 =z t+1) with probability r. The overall training procedure for EnGAN is detailed in Algorithm 1, with MALA referring to the above procedure for sampling by MCMC, with n mcmc steps. When n mcmc =0, we recover the base case where z is only sampled from the prior and passed through G, and no MCMC is done to clean up the sample. Require: Score penalty coefficient λ, number of energy function updates n ϕ per generator updates, number of MCMC steps n mcmc, number of training iterations T, Adam hyperparameters α, β 1 and β 2. Require: Energy function E θ with parameters θ, entropy statistics network T φ with parameters φ, generator network G ω with parameters ω, minibatch size m for t = 1,..., T do for 1,..., n ϕ do Sample minibatch of real data {x,..., Sample minibatch of latent variables {z DISPLAYFORM0 Per-dimension shuffle of the minibatch z of latent variables, obtaining {z DISPLAYFORM1 The gradient-based updates can be performed with any gradient-based learning rule. We used Adam in our experiments. Generative models trained with maximum likelihood often suffer from the problem of spurious modes and excessive entropy of the trained distribution, where the model incorrectly assigns high probability mass to regions not present in the data manifold. Typical energy-based models such as RBMs suffer from this problem partly because of the poor approximation of the negative phase gradient, as discussed above. To check if EnGAN suffers from spurious modes, we train the energy-based model on synthetic 2D datasets (swissroll, 25gaussians and 8gaussians) similar to BID10 and visualize the energy function. From the probaility density plots on Figure 1, we can see that the energy model doesn't suffer from spurious modes and learns a sharp energy distribution. Bottom: Corresponding probabiltiy density visualizations. Density was estimated using a sample based approximation of the partition function. GANs have been notoriously known to have issues with mode collapse, by which certain modes of the data distribution are not at all represented by the generative model. Similar to the mode dropping issue that occurs in GANs, our generator is prone to mode dropping as well, since it is matched with the energy model's distribution using a reverse KL penalty D KL [P G || P E]. Although the entropy maximization term attempts to fix this issue by maximizing the entropy of the generator's distribution, it is important to verify this effect experimentally. For this purpose, we follow the same experimental setup as BID21 and BID25. We train our generative model on the StackedMNIST dataset, which is a synthetic dataset created by stacking MNIST on different channels. The number of modes can be counted using a pretrained MNIST classifier, and the KL divergence can be calculated empirically between the mode count distribution produced by the generative model and true data (assumed to be uniform). Table 1: Number of captured modes and Kullblack-Leibler divergence between the training and samples distributions for ALI BID7, Unrolled GAN BID21, Vee-GAN BID25, PacGAN BID20, WGAN-GP BID10. Numbers except our model and WGAN-GP are borrowed from BID1 Table 1, we can see that our model naturally covers all the modes in that data, without dropping a single mode. Apart from just representing all the modes of the data distribution, our model also better matches the data distribution as evidenced by the very low KL divergence scores as compared to the baseline WGAN-GP.We noticed empirically that modeling 10 3 modes was quite trivial for benchmark methods such as WGAN-GP BID10. Hence, we also try evaluating our model on a new dataset with 10 4 modes (4 stacks). The 4-StackedMNIST was created to have similar statistics to the original 3-StackedMNIST dataset. We randomly sample and fix 128 × 10 4 images to train the generative model and take 26 × 10 4 samples for evaluations. Generative models trained with maximum likelihood have often been found to produce more blurry samples. Our energy model is trained with maximum likelihood to match the data distribution and the generator is trained to match the energy model's distribution with a reverse KL penalty. To evaluate if our generator exhibits blurriness issues, we train our EnGAN model on the standard benchmark 32x32 CIFAR10 dataset for image modeling. We additionally train our models on the 64x64 cropped CelebA -celebrity faces dataset to report qualitative samples from our model. Similar to recent GAN works BID22, we report both Inception Score (IS) and Frchet Inception Distance (FID) scores on the CIFAR10 dataset and compare it with a competitive WGAN-GP baseline. From TAB1, we can see that in addition to learning an energy function, EnGAN trains generative model producing samples comparable to recent adversarial methods such as WGAN-GP BID10 widely known for producing samples of high perceptual quality. Additionally, we attach samples from the generator trained on the CelebA dataset and the 3-StackedMNIST dataset for qualitative inspection. As shown below in Fig. 4, the visual quality of the samples can be further improved by using the proposed MCMC sampler. Figure 3: Left: 64x64 samples from the CelebA dataset Right: 28x28 samples from the 3-StackedMNIST dataset. All samples are produced by the generator in a single step, without MCMC fine-tuning (see Fig. 4 for that). Apart from the usefulness of energy estimates for relative density estimation (up to the normalization constant), energy functions can also be useful to perform unsupervised anomaly detection. Unsupervised anomaly detection is a fundamental problem in machine learning, with critical applications in many areas, such as cybersecurity, complex system management, medical care, etc. Density estimation is at the core of anomaly detection since anomalies are data points residing in low probability density areas. We test the efficacy of our energy-based density model for anomaly detection using two popular benchmark datasets: KDDCUP and MNIST.KDDCUP We first test our generative model on the KDDCUP99 10 percent dataset from the UCI repository BID19.Our baseline for this task is Deep Structured Energy-based Model for Anomaly Detection (DSEBM) BID30, which trains deep energy models such as Convolutional and Recurrent EBMs using denoising score matching instead of maximum likelihood, for performing anomaly detection. We also report scores on the state of the art DAGMM BID32, which learns a Gaussian Mixture density model (GMM) over a low dimensional latent space produced by a deep autoencoder. We train our model on the KDD99 data and use the score norm ||∇ x E θ (x)|| 2 2 as the decision function, similar to BID30. BID32. Values for our model are derived from 5 runs. For each individual run, the metrics are averaged over the last 10 epochs. +0.1990 F1 score) and is comparable to the current SOTA model (DAGMM) specifically designed for anomaly detection. MNIST Next we evaluate our generative model on anomaly detection of high dimensional image data. We follow the same experiment setup as BID29 and make each digit class an anomaly and treat the remaining 9 digits as normal examples. We also use the area under the precision-recall curve (AUPRC) as the metric to compare models. Table 4: Performance on the unsupervised anomaly detection task on MNIST measured by area under precision recall curve. Numbers except ours are obtained from BID29. Results for our model are averaged over last 10 epochs to account for the variance in scores. Table 4, it can be seen that our energy model outperforms VAEs for outlier detection and is comparable to the SOTA BiGAN-based anomaly detection methods for this dataset BID29 which train bidirectional GANs to learn both an encoder and decoder (generator) simultaneously. An advantage with our method is that it has theoretical justification for the usage of energy function as a decision function, whereas the BiGAN-σ model lacks justification for using a combination of the reconstruction error in output space as well as the discriminator's cross entropy loss for the decision function. To show that the Metropolis Adjusted Langevin Algorithm (MALA) performed in latent space produced good samples in observed space, we attach samples from the beginning (with z sampled from a Gaussian) and end of the chain for visual inspection. From the attached samples, it can be seen that the MCMC sampler appears to perform a smooth walk on the image manifold, with the initial and final images only differing in a few latent attributes such as hairstyle, color, face orientation, etc. Figure 4: Left: Samples at the beginning of the chain (i.e. simply from the ordinary generator, z ∼ N (0, I)). Right: Generated samples after 100 iterations of MCMC using the MALA sampler. We see how the chain is smoothly walking on the image manifold and changing semantically meaningful and coherent aspects of the images. We proposed EnGAN, an energy-based generative model that produces energy estimates using an energy model and a generator that produces fast approximate samples. This takes advantage of novel methods to maximize the entropy at the output of the generator using a GAN-like technique. We have shown that our energy model learns good energy estimates using visualizations in toy 2D data and through performance in unsupervised anomaly detection. We have also shown that our generator produces samples of high perceptual quality by measuring Inception and Frchet scores and shown that EnGAN is robust to the respective weaknesses of GAN models (mode dropping) and maximumlikelihood energy-based models (spurious modes). We found that running an MCMC in latent space rather than in data space (by composing the generator and the data-space energy to obtain a latentspace energy) works substantially better than running the MCMC in data-space. 7.1 MCMC IN DATA SPACE Figure 5: Samples from the beginning, middle and end of the chain performing MCMC sampling in visible space. Initial sample is from the generator (p G) but degrades as we follow MALA directly in data space. Compare with samples obtained by running the chain in latent space and doing the MH rejection according to the data space energy (Fig. 4). It can be seen that MCMC in data space has poor mixing and gets attracted to spurious modes. For all experiments we use Adam as the optimizer with α = 0.0001, β 1 = 0.5, β 2 = 0.9. We used n mcmc = 0 (no MCMC steps during training) for all scores reported in the paper. Toy Data: The generator, energy-model and the statistics network are simple 3-hidden layer MLPs with dimensionality 512. The input to the statistics network is a conatenation of the inputs x and latents z. For these experiments, we use the energy norm co-efficient λ = 0.1StackedMNIST:: In line with previous work, we adopt the same architectural choices for the generator and energy-model / discriminator as VeeGAN BID25. The statistics network is modeled similar to the energy-model, except with the final MLP which now takes as input both the latents z and reduced feature representation of x produced by the CNN. For the CIFAR10 experiments, we adopt the same'Standard CNN' architecture as in SpectralNorm BID22. We adapt the architecture for the Statistics Network similar to the StackedMNIST experiments as mentioned above. For these experiments, we use the energy norm co-efficient λ = 10Anomaly Detection: For the KDD99 dataset, we adopt the same architecture as BID29. We noticed that using n ψ = 1 and λ = 10 5 worked best for these experiments. A large energy norm coefficient was specifically necessary since the energy model overfit to some artifacts in the data and exploded in value. For the MNIST anomaly detection experiments, we use the same architecture as the StackedMNIST experiments.

We introduced entropy maximization to GANs, leading to a reinterpretation of the critic as an energy function.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.