Question: represent the scientific article to find a short tldr:

Question: To understand the inner work of deep neural networks and provide possible theoretical explanations, we study the deep representations through the untrained, random weight CNN-DCN architecture. As a convolutional AutoEncoder, CNN indicates the portion of a convolutional neural network from the input to an intermediate convolutional layer, and DCN indicates the corresponding deconvolutional portion. As compared with DCN training for pre-trained CNN, training the DCN for random-weight CNN converges more quickly and yields higher quality image reconstruction. Then, what happens for the overall random CNN-DCN? We gain intriguing that the image can be reconstructed with good quality. To gain more insight on the intermediate random representation, we investigate the impact of network width versus depth, number of random channels, and size of random kernels on the reconstruction quality, and provide theoretical justifications on empirical observations. We further provide a fast style transfer application using the random weight CNN-DCN architecture to show the potential of our observation. Deep neural networks have achieved impressive performance on various machine learning tasks. However, our understanding of how these deep learning models operate remains limited. Providing a theoretical explanation or empirical interpretation for their success is an important research area. Existing works Arora et al. (2015; 2014); propose mathematical models for learning architectures, however, the theoretical analysis of which fails to capture the state-of-the-art architectures.; leverage either compressive sensing or ordinary differential equations to facilitate the understanding of CNNs.; deliver rigorous proofs about the invertibility of convolutional generative models. Despite these promising progress, there is no solid theoretical foundation on why the overall random CNN-DCN architecture is capable for image reconstruction. In this paper, we bridge the gap between the empirical observation and theoretical explanation of CNNs, especially the invertibility of the overall random CNN-DCN architecture. To understand the deep representations of intermediate layers, a variety of visualization techniques have been developed in order to unveil the feature representation and hence the inner mechanism of convolutional neural networks (CNNs);;;. In this work we propose applying randomization on deconvolutional networks (DCNs) for a systematic investigation of deep representations, and provide insights on the intrinsic properties of deep convolutional networks. We first observe that training the DCN for reconstruction, the random CNN preserves richer information in the feature space. The training on DCN converges faster for the random CNN contrasted to pre-trained CNN and yields higher quality image reconstruction. It indicates there is rich information encoded in the random features; the pre-trained CNN discards some information irrelevant for classification and encodes relevant features in a way favorable for classification but harder for reconstruction. This leads us to be curious about what happens if we feed the images to a CNN-DCN architecture where both the CNN and the DCN have random weights. Our motivation for studying the overall random CNN-DCN architecture is threefold. First, a series of works empirically showed that a certain feature learning architecture with random weights allowed satisfactory discriminative validity on object recognition tasks , and certain convolutional pooling architectures even with random weights can be inherently frequency selective and translation invariant, leading to the potential application of fast search of network architectures. Second, studying a complex system with random weights rather than learned determin-istic ones may lead to a better understanding of the system even in the learned case. For example, in the field of compressed sensing, random sampling leads to breakthroughs in the understanding of the number of required measurements for a stable reconstruction of the signal;. For highly complicated systems with nonlinear operations along the hidden layers, there are already some investigations on random deep neural networks;; Ulyanov et al. (2017a). Third, as a reversible encoder-decoder architecture, deconvolution is a valuable visualization technique for studying the feature representation of deep convolutional nets. To our knowledge there is no existing work on the random deconvolutional networks in the literature. Our work on using deconvolution to study the random intermediate features of CNN provides new insights and inspires possible applications with untrained deep neural models. Our main and contributions are as follows. We study the overall random CNN-DCN architecture to investigate the randomness in deconvolutional networks, i.e. there is no training at all for inverting the inputs that passes their information through a random weight convolutional network. Surprisingly, the image is inverted with satisfactory quality. The geometric and photometric features of the inputs are well preserved given a sufficient number of channels. We provide empirical evidence as well as theoretical analysis on the reconstruction quality, and bound the error in terms of the number of random nonlinearities, the network architecture, the distribution of the random weights, and local similarity of the input which is high for natual images. Extensive empirical study by varying the network width, depth, or kernel size has been performed to show the effectiveness on the inversion. The CNN-DCN architecture with random weights can be very useful on texture synthesis, style transfer, image segmentation, image inpainting, etc. As an example, we illustrate how fast style transfer can be applied using random weight CNN-DCN architecture. Note that our approach can save a big amount of time and energy as we do not need to do the pre-training on deep models, and it is very flexible as we can easily try whatever nerual network architecture as we wish. Two techniques are closely related to our work, deconvolution and randomization. Deconvolution involves a CNN-DCN architecture, where CNN indicates the portion of a convolutional neural network from the input to an intermediate convolutional layer, and DCN indicates the corresponding deconvolutional network aiming to invert the intermediate features to the original images. Randomization indicates the stochastic assignment of weights to the deep neural network. As a generative model for encoder-decoder functions, deconvolutional networks (DCNs) are commonly used for deep feature visualization. propose to use a multi-layered deconvolutional network to project the feature activations back to the input pixel space, and show that the features have many intuitively desirable properties such as compositionality, increasing invariance and class discrimination for deeper layers. design a deconvolution variant to invert image representations learned from a pre-trained CNN, and conclude that features in higher layers preserve colors and rough contours of the images and discard information irrelevant for the classification task that the convolutional model is trained on. As there is no back propagation, their reconstruction is much quicker than the representation inverting method on gradient descent. Randomization on neural networks can be tracked back to the 1960's where the bottom-most layer of shallow networks consisted of random binary connections. In recent years, largely motivated by the fact that "randomization is computationally cheaper than optimization", randomization has been resurfacing repeatedly in the machine learning literature. For optimization problems such as regression or classification, this technique is used to stochastically assign a subset of weights in a feedforward network to derive a simpler optimization problem;. Specifically, they compute a weighted sum of the inputs after passing them through a bank of arbitrary randomized nonlinearities, such that the ing optimization task is formulated as a linear least-squares problem. Empirical comparisons as well as theoretical guarantees are provided for the approximation Rahimi & Recht (2008; 2009);. Other related works include random kernel approximation; and reservoir computing on random recurrent networks;. Specifically on convolutional neural networks (CNNs), there are a few works considering randomization. observe that, on a one-layer convolutional pooling architecture, random weights perform only slightly worse than pre-trained weights. prove that certain convolutional pooling architectures with random weights are inherently frequency selective and translation invariant, and argue that these properties underlie their performance. accomplish three popular visualization tasks, image inversion, texture synthesize and style transfer, using random weight CNNs. extend the scope from fully-connected and convolutional networks and prove that random networks induce representations which approximate the kernel space. combine compressive sensing with random-weight CNNs to investigate the CNN architectures. Dmitry et al. Ulyanov et al. (2017a) utilize randomly-initialized neural nets to finish denoising and inpainting tasks. Motivated by the intuition that "random net is theoretically easier to comprehend than the complicated well-trained net", and that it may reveal the intrinsic property of the network architecture, we use randomization to explore the convolution followed by deconvolution architecture, provide theoretical analysis on empirical observations, and show its application potentials by a style transfer case study. For the network architecture, we focus on VGG16 for the deconvolution. A convolutional layer is usually followed by a pooling layer, except for the last convolutional layer. For consistency, we will explore the "feature representation" after the convolutional layer but before the pooling layer. We build a CNN-DCN architecture on the layer of the feature representation to be studied. The convolution operator of a deconvolutional layer in DCN is the same as the convolution operator in CNN, and an upsampling operator is applied in DCN to invert the corresponding pooling operator in CNN. We will focus on the representations of the convolutional layers, and We first explore the reconstruction ability of random CNNs. We assign Gaussian random weights to the CNN part, and train the corresponding DCN to minimize the summation of the pixel-wise loss on the reconstructed images. Training. For each intermediate layer, using the feature vectors of all training images, we train the corresponding DCN such that the summation of L 2 -norm loss between the inputs and the outputs is minimized. Let Φ(x i, w) represent the output image of the DCN, in which x i is the i th input image and w the weights of the DCN. We train the DCN to get the desired weights w * that minimize the loss. Then for a feature vector of a certain layer, the corresponding DCN can predict an estimation of the expected pre-image, the average of all natural images which would have produced the current feature vector. training. The weight decay is set to 0.0004 to avoid overfitting. The maximum number of iterations is set at 200,000 empirically. We also consider another network architecture,. For the random weights, we try several Gaussian distributions with zero mean and various variance. We also try several other types of random distributions, Uniform, Logistic, Laplace, to have a sound exploration. See more details and comparisons in Appendix 2. In the following, we use CDk to represent a Conv[k]-DeConv[k] architecture. Take the VGG CD2 for elaboration, the loss curves during the training process are shown in Figure 10, which compares VGG and AlexNet on random as well as pre-trained weights. Here Conv2_Pretrained or Conv2_Random indicates whether the CNN is pre-trained or with random weights. We see that the training of DCN for reconstruction converges much quicker on random CNN and yields slightly lower loss. It indicates that by pre-training for classification, CNN encodes relevant features of the input image in a way favorable for classification but harder for reconstruction. And VGG has a much lower reconstruction loss than AlexNet. Reconstruction. We take 5,000 samples from the training set and validation set respectively from ImageNet, and compare their average reconstruction loss. The statistics are shown in Figure 11 ("Pre-trained net" represents pre-trained CNN while "random net" represents random CNN when we train the corresponding DCN for reconstruction). We see the pre-trained CNN and random CNN both have good generalization ability; a random VGG yields much less loss than the pre-trained VGG for the deconvolution reconstruction; for representations of deeper layers, the inverting loss increases significantly for pre-trained VGG but grows slowly for random VGG. The indicate that the random CNN encodes much richer information of the original images; the pre-trained CNN discards information not crucial for classification, especially on deeper layers, leading to a better classifier but a harder reconstruction task. Figure 4 shows reconstructions of various layers of the random VGG on images outside the training set. The reconstruction quality decays for intermediate representations of deeper layers. The VGG structure with random weights yields accurate reconstruction, even on CD5, which involves 26 convolution layers and 4 pooling layers. In Appendix 2, we also see that the reconstruction quality on VGG based deconvolution is better than that on AlexNet based deconvolution. The above inspire us to further explore what happens if both the CNN and DCN are of random weights. In this section we consider the reconstructions on purely random VGG CNN-DCN architecture (denoted by rrVGG for brevity), and find that the images can still be reconstructed with satisfactory quality! In other words, the CNN randomly extracts the image features and passes them to the DCN, then in an unsupervised manner the DCN reconstructs the input image by random feature extraction! Such intriguing show that the overall random CNN-DCN architecture substantially contributes to the geometric and photometric invariance for the image inversion. In the following, we will systematically explore the reconstruction ability of the rrVGG architecture with ReLU nonlinearity. We found that the network depth has a bigger impact than the network width, and the reconstruction quality decays with deeper layers; with plenty number of channels, an increasing number of random channels promotes the reconstruction quality; and the reconstruction quality decays with a larger kernel size. For evaluation, we use the structural similarity (SSIM) index , which is accurate by considering the correlation and dependency of local spatially close pixels, and consistent to the perceptron of human eyes. To remove the discrepancy on colors, we transform the inputs and outputs in grey-scale, and in case of negative SSIM value, we invert the luminosity of the grayscale image for calculation, so the value is in. A higher value indicates a higher similarity on the images. We first explore the impact of network depth and network width for the random reconstruction, using a cat image outside the training data as an example. The weights are random in N (0, 0.1) 1. We first study the reconstruction quality for different convolutional layers, as in Figure 5. Though there is no training at all, DCN can still perceive geometric positions and contours for CD1 to CD3. The deeper the random representations are, the coarser the reconstructed image is. We can still perceive a very rough contour for the random CD4 architecture, which is already 10 layers deep. Our follow-up theoretical analysis will show that depth does affect the , as it affects the size of receptive fields. In Figure 6, we build a Conv1-DeConv1 (CD1) architecture with different dimensions (width) using the actual width of VGG Conv1 to Conv5 for CD1 respectively. We see that the smaller the dimension (width) is, the coarser the image is. We investigate the reconstruction quality on the number of random channels using the rrVGG CD1 (Conv1-DeConv1) architecture. For simplicity, for each network instance we use the same number of channels in all layers except the output layer. We vary the number of random channels from 4, 8 up to 2048, and for each number of channels, we generate 30 rrVGG Conv1-DeConv1 networks and all random weights are in N (0, 0.1) distribution. For input images we randomly pick 50 samples from the ImageNet validation set. To reduce occasionality on the reconstruction, we transform the inputs and outputs in grey-scale and calculate the average SSIM value on each network, then we do statistics (mean and standard deviation) on the 30 average values. Figure 7 shows the trends on SSIM when the number of channels increases, (a) is for the original rrVGG network and (b) is for a variant of rrVGG network. The variant of rrVGG is almost the same as the original network except that the last convolutional layer is replaced by an average layer, which calculates the average over all the channels of the feature maps next to the last layer. We see that the increasing number of random channels promotes the reconstruction quality. Similar in spirit to the random forest method, different channels randomly and independently extract some feature from the previous layer. With sufficient number of random channels we may encode and transform all information to the next layer. In the next section, we will prove for the variant convolutional network, when the width of the random neural network goes to infinity, the output will converge to a fixed image close to the original image. In Figure 8, we pick some input images, and show the corresponding output images closest to the mean SSIM value for various number of channels. We transform the randomly-generated colored image to grey-scale image, for the ease of comparing the structural similarity. The SSIM value is on the top of each output image. The increasing number of channels promotes the random reconstruction quality. To show how the reconstruction quality decays with deeper convolutional layers, we also do experiments on the rrVGG CD2 architecture, and the quality decays by about a half as evaluated by SSIM. We expect that the reconstruction quality decays with larger kernel size, as a large kernel size can not consider the local visual feature of the input. In the extreme case when the kernel size equals the image dimension, the convolution operator actually combines all pixel values of the input to an output pixel using random weights. We use the rrVGG Conv1_1 DeConv1_1 architecture, which simply contains two convolutional operators. The random weights are in N (0, 0.1) distribution. For each kernel size, we randomly generate 30 networks for the reconstruction on 50 sample images as selected above. The verifies our assumption, as in Figure 7 (c). To show how our observation can be used for application, we provide a potential application using random CNN-DCN -Style Transfer with rrVGG. By choosing the suitable number of filters, the rrVGG CD1 architecture can achieve high-quality reconstruction. Besides, these reconstructions can also bring slight differences such as the color and texture, which is suited to exploring more interesting style transfer . And multiple rrV GG models can be efficiently acquired without training. Recent work Gatys et al. In this section, we provide theoretical analysis to explain the empirical . We will show that a slight variant of the random CNN architecture has the ability to reconstruct the input image. We also investigate how depth and width of the network will affect the reconstruction ability. Intuitively, content style rrV GG 1 rrV GG 2 rrV GG 3 Figure 9: Style transfer from several rrVGG models. Each model has the same architecture but different random weights. as the depth of the network increases, the receptive field of each output image pixel becomes larger, which makes the reconstruction harder whereas the width of the network, or equivalently, the number of channels, gives more basis or ways to reconstruct the original input. We theoretically show that the reconstruction ability of a random convolutional neural network rises when the number of channels in each layer increases and drops when the depth of the network increases. Note that DCN is also a kind of CNN with up-sampling layers, so our can be directly applied to the CNN-DCN architecture. For the following part, we will first show the convergence of the output image when the width of the network goes to infinity. Many researchers have worked on the infinite width fully connected networks. For instance,; focus on its relationship with Gaussian Process. They show the exact equivalence between infinitely wide deep networks and Gaussian Processes. Our work focuses on the random convolutional neural network without any training, which is different from the above-mentioned works and is in accordance with our previous empirical analysis. Then, we show the difference between the real output and the convergence value as a function of the width. Finally, we give an upper bound on the angle between the input and the convergence value. Thus, we can bound the reconstruction error. Notations: We use A:,j to denote the j th column vector of matrix A and x to denote the l 2 -norm of vector x. Let L be the number of layers in the neural network and X (i) ∈ R Ni×di be the feature maps in the i th layer, where N i is the number of channels and d i is the dimension of a single channel feature map (i.e. the width of the map times its height). is the output image. w (i,j), a row vector, is the j th convolutional filter of the i th layer if it is a convolutional layer. We use ReLU (x) = max(x, 0) as the activation function in the following analysis. Definition 1. Random CNN architecture To make it possible for property proof, this structure is different from the classic CNN structure in the following three points: 1) Different filters in the same layer are i.i.d. random vectors and filters in different layers are independent. The probability density function of each filter is isotropic. Let k 2) The last layer is the arithmetic mean of the channels of the previous layer, not the weighted combination. 3) Except for X (L−1), each layer of convolutional feature maps are normalized by a factor of, where N i is the number of channels of this layer. From the previous experiments, we see that when the number of channels increases, the quality of the output image improves. Here we prove that when the number of channels goes to infinity, the output will actually converge. Each pixel in the final convergence value is a constant times the weighted norm of its receptive field. Formally, we state our main for the convergence value of the random CNN as follows. Theorem 1. (Convergence Value) Suppose all the pooling layers use l 2 -norm pooling. When the number of filters in each layer of a random CNN goes to infinity, the output f corresponding to a fixed input will converge to a fixed image f * with probability 1, where f * = kz * and k is a constant only related to the CNN architecture and the distribution of random filters and z * i = l∈Ri n (l,i) X:,l 2, where R i is the index set of the receptive field of z * i and n (l,i) is the number of routes from the l th pixel of a single channel of the input image to the i th output pixel. The proof of Theorem 1 is in Appendix 3, Theorem 4. Here for the pooling layer, instead of average pooling, which calculates the arithmetic mean, we use l 2 -norm pooling which calculates the norm of the values in a patch. Intuitively, if most pixels of the input image are similar to their adjacent pixels, the above two pooling methods should have similar outputs. See details for average pooling in Appendix 3. Now we consider the case of a finite number of channels in each layer. We mainly focus on the difference between the real output and the convergence value as a function of the number of channels. We prove that for our random CNN architecture, as the number of channels increases, with high probability, the angle between the real output and the convergence value becomes smaller, which is in accordance with the variant rrVGG experiment shown in previous section. Theorem 2. (Multilayer Variance) Suppose all the pooling layers use l 2 -norm pooling. For a random CNN with L layers and N i filters in the i th layer, let Θ denote the angle between the output f and the convergence value f *, suppose that there is at most one route from an arbitrary input pixel to an arbitrary output pixel for simplicity, then with probability 1 − δ, where Here, (i) (x) actually measures the local similarity of x. The full definition of (i) (x) and the proof of this theorem is in Appendix 3. Finally, we focus on how well our random CNN architecture can reconstruct the original input image. From Theorem 2, we know that with high probability, the angle between the output of a random CNN with finite channels and the convergence value will be upper-bounded. Therefore, to evaluate the performance of reconstruction, we focus on the difference between the convergence value and the input image. We will show that if the input is an image whose pixels are similar to their adjacent pixels, then the angle between the input image X and the convergence value of the output image will be small. To show the essence more clearly, we state our for a two-layer random CNN and provide the multi-layer one in Appendix 3, which needs more complicated techniques but has the same insight as the two-layer one. Theorem 3. For a two-layer random CNN, suppose each layer has a zero-padding scheme to keep the output dimension equal to the dimension of the original input. The kernel size is r and stride is 1. The input image is X ∈ R d0, which has only one channel, whose entries are all positive. t = X t − X t means the difference between one pixel X t and the mean of the r-sized image patch whose center is X t. Let Φ be the angle between the input image X and the convergence value of the output image, we have cos The full proof of Theorem 3 is in Appendix 3. Note that when the kernel size r increases, t will become larger as an image only has local similarity, so that the lower bound of the cosine value becomes worse, which explains the empirical in previous section. In this work, we introduce a novel investigation on deep random representations through the convolution-deconvolution architecture, which to our knowledge is the first study on the randomness of deconvolutional networks in the literature. We extensively explore the potential of randomness for image reconstruction on deep neural networks, and found that images can be reconstructed with satisfactory quality when there are a sufficient number of channels. Extensive investigations have been performed to show the effectiveness of the reconstruction. We also provide theoretical analysis that a slight variant of the random CNN architecture has the ability to reconstruct the input image, and the output converges to the input image when the width of the network, i.e. number of channels, goes to infinity. We also bound the reconstruction error between the input and the convergence value as a function of the network width and depth. and. A convolutional layer is usually followed by a pooling layer, except for the last convolutional layer, Conv5. For consistency, we will explore the output after the convolutional layer but before the pooling layer. In what follows, "feature representation" or "image representation" denotes the feature vectors after the linear convolutional operator and the nonlinear activation operator but before the pooling operator for dimension reduction. We build a CNN-DCN architecture on the layer of feature representation to be studied. The convolution operator of a deconvolutional layer in DCN is the same as the convolution operator in CNN, and an upsampling operator is applied in DCN to invert the corresponding pooling operator in CNN, as designed in. We will focus on the representations of the convolutional layers, since build DCNs for each layer of the pre-trained AlexNet and find that the predicted image from the fully connected layers becomes very vague. For the activation operator, we apply the leaky ReLU nonlinearity with slope 0.2, that is, r(x) = x if x ≥ 0 and otherwise r(x) = 0.2x. At the end of the DCN, a final Crop layer is added to cut the output of DeConv1 to the same shape as the original images. We build deconvolutional networks on both VGG16 and AlexNet, and most importantly, we focus on the random features of the CNN structure when training the corresponding DCN. Then we do no training for deconvolution and explore the properties of the purely random CNN-DCN architecture on VGG16. For the random weights assigned to CNN or DCN, we try several Gaussian distributions with zero mean and various variance to see if they have different impact on the DCN reconstruction. Subsequent comparison shows that a small variance around 0.015 yields minimal inverting loss. We also try several other types of random distributions, Uniform, Logistic, Laplace, to study their impact. • The Uniform distribution is in [-0.04, 0.04), such that the interval equals [µ − 3δ, µ + 3δ] where µ = 0 and δ = 0.015 are parameters for Gaussian distribution. • The Logistic distribution is 0-mean and 0.015-scale of decay. It resembles the normal distribution in shape but has heavier tails. • The Laplace distribution is with 0 mean and 2 * λ 2 variance (λ = 0.015), which puts more probability density at 0 to encourage sparsity. For each intermediate layer, using the feature vectors of all training images, we train the corresponding DCN such that the summation of L 2 -norm loss between the inputs and the outputs is minimized. Let Φ(x i, w) represent the output image of the DCN, in which x i is the input of the i th image and w is the weights of the DCN. We train the DCN to get the desired weights w * that minimize the loss. Then for a feature vector of a certain layer, the corresponding DCN can predict an estimation of the expected pre-image, the average of all natural images which would have produced the given feature vector. training. The weight decay is set to 0.0004 to avoid overfitting. The maximum number of iterations is set at 200,000 empirically. Training. We observe similar for the training loss in different layers. Take the Conv2-DeConv2 architecture for elaboration, the loss curves during the training process are shown in Figure 10. Figure 10 (a) compares VGG and AlexNet on random as well as pre-trained weights. The training for reconstruction converges much quicker on random CNN and yields slightly lower loss, and this trend is more apparent on VGG. It indicates that by pre-training for classification, CNN encodes relevant features of the input image in a way favorable for classification but harder for reconstruction. Also, VGG yields much lower inverting loss as compared with AlexNet. Figure 10 (b) shows that random filters of different small-variance Gaussian distributions on CNN affect the initial training loss, but the loss eventually converges to the same magnitude. (The loss curve of N is not included as the loss is much larger even after the converge.) Figure 10 (c) shows that the four different random distributions with appropriate parameters acquire similar reconstruction loss. Generalization. We take 5000 samples from the training set and validation set respectively from ImageNet, and compare their average reconstruction loss. The statistics is as shown in Figure 11, where CDk represents a Conv[k]-DeConv[k] architecture. Figure 11 (a) shows that the VGG architecture is good in generalization for the reconstruction, and random VGG yields much less loss than pre-trained VGG. For representations of deeper layers, the inverting loss increases significantly for pre-trained VGG but grows slowly for random VGG. This means that in deeper layers, the pre-trained VGG discards much more information that is not crucial for classification, leading to a better classifier but a harder reconstruction task. Figure 11 (b) compares VGG and AlexNet on the CD3 architecture. It shows that the reconstruction quality on random compares favourably against that on pre-trained in VGG. Reconstruction. Figure 12 shows reconstructions from various layers of random VGG and random AlexNet, denoted by rwVGG and rwAlexNet respectively. 2 On both rwVGG and rwAlexNet, the reconstruction quality decays for representations of deeper layers. The rwVGG structure yields more accurate reconstruction, even on Conv5, which involves 26 convolution operations and 4 max pooling operations. Figure 13 shows reconstructions from a cat example image for various distributions of rwVGG CD2. Except for N, the reconstruction quality is indistinguishable by naked eyes. It shows that different random distributions work well when we set the random weights relatively sparse. In a nutshell, it is interesting that random CNN can speed up the training process of the DCN on both VGG and AlexNet, obtain higher reconstruction quality and generalize well for other inputs. Regarding weights in the convolutional part as a feature encoding of the original image, then the deconvolutional part can decode from the feature representations encoded by various methods. The fact that the random encoding of CNN is easier to be decoded indicates that the training for classification moves the image features of different categories into different manifolds that are moving further apart. Also, it may discard information irrelevant for the classification. The pre-trained CNN benefits the classification but is adverse to the reconstruction. For completeness, we repeat the notations and the definition of random CNN architecture. Notations: We use A:,j to denote the j th column vector of matrix A and use A ij to denote its entry. Let x i be the i th entry of vector x. Let L be the number of layers in the neural network and X (i) ∈ R Ni×di be the feature maps in the i th layer, where N i is the number of channels and d i is the dimension of a single channel feature map (i.e. the width of the map times its height). X = X is the input image and f = X (L) is the output image. For convenience, we also define convolutional feature maps to be the feature maps after convolutional operation and define pooled feature maps and up-sampled feature maps in the same way. In the i th layer, let r i be the fixed kernel size or the pool size (e.g. 3 × 3). If X (i+1) is convolutional feature maps, let Y (i) ∈ R Niri×di be the patched feature for pooling and up-sampling layers. For the j th pixel of the output image in the last layer, define its receptive filed on the input image in the first layer as X:,Rj = {X :,m | m ∈ R j}, where R j is a set of indexes. The activation function ReLU (x) = max(x, 0) is the element-wise maximum operation between x and 0 and (·) m is the element-wise power operation. Definition 2. Random CNN architecture This structure is different from the classic CNN in the following three points: • Different filters in the same layer are i.i.d. random vectors and filters in different layers are independent. The probability density function of each filter is isotropic. Let k 4 all exist. • The last layer is the arithmetic mean of the channels of the previous layer, not the weighted combination. • Except for X (L−1), each layer of convolutional feature maps are normalized by a factor of, where N i is the number of channels of this layer. A.3.1 CONVERGENCE Theorem 4. (Convergence Value) Suppose all the pooling layers use l 2 -norm pooling. When the number of filters in each layer of a random CNN goes to infinity, the output f corresponding to a fixed input will converge to a fixed image f * with probability 1, where f * = kz * and k is a constant only related to the CNN architecture and the distribution of random filters and z * i = l∈Ri n (l,i) X:,l 2, where n (l,i) is the number of routes from the l th input pixel to the i th output pixel. Here for the pooling layer, instead of average pooling, which calculates the arithmatic mean, we use l 2 -norm pooling which calculates the norm of the values in a patch. We also show the for average pooling in Theorem A.7. To prove the theorem, we first prove the following lemma. Lemma 5. Suppose w ∈ R n, n ≥ 2 is a random row vector and its probability density function is isotropic. Y ∈ R n×d is a constant matrix whose i th column vector is denoted by y i. z ∈ R d is a row vector and where θ ij is the angle between y i and y j. Proof. Note that max{·, ·} and (·) m are both element wise operations. The i th element of Eg m is (Eg m) i = Emax{wy i, 0} m. Since the probability density function of w is isotropic, we can rotate y i to y i without affecting the value of Emax{wy i, 0} m. Let Where the third equality uses the fact that the marginal distribution of w 1 is also isotropic. Similarly, we have: Eg i g j = Emax{wy i, 0}max{wy j, 0}. We can also rotate y i and y j to y i and y j. Let y i = (y i, 0, 0, ..., 0) T and y j = (y j cos θ ij, y j sin θ ij, 0, ..., 0) T and suppose the marginal probability density function of (w 1, w 2) is p(ρ) which does not depend on φ since it is isotropic, where ρ = w 2 1 + w 2 2 is the radial coordinate and φ is the angular coordinate. We have: Note that: We obtain the second part of this lemma. Now, we come to proof of Theorem A.1. Proof. According to Lemma A.2, if X (i+1) is convolutional feature maps, we can directly obtain: where we have fixed Y (i) and the expectation is taken over random filters in the i th layer only. Since different channels in X (i+1) are i.i.d. random variables, according to the strong law of large numbers, we have: which implies that with probability 1, Suppose that all N j for 1 ≤ j ≤ i have gone to infinity and z (i) has converged to z * (i), the above expression is the recurrence relation between z * (i+1) and z * (i) in a convolutional layer: If X (i+1) is up-sampled feature maps, a pixel X (i) jp will be up-sampled to a r-sized block {X jp and all the other elements are zeros. Definẽ D So far, we have obtained the recurrence relation in each layer. In order to get z * (i+1) given z * (i), we use the same sliding window scheme on z * (i) as that of the convolutional, pooling or upsampling operation on the feature maps. The only difference is that in a convolutional layer, instead of calculating the inner product of a filter and the vector in a sliding window, we simply calculate the l 2 -norm of the vector in the sliding window and then multiply it by k (i) 2. Note that z * can be directly obtained from the input image. Repeat this process layer by layer and we can obtain z * (L−2). According to Lemma A.2, we have: Suppose that z (L−2) has converged to z * (L−2), and by Definition A., we have f * = kz *. Note that z * is obtained through a multi-layer sliding window scheme similar to the CNN structure. It only depends on the input image and the scheme. It is easy to verify that z * i is the square root of the weighted sum of the square of input pixel values within the receptive field of the i th output pixel, where the weight of an input image pixel is the number of routes from it to the output pixel. Theorem 6. (Variance) For a two-layer random CNN with N filters in the first convolutional layer, let Θ denote the angle between the output f and the convergence value f *, then with probability 1 − δ, Proof. According to Theorem A.1, we have. For a two-layer CNN, we can directly obtain: Since different channels are i.i.d. random variables, we have EX According to Markov inequality, we have: 1 ) 2 ) z * 2, then with probability 1 − δ: To extend the above two-layer to a multi-layer one, we first prove the following lemma. Note that in this lemma, D (i) should be replaced byD (i) defined in the proof of Theorem A.1 if X (i) is up-sampled feature maps. is a linear mapping. For simplicity, suppose that for 2 for convolutional layers and k (i) = 1 for pooling and up-sampling layers and Proof. According to the definition of φ (i) (·) and Theorem A.1, we have: It is easy to verify that for any c ∈ R and x, y ∈ R di+1 we have x j, which is the average value of the m th patch.. We have: m, which implies that Theorem 8. (Multilayer Variance) Suppose all the pooling layers use l 2 -norm pooling. For a random CNN with L layers and N i filters in the i th layer, let Θ denote the angle between the output f and the convergence value f *, suppose that there is at most one route from an arbitrary input pixel to an arbitrary output pixel for simplicity, then with probability 1 − δ, where Proof. We will bound Θ recursively. Suppose that the angle between (z (i) ) 2 and (z * (i) ) 2 is θ i. We tional feature maps, we have obtained in the proof of Theorem A.1 that Using similar method to the proof of Theorem A.3 and let α i+1 denote the angle between g (i+1) and Eg (i+1), we can derive that with probability 1 − δ i+1, For a l 2 -norm pooling layer or an up-sampling layer, we have:, we have Let v denote the angle between z (L−2) and f, we have obtained its bound in Theorem A.3. With. With all the bounds above, define N by for simplicity, we can obtain the bound of Θ: with probability 1 − δ, Here, R t is the index set of the receptive field of f t and n (α,t) is the number of routes from X α to f t. Suppose that the receptive field of each f t has the same size and shape, X t is at a fixed relative position of the receptive field of f t and n (α,t) only depends on the relative position between X α and X t. Let X t = α∈R t n (α,t) Xα α∈R t n (α,t) be the weighted average and t = X t − X t. By using the same technique above, we can obtain that Note that although the bound is the same as the two-layer convolutional neural network, as the receptive field is enlarged, t can be much larger, so that the above bound will be worse. We also give the convergence value for average pooling in the next theorem. Theorem 10. (Convergence Value, average pooling) Suppose all the pooling layers use average pooling. When the number of filters in each layer of a random CNN goes to infinity, the output f corresponding to a fixed input will converge to a fixed image f * with probability 1. is convolutional feature maps, according to Lemma A.2, we have: where ϕ Note that: Suppose that all N j for 1 ≤ j ≤ i have gone to infinity and C (i) has converged to C * (i), the above expressions are the recurrence relation between C * (i+1) and C is average-pooled feature maps, we have: We have: which is the recurrence relation for an average pooling layer. For an up-sampling layer, a pixel X (i) jk will be up-sampled to a block {X jk and all the other elements are zeros. We have: otherwise. Note that we can directly calculate C * according to the input image. So we can recursively obtain C * (L−2) and thus z * (L−2). According to Lemma A.2, we have: Suppose that z (L−2) has converged to z * (L−2), and by Definition A., we have: f a.s. We can obtain the convergence value f * through the above process. We observe that by choosing a suitable number of random filters, the rrVGG Conv1-DeConv1 architecture can achieve high-quality reconstruction. The reconstructions also bring slight differences in the color and texture, which is suited for exploring more interesting style transfer . Hence we utilize the framework, as shown in , we also adopted the linear combination of both content loss and style loss,. L c indicates the content loss which is the euclidean distance between the content vector V c and stylized feature vector V new, where A i is the activation vector in i-th layer of the FVT. The style loss L s is obtained from the Gram matrix of the feature vectors. In Eq. equation 4, G represents the gram matrix. Inspired by Li et al.Huang & Belongie (2017, we also utilize the the mean value and standard deviation of feature vectors to calculate the style loss, the of which is similar to the gram loss L s . In Fig. 14, FVT (iterative optimization) only contains the convolutional layer from Conv2 to Conv5 and rrVGG contains Conv1 and DeConv1. In addition, we can also utilize more layers on rrVGG and FVT will contain less layers on the optimization network correspondingly, which will further speed up the style transfer process. In experiments, our framework is faster than the original optimization based approach and can transfer the arbitrary styles. As for the stylization effectiveness, we compared our with and Ulyanov et al. Ulyanov et al. (2017b). In Fig. 15, rrV GG 1 and rrV GG 2 columns denote the stylization acquired from our framework, applying two different rrVGG models. As shown in Fig. 15, our stylization are competitive to other well-trained approaches. Focused on rrV GG 1 column, our stylized is inclined to extract more features from the style image and slightly weaken the representation of content image. Since we utilize rrVGG CNN and DCN to complete the transformation between feature space and image space, some content information is possible to be lost during the reconstruction process. Despite that, our approach is still capable of generating high quality stylized images. In addition, we also investigate the stylized effectiveness when modifying the balance between style and content in FVT. As shown in Figure 16, the number below each column indicates the relative weightings between style and content reconstruction. In our framework, the transition from content to style is smooth with increasing ratio. As shown in Figure 17, our stylized is inclined to extract more features from the style image and slightly weaken the representation of the content image. content style rrVGG 1 rrVGG 2 rrVGG 3 Figure 17: Style transfer from several rrVGG models. Each model has the same architecture but different random weights. As proposed in our paper, in terms of different distributions and number of filters, rrVGG can reconstruct images with diverse textures and colors. In Fig. 14, replacing CNN and DCN parts with different rrVGG models, our framework can generate abundant stylized images depending on a single style image. Since rrVGG models are generated without training, it won't incur additional computational cost. As shown in Fig. 17, the rightmost three columns comprise the stylized images with different rrVGG model weights while the leftmost two columns represent input content and style images respectively. For each row, given content and style images, we choose three stylized images generated by our framework using different rrVGG models. For instance, in the 3-rd row of Fig. 17, the parameters of chosen rrVGG models are as following: rrVGG 1:(N (0, 0.01), filter size:3, filter num:128), rrVGG 2:(N (0, 0.01), filter size:5, filter num:256) and rrVGG 3:(N (0, 0.1), filter size:3, filter num:32). As shown in Fig. 17, those stylized images not only well preserve the style structure such as the shape of the curved lines, waves and abstract objects, but also exhibit novel combinations of the structure, shade and hue. Coupled with various rrVGG models, the proposed style transfer framework is able to unleash the diversity and variation inside a single style image, which works well in practice. Meanwhile, it's flexible as well as fast, since the FVT part can be implemented either by an optimization process or some feed-forward convolutional layers.

We investigate the deep representation of untrained, random weight CNN-DCN architectures, and show their image reconstruction quality and possible applications.


Question: Represent the article for finding a single-sentence TLDR

Question: The current trade-off between depth and computational cost makes it difficult to adopt deep neural networks for many industrial applications, especially when computing power is limited. Here, we are inspired by the idea that, while deeper embeddings are needed to discriminate difficult samples, a large number of samples can be well discriminated via much shallower embeddings. In this study, we introduce the concept of decision gates (d-gate), modules trained to decide whether a sample needs to be projected into a deeper embedding or if an early prediction can be made at the d-gate, thus enabling the computation of dynamic representations at different depths. The proposed d-gate modules can be integrated with any deep neural network and reduces the average computational cost of the deep neural networks while maintaining modeling accuracy. Experimental show that leveraging the proposed d-gate modules led to a ~38% speed-up and ~39% FLOPS reduction on ResNet-101 and ~46% speed-up and $\sim$36\% FLOPS reduction on DenseNet-201 trained on the CIFAR10 dataset with only ~2% drop in accuracy. Past studies such as BID15 have shown that deeper architectures often lead to better modeling performance; however, deeper architectures also pose a number of issues. Besides becoming more prone to overfitting and becoming more difficult to train, the trade-off between depth and computational cost makes it difficult to adopt deeper architectures for many industrial applications. He et al. BID6 tackled the former issue of degradation in learning deeper neural networks (e.g., vanishing gradient) by introducing the concept of residual learning, where learning is based on the residual mapping rather than directly on the unreferenced mapping. Following that, Xie et al. BID18 took advantage of the inception idea (i.e, split-transform-merge strategy) within a residual block structure to provide better subspace modeling while resolving the degradation problem at the same time, ing in a ResNext architecture with improved modeling accuracy. To tackle the issue of computational cost, a wide variety of methods have been proposed, including: precision reduction BID9, model compression BID5, teacher-student strategies BID7, and evolutionary algorithms BID12 BID13.More recently, conditional computation BID0 BID3 BID11 BID17 BID1 and early prediction BID16 methods have been proposed to tackle this issue, which involve the dynamic execution of different modules within a network. Conditional computation methods have largely been motivated by the idea that residual networks can be considered as an ensemble of shallower networks. As such, these methods take advantage of skip connections to determine which residual modules are necessary to be executed, with most leveraging reinforcement learning. In this study, we explore the idea of early prediction but instead draw inspiration from the soft-margin support vector BID2 theory for decision-making. Specifically, we introduce the concept of decision gates (d-gate), modules trained to decide whether a sample needs to be projected into a deeper embedding or if an early prediction can be made at the d-gate, thus enabling the conditional computation of dynamic representations at different depths. The proposed d-gate modules can be integrated with any deep neural network without the need to train networks from scratch, and thus reduces the average computational complexity of the deep neural networks while maintaining modeling accuracy. Deeper neural network architectures have been demonstrated to provide a better subspace embedding of data when compared to shallower architectures, ing in better discrimination of data space and better modeling accuracy. Inspired by soft-margin support vector BID2 theory, we make a hypothesis that while deeper subspace embeddings are necessary to discriminate samples that lie close to the decision boundaries in the lower embedding space, their effect on samples that already lie far from decision boundaries in the shallower embedding space may be insignificant and unnecessary. Therefore, an effective yet efficient mechanism for determining the distance between samples and the decision boundaries in the lower layers of the network would make it possible to perform early predictions on these samples without projecting them into a deeper embedding space. This approach would reduce the average computational cost of prediction significantly. However, devising an efficient way to determine whether a sample is a boundary sample is a challenging problem. Here, we formulate the early prediction problem as a risk minimization problem, and introduce a set of single-layer feed-forward networks (which we will refer to as decision gates (d-gate)) that are integrated directly into a deep neural network (see FIG0). The goal of d-gate modules is to not only decide whether a sample requires projection into a deep embedding space, but also minimize the risk of early wrong classifications as well. Specifically, we train d-gate modules that are integrated into a deep neural network via a hinge loss BID4 that minimizes the risk of early misclassification in lower embeddings while deciding whether the sample is a boundary sample: DISPLAYFORM0 where y is the ground truth label for the input data x andŷ is the predicted class label via the d-gate module with the set of weights w and biases b. The set of weights w has a dimensionality of f × c, where f denotes the number of input features to the d-gate module and c denotes the number of class labels in the classification task. This d-gate module provides an important benefit where the of w T x − b provides the distances of the sample to the corresponding decision boundary of each class label in the embedding space. Training the d-gate module in this way provides a linear classifier where samples that do not require deeper embeddings for discrimination are those with larger distances (i.e., with the positive sign) from the decision boundary. It is important to note that single layer nature of d-gate modules is designed to account for efficiency. The d-gate module is trained via the training data utilized to train the deep neural network and the objective for each d-gate module is to minimize the classification error on the training data. Therefore, the loss function on the training data can be formulated as: DISPLAYFORM1 where Y denotes the set of ground truth labels for all training data. What is most interesting about L(Y,Ŷ ; w, b) is the fact that L(·) is a convex function of w and b, and as such can be optimized via gradient descent. Therefore, traditional gradient descent can be adapted here, where a step is taken in the direction of a vector selected from the function's sub-gradient BID14 to find the optimized values. As a , the d-gate can be trained within a mini-batch training framework, which makes it is very convenient for utilizing in the training of deep neural networks with large datasets. In essence, the proposed d-gate module can compute the distance of each sample to the decision boundaries based on w T x − b; the calculated distances are compared with the decision threshold t of each d-gate to determine whether early prediction on the sample can be performed at the d-gate or have the sample moved to a deeper stage of the deep neural network to project into a better embedding space for improved prediction. The samples that are far from the decision boundaries to output larger values in w T x − b; therefore, if the d-gate distance for a sample satisfies the d-gate decision threshold, the class corresponding to the largest distance is assigned as the predicted class label for the sample in this early prediction step. formance of the network with decision gates trained via the proposed hinge loss is compared with decision gates trained via a conventional cross-entropy approach. It can be observed that the decision gates trained via the hinge loss provide greater computational efficiency with higher accuracy than if cross-entropy loss is leveraged. The efficacy of the proposed d-gate modules is examined with two different network architectures (ResNet101 BID6 and DenseNet201 BID8) on the CIFAR10 dataset. A key benefit of the proposed d-gate modules is that it enables fine control over the trade-off between modeling accuracy and computational cost by adjusting the d-gate decision thresholds. By decreasing the d-gate decision thresholds, the number of samples undergoing early prediction increases, thus reducing the average computational cost of network predictions greatly. For this study, we integrated two d-gate modules in ResNet-101 (after the first and second main blocks) and DenseNet-201 (after the first and second dense blocks), and explore different d-gate configurations. The networks are implemented in the Pytorch framework and the prediction speeds are reported based on single Nvidia Titan Xp GPU.It can be observed from TAB0 that the computational cost of ResNet network can be reduced by 67 MFLOPS while maintaining the same level of accuracy as to the original ResNet-101 by integrating two d-gate modules with decision thresholds of (t1, t2) = (2.5, 2.5). The integration of d-gate modules can reduce the computational cost of ResNet-101 network by ∼39% (i.e., lower by 1.95 GFLOPS) with 1.7% drop in accuracy compared to the original ResNet-101 (with distance thresholds (t1, t2) = (1.0, 2.0) in d-gate1 and d-gate2), ing in a ∼38% speed-up. The experiments for DenseNet-201 show that it is possible to reduce the number of FLOPs by 970 MFLOPs (∼36% reduction) with only a ∼2% drop in accuracy, leading to a ∼46% speed-up. Furthermore, a 2.3× speed-up can be achieved with d-gate modules compared to the original DenseNet-201 within a 3% accuracy margin. Based on the experimental , the proposed d-gate modules lead to a significant increase in prediction speed, making it well-suited for industrial applications. In addition to the d-gate modules being proposed, one of the key contributions of this paper is the introduction of a hinge loss for training the d-gate modules. Past studies BID10 have argued that crossentropy in a small margin between the decision boundaries and the training data. As such, it is very difficult to trust the confidence values of the Softmax layer to decide about the sample since there is no valuable information in the Softmax output. To demonstrate the effectiveness of the hinge loss leveraged in the proposed d-gates compared to the cross-entropy loss, an additional comparative experiment was conducted. More specifically, two decision gates were added to ResNet101 in the same way as reported. However, rather than train using the proposed hinge loss, the decision gates were instead trained via a cross-entropy loss. This enables us to compare the effect of hinge loss vs. cross-entropy loss on decision gate functionality. FIG1 demonstrates the accuracy vs. number of FLOPs for the network where the decision gates were trained based on the proposed hinge loss approach compared to trained using a regular cross-entropy training procedure. It can be observed that, with the same number of FLOPs in the network, the network where the decision gates were trained based on the proposed hinge loss provides much higher modeling accuracy compared to that trained via cross-entropy loss. The accuracy gap increases exponentially when the decision gates are configured such that the network uses fewer number of FLOPs. What this illustrates is the aforementioned issue with the use of cross-entropy loss and decision boundaries.

Represent the short summary for finding the corresponding scientific article (long)

This paper introduces a new dynamic feature representation approach to provide a more efficient way to do inference on deep neural networks.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: This work provides an additional step in the theoretical understanding of neural networks. We consider neural networks with one hidden layer and show that when learning symmetric functions, one can choose initial conditions so that standard SGD training efficiently produces generalization guarantees. We empirically verify this and show that this does not hold when the initial conditions are chosen at random. The proof of convergence investigates the interaction between the two layers of the network. Our highlight the importance of using symmetry in the design of neural networks. Building a theory that can help to understand neural networks and guide their construction is one of the current challenges of machine learning. Here we wish to shed some light on the role symmetry plays in the construction of neural networks. It is well-known that symmetry can be used to enhance the performance of neural networks. For example, convolutional neural networks (CNNs) (see) use the translational symmetry of images to classify images better than fully connected neural networks. Our focus is on the role of symmetry in the initialization stage. We show that symmetry-based initialization can be the difference between failure and success. On a high-level, the study of neural networks can be partitioned to three different aspects. Expressiveness Given an architecture, what are the functions it can approximate well? Training Given a network with a "proper" architecture, can the network fit the training data and in a reasonable time? Generalization Given that the training seemed successful, will the true error be small as well? We study these aspects for the first "non trivial" case of neural networks, networks with one hidden layer. We are mostly interested in the initialization phase. If we take a network with the appropriate architecture, we can always initialize it to the desired function. A standard method (that induces a non trivial learning problem) is using random weights to initialize the network. A different reasonable choice is to require the initialization to be useful for an entire class of functions. We follow the latter option. Our focus is on the role of symmetry. We consider the following class of symmetric functions S = S n = n ∑ i=0 a i · 1 |x|=i: a 1,..., a n ∈ {±1}, where x ∈ {0, 1} n and |x| = ∑ i x i. The functions in this class are invariant under arbitrary permutations of the input's coordinates. The parity function π(x) = (−1) |x| and the majority function are well-known examples of symmetric functions. Expressiveness for this class was explored by. They showed that the parity function cannot be represented using a network with limited "connectivity". Contrastingly, if we use a fully connected network with one hidden layer and a common activation function (like sign, sigmoid, or ReLU) only O(n) neurons are needed. We provide such explicit representations for all functions in S; see Lemmas 1 and 2. We also provide useful information on both the training phase and generalization capabilities of the neural network. We show that, with proper initialization, the training process (using standard SGD) efficiently converges to zero empirical error, and that consequently the network has small true error as well. Theorem 1. There exists a constant c > 1 so that the following holds. There exists a network with one hidden layer, cn neurons with sigmoid or ReLU activations, and an initialization such that for all distributions D over X = {0, 1} n and all functions f ∈ S with sample size m ≥ c(n+log(1/δ))/ε, after performing poly(n) SGD updates with a fixed step size h = 1/poly(n) it holds that is the network after training over S. The number of parameters in the network described in Theorem 1 is Ω(n 2). So in general one could expect overfitting when the sample size is as small as O(n). Nevertheless, the theorem provides generalization guarantees, even for such a small sample size. The initialization phase plays an important role in proving Theorem 1. To emphasize this, we report an empirical phenomenon (this is "folklore"). We show that a network cannot learn parity from a random initialization (see Section 5.3). On one hand, if the network size is big, we can bring the empirical error to zero (as suggested in), but the true error is close to 1/2. On the other hand, if its size is too small, the network is not even able to achieve small empirical error (see Figure 5). We observe a similar phenomenon also for a random symmetric function. An open question remains: why is it true that a sample of size polynomial in n does not suffice to learn parity (with random initialization)? A similar phenomenon was theoretically explained by and. The parity function belongs to the class of all parities where · is the standard inner product. This class is efficiently PAC-learnable with O(n) samples using Gaussian elimination. A continuous version of P was studied by and. To study the training phase, they used a generalized notion of statistical queries (SQ); see. In this framework, they show that most functions in the class P cannot be efficiently learned (roughly stated, learning the class requires an exponential amount of resources). This framework, however, does not seem to capture actual training of neural networks using SGD. For example, it is not clear if one SGD update corresponds to a single query in this model. In addition, typically one receives a dataset and performs the training by going over it many times, whereas the query model estimates the gradient using a fresh batch of samples in each iteration. The query model also assumes the noise to be adversarial, an assumption that does not necessarily hold in reality. Finally, the SQ-based lower bound holds for every initialization (in particular, for the initialization we use here), so it does not capture the efficient training process Theorem 1 describes. Theorem 1 shows, however, that with symmetry-based initialization, parity can be efficiently learned. So, in a nutshell, parity can not be learned as part of P, but it can be learned as part of S. One could wonder why the hardness proof for P cannot be applied for S as both classes consist of many input sensitive functions. The answer lies in the fact that P has a far bigger statistical dimension than S (all functions in P are orthogonal to each other, unlike S). The proof of the theorem utilizes the different behavior of the two layers in the network. SGD is performed using a step size h that is polynomially small in n. The analysis shows that in a polynomial number of steps that is independent of the choice of h the following two properties hold: (i) the output neuron reaches a "good" state and (ii) the hidden layer does not change in a "meaningful" way. These two properties hold when h is small enough. In Section 5.2, we experiment with large values of h. We see that, although the training error is zero, the true error becomes large. Here is a high level description of the proof. The neurons in the hidden layer define an "embedding" of the inputs space X = {0, 1} n into R (a.k.a. the feature map). This embedding changes in time according to the training examples and process. The proof shows that if at any point in time this embedding has good enough margin, then training with standard SGD quickly converges. This is explained in more detail in Section 3. It remains an interesting open problem to understand this phenomenon in greater generality, using a cleaner and more abstract language. To better understand the context of our research, we survey previous related works. The expressiveness and limitations of neural networks were studied in several works such as;; and. Constructions of small ReLU networks for the parity function appeared in several previous works, such as , , and. Constant depth circuits for the parity function were also studied in the context of computational complexity theory, see for example , and Håstad. The training phase of neural networks was also studied in many works. Here we list several works that seem most related to ours. analyzed SGD for general neural network architecture and showed that the training error can be nullified, e.g., for the class of bounded degree polynomials (see also). studied neural tangent kernels (NTK), an infinite width analogue of neural networks. showed that randomly initialized shallow ReLU networks nullify the training error, as long as the number of samples is smaller than the number of neurons in the hidden layer. Their analysis only deals with optimization over the first layer (so that the weights of the output neuron are fixed). provided another analysis of the latter two works. Allen-Zhu et al. (2018b) showed that over-parametrized neural networks can achieve zero training error, as as long as the data points are not too close to one another and the weights of the output neuron are fixed. provided guarantees for zero training error, assuming the two classes are separated by a positive margin. Convergence and generalization guarantees for neural networks were studied in the following works. gave data-dependent generalization bounds for GD. All these works optimized only over the hidden layer (the output layer is fixed after initialization). Margins play an important role in learning, and we also use it in our proof. , , and gave generalization bounds for neural networks that are based on their margin when the training ends. From a practical perspective, , and suggested different training algorithms that optimize the margin. As discussed above, it seems difficult for neural networks to learn parities. and demonstrated this using the language statistical queries (SQ). This is a valuable language, but it misses some central aspects of training neural networks. SQ seems to be closely related to GD, but does not seem to capture SGD. SQ also shows that many of the parities functions ⊗ i∈S x i are difficult to learn, but it does not imply that the parity function ⊗ i∈[n] x i is difficult to learn. demonstrated a similar phenomenon in a setting that is closer to the "real life" mechanics of neural networks. We suggest that taking the symmetries of the learning problem into account can make the difference between failure and success. Several works suggested different neural architectures that take symmetries into account; see , , and. Here we describe efficient representations for symmetric functions by networks with one hidden layer. These representations are also useful later on, when we study the training process. We study two different activation functions, sigmoid and ReLU (similar statement can be proved for other activations, like arctan). Each activation function requires its own representation, as in the two lemmas below. We start with the activation σ (ξ) =, since it helps to understand the construction for the ReLU activation. The building blocks of the symmetric functions are indicators of |x| = i for i ∈ {0, 1, . . ., n}. An indicator function is essentially the difference between two sigmoid functions: where A network with one hidden layer of n + 2 neurons with sigmoid activations and one bias neuron is sufficient to represent any function in S. The coefficients of the sigmoid gates are 0, ±1 in this representation. The proofs of this lemma and the subsequent lemmas appear in the appendix. A sigmoid function can be represented using ReLU(ξ) = max{0, ξ} as the difference between two ReLUs Hence, an indicator function can be represented using sign(1 |x|=i − 0.5) = sign(Γ i − 0.5) where The lemma shows that a network with one hidden layer of n + 3 ReLU neurons and one bias neuron is sufficient to represent any function in S. The coefficients of the ReLU gates are 0, ±1, ±2 in this representation. The goal of this section is to describe a small network with one hidden layer that (when initialized properly) efficiently learns symmetric functions using a small number of examples (the training is done via SGD). Here we specify the architecture, initialization and loss function that is implicit in our main (Theorem 1). To guarantee convergence of SGD, we need to start with "good" initial conditions. The initialization we pick depends on the activation function it uses, and is chosen with resemblance to Lemma 2 for ReLU. On a high level, this indicates that understanding the class of functions we wish to study in term of "representation" can be helpful when choosing the architecture of a neural network in a learning context. The network we consider has one hidden layer. We denote by w i j the weight between coordinate j of the input and neuron i in the hidden layer. We denote W this matrix of weights. We denote by b i the bias of neuron i of the hidden layer. We denote B this vector of weights. We denote by m i is the weight from neuron i in the hidden layer to the output neuron. We denote M this vector of weights. We denote by b the bias of the output neuron. Initialize the network as follows: The dimensions of W are (n + 3) × n. For all 1 ≤ i ≤ (n + 3) and 1 ≤ j ≤ n, we set w i j = 1 and b i = −i + 2. We set M = 0 and b = 0. To run SGD, we need to choose a loss function. We use the hinge loss, where v x = ReLU(W x + B) is the output of the hidden layer on input x and β > 0 is a parameter of confidence. A key property in the analysis is the'margin' of the hidden layer with respect to the function being learned. We are interested in the following set V in R d. Recall that W is the weight matrix between the input layer and the hidden layer, and that B is the relevant bias vector. Given W, B, we are interested in the set V = {v x : x ∈ X}, where v x = ReLU(W x + B). In words, we think of the neurons in the hidden layer as defining an "embedding" of X in Euclidean space. A similar construction works for other activation functions. We say that Y: V → {±1} agrees with f ∈ S if for all x ∈ X it holds that The following lemma bounds from below the margin of the initial V. Lemma 3. If Y is a partition that agrees with some function in S for the initialization described above then marg(Y) ≥ Ω(1/ √ n). Proof. By Lemmas 1 and 2, we see that any function in S can be represented with a vector of weights M, b ∈ [−1, 1] Θ(n) of the output neuron together with a bias. These M, b induce a partition we have our desired . Before analyzing the full behavior of SGD, we make an observation: if the weights of the hidden layer are fixed with the initialization described above, then Theorem 1 holds for SGD with batch size 1. This observation, unfortunately, does not suffice to prove Theorem 1. In the setting we consider, the training of the neural network uses SGD without fixing any weights. This more general case is handled in the next section. The rest of this subsection is devoted for explaining this observation. showed that that the perceptron algorithm makes a small number of mistakes for linearly separable data with large margin. For a comprehensive survey of the perceptron algorithm and its variants, see. Running SGD with the hinge loss induces the same update rule as in a modified perceptron algorithm, Algorithm 1. Initialize: Novikoff's proof can be generalized to any β > 0 and batches of any size to yield the following theorem; see; and appendix A. Theorem 2. For Y: V → {±1} with margin γ > 0 and step size h > 0, the modified perceptron algorithm performs at most updates and achieves a margin of at least γβ h 2β h+(Rh) 2, where R = max v∈V v. So, when the weights of the hidden layer are fixed, Lemma 3 implies that the number of SGD steps is at most polynomial in n. When we run SGD on the entire network, the layers interact. For a ReLU network at time t, the update rule for W is as follows. If the network classifies the input x correctly with confidence more than β, no change is made. Otherwise, we change the weights in M by ∆M = yv x h, where y is the true label and h is the step size. If also neuron i of the hidden fired on x, we update its incoming weights by ∆W i,: = ym i xh. These update rules define the following dynamical system: (a) where H is the Heaviside step function and • is the Hadamard pointwise product. A key observation in the proof is that the weights of the last layer ( and) are updated exactly as the modified perceptron algorithm. Another key statement in the proof is that if the network has reached a good representation of the input (i.e., the hidden layer has a large margin), then the interaction between the layers during the continued training does not impair this representation. This is summarized in the following lemma (we are not aware of a similar statement in the literature). Lemma 4. Let M = 0, b = 0, and V = {ReLU(W x + B): x ∈ X} be a linearly separable embedding of X and with margin γ > 0 by the hidden layer of a neural network of depth two with ReLU activation and weights given by W, B. Let R X = max x∈X x, let R = max v∈V v, and 0 < h ≤ γ 5/2 100R 2 R X be the integration step. Assuming R X > 1 and γ ≤ 1, and using β = R 2 h in the loss function, after t SGD iterations the following hold: -Each v ∈ V moves a distance of at most O(R 2 X h 2 Rt 3/2). -The norm M (t) is at most O(Rh √ t). -The training ends in at most O(R 2 /γ 2) SGD updates. Intuitively, this type of lemma can be useful in many other contexts. The high level idea is to identify a "good geometric structure" that the network reaches and enables efficient learning. Proof of Theorem 1. There is an unknown distribution D over the space X. We pick i.i.d. examples S = ((x 1, y 1),..., (x m, y m)) where m ≥ c n+log(1/δ) ε according to D, where y i = f (x i) for some f ∈ S. Run SGD for O(n 4) steps, where the step size is h = O(1/n 6) and the parameter of the loss function is β = R 2 h with R = n 3/2. We claim that it suffices to show that at the end of the training (i) the network correctly classifies all the sample points x 1,..., x m, and (ii) for every x ∈ X such that there exists 1 ≤ i ≤ m with |x| = |x i |, the network outputs y i on x as well. Here is why. The initialization of the network embeds the space X into n + 4 dimensional space (including the bias neuron of the hidden layer). Let V be the initial embedding V = {ReLU(W x + B ): x ∈ X}. Although |X| = 2 n, the size of V is n + 1. The VC dimension of all the boolean functions over V is n + 1. Now, m samples suffice to yield ε true error for an ERM when the VC dimension is n + 1; see e.g. Theorem 6.7 in. It remains to prove (i) and (ii) above. By Lemma 3, at the beginning of the training, the partition of V defined by the target f ∈ S has a margin of γ = Ω(1/ √ n). We are interested in the eventual V * = {ReLU(W * x + B *): x ∈ X} embedding of X as well. The modified perceptron algorithm together with Lemma 4 guarantees that after K ≤ 20R 2 /γ 2 = O(n 4) updates, (M *, b *) separates the embedded sample V * S = {ReLU(W * x i + B *): 1 ≤ i ≤ m} with a margin of at least 0.9γ/3. It remains to prove (ii). Lemma 4 states that as long as less than K = O(n 4) updates were made, the elements in V moved at most O(1/n 2). At the end of the training, the embedded sample V S is separated with a margin of at least 0.9γ/3 with respect to the hyperplane defined by M * and B *. Each v * x for x ∈ X moved at most O(1/n 2) < γ/4. This means that if |x| = |x i | then the network has the same output on x and x i. Since the network has zero empirical error, the output on this x is y i as well. A similar proof is available with sigmoid activation (with better convergence rate and larger allowed step size). Remark. The generalization part of the above proof can be viewed as a consequence of sample compression . Although the eventual network depends on all examples, the proof shows that its functionality depends on at most n + 1 examples. Indeed, after the training, all examples with equal hamming weight have the same label. Remark. The parameter β = R 2 h we chose in the proof may seem odd and negligible. It is a construct in the proof that allows us to bound efficiently the distance that the elements in V have moved during the training. For all practical purposes β = 0 works as well (see Figure 4). We accompany the theoretical with some experiments. We used a network with one hidden layer of 4n + 3 neurons, ReLU activation, and the hinge loss with β = n 3 h. In all the experiments, we used SGD with mini-batch of size one and before each epoch we randomized the sample. We observed similar behavior for larger mini-batches, other activation functions, and other loss functions. The graphs that appear in the appendix A present the training error and the true error 2 versus the epoch of the training process. In all the comparisons below, we chose a random symmetric function and a random sample from X. Figure 2 demonstrates our theoretical and also validates the performance of our initialization. In one setting, we trained only the second layer (freezed the weights of the hidden layer) which essentially corresponds to the perceptron algorithm. In the second setting, we trained both layers with a step size h = n −6 (as the theory suggests). As expected, performance in both cases is similar. We remark that SGD continues to run even after minimizing the empirical error. This happens because of the parameter β > 0. Here we experiment with two parameters in the proof, the step size h and the confidence parameter β. In Figure 3, we used three different step sizes, two of which much larger than the theory suggests. We see that the training error converges much faster to zero, when the step size is larger. This fast convergence comes at the expense of the true error. For a large step size, generalization cease to hold. Setting β = n 3 h is a construct in the proof. Figure 4 shows that setting β = 0 does not impair the performance. The difference between theory (requires β > 0) and practice (allows β = 0) can be explained as follows. The proof bounds the worst-case movement of the hidden layer, whereas in practice an average-case argument suffices. Figure 5 shows that even for n = 20, learning parity is hard from a random initialization. When the sample size is small the training error can be nullified but the true error is large. As the sample grows, it becomes much harder for the network to nullify even the training error. With our initialization, both the training error and true error are minimized quickly. Figure 6 demonstrates the same phenomenon for a random symmetric function. Our initialization also delivers satisfying when the input data it corrupted. In figure 7, we randomly perturb (with probability p = 1 10) the labels and use the same SGD to train the model. In figure 8, we randomly shift every entry of the vectors in the space X by ε that is uniformly distributed in [−0.1, 0.1] n. This work demonstrates that symmetries can play a critical role when designing a neural network. We proved that any symmetric function can be learned by a shallow neural network, with proper initialization. We demonstrated by simulations that this neural network is stable under corruption of data, and that the small step size is the proof is necessary. We also demonstrated that the parity function or a random symmetric function cannot be learned with random initialization. How to explain this empirical phenomenon is still an open question. The works and treated parities using the language of SQ. This language obscures the inner mechanism of the network training, so a more concrete explanation is currently missing. We proved in a special case that the standard SGD training of a network efficiently produces low true error. The general problem that remains is proving similar for general neural networks. A suggestion for future works is to try to identify favorable geometric states of the network that guarantee fast convergence and generalization. Proof. For all k ∈ A and x ∈ X of weight k, the first inequality holds since ∆ i (x) ≥ 0 for all i and x. For all k ∈ A and x ∈ X of weight k, = 2 exp(−2.5)/(1 − exp(−5)) < 0.17; the first equality follows from the definition, the second equality follows from σ (5(x + 0.5)) − σ (5(x − 0.5)) = σ (5(x + 0.5)) + σ (5(−x + 0.5)) − 1 for all x, the first inequality neglects the negative sums, and the second inequality follows because exp(ξ) > σ (ξ) for all ξ. Proof. The proof follows from two observations: For all i ∈ A and x of weight i it holds Γ i (x) = 1. Proof. We are interested in the maximal distance the embedding of an element x ∈ X has moved from its initial embedding: To simplify equations- discussed above, we assume that during the optimization process the norm of the weights W and B grow at a maximal rate: here the norm of a matrix is the 2 -norm. To bound these quantities, we follow the modified perceptron proof and add another quantity to bound. That is, the maximal norm R (t) of the embedded space X at time t satisfies (by assumption R X > 1) we used that the spectral norm of a matrix is at most its 2 -norm. We assume a worst-case where R (t) grows monotonically at a maximal rate. By the modified perceptron algorithm and choice β = R 2 h, Sample of size 10n whose input dimension is n = 30.

represent the tldr for finding the corresponding scientific article

When initialized properly, neural networks can learn the simple class of symmetric functions; when initialized randomly, they fail.

Represent the short summary for finding the corresponding scientific article (long)

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.