Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Motivated by applications to unsupervised learning, we consider the problem of measuring mutual information. Recent analysis has shown that naive kNN estimators of mutual information have serious statistical limitations motivating more refined methods. In this paper we prove that serious statistical limitations are inherent to any measurement method. More specifically, we show that any distribution-free high-confidence lower bound on mutual information cannot be larger than $O(\ln N)$ where $N$ is the size of the data sample. We also analyze the Donsker-Varadhan lower bound on KL divergence in particular and show that, when simple statistical considerations are taken into account, this bound can never produce a high-confidence value larger than $\ln N$. While large high-confidence lower bounds are impossible, in practice one can use estimators without formal guarantees. We suggest expressing mutual information as a difference of entropies and using cross entropy as an entropy estimator. We observe that, although cross entropy is only an upper bound on entropy, cross-entropy estimates converge to the true cross entropy at the rate of $1/\sqrt{N}$. Motivated by maximal mutual information (MMI) predictive coding BID11 BID16 BID13, we consider the problem of measuring mutual information. A classical approach to this problem is based on estimating entropies by computing the average log of the distance to the kth nearest neighbor in a sample BID7. It has recently been shown that the classical kNN methods have serious statistical limitations and more refined kNN methods have been proposed BID5. Here we establish serious statistical limitations on any method of estimating mutual information. More specifically, we show that any distribution-free high-confidence lower bound on mutual information cannot be larger than O(ln N) where N is the size of the data sample. Prior to proving the general case, we consider the particular case of the Donsker-Varadhan lower bound on KL divergence BID3 BID1. We observe that when simple statistical considerations are taken into account, this bound can never produce a highconfidence value larger than ln N. Similar comments apply to lower bounds based on contrastive estimation. The contrastive estimation lower bound given in BID13 does not establish mutual information of more than ln k where k is number of negative samples used in the contrastive choice. The difficulties arise in cases where the mutual information I(x, y) is large. Since I(x, y) = H(y) − H(y|x) we are interested in cases where H(y) is large and H(y|x) is small. For example consider the mutual information between an English sentence and its French translation. Sampling English and French independently will (almost) never yield two sentences where one is a plausible translation of the other. In this case the DV bound is meaningless and contrastive estimation is trivial. In this example we need a language model for estimating H(y) and a translation model for estimating H(y|x). Language models and translation models are both typically trained with crossentropy loss. Cross-entropy loss can be used as an (upper bound) estimate of entropy and we get an estimate of mutual information as a difference of cross-entropy estimates. Note that the upper-bound guarantee for the cross-entropy estimator yields neither an upper bound nor a lower bound guarantee for a difference of entropies. Similar observations apply to measuring the mutual information for pairs of nearby frames of video or pairs of sound waves for utterances of the same sentence. We are motivated by the problem of maximum mutual information predictive coding BID11 BID16 BID13. One can formally define a version of MMI predictive coding by considering a population distribution on pairs (x, y) where we think of x as past raw sensory signals (images or sound waves) and y as a future sensory signal. We consider the problem of learning stochastic coding functions C x and C y so as to maximize the mutual information I(C x (x), C y (y)) while limiting the entropies H(C x (x)) and H(C y (y)). The intuition is that we want to learn representations C x (x) and C y (y) that preserve "signal" while removing "noise". Here signal is simply defined to be a low entropy representation that preserves mutual information with the future. Forms of MMI predictive coding have been independently introduced in BID11 under the name "information-theoretic cotraining" and in BID13 under the name "contrastive predictive coding". It is also possible to interpret the local version of DIM (DIM(L)) as a variant of MMI predictive coding. A closely related framework is the information bottleneck BID17. Here one again assumes a population distribution on pairs (x, y). The objective is to learn a stochastic coding function C x so as to maximize I(C x (x), y) while minimizing I(C x (x), x). Here one does not ask for a coding function on y and one does not limit H(C x (x)).Another related framework is INFOMAX BID8 BID2. Here we consider a population distribution on a single random variable x. The objective is to learn a stochastic coding function C x so as to maximize the mutual information I(x, C x (x)) subject to some constraint or additional objective. As mentioned above, in cases where I(C x (x), C y (y)) is large it seems best to train a model of the marginal distribution of P (C y) and a model of the conditional distribution P (C y |C x) where both models are trained with cross-entropy loss. Section 5 gives various high confidence upper bounds on cross-entropy loss for learned models. The main point is that, unlike lower bounds on entropy, high-confidence upper bounds on cross-entropy loss can be guaranteed to be close to the true cross entropy. Out theoretical analyses will assume discrete distributions. However, there is no loss of generality in this assumption. Rigorous treatments of probability (measure theory) treat integrals (either Riemann or Lebesgue) as limits of increasingly fine binnings. A continuous density can always be viewed as a limit of discrete distributions. Although our proofs are given for discrete case, all our formal limitations on the measurement of mutual information apply to continuous case as well. See BID9 for a discussion of continuous information theory. Additional comments on this point are given in section 4. Mutual information can be written as a KL divergence. DISPLAYFORM0 Here P X,Y is a joint distribution on the random variables X and Y and P X and P Y are the marginal distributions on X and Y respectively. The DV lower bound applies to KL-divergence generally. To derive the DV bound we start with the following observation for any distributions P, Q, and G on the same support. Our theoretical analyses will assume discrete distributions. DISPLAYFORM1 Note that achieves equality for G(z) = P (z) and hence we have DISPLAYFORM2 Here we can let G be a parameterized model such that G(z) can be computed directly. However, we are interested in KL(P X,Y, P X P Y) where our only access to the distribution P is through sampling. If we draw a pair (x, y) and ignore y we get a sample from P X. We can similarly sample from P Y. So we are interested in a KL-divergence KL(P, Q) where our only access to the distributions P and Q is through sampling. Note that we cannot evaluate by sampling from P because we have no way of computing Q(z). But through a change of variables we can convert this to an expression restricted to sampling from Q. More specifically we define G(z) in terms of an unconstrained function F (z) as DISPLAYFORM3 Substituting FORMULA3 into FORMULA2 gives DISPLAYFORM4 Equation FORMULA4 is the Donsker-Varadhan lower bound. Applying this to mutual information we get DISPLAYFORM5 This is the equation underlying the MINE approach to maximizing mutual information BID1. It would seem that we can estimate both terms in through sampling and be able to maximize I(X, Y) by stochastic gradient ascent on this lower bound. In this section we show that the DV bound cannot be used to measure KL-divergences of more than tens of bits. In fact we will show that no high-confidence distribution-free lower bound on KL divergence can be used for this purpose. As a first observation note that involves E z∼Q e F (z). This expression has the same form as the moment generating function used in analyzing large deviation probabilities. The utility of expectations of exponentials in large deviation theory is that such expressions can be dominated by extremely rare events (large deviations). The rare events dominating the expectation will never be observed by sampling from Q. It should be noted that the optimal value for F (z) in is ln(P (z)/Q(z)) in which case the right hand side of simplifies to KL(P, Q). But for large KL divergence we will have that F (z) = ln(P (z)/Q(z)) is typically hundreds of bits and this is exactly the case where E z∼Q e F (z) cannot be measured by sampling from Q. If E z∼Q e F (z) is dominated by events that will never occur in sampling from Q then the optimization of F through the use of and sampling from Q cannot possibly lead to a function F (z) that accurately models the desired function ln(P (z)/Q(z)).To quantitatively analyze the risk of unseen outlier events we will make use of the following simple lemma where we write P z∼Q (Φ[z]) for the probability over drawing z from Q that the statement DISPLAYFORM0 Outlier Risk Lemma: For a sample S ∼ Q N with N ≥ 2, and a property Φ[z] such that P z∼Q (Φ[z]) ≤ 1/N, the probability over the draw of S that no z ∈ S satisfies Φ[z] is at least 1/4. Proof: The probability that Φ[z] is unseen in the sample is at least (1 − 1/N) N which is at least 1/4 for N ≥ 2 and where we have DISPLAYFORM1 We can use the outlier risk lemma to perform a quantitative risk analysis of the DV bound. We can rewrite as DISPLAYFORM2 We can try to estimate B(P, Q, G) from samples S P and S Q, each of size N, from the population distributions P and Q respectively. DISPLAYFORM3 While B(P, Q, F) is a lower bound on KL(P, Q), the sample estimateB(S P, S Q, F) is not. To get a high confidence lower bound on KL(P, Q) we have to handle unseen outlier risk. For a fair comparison with our analysis of cross-entropy estimators in section 5, we will limit the outlier risk by bounding F (z) to the interval [0, F max]. The largest possible value ofB(S P, S q, F) occurs when F (z) = F max for all z ∈ S P and F (z) = 0 for all z ∈ S Q. In this case we getB(S P, S Q, F) = F max. But by the outlier risk lemma there is still at least a 1/4 probability that DISPLAYFORM4 Any high confidence lower boundB(S P, S Q, F) must account for the unseen outlier risk. In particular we must haveB DISPLAYFORM5 Our negative can be strengthened by considering the preliminary bound where G(z) is viewed as a model of P (z). We can consider the extreme case of perfect modeling of the population P with a model G(z) where G(z) is computable. In this case we have essentially complete access to the distribution P. But even in this setting we have the following negative . Theorem 1 Let B be any distribution-free high-confidence lower bound on KL(P,Q) computed with complete knowledge of P but only a sample from Q.More specifically, let B(P, S, δ) be any real-valued function of a distribution P, a multiset S, and a confidence parameter δ such that, for any P, Q and δ, with probability at least (1 − δ) over a draw of S from Q N we have KL(P, Q) ≥ B(P, S, δ). For any such bound, and for N ≥ 2, with probability at least 1 − 4δ over the draw of S from Q N we have B(P, S, δ) ≤ ln N.Proof. Consider distributions P and Q and N ≥ 2. DefineQ bỹ DISPLAYFORM6 We now have KL(P,Q) ≤ ln N. We will prove that from a sample S ∼ Q N we cannot reliably distinguish between Q andQ.We first note that by applying the high-confidence guarantee of the bound toQ have DISPLAYFORM7 The distributionQ equals the marginal on z of a distribution on pairs (s, z) where s is the value of Bernoulli variable with bias 1/N such that if s = 1 then z is drawn from P and otherwise z is drawn from Q. By the outlier risk lemma the probability that all coins are zero is at least 1/4. Conditioned on all coins being zero the distributionsQ N and Q N are the same. Let Pure(S) represent the event that all coins are 0 and let Small(S) represent the event that B(P, S, δ) ≤ ln N. We now have DISPLAYFORM8 Mutual information is a special case of KL-divergence. It is possible that tighter lower bounds can be given in this special case. In this section we show similar limitations on lower bounding mutual information. We first note that a lower bound on mutual information implies a lower bound on entropy. The mutual information between X and Y cannot be larger than information content of X alone. So a lower bound on I(X, Y) gives a lower bound on H(X). We show that any distribution-free high-confidence lower bound on entropy requires a sample size exponential in the size of the bound. The above argument seems problematic for the case of continuous densities as differential entropy can be negative. However, for the continuous case we have DISPLAYFORM0 where C x and C y range over all maps from the underlying continuous space to discrete sets (all binnings of the continuous space). Hence an O(ln N) upper bound on the measurement of mutual information for the discrete case applies to the continuous case as well. The type of a sample S, denoted T (S), is defined to be a function on positive integers (counts) where T (S)(i) is the number of elements of S that occur i times in S. For a sample of N draws we have N = i iT (S)(i). The type T (S) contains all information relevant to estimating the actual probability of the items of a given count and of estimating the entropy of the underlying distribution. The problem of estimating distributions and entropies from sample types has been investigated by various authors BID12 BID15 BID14 BID0. Here we give the following negative on lower bounding the entropy of a distribution by sampling. Theorem 2 Let B be any distribution-free high-confidence lower bound on H(P) computed from a sample type T (S) with S ∼ P N.More specifically, let B(T, δ) be any real-valued function of a type T and a confidence parameter δ such that for any P, with probability at least (1 − δ) over a draw of S from P N, we have For any such bound, and for N ≥ 50 and k ≥ 2, with probability at least 1 − δ − 1.01/k over the draw of S from P N we have B(T (S), δ) ≤ ln 2kN 2.Proof: Consider a distribution P and N ≥ 100. If the support of P has fewer than 2kN 2 elements then H(P) < ln 2kN 2 and by the premise of the theorem we have that, with probability at least 1 − δ over the draw of S, B(T (S), δ) ≤ H(P) and the theorem follows. If the support of P has at least 2kN 2 elements then we sort the support of P into a (possibly infinite) sequence x 1, x 2, x 3,... so that P (x i) ≥ P (x i+1). We then define a distributionP on the elements x 1,..., x 2kN 2 bỹ DISPLAYFORM0 We will let Small(S) denote the event that B(T (S), δ) ≤ ln 2kN 2 and let Pure(S) abbreviate the event that no element x i for i > kN 2 occurs twice in the sample. SinceP has a support of size 2kN 2 we have H(P) ≤ ln 2kN 2. Applying the premise of the lemma toP gives DISPLAYFORM1 For a type T let P S∼P N (T) denote the probability over drawing S ∼ P N that T (S) = T. We now have DISPLAYFORM2 This gives the following. DISPLAYFORM3 For i > kN 2 we haveP (x i) ≤ 1/(kN 2) which gives DISPLAYFORM4 Using (1 − P) ≥ e −1.01 P for P ≤ 1/100 we have the following birthday paradox calculation. DISPLAYFORM5 Applying the union bound to FORMULA17 and FORMULA21 gives. DISPLAYFORM6 By a derivation similar to that of we get DISPLAYFORM7 Combining FORMULA19, FORMULA1 and FORMULA1 gives DISPLAYFORM8 Since mutual information can be expressed as a difference of entropies, the problem of measuring mutual information can be reduced to the problem of measuring entropies. In this section we show that, unlike high-confidence distribution-free lower bounds, high-confidence distribution-free upper bounds on entropy can approach the true cross entropy at modest sample sizes even when the true cross entropy is large. More specifically we consider the cross-entropy upper bound. DISPLAYFORM0 For G = P we get H(P, G) = H(P) and hence we have DISPLAYFORM1 In practice P is a population distribution and G is model of P. For example P might be a population distribution on paragraphs and G might be an autoregressive RNN language model. In practice G will be given by a network with parameters Φ. In this setting we have the following upper bound entropy estimator. Ĥ DISPLAYFORM2 The gap betweenĤ(P) and H(P) depends on the expressive power of the model class. The statistical limitations on distribution-free high-confidence lower bounds on entropy do not arise for cross-entropy upper bounds. For upper bounds we can show that naive sample estimates of the cross-entropy loss produce meaningful (large entropy) . We first define the cross-entropy estimator from a sample S.Ĥ DISPLAYFORM3 We can bound the loss of a model G by ensuring a minimum probability e −Fmax where F max is then the maximum possible log loss in the cross-entropy objective. In language modeling a loss bound exists for any model that ultimately backs off to a uniform distribution on characters. Given a loss bound of F max we have thatĤ(S, G) is just the standard sample mean estimator of an expectation of a bounded variable. In this case we have the following standard confidence interval. Theorem 3 For any population distribution P, and model distribution G with −ln G(x) bounded to the interval [0, F max], with probability at least 1 − δ over the draw of S ∼ P N we have DISPLAYFORM4 It is also possible to give PAC-Bayesian bounds on H(P, G Φ) that take into account the fact that G Φ is typically trained so as to minimize the empirical loss on the training data. The PAC-Bayesian bounds apply to"broad basin" losses and loss estimates such as the following. DISPLAYFORM0 Under mild smoothness conditions on G Φ (x) as a function of Φ we have DISPLAYFORM1 An L2 PAC-Bayesian generalization bound BID10 ) gives that for any parameterized class of models and any bounded notion of loss, and any λ > 1/2 and σ > 0, with probability at least 1 − δ over the draw of S from P N we have the following simultaneously for all parameter vectors Φ. DISPLAYFORM2 It is instructive to set λ = 5 in which case the bound becomes. DISPLAYFORM3 While this bound is linear in 1/N, and tighter in practice than square root bounds, note that there is a small residual gap when holding λ fixed at 5 while taking N → ∞. In practice the regularization parameter λ can be tuned on holdout data. One point worth noting is the form of the dependence of the regularization coefficient on F max, N and the basin parameter σ. It is also worth noting that the bound can be given in terms of "distance traveled" in parameter space from an initial (random) parameter setting Φ 0. DISPLAYFORM4 Evidence is presented in BID4 that the distance traveled bounds are tighter in practice than traditional L2 generalization bounds. Recall that in MMI predictive coding we assume a population distribution on pairs (x, y) where we think of x as past raw sensory signals (images or sound waves) and y as a future sensory signal. We then consider the problem of learning stochastic coding functions C x and C y that maximizes the mutual information I(C x (x), C y (y)) while limiting the entropies H(C x (x)) and H(C y (y)). Here we propose representing the mutual information as a difference of entropies. I(C x (x), C y (y)) = H(C y (y)) − H(C y (y)|C x (x))When the coding functions are parameterized by a function Ψ, the above quantities become a function of Ψ. We can then formulate the following nested optimization problem. The above quantities are expectations over the population distribution on pairs (x, y). In practice we have only a finite sample form the population. But the preceding section presents theoretical evidence that, unlike lower bound estimators, upper bound cross-entropy estimators can meaningfully estimate large entropies from feasible samples. DISPLAYFORM0 Maximum mutual information (MMI) predictive coding seems well motivated as a method of unsupervised pretraining of representations that maintain semantic signal while dropping uninformative noise. However, the maximization of mutual information is a difficult training objective. We have given theoretical arguments that representing mutual information as a difference of entropies, and estimating those entropies by minimizing cross-entropy loss, is a more statistically justified approach than maximizing a lower bound on mutual information. Unfortunately cross-entropy upper bounds on entropy fail to provide either upper or lower bounds on mutual information -mutual information is a difference of entropies. We cannot rule out the possible existence of superintelligent models, models beyond current expressive power, that dramatically reduce cross-entropy loss. Lower bounds on entropy can be viewed as proofs of the non-existence of superintelligence. We should not surprised that such proofs are infeasible.

Represent the TLDR for finding the corresponding scientific article

We give a theoretical analysis of the measurement and optimization of mutual information.

represent the tldr for finding the corresponding scientific article
To give you a sense - "Conditional VAE on top of latent spaces of pre-trained generative models that enables  transfer between drastically different domains while preserving locality and semantic alignment." should be close to "Domain transfer is a exciting and challenging branch of machine learning because models must learn to smoothly transfer between domains, preserving local variations and capturing many aspects of variation without labels. However, most successful applications to date require the two domains to be closely related (ex. image-to-image, video-video), utilizing similar or shared networks to transform domain specific properties like texture, coloring, and line shapes. Here, we demonstrate that it is possible to transfer across modalities (ex. image-to-audio) by first abstracting the data with latent generative models and then learning transformations between latent spaces. We find that a simple variational autoencoder is able to learn a shared latent space to bridge between two generative models in an unsupervised fashion, and even between different types of models (ex. variational autoencoder and a generative adversarial network). We can further impose desired semantic alignment of attributes with a linear classifier in the shared latent space. The proposed variation autoencoder enables preserving both locality and semantic alignment through the transfer process, as shown in the qualitative and quantitative evaluations. Finally, the hierarchical structure decouples the cost of training the base generative models and semantic alignments, enabling computationally efficient and data efficient retraining of personalized mapping functions. Domain transfer has long captured the imagination of inventors and artists alike. The early precursor of the phonograph, the phonautograph, was actually inspired by the idea of "words which write themselves", where the shape of audio waveforms would transform into the shape of writing, capturing the content and character of the speaker's voice in shape and stroke of the written characters BID9. While perhaps fanciful at the time, modern deep learning techniques have shown similar complex transformations are indeed possible. Deep learning enables domain transfer by learning a smooth mapping between two domains such that the variations in one domain are reflected in the other. This has been demonstrated to great effect within a data modality, for example transferring between two different styles of image BID12 BID18, video BID26 and music BID23. The works have been the basis of interesting creative tools, as small intuitive changes in the source domain are reflected by small intuitive changes in the target domain. Furthermore, the strong conditioning signal of the source domain makes learning transformations easier than learning a full generative model in each domain. Despite these successes, this line of work in domain transfer has several limitations. The first limitation is that it requires that two domains should be closely related (e.g. image-to-image or videoto-video). This allows the model to focus on transferring local properties like texture and coloring instead of high-level semantics. For example, directly applying these image-to-image transfer such as CycleGAN or its variants to images from distant domains leads to distorted and unrealistic . This agrees with the findings of BID3 who show that CycleGAN transformations are more akin to adversarial examples than style transfer, as the model Our method aims at transfer from one domain to another domain such that the correct semantics (e.g., label) is maintained across domains and local changes in the source domain should be reflected in the target domain. To achieve this, we train a model to transfer between the latent spaces of pre-trained generative models on source and target domains. (a) The training is done with three types of loss functions: The VAE ELBO losses to encourage modeling of z 1 and z 2, which are denoted as L2 and KL in the figure. The Sliced Wasserstein Distance loss to encourage cross-domain overlapping in the shared latent space, which is denoted as SWD. The classification loss to encourage intra-class overlap in the shared latent space, which is denoted as Classifier. The training is semi-supervised, since and requires no supervision (classes) while only needs such information. (b) To transfer data from one domain x 1 (an image of digit "0") to another domain x 2 (an audio of human saying "zero", shown in form of spectrum in the example), we first encode x 1 to z 1 ∼ q(z 1 |x 1), which we then further encode to a shared latent vector z using our conditional encoder, z ∼ q(z |z 1, D = 1), where D donates the operating domain. We then decode to the latent space of the target domain z 2 = g(z|z, D = 2) using our conditional decoder, which finally is used to generate the transferred audio x 2 = g(x 2 |z 2).learns to hide information about the source domain in near-imperceptible high-frequency variations of the target domain. The second limitation is data efficiency. Most conditional GAN techniques, such as Pix2Pix BID12 and vid2vid BID26, require very dense supervision from large volumes of paired data. This is usually accomplished by extracting features, such as edges or a segmentation map, and then training the conditional GAN to learn the inverse mapping back to pixels. For many more interesting transformations, no such easy alignment procedure exists, and paired data is scarce. We demonstrate the limitation of existing approaches in Appendix C.For multi-modal domain transfer, we seek to train a model capable of transferring instances from a source domain (x 1) to a target domain (x 2), such that local variations in source domain are transferred to local variations in the target domain. We refer to this property as locality. Thus, local interpolation in the source domain would ideally be similar to local interpolation in target domain when transferred. There are many possible ways that two domains could align such that they maintain locality, with many different alignments of semantic attributes. For instance, for a limited dataset, there is no a priori reason that images of the digit "0" and spoken utterances of the digit "0" would align with each other. Or more abstractly, there may be no agreed common semantics for images of landscapes and passages of music, and it is at the liberty of the user to define such connections based on their own intent. Our goal in modeling is to respect the user's intent and make sure that the correct semantics (e.g., labels) are shared between the two domains after transfer. We refer to this property as semantic alignment. A user can thus sort a set of data points from in each domain into common bins, which we can use to constrain the cross-domain alignment. We can quantitatively measure the degree of semantic alignment by using a classifier to label transformed data and measuring the percentage of data points that fall into the same bin for the source and target domain. Our goal can thus be stated as learning transformations that preserve locality and semantic alignment, while requiring as few labels from a user as possible. To achieve this goal and tackle prior limitations, we propose to abstract the domain domains with independent latent variable models, and then learn to transfer between the latent spaces of those models. Our main contributions include:• We propose a shared "bridging" VAE to transfer between latent generative models. Locality and semantic alignment of transformations are encouraged by applying a sliced-wasserstein distance, and a classification loss respectively to the shared latent space.• We demonstrate with qualitative and quantitative that our proposed method enables transfer both within a modality (image-to-image) and between modalities (image-to-audio).• Since we training a smaller secondary model in latent space, we find improvements in training efficiency, measured by both in terms of the amount of required labeled data and well training time. Figure 1 diagrams our hierarchical approach. We first independently pre-train separate generative models, either a VAE or GAN, for both the source and target domain. For VAEs, data is encoded from the data domain to the latent space through a learned encoding function z ∼ q(z|x), and decoded back to the data space with a decoder functionx ∼ g(x|z). For GANs, we choose latent samples from a spherical Gaussian prior z ∼ p(z) and then use rejection sampling to only select latent samples whose associated data x = g(z) is classified with high confidence by an auxiliary classifier. We then add the bridging conditional VAE with shared weights, tasked with modeling both latent spaces z 1, z 2. The VAE has a single shared latent space z, that corresponds to both domains. Sharing the weights encourages the model to seek common structures between the latent domains, but we also find it helpful to condition both the encoder q shared (z |z, D) and decoder g shared (z|z, D), with an additional one-hot domain label, D, to allow the model some flexibility to adapt to variations particular to each domain. While the low-level VAEs have a spherical Gaussian prior, we penalize the KL-Divergence to be less than 1, allowing the models to achieve better reconstructions and retain some structure of the original dataset for the bridging VAE to model. Full architectural and training details can be found in the Appendix. The conditional bridging VAE objective consists of three types of loss loss terms:1. Evidence Lower Bound (ELBO). Standard value that is maximized for training a VAE. DISPLAYFORM0 where the likelihood π(z; g) is a spherical Gaussian N (z; g, σ 2 I), and σ and β KL are hyperparmeters set to 1 and 0.1 respectively to encourage reconstruction accuracy. 2. Sliced Wasserstein Distance (SWD) BID2. The distribution distance between mini-batches of samples from each domain in the shared latent space (z 1, z 2). DISPLAYFORM1 where Ω is a set of random unit vectors, proj(A, a) is the projection of A on vector a, and W 2 2 (A, B) is the quadratic Wasserstein distance. 3. Classification Loss (Cls). For each domain d ∈ {1, 2}, we enforce semantic alignment with attribute labels y and a classification loss in the shared latent space: DISPLAYFORM2 where H is the cross entropy loss, f (z) is a one-layer linear classifier. The shared latent space where the blue line is the decision boundary of the classifier. Here, the points from both domains are overlapping, class-separated, spread evenly, and maintain the continuity of color gradient. Including terms for both domains, the total training loss is then DISPLAYFORM0 Where β SWD and β Cls are scalar loss weights. The transfer procedure is illustrated FIG1 using synthetic data. For reconstructions, data x 1 is passed through two encoders, DISPLAYFORM1 For transformations, the encoding is the same, but decoding uses decoders (and conditioning) from the second domain,ẑ 2 ∼ g shared (ẑ 2 |z, D = 2),x 2 ∼ g(x 2 |ẑ 2). Further analysis of this example and intuition behind loss terms is summarized in FIG1 and detailed in Appendix A. We discuss two aspects of existing research focuses that are related to our work, followed by how our work differentiates itself from them in order to deal with the challenge identified in this paper. Deep latent generative models are usually constructed to transfer a simple, tractable distribution p(z) into the approximation of population distribution p * (x), through an expressive neural network function. Such models include VAE BID15 and GAN BID10. GANs are trained with an accompany classifier that attempts to distinguish between samples from the decoder and the true dataset. VAEs, in contrast, are trained with an encoder distribution q(z|x) as an approximation to the posterior p(z|x) using variational approximation through the use of evidence lower bound (ELBO). These classes of models have been thoroughly investigated in many applications and variants BID11 BID17 BID1 including conditional generation BID22, generation of one domain conditioned on another BID4 BID24, generation of high-quality images BID13 and long-range structure in music. In terms of overall VAE structure, BID29 studies options to build hierarchical VAEs. The domain transfer enables transfer between images BID12 BID18, audio BID23 and video BID26, intuitively mapping between two domains where the variations in one domain should be reflected in the other. Besides visually appealing , domain transfer also enables application such as image colorization BID28. Domain transfer is also proposed to be done through jointly training of generative models BID21. Also, the behavior of domain transfer models also attracts attention. For example, BID3 suggests that image transfer does only local, texture level transfer. To enable transfer between possibly drastically different domains, Our work proposes to use VAE in modeling the latent space of pre-trained generative models, in several aspects differentiating itself from related work. Generally, the modeling of latent space is different from modeling directly the data domains as most of latent generative models naturally do, also, the transfer in a more abstract semantics and between heterogeneous domains differs from most domain transfer method which focusing on locality and similar domains. More specifically, regarding modeling latent spaces, BID29 suggests training a hierarchical VAE on a single domain should be done end-to-end, whose extension to multiple domains seems non-trivial and likely to suffer from data efficient issues. Instead, our proposed work, though enabling separation of pre-trained model and conditional shared VAE, apply to domain transfer setting while overcoming this shortcoming. Moreover, regarding shared latent space, BID20 proposes to use shared latent space for two generative models on data domains. It requires joint training of generative models on both domains using dense supervision which is infeasible for drastically different domains. Our work that leverages pre-trained generative model and model the latent space instead of data domains addresses to this limitation. While the end goal of our method is to enable creative mapping between datasets with arbitrary alignments, for quantitative studies we restrict ourselves to three domains where there exist a somewhat natural alignment to compare against:1. MNIST , which contains images of hand-written digits of 10 classes from "0" to "9". 2. Fashion MNIST BID27, which contains fashion related objects such as shoes, tshirts, categorized into 10 classes. The structure of data and the size of images are identical to MNIST. 3. SC09, a subset of Speech Commands Dataset 1, which contains the record of audio of humans saying digits from "0" to "'9". For MNIST and Fashion MNIST, we prepare VAE with MLP encoder and decoder following setting up in. More specifically, we use stacks of fully-connected linear layers activated by ReLU, together with a "Gated Mixing Layer". The full network architecture of the bridging VAE is detailed in Appendix B. For SC09 we use the publicly available WaveGAN BID6 2. We would like to emphasize that we only use class level supervision for enforcing semantic alignment with the latent classifier. We examine three scenarios of domain transfer:1. MNIST ↔ MNIST. We first train two lower-level VAEs from different initial conditions. The bridging autoencoder is then tasked with transferring between latent spaces while maintaining the digit class from source to target. 2. MNIST ↔ Fashion MNIST. In this scenario, we specify a global one-to-one mapping between 10 digit classes and 10 fashion object classes (See TAB3 in Appendix for details).Under review as a conference paper at ICLR 2019The bridging autoencoder is tasked with preserving this mapping as it transfers between images of digits and clothing.3. MNIST ↔ SC09. For the speech dataset, we first train a GAN to generate audio waveforms BID6 of spoken digits. We chose to use a WaveGAN because we wanted a global latent variable for the full waveform (as opposed to a distributed latent code as in BID7). It also gives us an opportunity to explore transferring between different classes of models. The bridging autoencoder is then tasked with transferring between a VAE of written digits and a GAN of spoken digits. For reconstructions and domain transfer, we present both qualitative and quantitative . Quantitative measurements of semantic alignment are performed with pre-trained classifiers in each data domain. Given that the datasets have pre-aligned classes, when evaluating transferring from data x d1 to x d2, the reported accuracy is the portion of instances that x d1 and x d2 have the same predicted class. Qualitative reconstruction are shown in FIG2 and the quantitative reconstruction accuracies are given in TAB0. For domain transfer, qualitative are shown in FIG4 and the quantitative transfer accuracies are given in Table 2. Within each group, on the left is the data in the source domain and on the right is the data in the target domain. We see that transfer maintains the label, yet still maintains diversity of samples, reflecting the transfer of a broad range of attributes. Table 2: Domain Transfer Accuracy for MNIST ↔ MNIST, MNIST ↔ Fashion MNIST and MNIST ↔ SC09 transfer respectively. We compare to pre-existing approaches trained on raw-pixels for MNIST ↔ Fashion MNIST only, MNIST → MNIST involves transferring between pretrained models with different initial conditions which is not directly comparable, and in MNIST → SC09, the two data domains were too distinct to provide any reasonable transfer with existing methods. Further comparisons can be found in Appendix C. DISPLAYFORM0 Interpolation can act as a good proxy for locality and local smoothness of the latent transformations, as by definition good interpolations require that small changes in the source domain are reflected by small changes in the target domain. We show inter-class and inter-class interpolation in FIG5 and FIG6 respectively. Particularly, we are interested in two comparing three rows of interpolations: the interpolation in the source domain's latent space, which acts a baseline for smoothness of interpolation for a pre-trained generative model, transfer fixed points to the target domain's latent space and interpolate in that space, and transfer all points of the source interpolation to the target domain's latent space, which shows how the transferring warps the latent space. We use spherical interpolation (e.g., √ pv 1 + (1 − p)v 2 )since we are interpolating in the Gaussian latent space. Note in FIG5 that the second and third rows have comparably smooth trajectories, reflecting that locality has been preserved. For inter-class interpolation in FIG5 interpolation is smooth within a class, but between classes the second row blurs pixels to create blurry combinations of digits, while the full transformation in the third row makes sudden transitions between classes. This is expected from our training procedure as the bridging autoencoder is modeling the marginal posterior of each latent space, and thus always stays on the manifold of the actual data during interpolation. Since our method is a semi-supervised method, we want to know how effectively our method leverages the labeled data. In Table 3 we show for the MNIST → MNIST setting the performance measured by transfer accuracy with respect to the number of labeled data points. Labels are distributed 3 In FIG2 and 4 we show the spectrum of audio samples for demonstration purpose. The corresponding audio samples themselves are available here: https://drive.google.com/drive/u/8/folders/ 12u6fKvg0St6gjQ_c2bThX9B2KRJb7Cvk FIG5, except that interpolation now happens between classes. It can be shown that, unlike regular generative model (row 1 and row 2 in each group) that exhibits pixel (data) level interpolation, especially the blurriness and distortion half way between instances of different labels, our proposed transfer (row 3) resorts to produce high-quality, in-domain data. This is an expected behavior since our proposed method learns to model the marginalized posterior of data distribution.evenly among classes. The accuracy of transformations grows monotonically with the number of labels and reaches over 50% with as few as 10 labels per a class. Without labels, we also observe accuracies greater than chance due to unsupervised alignment introduced by the SWD penalty in the shared latent space. Table 3: MNIST → MNIST transfer accuracy as a function of labeled data points. The supervised data points are split evenly among all 10 classes. Besides data efficiency, pre-training the base generative models has computational advantages. For large generative models that take weeks to train, it would be infeasible to retrain the entire model for each new cross-domain mapping. The bridging autoencoder avoids this cost by only retraining the latent transfer mappings. As an example from these experiments, training the bridging autoencoder for MNIST ↔ SC09 takes about one hour on a single GPU, while retraining the SC09 WaveGAN takes around four days. Finally, we perform an ablation study to confirm the benefits of each architecture component to transfer accuracy. For consistency, we stick to the MNIST → MNIST setting with fully labeled data. In Table 4, we see that the largest contribution to performance is the giving the bridging VAE a domain conditioning signal, allowing it to share weights between domains, but also adapt to the specific structure of each domain. Further, the increased overlap in the shared latent space induced by the SWD penalty is reflected in the greater transfer accuracies. Data Domain Vanilla, Unconditional VAE Conditional VAE Conditional VAE + SWD Accuracy 0.149 0.849 0.980 Table 4: Ablation study of MNIST → MNIST transfer accuracies. We have demonstrated an approach to learn mappings between disparate domains by bridging the latent codes of each domain with a shared autoencoder. We find bridging VAEs are able to achieve high transfer accuracies, smoothly map interpolations between domains, and even connect different model types (VAEs and GANs). Here, we have restricted ourselves to datasets with intuitive classlevel mappings for the purpose of quantitative comparisons, however, there are many interesting creative possibilities to apply these techniques between domains without a clear semantic alignment. As a semi-supervised technique, we have shown bridging autoencoders to require less supervised labels, making it more feasible to learn personalized cross-modal domain transfer based on the creative guidance of individual users. We want to archive following three goals for the proposed VAE for latent spaces:1. It should be able to model the latent space of both domains, including modeling local changes as well.2. It should encode two latent spaces in a way to enable domain transferability. This means encoded z 1 and z 2 in the shared latent space should occupy overlapped spaces.3. The transfer should be kept in the same class. That means, regardless of domain, zs for the same class should occupy overlapped spaces. With these goals in mind, we propose to use an optimization target composing of three kinds of losses. In the following text for notational convenience, we denote approximated posterior DISPLAYFORM0, 2}, the process of sampling z d from domain d.1. Modeling two latent spaces with local changes. VAEs are often used to model data with local changes in mind, usually demonstrated with smooth interpolation, and we believe this property also applies when modeling the latent space of data. Consider for each domain d ∈ {1, 2}, the VAE is fit to data to maximize the ELBO (Evidence Lower Bound) DISPLAYFORM1 where both q and g are fit to maximize L ELBO d. Notably, the latent space zs are continuous, so we choose the likelihood π(z; g) to be the product of N (z; g, σ 2 I), where we set σ to be a constant that effectively sets log π(z; g) = ||z − g|| 2, which is the L2 loss in FIG0. Also, D KL is denoted as KL loss in FIG0.2. Cross-domain overlapping in shared latent space. Formally, we propose to measure the cross-domain overlapping through the distance between following two distributions as a proxy: the distribution of z from source domain (e.g., z 1 ∼ Z 1) and that from the target domain (e.g., z 2 ∼ Z 1). We use Wasserstein Distance to measure the distance of two sets of samples (this notion straightforwardly applies to the mini-batch setting) S 1 and S 2, where S 1 is sampled from the source domain z 1 ∼ Z 1 and S 1 from the target domain z 2 ∼ Z d. For computational efficiency and inspired by BID5, we use SWD, or Sliced Wasserstein Distance BID2 between S 1 and S 2 as a loss term to encourage cross-domain overlapping in shared latent space. This means in practice we introduce the loss term DISPLAYFORM2 where Ω is a set of random unit vectors, proj(A, a) is the projection of A on vector a, and W 2 2 (A, B) is the quadratic Wasserstein distance, which in the one-dimensional case can be easily solved by monotonically pairing points in A and B, as proven in BID5.3. Intra-class overlapping in shared latent space. We want that regardless of domain, zs for the same class should occupy overlapped spaces, so that instance of a particular class should retain its label through the transferring. We therefore introduce the following loss term for both domain DISPLAYFORM3 where H is the cross entropy loss, f (z) is a one-layer linear classifier, and l x is the one-hot representation of label of x where x is the data associated with z. We intentionally make classifier f as simple as possible in order to encourage more capacity in the VAE instead of the classifier. Notably, unlike previous two categories of losses that are unsupervised, this loss requires labels and is thus supervised. In Figure 7 we show the intuition to design and the contribution to performance from each loss terms. DISPLAYFORM4 Figure 7: Synthetic data to demonstrate the transfer between 2-D latent spaces with 2-D shared latent space. Better viewed with color and magnifier. Columns (a) -(e) are synthetic data in latent space, reconstructed latent space points using VAE, domain 1 transferred to domain 2, domain 2 transferred to domain 1, shared latent space, respectively, follow the same arrangement as FIG1. Each row represent a combination of our proposed components as follows: Regular, unconditional VAE.Here transfer fails and the shared latent space are divided into region for two domains. Conditional VAE. Here exists an overlapped shared latent space. However the shared latent space are not mixed well. Conditional VAE + SWD. Here the shared latent space are well mixed, preserving the local changes across domain transfer. Conditional + SWD + Classification. This is the best scenario that enables both domain transfer and class preservation as well as local changes. It is also highlighted in FIG1. An overall observation is that each proposed component contributes positively to the performance in this synthetic data, which serves as a motivation for our decision to include all of them. The model architecture of our proposed VAE is illustrated in Figure B. The model relies on Gated Mixing Layers, or GML. We find empirically that GML improves performance by a large margin than linear layers, for which we hypothesize that this is because both the latent space (z 1, z 2) and the shared latent space z are Gaussian space, GML helps optimization by starting with a good initialization. We also explore other popular network components such as residual network and batch normalization, but find that they are not providing performance improvements. Also, the condition is fed to encoder and decoder as a 2-length one hot vector indicating one of two domains. For all settings, we use the dimension of shared latent space 100, β SWD = 1.0 and β CLs = 0.05, Specifically, for MNIST ↔ MNIST and MNIST ↔ Fashion MNIST, we use the dimension of shared latent space 8, 4 layers of FC (fully connected layers) of size 512 with ReLU, β KL = 0.05, β SWD = 1.0 and β CLs = 0.05; while for MNIST ↔ SC09, we use the dimension of shared latent space 16, 8 layers of FC (fully connected layers) of size 1024 with ReLU β KL = 0.01, β SWD = 3.0 and β CLs = 0.3. The difference is due to that GAN does not provide posterior, so the latent space points estimated by the classifier is much harder to model. For optimization, we use Adam optimizer BID14 with learning rate 0.001, beta 1 = 0.9 and beta 2 = 0.999. We train 50000 batches with batch size 128. We do not employ any other tricks for VAE training. We compare our with two existing approaches, Pix2Pix BID12 on the left and CycleGAN on the right, on the same MNIST ↔ Fashion MNIST transfer settings used in Figured 4. We show qualitative from applying Pix2Pix and CycleGAN in Figure 9, which can be compared with Figured 4, as well as quantitative in Table 5. Both qualitative and quantitative shows the limitation of existing methods and our proposed approach's advantage over them. Figure 9: Qualitative from applying Pix2Pix BID12 on the left and Cycle-GAN on the right, on the same settings used in Figured 4. Visually, both existing transfer approaches suffer from less desirable overall visual quality and less diversity in local changes, compared to our proposed approach. Particularly, Pix2Pix more or less makes semantic labels correct but suffers from mode collapses in each label, while CycleGAN has slightly better quality but suffers from label collapse, which is observable here that most of digits are transferred to Dress and leads to bad transfer accuracy. Transfer Accuarcy FID (Fréchet Inception Distance)Pix2Pix BID12 0.773 0.0786 CycleGAN 0.075 0.3333 This work 0.945 0.0055 Table 5: Quantitative of methods using Transfer Accuracy and Fréchet Inception Distance (FID). Transfer Accuracy is calculated using the same protocol as Table 2 where a higher value indicates the better performance, while FID is computed using the activation of the classifier on the target domain (Fashion MNIST) where a lower value indicates a better image quality. Quantitatively both existing methods perform worse than our proposed method. Here Pix2Pix more or less makes semantic labels correct but still suffers from lower accuracy and image quality. while CycleGAN suffers from label collapse, which leads to an even lower transfer accuracy and FID."

We establish that the scaling laws derived in (Bora et al., 2017) are optimal or near-optimal in the absence of further assumptions.


Question: Represent the scientific article to find a short TLDR

Question: In this paper, we propose a neural network framework called neuron hierarchical network (NHN), that evolves beyond the hierarchy in layers, and concentrates on the hierarchy of neurons. We observe mass redundancy in the weights of both handcrafted and randomly searched architectures. Inspired by the development of human brains, we prune low-sensitivity neurons in the model and add new neurons to the graph, and the relation between individual neurons are emphasized and the existence of layers weakened. We propose a process to discover the best base model by random architecture search, and discover the best locations and connections of the added neurons by evolutionary search. Experiment show that the NHN achieves higher test accuracy on Cifar-10 than state-of-the-art handcrafted and randomly searched architectures, while requiring much fewer parameters and less searching time. Neural networks can be designed either by human experts or search algorithms, both of which have gained great success in image classification and language modeling BID45 BID33. Network architectures designed by both means are mostly layer-based or block-based, which means that the fundamental components are either operation layers or blocks that consist of several layers. A clear tendency can be observed that models with more parameters generally have better performances. It is a well-established fact that redundancy of parameters exists widely in handcrafted neural networks BID16 BID41 BID25. We find that such claim holds for architectures discovered by random search or evolutionary search as well. The pruning of unimportant neurons emphasizes the hierarchical relation between individual neurons. Additionally, the decrease in accuracy after parameter reduction is generally inevitable. Therefore, we propose a heuristic procedure to construct neuron-based network architectures by pruning redundant connections and neurons in layer-based models and adding new neurons to strengthen the neuron hierarchy while achieving competitive performances as layer-hierarchy models. Experiments show that NHN achieves higher test accuracy than DenseNet, SMASH BID4 and hierarchical representation with much fewer parameters. Handcrafted architectures. Successful convolutional neural networks (CNNs) designed by human experts can be sketchily categorized by the way data flow through the networks, i.e., plain networks and branching networks. A notable example of plain networks would be VGG nets BID37, where there are only one input and output path in each hidden layer. However, in a branching network, the computation flow splits somewhere in the network and merges in a latter layer BID0. The splitting and aggregation may occur multiple times in a single network. Many have discovered numerous branching network architectures whose performances surpass plain ones while requiring fewer parameters. Skip connections BID17 BID18 are increasingly popular in improving the performance of deep neural networks, and it becomes common to observe additional convolutions (or other forms of operations) stacked between large layers BID24 BID40. In fact, the "stacked-between" operations can be considered as part of a generalized residual block. Multi-branching computation graphs benefit addressing the gradient vanishing problem during the gradient descent training BID18. The distinguished techniques mentioned above (plus more listed in Table 1) share the same idea of weakening the hierarchy between layers by introducing complex paths to the data flow. The idea is further highlighted by architecture search algorithms. Random and evolutionary architectures. Machine learning algorithms evolve fast. Designing neural networks that perform remarkably on a given task requires ample experience. It has been found that neural networks are not only good at autonomically extracting useful features from raw data, but also capable of finding optimal network architectures to that end. Neural architecture search (NAS) has been attested to its ability to design network architectures for language modeling and image classification. However, candidate models have to be entirely randomly generated and fully trained, therefore NAS is extremely computation intensive, which dims its competitive performances to handcrafted architectures. Many efforts have been devoted to reducing the computational costs of NAS while ensuring sufficient capacity of search space BID47 BID5. Two major ideas to achieve the purpose are to design individual reusable cells rather than the entire networks or to share trained weights between models. Recently, BID33 proposed to describe each candidate model as a subgraph of a single directed acyclic graph (DAG). By sharing weights between submodels, the searching time of NAS is reduced by 1000X in term of GPU-hours. Genetic algorithms are also applied in searching the optimal architectures of CNNs BID29 BID43 BID34. BID35 proposed regularized evolution (RE) to remove the oldest models from the population, instead of removing the worst ones as in traditional tournament selection BID14. The best CNN model discovered by RE achieves the state-of-the-art performance on Cifar-10, i.e., 2.13% test accuracy on average. However, the number of parameters it requires is as large as nearly 35 million. Convolutional neural fabrics (CNF) BID36 BID42 and other forms of random search methods BID2 BID4 BID10 have been investigated as well. Layer-wise to neuron-wise hierarchy. Take the best model discovered by macro search in ENAS BID33 for example. The 12-layer CNN model contains over 21 million parameters and achieves 4.23% test accuracy on Cifar-10. If we remove 75% − 90% parameters in all 3 × 3 and 5 × 5 convolutions, the test accuracy is hardly compromised after the same duration of retraining. Even though the architectures in all the search methods are described as directed graphs, each node in these graphs represents either an operation layer (e.g. convolution or pooling layer) or an operation block (e.g. residual block). None of the nodes stands for an actual individual neuron in the network. On one hand, random search and evolutionary search tend to discover architectures that contain complex branching paths. On the other hand, the pruned versions of such architectures work nearly as well as intact ones. These facts bring about the hypothesis that the hierarchy of neurons should work as well as the hierarchy of layers. Please note that we do not simply replace layers with individual neurons, considering layers are composed of abundant neurons. We need the sufficient number of neurons to meet the feature representation requirements. A good hierarchical architecture of neurons may be discovered by either random search or evolutionary search, or combined. The search process must be carefully designed. In this paper, we propose a three-step course to discover the optimal neuron hierarchy for image classification (see FIG1, i.e., discover the optimal layer hierarchy with ENAS, prune unimportant neurons in the discovered layer-wise model, and randomly add new neurons to the pruned model to enrich the expressive capacity of neuron hierarchy networks. It is worth pointing out that the three-step procedure is also an imitation of the natural development of human brains . For example, the creation and searching by ENAS correspond to the mass neurogenesis before birth. The pruning of unimportant neurons corresponds to the apoptosis before puberty BID11. The addition of new neurons to the pruned model corresponds to the persisting neurogenesis during adulthood BID30. Although the existence of neurogenesis in mature brains is being debated in the field of neuroscience BID38 BID3, the software simulation of such process by our work indicates that it is helpful in improving the learning capability of neural networks. In this section, we propose a heuristic algorithm to derive network architectures that emphasize the relation between individual neurons based on traditional layer-based models by network pruning and adding new neurons. We directly employ the macro search method proposed by BID33 to discover a 12-layer CNN architecture for image classification. Note that we need to distinguish the important neurons from the less important ones, thus we are not performing Dropout BID39 during the searching and re-training in ENAS. Even though Dropout has proved to be helpful in increasing the generalization power of networks, it distributes the feature extracting capabilities evenly in all neurons, which makes it hard for us to tell which neurons are more important than others. We will, however, employ Dropout in the final fine-tuning stage. We use the cosine schedule BID28 ) that involves 4 warm restarts (for a total of 310 epochs) to adjust the learning rate during training, as in BID33, with the maximum value of 0.05 and minimum of 0.001. Cutout is used to enrich the sample capacity and improve the generalization of the model. The architecture discovered by ENAS is illustrated in FIG2 (the large blocks connected with solid lines) Network pruning BID16 a; BID25 has been used to get rid of unimportant parameters in CNNs. When followed by quantization and sparsification techniques, the network pruning methods can considerably reduce the memory and bandwidth consumption of deep networks in distributed systems. But it requires pre-learning of the connectivity and a very long time of re-training to restore the performance. We employ the dynamic network pruning (DNP) method proposed by to remove redundant weights. Specifically, during the search of optimal layer-hierarchy model, we replace the global average pooling BID24 with grouped convolution BID21, in which the group number equals the number of both input and output channels and the kernel sizes equal the sizes of input feature maps, so that the grouped convolution exactly maps each feature map to a single value. We also smooth out all 3 × 3 and 5 × 5 convolution kernels with 3 × 3 Gaussian filters after initialization. After training the optimal model discovered by ENAS from scratch, we find that only a small portion of trained kernels have apparent texture appearance, while a majority of them are practically flat. We remove nearly 85% of the kernels based on standard deviation thresholds. Accuracy drop is observed on the pruned model, it expresses the need for finding a way to improve the representation capability of our model, which leads to the idea of adding additional neurons to the network. In order to distinguish the newly added neurons from the existing neurons, we refer to the former ones as add-on neurons. The add-on neurons are free to be inserted between two arbitrary neurons that are originally located in different layers, since a connection within a layer could cause confusion about computation order. The input and output neurons of an add-on neuron must share the same data dimension, so that we can define a standard rule to create add-on neurons. Take the best architecture discovered by ENAS for example. The 12-layer architecture is divided by two average pooling layers into three segments, with each containing 4 convolution cells. The data dimensions are uniformed in each segment, therefore add-on neurons are not allowed to be inserted across separate segments. Two types of operations are allowed in an add-on neuron, i.e., 3 × 3 convolution and 5 × 5 convolution. Algorithm 1: Location-Direction Search (LDS) Input: A directed acyclic graph G, in which each node represents a layer in the layer-hierarchy network. Output:G = G + A, in which A represents the discovered add-on neurons and their connections to the input network G. Require: Population capacity K, cycle number T, train samples B train, test samples B test Location search Initialize: DISPLAYFORM0 Location and direction search. In order to increase the search speed for add-on neurons, we propose a location-direction search (LDS) method to search the specific positions and connections independently, which means that we first search for the most possible locations where the add-on neurons may occur, then decide on their specific input and output connections. The location search is carried out in an evolutionary fashion, as described in Algorithm 1. First, we define an initial population (GenerateLoc). Each individual in the population stands for a possible combination of add-on locations. During each evolution cycle, we train every model in the population for several iterations (M) and we crossover the best models (Recombine) to reproduce new models, mutate the best models (M utate) to produce new models, and remove the worst models from the population. The crossover (Recombine) is performed as follows. DISPLAYFORM1 Let p 1 l, p 2 l, p 3 l, p 4 l denote the best 4 models, in which lower indices are assigned to better models. The crossover pairs to generate 5 new models are selected as {After the best combination of add-on locations is discovered, an evolutionary search on add-on directions is performed. The direction of an add-on neuron (i.e. to choose which neurons to connect with) at given location is selected sparsely (GenerateDir), since we do not want add-on neurons at the same location to marginally form a new layer. Because the potential number of different combinations of add-on directions can be extremely large, we simply re-generate all combinations at each evolution cycle to find the best one. No mutation or crossover will be performed during direction search. For example, let us employ LDS to the first segment in FIG2, i.e., layers {0, 1, 2, 3}, as separately shown in FIG3. We define the location where one or several add-on neurons may occur as a new sparse convolution layer (e.g., a 0 in FIG2). All possible locations for add-on neurons are {,,,,,}, in which tuple (a, b) means that add-on neurons are inserted between layer a and b, as illustrated as gray blocks in FIG3. If we enforce a restraint that only one input add-on location with the same kernel size is allowed for each existing layer and include the pooling layer into search space, the overall location search complexity will be 1.9 × 10 8 (which is (4!) 3 · (4!) 3, since there are three segments in the network and we are using two different kernel sizes of add-on neurons). When the optimal add-on locations are determined, we perform the direction search with the sparsity of 0.1, which means that we will leave out 90% of all possible connections and leave the remaining effective neurons as the search . Keeping neurons with high sensitivities is recommended, because we need the add-on neurons to improve the performance of the model. Therefore, the standard deviation thresholds are used to determine the optimal add-on directions. Weight sharing in add-on search. We adopt the idea of sharing weights between submodels by BID33 to increase the search speed in LDS. Specifically, we define all possible connections at all possible locations beforehand. Let W = {w l,d |l ∈ L, d ∈ D} denote all shared weights for LDS, in which w l,d denotes the weight at location l and direction d, and L and D stand for all possible locations and directions, respectively. When generating new subgraphs for location or direction search, we simply select the corresponding weights in W k,t = {w l,d |l ∈ L k,t, d ∈ D k,t}, in which (k, t) stands for the kth model in the search population at evolution cycle t. In general, all w l,d in W are trained in an asynchronous fashion. The proposed approach for neuron hierarchical networks can be described as Algorithm 2. Require: Number of layers L Initialize: Layer-hierarchy network G Layer hierarchy search DISPLAYFORM0 TrainG from scratch returnG Table 1: Cifar-10 test errors of NHN and other methods. The first block represents handcrafted architectures. The second block represents architectures discovered by random search methods. The third block represents evolutionary architectures. Note that "d/o" stands for Dropout, "s/s" stands for Shake-Shake regulation, "c/o" stands for Cutout, and "d/p" stands for Drop-path. Score = (e/2) 2 + (n/25) 2, where e denotes the test error in percentage and n denotes the number of parameters in million. Smaller score indicates better overall performance. *The performance of Evolving DNN on Cifar-10 is reported by. Parameters Error ScoreResNet-110 BID18 1.7M 6.43% 3.22 pre-act-ResNet-1001 BID19 10.2M 4.62% 2.35 WRN-40-10 + d/o BID46 56M 3.80% 2.94 WRN-28-20 + SGDR BID28 145.8M 3.74% 6.12 DenseNet 27.2M 3.74% 2.16 ResNeXt 68.1M 3.58% 3.26 DenseNet + s/s BID13 26.2M 2.86% 1.77CNF BID36 21.2M 7.43% 3.81 MetaQNN BID2 11.2M 6.92% 3.49 Budgeted CNF BID42 7.6M 5.12% 2.58 BID23 38.6M 4.60% 2.77 PPP-Net BID10 11.3M 4.36% 2.23 ENAS macro BID33 21.3M 4.23% 2.28 SMASH BID4 16.0M 4.03% 2.11 NAS + more filters 37.4M 3.65% 2.36 Block-QNN-S + more filters BID47 39.8M 3.54% 2.38 ENAS micro BID33 4.6M 3.54% 1.78 EAS by NT (DenseNet) BID5 10.7M 3.44% 1.77 Progressive NAS 3.2M 3.41% 1.71 CL on ResNeXt BID0 34.4M 3.17% 2.10 DISPLAYFORM0 Genetic CNN -7.10% -Evolving DNN * -7.30% -Large-scale evolution BID34 -5.40% -Evo. search on hier. repr. 61.3M 3.63% 3.05 AmoebaNet + c/o BID35 34 We are using the Cifar-10 dataset to train and evaluate our neuron hierarchy network. The Cifar-10 dataset consists of 50k 32 × 32 3-channel training images in 10 classes, and 10k test images. The standard pre-processing is applied, i.e., subtracting the mean and dividing the standard deviation on each channel. The standard data augmentation is used for the training set, i.e., padding the images to 40 × 40 and randomly cropping down to 32 × 32, then flipping horizontally by a probability of 0.5. All experiments are run on a single NVIDIA GTX 1080Ti video card, which has sufficient GPU memory for the searching and training of the NHN.Layer hierarchy search. We use the same hyperparameters as in BID33 to perform the macro search on the entire architecture, as explained in Section 2.1. It takes ENAS no more than 16 hours to find a layer-hierarchy model with test accuracy of 96.55% on Cifar-10 with Cutout. If trained without Cutout, the model's test accuracy is 95.08%. The re-training of the best model from scratch takes almost 9 hours. Network pruning. We perform the DNP method to prune low sensitivity connections in the best model discovered by ENAS, as described in Section 2.2. We also clean up the network by removing all the neurons with zero input or output connection after DNP. We prune nearly 85% of input connections of all 3 × 3 and 5 × 5 convolution layers on average. In total, more than 70% of the parameters are pruned. The overall pruning ratio decreases because of the immense applications of 1 × 1 convolutions in ENAS, which are not directly removed by DNP. Evident impact on the performance by pruning is observed. We re-train the pruned network for another 160 epochs, which takes about 5 hours. The test accuracy is slightly dropped to 96.20% (with Cutout).Neuron hierarchy search. The population capacity (K) for location search is set to 20. The individual model p k l is described as the gene code of length of 24, in which the first 12 genes indicate whether an add-on location exists with kernel size of 3 × 3 at each existing layer (since there are 3 segments and we need 4 genes for each segment), and the last 12 genes are for 5 × 5. A positive gene indicates there are add-on neurons connecting into this layer and the value denotes the input layer index. A negative gene value (-1) means that no add-on neuron is connected to this layer. Out of avoiding too many add-on neurons being added to the model, the gene value of -1 is generated by a probability of 0.8, and the rest of the values are sampled uniformly. Each model is trained for 25 iterations (M) with the batch size of 100, so that a complete training cycle of the population exactly traverses the entire training set of Cifar-10. Submodels are evaluated by single iteration (N = 1) with the batch size of 500. Considering the sparsity of add-on locations and that a slight change in the gene value could lead to dramatic changes in performance, we simply re-generate 5 new models instead of performing mutation on the best models. A cosine schedule for learning rate that gradually decreases from 0.05 to 0.001 is used. The location search takes around 10 hours (300 evolution cycles) to complete. The best 3 sets of discovered locations are illustrated in FIG2 (the small blocks). It is apparent that having more add-on neurons at low-level layers works better for image classification. The direction search is performed as described in Section 2.3. The direction search takes about 9 hours (300 cycles) to finish. Only weights in add-on neurons are trained and weights in existing neurons are fixed during the LDS, because we are trying to find the add-on neurons that are most compatible with existing neurons. It also saves a great number of computation resources when the majority of parameters are not trained. After the best locations and directions of add-on neurons have been discovered, we re-train the entire network from scratch. Cutout BID9 and Dropout BID39 are used for better generalization. The training schedule resembles the one used by BID33. Nesterov BID31 ) is used as the gradient descent method, in which the momentum term is set to 0.9 and l 2 regularization of 2 × 10 −4 is used. Gradient clipping BID32 with the threshold of 5.0 is used to overcome the potential gradient explosion. In fact, most of the weights (i.e. the weights from the original layer-wise network) have undergone three passes of training, i.e., during ENAS, after DNP, and after LDS. It is reasonable to believe that the overall training has seen several warm restarts, hence the reason why no warm restart is introduced in the training stages after DNP and LDS. The final training takes 6 hours (260 epochs) to complete. The final test accuracy of NHN is 96.52%. The performance of NHN in comparison with other methods is listed in Table 1. The total number of parameters in the NHN is approximately 7M. We can see that the test accuracy of NHN is better than most of the methods listed in Table 1. However, several methods achieve lower test errors than NHN, e.g. DenseNet with Shake-Shake, connectivity learning on ResNeXt and AmoebaNet with Cutout. However, the amounts of parameters of these methods are huge. In order to examine the overall performances of methods in terms of test errors and parameter numbers, we define the performance score as follows. score = (e/2) 2 + (n/25) 2, where e denotes test error (%) and n denotes number of parameters (M). A lower score value indicates better performance of the model. We can see that except for Progressive NAS which has marginally better performance score, the NHN is highly competitive against all other methods in terms of low test error and small parameter number. On the whole, the searching and training of the neuron hierarchy network take approximately 46 hours in total, in which the search time of neuron hierarchy is 10 hours. The search time of NHN, however, is indeed more than that of ENAS, since we actually perform a complete ENAS process among other operations during the search. When comparing to NAS ) (which takes 450 GPUs to search for 3-4 days), the speedup is as high as 700X-940X and parameter number is reduced by 81%. NHN achieves higher accuracy than SMASH BID4 with 56% fewer parameters. Moreover, it also has higher accuracy than hierarchical representation with 89% fewer parameters and a speedup in search time of 152X, when the latter costs 1.5 days on 200 GPUs to evaluate 7000 samples. Our approach achieves more than 5X speedup with 35% fewer parameters compared to the EAS by NT BID5, which costs 2 days on 5 GPUs to achieve similar accuracy. Compared to another method that achieves similar accuracy, Progressive NAS, NHN achieves speedup in GPU time of 50X. BID5 proposed a network architecture search method which allows layer transformations during architecture search by enforcing Actor Networks for widening or deepening. Although it allows more neurons and parameters being added to the network during architecture search, the models it discovers are still layer-based, thus redundancy in weights cannot be overlooked. BID0 proposed to learn the connectivity in ResNeXt. They create a uniform grid on which each node represents a residual block, and connections are only allowed between neighboring rows of nodes, whereas in our work, each node represents an actual neuron, the network is not necessarily described as a grid, and connections are not under strict restrains. The performance of the best architecture discovered by connectivity learning is impressive, but the method only applies to ResNeXt. We did not choose fixed or handcrafted architectures as the base model because we believe that experiments conducted on a randomly searched model would be more compelling. There are also pruning-related issues with fixed models, for example, performances of ResNets and DenseNet are extremely sensitive to pruning. The training from scratch on the pruned architecture is crucial, because without it, the model only has a test accuracy of 15.7%. The NHN is not built upon the by ENAS micro search even though it presents higher test accuracy while requiring fewer parameters than macro search. It is mainly due to the mass employment of depthwise separable convolution BID7 in which kernels are pairs of vectors and cannot be directly pruned. If we replace all the depthwise separable convolutions with normal convolutions, the micro model merely gains accuracy advantage of 0.3% over the macro model. However, it instead contains 67.8M parameters, which is more than 4 times of macro (16.3M). Also, it will consume more than 28GB of memory space to perform the layer-hierarchy search. LDS FIG2 show that add-on neurons at lower layers work better, which indicates that rich representation of low-level features is crucial to the performance of NHN. When comparing the final test accuracy (96.52%) to the network without any add-on neuron (96.20%), we know that add-on neurons are helpful in increasing the performance of NHN. In fact, perturbation on the add-on genes discovered by LDS almost always leads to degradation of performance, and the total ablation of added neurons in the final model causes accuracy drop of 1.08%, which proves that the search are optimal. The main goal of this paper is neither to comprehensively discuss the properties of neuron fields BID12, nor to investigate a training method on an entirely randomly generated neuron graph. We'd like to point out that it is quite possible to directly generate a large number of free neurons with somewhat arbitrary connections and train this "random neuron field" to address the same task presented in this work. However, because modern GPUs, or to be more precise, the computation softwares that run on these GPUs are mainly designed for dense 4-d tensor calculation. It is hard to efficiently train such random neuron field at present. Therefore, as sophisticated as our approach may seem, it's an efficient method to construct network architectures that highlight the significance of individual neurons and perform competitively against other state-of-the-art methods. Neural networks that are designed by human experts and search algorithms perform outstandingly in image classification. However, redundancy in parameters exists widely in layer-based architectures. We propose a heuristic method to construct neuron-based architectures, namely, neuron hierarchical networks (NHN), that obviate the redundancy in weights and emphasize the relation between individual neurons. Experiments show that the NHN discovered based on ENAS and by locationdirection search (LDS) outperforms the original ENAS architecture and many other handcrafted and randomly searched models in Cifar-10 classification, while requiring much fewer parameters. Also, the search time of NHN is very efficient compared to several state-of-the-art network architecture search methods, while achieving competitive performance.

By breaking the layer hierarchy, we propose a 3-step approach to the construction of neuron-hierarchy networks that outperform NAS, SMASH and hierarchical representation with fewer parameters and shorter searching time.

represent the short summary for finding the corresponding scientific article (long)

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.