Question: Represent the scientific article to find a short TLDR

Question: There exists a plethora of techniques for inducing structured sparsity in parametric models during the optimization process, with the final goal of resource-efficient inference. However, to the best of our knowledge, none target a specific number of floating-point operations (FLOPs) as part of a single end-to-end optimization objective, despite reporting FLOPs as part of the . Furthermore, a one-size-fits-all approach ignores realistic system constraints, which differ significantly between, say, a GPU and a mobile phone -- FLOPs on the former incur less latency than on the latter; thus, it is important for practitioners to be able to specify a target number of FLOPs during model compression. In this work, we extend a state-of-the-art technique to directly incorporate FLOPs as part of the optimization objective and show that, given a desired FLOPs requirement, different neural networks can be successfully trained for image classification. Neural networks are a class of parametric models that achieve the state of the art across a broad range of tasks, but their heavy computational requirements hinder practical deployment on resourceconstrained devices, such as mobile phones, Internet-of-things (IoT) devices, and offline embedded systems. Many recent works focus on alleviating these computational burdens, mainly falling under two non-mutually exclusive categories: manually designing resource-efficient models, and automatically compressing popular architectures. In the latter, increasingly sophisticated techniques have emerged BID3 BID4 BID5, which have achieved respectable accuracy-efficiency operating points, some even Pareto-better than that of the original network; for example, network slimming BID3 reaches an error rate of 6.20% on CIFAR-10 using VGGNet BID9 with a 51% FLOPs reduction-an error decrease of 0.14% over the original. However, to the best of our knowledge, none of the methods impose a FLOPs constraint as part of a single end-to-end optimization objective. MorphNets BID0 apply an L 1 norm, shrinkage-based relaxation of a FLOPs objective, but for the purpose of searching and training multiple models to find good network architectures; in this work, we learn a sparse neural network in a single training run. Other papers directly target device-specific metrics, such as energy usage BID15, but the pruning procedure does not explicitly include the metrics of interest as part of the optimization objective, instead using them as heuristics. Falling short of continuously deploying a model candidate and measuring actual inference time, as in time-consuming neural architectural search BID11, we believe that the number of FLOPs is reasonable to use as a proxy measure for actual latency and energy usage; across variants of the same architecture, Tang et al. suggest that the number of FLOPs is a stronger predictor of energy usage and latency than the number of parameters BID12.Indeed, there are compelling reasons to optimize for the number of FLOPs as part of the training objective: First, it would permit FLOPs-guided compression in a more principled manner. Second, practitioners can directly specify a desired target of FLOPs, which is important in deployment. Thus, our main contribution is to present a novel extension of the prior state of the art BID6 to incorporate the number of FLOPs as part of the optimization objective, furthermore allowing practitioners to set and meet a desired compression target. Formally, we define the FLOPs objective L f lops: f × R m → N 0 as follows: DISPLAYFORM0 where L f lops is the FLOPs associated with hypothesis h(·; θ θ θ):= p(·|θ θ θ), g(·) is a function with the explicit dependencies, and I is the indicator function. We assume L f lops to depend only on whether parameters are non-zero, such as the number of neurons in a neural network. For a dataset D, our empirical risk thus becomes DISPLAYFORM1 Hyperparameters λ f ∈ R + 0 and T ∈ N 0 control the strength of the FLOPs objective and the target, respectively. The second term is a black-box function, whose combinatorial nature prevents gradient-based optimization; thus, using the same procedure in prior art BID6, we relax the objective to a surrogate of the evidence lower bound with a fully-factorized spike-and-slab posterior as the variational distribution, where the addition of the clipped FLOPs objective can be interpreted as a sparsity-inducing prior p(θ θ θ) DISPLAYFORM2 where denotes the Hadamard product. To allow for efficient reparameterization and exact zeros, Louizos et al. BID6 propose to use a hard concrete distribution as the approximation, which is a stretched and clipped version of the binary Concrete distribution BID7: ifẑ ∼ BinaryConcrete(α, β), thenz:= max(0, min(1, (ζ − γ)ẑ + γ)) is said to be a hard concrete r.v., given ζ > 1 and γ < 0. Define φ φ φ:= (α α α, β), and let ψ(φ φ φ) = Sigmoid(log α α α − β log DISPLAYFORM3 ψ(·) is the probability of a gate being non-zero under the hard concrete distribution. It is more efficient in the second expectation to sample from the equivalent Bernoulli parameterization compared to hard concrete, which is more computationally expensive to sample multiple times. The first term now allows for efficient optimization via the reparameterization trick BID2; for the second, we apply the score function estimator (REINFORCE) BID14, since the FLOPs objective is, in general, nondifferentiable and thus precludes the reparameterization trick. High variance is a non-issue because the number of FLOPs is fast to compute, hence letting many samples to be drawn. At inference time, the deterministic estimator isθ θ θ:= θ θ θ max(0, min(1, Sigmoid(log α α α)(ζ − γ) + γ)) for the final parametersθ θ θ. In practice, computational savings are achieved only if the model is sparse across "regular" groups of parameters, e.g., each filter in a convolutional layer. Thus, each computational group uses one hard concrete r.v. BID6 -in fully-connected layers, one per input neuron; in 2D convolution layers, one per output filter. Under convention in the literature where one addition and one multiplication each count as a FLOP, the FLOPs for a 2D convolution layer h conv (·; θ θ θ) given a random draw z is then defined as L f lops (h conv, z) = (K w K h C in + 1)(I w − K w + P w + 1)(I h − K h + P h + 1) z 0 for kernel width and height (K w, K h), input width and height (I w, I h), padding width and height (P w, P h), and number of input channels C in. The number of FLOPs for a fully-connected layer h f c (·; θ θ θ) is L f lops (h f c, z) = (I n + 1) z 0, where I n is the number of input neurons. Note that these are conventional definitions in neural network compression papers-the objective can easily use instead a number of FLOPs incurred by other device-specific algorithms. Thus, at each training step, we compute the FLOPs objective by sampling from the Bernoulli r.v.'s and using the aforementioned definitions, e.g., L f lops (h conv, ·) for convolution layers. Then, we apply the score function estimator to the FLOPs objective as a black-box estimator. We report on MNIST, CIFAR-10, and CIFAR-100, training multiple models on each dataset corresponding to different FLOPs targets. We follow the same initialization and hyperparameters as Louizos et al. BID6, using Adam BID1 with temporal averaging for optimization, a weight decay of 5 × 10 −4, and an initial α that corresponds to the original dropout rate of that layer. We similarly choose β = 2/3, γ = −0.1, and ζ = 1.1. For brevity, we direct the interested reader to their repository BID0 for specifics. In all of our experiments, we replace the original L 0 penalty with our FLOPs objective, and we train all models to 200 epochs; at epoch 190, we prune the network by weights associated with zeroed gates and replace the r.v.'s with their deterministic estimators, then finetune for 10 more epochs. For the score function estimator, we draw 1000 samples at each optimization step-this procedure is fast and has no visible effect on training time. BID10 7-13-208-16 1.1% 254K SBP BID8 3-18-284-283 0.9% 217K BC-GNJ BID5 8-13-88-13 1.0% 290K BC-GHS BID5 5-10-76-16 1.0% 158K L 0 BID6 20-25-45-462 0.9% 1.3M L 0 -sep BID6 9-18-65-25 1.0% 403K DISPLAYFORM0 We choose λ f = 10 −6 in all of the experiments for LeNet-5-Caffe, the Caffe variant of LeNet-5. BID0 We observe that our methods TAB0, bottom three rows) achieve accuracy comparable to those from previous approaches while using fewer FLOPs, with the added benefit of providing a tunable "knob" for adjusting the FLOPs. Note that the convolution layers are the most aggressively compressed, since they are responsible for most of the FLOPs in this model. Orig. in TAB1 denotes the original WRN-28-10 model BID16, and L 0 -* refers to the L 0 -regularized models BID6; likewise, we augment CIFAR-10 and CIFAR-100 with standard random cropping and horizontal flipping. For each of our (last two rows), we report the median error rate of five different runs, executing a total of 20 runs across two models for each of the two datasets; we use λ f = 3 × 10 −9 in all of these experiments. We also report both the expected FLOPs and actual FLOPs, the former denoting the number of FLOPs, on average, at training time under stochastic gates and the latter denoting the number of FLOPs at inference time. We restrict the FLOPs calculations to the penalized non-residual convolution layers only. For CIFAR-10, our approaches in Pareto-better models with decreases in both error rate and the actual number of inference-time FLOPs. For CIFAR-100, we do not achieve a Pareto-better model, since our approach trades accuracy for improved efficiency. The acceptability of the tradeoff depends on the end application.

We extend a state-of-the-art technique to directly incorporate FLOPs as part of the optimization objective, and we show that, given a desired FLOPs requirement, different neural networks are successfully trained.

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Unpaired image-to-image translation among category domains has achieved remarkable success in past decades. Recent studies mainly focus on two challenges. For one thing, such translation is inherently multimodal due to variations of domain-specific information (e.g., the domain of house cat has multiple fine-grained subcategories). For another, existing multimodal approaches have limitations in handling more than two domains, i.e. they have to independently build one model for every pair of domains. To address these problems, we propose the Hierarchical Image-to-image Translation (HIT) method which jointly formulates the multimodal and multi-domain problem in a semantic hierarchy structure, and can further control the uncertainty of multimodal. Specifically, we regard the domain-specific variations as the of the multi-granularity property of domains, and one can control the granularity of the multimodal translation by dividing a domain with large variations into multiple subdomains which capture local and fine-grained variations. With the assumption of Gaussian prior, variations of domains are modeled in a common space such that translations can further be done among multiple domains within one model. To learn such complicated space, we propose to leverage the inclusion relation among domains to constrain distributions of parent and children to be nested. Experiments on several datasets validate the promising and competitive performance against state-of-the-arts. Image-to-image translation is the process of mapping images from one domain to another, during which changing the domain-specific aspect and preserving the domain-irrelevant information. It has wide applications in computer vision and computer graphics;; Zhu et al. (2017a);; such as mapping photographs to edges/segments, colorization, super-resolution, inpainting, attribute and category transfer, style transfer, etc. In this work, we focus on the task of attribute and category transfer, i.e. a set of images sharing the same attribute or category label is defined as a domain 1. Such task has achieved significant development and impressive in terms of image quality in recent years, benefiting from the improvement of generative adversarial nets (GANs);. Representative methods include pix2pix , , CycleGAN Zhu et al. (2017a), , and. More recently the study of this task mainly focus on two challenges. The first is the ability of involving translation among several domains into one model. It is quite a practical need for users. Using most methods, we have to train a separate model for each pair of domains, which is obviously inefficient. To deal with such problem, leverages one generator to transform an image to any domain by taking both the image and the target domain label as conditional input supervised by an auxiliary domain classifier. Another challenge is the multimodal problem, which is early addressed by BicycleGAN Zhu et al. (2017b). Most techniques including aforementioned StarGAN can only give a single determinate output in target domain given an image from source domain. However, for many translation task, the mapping is naturally multimodal. As shown in Fig.1, a cat could have many possible appearances such as being a Husky, a Samoyed or other dogs when translated to the dog domain. To address Figure 1: An illustration of a hierarchy structure and the distribution relationship in a 2D space among categories in such hierarchy. Multi-domain translation is shown in the horizontal direction (blue dashed arrow) while multimodal translation is indicated in the vertical direction (red dashed arrow). Since one child category is a special case of its parent, in the distribution space it is a conditional distribution of its parent, leading to the nested relationship between them. this issue, recent works including BicycleGAN Zhu et al. (2017b), and model a continuous and multivariant distribution independently for each domain to represent the variations of domain-specific information, and they have achieved diverse and high-quality for several two-domain translation tasks. In this paper, we aim at involving the abilities of both multi-domain and multimodal translation into one model. As shown in Fig.1, it is noted that categories have the natural hierarchical relationships. For instance, the cat, dog and bird are three special children of the animal category since they share some common visual attributes. Furthermore, in the dog domain, some samples are named as husky and some of them are called samoyed due to the appearance variations of being the dog. Of course, one can continue to divide the husky to be finer-grained categories based on the variations of certain visual attributes. Such hierarchical relationships widely exist among categories in real world since it is a natural way for our human to understand objects according to our needs in that time. We go back to the image translation task, the multi-domain and multimodal issues can be understood from horizontal and vertical views respectively. From the horizontal view as the blue dashed arrow indicates, multi-domain translation is the transformation in a flat level among categories. From the vertical view as the red dashed arrow indicates, multimodal translation further considers variations within target category, i.e. the multimodal issue is actually due to the multi-granularity property of categories. In the extreme case, every instance is a variation mode of the domain-specific information. Inspired by these observations, we propose a Hierarchical Image-to-image Translation (HIT) method which translates object images among both multiple category domains in a same hierarchy level and their children domains. To this end, our method models the variations of all domains in forms of multiple continuous and multivariant Gaussian distributions in a common space. This is different from previous methods which model the same Gaussian distribution for two domains in independent spaces and thus can not work with only one generator. Due to the hierarchical relationships among domains, distribution of a child domain is the conditional one of its parent domain. Take such principle into consideration, distributions of domains should be nested between a parent and its children, as a 2D illustration shown in Fig.1. To effectively supervise the learning of such distributions space, we further improve the traditional conditional GAN framework to possess the hierarchical discriminability via a hierarchical classifier. Experiments on several categories and attributes datasets validate the competitive performance of HIT against state-of-the-arts. Conditional Generative Adversarial Networks. is probably one of the most creative frameworks recently for the deep learning community. It contains a generator and a discriminator. The generator is trained to fool the discriminator, while the discriminator in turn tries to distinguish the real and generated data. Various GANs have been proposed to improve the training stability, including better network architectures;;;; , more reasonable distribution metrics;; and normalization schemes;. With these improvements, GANs have been applied to many conditional tasks , such as image generation given class labels , super resolution , text2image , 3D reconstruction from 2D input and image-to-image translation introduced in the following. Image-to-image Translation. Pix2pix is the first unified framework for the task of image-to-image translation based on conditional GANs, which combines the adversarial loss with a pixel-level L1 loss and thus requires the pairwise supervision information between two domains. To address this issue, unpaired methods are proposed including , , and CycleGAN Zhu et al. (2017a). UNIT combines the varitional auto-encoder and GAN framework, and proposes to share partial network weights of two domains to learn a common latent space such that corresponding images in two domains can be matched in this space. DiscoGAN, DualGAN and CycleGAN leverage a cycle consistency loss which enforces that we can re-translate the target image back to the original image. More recently, works in this area mainly focus on the problems of multi-domain and multimodal. To deal with translation among several domains in one generator, takes target label and input image as conditions, and uses an auxiliary classifier to classify translated image into its belonged domain. As for the multimodal issue, BicycleGAN Zhu et al. (2017b) proposes to model continuous and multivariant distributions. However, BicycleGAN requires input-output paired annotations. To overcome this problem, and adopt a disentangled representation for learning diverse translation from unpaired training data. propose to interpolate the latent codes between input and referred image to realize generation of diverse images. Different from all aforementioned works, we aim at realizing both multi-domain and multimodal translation in one model using the natural hierarchical relationships among domains defined by category or attribute. Hierarchy-regularized Learning. Hierarchical learning is a natural learning manner for human beings and we often describe objects in the world from abstract to detailed according to our needs of the time. propose to use generative models to disentangle the factors from low-level representations to high-level ones that can construct a specific object. uses an unsupervised generative framework to hierarchically disentangle the , object shape and appearance from an image. In natural language processing, propose a probabilistic word embedding method to capture the semantics described by the WordNet hierarchy. Our method first introduces such semantic hierarchy to learn a both multi-domain and multimodal translation model. 3.1 PROBLEM FORMULATION Let x i ∈ X i be an image from domain i. Our goal is to estimate the conditional probability p(x j |x i) by learning an image-to-image translation model p(x i→j |x i), where x i→j is a sample produced by translating x i to domain X j. Generally speaking, p(x j |x i) are multimodal due to the intra-domain variations. To deal with the multimodal problem, similar to, we assume that x i is disentangled by an encoder E into the content part c ∈ C that is shared by all domains (i.e. domain-irrelevant) and the style part s i ∈ S i that is specific to domain X i (i.e. domain-specific). By modeling S j as a continuous distribution such as a Gaussian N j, x i can be simply translated to domain X j by G(c, s j) where s j is randomly sampled from N j and G is a decoder. We further assume G and E are deterministic and mutually inverse, i.e. E = G −1 and G = E −1. Besides, we assume that c is a high-dimensional feature map while s i is a low-dimensional vector such that the complex spatial structure of objects can be preserved and the style parts could focus more on the relatively small scale but discriminative domain-specific information. Different from, we aim to translate not only between two domains but among multiple ones. To this end, we need to model Gaussians of styles for all domains in a common space (not independently in two spaces like) such that the single decoder G could generate target image based on which Gaussian is sampled. In the multi-domain and multimodal Figure 2: Overview of the whole framework of the proposed method, which mainly consists of five modules: an encoder, a domain distributions modeling module, a decoder, a discriminator and a hierarchical classifier. Given images from different categories, the encoder extracts domain-irrelevant and domain-specific features respectively from the content and style branches. Then the decoder takes them as input to reconstruct the inputs supervised by the reconstruction losses. To realize the multimodal and multi-domain translation, domain distributions are modeled in a common space based on the semantic hierarchy structure and elaborately designed nested loss. Combining the domain-irrelevant features and sampled styles from any distribution, the decoder could translate them to the target domain, guided by the adversarial loss and hierarchical classification loss. settings, it is noted that categories have the hierarchical relationships. As we introduced in Fig.1, multi-domain translation is in the horizontal direction among categories in a particular hierarchy level, and multimodal translation is in the vertical direction since samples can be further divided into multiple child modes. Therefore, distribution of a parent domain covers several conditional distributions, leading to the nested relationship. In this paper, we model all category domains in a given hierarchy structure as nested Gaussian distributions in a common space, realizing Hierarchical Image-to-image Translation (HIT) between any two domains. In such settings, N. The framework is trained with adversarial loss that ensures the translated images approximate the manifold of real images, hierarchical cross-entropy loss that makes the generation conditioned on the sampled domain, nested loss that constrains distributions of domains to satisfy their hierarchical relationships, as well as bidirectional reconstruction losses that ensure enough and meaningful information be encoded. In math, the relation between a parent node u and a child node v in the hierarchy is called partial order relation , defined as v u. In the application of taxonomy, for concept u and v, v u means every instance of category v is also an instance of category u, but not vise versa. We call such partial order on probability densities as the notion of nested (encapsulation called by). Let g and f be the densities of u and v respectively, if v u, then f g, i.e. f is nested in g. Quantitatively measuring the loss violate the nested relation between f and g is not easy. According to the definition of partial order, strictly measuring that can be done as: where {x : f (x) > η} is the set where f is greater than a nonnegative threshold η. Threshold η indicates the nested degree required by us. Small value of η means high requirement for the overlap between f and g to satisfy f g. Eqn. describes how many regions with densities greater than η of f are not nested in those of g. Eqn. is difficult to be computed for most distributions including Gaussians. Inspired by the work in word embedding , we turn to use a thresholded divergence: where D(·||·) is a divergence measure between densities, we use the KL divergence considering its simple formulation for Gaussians. Such loss is a soft measure of violation of the nested relation. If f = g, then D(f ||g) = 0. In case of f g, D(f ||g) would be positive but smaller than a threshold α. As for the effectiveness of using α, please make a reference to. To learn the nested distributions for domains in the hierarchy shown in Fig.2, the penalty described by Eqn. between a positive pair of distributions (N i N j) should be minimized, while that between a negative pair (N i N j) should be greater than a margin m: where P and N denote the numbers of positive and negative pairs respectively. Apart from the proposed nested loss in Eqn., our HIT is equipped with an adversarial loss and a hierarchical classification loss to distinguish which domain the generated images belong to, and two general reconstruction losses applied on both images and features. Adversarial loss. GAN is an effective objective to match the generated images to the real data manifold. The discriminator D tries to classify natural images as real and distinguish generated ones as fake, while the generator G learns to improve image quality to fool D, defined as: Hierarchical classification loss. Similar to , we introduce an auxiliary classifier D cls on top of D and impose the domain classification loss when optimizing G and D, i.e. using real images to train D cls and generated ones to optimize G with such classification loss. Differently, our classifier is hierarchical. In general, the deeper of categories in the hierarchy, the more difficult to distinguish. To alleviate such problem, the loss is cumulative, i.e. classification loss of l-th level is the summation of losses of all levels above l-th with more than two categories. where y k j is the label of domain X j in k-th level. Bidirectional reconstruction loss. To ensure meaningful information encoded and inverse between G and E, we encourage reconstruction of both images and latent features. -Image reconstruction loss: -Feature reconstruction loss: Full objectives. To learn E, G and N l j, we need to optimize the following terms: where λ 1, λ 2 and λ 3 are loss weights of different terms. D is updated with the following losses: and translation works. As shown in Fig.2, we add a distribution modeling module where a pair of mean vector and diagonal covariance matrix of Gaussian for each domain is parameterized to learn. More training details are given in the Appendix. Style adversarial loss. Eqn. and Eqn. match the generated images to the distribution of a target domain. Such loss functions can also be applied on the encoded style codes, i.e. matching s i (act as fake data) of input images to domain Gaussians N l i (act as real data) they belong to. By doing so, it is found that the performance of style transfer between a pair of real images would become better. However, such loss would lead to the training collapse on small scale datasets. Therefore, it is recommended to equip it to our framework on datasets with enough training data. We conduct experiments on hierarchical annotated data from , and. Typical examples are shown in Fig.8, Fig.9 and Fig.10 in Appendix. CelebA provides more than 200K face images with 40 attribute annotations. Following the official train/test protocol and imitating the category hierarchy, we define a hierarchy based on attribute annotations. Specifically, all faces are first clustered into male and female and are further classified according to the age and hair color in the next two levels. Following, we collect images from 3 super domains including house cats, dogs and big cats of ImageNet. Each super domain contains 4 fine-grained categories, which thus construct in a three-level hierarchy (root is animal). All images split by official train/test protocol are processed by a pre-trained faster-rcnn head detector and then cropped as the inputs for translation. ShapeNet is constitutive of 51,300 3D models covering 55 common and 205 finer-grained categories. 12 2D images with different poses are obtained for each 3D model. A three-level hierarchy of furniture containing different kinds of tables and sofas are defined. Ratio of train/test split is 4:1. 4.3 EVALUATION METRICS Frankly speaking, quantitatively evaluating the quality of generated images is not easy. Recent proposed metrics may be fooled by artifacts in some extent. In this paper, we use the Inception Score (IS) and Frchet Inception Distance (FID) to evaluate the semantics of generated images, and leverage the (LPIPS) to measure the semantic diversity of generated visual modes. 4.4 COMPARED BASELINES We mainly compared methods proposed for the objectives of either multi-domain or multimodal translation. Considering the unpaired training settings, the multi-domain method and multimodal method are compared in this paper. Since MUNIT needs to train a model for each pair of domains, it is trained for domain pairs of male/female, young/old and black/golden hair on CelebA, house cat/dog, house cat/big cat and big cat/dog on ImageNet, and sofa/table on ShapeNet, respectively. The average of evaluations on all domain pairs for each dataset is reported. As for StarGAN, it is trained on CelebA as done in its opened source codes. Translations among house cat, dog and big cat domains on ImageNet, and between sofa and table domains on ShapeNet are learned for StarGAN. As comparison, of our HIT in corresponding domain levels for each dataset are reported. 4.5 Table. 1 shows the quantitative comparisons of the proposed HIT with the baselines. Fig.3 shows qualitative on CelebA. It is observed that StarGAN achieves outstanding image quality espe- cially on the fine-grained translations among attribute domains, while the advantages of multimodal methods are generating multiple translations with intra-domain mode variations at the cost of image quality. The image quality of MUNIT is not satisfactory on CelebA both in quantitatively in Table. 1 and in qualitatively in Fig.3. The reasons for this may be that using only the adversarial learning to find fine-grained attribute differences between domains is not stable while multi-domain classifier is good at such task. Besides MUNIT obtains the best diversity performance. It is reasonable as it only involves two domains in one model and equips a triplet of backbone including encoder, decoder and discriminator for each domain. Our method considers both multimodal and multi-domain translation within only one triplet of such backbone, which has high requirement for capacity of networks. It performs in trade-off between image quality and diversity. From Fig.3, artifacts accompanying the generated faces for MUNIT may overestimate the LPIPS on CelebA. Fig.4 and Fig.5 further shows the qualitative of our HIT on ImageNet and ShapeNet datasets respectively. Generally speaking, translation among such categories with large variations is much more challenging than that for face data (several times of increase of the FID in Table. 1 can be found). Even so, our HIT achieves promising qualitative . Besides, using the fixed styles from a particular category distribution (same columns in Fig.4 and Fig.5), the generated images indeed have similar styles of that category and dissimilar content appearances (e.g. pose, expression), demonstrating good disentanglement of content and style of images. Walking in the path towards leaf-level, translated images would have fewer variations with more conditions being specified by the categories in high levels. In other words, distributions in low levels are local modes of its ancestor domains in high levels, leading to the nested relationship. Results in Fig.6 validate the learned distributions of styles in different levels are exactly nested. In the Appendix, we give an experimental parameter-sensitiveness analysis of m and α which constrain the nested relationships among distributions. In Fig.7(a), we further study the smoothness of learned distributions. It is observed one can conduct smooth translation via interpolations between styles from different attribute domains. Besides, with the help of additional style adversarial loss introduced in Sec.4.1, our method can provide users more controlled translation as done in, i.e. use the styles of referenced real images instead of sampling them from distributions. Fig.7(b) shows some example . We can find that the semantics of gender, age and hair color are all correctly transferred to the input images. In this paper we propose the Hierarchical Image-to-image Translation (HIT) method which incorporates multi-domain and multimodal translation into one model. Experiments on three datasets especially on CelebA show that the proposed method can well achieve such granularity controlled translation objectives, i.e. the variation modes of outputs can be specified owe to the nested distributions. However, current work has a limitation, i.e. the assumption of single Gaussian for each category domain. On one hand, though Gaussian distribution prior is a good approximation for many data, it may not be applicable when scale of available training data is small but variations within domain are large such as the used hierarchical data on ImageNet and ShapeNet in this paper. On the other hand, the parent distributions should be mixture of Gaussians given multiple single Gaussians of its children. This issue would lead to sparse sampling around the centers of parent distributions and poor nested if samples are not enough to fulfill the whole space. We have made efforts to the idea of mixture of Gaussians and found that it is hard to compute the KL divergence between two mixture of Gaussians which does not have an analytical solution. Besides, the re-parameterize trick for distribution sampling during SGD optimization can not be transferred to the case of mixture of Gaussians. A better assumption to realize the nested relationships among parent-children distributions is a promising direction for our future research. We use the Adam optimizer with β 1 = 0.5, β 2 = 0.999, and initial learning rate of 0.0001. We train HIT on all datasets for 300K iterations and half decay the learning rate every 100K iterations. We set batch size to 16. The loss weights λ 1, λ 2 andλ 3 in Eqn. are set as 1, 10 and 1 respectively. α and m in Eqn. are empirically set as 50, 200 respectively. Random mirroring is applied during training. In this section, Fig.8, Fig.9 and Fig.10 provide leaf-level examples for better understanding the nested relationships among categories in different hierarchy levels. Take the CelebA for example, the root category face has two children distinguished by gender attribute. For each of the two super categories, it includes two finer granular children which are further divided by the age attribute (young/old). Finally, in the leaf-level, each local branch are classified according to their hair colors, i.e. black, golden and brown hair. Within each leaf-level category, samples mainly contain intraclass variations caused by identities, expressions, poses, etc. The impacts of hyper-parameters in the nested distribution learning on word embedding task have been studied in. In this section, we further make an analysis of them in current image generation task. Fig.11 and Fig.12 show the impacts of hyper-parameters m and α in the nested loss of Eqn.. It is observed that distribution margin m has larger impact than nested threshold α. With too large settings of m, distributions which do not have nested relationship would be pushed far away, leading to sparse space. Sampling in such space would make the learning of generator quite difficult. In contrast, with too small settings of m, the discriminabilities of distributions may be poor. Therefore, a trade-off value 200 is set for m in this paper. As for nested threshold α, a relative small or large value performs well in terms of the image quality metric. However, in theory, large value setting of α would relax the nested constraint too much, in small overlap between parent and children distributions. Therefore, we recommend to set α in the left half axis of α. When α is set as 0, it means the parent and children distributions are all overlapped, which is too strict to learn. Finally, we set α as 50, and the ratio of 1:4 between α and m is consistent with the observation in.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Granularity controled multi-domain and multimodal image to image translation method

represent this text to find a much much longer article about it with say hundreds-thousands of words

This paper proposes a deep generative classifier which is effective to detect out-of-distribution samples as well as classify in-distribution samples, by integrating the concept of Gaussian discriminant analysis into deep neural networks.


Question: represent the article for finding a single-sentence tldr

Question: Recurrent Neural Networks (RNNs) are very successful at solving challenging problems with sequential data. However, this observed efficiency is not yet entirely explained by theory. It is known that a certain class of multiplicative RNNs enjoys the property of depth efficiency --- a shallow network of exponentially large width is necessary to realize the same score function as computed by such an RNN. Such networks, however, are not very often applied to real life tasks. In this work, we attempt to reduce the gap between theory and practice by extending the theoretical analysis to RNNs which employ various nonlinearities, such as Rectified Linear Unit (ReLU), and show that they also benefit from properties of universality and depth efficiency. Our theoretical are verified by a series of extensive computational experiments. Recurrent Neural Networks are firmly established to be one of the best deep learning techniques when the task at hand requires processing sequential data, such as text, audio, or video BID10 BID18 BID7. The ability of these neural networks to efficiently represent a rich class of functions with a relatively small number of parameters is often referred to as depth efficiency, and the theory behind this phenomenon is not yet fully understood. A recent line of work BID12 BID5 focuses on comparing various deep learning architectures in terms of their expressive power. It was shown in that ConvNets with product pooling are exponentially more expressive than shallow networks, that is there exist functions realized by ConvNets which require an exponentially large number of parameters in order to be realized by shallow nets. A similar also holds for RNNs with multiplicative recurrent cells BID12. We aim to extend this analysis to RNNs with rectifier nonlinearities which are often used in practice. The main challenge of such analysis is that the tools used for analyzing multiplicative networks, namely, properties of standard tensor decompositions and ideas from algebraic geometry, can not be applied in this case, and thus some other approach is required. Our objective is to apply the machinery of generalized tensor decompositions, and show universality and existence of depth efficiency in such RNNs. Tensor methods have a rich history of successful application in machine learning. BID21, in their framework of TensorFaces, proposed to treat facial image data as multidimensional arrays and analyze them with tensor decompositions, which led to significant boost in face recognition accuracy. BID0 ) employed higher-order co-occurence data and tensor factorization techniques to improve on word embeddings models. Tensor methods also allow to produce more accurate and robust recommender systems by taking into account a multifaceted nature of real environments BID6.In recent years a great deal of work was done in applications of tensor calculus to both theoretical and practical aspects of deep learning algorithms. BID15 represented filters in a convolutional network with CP decomposition BID11 BID1 which allowed for much faster inference at the cost of a negligible drop in performance. BID19 proposed to use Tensor Train (TT) decomposition BID20 to compress fully-connected layers of large neural networks while preserving their expressive power. Later on, TT was exploited to reduce the number of parameters and improve the performance of recurrent networks in long-term forecasting BID24 and video classification BID23 problems. In addition to the practical benefits, tensor decompositions were used to analyze theoretical aspects of deep neural nets. ) investigated a connection between various network architectures and tensor decompositions, which made possible to compare their expressive power. Specifically, it was shown that CP and Hierarchial Tucker BID9 decompositions correspond to shallow networks and convolutional networks respectively. Recently, this analysis was extended by BID12 who showed that TT decomposition can be represented as a recurrent network with multiplicative connections. This specific form of RNNs was also empirically proved to provide a substantial performance boost over standard RNN models BID22.First on the connection between tensor decompositions and neural networks were obtained for rather simple architectures, however, later on, they were extended in order to analyze more practical deep neural nets. It was shown that theoretical can be generalized to a large class of CNNs with ReLU nonlinearities and dilated convolutions BID5, providing valuable insights on how they can be improved. However, there is a missing piece in the whole picture as theoretical properties of more complex nonlinear RNNs have yet to be analyzed. In this paper, we elaborate on this problem and present new tools for conducting a theoretical analysis of such RNNs, specifically when rectifier nonlinearities are used. Let us now recall the known about the connection of tensor decompositions and multiplicative architectures, and then show how they are generalized in order to include networks with ReLU nonlinearities. Suppose that we are given a dataset of objects with a sequential structure, i.e. every object in the dataset can be written as DISPLAYFORM0 We also introduce a parametric feature map f θ: R N → R M which essentially preprocesses the data before it is fed into the network. Assumption 1 holds for many types of data, e.g. in the case of natural images we can cut them into rectangular patches which are then arranged into vectors x (t). A typical choice for the feature map f θ in this particular case is an affine map followed by a nonlinear activation: f θ (x) = σ(Ax + b). To draw the connection between tensor decompositions and feature tensors we consider the following score functions (logits 1): DISPLAYFORM1 where W ∈ R M ×M ×···×M is a trainable T -way weight tensor and Φ(X) ∈ R M ×M ×···×M is a rank 1 feature tensor, defined as DISPLAYFORM2 where we have used the operation of outer product ⊗, which is important in tensor calculus. For a tensor A of order N and a tensor B of order M their outer product C = A ⊗ B is a tensor of order N + M defined as: DISPLAYFORM3 It is known that equation 2 possesses the universal approximation property (it can approximate any function with any prescribed precision given sufficiently large M) under mild assumptions on f θ BID8 ). Working the entire weight tensor W in eq. FORMULA1 is impractical for large M and T, since it requires exponential in T number of parameters. Thus, we compactly represent it using tensor decompositions, which will further lead to different neural network architectures, referred to as tensor networks BID2. The most basic decomposition is the so-called Canonical (CP) decomposition BID11 BID1 which is defined as follows DISPLAYFORM0 where v (t) r ∈ R M and minimal value of R such that decomposition equation 5 exists is called canonical rank of a tensor (CP-rank). By substituting eq. into eq. we find that DISPLAYFORM1 In the equation above, outer products ⊗ are taken between scalars and coincide with the ordinary products between two numbers. However, we would like to keep this notation as it will come in handy later, when we generalize tensor decompositions to include various nonlinearities. TT-decomposition Another tensor decomposition is Tensor Train (TT) decomposition which is defined as follows DISPLAYFORM2 where g (t)rt−1rt ∈ R M and r 0 = r T = 1 by definition. If we gather vectors g (t)rt−1rt for all corresponding indices r t−1 ∈ {1, . . ., R t−1} and r t ∈ {1, . . ., R t} we will obtain three-dimensional tensors G (t) ∈ R M ×Rt−1×Rt (for t = 1 and t = T we will get matrices DISPLAYFORM3 is called TT-cores and minimal values of {R t} T −1 t=1 such that decomposition equation 7 exists are called TT-ranks. In the case of TT decomposition, the score function has the following form: DISPLAYFORM4 Now we want to show that the score function for Tensor Train decomposition exhibits particular recurrent structure similar to that of RNN. We define the following hidden states: DISPLAYFORM0 r0r1, DISPLAYFORM1 Such definition of hidden states allows for more compact form of the score function. Lemma 3.1. Under the notation introduced in eq., the score function can be written as DISPLAYFORM2 Proof of Lemma 3.1 as well as the proofs of our main from Section 5 were moved to Appendix A due to limited space. Note that with a help of TT-cores we can rewrite eq. in a more convenient index form: DISPLAYFORM3 where the operation of tensor contraction is used. Combining all weights from G (t) and f θ (·) into a single variable Θ (t)G and denoting the composition of feature map, outer product, and contraction as g: DISPLAYFORM4 Rt we arrive at the following vector form: G depend on the time step. However, if we set DISPLAYFORM5 DISPLAYFORM6 we will get simplified hidden state equation used in standard recurrent architectures: DISPLAYFORM7 Note that this equation is applicable to all hidden states except for the first DISPLAYFORM8 and for the last DISPLAYFORM9, due to two-dimensional nature of the corresponding TT-cores. However, we can always pad the input sequence with two auxiliary vectors x and x (T +1) to get full compliance with the standard RNN structure. In the previous section we showed that tensor decompositions correspond to neural networks of specific structure, which are simplified versions of those used in practice as they contain multiplicative nonlinearities only. One possible way to introduce more practical nonlinearities is to replace outer product ⊗ in eq. FORMULA5 and eq. with a generalized operator ⊗ ξ in analogy to kernel methods when scalar product is replaced by nonlinear kernel function. Let ξ: R × R → R be an associative and commutative binary operator (∀x, y, z ∈ R : ξ(ξ(x, y), z) = ξ(x, ξ(y, z)) and ∀x, y ∈ R: ξ(x, y) = ξ(y, x)). Note that this operator easily generalizes to the arbitrary number of operands due to associativity. For a tensor A of order N and a tensor B of order M we define their generalized outer product C = A ⊗ ξ B as an (N + M) order tensor with entries given by: DISPLAYFORM0 Now we can replace ⊗ in eqs. FORMULA5 and FORMULA0 with ⊗ ξ and get networks with various nonlinearities. For example, if we take ξ(x, y) = max(x, y, 0) we will get an RNN with rectifier nonlinearities; if we take ξ(x, y) = ln(e x + e y) we will get an RNN with softplus nonlinearities; if we take ξ(x, y) = xy we will get a simple RNN defined in the previous section. Concretely, we will analyze the following networks. • Score function: DISPLAYFORM0 • Parameters of the network: DISPLAYFORM1 Generalized RNN with ξ-nonlinearity• Score function: DISPLAYFORM2 • Parameters of the network: DISPLAYFORM3 Note that in eq. FORMULA0 we have introduced the matrices C (t) acting on the input states. The purpose of this modification is to obtain the plausible property of generalized shallow networks being able to be represented as generalized RNNs of width 1 (i.e., with all R i = 1) for an arbitrary nonlinearity ξ. In the case of ξ(x, y) = xy, the matrices C (t) were not necessary, since they can be simply absorbed by G (t) via tensor contraction (see Appendix A for further clarification on these points).Initial hidden state Note that generalized RNNs require some choice of the initial hidden state h. We find that it is convenient both for theoretical analysis and in practice to initialize h as unit of the operator ξ, i.e. such an element u that ξ(x, y, u) = ξ(x, y) ∀x, y ∈ R. Henceforth, we will assume that such an element exists (e.g., for ξ(x, y) = max(x, y, 0) we take u = 0, for ξ(x, y) = xy we take u = 1), and set h = u. For example, in eq. FORMULA10 it was implicitly assumed that h = 1. Introduction of generalized outer product allows us to investigate RNNs with wide class of nonlinear activation functions, especially ReLU. While this change looks appealing from the practical viewpoint, it complicates following theoretical analysis, as the transition from obtained networks back to tensors is not straightforward. In the discussion above, every tensor network had corresponding weight tensor W and we could compare expressivity of associated score functions by comparing some properties of this tensors, such as ranks BID12. This method enabled comprehensive analysis of score functions, as it allows us to calculate and compare their values for all possible input sequences X = x,..., x (T). Unfortunately, we can not apply it in case of generalized tensor networks, as the replacement of standard outer product ⊗ with its generalized version ⊗ ξ leads to the loss of conformity between tensor networks and weight tensors. Specifically, not for every generalized tensor network with corresponding score function (X) now exists a weight tensor W such that (X) = W, Φ(X). Also, such properties as universality no longer hold automatically and we have to prove them separately. Indeed as it was noticed in shallow networks with ξ(x, y) = max(x, 0) + max(y, 0) no longer have the universal approximation property. In order to conduct proper theoretical analysis, we adopt the apparatus of so-called grid tensors, first introduced in.Given a set of fixed vectors X = x,..., x (M) referred to as templates, the grid tensor of X is defined to be the tensor of order T and dimension M in each mode, with entries given by: DISPLAYFORM0 where each index i t can take values from {1, . . ., M}, i.e. we evaluate the score function on every possible input assembled from the template vectors {x DISPLAYFORM1 . To put it simply, we previously considered the equality of score functions represented by tensor decomposition and tensor network on set of all possible input sequences X = x,..., x (T), x (t) ∈ R N, and now we restricted this set to exponentially large but finite grid of sequences consisting of template vectors only. Define the matrix F ∈ R M ×M which holds the values taken by the representation function f θ: R N → R M on the selected templates X: DISPLAYFORM2 Using the matrix F we note that the grid tensor of generalized shallow network has the following form (see Appendix A for derivation): DISPLAYFORM3 Construction of the grid tensor for generalized RNN is a bit more involved. We find that its grid tensor Γ (X) can be computed recursively, similar to the hidden state in the case of a single input sequence. The exact formulas turned out to be rather cumbersome and we moved them to Appendix A. With grid tensors at hand we are ready to compare the expressive power of generalized RNNs and generalized shallow networks. In the further analysis, we will assume that ξ(x, y) = max(x, y, 0), i.e., we analyze RNNs and shallow networks with rectifier nonlinearity. However, we need to make two additional assumptions. First of all, similarly to we fix some templates X such that values of the score function outside of the grid generated by X are irrelevant for classification and call them covering templates. It was argued that for image data values of M of order 100 are sufficient (corresponding covering template vectors may represent Gabor filters). Secondly, we assume that the feature matrix F is invertible, which is a reasonable assumption and in the case of f θ (x) = σ(Ax + b) for any distinct template vectors X the parameters A and b can be chosen in such a way that the matrix F is invertible. As was discussed in section 4.2 we can no longer use standard algebraic techniques to verify universality of tensor based networks. Thus, our first states that generalized RNNs with ξ(x, y) = max(x, y, 0) are universal in a sense that any tensor of order T and size of each mode being m can be realized as a grid tensor of such RNN (and similarly of a generalized shallow network).Theorem 5.1 (Universality). Let H ∈ R M ×M ×···×M be an arbitrary tensor of order T. Then there exist a generalized shallow network and a generalized RNN with rectifier nonlinearity ξ(x, y) = max(x, y, 0) such that grid tensor of each of the networks coincides with H.Part of Theorem 5.1 which corresponds to generalized shallow networks readily follows from (, Claim 4). In order to prove the statement for the RNNs the following two lemmas are used. Lemma 5.1. Given two generalized RNNs with grid tensors Γ A (X), Γ B (X), and arbitrary ξ-nonlinearity, there exists a generalized RNN with grid tensor Γ C (X) satisfying DISPLAYFORM0 This lemma essentially states that the collection of grid tensors of generalized RNNs with any nonlinearity is closed under taking arbitrary linear combinations. Note that the same clearly holds for generalized shallow networks because they are linear combinations of rank 1 shallow networks by definition. Lemma 5.2. Let E (j1j2...j T) be an arbitrary one-hot tensor, defined as DISPLAYFORM1 Then there exists a generalized RNN with rectifier nonlinearities such that its grid tensor satisfies DISPLAYFORM2 This lemma states that in the special case of rectifier nonlinearity ξ(x, y) = max(x, y, 0) any basis tensor can be realized by some generalized RNN.Proof of Theorem 5.1. By Lemma 5.2 for each one-hot tensor E (i1i2...i T) there exists a generalized RNN with rectifier nonlinearities, such that its grid tensor coincides with this tensor. Thus, by Lemma 5.1 we can construct an RNN with DISPLAYFORM3 For generalized shallow networks with rectifier nonlinearities see the proof of, Claim 4).The same regarding networks with product nonlinearities considered in BID12 directly follows from the well-known properties of tensor decompositions (see Appendix A).We see that at least with such nonlinearities as ξ(x, y) = max(x, y, 0) and ξ(x, y) = xy all the networks under consideration are universal and can represent any possible grid tensor. Now let us head to a discussion of expressivity of these networks. As was discussed in the introduction, expressivity refers to the ability of some class of networks to represent the same functions as some other class much more compactly. In our case the parameters defining size of networks are ranks of the decomposition, i.e. in the case of generalized RNNs ranks determine the size of the hidden state, and in the case of generalized shallow networks rank determines the width of a network. It was proven in BID12 that ConvNets and RNNs with multiplicative nonlinearities are exponentially more expressive than the equivalent shallow networks: shallow networks of exponentially large width are required to realize the same score functions as computed by these deep architectures. Similarly to the case of ConvNets, we find that expressivity of generalized RNNs with rectifier nonlinearity holds only partially, as discussed in the following two theorems. For simplicity, we assume that T is even. Theorem 5.2 (Expressivity 1). For every value of R there exists a generalized RNN with ranks ≤ R and rectifier nonlinearity which is exponentially more efficient than shallow networks, i.e., the corresponding grid tensor may be realized only by a shallow network with rectifier nonlinearity of width at least DISPLAYFORM0 This states that at least for some subset of generalized RNNs expressivity holds: exponentially wide shallow networks are required to realize the same grid tensor. Proof of the theorem is rather straightforward: we explicitly construct an example of such RNN which satisfies the following description. Given an arbitrary input sequence X = x,... x (T) assembled from the templates, these net- DISPLAYFORM1, and 1 in every other case, i.e. they measure pairwise similarity of the input vectors. A precise proof is given in Appendix A. In the case of multiplicative RNNs BID12 almost every network possessed this property. This is not the case, however, for generalized RNNs with rectifier nonlinearities. In other words, for every rank R we can find a set of generalized RNNs of positive measure such that the property of expressivity does not hold. In the numerical experiments in Section 6 and Appendix A we validate whether this can be observed in practice, and find that the probability of obtaining CP-ranks of polynomial size becomes negligible with large T and R. Proof of Theorem 5.3 is provided in Appendix A.Shared case Note that all the RNNs used in practice have shared weights, which allows them to process sequences of arbitrary length. So far in the analysis we have not made such assumptions about RNNs (i.e., G = · · · = G (T −1) ). By imposing this constraint, we lose the property of universality; however, we believe that the statements of Theorems 5.2 and 5.3 still hold (without requiring that shallow networks also have shared weights). Note that the example constructed in the proof of Theorem 5.3 already has this property, and for Theorem 5.2 we provide numerical evidence in Appendix A. In this section, we study if our theoretical findings are supported by experimental data. In particular, we investigate whether generalized tensor networks can be used in practical settings, especially in problems typically solved by RNNs (such as natural language processing problems). Secondly, according to Theorem 5.3 for some subset of RNNs the equivalent shallow network may have a low rank. To get a grasp of how strong this effect might be in practice we numerically compute an estimate for this rank in various settings. Performance For the first experiment, we use two computer vision datasets MNIST BID16 and CIFAR-10 , and natural language processing dataset for sentiment analysis IMDB BID17. For the first two datasets, we cut natural images into rectangular patches which are then arranged into vectors x (t) (similar to BID12) and for IMDB dataset the input data already has the desired sequential structure. Figure 2 depicts test accuracy on IMDB dataset for generalized shallow networks and RNNs with rectifier nonlinearity. We see that generalized shallow network of much higher rank is required to get the level of performance close to that achievable by generalized RNN. Due to limited space, we have moved the of the experiments on the visual datasets to Appendix B. Expressivity For the second experiment we generate a number of generalized RNNs with different values of TT-rank r and calculate a lower bound on the rank of shallow network necessary to realize the same grid tensor (to estimate the rank we use the same technique as in the proof of Theorem 5.2). FIG4 shows that for different values of R and generalized RNNs of the corresponding rank there exist shallow networks of rank 1 realizing the same grid tensor, which agrees well with Theorem 5.3. This looks discouraging, however, there is also a positive observation. While increasing rank of generalized RNNs, more and more corresponding shallow networks will necessarily have exponentially higher rank. In practice we usually deal with RNNs of R = 10 2 − 10 3 (dimension of hidden states), thus we may expect that effectively any function besides negligible set realized by generalized RNNs can be implemented only by exponentially wider shallow networks. The numerical for the case of shared cores and other nonlinearities are given in Appendix B. In this paper, we sought a more complete picture of the connection between Recurrent Neural Networks and Tensor Train decomposition, one that involves various nonlinearities applied to hidden states. We showed how these nonlinearities could be incorporated into network architectures and provided complete theoretical analysis on the particular case of rectifier nonlinearity, elaborating on points of generality and expressive power. We believe our will be useful to advance theoretical understanding of RNNs. In future work, we would like to extend the theoretical analysis to most competitive in practice architectures for processing sequential data such as LSTMs and attention mechanisms. A PROOFS Lemma 3.1. Under the notation introduced in eq., the score function can be written as DISPLAYFORM0 Proof. DISPLAYFORM1 rt−1rt h r1 DISPLAYFORM2 r1r2 h r1 DISPLAYFORM3 =... DISPLAYFORM4 Proposition A.1. If we replace the generalized outer product ⊗ ξ in eq. with the standard outer product ⊗, we can subsume matrices C (t) into tensors G (t) without loss of generality. Proof. Let us rewrite hidden state equation eq. after transition from ⊗ ξ to ⊗: DISPLAYFORM5 We see that the obtained expression resembles those presented in eq. with TT-cores G (t) replaced byG (t) and thus all the reasoning applied in the absence of matrices C (t) holds valid. Proposition A.2. Grid tensor of generalized shallow network has the following form (eq.): DISPLAYFORM6 denote an arbitrary sequence of templates. Corresponding element of the grid tensor defined in eq. FORMULA1 has the following form: DISPLAYFORM7 Proposition A.3. Grid tensor of a generalized RNN has the following form: DISPLAYFORM8 Proof. Proof is similar to that of Proposition A.2 and uses eq. FORMULA0 to compute the elements of the grid tensor. Lemma 5.1. Given two generalized RNNs with grid tensors Γ A (X), Γ B (X), and arbitrary ξ-nonlinearity, there exists a generalized RNN with grid tensor Γ C (X) satisfying DISPLAYFORM9 Proof. Let these RNNs be defined by the weight parameters DISPLAYFORM10 and DISPLAYFORM11 We claim that the desired grid tensor is given by the RNN with the following weight settings. DISPLAYFORM12 It is straightforward to verify that the network defined by these weights possesses the following property: DISPLAYFORM13, 0 < t < T, and h DISPLAYFORM14 B, concluding the proof. We also note that these formulas generalize the well-known formulas for addition of two tensors in the Tensor Train format .Proposition A.4. For any associative and commutative binary operator ξ, an arbitrary generalized rank 1 shallow network with ξ-nonlinearity can be represented in a form of generalized RNN with unit ranks (R 1 = · · · = R T −1 = 1) and ξ-nonlinearity. DISPLAYFORM15 be the parameters specifying the given generalized shallow network. Then the following weight settings provide the equivalent generalized RNN (with h being the unity of the operator ξ). DISPLAYFORM16 Indeed, in the notation defined above, hidden states of generalized RNN have the following form:Theorem 5.3 (Expressivity 2). For every value of R there exists an open set (which thus has positive measure) of generalized RNNs with rectifier nonlinearity ξ(x, y) = max(x, y, 0), such that for each RNN in this open set the corresponding grid tensor can be realized by a rank 1 shallow network with rectifier nonlinearity. Proof. As before, let us denote by I (p,q) a matrix of size p × q such that I (p,q) ij = δ ij, and by a (p1,p2,...p d) we denote a tensor of size p 1 × · · · × p d with each entry being a (sometimes we will omit the dimensions when they can be inferred from the context). Consider the following weight settings for a generalized RNN. DISPLAYFORM17 The RNN defined by these weights has the property that Γ (X) is a constant tensor with each entry being 2(M R) T −1, which can be trivially represented by a rank 1 generalized shallow network. We will show that this property holds under a small perturbation of C (t), G (t) and F. Let us denote each of these perturbation (and every tensor appearing size of which can be assumed indefinitely small) collectively by ε. Applying eq. FORMULA0 we obtain (with ξ(x, y) = max(x, y, 0)). where we have used a simple property connecting ⊗ ξ with ξ(x, y) = max(x, y, 0) and ordinary ⊗: if for tensors A and B each entry of A is greater than each entry of B, A ⊗ ξ B = A ⊗ 1. The obtained grid tensors can be represented using rank 1 generalized shallow networks with the following weight settings. λ = 1, DISPLAYFORM18 DISPLAYFORM19 ε (2(MR) T−1 + ε), t = 1, 0, t > 1, where F ε is the feature matrix of the corresponding perturbed network. In this section we provide the additional computational experiments, aimed to provide more thorough and complete analysis of generalized RNNs. Different ξ-nonlinearities In this paper we presented theoretical analysis of rectifier nonlinearity which corresponds to ξ(x, y) = max(x, y, 0). However, there is a number of other associative binary operators ξ which can be incorporated in generalized tensor networks. Strictly speaking, every one of them has to be carefully explored theoretically in order to speak about their generality and expressive power, but for now we can compare them empirically. Table 1 shows the performance (accuracy on test data) of different nonlinearities on MNIST, CIFAR-10, and IMDB datasets for classification. Although these problems are not considered hard to solve, we see that the right choice of nonlinearity can lead to a significant boost in performance. For the experiments on the visual datasets we used T = 16, m = 32, R = 64 and for the experiments on the IMDB dataset we had T = 100, m = 50, R = 50. Parameters of all networks were optimized using Adam (learning rate α = 10 −4) and batch size 250. Expressivity in the case of shared cores We repeat the expressivity experiments from Section 6 in the case of equal TT-cores (G = · · · = G (T −1) ). We observe that similar to the case of different cores, there always exist rank 1 generalized shallow networks which realize the same score function as generalized RNN of higher rank, however, this situation seems too unlikely for big values of R. Figure 4: Distribution of lower bounds on the rank of generalized shallow networks equivalent to randomly generated generalized RNNs of ranks (M = 6, T = 6, ξ(x, y) = max(x, y, 0)).Figure 5: Distribution of lower bounds on the rank of generalized shallow networks equivalent to randomly generated generalized RNNs of ranks (M = 6, T = 6, ξ(x, y) = x 2 + y 2 ).

Represent the TLDR for finding the corresponding scientific article

Analysis of expressivity and generality of recurrent neural networks with ReLu nonlinearities using Tensor-Train decomposition.

Represent the short summary for finding the corresponding scientific article (long)

we define the filter-level pruning problem for binary neural networks for the first time and propose method to solve it.


Question: represent the scientific article to find a short tldr

Question: While deep neural networks have proven to be a powerful tool for many recognition and classification tasks, their stability properties are still not well understood. In the past, image classifiers have been shown to be vulnerable to so-called adversarial attacks, which are created by additively perturbing the correctly classified image. In this paper, we propose the ADef algorithm to construct a different kind of adversarial attack created by iteratively applying small deformations to the image, found through a gradient descent step. We demonstrate our on MNIST with convolutional neural networks and on ImageNet with Inception-v3 and ResNet-101. In a first observation in BID25 it was found that deep neural networks exhibit unstable behavior to small perturbations in the input. For the task of image classification this means that two visually indistinguishable images may have very different outputs, ing in one of them being misclassified even if the other one is correctly classified with high confidence. Since then, a lot of research has been done to investigate this issue through the construction of adversarial examples: given a correctly classified image x, we look for an image y which is visually indistinguishable from x but is misclassified by the network. Typically, the image y is constructed as y = x + r, where r is an adversarial perturbation that is supposed to be small in a suitable sense (normally, with respect to an p norm). Several algorithms have been developed to construct adversarial perturbations, see BID9 BID18; BID14; BID16; BID5 and the review paper BID0.Even though such pathological cases are very unlikely to occur in practice, their existence is relevant since malicious attackers may exploit this drawback to fool classifiers or other automatic systems. Further, adversarial perturbations may be constructed in a black-box setting (i.e., without knowing the architecture of the DNN but only its outputs) BID19 BID17 and also in the physical world BID14 BID1 BID3 BID24. This has motivated the investigation of defenses, i.e., how to make the network invulnerable to such attacks, see BID13; BID4; BID16; BID27; BID28; BID20; BID2; BID12. In most cases, adversarial examples are artificially created and then used to retrain the network, which becomes more stable under these types of perturbations. Most of the work on the construction of adversarial examples and on the design of defense strategies has been conducted in the context of small perturbations r measured in the ∞ norm. However, this is not necessarily a good measure of image similarity: e.g., for two translated images x and y, the norm of x−y is not small in general, even though x and y will look indistinguishable if the translation is small. Several papers have investigated the construction of adversarial perturbations not designed for norm proximity BID21 BID24 BID3 BID6 BID29.In this work, we build up on these ideas and investigate the construction of adversarial deformations. In other words, the misclassified image y is not constructed as an additive perturbation y = x + r, but as a deformation y = x • (id + τ), where τ is a vector field defining the transformation. In this case, the similarity is not measured through a norm of y − x, but instead through a norm of τ, which quantifies the deformation between y and x. We develop an efficient algorithm for the construction of adversarial deformations, which we call ADef. It is based on the main ideas of DeepFool BID18, and iteratively constructs the smallest deformation to misclassify the image. We test the procedure on MNIST (LeCun) (with convolutional neural networks) and on ImageNet (with Inception-v3 BID26 and ResNet-101 BID10). The show that ADef can succesfully fool the classifiers in the vast majority of cases (around 99%) by using very small and imperceptible deformations. We also test our adversarial attacks on adversarially trained networks for MNIST. Our implementation of the algorithm can be found at https://gitlab.math. ethz.ch/tandrig/ADef.The of this work have initially appeared in the master's thesis BID8, to which we refer for additional details on the mathematical aspects of this construction. While writing this paper, we have come across BID29, in which a similar problem is considered and solved with a different algorithm. Whereas in BID29 the authors use a second order solver to find a deforming vector field, we show how a first order method can be formulated efficiently and justify a smoothing operation, independent of the optimization step. We report, for the first time, success rates for adversarial attacks with deformations on ImageNet. The topic of deformations has also come up in BID11, in which the authors introduce a class of learnable modules that deform inputs in order to increase the performance of existing DNNs, and BID7, in which the authors introduce a method to measure the invariance of classifiers to geometric transformations. Let K be a classifier of images consisting of P pixels into L ≥ 2 categories, i.e. a function from the space of images X = R cP, where c = 1 (for grayscale images) or c = 3 (for color images), and into the set of labels L = {1, . . ., L}. Suppose x ∈ X is an image that is correctly classified by K and suppose y ∈ X is another image that is imperceptible from x and such that K(y) = K(x), then y is said to be an adversarial example. The meaning of imperceptibility varies, but generally, proximity in p -norm (with 1 ≤ p ≤ ∞) is considered to be a sufficient substitute. Thus, an adversarial perturbation for an image x ∈ X is a vector r ∈ X such that K(x + r) = K(x) and r p is small, where DISPLAYFORM0 Given such a classifier K and an image x, an adversary may attempt to find an adversarial example y by minimizing x − y p subject to K(y) = K(x), or even subject to K(y) = k for some target label k = K(x). Different methods for finding minimal adversarial perturbations have been proposed, most notably FGSM BID9 and PGD BID16 for ∞, and the DeepFool algorithm BID18 DISPLAYFORM1 extending ξ by zero outside of 2. Deformations capture many natural image transformations. For example, a translation of the image ξ by a vector v ∈ R 2 is a deformation with respect to the constant vector field τ = v. If v is small, the images ξ and ξ v may look similar, but the corresponding perturbation ρ = ξ v − ξ may be arbitrarily large in the aforementioned L p -norms. Figure 1 shows three minor deformations, all of which yield large L ∞ -norms. In the discrete setting, deformations are implemented as follows. We consider square images of W × W pixels and define the space of images to be DISPLAYFORM2 In what follows we will only consider the set T of vector fields that do not move points on the grid {1, . . ., W} 2 outside of [1, W] 2. More precisely, DISPLAYFORM3 An image x ∈ X can be viewed as the collection of values of a function ξ: DISPLAYFORM4 for s, t = 1,..., W. Such a function ξ can be computed by interpolating from x. Thus, the deformation of an image x with respect to the discrete vector field τ can be defined as the discrete deformed image x τ in X by DISPLAYFORM5 It is not straightforward to measure the size of a deformation such that it captures the visual difference between the original image x and its deformed counterpart x τ. We will use the size of the corresponding vector field, τ, in the norm defined by DISPLAYFORM6 as a proxy. The p -norms defined in, adapted to vector fields, can be used as well. (We remark, however, that none of these norms define a distance between x and x τ, since two vector fields τ, σ ∈ T with τ T = σ T may produce the same deformed image x τ = x σ .) We will now describe our procedure for finding deformations that will lead a classifier to yield an output different from the original label. DISPLAYFORM0 Let x ∈ X be the image of interest and fix ξ: 2 → R c obtained by interpolation from x. Let l = K(x) denote the true label of x, let k ∈ L be a target label and set f = F k − F l. We assume that x does not lie on a decision boundary, so that we have f (x) < 0.We define the function g: DISPLAYFORM1 We can use a linear approximation of g around the zero vector field as a guide: DISPLAYFORM2 for small enough τ ∈ T and D 0 g: T → R the derivative of g at τ = 0. Hence, if τ is a vector field such that DISPLAYFORM3 and τ T is small, then the classifier K has approximately equal confidence for the deformed image x τ to have either label l or k. This is a scalar equation with unknown in T, and so has infinitely many solutions. In order to select τ with small norm, we solve it in the least-squares sense. In view of, we have DISPLAYFORM4. Thus, by applying the chain rule to g(τ) = f (x τ), we obtain that its derivative at τ = 0 can, with a slight abuse of notation, be identified with the vector field DISPLAYFORM5 where ∇f (x) s,t ∈ R 1×c is the derivative of f in x calculated at (s, t). With this, DISPLAYFORM6, and the solution to in the least-square sense is given by DISPLAYFORM7 Finally, we define the deformed image x τ ∈ X according to.One might like to impose some degree of smoothness on the deforming vector field. In fact, it suffices to search in the range of a smoothing operator S: T → T. However, this essentially amounts to applying S to the solution from the larger search space DISPLAYFORM8 where S denotes the componentwise application of a two-dimensional Gaussian filter ϕ (of any standard deviation). Then the vector field DISPLAYFORM9 also satisfies, since S is self-adjoint. We can hence replace τ byτ to obtain a smooth deformation of the image x. We iterate the deformation process until the deformed image is misclassified. More explicitly, let x = x and for n ≥ 1 let τ (n) be given by for x (n−1). Then we can define the iteration as DISPLAYFORM10 ). The algorithm terminates and outputs an adversarial example y = x DISPLAYFORM11 The iteration also terminates if x (n) lies on a decision boundary of K, in which case we propose to introduce an overshoot factor 1 + η on the total deforming vector field. Provided that the number of iterations is moderate, the total vector field can be well approximated by τ * = τ +· · ·+τ (n) and the process can be altered to output the deformed image DISPLAYFORM12 The target label k may be chosen in each iteration to minimize the vector field to obtain a better approximation in the linearization. More precisely, for a candidate set of labels k 1,..., k m, we compute the corresponding vectors fields τ 1,..., τ m and select DISPLAYFORM13 The candidate set consists of the labels corresponding to the indices of the m smallest entries of F − F l, in absolute value. DISPLAYFORM14 By equation, provided that ∇f is moderate, the deforming vector field takes small values wherever ξ has a small derivative. This means that the vector field will be concentrated on the edges in the image x (see e.g. the first row of figure 2). Further, note that the of a deformation is always a valid image in the sense that it does not violate the pixel value bounds. This is not guaranteed for the perturbations computed with DeepFool. We evaluate the performance of ADef by applying the algorithm to classifiers trained on the MNIST (LeCun) and ImageNet datasets. Below, we briefly describe the setup of the experiments and in tables 1 and 2 we summarize their . MNIST: We train two convolutional neural networks based on architectures that appear in BID16 and BID27 respectively. The network MNIST-A consists of two convolutional layers of sizes 32 × 5 × 5 and 64 × 5 × 5, each followed by 2 × 2 max-pooling and a rectifier activation function, a fully connected layer into dimension 1024 with a rectifier activation function, and a final linear layer with output dimension 10. The network MNIST-B consists of two convolutional layers of sizes 128 × 3 × 3 and 64 × 3 × 3 with a rectifier activation function, a fully connected layer into dimension 128 with a rectifier activation function, and a final linear layer with output dimension 10. During training, the latter convolutional layer and the former fully connected layer of MNIST-B are subject to dropout of drop probabilities 1 /4 and 1 /2. We use ADef to produce adversarial deformations of the images in the test set. The algorithm is configured to pursue any label different from the correct label (all incorrect labels are candidate labels). It performs smoothing by a Gaussian filter of standard deviation 1 /2, uses bilinear interpolation to obtain intermediate pixel intensities, and it overshoots by η = 2 /10 whenever it converges to a decision boundary. An image from the ILSVRC2012 validation set, the output of ADef with a Gaussian filter of standard deviation 1, the corresponding vector field and perturbation. The rightmost image is a close-up of the vector field around the nose of the ape. Second row: A larger deformation of the same image, obtained by using a wider Gaussian filter (standard deviation 6) for smoothing. We apply ADef to pretrained Inception-v3 BID26 and ResNet-101 BID10 ) models to generate adversarial deformations for the images in the ILSVRC2012 validation set. The images are preprocessed by first scaling so that the smaller axis has 299 pixels for the Inception model and 224 pixels for ResNet, and then they are center-cropped to a square image. The algorithm is set to focus only on the label of second highest probability. It employs a Gaussian filter of standard deviation 1, bilinear interpolation, and an overshoot factor η = 1 /10.We only consider inputs that are correctly classified by the model in question, and, since τ * = τ +· · ·+τ (n) approximates the total deforming vector field, we declare ADef to be successful if its output is misclassified and τ * T ≤ ε, where we choose ε = 3. Observe that, by, a deformation with respect to a vector field τ does not displace any pixel further away from its original position than τ T. Hence, for high resolution images, the choice ε = 3 indeed produces small deformations if the vector fields are smooth. In appendix A, we illustrate how the success rate of ADef depends on the choice of ε. When searching for an adversarial example, one usually searches for a perturbation with ∞ -norm smaller than some small number ε > 0. Common choices of ε range from 1 /10 to 3 /10 for MNIST classifiers BID9 BID16 BID28 BID27 BID12 and 2 /255 to 16 /255 for ImageNet classifiers BID9 BID13 BID27 BID12. TAB1 shows that on average, the perturbations obtained by ADef are quite large compared to those constraints. However, as can be seen in FIG0, the relatively high resolution images of the ImageNet dataset can be deformed into adversarial examples that, while corresponding to large perturbations, are not visibly different from the original images. In appendices B and C, we give more examples of adversarially deformed images. In addition to training MNIST-A and MNIST-B on the original MNIST data, we train independent copies of the networks using the adversarial training procedure described by BID16. That is, before each step of the training process, the input images are adversarially perturbed using the PGD algorithm. This manner of training provides increased robustness against adversarial perturbations of low ∞ -norm. Moreover, we train networks using ADef instead of PGD as an adversary. In table 2 we show the of attacking these adversarially trained networks, using ADef on the one hand, and PGD on the other. We use the same configuration for ADef as above, and for PGD we use 40 iterations, step size 1 /100 and 3 /10 as the maximum ∞ -norm of the perturbation. Interestingly, using these configurations, the networks trained against PGD attacks are more resistant to adversarial deformations than those trained against ADef. ADef can also be used for targeted adversarial attacks, by restricting the deformed image to have a particular target label instead of any label which yields the optimal deformation. FIG1 demonstrates the effect of choosing different target labels for a given MNIST image, and FIG2 shows the of targeting the label of lowest probability for an image from the ImageNet dataset. In this work, we proposed a new efficient algorithm, ADef, to construct a new type of adversarial attacks for DNN image classifiers. The procedure is iterative and in each iteration takes a gradient descent step to deform the previous iterate in order to push to a decision boundary. We demonstrated that with almost imperceptible deformations, state-of-the art classifiers can be fooled to misclassify with a high success rate of ADef. This suggests that networks are vulnerable to different types of attacks and that simply training the network on a specific class of adversarial examples might not form a sufficient defense strategy. Given this vulnerability of neural networks to deformations, we wish to study in future work how ADef can help for designing possible defense strategies. Furthermore, we also showed initial on fooling adversarially trained networks. Remarkably, PGD trained networks on MNIST are more resistant to adversarial deformations than ADef trained networks. However, for this to be more conclusive, similar tests on ImageNet will have to be conducted. We wish to study this in future work. T from the MNIST experiments. Deformations that fall to the left of the vertical line at ε = 3 are considered successful. The networks in the first column were trained using the original MNIST data, and the networks in the second and third columns were adversarially trained using ADef and PGD, respectively. Figures 5 and 6 show the distribution of the norms of the total deforming vector fields, τ *, from the experiments in section 3. For networks that have not been adversarially trained, most deformations fall well below the threshold of = 3. Out of the adversarially trained networks, only MNIST-A trained against PGD is truly robust against ADef. Further, a comparison between the first column of figure 5 and figure 6 indicates that ImageNet is much more vulnerable to adversarial deformations than MNIST, also considering the much higher resolution of the images in ImageNet. Thus, it would be very interesting to study the performance of ADef with adversarially trained network for ImageNet, as mentioned in the Conclusion. The standard deviation of the Gaussian filter used for smoothing in the update step of ADef has significant impact on the ing vector field. To explore this aspect of the algorithm, we repeat the experiment from section 3 on the Inception-v3 model, using standard deviations σ = 0, 1, 2, 4, 8 (where σ = 0 stands for no smoothing). The are shown in table 3, and the effect of varying σ is illustrated in figures 7 and 8. We observe that as σ increases, the adversarial distortion steadily increases both in terms of vector field norm and perturbation norm. Likewise, the success rate of ADef decreases with larger σ. However, from figure 8 we see that the constraint τ * T ≤ 3 on the total vector field may provide a rather conservative measure of the effectiveness of ADef in the case of smooth high dimensional vector fields. Figures 9 and 10 show adversarial deformations for the models MNIST-A and MNIST-B, respectively. The attacks are performed using the same configuration as in the experiments in section 3. Observe that in some cases, features resembling the target class have appeared in the deformed image. For example, the top part of the 4 in the fifth column of figure 10 has been curved slightly to more resemble a 9. Figures 11 -15 show additional deformed images ing from attacking the Inception-v3 model using the same configuration as in the experiments in section 3. Similarly, figures 16 -20 show deformed images ing from attacking the ResNet-10 model. However, in order to increase variability in the output labels, we perform a targeted attack, targeting the label of 50th highest probability. Deformed: hartebeest

We propose a new, efficient algorithm to construct adversarial examples by means of deformations, rather than additive perturbations.

Represent the TLDR for finding the corresponding scientific article.

This paper identifies classes of problems for which adversarial examples are inescapable, and derives fundamental bounds on the susceptibility of any classifier to adversarial examples.