Question: Represent the scientific article to find a short TLDR

Question: Adversaries in neural networks have drawn much attention since their first debut. While most existing methods aim at deceiving image classification models into misclassification or crafting attacks for specific object instances in the object setection tasks, we focus on creating universal adversaries to fool object detectors and hide objects from the detectors. The adversaries we examine are universal in three ways: They are not specific for specific object instances; They are image-independent; They can further transfer to different unknown models. To achieve this, we propose two novel techniques to improve the transferability of the adversaries: \textit{piling-up} and \textit{monochromatization}. Both techniques prove to simplify the patterns of generated adversaries, and ultimately in higher transferability. Despite the success of machine learning and deep learning models, recently it has been shown that these models are susceptible and sensitive to what is termed as adversarial examples, a.k.a. adversaries BID32 BID10. Adversaries are usually derived from ordinary data and retain the same semantic content, but can in wrong predictions. Previous studies have shown that adversarial examples can be crafted efficiently and successfully in some conditions, which poses significant security threats BID14. Formally speaking, given a model y = F (x), input X and original or ground-truth output Y = F (X), adversaries are modified versions of the original data, denoted as X + ∆X such that F (X + ∆X) = Y. Generally, ∆X is constrained by its norm value (e.g. L ∞) or other metrics to preserve the original semantic meaning of input X.Existing studies on adversarial examples focus on designing effective and efficient methods to craft ∆X, e.g. L-BFGS BID32, FGSM BID10, iterative methods BID13; defense methods including defensive distillation BID24, random transformation BID35, JPEG-compression and etc.; how to improve the transferability of attacks crafted on one model to deceive another model, both for differently initialized and trained models, and models of different architecture BID19 BID23 BID33 BID34. Up till now, these efforts mainly focus on image classification models. More recent work has studied the robustness of object detectors and tried to fool these models BID21 BID3 BID6 BID16 a; BID28. However, most of these works only attack specific object instances. Few proposed methods have attempted to attack multiple objects and images or verify the capacity to transfer to another model. In this work, we aim to craft universal and transferable adversaries to fool object detectors and conceal objects. As far as we know, we are the first to carry out such large-scale attacks on object detectors. Our target is three-fold: The adversary should work for different objects, regardless of their types, positions, sizes, and etc.. The adversary is not limited to one image only, i.e. achieving image-independence. The adversary should be able to attack detectors that they are not crafted on, i.e. achieving black-box attack. Specifically, we craft an adversarial mask of the same size as input image, denoted as ∆X ∈ Himage×Wimage×3, and impose a norm-value constraint, ||∆X|| ∞ ≤. Such an adversarial mask is in fact similar to what the community has used to fool image classification models. However, optimizing over it is a non-trivial task. A full-sized mask would introduce a total amount of 0.5M parameters, putting our method on risk of overfitting. Further, using the concept of Effective Receptive Field BID22, we found that gradients obtained through back propagation are sparse in spatial positions, making optimization difficult. To achieve our objective, we propose to use the following techniques: Optimizing ∆X over a set of images; Using identical small patches that are piled-up to form the full-sized mask ∆X; Crafting monochromatic masks instead of colorful ones as done in previous work. Our motivation is that piling-up identical small patches in a grid can incorporate translation invariance in a similar way to Convolutional Neural Networks (CNNs), which is also connected with the intuition that any part of the mask should perform equally to attack an object in any position. Constraining the adversarial mask to monochrome further forces the mask to learn coarse-grained patterns that may be universal. In experiments, we compare with decent baseline methods and found that our methods can consistently surpasses them. While our adversarial mask can conceal as many as 80% objects from YOLO V3 BID25, on which it is crafted, it can also hide more than 40% objects from the eyes of Faster-RCNN BID27, in a black-box setting. Further, we compare the patterns generated by different methods and carry out detailed analysis. We found that our techniques did help in crafting more coarse-grained patterns. These patterns have generic appearance, which we attribute as the key for good transferability. In , we make the following contributions in this work: We successfully craft universal adversarial mask that can fool object detectors that are independent in object-level, image-level and model-level. We show that, with the proposed techniques, we can learn and generate masks that have generic and coarse-grained patterns. The pattern we generate is different from those in previous works by large, which may be the key for better transferability. Norm-Ball Attack BID29 first demonstrates how deep learning models can be fooled by images, denoted as X ∈ H×W ×3, that are mixed with imperceptible perturbations, denoted as ∆X ∈ R H×W ×3. Later, various methods for crafting such perturbations have been proposed BID32 BID10 BID19 BID13 BID0 BID2 BID33 BID5. A major common characteristic for these methods is that, the crafted perturbations satisfied the following constraint: ||∆X|| ∞ ≤, where measures how much the images are perturbed. These efforts mainly focus on image classification models. Few shed light on object detectors. We also refer readers to these comprehensive surveys for more detailed introduction BID11 BID7 BID14. In real world application, the attackers usually have no knowledge about the target models, including their architecture, hyper-parameters, and learned parameters. Such situation is termed as black-box attack. Transferability between different models is thus a proxy for black-box methods, and several methods have been proposed. Ensemble attack BID33 is based on the assumption that if an adversary can fool a set of N models, it is more likely to be able to generalize well and fool a N + 1-th one. BID34 analyze the cosine similarity between gradient obtained from different models and propose to smooth the loss landscape BID31 to improve the generalization capacity among models. Specifically, they optimize over a set of data points sampled from the norm-ball of the target image. Another similar work BID1 demonstrates how to generate image-independent adversaries for image classification. They optimize an adversarial patch that has not norm-value constraint but can only modify a small region of the target images. By optimizing over a set of images, the trained patch can transfer to new images successfully. Attack on Object Detector Methods to attack object detectors can be categorized into two classes: stickers that are glued onto target objects to interfere with classification or onto s as counterfeit objects BID3 BID6; perturbation masks that are aligned to and trained for one specific object BID21 or one image only BID15 BID1. In a nutshell, these methods are specific to designated object instances, which means that to successfully fool detectors, one needs to craft adversaries and attacks the target objects one-by-one. BID20 is the first to explore the possibility of transferability. However, the success rate it not very promising. Recently, BID28 demonstrate the effects of feature inference, where randomly transplanted generic objects prove to have non-local adversarial effects, distorting detection even far from the original transplantation position. More concretely, features attained from areas that do not belong to the object of interest have an impact on the detectors' behavior. This holds true both for pixels inside the region-of-interest (ROI) of the object and for those outside of it. The wide-range existence of such phenomenon is a proof that the obejct detectors are fragile and sensitive. Note that the probing approaches used in BID28 are not practical attack strategies, as the authors' method is a type of random search and the are random. Such method may also be dependent on the architecture of target models, as we implemented this method on YOLO V3 but did not observe similar . Besides, BID28 did not study how the objectiveness is influenced in this setting. Extending from BID28, we use a learned mask to probe how to hide an object by modifying its surroundings in a systematic way. Object detection aims to localize the existence of objects of interest, and recognize the categories of them. There are mainly two branches, i.e. region-proposal based methods, including RCNN by BID9, Fast-RCNN by BID8 and Faster-RCNN by BID27, and unified methods including SSD by BID18 and YOLO by BID26. In our research, we experiment with YOLOv3 as it runs the fastest and also performs at state-of-the-art level. We do experiment to see how well the adversaries crafted on YOLOv3, representative of unified methods, can transfer to Faster-RCNN, which is also a representative method for regionproposal based methods. We briefly introduce the core concept of YOLOv3 and Faster-RCNN.YOLO V3 performs two functions: spotting the existence of objects of interest, i.e. those in a pre-defined list; classifying spotted objects into the correct categories. Input images are first fed into the backbone network, producing a sequence of H × W × C feature maps. Each 1 × 1 × C vector represents the potential object at the corresponding position. Classifiers, which are 1 × 1 convolutional layers in practice, predict the existence of objects, its types and positions. Non-Maximal Suppression (NMS) is performed to deliver the final . YOLO V3 has a set of 3 classifiers, each deployed in different layers and aimed at objects of different sizes. In total, there are N P = 10647 such prediction points, also termed as anchor. In essence, YOLO V3 can be viewed as a multi-head image classifier. Faster-RCNN incorporates a Region-Proposal Network (RPN) to make detection proposals, which are bounding boxes indicating the existence of objects. Sub-regions are cropped from a shared feature map to perform classification. However, the detection and localization of objects in Faster-RCNN is solely dependent on RPN, which works in the same way as YOLO V3. In this section, we introduce how to obtain adversarial masks, DISPLAYFORM0 H×W ×3, and further introduce the two techniques we propose to generate adversarial masks that better transfer to other settings. Note that the attack is performed on YOLO V3, and therefore H = W = 416. The simplest way is to follow the tradition in adversaries for image classification, model the mask as a 416 × 416 × 3 parameter, and optimize over some metric. We denote it as DISPLAYFORM0 416×416×3. To conceal objects, we minimize the objectiveness score produced by the model. We set the minimization target as the average log-likelihood of the top-200 anchors of highest scores in YOLO V3. This is an adaptation of Online Hard Example Mining (OHEM) BID30 to balance the number of different categories. In our case, such Online Hard Positive Mining (OHPM) can avoid the overwhelming effects of the large number of negative anchors 1. Optimization is done over a set of training images in the way of BID1. Data augmentation is used to improve the robustness of the trained mask. Specifically, we minimize the following target: DISPLAYFORM1 where x's are images sampled from the training set, A is a randomly composed data augmentation scheme(rotation, translation, scaling), p top−i is the probability value of the i−th highest scored anchor, N = 200, Clip is a per-pixel clipping to ensure the attacked image is still in valid scope, and other symbols are as defined above. In practice, the norm-value constraint is done by applying an element-wise tanh function to the parametrized mask and then multiply it with a designated distortion rate, which is proposed in BID2. Training is continued until performance on a held-out test-set is not improved further. As there are no other baselines, the basic setting of full-mask will in practice serve as a baseline for the two newly proposed techniques. The baseline setting of parameterization would in a total number of 519K parameters. Although it allows for fine-grained patterns and thus stronger capacity, such exploitation of details may make it difficult to transfer to other models BID13. Besides, an ideal adversarial mask should be translation-invariant, as it should be able to attack objects in any positions. To explicitly encode such intuitions, we propose a pile-up configuration to obtain adversarial masks. As shown in FIG0, we parametrize a much smaller mask, denoted as m pile ∈ R 416r × 416r ×3, where r ∈ measures the size of the mask. To obtain a full-sized mask, we duplicate and pile up these small masks in a grid-aligned way. Specifically, we stack them into a 1 r × 1 r grid. We denote the aforementioned pile-up process as a function: y = pile(x).In the case of pile-up configuration, the adversarial mask is obtained by: DISPLAYFORM0 During training, the mask is applied to input images by addition followed by clipping: x * = Clip max=1 min=0 {x + ∆X}. Other training details are the same as Section 4.1. Gradients are averaged over the grid cells. The motivation for a monochromatic adversarial mask is two-fold. On the one hand, such adversaries require much fewer parameters. On the other hand, monochromatic patterns are less conspic- Green bounding boxes represent objects that are detected both before and after attack; Red ones stand for those detected before attack, but concealed after attack.uous and may better blend into objects, which may show potential in physical world application. Monochromatic mask can further be interpreted as changes of brightness. We implement monochromatic adversarial masks by setting the values of the three color channels as the same. Training is the same as the one described above. Note that technique 1 and technique 2 can be combined together. In such case, there are only 10816 parameters, and only 2.1% of the baseline full-mask setting. Later we would show that such constraints in simplified and even stylish adversarial patterns. We design experiments to answer the following questions: (Q1) How effective are our trained adversarial masks on YOLO V3 and Faster-RCNN respectively? How successful it the transfer to Faster-RCNN? (Q2) How do the techniques we employ help in generating effective attacks?Empirically, we show that: All three methods can achieve decent performance in the task of hiding objects from detectors. The two proposed techniques improve transferability significantly. Adversaries generated with the two proposed techniques demonstrate repetitive and coarse-grained patterns, which seems more robust than the finer ones. Samples for detection are shown in FIG1. All experiments are based on off-the-shelf PyTorch implementations of YOLO V3 2 and Faster-RCNN 3. Models are pretrained on COCO Detection dataset BID17, with an mAP value of 33.0 for YOLO V3 and 37.0 for Faster-RCNN on test set. For the pile-up configuration, we set r = 0.25. We randomly sample images from the validation set of the COCO Detection dataset as training set and test set for our methods, 512 for each. The adversaries are constructed by applying mini-batch SGD with a batch size of 16, a learning rate of 1e + 2, and momentum of 5e − 1, until convergence. As we aim at hiding objects from detectors, we propose to use the average number of detected objects per image as our main evaluation metric. Previous work on interfering with object detectors at large scale BID28 ) uses more detailed evaluation method, taking into accounts cases like mis-classification of detected objects. We argue that our metric is suitable enough for our task, as it directly measures our main goal of making objects disappear. To make comparison easier, we use a derived metric in practice, where we compute the ratio of attacked image to clean image. We term it as Detection Rate, measuring the proportion of objects that are still detected when being attacked. Here, the lower the measure, the better. For a more comprehensive comparison, we train the adversaries with different values of, and plot a curve to characterize its dynamics. Samples for different levels of distortion are shown appendix. B.Overall performance evaluations are shown in FIG2.Further, we perform experiments under the setting of black box attack to truly evaluate the effectiveness of our methods: we transfer the adversaries trained on YOLO V3 to Faster-RCNN and compute the detection rate for each sample. Similarly, we evaluate over different levels of distortion. Results are shown in FIG2 (Right). The second observation is that, while the full-mask attack has much more parameters than the other two methods, it achieves slightly lower success rate 4 at concealing objects. This may seem unreasonable at first glance as more parameters means stronger capacity. However, as we show in the next section, this may be due to the fact that the full-mask attack is much harder in training. We also notice that the colorful pile-up setting performs better that the other two methods by a significant margin. This can be explained by the fact that it has more parameters but not too many, therefore containing enough capacity and still being easy to train. From FIG2, one basic observation is that, even with mild distortion (≤ 16 255), the best performing method can still conceal 40% of the objects. The success rate of the proposed methods perform better than the white noise baseline, demonstrating some inherent potential in transferring. The most important observation is that, the two proposed techniques are significantly better than the full-mask baseline. At the same time, the monochromatic is better than the pile-up configuration. The comparison of the supports our arguments that, pile-up and monochromatization are both effective technique in improving transferability, while monochromatization can further push the envelope. Specifically, pile-up setting can reduce detection rate by a large margin over the fullmask baseline, ranging from 5% to 30%, depending on the value. Monochromatization can further reduce by 5%.We also notice that the detection rate bounces back for pile-up setting when ≥ 24 255. We manually check the attacked images and detection , and found that the repetitive circle patterns in the trained pattern are sometimes mistaken as round objects, e.g. apple and orange, ing in higher detection rate. We consider that this is one defect in the evaluation metric we use. We manually check all the attacked images again and found that this phenomenon only occurs in the colorful pile-up masks when ≥ 24 255. In this section, we visually evaluate how the two techniques play their role in improving transferability. Especially, we discuss about how pile-up helps in significantly improve transferability, as is shown in the experiments. Further, we study a strong method for comparison to provide further insight into adversaries for object detectors. To observe what impact the two proposed techniques actually have, we visualize the trained adversaries in FIG4 for full-mask, pile-up, and monochromitization respectively. We notice that, as we train with our techniques, the generated adversaries are much different from naively trained ones. Specifically, when we use pile-up configuration, the adversaries are repetitive as expected by design, and more smooth. We zoom in to compare the pixel-level landscape, and found that while full-mask consists of tiny color lumps that are nearly as complex as white noise, the pile-up mask is much more smooth, containing less fine-grained details. When we monochromatize the mask, the mask becomes highly repetitive and stylish. The zoom-in view shows that the patterns are even more coarse-grained than using pile-up solely. We attribute the success in improving transferability to such highly simplified patterns for adversarial attacks. In this section, we try to give a theoretical explanation as to why the less parametrized pile-up configuration can perform better even in the white-box setting. It would not be surprising for performing worse on Faster-RCNN, which can be attributed as overfitting, due to the large number of parameters. Here we introduce our hypothesis that the full-mask adversary is harder to train, and the reason may lie in the sparsity of gradients. For crafting adversaries, most existing methods, including ours, are based on gradients propagated from the last layer, and thus the quality of the gradients are important. We found that for each image, the gradients propagated from the classifier layer only cover a small region of the adversarial mask. The large majority of the parameters in full-mask setting are not getting gradients at all 5. Under such condition, using adaptive training methods e.g. Adam would in erroneous estimation of amounts of recent updates; using momentum based methods would in over-update; using vanilla SGD would only update a small fraction of the mask. There is a related concept named Effective Receptive Field (ERF) BID22, which essentially computes the gradient of a certain neuron over the input image. In fact, the objectiveness score is the activation value of that classifier neuron. Therefore, we can compute ERF for each anchor to analyze how the full-mask attack is updated. Some examples is shown in appendix. A. We notice that the gradients basically only cover the object region (not even the bounding box!). Significant variance exists for each pixel across different samples. As objects in images usually take up small fractions, the updates of such a large mask may thus be inefficient and difficult. Different from full-mask setting, piled-up small and identical patches, in turn, can gather the gradients up, making the updates more efficient and accurate. We assume this is the main reason why pile-up configuration can beat full-mask setting by large. As far as we know, there are no other methods that perform similar tasks to our target setting. Therefore, we use white noise of the same distortion level as the baseline for our three methods. We also contend that our main contribution rests in proposing and exploring two techniques to improve transferability, and that therefore, the full-mask method itself is a strong and appropriate baseline such that improvements over it provide appropriate experimental analysis. However, we also provide experimentation below with a method adapted from adversaries for image classification approaches, to establish another benchmark for universal attack on object detectors. Following Adversarial Patch approach in BID1, we train a patch to conceal neighboring objects. One may argue that by placing the patch onto objects to conceal it can also serve as a baseline. While there is existing work on specific objects by applying stickers, they are specially designed for each object instance and still change the object a lot. One could argue that altering a scene with such large distortion does not make sense in the real world as it's too conspicuous or one could simply cover the object with a cloth. Therefore, we consider it interesting if we can design an object that can conceal neighboring objects contactlessly. Basically we follow the original training method BID1. We found that the patch is indeed able to conceal other objects contactlessly. The training setting and more detailed and are in appendix. C.We notice that the success rate depends on the distance to the objects. For objects that are close enough, the success rate can be as high as 50%. An interesting observation is that the trained patch contains circular patterns that are similar to those in the pile-up and monochromatization setting. Overall, it's a well-performing baseline, but it's essentially another type of attack. We have just included it here to provide a better understanding of the task of concealing objects. This paper provides effective methods to fool object detectors. We admit that one major limitation is: object detectors themselves are not robust enough yet. Current image classification can attain a top-1 accuracy higher than 80% and top-5 accuracy, which has surpassed the human level. Therefore, the wide-range existence of adversaries are intriguing: how are these intricate models fooled? On the contrary, the performance of object detectors are still far from human level. Though the experiments presented here show that our methods can beat baselines directly adapted from methods for attacking image classification models, the mechanism behind errors in object detectors still remains unknown. We randomly pick 100 samples to analyze how gradients are being propagated. Among the samples we analyzed, we randomly pick 4 as representatives and show them in Fig.5. For each image sample, we randomly select one object that's detected, compute gradients of its objectiveness scores, and visualize the gradients propagated to the adversarial mask. We manually check all the samples and found that only the object area can obtain gradients that are not negligible. We need more detailed specification of being negligible. For object area, i.e. pixels belonging to the object we study, the average norm of gradient has a magnitude from 1e1 to 1e2. For area outside object, the magnitude drops to 1e − 6. Although some current optimization methods suggest that different parameters can have different level of gradients, e.g. Adam BID12, this is not the case for us. In our case, parameters are not receiving gradients of different magnitudes. On the contrary, parameters are receiving gradients of unstable magnitudes, while intuitively, there should be some level of symmetry or repetition in the pattern of the mask. We argue that such unstable gradient flow may make it hard to train, and finally in lower performance despite larger potential capacity. For better illustration of how much images are distorted, we randomly select some samples ans show them in FIG5 respectively. (Zoom-in for better view.) APPENDIX C ADVERSARIAL PATCH THAT HIDES NEIGHBORING OBJECTS C.1 TRAINING We parametrize the artificial object as a tensor p ∈ h×w×3 in round shape, and train the patch with gradient descent methods on a training set of images. Standard data augmentations are performed to improve robustness, including scaling and rotation. However, we need to adapt the training details for better suitability. Specifically, the artificial object is randomly placed around objects, but not overlapping with objects. Specifically, for each image, we first randomly select one object, around which we place the artificial object. Then we rescale the artificial object to a proper size: r = max(min(0.25, w object, h object), 0.1) × U (0.9, 1.1) where U (a, b) is a uniform random variable used as scaling factor for data augmentation, w object, h object ∈ are the size of the selected object as proportion to input image size, r is the ratio of the proper size to the image size. The size ratio makes sure that the size of the artificial object is neither too big nor too small. Then we randomly rotate the object, ranging from − 1 8 π to 1 8 π. The last step is to pick a point to place the artificial object. We enlarge the bounding box of the selected object in-place by 10%, and uniformly sample one point from the periphery. We place the artificial such that the center of it are located at the sampled point. We denote the transformation aforementioned as function A. For training, we minimize the expected log probability of objectiveness of all predictions in YOLOv3 over transformations and images in the training set: DISPLAYFORM0 where p i denote the probability of being an object for prediction point i in YOLOv3, and N denotes the number of prediction points in the model 6. In practice, most prediction points (around 99.9%) in YOLOv3 are negative, which would obfuscate the positives that are in minority and be disastrous. We alternatively optimize over the top 12 positive prediction points. We explore the effect of different sizes and distances to target objects. We measure the size of the object as proportion of its diameter to the side length of the input image. The distance is computed as the absolute distance between the center of the trained adversarial object, and the center of the bounding box of the detected object, divided by the length of the bounding box's diagonal. For better vision, we take the logarithm of distance.6 N = 10647 in the case of YOLOv3Original Image Gradient Map Figure 5: Left: detection for original image; Red BBox: the target object from which we back-prop gradients; Right: the gradient flowing from the object bounded by red bounding box. Pixel values are normalized using the min-max rule. Figure 10: Overall performance of adversarial patch of different sizes and distances from target objects. We use two baselines: black-hole that replaces all pixels in a sub-region with 0; white-noise that replaces all pixels in a sub-region with random Gaussian noise. We evaluate the performance with the using held-out test set. Quantitative are shown in FIG0. One important is that, although baseline methods similarly change the image significantly by replacing a sub-region completely, they can barely affect the detection . We also include some samples from the test set in FIG0. The trained object can indeed conceal other objects in a contactless way. When we look can the effect of size, we notice that the trained object, of a reasonable size (0.28), can conceal more than 50% of the existence of other objects when simply placed around them. We did not consider patches of size larger than 0.30, as we consider it impractical under the real world setting. Distance also plays an important role. As we randomly place the object for training, where the actual size ranges from 0.10 to 0.25, we notice that for neighboring objects, more than half the objects can be concealed. As the distance to the target objects grows, success rate drops. Figure 11: Demonstration of successful attack. Top: original image and its detection ; Bottom: attacked image and its detection .

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We focus on creating universal adversaries to fool object detectors and hide objects from the detectors.

represent the tldr for finding the corresponding scientific article

Pseudo-labeling has shown to be a weak alternative for semi-supervised learning. We, conversely, demonstrate that dealing with confirmation bias with several regularizations makes pseudo-labeling a suitable approach.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: This work presents the Poincaré Wasserstein Autoencoder, a reformulation of the recently proposed Wasserstein autoencoder framework on a non-Euclidean manifold, the Poincaré ball model of the hyperbolic space H n. By assuming the latent space to be hyperbolic, we can use its intrinsic hierarchy to impose structure on the learned latent space representations. We show that for datasets with latent hierarchies, we can recover the structure in a low-dimensional latent space. We also demonstrate the model in the visual domain to analyze some of its properties and show competitive on a graph link prediction task. Variational Autoencoders (VAE) (17; 28) are an established class of unsupervised machine learning models, which make use of amortized approximate inference to parametrize the otherwise intractable posterior distribution. They provide an elegant, theoretically sound generative model used in various data domains. Typically, the latent variables are assumed to follow a Gaussian standard prior, a formulation which allows for a closed form evidence lower bound formula and is easy to sample from. However, this constraint on the generative process can be limiting. Real world datasets often possess a notion of structure such as object hierarchies within images or implicit graphs. This notion is often reflected in the interdependence of latent generative factors or multimodality of the latent code distribution. The standard VAE posterior parametrizes a unimodal distribution which does not allow structural assumptions. Attempts at resolving this limitation have been made by either "upgrading" the posterior to be more expressive or imposing structure by using various structured priors,. Furthermore, the explicit treatment of the latent space as a Riemannian manifold has been considered. For instance, the authors of show that the standard VAE framework fails to model data with a latent spherical structure and propose to use a hyperspherical latent space to alleviate this problem. Similarly, we believe that for datasets with a latent tree-like structure, using a hyperbolic latent space, which imbues the latent codes with a notion of hierarchy, is beneficial. There has recently been a number of works which explicitly make use of properties of non-Euclidean geometry in order to perform machine learning tasks. The use of hyperbolic spaces in particular has been shown to yield improved on datasets which either present a hierarchical tree-like structure such as word ontologies or feature some form of partial ordering. However, most of these approaches have solely considered deterministic hyperbolic embeddings. In this work, we propose the Poincaré Wasserstein Autoencoder (PWA), a Wasserstein autoencoder model which parametrizes a Gaussian distribution in the Poincaré ball model of the hyperbolic space H n. By treating the latent space as a Riemannian manifold with constant negative curvature, we can use the norm ranking property of hyperbolic spaces to impose a notion of hierarchy on the latent space representation, which is better suited for applications where the dataset is hypothesized to possess a latent hierarchy. We demonstrate this aspect on a synthetic dataset and evaluate it using a distortion measure for Euclidean and hyperbolic spaces. We derive a closed form definition of a Gaussian distribution in hyperbolic space H n and sampling procedures for the prior and posterior distributions, which are matched using the Maximum Mean Discrepancy (MMD) objective. We also compare the PWA to the Euclidean VAE visually on an MNIST digit generation task as well quantitatively on a semi-supervised link prediction task. The rest of this paper is structured as follows: we review related work in Section 2, give an overview of the mathematical tools required to work with Riemannian manifolds as well as define the notion of probability distributions on Riemannian manifolds in Section 3. Section 4 describes the model architecture as well as the intuition behind the Wasserstein autoencoder approach. Furthermore, we derive a method to obtain samples from prior and posterior distributions in order to estimate the PWA objective. We present the performed experiments in and discuss the observed in Section 5 and a summary of our in Section 6. Amortized variational inference There has been a number of extensions to the original VAE framework. These extensions address various problematic aspects of the original model. The first type aims at improving the approximation of the posterior by selecting a richer family of distributions. Some prominent examples include the Normalizing Flow model as well as its derivates,,. A second direction aims at imposing structure on the latent space by selecting structured priors such as the mixture prior,, learned autoregressive priors or imposing informational constraints on the objective,. The use of discrete latent variables has been explored in a number of works. The approach conceptually most similar to ours but with a hyperspherical latent space and a von-Mises variational distribution has been presented in. Hyperbolic geometry The idea of graph generation in hyperbolic space and analysis of complex network properties has been studied in. The authors of have recently used both the Poincaré model and the Lorentz model of the hyperbolic space to develop word ontology embeddings which carry hierarchical information encoded by the embedding norm. The general idea of treating the latent space as a Riemannian manifold has been explored in. A model for Bayesian inference for Riemannian manifolds relying on particle approximations has been proposed in. Finally the natural gradient method is a prime example for using the underlying information geometry imposed by the Fisher information metric to enhance learning performance. Three concurrent works have explored an idea similar to ours. propose to train a VAE with a hyperbolic latent space using the traditional evidence lower bound (ELBO) formulation. They approximate the ELBO using MCMC samples as opposed to our approach, which uses a Wasserstein formulation of the problem. propose to use a wrapped Gaussian distribution to obtain samples on the Lorentz model of hyperbolic latent space. The samples are generated in Euclidean space using classical methods and then projected onto the manifold under a concatenation of a parallel transport and the exponential map at the mean. The authors of also propose a similar approach but use an adversarial autoencoder model in their work instead. In this section, we briefly outline some of the concepts from differential geometry, which are necessary to formally define our model. A Riemannian manifold is defined as a the tuple (M, g), where for every point x belonging to the manifold M, a tangent space T x M is defined, which corresponds to a first order local linear approximation of M at point x. The Riemannian metric g is a collection of inner products ·|· x: T x M × T x M → R on the tangent spaces T x M. We denote by α(t) ∈ M to be smooth curves on the manifold. By computing the speed vectorα(t) at every point of the curve, the Riemannian metric allows the computation of the curve length: Given a smooth curve α(a, b) → M, the distance is defined by the infimum over α(The smooth curves of shortest distance between two points on a manifold are called geodesics. Given a point x ∈ M, the exponential map exp x (v): T x M → M gives a way map a vector v in the tangent space T x M at point x to the corresponding point on the manifold M. For the Poincaré ball model of the hyperbolic space, which is geodesically complete, this map is well defined on the whole tangent space T x M. The logarithmic map log x (v) is the inverse mapping from the manifold to the tangent space. The parallel transport P x0→x: T x0 M → T x M defines a linear isometry between two tangent spaces of the manifold and allows to move tangent vectors along geodesics. Hyperbolic spaces are one of three existing types of isotropic spaces: the Euclidean spaces with zero curvature, the spherical spaces with constant positive curvature and the hyperbolic spaces which feature constant negative curvature. The Poincaré ball is one of the five isometric models of the hyperbolic space. The model is defined by the tuple (B n, g H) where B n is the open ball of radius 1, 1 g H is the hyperbolic metric and g E = I n is the Riemannian metric on the flat Euclidean manifold. The geodesic distance on the Poincaré ball is given by In order to perform arithmetic operations on the Poincaré ball model, we rely on the concept of gyrovector spaces, which is a generalization of Euclidean vector spaces to models of hyperbolic space based on Möbius transformations. First proposed by, they have been recently used to describe typical neural network operations in the Poincaré ball model of hyperbolic space. In order to perform the reparametrization in hyperbolic space, we use the gyrovector addition and Hadamard product defined as a diagonal matrix-gyrovector multiplication. Furthermore, we make use of the exponential exp µ and logarithm log µ map operators in order to map points onto the manifold and perform the inverse mapping back to the tangent space. The Gaussian decoder network is symmetric to the encoder network. The Gaussian distribution is a common choice of prior for VAE style models. Similarly to the VAE, we can select a generalization of the Gaussian distribution in the hyperbolic space as prior for our model. In particular, we choose the maximum entropy generalization of the Gaussian distribution on the Poincaré ball model. The Gaussian probability density function in hyperbolic space is defined via the Fréchet mean µ and dispersion parameter σ > 0, analogously to the density in the Euclidean space. The main difference compared to Euclidean space is the use of the geodesic distance d(x, µ) in the exponent and a different dispersion dependent normalization constant Z(σ) which accounts for the underlying geometry. In order to compute the normalization constant, we use hyperbolic 1 this can be generalized to radius for curvature c. Throughout this paper, we assume the Poincaré ball radius to be c = 1 and omit it from the notation. polar coordinates where r = d(x, µ) is the geodesic distance between the x and µ. This allows the decomposition of Z(σ) into radial and angular components. We derive the closed form of the normalization constant in appendix A. For a two-dimensional space, the normalization constant is given by: Dispersion representation The closed form of the hyperbolic Gaussian distribution is only defined for a scalar dispersion value. This can be a limitation on the expressivity of the learned representations. However, the variational family which is implicitly given by the hyperbolic reparametrization allows for vectorial or even full covariance matrix representations, which can be more expressive. Since the maximum mean discrepancy can be estimated via samples, we do not require a closed form definition of the posterior density as is the case with training using the evidence lower bound. This allows the model to learn richer latent space representations. Our model mimics the general architecture of a variational autoencoder. The encoder parametrizes the posterior variational distribution q φ (z|x) and the decoder parametrizes the unit variance Gaussian likelihood p θ (x|z). In order to accomodate the change in the underlying geometry of the latent space, we introduce the maps into hyperbolic space and back to the tangent space. Both the encoder and decoder network consist of three fully-connected layers with ReLU activations. We use the recently proposed hyperbolic feedforward layer for the encoding of the variational family parameters (µ H, σ). For the decoder f θ (x|z), we use the logarithm map at the origin log 0 (z) to map the posterior sample z back into the tangent space. Mean and variance parametrization In order to obtain posterior samples in hyperbolic space, the parametrization of the mean uses a hyperbolic feedforward layer (W, b H) as the last layer of the encoder network (proposed in). The weight matrix parameters are Euclidean and are subject to standard Euclidean optimization procedures (we use Adam) while the bias parameters are hyperbolic, requiring the use of Riemannian stochastic gradient descent (RSGD). The outputs of the underlying Euclidean network h are projected using the exponential map at the origin and transformed using the hyperbolic feedforward layer map where ϕ h is the hyperbolic nonlinearity 2: The reparametrization trick is a common method to make the sampling operation differentiable by using a differentiable function g(, θ) to obtain a reparametrization gradient for backpropagation through the stochastic layer of the network. For the location-scale family of distributions, the reparametrization function g(, θ) can be written as z = µ + σ in the Euclidean space where ∼ N (0, I). We adapt the reparametrization trick for the Gaussian distribution in the hyperbolic space by using the framework of gyrovector operators. We obtain the posterior samples for the parametrized mean µ H (x) and dispersion σ(x) using the following relation: We can motivate the reparametrization with the help of Fig. 1, which depicts the reparametrization in a graphical fashion. In a first step, we sample from the hyperbolic standard prior ∼ N H using a rejection sampling procedure we describe in Algorithm 1. The samples are projected to the tangent space using the logarithm map log 0 at the origin, where they are scaled using the dispersion parameter. The scaled samples are then projected back to the manifold using the exponential map exp 0 and translated using µ. We choose the hyperbolic standard prior N H (0, I) as prior p(z). In order to generate samples from the standard prior, we use an approach based on the volume ratio of spheres in H d to obtain the quasi-uniform samples on the Poincaré disk and subsequently use a rejection sampling procedure to obtain radius samples. We use the quasi-uniform distribution as a proposal distribution for the radius. Using the decomposition into radial and angular components, we can sample a direction from the unit sphere uniformly and simply scale using the sampled radius to obtain the samples from the prior. An alternative choice of prior is the wrapped Gaussian distribution. Evidence Lower Bound The variational autoencoder relies on the evidence lower bound (ELBO) reformulation in order to perform tractable optimization of the Kullback-Leibler divergence (KLD) between the true and approximate posteriors. In the Euclidean VAE formulation, the KLD integral has a closed-form expression, which simplifies the optimization procedure considerably. The definition of the evidence lower bound can be extended to non-Euclidean spaces by using the following formulation with the volume element of the manifold dvol g H induced by the Riemannian metric g H. By substituting the hyperbolic Gaussian into we obtain the following expressions for E q φ (z) log q φ (z|x): Due to the nonlinearity of the geodesic distance in the exponent, we cannot derive a closed form solution of the expectation expression E q φ (z) [log q φ (z)]. One possibility is to use a Taylor expansion of the first two moments of the expectation of the squared logarithm. This is however problematic from a numerical standpoint due to the small convergence radius of the Taylor expansion. The ELBO can be approximated using Monte-Carlo samples, as is done in. We have considered this approach to be suboptimal due to large variance associated with one-sample MC approximations of the integral. Wasserstein metric In order to circumvent the high variance associated with the MC approximation we propose to use a Wasserstein Autoencoder (WAE) formulation of the variational inference problem. The authors of the WAE framework propose to solve the optimal transport problem for matching distributions in the latent space instead of the more difficult problem of matching the data distribution p(x) to the distribution generated by the model p y (z) as is done in the generative adversarial network (GAN) literature. Kantorovich's formulation of the optimal transport problem is given by: Γ∈p(x∼px,y∼py) where c(x, y) is the cost function, p(x ∼ p x, y ∼ p y) is the set of joint distributions of the variables x ∼ p x and y ∼ p y. Solving this problem requires a search over all possible couplings Γ of the two distributions which is very difficult from an optimization perspective. The issue is circumvented in a WAE model as follows. The generative model of a variational autoencoder is defined by two steps. First we sample a latent variable z from the latent space distribution p(z). In a second step, we map it to the output space using a deterministic parametric decoder f θ (x|z). The ing density is given by: Under this model, the optimal transport cost takes the following simpler form due to the fact that the transportation plan factors through the map f θ. The optimization procedure is over the encoders q φ (x) instead of the couplings between p x and p y. The WAE objective is derived from the optimal transport cost by relaxing the constraint on the posterior q. The constraint is relaxed by using a Lagrangian multiplier and an appropriate divergence measure. The Maximum Mean Discrepancy (MMD) metric with an appropriate positive definite RKHS 3 kernel is an example of such a divergence measure. MMD is known to perform well when matching high-dimensional standard normal distributions. MMD is a metric on the space of probability distributions under the condition that the selected RKHS kernel is characteristic. Geodesic kernels are generally not positive definite, however it has been shown that the Laplacian kernel k(x, y) = exp(−λ(d H (x, y))) is positive definite if the metric of the underlying space is conditionally negative definite. In particular, this holds for hyperbolic spaces. In practice, there is a high probability that a geodesic RBF kernel is also positive definite depending on the dataset topology. We choose the Laplacian kernel as it also features heavier tails than the Gaussian RBF kernel, which has a positive effect on outlier gradients. The MMD loss function is defined over two probability measures p and q in an RKHS unit ball F as follows: There exists an unbiased estimator for D MMD (p, q φ). A finite sample estimate can be computed based on minibatch samples from the prior z ∼ p(z) via the rejection sampling procedure described Parameter updates The hyperbolic geometry of the latent space requires us to perform Riemannian stochastic gradient descent (RSGD) updates for a subset of the model parameters, specifically the bias parameters of µ. We perform full exponential map updates using gyrovector arithmetic for the gradients with respect to the hyperbolic parameters similar to instead of using a retraction approximation as in. In order to avoid numerical problems at the origin and far away from the origin of the Poincaré ball, we perturb the operands if the norm is close to 0 or 1 respectively. The Euclidean parameters are updated in parallel using the Adam optimization procedure. To determine the capability of the model to retrieve an underlying hierarchy, we have setup two experiments in which we measure the average distortion of the respective latent space embeddings. We measure the distortion between the input and latent spaces using the following distortion metric, where subscript U denotes the distances in the input space and V the distances in the latent space. Noisy trees The first dataset is a set of synthetically generated noisy binary trees. The vertices of the main tree are generated from a normal distribution where the mean of the child nodes corresponds to the parent sample x i = N (x p(i), σ i ) and p(i) denotes the index of the parent node. In addition to the main tree, we add To encourage a good embedding in a hyperbolic space, we enforce the norms of the tree vertices to grow monotonously with the depth of the tree by rejecting samples whose norms are smaller than the norm of the parent vertices. We have trained our model on 100 generated trees for 100 epochs. The tree vertex variance was set to σ i = 1 and the noise variance to σ j = 0.1. We have also normalized the generated vertices to zero mean and unit variance. Table 1 compares the distortion values of the test set latent space embeddings obtained by using the Euclidean VAE model compared to the PWA model. We can see that the PWA model shows less distortion when embedding trees into the latent space of dimension d = 2, which confirms our hypothesis that a hyperbolic latent space is better suited to data with latent hierarchies. As a reference, we provide the distortion scores obtained by the classical T-SNE dimensionality reduction technique. In this experiment, we apply our model to the task of generating MNIST digits in order to get an intuition for the properties of the latent hyperbolic geometry. In particular, we are interested in the visual distibution of the latent codes in the Poincaré disk latent space. While the MNIST latent space is not inherently hierarchically structured -there is no obvious norm ranking that can be imposed -we can use it to compare our model to the Euclidean VAE approach. We train the models on dynamically binarized MNIST digits and evaluate the generated samples qualitatively as well as quantitatively via the reconstruction error scores. We can observe in Appendix B that the samples present a deteriorating quality as the dimensionality increases despite the lower reconstruction error. This can be explained by the issue of dimension mismatch between the selected latent space dimensionality d z and the intrinsic latent space dimensionality d I documented in and can be alleviated by an additional p-norm penalty on the variance. We have not observed a significant improvement by applying the L2-penalty for higher dimensions. We have also performed an experiment using a two-dimensional latent space. We can observe that the structure imposed by the Poincaré disk pushes the samples towards the outside of the disk. This observation can be explained by the fact that hyperbolic spaces grow exponentially. In order to generate quality samples using the prior, some overlap is required with the approximate posterior in the latent space. The issue is somewhat alleviated in higher dimensions as the distribution shifts towards the ball surface. In this experiment, we aim at exploring the advantages of using a hyperbolic latent space on the task of predicting links in a graph. We train our model on three different citation network datasets: Cora, Citeseer and Pubmed. We use the Variational Graph Auto-Encoder (VGAE) framework and train the model in an unsupervised fashion using a subset of the links. The performance is measured in terms of average precision (AP) and area under curve (AUC) on a test set of links that were masked during training. Table 1 shows a comparison to the baseline with a Euclidean latent space (N -VGAE), showing improvements on the Cora and Citeseer datasets. We also compare our to the obtained using a hyperspherical autoencoder (S-VGAE). It should be noted that we have used a smaller dimensionality for the hyperbolic latent space (16 vs 64 and 32 for the Euclidean and hyperspherical cases respectively), which could be attributed to the fact that a dataset with a hierarchical latent manifold requires latent space embeddings of smaller dimensionality to efficiently encode the information (analogously to the of). We can observe that the PWA outperforms the Euclidean VAE on two of the three datasets. The hyperspherical graph autoencoder (S-VGAE) outperforms our model. One hypothesis which explains this is the fact that the structure of the citation networks has a tendency towards a positive curvature rather than a negative one. It is worth noting that it is not entirely transparent whether the use of Graph Convolutional Networks, which present a very simple local approximation of the convolution operator on graphs, allows to preserve the curvature of the input data. We have presented an algorithm to perform amortized variational inference on the Poincaré ball model of the hyperbolic space. The underlying geometry of the hyperbolic space allows for an improved performance on tasks which exhibit a partially hierarchical structure. We have discovered certain issues related to the use of the MMD metric in hyperbolic space. Future work will aim to circumvent these issues as well as extend the current . In particular, we hope to demonstrate the capabilities of our model on more tasks hypothesized to have a latent hyperbolic manifold and explore this technique for mixed curvature settings. A PRIOR REJECTION SAMPLING H (r|0, 1) Result: n samples from prior p(z) while i < n do sampleφ ∼ N (0, I d); compute direction on the unit sphereφ =φ ||φ||; sample u ∼ U; get uniform radius samples r i ∈ [0, r max] via ratio of hyperspheres; where erfc is the complementary error function. In this list of gyrovector operations and throughout this paper, we assume the Poincaré ball radius to be c = 1 and omit it from the notation. Gyrovector addition: x ⊕ y = (1 + 2 x, y + ||y|| 2)x + (1 − ||x|| 2)y 1 + 2 x, y + ||x|| 2 ||y|| 2 ] Matrix-gyrovector product:

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Wasserstein Autoencoder with hyperbolic latent space

Represent the TLDR for finding the corresponding scientific article

We find movement in function space is not proportional to movement in parameter space during optimization. We propose a new natural-gradient style optimizer to address this.


Question: Represent the text to find a very short summary (1-2 sentences max).

Question: In this paper, we introduce a method to compress intermediate feature maps of deep neural networks (DNNs) to decrease memory storage and bandwidth requirements during inference. Unlike previous works, the proposed method is based on converting fixed-point activations into vectors over the smallest GF finite field followed by nonlinear dimensionality reduction (NDR) layers embedded into a DNN. Such an end-to-end learned representation finds more compact feature maps by exploiting quantization redundancies within the fixed-point activations along the channel or spatial dimensions. We apply the proposed network architecture to the tasks of ImageNet classification and PASCAL VOC object detection. Compared to prior approaches, the conducted experiments show a factor of 2 decrease in memory requirements with minor degradation in accuracy while adding only bitwise computations. Recent achievements of deep neural networks (DNNs) make them an attractive choice in many computer vision applications including image classification BID6 and object detection BID9. The memory and computations required for DNNs can be excessive for low-power deployments. In this paper, we explore the task of minimizing the memory footprint of DNN feature maps during inference and, more specifically, finding a network architecture that uses minimal storage without introducing a considerable amount of additional computations or on-the-fly heuristic encoding-decoding schemes. In general, the task of feature map compression is tightly connected to an input sparsity. The input sparsity can determine several different usage scenarios. This may lead to substantial decrease in memory requirements and overall inference complexity. First, a pen sketches are spatially sparse and can be processed efficiently by recently introduced submanifold sparse CNNs BID4. Second, surveillance cameras with mostly static input contain temporal sparsity that can be addressed by Sigma-Delta networks BID15. A more general scenario presumes a dense input e.g. video frames from a high-resolution camera mounted on a moving autonomous car. In this work, we address the latter scenario and concentrate on feature map compression in order to minimize memory footprint and bandwidth during DNN inference which might be prohibitive for high-resolution cameras. We propose a method to convert intermediate fixed-point feature map activations into vectors over the smallest finite field called the Galois field of two elements (GF) or, simply, binary vectors followed by compression convolutional layers using a nonlinear dimensionality reduction (NDR) technique embedded into DNN architecture. The compressed feature maps can then be projected back to a higher cardinality representation over a fixed-point (integer) field using decompression convolutional layers. Using a layer fusion method, only the compressed feature maps need to be kept for inference while adding only computationally inexpensive bitwise operations. Compression and decompression layers over GF can be repeated within the proposed network architecture and trained in an end-to-end fashion. In brief, the proposed method resembles autoencoder-type BID7 structures embedded into a base network that work over GF. Binary conversion and compression-decompression layers are implemented in the Caffe BID12 framework and publicly available 1.The rest of the paper is organized as follows. Section 2 reviews related work. Section 3 gives notation for convolutional layers, describes conventional fusion and NDR methods, and explains the proposed method including details about network training and the derived architecture. Section 4 presents experimental on ImageNet classification and PASCAL VOC object detection using SSD BID13, memory requirements, and obtained compression rates. Feature Map Compression using Quantization. Unlike a weight compression, surprisingly few papers consider feature map compression. This can most likely be explained by the fact that feature maps have to be compressed for every network input as opposed to offline weight compression. Previous feature map compression methods are primarily developed around the idea of representation approximation using a certain quantization scheme: fixed-point quantization BID2 BID5, binary quantization BID10 BID17 BID20 BID19, and power-of-two quantization BID14. The base floatingpoint network is converted to the approximate quantized representation and, then, the quantized network is retrained to restore accuracy. Such methods are inherently limited in finding more compact representations since the base architecture remains unchanged. For example, the dynamic fixed-point scheme typically requires around 8-bits of resolution to achieve baseline accuracy for state-of-the-art network architectures. At the same time, binary networks experience significant accuracy drops for large-scale datasets or compact (not over-parametrized) network architectures. Instead, our method can be considered in a narrow sense as a learned quantization using binary representation. Embedded NDR Layers. Another interesting approach is implicitly proposed by BID11. Although the authors emphasized weight compression rather than feature map compression, they introduced NDR-type layers into network architecture that allowed to decrease not only the number of weights but also feature map sizes by a factor of 8, if one keeps only the outputs of socalled "squeeze" layers. The latter is possible because such network architecture does not introduce any additional convolution recomputations since "squeeze" layer computations with a 1×1 kernel can be fused with the preceding "expand" layers. Our work goes beyond this approach by extending NDR-type layers to work over GF to find a more compact feature map representation. Hardware Accelerator Architectures. BID8 estimated that off-chip DRAM access requires approximately 100× more power than local on-chip cache access. Therefore, currently proposed DNN accelerator architectures propose various schemes to decrease memory footprint and bandwidth. One obvious solution is to keep only a subset of intermediate feature maps at the expense of recomputing convolutions BID0. The presented fusion approach seems to be oversimplified but effective due to high memory access cost. Our approach is complementary to this work but proposes to keep only compressed feature maps with minimum additional computations. Another recent work BID16 exploits weight and feature map sparsity using a more efficient encoding for zeros. While this approach targets similar goals, it requires having high sparsity, which is often unavailable in the first and the largest feature maps. In addition, a special control and encoding-decoding logic decrease the benefits of this approach. In our work, compressed feature maps are stored in a dense form without the need of special control and enconding-decoding logic. The input feature map of lth convolutional layer in commonly used DNNs can be represented by a tensor X l−1 ∈ RĆ ×H×W, whereĆ, H and W are the number of input channels, the height and the width, respectively. The input X l−1 is convolved with a weight tensor W l ∈ R C×Ć×H f ×W f, where C is the number of output channels, H f and W f are the height and the width of filter kernel, respectively. A bias vector b ∈ R C is added to the of convolution operation. Once all C channels are computed, an element-wise nonlinear function is applied to the of the convolution operations. Then, the cth channel of the output tensor X l ∈ R C×H×W can be computed as DISPLAYFORM0 where * denotes convolution operation and g is some nonlinear function. In this paper, we assume g is the most commonly used rectified linear unit (ReLU) defined as g(x) = max (0, x) such that all activations are non-negative. We formally describe previously proposed methods briefly reviewed in Section 2 using the unified model illustrated in FIG0. To simplify notation, biases are not shown. Consider a network built using multiple convolutional layers and processed according to. Similar to BID0, calculation of N sequential layers can be fused together without storing intermediate feature maps DISPLAYFORM0 For example, fusion can be done in a channel-wise fashion using memory buffers which are much smaller than the whole feature map. Then, feature map X l ∈ R can be quantized intoX l ∈ Q using a nonlinear quantization function q where Q is a finite field over integers. The quantization step may introduce a drop in accuracy due to imperfect approximation. The network can be further finetuned to restore some of the original accuracy BID2 BID5. The network architecture is not changed after quantization and feature maps can be compressed only up to a certain suboptimal bitwidth resolution. The next step implicitly introduced by Iandola et al. FORMULA0 is to perform NDR using an additional convolutional layer. A mappingX DISPLAYFORM1 ×H×W can be performed using projection weights P l ∈ RC ×C×H f ×W f, where the output channel dimensionC < C. Then, only compressed feature mapŶ l needs to be stored in the memory buffer. During the inverse steps, the compressed feature map can be projected back onto the higher-dimensional tensorX l+1 ∈ Q using weights R l ∈ R C×C×H f ×W f and, lastly, converted back to X l+1 ∈ R using an inverse quantization function q −1 . In the case of a fully quantized network, the inverse quantization can be omitted. In practice, the number of bits for the feature map quantization step depends on the dataset, network architecture and desired accuracy. For example, over-parameterized architecture like AlexNet may require only 1 or 2 bits for small-scale datasets (CIFAR-10, MNIST, SVHN), but experience significant accuracy drops for large-scale datasets like ImageNet. In particular, the modified AlexNet (with the first and last layers kept in full-precision) top-1 accuracy is degraded by 12.4% and 6.8% for 1-bit XNOR-Net BID17 and 2-bit DoReFa-Net BID20, respectively. At the same time, efficient network architectures e.g. BID11 using NDR layers require 6-8 bits for the fixed-point quantization scheme on ImageNet and fail to work with lower precision activations. In this paper, we follow the path to select an efficient base network architecture and then introduce additional compression layers to obtain smaller feature maps as opposed to initially selecting an over-parametrized network architecture for quantization. Consider a scalar x from X l ∈ R. A conventional feature map quantization step can be represented as a scalar-to-scalar mapping or a nonlinear functionx = q(x) such that DISPLAYFORM0 wherex is the quantized scalar, Q is the GF(2 B) finite field for fixed-point representation and B is the number of bits. We can introduce a newx representation by a linear binarization function b defined bŷ DISPLAYFORM1 where ⊗ is a bitwise AND operation, DISPLAYFORM2 An inverse linear function b −1 can be written as DISPLAYFORM3 Equations FORMULA4 - FORMULA6 show that a scalar over a higher cardinality finite field can be linearly converted to and from a vector over a finite field with two elements. Based on these derivations, we propose a feature map compression method shown in Figure 2. Similar to BID2, we quantize activations to obtainX l and, then, apply transformation. The ing feature map can be represented asX DISPLAYFORM4 B×C×H×W. For implementation convenience, a new bit dimension can be concatenated along channel dimension ing in the feature mapX l ∈ B BC×H×W. Next, a single convolutional layer using weights P l or a sequence of layers with P l i weights can be applied to obtain a compressed representation over GF. Using the fusion technique, only the compressed feature mapsỸ l ∈ B need to be stored in memory during inference. Non-compressed feature maps can be processed using small buffers e.g. in a sequential channel-wise fashion. Lastly, the inverse function b −1 from using convolutional layers R l i and inverse of quantization q −1 undo the compression and quantization steps. Figure 2: Scheme of the proposed method. The graph model shown in FIG1 explains details about the inference (forward pass) and backpropagation (backward pass) phases of the newly introduced functions. The inference phase represents- as explained above. Clearly, the backpropagation phase may seem not obvious at a first glance. One difficulty related to the quantized network is that quantization function itself is not differentiable. But many studies e.g. BID2 show that a mini-batch-averaged floating-point gradient practically −1 can be represented as gates that make hard decisions similar to ReLU. The gradient of b −1 can then be calculated using of BID1 aŝ DISPLAYFORM0 Lastly, the gradient of b is just a scaled sum of the gradient vector calculated bỹ DISPLAYFORM1 where x 0 is a gradient scaling factor that represents the number of nonzero elements inx. Practically, the scaling factor can be calculated based on statistical information only once and used as a static hyperparameter for gradient normalization. Since the purpose of the network is to learn and keep only the smallestỸ l, the choice of P l and R l initialization is important. Therefore, we can initialize these weight tensors by an identity function that maps the non-compressed feature map to a truncated compressed feature map and vice versa. That provides a good starting point for training. At the same time, other initializations are possible e.g. noise sampled from some distribution studied by BID1 can be added as well. To highlight benefits of the proposed approach, we select a base network architecture with the compressed feature maps according to Section 3.2. The selected architecture is based on a quantized SqueezeNet network. It consists of a sequence of "fire" modules where each module contains two concatenated "expand" layers and a "squeeze" layer illustrated in FIG2. The "squeeze" layers perform NDR over the field of real or, in case of the quantized model, integer numbers. Specifically, the size of concatenated "expand 1×1" and "expand 3×3" layers is compressed by a factor of 8 along channel dimension by "squeeze 1×1" layer. Activations of only the former one can be stored during inference according to the fusion method. According to the analysis presented in BID5, activations quantized to 8-bit integers do not experience significant accuracy drop. The given base architecture is extended by the proposed method. The quantized "squeeze" layer feature map is converted to its binary representation following Figure 2. Then, the additional compression rate is defined by selecting parameters of P l i. In the simplest case, only a single NDR layer can be introduced with the weights P l. In general, a number of NDR layers can be added with 1×1, 3×3 and other kernels with or without pooling at the expense of increased computational cost. For example, 1×1 kernels allow to learn optimal quantization and to compensate redundancies along channel dimension only. But 3×3 kernels can address spatial redundancies and, while being implemented with stride 2 using convolutional-deconvolutional layers, decrease feature map size along spatial dimensions. We implemented the new binarization layers from Section 3 as well as quantization layers using modified BID5 code in the Caffe BID12 framework. The latter code is modified to accurately support binary quantization during inference and training. SqueezeNet v1.1 is selected as a base floating-point network architecture, and its pretrained weights were downloaded from the publicly available source 2. In the experiments, we compress the "fire2/squeeze" and "fire3/squeeze" layers which are the largest of the "squeeze" layers due to high spatial dimensions. The input to the network has a resolution of 227×227, and the weights are all floating-point. The quantized and compressed models are retrained for 100,000 iterations with a mini-batch size of 1024 on the ImageNet BID18 (ILSVRC2012) training dataset, and SGD solver with a step-policy learning rate starting from 1e-3 divided by 10 every 20,000 iterations. Although this large mini-batch size was used by the original model, it helps the quantized and compressed models to estimate gradients as well. The compressed models were derived and retrained iteratively from the 8-bit quantized model. TAB0 reports top-1 and top-5 inference accuracies of 50,000 images from ImageNet validation dataset. The fourth column in TAB0 represents speed in terms of frames per second (FPS) of network forward pass with mini-batch size of 1 using NVIDIA Tesla P100 GPU. All quantized computations are emulated on GPU using fp32. Hence, it shows a measure of emulation speed for additional layers rather than speed of the optimized layers. According to TAB0, the quantized models after retraining experience -0.2%, 0.6% and 3.5% top-1 accuracy drops for 8-bit, 6-bit and 4-bit quantization, respectively. For comparison, the quantized models without retraining are shown in parentheses. The proposed compression method using 1 × 1 kernels allows us to restore corresponding top-1 accuracy by 1.0% and 2.4% for 6-bit and 4-bit versions at the expense of a small increase in the number of weights and bitwise convolutions. Moreover, we evaluated a model with a convolutional layer followed by a deconvolutional layer both with a 3 × 3 stride 2 kernel at the expense of a 47% increase in weight size for 6-bit activations. That allowed us to decrease feature maps in spatial dimensions by exploiting local spatial quantization redundancies. Then, the feature map size is further reduced by a factor of 4, while top-1 accuracy dropped by 4.3% and 4.6% for 8-bit and 6-bit activations, respectively. A comprehensive comparison with the state-of-the-art networks is given in Appendix A. Accuracy experiments. We evaluate object detection using Pascal VOC BID3 dataset which is a more realistic application for autonomous cars where the high-resolution cameras emphasize feature map compression benefits. The VOC2007 test dataset contains 4,952 images and a training dataset of 16,551 images is a union of VOC2007 and VOC2012. We adopted SSD512 model BID13 for the proposed architecture. SqueezeNet pretrained on ImageNet is used as a feature extractor instead of the original VGG-16 network. This reduces number of parameters and overall inference time by a factor of 4 and 3, respectively. The original VOC images are rescaled to 512×512 resolution. As with ImageNet experiments, we generated several models for comparisons: a base floating-point model, quantized models, and compressed models. We apply quantization and compression to the "fire2/squeeze" and "fire3/squeeze" layers which represent, if the fusion technique is applied, more than 80% of total feature map memory due to their large spatial dimensions. Typically, spatial dimensions decrease quadratically because of max pooling layers compared to linear growth in the depth dimension. The compressed models are derived from the 8-bit quantized model, and both are retrained for 10,000 mini-batch-256 iterations using SGD solver with a step-policy learning rate starting from 1e-3 divided by 10 every 2,500 iterations. TAB1 presents mean average precision (mAP) for the original VGG-16 and SqueezeNetbased models as well as size of the weights, feature maps to compress and the speed metric with Section 4.1 assumptions. The 8-bit quantized model with retraining drops accuracy by less than 0.04%, while 6-bit, 4-bit and 2-bit models decrease accuracy by 0.5%, 2.2% and 12.3%, respectively. For reference, mAPs for the quantized models without retraining are shown in parentheses. Using the proposed compression-decompression layers with a 1×1 kernel, mAP for the 6-bit model is increased by 0.5% and mAP for the 4-bit is decreased by 0.5%. We conclude that compression along channel dimension is not beneficial for SSD unlike ImageNet classification either due to low quantization redundancy in that dimension or the choice of hyper-parameters e.g. mini-batch size. Then, we evaluate the models with spatial-dimension compression which is intuitively appealing for high-resolution images. Empirically, we found that a 2×2 kernel with stride 2 performs better than a corresponding 3×3 kernel while requiring less parameters and computations. According to TAB1, an 8-bit model with 2×2 kernel and downsampling-upsampling layers achieves 1% higher mAP than a model with 3×3 kernel and only 3.7% lower than the base floating-point model. Memory Requirements. TAB2 summarizes memory footprint benefits for the evaluated SSD models. Similar to the previous section, we consider only the largest feature maps that represent more than 80% of total activation memory. Assuming that the input frame is stored separately, the fusion technique allows to compress feature maps by a factor of 19. Note that no additional recomputations are needed. Second, conventional 8-bit and 4-bit fixed-point models decrease the size of feature maps by a factor of 4 and 8, respectively. Third, the proposed model with 2×2 stride 2 kernel gains another factor of 2 compression compared to 4-bit fixed-point model with only 1.5% degradation in mAP. This is similar to ImageNet experiments which showed relatively limited compression gain along channel dimension only. At the same time, learned quantization along combined channel and spatial dimensions pushes further compression gain. In total, the memory footprint for this feature extractor is reduced by two orders of magnitude. We introduced a method to decrease memory storage and bandwidth requirements for DNNs. Complementary to conventional approaches that use fused layer computation and quantization, we presented an end-to-end method for learning feature map representations over GF within DNNs. Such a binary representation allowed us to compress network feature maps in a higher-dimensional space using autoencoder-inspired layers embedded into a DNN along channel and spatial dimensions. These compression-decompression layers can be implemented using conventional convolutional layers with bitwise operations. To be more precise, the proposed representation traded cardinality of the finite field with the dimensionality of the vector space which makes possible to learn features at the binary level. The evaluated compression strategy for inference can be adopted for GPUs, CPUs or custom accelerators. Alternatively, existing binary networks can be extended to achieve higher accuracy for emerging applications such as object detection and others. We compare recently reported ImageNet for networks that compress feature maps as well as several configurations of the proposed approach. Most of the works use the over-parametrized AlexNet architecture while ours is based on SqueezeNet. TAB3 shows accuracy for base networks as well as their quantized versions. Binary XNOR-Net BID17 estimates based on AlexNet as well as ResNet-18. DoReFa-Net BID20 ) is more flexible and can adjust the number of bits for weights and activations. Since its accuracy is limited by the number of activation bits, we present three cases with 1-bit, 2-bit, and 4-bit activations. The most recent work BID19 solves the problem of binarizing the last layer weights, but weights of the first layer are full-precision. Overall, AlexNet-based low-precision networks achieve 43.6%, 49.8%, 53.0% top-1 accuracy for 1-bit, 2-bit and 4-bit activations, respectively. Around 70% of the memory footprint is defined by the first two layers of AlexNet. The fusion technique is difficult in such architectures due to large kernel sizes (11×11 and 5×5 for AlexNet) which can cause extra recomputations. Thus, activations require 95.4KB, 190.0KB and 381.7KB of memory for 1-bit, 2-bit and 4-bit models, respectively. The NIN-based network from BID19 with 2-bit activations achieves 51.4% top-1 accuracy, but its activation memory is larger than AlexNet due to late pooling layers. BID17 whether the activations were binarized or not for the first and the last layer. 3 Weights are not binarized for the first layer. 4 Number of bits for the compressed "fire2,3/squeeze" layers and, in parentheses, for quantized only layers. 5 For comparison, accuracy in parentheses represents for the corresponding model in TAB0 The SqueezeNet-based models in TAB3 are finetuned from the corresponding models in TAB0 for 40,000 iterations with a mini-batch size of 1024, and SGD solver with a step-policy learning rate starting from 1e-4 divided by 10 every 10,000 iterations. The model with fusion and 8-bit quantized weights and activations, while having an accuracy similar to floating-point model, outperforms the state-of-the-art networks in terms of weight and activation memory. The proposed four models from TAB0 further decrease activation memory by adding compression-decompression layers to "fire2,3/squeeze" modules. This step allowed to shrink memory from 189.9KB to 165.4KB, 140.9KB, 116.4KB and 110.3KB depending on the compression configuration. More compression is possible, if apply the proposed approach to other "squeeze" layers.

represent the tldr for finding the corresponding scientific article

Feature map compression method that converts quantized activations into binary vectors followed by nonlinear dimensionality reduction layers embedded into a DNN

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose accelerating Batch Normalization (BN) through sampling less correlated data for reduction operations  with regular execution pattern, which achieves up to 2x and 20% speedup for BN itself and the overall training, respectively.