Question: Represent the scientific article to find a short TLDR

Question: For fast and energy-efficient deployment of trained deep neural networks on resource-constrained embedded hardware, each learned weight parameter should ideally be represented and stored using a single bit. Error-rates usually increase when this requirement is imposed. Here, we report large improvements in error rates on multiple datasets, for deep convolutional neural networks deployed with 1-bit-per-weight. Using wide residual networks as our main baseline, our approach simplifies existing methods that binarize weights by applying the sign function in training; we apply scaling factors for each layer with constant unlearned values equal to the layer-specific standard deviations used for initialization. For CIFAR-10, CIFAR-100 and ImageNet, and models with 1-bit-per-weight requiring less than 10 MB of parameter memory, we achieve error rates of 3.9%, 18.5% and 26.0% / 8.5% (Top-1 / Top-5) respectively. We also considered MNIST, SVHN and ImageNet32, achieving 1-bit-per-weight test of 0.27%, 1.9%, and 41.3% / 19.1% respectively. For CIFAR, our error rates halve previously reported values, and are within about 1% of our error-rates for the same network with full-precision weights. For networks that overfit, we also show significant improvements in error rate by not learning batch normalization scale and offset parameters. This applies to both full precision and 1-bit-per-weight networks. Using a warm-restart learning-rate schedule, we found that training for 1-bit-per-weight is just as fast as full-precision networks, with better accuracy than standard schedules, and achieved about 98%-99% of peak performance in just 62 training epochs for CIFAR-10/100. For full training code and trained models in MATLAB, Keras and PyTorch see https://github.com/McDonnell-Lab/1-bit-per-weight/. Fast parallel computing resources, namely GPUs, have been integral to the resurgence of deep neural networks, and their ascendancy to becoming state-of-the-art methodologies for many computer vision tasks. However, GPUs are both expensive and wasteful in terms of their energy requirements. They typically compute using single-precision floating point (32 bits), which has now been recognized as providing far more precision than needed for deep neural networks. Moreover, training and deployment can require the availability of large amounts of memory, both for storage of trained models, and for operational RAM. If deep-learning methods are to become embedded in resourceconstrained sensors, devices and intelligent systems, ranging from robotics to the internet-of-things to self-driving cars, reliance on high-end computing resources will need to be reduced. To this end, there has been increasing interest in finding methods that drive down the resource burden of modern deep neural networks. Existing methods typically exhibit good performance but for the ideal case of single-bit parameters and/or processing, still fall well-short of state-of-the-art error rates on important benchmarks. In this paper, we report a significant reduction in the gap (see Figure 1 and Results) between Convolutional Neural Networks (CNNs) deployed using weights stored and applied using standard precision (32-bit floating point) and networks deployed using weights represented by a single-bit each. In the process of developing our methods, we also obtained significant improvements in error-rates obtained by full-precision versions of the CNNs we used. In addition to having application in custom hardware deploying deep networks, networks deployed using 1-bit-per-weight have previously been shown BID21 to enable significant speedups on regular GPUs, although doing so is not yet possible using standard popular libraries. Aspects of this work was first communicated as a subset of the material in a workshop abstract and talk BID19. Figure 1: Our error-rate gaps between using full-precision and 1-bit-per-weight. All points except black crosses are data from some of our best reported in this paper for each dataset. Black points are on the full ImageNet dataset, in comparison with of BID22 (black crosses). The notation 4x, 10x and 15x corresponds to network width (see Section 4).1.1 RELATED WORK In 2015, a new form of CNN called a "deep residual network," or "ResNet" BID8 was developed and used to set many new accuracy records on computer-vision benchmarks. In comparison with older CNNs such as Alexnet BID14 and VGGNet BID24, ResNets achieve higher accuracy with far fewer learned parameters and FLOPs (FLoating-point OPerations) per image processed. The key to reducing the number of parameters in ResNets was to replace "all-to-all layers" in VGG-like nets with "global-average-pooling" layers that have no learned parameters BID17 BID25, while simultaneously training a much deeper network than previously. The key new idea that enabled a deeper network to be trained effectively was the introduction of so-called "skip-connections" BID8. Many variations of ResNets have since been proposed. ResNets offer the virtue of simplicity, and given the motivation for deployment in custom hardware, we have chosen them as our primary focus. Despite the increased efficiency in parameter usage, similar to other CNNs the accuracy of ResNets still tends to increase with the total number of parameters; unlike other CNNs, increased accuracy can either from deeper or wider networks.In this paper, we use Wide Residual Networks, as they have been demonstrated to produce better accuracy in less training time than deeper networks. Achieving the best accuracy and speed possible when deploying ResNets or similar networks on hardware-constrained mobile devices will require minimising the total number of bits transferred between memory and processors for a given number of parameters. Motivated by such considerations, a lot of recent attention has been directed towards compressing the learned parameters (model compression) and reducing the precision of computations carried out by neural networks-see BID12 for a more detailed literature review. Recently published strategies for model compression include reducing the precision (number of bits used for numerical representation) of weights in deployed networks by doing the same during training BID2 BID12 BID20 BID22, reducing the number of weights in trained neural networks by pruning BID6 BID13, quantizing or compressing weights following training BID6 BID29, reducing the precision of computations performed in forward-propagation during inference BID2 BID12 BID20 BID22, and modifying neural network architectures BID10. A theoretical analysis of various methods proved on the convergence of a variety of weight-binarization methods BID16.From this range of strategies, we are focused on an approach that simultaneously contributes two desirable attributes: simplicity, in the sense that deployment of trained models immediately follows training without extra processing; implementation of convolution operations can be achieved without multipliers, as demonstrated by BID22. Our strategy for improving methods that enable inference with 1-bit-per-weight was threefold:1. State-of-the-art baseline. We sought to begin with a baseline full-precision deep CNN variant with close to state-of-the-art error rates. At the time of commencement in 2016, the state-of-the-art on CIFAR-10 and CIFAR-100 was held by Wide Residual Networks, so this was our starting point. While subsequent approaches have exceeded their accuracy, ResNets offer superior simplicity, which conforms with our third strategy in this list.2. Make minimal changes when training for 1-bit-per-weight. We aimed to ensure that training for 1-bit-per-weight could be achieved with minimal changes to baseline training.3. Simplicity is desirable in custom hardware. With custom hardware implementations in mind, we sought to simplify the design of the baseline network (and hence the version with 1-bit weights) as much as possible without sacrificing accuracy. Although this paper is chiefly about 1-bit-per-weight, we exceeded our objectives for the fullprecision baseline network, and surpassed reported error rates for CIFAR-10 and CIFAR-100 using Wide ResNets. This was achieved using just 20 convolutional layers; most prior work has demonstrated best wide ResNet performance using 28 layers. Our innovation that achieves a significant error-rate drop for CIFAR-10 and CIFAR-100 in Wide ResNets is to simply not learn the per-channel scale and offset factors in the batch-normalization layers, while retaining the remaining attributes of these layers. It is important that this is done in conjunction with exchanging the ordering of the final weight layer and the global average pooling layer (see FIG2 .We observed this effect to be most pronounced for CIFAR-100, gaining around 3% in test-error rate. But the method is advantageous only for networks that overfit; when overfitting is not an issue, such as for ImageNet, removing learning of batch-norm parameters is only detrimental. Ours is the first study we are aware of to consider how the gap in error-rate for 1-bit-per-weight compared to full-precision weights changes with full-precision accuracy across a diverse range of image classification datasets (Figure 1).Our approach surpasses by a large margin all previously reported error rates for CIFAR-10/100 (error rates halved), for networks constrained to run with 1-bit-per-weight at inference time. One reason we have achieved lower error rates for the 1-bit case than previously is to start with a superior baseline network than in previous studies, namely Wide ResNets. However, our approach also in smaller error rate increases relative to full-precision error rates than previously, while training requires the same number of epochs as for the case of full-precision weights. Our main innovation is to introduce a simple fixed scaling method for each convolutional layer, that permits activations and gradients to flow through the network with minimum change in standard deviation, in accordance with the principle underlying popular initialization methods BID7. We combine this with the use of a warm-restart learning-rate method BID18 ) that enables us to report close-to-baseline for the 1-bit case in far fewer epochs of training than reported previously. We follow the approach of BID2; BID22; BID20, in that we find good when using 1-bit-per-weight at inference time if during training we apply the sign function to real-valued weights for the purpose of forward and backward propagation, but update full-precision weights using SGD with gradients calculated using full-precision. However, previously reported methods for training using the sign of weights either need to train for many hundreds of epochs BID2 BID20, or use computationallycostly normalization scaling for each channel in each layer that changes for each minibatch during training, i.e. the BWN method of BID22. We obtained our using a simple alternative approach, as we now describe. We begin by noting that the standard deviation of the sign of the weights in a convolutional layer with kernels of size F × F will be close to 1, assuming a mean of zero. In contrast, the standard deviation of layer i in full-precision networks is initialized in the method of BID7 to 2/(F 2 C i−1), where C i−1 is the number of input channels to convolutional layer i, and i = 1,.., L, where L is the number of convolutional layers and C 0 = 3 for RGB inputs. When applying the sign function alone, there is a mismatch with the principled approach to controlling gradient and activation scaling through a deep network BID7. Although the use of batch-normalization can still enable learning, convergence is empirically slow and less effective. To address this problem, for training using the sign of weights, we use the initialization method of BID7 for the full-precision weights that are updated, but also introduce a layer-dependent scaling applied to the sign of the weights. This scaling has a constant unlearned value equal to the initial standard deviation of 2/(F 2 C i−1) from the method of BID7. This enables the standard deviation of forward-propagating information to be equal to the value it would have initially in full-precision networks. In implementation, during training we multiply the sign of the weights in each layer by this value. For inference, we do this multiplication using a scaling layer following the weight layer, so that all weights in the network are stored using 0 and 1, and deployed using ±1 (see https://github.com/McDonnell-Lab/1-bit-per-weight/). Hence, custom hardware implementations would be able to perform the model's convolutions without multipliers BID22, and significant GPU speedups are also possible BID21.The fact that we scale the weights explicitly during training is important. Although for forward and backward propagation it is equivalent to scale the input or output feature maps of a convolutional layer, doing so also scales the calculated gradients with respect to weights, since these are calculated by convolving input and output feature maps. As a consequence, learning is dramatically slower unless layer-dependent learning rates are introduced to cancel out the scaling. Our approach to this is similar to the BWN method of BID22, but our constant scaling method is faster and less complex. In summary, the only differences we make in comparison with full-precision training are as follows. Let W i be the tensor for the convolutional weights in the i-th convolutional layer. These weights are processed in the following way only for forward propagation and backward propagation, not for weight updates: DISPLAYFORM0 where F i is the spatial size of the convolutional kernel in layer i; see FIG1. Our ResNets use the'post-activation' and identity mapping approach of for residual connections. For ImageNet, we use an 18-layer design, as in BID8. For all other datasets we mainly use a 20-layer network, but also report some for 26 layers. Each residual block includes two convolutional layers, each preceded by batch normalization (BN) and Rectified Linear Unit (ReLU) layers. Rather than train very deep ResNets, we use wide residual networks (Wide ResNets). Although and others reported that 28/26-layer networks in better test accuracy than 22/20-layer 1 networks, we found for CIFAR-10/100 that just 20 layers typically produces best , which is possibly due to our approach of not learning the batch-norm scale and offset parameters. Our baseline ResNet design used in most of our experiments (see Figures 3 and 4) has several differences in comparison to those of;. These details are articulated in Appendix A, and are mostly for simplicity, with little impact on accuracy. The exception is our approach of not learning batch-norm parameters. We trained our models following, for most aspects, the standard stochastic gradient descent methods used by for Wide ResNets. Specifically, we use cross-entropy loss, minibatches of size 125, and momentum of 0.9 (both for learning weights, and in situations where we learn batch-norm scales and offsets). For CIFAR-10/100, SVHN and MNIST, where overfitting is evident in Wide ResNets, we use a larger weight decay of 0.0005. For ImageNet32 and full ImageNet we use a weight decay of 0.0001. Apart from one set of experiments where we added a simple extra approach called cutout, we use standard'light' data augmentation, including randomly flipping each image horizontally with probability 0.5 for CIFAR-10/100 and ImageNet32. For the The design is mostly a standard pre-activation ResNet. The first (stand-alone) convolutional layer ("conv") and first 2 or 3 residual blocks have 64k (ImageNet) or 16k (other datasets) output channels. The next 2 or 3 blocks have 128k or 32k output channels and so on, where k is the widening parameter. The final (stand-alone) convolutional layer is a 1 × 1 convolutional layer that gives N output channels, where N is the number of classes. Importantly, this final convolutional layer is followed by batch-normalization ("BN") prior to global-average-pooling ("GAP") and softmax ("SM"). The blocks where the number of channels double are downsampling blocks (details are depicted in FIG4) that reduce each spatial dimension in the feature map by a factor of two. The rectified-linearunit ("RELU") layer closest to the input is optional, but when included, it is best to learn the BN scale and offset in the subsequent layer., downsampling (stride-2 convolution) is used in the convolutional layers where the number of output channels increases. The corresponding downsampling for skip connections is done in the same residual block. Unlike standard pre-activation ResNets we use an average pooling layer ("avg pool") in the residual path when downsampling. same 3 datasets plus SVHN, we pad by 4 pixels on all sides (using random values between 0 and 255) and crop a 32 × 32 patch from a random location in the ing 40 × 40 image. For full ImageNet, we scale, crop and flip, as in BID8. We did not use any image-preprocessing, i.e. we did not subtract the mean, as the initial BN layer in our network performs this role, and we did not use whitening or augment using color or brightness. We use the initialization method of BID7.We now describe important differences in our training approach compared to those usually reported. When training on CIFAR-10/100 and SVHN, in all batch-norm layers (except the first one at the input when a ReLU is used there), we do not learn the scale and offset factor, instead initializating these to 1 and 0 in all channels, and keeping those values through training. Note that we also do not learn any biases for convolutional layers. The usual approach to setting the moments for use in batch-norm layers in inference mode is to keep a running average through training. When not learning batch-normalization parameters, we found a small benefit in calculating the batch-normalization moments used in inference only after training had finished. We simply form as many minibatches as possible from the full training-set, each with the same data augmentation as used during training applied, and pass these through the trained network, averaging the returned moments for each batch. For best when using this method using matconvnet, we found it necessary to ensure the parameter that is used to avoid divisions by zero is set to 1 × 10 −5; this is different to the way it is used in keras and other libraries. FORMULA0 is that we exchange the ordering of the global average pooling layer and the final weight layer, so that our final weight layer becomes a 1 × 1 convolutional layer with as many channels as there are classes in the training set. This design is not new, but it does seem to be new to ResNets: it corresponds to the architecture of BID17, which originated the global average pooling concept, and also to that used by BID25.As with all other convolutional layers, we follow this final layer with a batch-normalization layer; the benefits of this in conjunction with not learning the batch-normalization scale and offset are described in the Discussion section. We use a'warm restarts' learning rate schedule that has reported state-of-the-art Wide ResNet BID18 whilst also speeding up convergence. The method constantly reduces the learning rate from 0.1 to 1 × 10 −4 according to a cosine decay, across a certain number of epochs, and then repeats across twice as many epochs. We restricted our attention to a maximum of 254 epochs (often just 62 epochs, and no more than 30 for ImageNet32) using this method, which is the total number of epochs after reducing the learning rate from maximum to minimum through 2 epochs, then 4, 8, 16, 32, 64 and 128 epochs. For CIFAR-10/100, we typically found that we could achieve test error rates after 32 epochs within 1-2% of the error rates achievable after 126 or 254 epochs. In the literature, most experiments with CIFAR-10 and CIFAR-100 use simple "standard" data augmentation, consisting of randomly flipping each training image left-right with probability 0.5, and padding each image on all sides by 4 pixels, and then cropping a 32 × 32 version of the image from a random location. We use this augmentation, although with the minor modification that we pad with uniform random integers between 0 and 255, rather than zero-padding. Additionally, we experimented with "cutout" BID3. This involves randomly selecting a patch of each raw training image to remove. The method was shown to combine with other state-of-the-art methods to set the latest state-of-the-art on CIFAR-10/100 (see TAB0). We found better using larger cutout patches for CIFAR-100 than those reported by BID3; hence for both CIFAR-10 and CIFAR-100 we choose patches of size 18 × 18. Following the method of BID3, we ensured that all pixels are chosen for being included in a patch equally frequently throughout training by ensuring that if the chosen patch location is near the image border, the patch impacts on the image only for the part of the patch inside the image. Differently to BID3, as for our padding, we use uniform random integers to replace the image pixel values in the location of the patches. We did not apply cutout to other datasets. We conducted experiments on six databases: four databases of 32×32 RGB images-CIFAR-10, CIFAR-100, SVHN and ImageNet32-and the full ILSVRC ImageNet database BID23, as well as MNIST BID15. Details of the first three 32×32 datasets can be found in many papers, e.g.. ImageNet32 is a downsampled version of ImageNet, where the training and validation images are cropped using their annotated bounding boxes, and then downsampled to 32 × 32 BID1; see http://image-net.org/download-images. All experiments were carried out on a single GPU using MATLAB with GPU acceleration from MatConvNet and cuDNN.We report for Wide ResNets, which (except when applied to ImageNet) are 4× and 10× wider than baseline ResNets, to use the terminology of, where the baseline has 16 channels in the layers at the first spatial scale. We use notation of the form 20-10 to denote Wide ResNets with 20 convolutional layers and 160 channels in the first spatial scale, and hence width 10×. For the full ImageNet dataset, we use 18-layer Wide ResNets with 160 channels in the first spatial scale, but given the standard ResNet baseline is width 64, this corresponds to width 2.5× on this dataset. We denote these networks as 18-2.5. TAB0 lists our top-1 error rates for CIFAR-10/100; C10 indicates CIFAR-10, C100 indicates CIFAR-100; the superscript + indicates standard crop and flip augmentation; and ++ indicates the use of cutout. TAB1 lists error rates for SVHN, ImageNet32 (I32) and full ImageNet; we did not use cutout on these datasets. Both top-1 and top-5 are tabulated for I32 and ImageNet. For ImageNet, we provide for single-center-crop testing, and also for multi-crop testing. In the latter, the decision for each test image is obtained by averaging the softmax output after passing through the network 25 times, corresponding to crops obtained by rescaling to 5 scales as described by BID8, and from 5 random positions at each scale. Our full-precision ImageNet error rates are slightly lower than expected for a wide ResNet according to the of, probably due to the fact we did not use color augmentation. TAB2 shows comparison from the original work on Wide ResNets, and subsequent papers that have reduced error rates on the CIFAR-10 and CIFAR-100 datasets. We also show the only , to our knowledge, for ImageNet32. The current state-of-the-art for SVHN without augmentation is 1.59% BID11, and with cutout augmentation is 1.30% BID3. Our full-precision for SVHN (1.75%) is only a little short of these even though we used only a 4× ResNet, with less than 5 million parameters, and only 30 training epochs. Table 4 shows comparison for previous work that trains models by using the sign of weights during training. Additional appear in BID12, where activations are quantized, and so the error rates are much larger. Inspection of Tables 1 to 4 reveals that our baseline full-precision 10× networks, when trained with cutout, surpass the performance of deeper Wide ResNets trained with dropout. Even without the use Table 4: Test error rates using 1-bit-per-weight at test time and propagation during training. Method C10 C100 SVHN ImageNet BC BID2 8.27 -2.30 -Weight binarization BID20 8.25 ---BWN -Googlenet BID22 9.88 --34.5 / 13.9 (full ImageNet) VGG+HWGQ BID0 7.49 ---BC with ResNet + ADAM BID16 of cutout, our 20-10 network surpasses by over 2% the CIFAR-100 error rate reported for essentially the same network by and is also better than previous Wide ResNet on CIFAR-10 and ImageNet32. As elaborated on in Section 5.3, this improved accuracy is due to our approach of not learning the batch-norm scale and offset parameters. For our 1-bit networks, we observe that there is always an accuracy gap compared to full precision networks. This is discussed in Section 5.1.Using cutout for CIFAR-10/100 reduces error rates as expected. In comparison with training very wide 20× ResNets on CIFAR-100, as shown in TAB0, it is more effective to use cutout augmentation in the 10× network to reduce the error rate, while using only a quarter of the weights. Clearly, even for width-4 ResNets, the gap in error rate between full precision weights and 1-bit-per-weight is small. Also noticeable is that the warm-restart method enables convergence to very good solutions after just 30 epochs; training longer to 126 epochs reduces test error rates further by between 2% and 5%. It can also be observed that the 20-4 network is powerful enough to model the CIFAR-10/100 training sets to well over 99% accuracy, but the modelling power is reduced in the 1-bit version, particularly for CIFAR-100. The reduced modelling capacity for single-bit weights is consistent with the gap in test-error rate performance between the 32-bit and 1-bit cases. When using cutout, training for longer gives improved error rates, but when not using cutout, 126 epochs suffices for peak performance. Finally, for MNIST and a 4× wide ResNet without any data augmentation, our full-precision method achieved 0.71% after just 1 epoch of training, and 0.28% after 6 epochs, whereas our 1-bit method achieved 0.81%, 0.36% and 0.27% after 1, 6 and 14 epochs. In comparison, 1.29% was reported for the 1-bit-per-weight case by BID2, and 0.96% by BID12. Error rate (%) C100-1-bit C100-1-bit-Cutout C100-32-bits C100-32-bits-Cutout C10-1-bit C10-1-bit-Cutout C10-32-bits C10-32-bits-Cutout FIG6: Convergence through training. Left: Each marker shows the error rates on the test set and the training set at the end of each cycle of the warm-restart training method, for 20-4 ResNets (less than 5 million parameters). Right: each marker shows the test error rate for 20-10 ResNets, with and without cutout. C10 indicates CIFAR-10, and C100 indicates CIFAR-100. For CIFAR-10/100, both our full-precision Wide ResNets and our 1-bit-per-weight versions benefit from our method of not learning batch-norm scale and offset parameters. Accuracy in the 1-bit-perweight case also benefits from our use of a warm-restart training schedule BID18. To demonstrate the influence of these two aspects, in FIG8 we show, for CIFAR-100, how the test error rate changes through training when either or both of these methods are not used. We did not use cutout for the purpose of this figure. The comparison learning-rate-schedule drops the learning rate from 0.1 to 0.01 to 0.001 after 85 and 170 epochs. It is clear that our methods lower the final error rate by around 3% absolute by not learning the batch-norm parameters. The warm-restart method enables faster convergence for the full-precision case, but is not significant in reducing the error rate. However, for the 1-bit-per-weight case, it is clear that for best it is best to both use warm-restart, and to not learn batch-norm parameters. It is to be expected that a smaller error rate gap will between the same network using fullprecision and 1-bit-per-weight when the test error rate on the full precision network gets smaller. Indeed, TAB0 quantify how the gap in error rate between the full-precision and 1-bit-perweight cases tends to grow as the error-rate in the full-precision network grows. To further illustrate this trend for our approach, we have plotted in Figure 1 the gap in Top-1 error rates vs the Top-1 error rate for the full precision case, for some of the best performing networks for the six datasets we used. Strong from this data can only be made relative to alternative methods, but ours is the first study to our knowledge to consider more than two datasets. Nevertheless, we have also plotted in Figure 1 the error rate and the gap reported by BID22 for two different networks using their BWN 1-bit weight method. The reasons for the larger gaps for those points is unclear, but what is clear is that better full-precision accuracy in a smaller gap in all cases. A challenge for further work is to derive theoretical bounds that predict the gap. How the magnitude of the gap changes with full precision error rate is dependent on many factors, including the method used to generate models with 1-bit-per-weight. For high-error rate cases, the loss function throughout training is much higher for the 1-bit case than the 32-bit case, and hence, the 1-bit-per-weight network is not able to fit the training set as well as the 32-bit one. Whether this is because of the loss of precision in weights, or due to the mismatch in gradients inherent is propagating with 1-bit weights and updating full-precision weights during training is an open question. If it is the latter case, then it is possible that principled refinements to the weight update method we used will further reduce the gap. However, it is also interesting that for our 26-layer networks applied to CIFAR-10/100, that the gap is much smaller, despite no benefits in the full precision case from extra depth, and this also warrants further investigation. Our approach differs from the BWN method of BID22 for two reasons. First, we do not need to calculate mean absolute weight values of the underlying full precision weights for each output channel in each layer following each minibatch, and this enables faster training. Second, we do not need to adjust for a gradient term corresponding to the appearance of each weight in the mean absolute value. We found overall that the two methods work equally effectively, but ours has two advantages: faster training, and fewer overall parameters. As a note we found that the method of BID22 also works equally effectively on a per-layer basis, rather than per-channel. We also note that the focus of BID22 was much more on the case that combines single-bit activations and 1-bit-per-weight than solely 1-bit-per-weight. It remains to be tested how our scaling method compares in that case. It is also interesting to understand whether the use of batch-norm renders scaling of the sign of weights robust to different scalings, and whether networks that do not use batch-norm might be more sensitive to the precise method used. The unusual design choice of not learning the batch normalization parameters was made for CIFAR-10/100, SVHN and MNIST because for Wide ResNets, overfitting is very evident on these datasets (see FIG6 ; by the end of training, typically the loss function becomes very close to zero, corresponding to severe overfitting. Inspired by label-smoothing regularization BID26 that aims to reduce overconfidence following the softmax layer, we hypothesized that imposing more control over the standard deviation of inputs to the softmax might have a similar regularizing effect. This is why we removed the final all-to-all layer of our ResNets and replaced it with a 1×1 convolutional layer followed by a batch-normalization layer prior to the global average pooling layer. In turn, not learning a scale and offset for this batch-normalization layer ensures that batches flowing into the softmax layer have standard deviations that do not grow throughout training, which tends to increase the entropy of predictions following the softmax, which is equivalent to lower confidence .After observing success with these methods in 10× Wide ResNets, we then observed that learning the batch-norm parameters in other layers also led to increased overfitting, and increased test error rates, and so removed that learning in all layers (except the first one applied to the input, when ReLU is used there).As shown in FIG8, there are significant benefits from this approach, for both full-precision and 1-bit-per-weight networks. It is why, in TAB2, our surpass those of on effectively the same Wide ResNet (our 20-10 network is essentially the same as the 22-10 comparison network, where the extra 2 layers appear due to the use of learned 1×1 convolutional projections in downsampling residual paths, whereas we use average pooling instead).As expected from the motivation, we found our method is not appropriate for datasets such as ImageNet32 where overfitting is not as evident, in which case learning the batch normalization parameters significantly reduces test error rates. Here we compare our approach with SqueezeNet BID13, which reported significant memory savings for a trained model relative to an AlexNet. The SqueezeNet approach uses two strategies to achieve this: replacing many 3x3 kernels with 1x1 kernels; deep compression BID6.Regarding SqueezeNet strategy, we note that SqueezeNet is an all-convolutional network that closely resembles the ResNets used here. We experimented briefly with our 1-bit-per-weight approach in many all-convolutional variants-e.g. plain all-convolutional BID25, SqueezeNet BID13, MobileNet BID10, ResNexT BID27 and found its effectiveness relative to full-precision baselines to be comparable for all variants. We also observed in many experiments that the total number of learned parameters correlates very well with classification accuracy. When we applied a SqueezeNet variant to CIFAR-100, we found that to obtain the same accuracy as our ResNets for about the same depth, we had to increase the width until the SqueezeNet had approximately the same number of learned parameters as the ResNet. We conclude that our method therefore reduces the model size of the baseline SqueezeNet architecture (i.e. when no deep compression is used) by a factor of 32, albeit with an accuracy gap. Regarding SqueezeNet strategy, the SqueezeNet paper reports deep compression BID6 was able to reduce the model size by approximately a factor of 10 with no accuracy loss. Our method reduces the same model size by a factor of 32, but with a small accuracy loss that gets larger as the full-precision accuracy gets smaller. It would be interesting to explore whether deep compression might be applied to our 1-bit-per-weight models, but our own focus is on methods that minimally alter training, and we leave investigation of more complex methods for future work. Regarding SqueezeNet performance, the best accuracy reported in the SqueezeNet paper for ImageNet is 39.6% top-1 error, requiring 4.8MB for the model's weights. Our single-bit-weight models achieve better than 33% top-1 error, and require 8.3 MB for the model's weights. In this paper we focus only on reducing the precision of weights to a single-bit, with benefits for model compression, and enabling of inference with very few multiplications. It is also interesting and desirable to reduce the computational load of inference using a trained model, by carrying out layer computations in very few numbers of bits BID12 BID22 BID0. Facilitating this requires modifying non-linear activations in a network from ReLUs to quantized ReLUs, or in the extreme case, binary step functions. Here we use only full-precision calculations. It can be expected that combining our methods with reduced precision processing will inevitably increase error rates. We have addressed this extension in a forthcoming submission. optional ReLU is used, unlike all other batch-normalization layers, we enable learning of the scale and offset factors. This first BN layer enables us to avoid doing any pre-processing on the inputs to the network, since the BN layer provides necessary normalization. When the optional ReLU is included, we found that the learned offset ensures the input to the first ReLU is never negative. In accordance with our strategy of simplicity, all weight layers can be thought of as a block of three operations in the same sequence, as indicated in FIG2. Conceptually, batch-norm followed by ReLU can be thought of as a single layer consisting of a ReLU that adaptively changes its centre point and positive slope for each channel and relative to each mini-batch. We also precede the global average pooling layer by a BN layer, but do not use a ReLU at this point, since nonlinear activation is provided by the softmax layer. We found including the ReLU leads to differences early in training but not by the completion of training. The Wide ResNet of is specified as having a first convolutional layer that always has a constant number of output channels, even when the number of output channels for other layers increases. We found there is no need to impose this constraint, and instead always allow the first layer to share the same number of output channels as all blocks at the first spatial scale. The increase in the total number of parameters from doing this is small relative to the total number of parameters, since the number of input channels to the first layer is just 3. The benefit of this change is increased simplicity in the network definition, by ensuring one fewer change in the dimensionality of the residual pathway. We were interested in understanding whether the good we achieved for single-bit weights were a consequence of the skip connections in residual networks. We therefore applied our method to plain all-convolutional networks identical to our 4× residual networks, except with the skip connections removed. Initially, training indicated a much slower convergence, but we found that altering the initial weights standard deviations to be proportional to 2 instead of √ 2 helped, so this was the only other change made. The change was also applied in Equation.As summarized in Figure 7, we found that convergence remained slower than our ResNets, but there was only a small accuracy penalty in comparison with ResNets after 126 epochs of training. This is consistent with the findings of BID8 where only ResNets deeper than about 20 layers showed a significant advantage over plain all-convolutional networks. This experiment, and others we have done, support our view that our method is not particular to ResNets. Classification error rate (%) C100-32-bits-ResNet C100-1-bit-ResNet C10-32-bits-ResNet C10-1-bit-ResNet C100-32-bits-Plain C100-1-bit-Plain C10-32-bits-Plain C10-1-bit-Plain Figure 7: Residual networks compared with all-convolutional networks. The data in this figure is for networks with width 4×, i.e. with about 4.3 million learned parameters.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We train wide residual networks that can be immediately deployed using only a single bit for each convolutional weight, with signficantly better accuracy than past methods.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the article for finding a single-sentence TLDR.

Question: This paper presents a system for immersive visualization of Non-Euclidean spaces using real-time ray tracing. It exploits the capabilities of the new generation of GPU’s based on the NVIDIA’s Turing architecture in order to develop new methods for intuitive exploration of landscapes featuring non-trivial geometry and topology in virtual reality.

represent the short summary for finding the corresponding scientific article (long)

For example, ' We demonstrate a gated recurrent asynchronous spiking neural network that corresponds to an LSTM unit.' should be close to 'Spiking neural networks are being investigated both as biologically plausible models of neural computation and also as a potentially more efficient type of neural network. While convolutional spiking neural networks have been demonstrated to achieve near state-of-the-art performance, only one solution has been proposed to convert gated recurrent neural networks, so far. Recurrent neural networks in the form of networks of gating memory cells have been central in state-of-the-art solutions in problem domains that involve sequence recognition or generation. Here, we design an analog gated LSTM cell where its neurons can be substituted for efficient stochastic spiking neurons. These adaptive spiking neurons implement an adaptive form of sigma-delta coding to convert internally computed analog activation values to spike-trains. For such neurons, we approximate the effective activation function, which resembles a sigmoid. We show how analog neurons with such activation functions can be used to create an analog LSTM cell; networks of these cells can then be trained with standard backpropagation. We train these LSTM networks on a noisy and noiseless version of the original sequence prediction task from , and also on a noisy and noiseless version of a classical working memory reinforcement learning task, the T-Maze. Substituting the analog neurons for corresponding adaptive spiking neurons, we then show that almost all ing spiking neural network equivalents correctly compute the original tasks. With the manifold success of biologically inspired deep neural networks, networks of spiking neurons are being investigated as potential models for computational and energy efficiency. Spiking neural networks mimic the pulse-based communication in biological neurons, where in brains, neurons spike only sparingly -on average 1-5 spikes per second BID0. A number of successful convolutional neural networks based on spiking neurons have been reported BID7 BID13 BID6 BID15 BID12, with varying degrees of biological plausibility and efficiency. Still, while spiking neural networks have thus been applied successfully to solve image-recognition tasks, many deep learning algorithms use recurrent neural networks (RNNs), in particular using Long Short-Term Memory (LSTM) layers BID11. Compared to convolutional neural networks, LSTMs use memory cells to store selected information and various gates to direct the flow of information in and out of the memory cells. To date, the only spike-based version of LSTM has been realized for the IBM TrueNorth platform Shrestha et al.: this work proposes a method to approximate LSTM specifically for the constrains of this neurosynaptic platform by means of a store-and-release mechanism that synchronizes the modules. This translates to a frame-based rate coding computation, which is less biological plausible and energy efficient than an asynchronous approach, as the one proposed here. Here, we demonstrate a gated recurrent spiking neural network that corresponds to an LSTM unit with a memory cell and an input gate. Analogous to recent work on spiking neural networks (O '; BID6 BID19 BID20, we first train a network with modified LSTM units that computes with analog values, and show how this LSTMnetwork can be converted to a spiking neural network using adaptive stochastic spiking neurons that encode and decode information in spikes using a form of sigma-delta coding BID18 BID19 BID14 . In particular, we develop a binary version of the adaptive sigma-delta coding proposed in BID19 : we approximate the shape of the transfer function that this model of fast-adapting spiking neurons exhibits, and we assemble the analog LSTM units using just this transfer function. Since input-gating is essential for maintaining memorized information without interference from unrelated sensory inputs BID11, and to reduce complexity, we model a limited LSTM neuron consisting of an input cell, input gating cell, a Constant Error Carousel (CEC) and output cell. The ant analog LSTM network is then trained on a number of classical sequential tasks, such as the noise-free and noisy Sequence Prediction and the T-Maze task BID11 BID1. We demonstrate how nearly all the corresponding spiking LSTM neural networks correctly compute the same function as the analog version. Note that the conversion of gated RNNs to spike-based computation implies a conversion of the neural network from a time step based behavior to the continuous-time domain: for RNNs, this means having to consider the continuous signal integration in the memory cell. We solve the time conversion problem by approximating analytically the spiking memory cell behavior through time. Together, this work is a first step towards using spiking neural networks in such diverse and challenging tasks like speech recognition and working memory cognitive tasks. To construct an Adapting Spiking LSTM network, we first describe the Adaptive Spiking Neurons and we approximate the corresponding activation function. Subsequently, we show how an LSTM network comprised of a spiking memory cell and a spike-driven input-gate can be constructed and we discuss how analog versions of this LSTM network are trained and converted to spiking versions. Adaptive Spiking Neuron. The spiking neurons that are used in this paper are Adaptive Spiking Neurons (ASNs) as described in BID2. This is a variant of an adapting Leaky Integrate & Fire (LIF) neuron model that includes fast adaptation to the dynamic range of input signals. The ASNs used here communicate with spikes of a fixed height h = 1 (binary output), as suggested by BID20. The behavior of the ASN is determined by the following equations: incoming postsynaptic current: DISPLAYFORM0 input signal: DISPLAYFORM1 threshold: DISPLAYFORM2 internal state: DISPLAYFORM3 where w i is the weight (synaptic strength) of the neuron's incoming connection; t i s < t denote the spike times of neuron i, and t s < t denote the spike times of the neuron itself; φ(t) is an exponential smoothing filter with a short time constant τ φ; ϑ 0 is the resting threshold; m f is a variable controlling the speed of spike-rate adaptation; τ β, τ γ, τ η are the time constants that determine the rate of decay of I(t), ϑ(t) andŜ(t) respectively (see BID2 and BID19 for more details). As in BID2, the ASN emits spikes following a stochastic firing condition defined as: DISPLAYFORM4 where V (t) is the membrane potential defined as the difference between S(t) andŜ(t), λ 0 = 0.005 is a normalization parameter and ∆V = 0.1 is a scaling factor that defines the slope of the stochastic area. Activation function of the Adaptive Analog Neuron. In order to create a network of ASNs that performs correctly on typical LSTM tasks, our approach is to train a network of Adaptive Analog Neurons (AANs) and then convert the ing analog network into a spiking one, similar to O'; BID6; BID19. We define the activation function of the AANs as the function that maps the input signal S to the average PSC I that is perceived by the next (receiving) ASN in a defined time window. We normalize the obtained spiking activation function at the point where it reaches a plateau. We then fit the normalized spiking activation function with a sum-of-exponentials shaped function as: DISPLAYFORM5 with derivative: DISPLAYFORM6 where, for the neuron parameters used, we find a = 148.7, b = −10.16, c = 3.256 and d = −1.08.Using this mapping from the AAN to the ASN (see Figure 1), the activation function can be used during training: thereafter, the ASNs are used as "drop in" replacements for the AANs in the trained network. Unless otherwise stated, the ASNs use τ η = τ β = τ γ = 10 ms, and ϑ 0 and m f are set to 0.3 and 0.18 for all neurons. The spike height, h, is found such that ASN(4.8) = 1. Note that the spike height h is a normalization parameter for the activation function of the ASN model: in order to have binary communication across the network, the output weights are simply scaled by h. Adaptive Spiking LSTM. An LSTM cell usually consists of an input and output gate, an input and output cell and a CEC BID11. Deviating from the original formulation, and more recent versions where forget gates and peepholes were added Gers et al. FORMULA1, the Adaptive Spiking LSTM as we present it here only consists of an input gate, input and output cells, and a CEC. As noted, to obtain a working Adaptive Spiking LSTM, we first train its analog equivalent, the Adaptive Analog LSTM. Figure 2 shows the schematic of the Adaptive Analog LSTM and its spiking analogue. It is important to note that we aim for a one-on-one mapping from the Adaptive Analog LSTM to the Adaptive Spiking LSTM. This means that while we train the Adaptive Analog LSTM network with the standard time step representation, the conversion to the continuous-time spiking domain is achieved by presenting each input for a time window of size ∆t. Sigmoidal ASN. The original formulation of LSTM uses sigmoidal activation functions in the input gate and input cell. However, the typical activation function of real neurons resembles a half-sigmoid and we find that the absence of a gradient for negative input is problematic during training. Here, we approximate a sigmoidal-shaped activation function by exploiting the stochastic firing condition of the ASN. Indeed, Figure 1 shows that the ASN has a non-null probability to fire even under the threshold ϑ 0. Therefore, the AAN transfer function of Eq. 6 holds a gradient in that area. Together with the maximal activation being normalized to 1 (see Eq. 6 for lim S→∞) the AAN transfer function represents a good candidate for LSTM operations such as closing and opening the gates. Spiking input gate and spiking input cell. The AAN functions are used in the Adaptive Analog LSTM cell for the input gate and input cell. The activation value of the input cell is multiplied by the activation value of the input gate, before it enters the CEC, see Figure 2. In the spiking version of the input gate, the outgoing signal from the ASN is accumulated in an intermediate neuron (ASN * in Figure 2). The internal stateŜ of this neuron is then multiplied with the spikes that move from the ASN of the input cell to the ASN of the output cell. This leads to a direct mapping from the Adaptive Analog LSTM to the Adaptive Spiking LSTM.Spiking Constant Error Carousel (CEC) and spiking output cell. The Constant Error Carousel (CEC) is the central part of the LSTM cell and avoids the vanishing gradient problem BID11. In the Adaptive Spiking LSTM, we merge the CEC and the output cell to one ASN with an internal state that does not decay -in the brain could be implemented by slowly decaying (seconds) neurons BID5. The value of the CEC in the Adaptive Analog LSTM corresponds with state I of the ASN output cell in the Adaptive Spiking LSTM.In the Adaptive Spiking LSTM, we set τ β in Equation 1 to a very large value for the CEC cell to obtain the integrating behavior of a CEC. Since no forget gate is implemented this in a spiking CEC neuron that fully integrates its input. When τ β is set to ∞, every incoming spike is added to a non-decaying PSC I. So if the state of the sending neuron (ASN in in FIG1) has a stable inter-spike interval (ISI), then I of the receiving neuron (ASN out) is increased with incoming spike height h every ISI, so h ISI per time step. For a stochastic neuron, this corresponds to the average increase per time step. The same integrating behavior needs to be translated to the analog CEC. Since the CEC cell of the Adaptive Spiking LSTM integrates its input S every time step by S τη, we can map this to the CEC of the Adaptive Analog LSTM. The CEC of a traditional LSTM without a forget gate is updated every time step by CEC(t) = CEC(t − 1) + S, with S its input value. The CEC of the Adaptive Analog LSTM is updated every time step by CEC(t) = CEC(t − 1) + S τη. This is depicted in Figure 2 via a weight after the input gate with value 1 τη. To allow a correct continuous-time representation after the spike-coding conversion, we divide the incoming connection weight to the CEC, W CEC, by the time window ∆t. In our approach then, we train the Adaptive Analog LSTM as for the traditional LSTM (without the τ η factor), which effectively corresponds to set a continuous-time time window ∆t = τ η. Thus, to select a different ∆t, in the spiking version W CEC has to be set to W CEC = τ η /∆t. The middle plot in FIG1 shows that setting τ β to ∞ for ASN out in a spiking network in the same behavior as using an analog CEC that integrates with CEC(t) = CEC(t − 1) + S, since the slope of the analog CEC is indeed the same as the slope of the spiking CEC. Here, every time step in the analog experiment corresponds to ∆t = 200 ms. However, the spiking CEC still produces an error with respect to the analog CEC (the error increases for lower ∆ts, e.g. it doubles when going from 200ms to 50ms). This is because of two reasons: first, the stochastic firing condition in an irregular ISI; second, the adapting behavior of the ASN produces a transitory response that is not represented by the AAN transfer function. For these reasons, by choosing bigger time windows ∆t more stable responses are obtained. Learning rule used for training the spiking LSTM To train the analog LSTMs on the supervised tasks, a customized truncated version of real-time recurrent learning (RTRL) was used. This is the same algorithm used in Gers et al. FORMULA1, where the partial derivatives w.r.t. the weights W xc and W xi (see Figure 2) are truncated. For the reinforcement learning (RL) tasks we used RL-LSTM Bakker FORMULA1, which uses the same customized, truncated version of RTRL that was used for the supervised tasks. RL-LSTM also incorporates eligibility traces to improve training and Advantage Learning BID10. All regular neurons in the network are trained with traditional backpropagation. Since the presented Adaptive Analog LSTM only has an input gate and no output or forget gate, we present four classical tasks from the LSTM literature that do not rely on these additional gates. Sequence Prediction with Long Time Lags. The main concept of LSTM, the ability of a CEC to maintain information over long stretches of time, was demonstrated in in a Sequence Prediction task: the network has to predict the next input of a sequence of p + 1 possible input symbols denoted as a 1,..., a p−1, a p = x, a p+1 = y. In the noise free version of this task, every symbol is represented by the p + 1 input units with the i − th unit set to 1 and all the others to 0. At every time step a new input of the sequence is presented. As in the original formulation, we train the network with two possible sequences, (x, a 1, a 2, ..., a p−1, x) and (y, a 1, a 2, ..., a p−1, y), chosen with equal probability. For both sequences the network has to store a representation of the first element in the memory cell for the entire length of the sequence (p). We train 20 networks on this task for a total of 100k trials, with p = 100, on an architecture with p + 1 input units and p + 1 output units. The input units are fully connected to the output units without a hidden layer. The same sequential network construction method from the original paper was used to prevent the "abuse problem": the Adaptive Analog LSTM cell is only included in the network after the error stops decreasing BID11. In the noisy version of the sequence prediction task, the network still has to predict the next input of the sequence, but the symbols from a 1 to a p−1 are presented in random order and the same symbol can occur multiple times. Therefore, only the final symbols a p and a p+1 can be correctly predicted. This version of the sequence prediction task avoids the possibility that the network learns local regularities in the input stream. We train 20 networks with the same architecture and parameters of the previous task, but for 200k trials. For both noise-free and noisy tasks we considered the network converged when the average error over the last 100 trials was less than 0.25.T-Maze task. In order to demonstrate the generality of our approach, we trained a network with Adaptive Analog LSTM cells on a Reinforcement Learning task, originally introduced in BID1. In the T-Maze task, an agent has to move inside a maze to reach a target position in order to BID11 and current implementation); while for the T-Maze tasks it corresponds to the total number of steps BID1. ASN accuracy (%), total number of spikes per task and firing rate (Hz) are also reported. Note that the firing rate for both the sequence prediction tasks are computed without taking into account the input and output neurons not active in a specific time frame. Task be rewarded while maintaining information during the trial. The maze is composed of a long corridor with a T-junction at the end, where the agent has to make a choice based on information presented at the start of the task. The agent receives a reward of 4 if it reaches the target position and −0.4 if it moves against the wall. If it moves to the wrong direction at the T-junction it also receives a reward of −0.4 and the system is reset. The larger negative reward value, w.r.t. the one used in BID1, is chosen to encourage Q-values to differentiate more during the trial. The agent has 3 inputs and 4 outputs corresponding to the 4 possible directions it can move to. At the beginning of the task the input can be either 011 or 110 (which indicates on which side of the T-junction the reward is placed).Here, we chose the corridor length N = 20. A noiseless and a noisy version of the task were defined: in the noiseless version the corridor is represented as 101, and at the T-junction 010; in a noisy version the input in the corridor is represented as a0b where a and b are two uniformly distributed random variables in a range of. While the noiseless version can be learned by LSTM-like networks without input gating BID16, the noisy version requires the use of such gates. The network consists of a fully connected hidden layer with 12 AAN units and 3 Adaptive Analog LSTMs. To increase the influence of the LSTM cell in the network, we normalized the activation functions of the AAN output cell and ASN output cell at S = 1. The same training parameters are used as in Bakker FORMULA1; we train 20 networks for each task and all networks have the same architecture. As a convergence criteria we checked whenever the network reached on average a total reward greater than 3.5 in the last 100 trials. As shown in TAB0, all of the networks that were successfully trained for the noise-free and noisy Sequence Prediction tasks could be converted into spiking networks. FIG3 shows the last 6 inputs of a noise-free Sequence Prediction task before (left) and after (right) the conversion, demonstrating the correct predictions made in both cases. Indeed, for the 19 successful networks, after presenting either x or y as the first symbol of the sequence, the average error over the last 200ms was always below the chosen threshold of 0.25. As it can be seen in Figure 6, the analog and the spiking CEC follow a comparable trend during the task, reaching similar values at the end of the simulation. Note that, in the noisy task, all the successfully trained networks were still working after the conversion: in this case, due to the input noise, the CEC values are always well separated. Finally, we found that the number of trials needed to reach the convergence criterion were, on average, lower than the one reported in BID11.Similar were obtained for the T-Maze task: all the networks were successful after the conversion in both the noise-free and noisy conditions. FIG4 shows the Q-values of a noisy T-Maze task, demonstrating the correspondence between the analog and spiking representation even in presence of noisy inputs. However, we notice that the CEC of the spiking LSTMs reach different values compared to their analog counterparts. This is probably due to the increased network and task complexity. In general, we see that the spiking CEC value is close to the analog CEC value, while always exhibiting some deviations. Moreover, TAB0 reports the average firing rate computed per task, showing reasonably low values compatible with the one recorder from real neurons. Gating is a crucial ingredient in recurrent neural networks that are able to learn long-range dependencies BID11. Input gates in particular allow memory cells to maintain information over long stretches of time regardless of the presented -irrelevant -sensory input BID11. The ability to recognize and maintain information for later use is also that which makes gated RNNs like LSTM so successful in the great many sequence related problems, ranging from natural language processing to learning cognitive tasks BID1.To transfer deep neural networks to networks of spiking neurons, a highly effective method has been to map the transfer function of spiking neurons to analog counterparts and then, once the network has been trained, substitute the analog neurons with spiking neurons; BID6; BID19. Here, we showed how this approach can be extended to gated memory units, and we demonstrated this for an LSTM network comprised of an input gate and a CEC. Hence, we effectively obtained a low-firing rate asynchronous LSTM network. The most complex aspect of a gating mechanism turned out to be the requirement of a differentiable gating function, for which analog networks use sigmoidal units. We approximated the activation function for a stochastic Adaptive Spiking Neurons, which, as many real neurons, approximates a half-sigmoid (Fig. 1). We showed how the stochastic spiking neuron has an effective activation even below the resting threshold ϑ 0. This provides a gradient for training even in that area. The ant LSTM network was then shown to be suitable for learning sequence prediction tasks, both in a noise-free and noisy setting, and a standard working memory reinforcement learning task. The learned network could then successfully be mapped to its spiking neural network equivalent for at least 90% of the trained analog networks. Figure 6: The values of the analog CECs and spiking CECs for the noise-free Sequence Prediction (left, only one CEC cell was used) and noise-free T-maze (right, three CEC cells were used) tasks. The spiking CEC is the internal stateŜ of the output cell of the Adaptive Spiking LSTM.We also showed that some difficulties arise in the conversion of analog to spiking LSTM. Principally, the ASN activation function is derived for steady-state adapted spiking neurons, and this difference causes an error that may be large for fast changing signals. Analog-valued spikes as explored in BID19 could likely resolve this issue, at the expense of some loss of representational efficiency. Although the adaptive spiking LSTM implemented in this paper does not have output gates BID11, they can be included by following the same approach used for the input gates: a modulation of the synaptic strength. The reasons for our approach are multiple: first of all, most of the tasks do not really require output gates; moreover, modulating each output synapse independently is less intuitive and biologically plausible than for the input gates. A similar argument can be made for the forget gates, which were not included in the original LSTM formulation: here, the solution consists in modulating the decaying factor of the CEC.Finally, which gates are really needed in an LSTM network is still an open question, with answers depending on the kind of task to be solved BID9 BID21. For example, the AuGMEnT framework does not use gates to solve many working memory RL tasks BID16. In addition, it has been shown by BID4; BID9 that a combination of input and forget gates can outperform LSTM on a variety of tasks while reducing the LSTM complexity.'

Immersive Visualization of the Classical Non-Euclidean Spaces using Real-Time Ray Tracing.

Represent the short summary for finding the corresponding scientific article (long)

Interactively generating image from incrementally growing scene graphs in multiple steps using GANs while preserving the contents of image generated in previous steps


Question: Represent the scientific article to find a short TLDR!

Question: We propose the set autoencoder, a model for unsupervised representation learning for sets of elements. It is closely related to sequence-to-sequence models, which learn fixed-sized latent representations for sequences, and have been applied to a number of challenging supervised sequence tasks such as machine translation, as well as unsupervised representation learning for sequences. In contrast to sequences, sets are permutation invariant. The proposed set autoencoder considers this fact, both with respect to the input as well as the output of the model. On the input side, we adapt a recently-introduced recurrent neural architecture using a content-based attention mechanism. On the output side, we use a stable marriage algorithm to align predictions to labels in the learning phase. We train the model on synthetic data sets of point clouds and show that the learned representations change smoothly with translations in the inputs, preserve distances in the inputs, and that the set size is represented directly. We apply the model to supervised tasks on the point clouds using the fixed-size latent representation. For a number of difficult classification problems, the are better than those of a model that does not consider the permutation invariance. Especially for small training sets, the set-aware model benefits from unsupervised pretraining. Autoencoders are a class of machine learning models that have been used for various purposes such as dimensionality reduction, representation learning, or unsupervised pretraining (see, e.g., BID13 ; BID1 ; BID6 ; BID10). In a nutshell, autoencoders are feed-forward neural networks which encode the given data in a latent, fixed-size representation, and subsequently try to reconstruct the input data in their output variables using a decoder function. This basic mechanism of encoding and decoding is applicable to a wide variety of input distributions. Recently, researchers have proposed a sequence autoencoder BID5, a model that is able to handle sequences of inputs by using a recurrent encoder and decoder. Furthermore, there has been growing interest to tackle sets of elements with similar recurrent architectures BID21 ). In this paper, we propose the set autoencoder -a model that learns to embed a set of elements in a permutation-invariant, fixed-size representation using unlabeled training data only. The basic architecture of our model corresponds to that of current sequence-to-sequence models BID20 BID3 BID23: It consists of a recurrent encoder that takes a set of inputs and creates a fixed-length embedding, and a recurrent decoder that uses the fixedlength embedding and outputs another set. As encoder, we use an LSTM network with an attention mechanism as in BID21. This ensures that the embedding is permutation-invariant in the input. Since we want the loss of the model to be permutation-invariant in the decoder output, we re-order the output and align it to the input elements, using a stable matching algorithm that calculates a permutation matrix. This approach yields a loss which is differentiable with respect to the model's parameters. The proposed model can be trained in an unsupervised fashion, i.e., without having a labeled data set for a specific task. In a series of experiments, we analyze the properties of the embedding. For example, we show that the learned embedding is to some extent distance-preserving, i.e., the distance between two sets of elements correlates with the distances of their embeddings. Also, the embedding is smooth, i.e., small changes in the input set lead to small changes of the respective embedding. Furthermore, we show Figure 1: Example of a sequence-to-sequence translation model. The encoder receives the input characters ["g","o"]. Its internal state is passed to the decoder, which outputs the translation, i.e., the characters of the word "aller".that pretraining in an unsupervised fashion can help to increase the performance on supervised tasks when using the fixed-size embedding as input to a classification or regression model, especially if training data is limited. The rest of the paper is organized as follows. Section 2 introduces the preliminaries and briefly discusses related work. In Section 3, we present the details of the set autoencoder. Section 4 presents experimental setup and . We discuss the and conclude the paper in Section 5.2 RELATED WORK Sequence-to-sequence models have been applied very successfully in various domains such as automatic translation BID20, speech recognition BID3, or image captioning BID23. In all these domains, the task is to model P (Y |X), i.e., to predict an output sequence Y = (y 1, y 2, . . ., y m) given an input sequence X = (x 1, x 2, . . ., x n). Figure 1 shows the basic architecture of a sequence-to-sequence model. It consists of an encoder and a decoder, both of which are usually recurrent neural networks (RNNs). In the figure, the sequence-to-sequence model translates the input sequence X = (g, o) to the output sequence Y = (a, l, l, e, r). One by one, the elements of the input sequence are passed to the encoder as inputs. The encoder always updates its internal state given the previous state and the new input. Now, the last internal state of the encoder represents a fixed-size embedding of the input sequence (and is sometimes referred to as the thought vector). The decoder network's internal state is now initialized with the thought vector, and a special "start" token is passed as the input. One by one, the decoder will now output the tokens of the output sequence, each of which is used as input in the next decoder step. By calculating a loss on the output sequence, the complete sequence-to-sequence model can be trained using backpropagation. A special case of a sequence-to-sequence model is the sequence autoencoder BID5, where the task is to reconstruct the input in the output. For a more formal description of sequence-to-sequence models, please refer to BID20. Researchers have tackled sets of elements directly with neural networks, without using explicit but lossy set representations such as the popular bag-of-words-model BID12. Vinyals et al. raise the question of how the sequence-to-sequence architecture can be adapted to handle sets. They propose an encoder that achieves the required permutation-invariance to the input elements by using a content-based attention mechanism. Using a pointer network BID22 as decoder, the model can then be trained to sort input sets and outperforms a model without a permutation-invariant encoder. The proposed attention-based encoder has been used successfully in other tasks such as one-shot or few-shot learning BID3 ). Another approach BID18 tackles sets of fixed size by introducing a permutation-equivariant 1 layer in standard neural networks. For sets containing more than a few elements, the proposed layer helps to solve problems like point cloud classification, calculating sums of images depicting numbers, or set anomaly detection. The proposed models can fulfill complex supervised tasks and operate on sets of elements by exploiting permutation equi-or invariance. However, they do not make use of unlabeled data. The objective of the set autoencoder is very similar to that of the sequence autoencoder BID5: to create a fixed-size, permutation-invariant embedding for an input set X = {x 1, x 2, . . ., x n}, x i ∈ R d by using unsupervised learning, i.e., unlabeled data. The motivation is that unlabeled data is much easier to come by, and can be used to pretrain representations, which facilitate subsequent supervised learning on a specific task BID6. In contrast to the sequence autoencoder, the set autoencoder needs to be permutation invariant, both in the input and the output set. The first can be achieved directly by employing a recurrent encoder architecture using content-based attention similar to the one proposed by BID21 ) (see Section 3.1). Achieving permutation invariance in the output set is not straightforward. When training a set autoencoder, we need to provide the desired outputs Y in some order. By definition, all possible orders should be equally good, as long as all elements of the input set and no surplus elements are present. In theory, the order in which the outputs are presented to the model (or, more specifically: to the loss function) should be irrelevant: by using a chain rule-based model, the RNN can, in principle, model any joint probability distribution, and we could simply enlarge the data set by including all permutations. Since the number of permutations grows exponentially in the number of elements, this is not a feasible way: The data set quickly becomes huge, and the model has to learn to create every possible permutation of output sets. Therefore, we need to tackle the problem of random permutations in the outputs differently, while maintaining a differentiable architecture (see Section 3.2). The encoder takes the input set and embeds it into the fixed-sized latent representation. This representation should be permutation invariant to the order of the inputs. We use an architecture with content-based attention mechanism similar to the one proposed in BID21 ) (see FIG2): First, each element x i of the input set X is mapped to a memory slot m i ∈ R l using a mapping function f inp (Eq. 1). We use a linear mapping as f inp, the same for all i 3. Then, an LSTM network BID14 BID8 with l cells performs n steps of calculation. In each step, it calculates its new cell state c t ∈ R l and hidden state h t ∈ R l using the previous cell-and hidden state c t−1 and h t−1, as well as the previous read vector r t−1, all of which are initialized to zero in the first step. The new read vector r t is then calculated as weighted combination of all memory locations, using the attention mechanism (Eq. 5). For each memory location i, the attention mechanism calculates a scalar value e i,t which increases based on the similarity between the memory value m i and the hidden state h t, which is interpreted as a query to the memory (Eq. 3). We set f dot to be a dot product. Then, the normalized weighting a i for all memory locations is calculated using a softmax (Eq. 4). After n steps, the concatenation of c n,h n and r n can be seen as a fixed-size embedding of X 4. Note that permuting the elements of X has no effect on the embedding, since the memory locations m i are weighted by content, and the sum in Eq. 5 is commutative. DISPLAYFORM0 DISPLAYFORM1 DISPLAYFORM2 DISPLAYFORM3 Section 3.1 defined the forward propagation from the input set X to the fixed-size embedding [c t, h t, r t]. We now define the output of the set autoencoder that allows us to train the model using a loss function L. Like in the original sequence-to-sequence model, the core of the decoder is an LSTM network (see Figure 3): DISPLAYFORM0 DISPLAYFORM1 In each step, the decoder LSTM calculates its internal cell stateĉ t and its hidden stateĥ t (Eq. 6). The cell-and hidden state are initialized with the cell-and hidden state from the embedding, produced by the encoder (Eq. 7 and Eq. 8). In the first step, the decoder also gets an additional inputr 0, which is set to be the last read vector of the encoder (Eq. 9). In all following steps,r t is a vector of all zeros. We calculate the decoder's output o t at step t by using the linear function f out (Eq. 10). Each output element o t is of the same dimensionality as the input elements x i. The underlying idea is that f out is the "reverse" operation to f inp, such that encoder and decoder LSTM can operate on similar representations. Furthermore, in each step, the function f eos calculates ω t (Eq. 11), which we interpret as the probability that o t is an element of the set. In other words, if ω t = 0, we can stop sampling. In principle, we could use the LSTM's output sequence O = (o 1, o 2, . . ., o m) directly as elements of the output set. However, this does not take into account the following issues: First, the number m of outputs produced by the decoder should be equal to the size n of the input set X. This can be achieved by learning to output the correct ω t (see Eq. 12 below). Second, the order of the set elements should be irrelevant, i.e., the loss function should be permutation-invariant in the outputs. To address the latter point, we introduce an additional mapping layer D = (d 0, d 1, . . ., d n) between the decoder output and the loss function. The mapping rearranges the first n decoder outputs in the order of the inputs X. That is, it should make sure that the distance between d i and x i is small for all i. The mapping is defined as: d i = n j=1 o j w ij Here, w ij are the elements of a permutation matrix W of size n × n with the properties DISPLAYFORM0 In other words, each output o i is mapped to exactly one d j, and vice versa. For now, we assume that W has been parametrized such that the re-ordered elements in D match the elements in input set X well (see Section 3.3). Then, the set autoencoder loss function can be calculated as DISPLAYFORM1 The function L(x i, d i) is small if x i and d i are similar, and large if x i and d i are dissimilar. In other words, for each element in the input set, the distance to a matching output element (as mapped by W) will be decreased by minimizing L. For discrete elements, L can be calculated as the cross entropy loss L(x, d) = − i x i log d i. For elements that are vectors of real numbers, L is a norm of these vectors, e.g., L(x, d) = ||x − d||. The function L eos calculates the cross-entropy loss between ω i and ω * t, where ω * i indicates if an i th element should be present in the output, i.e., ω * i = 1 if i <= n, 0 else (recall that the decoder can produce m outputs, and m is not necessarily equal to n). Since the whole set autoencoder is differentiable, we train all weights except W using gradient descent. Re-ordering the decoder outputs resembles a point cloud correspondence problem from the domain of computer vision BID15 BID19. Methods like the iterative closest points algorithm Figure 4: Examples of sets in shapes data set BID2 find closest points between two sets, and find a transformation that aligns the second set to the first. Since we are only interested in the correspondence step, we notice its similarity to the stable marriage problem BID11: We want to find matching pairs P i = {man i, woman i} of two sets of men and women, such that there are no two pairs P i, P j where element man i prefers woman j over woman i, and, at the same time, woman j prefers man i over man j.5 To solve this problem greedily, we can use the Gale-Shapely algorithm BID7, which has a run time complexity of O(n 2) (see Algorithm 1) 6. Since its solution is permutation invariant in the set that proposes first BID11 )(p. 10), we consider the input elements x i to be the men, and let them propose first. After the stable marriage step, w ij = 1 if x i is "engaged" to o j. We use a number of synthetic data sets of point clouds for the unsupervised experiments. Each data set consists of sets of up to k items with d dimensions. In the random data sets, the k values of each element are randomly drawn from a uniform distribution between -0.5 and 0.5. In the following experiments, we set k = 16 and d ∈ {1, 2, 3}. In other words, we consider sets of up to 16 elements that are randomly distributed along a zero-centered 1d-line, 2d-plane, or 3-d cube with side length 1. We choose random distributions to evaluate the architecture's capability of reconstructing elements from sets, rather than learning common structure within those sets. In the shapes data set, we create point clouds with up to k = 32 elements of d = 2 dimensions. In each set, the points form either a circle, a square, or a cross (see Figure 4). The shapes can occur in different positions and vary in size. To convey enough information, each set consists of at least 10 points. Each data set contains 500k examples in the training set, 100k examples in the validation set, and another 500k examples in the test set. For each of the data sets we train a set autoencoder to minimize the reconstruction error of the sets, i.e., to minimize Eq. 12 (please refer to the appendix for details of the training procedure, including all hyperparameters). Figure 5 shows the mean euclidean distance of the reconstructed elements for the three random data sets (left-hand side) and the shapes data set (right-hand side), for different set sizes. For the random data sets, the mean error increases with the number of dimensions d of the elements, and with the number of elements within a set. This is to be expected, since all values are completely uncorrelated. For the shapes data set, the average error is lower than the errors for the 2d-random data set with more than two elements. Furthermore, the error decreases with the number of elements in the set. We hypothesize that with a growing number of points, the points become more evenly distributed along the outlines of the shapes, and it is therefore easier for the model to reconstruct them (visual inspection of the reconstructions suggests that the model tends to distribute points more uniformly on the shapes' outlines). We now take a closer look at the embeddings of the set autoencoder (i.e., the vector [c t, h t, r t]) for random sets. Some of the embedding variables have a very strong correlation with the set size (Pearson correlation coefficients of > 0.97 and <-0.985, respectively). In other words, the size of the set is encoded almost explicitly in the embedding. The embeddings seem to be reasonably smooth (see FIG4). We take a set with a single 2d-element and calculate its embeddings (leftmost points in FIG4). We then move this element smoothly in the 2d-plane and observe the ing embeddings. Most of the time, the embeddings change smoothly as well. The discontinuities occur when the element crosses the center region of the 2d plane. Apart from this center region, the embedding preserves a notion of distance of two item sets. This becomes apparent when looking at the correlations between the euclidean distances of two sets X 1 and X 2 and their corresponding embeddings enc(X 1) and enc(X 2). 7 The correlation coefficients for random sets of size one to four are 0.81, 0.71, 0.67, and 0.64. In other words, similar sets tend to yield similar embeddings. Vinyals et al. show that order matters for sequence-to-sequence and set-to-set models. This is the case both for the order of input sequences -specific orders improve the performance of the model's taskas well as for the order of the output sets, i.e., the order in which the elements of the set are processed to calculate the loss function. Recall that the proposed set autoencoder is invariant to the order of the elements both in the input set (using the attention mechanism) and the target set (by reordering the outputs before calculating the loss). Nevertheless, we observe that the decoder learns to output elements in a particular order: We now consider sets with exactly 8 random 2-dimensional elements. We use a pretrained set autoencoder from above, encode over 6,000 sets, and subsequently reconstruct the sets using the decoder. Figure 7 shows heat maps of the 2d-coordinates of the i'th reconstructed element. The first reconstructed element tends to be in the center right area. Accordingly, the second element tends to be in the lower-right region, the third element in the center-bottom region, and so on. The decoder therefore has learned to output the elements within a set in a particular order. Note that most of the distributions put some mass in every area, since the decoder must be able to reproduce sets where the items are not distributed equally in all areas (e.g., in a set where all elements are in the top right corner, the first element must be in the top right corner as well). Figure 8 shows the effect of the set size n on the distribution of the first element. If there is only one element (n = 1), it can be anywhere in the 2d plane. The more elements there are, the more likely it is that at least one of them will be in the center right region, so the distribution of the first element gets more peaked. We speculate that the roughly circular arrangement of the element distributions (which occurs for other set sizes as well) could be an implicit of using the cosine similarity f dot in the attention mechanism of the encoder. Also, this behavior is likely to be the reason for the discontinuity in FIG4 around. 7 We align the elements of X 1 and X 2 using the Gale-Shapely algorithm before calculating distances. DISPLAYFORM0 Figure 7: Heat maps of the location of the ith element in sets of size 8 with 2d-elements (decoder output). Darker shadings indicate more points at these coordinates. n=1 n=6 n=11 n=16Figure 8: Heat maps of the location of the first element in sets of various sizes n with 2d-elements (decoder output). Darker shadings indicate more points at these coordinates. We derive a number of classification and regression tasks based on the data sets in Section 4.1. On the random data sets, we define a number of binary classification tasks. The 1d-, 2d-or 3d-area is partitioned into two, four, or eight areas of equal sizes. Then, two classes of sets are defined: A set is of class 1, if all of its elements are within i of the j defined areas. All other sets are of class two. For example, if d = 2, i = 2, and j = 4, a set is of class 1 if all its elements are in the top left or bottom left area, or any other combination of two areas. 8 Furthermore, we define two regression tasks on the random data sets: The target for the first one is the maximum distance between any two elements in the set, the second one is the volume of the d-dimensional bounding box of all elements. On the shapes data set, the three-class classification problem is to infer the prototypical shape represented by the elements in the set. In the following, we use a set autoencoder as defined above, add a standard two-layer neural network f supervised (enc(X)) on top of the embedding, and use an appropriate loss function for the task (for implementation details see supplementary material). We compare the of the set autoencoder (referred to as set-AE) to those of two vanilla sequence autoencoders, which ignore the fact that the elements form a set. The first autoencoder (seq-AE) gets the same input as the set model, the input for the second autoencoder has been ordered along the first element dimensions (seq-AE (ordered)). Furthermore, we consider three training fashions: For direct training, we train the respective model in a purely supervised fashion on the available training data (in other words: the models are not trained as autoencoders). In the pretrained fashion (pre), we initialize the encoder weights using unsupervised pretraining on the respective (unlabeled) 500k training set, and subsequently train the parameters of f supervised, holding the encoder weights fixed. In the fine tuning setting (fine), we continue training from the pretrained model, and fine-tune all weights. Tables 1(a) and (b) show the accuracy on the test set for the i-of-j-areas classification tasks, for 1,000 and 10,000 training examples. The plain sequence autoencoder is only competitive for the most simple task (first row). For the simple and moderately complex tasks (first three rows), the ordered sequence autoencoder and the set autoencoder reach high accuracies close to 100%, both Table 3: Accuracy for object shape classification task, higher is better. Averaged over 10 runs.for the small and the large training set. When the task gets more difficult (higher i, j, or d), the set autoencoder clearly outperforms both other models. For the small training set, the pre and fine training modes of the set autoencoder usually lead to better than direct training. In other words, the unsupervised pretraining of the encoder weights leads to a representation which can be used to master the classification tasks with a low number of labeled examples. For the larger training set, unsupervised pretraining is still very useful for the more complicated classification tasks. On the other hand, unsupervised pretraining only helps in a few rare cases if the elements are treated as a sequence -the representation learned by the sequence autoencoders does not seem to be useful for the particular classification tasks. The or the regression task are shown in Tables 2 (a) and (b). Again, the ordered sequence autoencoder shows good for small d (recall that the first element dimension is the one that has been ordered), but fails to compete with the set-aware model in the higher dimensions. However, unsupervised pretraining helps the set model in the regression task only for small d. Tables 3 (a) and (b) show the for the shapes classification task. Here, the ordered sequence autoencoder with fine tuning clearly dominates both other models. The set model is unable to capitalize on the proper handling of permutation invariance. In sum, the show that unsupervised pretraining of the set autoencoder creates representations that can be useful for subsequent supervised tasks. This is primarily the case, if the supervised task requires knowledge of the individual locations of the elements, as in the i-of-j-areas classification task. If the precise locations of a subset of the elements are required (as in the bounding box or maximum distance regression tasks), direct training yields better . We hypothesize that failure of the set-aware model on the shapes classification is due to the linear mapping functions f inp and f out: They might be too simple to capture the strong, but non-linear structures in the data. We presented the set autoencoder, a model that can be trained to reconstruct sets of elements using a fixed-size latent representation. The model achieves permutation invariance in the inputs by using a content-based attention mechanism, and permutation invariance in the outputs, by reordering the outputs using a stable marriage algorithm during training. The fixed-size representation possesses a number of interesting attributes, such as distance preservation. We show that, despite the output permutation invariance, the model learns to output elements in a particular order. A series of experiments show that the set autoencoder learns representations that can be useful for tasks that require information about each set element, especially if the tasks are more difficult, and few labeled training examples are present. There are a number of directions for future research. The most obvious is to use non-linear functions for f inp and f out to enable the set autoencoder to capture non-linear structures in the input set, and test the performance on point clouds of 3d data sets such as ShapeNet BID4. Also, changes to the structure of the encoder/decoder (e.g., which variables are interpreted as query or embedding) and alternative methods for aligning the decoder outputs to the inputs can be investigated. Furthermore, more research is necessary to get a better understanding for which tasks the permutation invariance property is helpful, and unsupervised pretraining can be advantageous. BID0 to implement all models. For the implementation and experiments, we made the following design choices:Model Architecture• Both the encoder and the decoder LSTMs are have peephole connections BID8. We use the LSTM implementation of Tensorflow • The input mapping f inp and output mapping f out functions are simple linear functions. Note that we can not re-use f inp on the decoder side to transform the supervised labels in a "backwards" fashion: In this case, learning could parametrize f inp such that all set elements are mapped to a the same value, and the decoder learns to output this element only.• For the supervised experiments (classification and regression), we add a simple two-layer neural network f supervised on top of the embedding. The hidden layer of this network has the same number of units as the embedding, and uses ReLu activations BID17. For the two-class problems (i of j areas), we use a single output neuron and a cross-entropy loss, for the multi-class problems (object shapes) we use three output neurons and a cross-entropy loss. For the regression problems (bounding box and maximum distance), we optimize the mean squared error.• All parameters are initialized using Xavier initialization BID9 • Batch handling: For efficiency, we use minibatches. Therefore, within a batch, there can be different set sizes n i for each example i in the set. For simplicity, the encoder keeps processing all sets in a batch, i.e., it always performs n = k steps, where k is the maximum set size. Preliminary experiments showed only minor variations in the performance when processing is stopped after n i steps, where n i corresponds to the actual size of set i in the minibatch. • The number l of LSTM cells is automatically determined by the dimensionality d and maximum set size k of the input. We set l = k * d, therefore c t, h t,ĉ t,ĥ t ∈ R l. As a consequence, the embedding for all models (set-and sequence autoencoder) could, in principle, comprise the complete information of the set (recall that the goal was not to find the most compact or efficient embedding) • For simplicity, the dimensionality of each the memory cell m i and the read vector r i is equal to the number of LSTM cells, i.e., m i, r i ∈ R l (in principle, the memory could have any other dimensionality, but this would require an additional mapping step, since the query needs to be of the same dimensionality as the memory). We use Adam BID16 to optimize all parameters. We keep Adam's hyperparameters (except for the learning rate) at Tensorflow's default values (β 1 = 0.9, β 2 = 0.999, = 1e − 08). We use minibatch training with a batch size of 100. We keep track of the optimization objective during Table 4: Training hyperparameters.training and reduce the learning rate by 33% / stop training once there has been no improvement for a defined number of epochs, depending on the training mode (see Table 4). For the classification tasks, we couple the learning rate decrease/early stopping mechanism to the missclassification error (1 − accuracy) rather than the loss function. • The values of stalled-epochs-before-X are much higher for the supervised learning scenarios, since the training sets are much smaller (e.g., when using 1,000 examples and a batch size of 100, a single epoch only in 10 gradient update steps).• It is possible that the for supervised training with fine tuning improve if the encoder weights are regularized as well (the weights are prone to overfitting, since we use a low number of training examples).

We propose the set autoencoder, a model for unsupervised representation learning for sets of elements.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.