Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Watermarks have been used for various purposes. Recently, researchers started to look into using them for deep neural networks. Some works try to hide attack triggers on their adversarial samples when attacking neural networks and others want to watermark neural networks to prove their ownership against plagiarism. Implanting a backdoor watermark module into a neural network is getting more attention from the community. In this paper, we present a general purpose encoder-decoder joint training method, inspired by generative adversarial networks (GANs). Unlike GANs, however, our encoder and decoder neural networks cooperate to find the best watermarking scheme given data samples. In other words, we do not design any new watermarking strategy but our proposed two neural networks will find the best suited method on their own. After being trained, the decoder can be implanted into other neural networks to attack or protect them (see Appendix for their use cases and real implementations). To this end, the decoder should be very tiny in order not to incur any overhead when attached to other neural networks but at the same time provide very high decoding success rates, which is very challenging. Our joint training method successfully solves the problem and in our experiments maintain almost 100\% encoding-decoding success rates for multiple datasets with very little modifications on data samples to hide watermarks. We also present several real-world use cases in Appendix. Security issues of deep learning have been very actively being studied. It had been already demonstrated that deep learning methods are vulnerable to some carefully devised adversarial attacks BID7 BID4 BID6 BID1. At the same time, many researchers are also studying about how to make them more robust against such attacks. A couple of recent works, for example, proposed to use watermarks BID9 BID0 to protect neural networks. At the same time, other work wanted to use a similar watermark technique to attack neural networks BID9.The method of adding watermarks to data samples can be used in various ways to protect deep learning models. First, the decoder can be implanted into a trained deep learning model and later one can prove the ownership, when other people copied the model, by showing that the copied model reacts to one's watermarked samples. Second, the implanted decoder may allow only legitimately watermarked samples and reject other non-watermarked samples. In this case, only people that have the encoder can access the deep learning model. However, there is one very strict requirement that the decoder should be tiny to minimize the incurred overheads by attaching it as part of the main deep learning model. Similar techniques can also be used to attack neural networks. In this paper, we do not propose any specific watermarking techniques. Instead, we want the encoder and decoder discuss and decide their watermarking method. Inspired from generative adversarial networks (GANs) BID3, the encoder and decoder work for the same goal and are jointly trained. They do not perform the adversarial game of GANs. Their relationship is rather cooperative than adversarial in our method. The decoder is a tiny neural network to decode watermarks and the encoder is a high-capacity neural network that can watermark samples in such a way that the tiny neural network can successfully decode. Therefore, those two neural networks should cooperate to find such a watermarking scheme -in GANs, one neural network (generator) tries to fool the other neural network (discriminator). Because the decoder has a limited capacity due to its tiny neural network size, the encoder should not decide the watermarking scheme alone. The encoder should receive feedback from the decoder to revise its watermarking scheme. After training them, one should keep the encoder in a secure place but can deploy the decoder to as many places as one wants. We also show that our method can be used for both defences and attacks (refer to Appendix for some of these examples we implemented using our proposed method).We adopt residual blocks BID5 to design the encoder. Each residual block of the encoder is supposed to learn f (x)+x where x is an input to the block. One can consider f (x) as a watermark signal discovered by the joint training of the encoder and the decoder. The signal produced by f (x) should be strong enough to be detected by the decoder but weak enough not to be detected by human eyes. We design our training loss definition to achieve this goal. The encoder should modify original samples to implant watermarks. As more modifications are allowed, stronger watermarks will be implanted but they can be readily detected by human eyes. Our loss definition has a parameter that can be set by user to limit the modifications by the encoder. Our experiments show that we can find a well-balanced watermarking scheme that be detected only by the decoder. We tested many different datasets: face recognition(VGG-Face Data-set), speech recognition BID11, images with general objects BID7, and flowers (Flowers Data-set). Two of them are reported in the main paper with the comparison with other watermarking methods and others are introduced in Appendix. During experiments, our methods marked 100% decoding success rates for all datasets (in at least one hyper-parameter configuration). This well outperforms other baseline methods. In addition, we also found that different watermarking schemes are trained for different datasets. For instance, the encoder modified the tone of colors for the face recognition images. For the general object images, however, the encoder explicitly marks some dots rather than modifying their color tones (see FIG1 and Figure 4). This proves our goal that two neural networks cooperate to find the best suited watermarking method for each dataset. Watermarking data samples, such as images, videos, etc., is a long-standing research problem. In many cases, watermarking systems merge a specific watermark signal s (set by user) and a data sample x to produce a watermarked sample x, i.e., x = encode(x, s) and s = decode(x). In general, the signal s is secret and later used to check where the watermarked sample x is originated from. There exist many different watermarking techniques for relational databases, images, videos, and so forth. However, watermarking deep neural networks is still under-explored except for a couple of recent papers BID9 BID0. For instance, one can implant a certain signal on neural network weights -technically, this is similar to implanting a signal on a column of table for watermarking a relational database. However, the signal on the weights will disappear after finetuning the neural network which can preserve its accuracy but reorganize its weight values. Instead, we need a more robust way for watermarking neural networks. To this end, a backdoor based watermarking method has been recently proposed BID9 BID0. In general, a backdoor means a certain malware piece that can be exploited to avoid authentication processes in computer security. In their contexts, however, a neural network backdoor means a way to control the final prediction of a target neural network -for instance, retraining a target neural network so that it classifies a certain type of cats as dogs. The authors want to use the backdoor mechanism to protect the ownership of a neural network. Because the backdoor reacts to the samples specially watermarked by the owner, the proof of its ownership is available when other people copied the neural network. Of course, if the backdoor is successfully identified and removed, the proof of the ownership is not possible. However, this incurs additional costs and greatly decreases the motivation of copying the model. The same watermarking technique can be used for attacks. In BID9, the attacker implants an attack trigger into a data sample using a simple watermarking technique and the target neural network is already compromised by the attack to make it react to their trigger. Their goal is to induce the compromised neural network outputs a certain label encoded in the attack trigger and preferred by the attacker. Because this paper uses a very strong watermark signal, their watermarked images are visually impaired. Due to its strong watermarks, however, their attack shows very high success rates. GANs are one of the most successful generative models. They consist of two neural networks, one generator and one discriminator. They perform the following zero-sum minimax game: DISPLAYFORM0 where p(z) is a prior distribution, G(·) is a generator function, and D(·) is a discriminator function whose output spans DISPLAYFORM1 indicates that the discriminator D classifies a sample x as generated (resp. real).The generator tries to obfuscate the task of the discriminator by producing realistic fake samples. We redesign the adversarial game model for our purposes. In our case, two neural networks, one encoder and one decoder, perform a cooperative game. A watermarking framework consists of encoder and decoder. The encoder modifies original samples by adding a watermark signal into them and the decoder is a binary classification to detect the presence of the watermark signal. Watermarks are used for various purposes in deep learning. They were used for both of defenses and attacks for deep neural networks. In our case, we are interested in developing a pair of encoder and decoder and the decoder should be pluggable to other neural networks (as in the malware piece or backdoor in computer security). Our encoder-decoder pair can be used for both defenses and attacks (refer to Appendix for our case studies).There are several watermarking methods based on CNNs that can be described by x = encode(x, s) and s = decode(x) BID10. However, existing methods do not care about the size of the decoder and we are not interested in implanting a watermark signal s into a data sample x. We let the encoder modify x in a way that the decoder wants and the decoder performs the binary classification of watermarked or non-watermarked. Thus, our model can be described as x = encode(x) and decode(x) ∈ {0, 1} without s. In real world applications, this binary classification decoder suffices (refer to our use cases in Appendix) and the decoder should be so tiny that it does not incur any overheads when attached to other main neural networks -this is a strong requirement especially for the backdoor based watermarking method. Our goal is to develop a watermarking framework that consists of one large encoder (a fatty network) and one tiny decoder (a skinny network) and they should decide their own watermarking scheme without human efforts. Our overall idea is greatly inspired by generative adversarial networks (GANs). In GANs, there are two neural networks, generator and discriminator, that are comparable to each other in terms of their neural network capacity. In our case, however, the encoder and decoder are highly imbalanced in their neural network capacity and they perform a cooperative game (rather than the zero-sum adversarial game of GANs).In our method, the encoder should be capable of generating simple but robust watermarked samples because the decoder has very low capacity and as a , it may not be able to decode complicated watermarks. Therefore, the encoder should be large and trained enough to find the watermarking. + Figure 1: The proposed encoder architecture. Based on the attention map, a series of residual blocks generate a watermark signal specific to the input sample x which will be later merged with the generated watermark signal. All those convolutions in this encoder use the stride of 1 and the channel of 3 to maintain the identical input and output dimensions.mechanism suitable for the low-capacity decoder. In other words, we do not teach any watermarking mechanism but let them discover on their own considering the neural network capacity difference. The encoder (comparable to the generator of GANs) should modify original samples to implant a watermark signal. We adopt residual blocks to design the encoder as shown in Figure 1. Residual blocks are proven to be effective in designing a deep architecture and adopted by many works (e.g., ResNet BID5). Each residual block that can be described as x + f (x) is suitable to perform the watermarking task. After the multiple stages of residual blocks, the encoder generates a watermark signal 1 that will be merged with the original sample x. We use the multiple residual blocks because it is very unlikely that one residual block is able to generate a robust watermark signal. The overall watermarked sample generation process can be described as follows: DISPLAYFORM0 where x is the original sample; A is the attention map of x produced after two convolutions, one activation, and a softmax; means the Hadamard product; f i (·) represent an additive term by i-th residual block. In particular, we use the swish activation BID14. Thus, one can consider our generated watermark signal is an ensemble of all those additive terms . Note that our watermark signal is generated after ignoring unimportant parts of x after the element-wise product with the attention map. After merging the input sample x and the generated watermark, we have one post-processing block to refine the watermarked sample. This process includes a couple of more convolutions. In FIG0, and 4, we show watermarking examples for various datasets. In FIG0, watermarks are generated for the parts where the attention map focuses on. In FIG1, watermakrs are dispersed over many pixels and in this case, the attention also provides similar weights for those pixels. The decoder (comparable to the discriminator of GANs) should classify if an input sample has a watermark signal or not. We adopt the discriminator of DCGAN BID13 (after shrinking its size) as decoder. Its discriminator follows a standard CNN architecture. One of its ) and (e) are generated watermarks (before being merged with images). These are cases where watermarks are generated, aided by attention. For (e), there is a watermark in the most lower right corner and its attention also focuses on the same area. However, attention maps sometimes provide similar weights over almost all pixels, in which cases watermarks are scattered over all pixels -examples in FIG1 correspond to this case.advantageous is that it is very hard to identity the decoder after being implanted into a neural network model because it is tiny and uses only very standard neural operators. We perform experiments by varying the number of convolution layers in order to find the smallest decoder configuration. We introduce our training method. The main training loss can be described as follows: DISPLAYFORM0 where E(·) is an encoder, and D(·) is a decoder, and x is a data sample. This loss definition looks similar to the one in GANs. However, we do not perform the minimax zero-sum game of GANs. Both the encoder and decoder cooperate to find the best performing watermarking scheme. Its equilibrium state analysis is rather meaningless because they do not perform the zero-sum adversarial game of GANs. It is obvious that the main loss representing equation 3 will be optimized when the decoding success rate of watermarked and non-watermarked cases is 100%. The main loss can be implemented using the cross-entropy loss as in other GANs. In addition, we also use one more regularization term to limit the modification by the encoder. Let L be the main loss in equation 3. The final loss term is defined as follows: DISPLAYFORM1 where φ(·) s,t means the feature map taken after t-th convolution (after activation) before s-th maxpooling layer in the VGG19 network 2, γ is the maximum margin in the hinge-loss based regularization. We allow the modification up to γ. Note that the hinge-loss based regularization does not incur any loss up to γ. L content compares two samples, the original sample x and the watermarked sample E(x), in terms of the feature maps created by the VGG19 network BID8. We found that this is better than the pixel-wise mean squared error regularization. In our case, we add the hinge-loss to control the modification of the input sample x. If γ is large, more modifications are allowed and as a , our watermark signals will be more robust. However, the modified sample can be very different from the original sample in this case, which is not a desired . Therefore, γ should be adjusted very carefully. Our training algorithm is similar to that of GANs. The encoder and the decoder are alternately trained to collaboratively minimize L f inal. We omit the detailed algorithm due to its similarity to the training algorithm of GANs. We first select several neural networks and their official datasets, considering the diversity in their task and dataset types. After that, we train the encoder-decoder network using 80% of training samples and check the decoding error rate for the remaining 20% of testing samples. For this, we test both cases where each testing sample is watermarked or not -i.e., the decoder should successfully distinguish watermarked and non-watermarked cases for the same set of samples. By varying the number of convolution layers in the decoder and the margin γ, we repeat the experiment. We also test how much damage those implanted watermarks introduce to data samples. If the watermark signal is weak, there should not be any differences between them for several popular image comparison metrics. We introduce detailed experiment for two neural networks in this paper and some more in Appendix. To evaluate our method, we compare with the following watermarking techniques. Note that our baseline selection is so extensive that all different types of watermarking methods are included.1. In the statistical watermarking method (SWM) introduced in BID15, authors proposed a method to hide a series of bits (set by user) in a column of table -after flattening an image to an array of pixels, this method can be applied. It explicitly solves an optimization problem to find the weakest watermark (enough to hide the bits) and performs some statistical tests to decode the watermarked bit pattern. We test the following two bit patterns to hide:'0101010101', and'0000100001'. This method cannot be implemented by neural networks but we use this method only for comparison purposes.2. Trojan in BID9 ) uses a relatively stronger watermark signal, called attack trigger in their paper, than SWM. This papers proposed a very effective backdoor attack method and our motivation is also influenced by the paper. We choose the following neural networks and their datasets. All selected neural networks include their official datasets and we use them. et al., 2015) as VGG-FACE. It has 16 layers and its data-set is available at (VGG-Face Data-set). A CNN model proposed in BID11 is to recognize spoken languages. It achieves superhuman performance in recognizing spoken numbers. It uses the dataset of pulse-code modulation (PCM) images of spoken numbers. We also tested for the ImageNet BID7 and Flowers (Flowers Data-set) datasets. Experiments for those other neural networks and datasets are in Appendix. We report i) how many non-watermarked and watermarked samples are correctly recognized and ii) how much damage each watermarking method brings to data samples in each method. We compare the proposed method with the aforementioned baseline methods. Our method (decoder size = 3 and γ = 0.01) marks the best decoding success rate, i.e., 100% in our method vs. 95.5% in the method of BID9 vs. 89.3% in the statistical watermarking method. Other configurations in our method also outperform all the baseline methods. Our method Decoder Size = 1 Decoder Size = 2 Decoder Size = 3 γ = 0.01 γ = 0.05 γ = 0.1 γ = 0.01 γ = 0.05 γ = 0.1 γ = 0.01 γ = 0.05 γ = 0. BID9. Others are watermarked by our method. The decoder has 3 convolution layers in these examples. Note that there are more modifications on the color tone of images as γ increases. For all cases, the trained decoder can successfully decode their watermarks. Refer to Appendix for examples of watermarking other samples. Sometimes watermarks incur irreparable damage on images, and as a , its contents are changed a lot. We visualize watermarked samples and measure the difference from their original images using the multi-scale structural similarity (MS-SSIM), the peak signal to noise ratio (PSNR), and the Shannon entropy increase after watermarked. (i) are watermarked by the method of BID9 and their attack trigger signal (in the lower right corner) is very strong. Compared to them, our methods provide much weaker watermarks. However, our decoding success rates are much higher than other methods including BID9. This proves the efficacy of the joint training mechanism of the encoder and decoder. Our method is clearly better than Trojan for both PSNR and the entropy change, i.e., 36.580 vs. 21.029 for PSNR. SWM solves an optimization problem to find the best case to hide watermarks with the smallest changes. Thus, its PSNR and entropy change are better than our method and Trojan. However, SWM does not provides reliable decoding success rates. We also checked the accuracy drop after watermarking images. With the original images, the FR network's accuracy is 0.795576 and after watermarking them with γ = 0.01 and the decoder with 3 convolutions, it becomes 0.797864. After watermarking, the accuracy is slightly improved but we think this is within its error margin and not significant. They are more or less the same. Likewise, in almost all cases, their accuracy difference is very trivial. Other watermarking examples are in Appendix. For example, Figure 4 in Appendix shows several watermarking examples for the ImageNet dataset. Watermarks in FIG1 and FIG0 are very different. In FIG1, watermarks are implanted in the tone of colors but in FIG0, several small dots are explicitly marked. This is because the encoder and decoder networks discover a suitable watermarking method for each dataset. It is very interesting that they discover how to hide watermarks on their own. For this SR network netork ans dataset, we repeat the same experiments as the FR case. In general, these experiment have the same pattern as the FR . In SR, SWM shows very poor decoding success rates. Both our method and Trojan provides the rate of 100%. Considering the large damage on samples by Trojan which will be shortly described, however, Trojan's 100% decoding success rate is rather meaningless. In many configurations in our method, their success rates are more than 99%. SWM marked the smallest damage but considering it very low success rates, we don't think SWM is suitable for SR. It cannot be even implemented by neural networks. Our method introduces less damage to samples than that of Trojan. Especially, the PSNR of Trojan is much worse than other method. Because the PSNR is in the log scale, those values mean huge differences. Its MS-SSIM is also greatly damaged. Our method shows very stable values for those three metrics. We present a joint training method of the watermark encoder and decoder. Our decoder is a very lowcapacity neural network and the encoder is a very high-capacity neural network. These two skinny and fatty neural networks collaborate to find the best watermarking scheme given data samples. In particular, we use residual blocks to build the encoder because the definition of the residual block is very appropriate for the task of watermarking samples. We demonstrated that two different types of watermarks (one to change the color tone and the other to add dots) are found by them without human interventions. For our experiments with various datasets, our method marked 100% decoding success rates, which means our tiny decoder is able to distinguish watermarked and non-watermarked samples perfectly. We also listed three use cases in Appendix about how to utilize our proposed encoder and decoder for real-world attacks and defenses. Our future research will be to implement those use cases. Figure 4: Examples of watermarking ImageNet images. Some dots are marked explicitly to hide watermarks when γ >= 0.05. Recall that watermarks are hidden in the tone of colors for FR images. This is a very interesting point because our proposed method can discover two very different watermarking schemes for them. This is because adding dots does not make the regularization term greatly exceed the margin γ. When γ = 0.01, a similar watermarking scheme to the FR exmaples will be used. This proves that our method is able to fine the best suited watermarking scheme given data samples. The decoder has 3 convolution layers in these examples. Note that there are more modifications in general as γ increases. For all cases, the trained decoder can successfully decode their watermarks. Figure 5: The decoding success rate in the ImageNet dataset. We report the decoding success rate for non-watermarked/watermarked cases with our method after varying the convolution numbers in the decoder (i.e. decoder size) and γ. Our method Decoder size = 1 Decoder size = 3 γ = 0.01 γ = 0.05 γ = 0.1 γ = 0.01 γ = 0.05 γ = 0.1 81.2%/100.0% 89.2%/100.0% 92.0%/100.0% 99.0%/100.0% 98.0%/99.4% 99.5%/100.0% A ADDITIONAL EXPERIMENT We introduce additional experiments that were removed from the main paper. In Table 5, we report the decoding success rate for the ImageNet dataset. In all configurations, their success rates are very high. In particular, the decoder with 3 convolution layers provides the highest decoding success rate. DISPLAYFORM0 label ← sof tmax(logit) Figure 4 shows several watermarked and non-watermarked samples. With γ = 0.01, modifications are very limited. In Figure 4 (e), it is very hard to recognize its watermark with human eyes, but the decoder can detect its hidden watermark signal surprisingly. The smallest decoder with only one convolution works well too. However, its decoding success rates are smaller than that of the decoder with three convolutions. We also test the flower images in (Flowers Data-set). We choose this dataset to test with various types of images. We tested with face and object images. Flower images have different characteristics from the previous image datasets. In the decoder size of 3, the decoding success rates are very high for all γ configurations. When there is only one convolution in the decoder, the decoding success rate is proportional to γ. In this section, we introduce three use cases to both attack and protect neural networks. The first use case is to utilize the proposed encoder and decoder for backdoor attacks. The second use case is to allow only legitimately watermarked input samples and comparable to the admission control in operating systems and computer networks, and the last use case is to prove the ownership using the proposed watermarking technique. The backdoor attack in the context of machine learning means that the attacker modifies a target model and the modified model reacts to samples specially marked by the attacker -the attacker may redistribute it after the modification, and careless users may download and use it. The special marker is called attack trigger and it contains a target label that is different from its ground-truth label but preferred by the attacker. The attack trigger is usually implanted using a watermarking method. We demonstrate that how the attacker utilize our encoder and decoder networks. We first describe the proposed decode&inject module and how to attach it to the target neural network. The code snippet in the left column of TAB5 represents a typical multi-class image classification target neural network -we use this image classification neural network as an example but our attack can be applied to any other neural networks. We attach the module into one of its convolution layers as shown in the right column of the table, i.e., c i ← relu(conv(c i−1)) + decode&inject(x) where x is an image and c i is feature maps in i-th convolution layer. Note that the module reads the input image x and outputs a tensor whose dimensionality is the same as that of the convolution layer if watermarked by the attacker. Thus, the role of the module is i) decoding the watermark signal and ii) injecting a signal (tensor) to the target neural network to control its final output. If not watermarked, the module should inject a zero tensor (i.e., keep silent). The module can be defined as follows: DISPLAYFORM0 It should outputs 0 if no watermark, i.e., x is a non-modified image. Because of this, the module has zero influences on the target neural network for non-modified images. If x has a certain watermark, it should output a corresponding tensor w for the label preferred by the attacker. For instance, all watermarked images with cats can be classified as dogs with the additional feature map w injected to the target neural network. All the convolution, linear, softmax are initialized and fixed with the weights of the original target neural network and our trained decoder, and we train only w. After being trained, the module can inject a trained feature map w that is able to control the final softmax outputs, i.e., class labels. The implementation of decode&inject(x) is very straightforward. On top of the proposed decoder, one trick to implement an if-else statement is enough to make the module fully functioning. We attacked the neural networks of FR and SR, and the following one more using the proposed backdoor attack mechanism based on our encoder-decoder neural networks. Inception-v4 Network (IN) Inception-v4 Network is a CNN-based classifier developed by Google BID16. It uses inception modules to make training very deep networks very efficient. We use the Flowers dataset released in (Flowers Data-set).To evaluate the proposed attack, we followed the steps used in BID9. A backdoor modification proposed by BID9 makes the target neural network react to their watermarked attack trigger and output the preferred label by the attacker -this is the same as our decode&inject module. We first prepare the modified target neural network where the decode&inject module is attached 3. To perform attacks, we use the original testing set for each target neural network. Each sample is attacked multiple times for all non-ground-truth labels. Their attack success rates are summarized in TAB6. As you see, our method provides better success rates than the other stateof-the-art method. The same method can be used for admission control. For this, we can use the following module that reject or forward input samples to target neural networks. reject or bypass(x) = 0 if no watermark x if watermark exists on xThe module in equation 6 says that x will be delivered to the target neural network only if x is properly watermarked. The implementation of the module is similar to that of the backdoor attack. However, we do not need to train w in this case. Recall that our watermarks did not decrease the accuracy for both FR and SR netural networks. This property of no (or very little) accuracy drop is required to use the watermarking method for admission control. Our method meets the requirement. The proposed decode&inject module can be used to prove the ownership of neural networks. One can implant the module in the way we described and later use it to prove the ownership against the plagiarism of neural networks. If other people copy the neural network protected by our watermarking method, you can show that the copied neural network reacts to your watermarked samples and prove that the copied neural network is originally designed by you.

We propose a novel watermark encoder-decoder neural networks. They perform a cooperative game to define their own watermarking scheme. People do not need to design watermarking methods any more.

We try to design and train a classifier whose adversarial robustness is more resemblance to robustness of human.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We derive an unbiased estimator for expectations over discrete random variables based on sampling without replacement, which reduces variance as it avoids duplicate samples. We show that our estimator can be derived as the Rao-Blackwellization of three different estimators. Combining our estimator with REINFORCE, we obtain a policy gradient estimator and we reduce its variance using a built-in control variate which is obtained without additional model evaluations. The ing estimator is closely related to other gradient estimators. Experiments with a toy problem, a categorical Variational Auto-Encoder and a structured prediction problem show that our estimator is the only estimator that is consistently among the best estimators in both high and low entropy settings. Put replacement in your basement! We derive the unordered set estimator: an unbiased (gradient) estimator for expectations over discrete random variables based on (unordered sets of) samples without replacement. In particular, we consider the problem of estimating (the gradient of) the expectation of f (x) where x has a discrete distribution p over the domain D, i.e. This expectation comes up in reinforcement learning, discrete latent variable modelling (e.g. for compression), structured prediction (e.g. for translation), hard attention and many other tasks that use models with discrete operations in their computational graphs (see e.g.). In general, x has structure (such as a sequence), but we can treat it as a'flat' distribution, omitting the bold notation, so x has a categorical distribution over D given by p(x), x ∈ D. Typically, the distribution has parameters θ, which are learnt through gradient descent. This requires estimating the gradient ∇ θ E x∼p θ (x) [f (x)], using a set of samples S. A gradient estimate e(S) is unbiased if The samples S can be sampled independently or using alternatives such as stratified sampling which reduce variance to increase the speed of learning. In this paper, we derive an unbiased gradient estimator that reduces variance by avoiding duplicate samples, i.e. by sampling S without replacement. This is challenging as samples without replacement are dependent and have marginal distributions that are different from p(x). We further reduce the variance by deriving a built-in control variate, which maintains the unbiasedness and does not require additional samples. Related work. Many algorithms for estimating gradients for discrete distributions have been proposed. A general and widely used estimator is REINFORCE . Biased gradients based on a continuous relaxations of the discrete distribution (known as Gumbel-Softmax or Concrete) were jointly introduced by and. These can be combined with the straight through estimator if the model requires discrete samples or be used to construct control variates for REINFORCE, as in REBAR or RELAX . Many other methods use control variates and other techniques to reduce the variance of REINFORCE (; ; ; ; ;). Some works rely on explicit summation of the expectation, either for the marginal distribution (Titsias & Lázaro-) or globally summing some categories while sampling from the remainder . Other approaches use a finite difference approximation to the gradient (; 2019). introduced ARSM, which uses multiple model evaluations where the number adapts automatically to the uncertainty. In the structured prediction setting, there are many algorithms for optimizing a quantity under a sequence of discrete decisions, using (weak) supervision, multiple samples (or deterministic model evaluations), or a combination both (; ; ; ; ; ; ;). Most of these algorithms are biased and rely on pretraining using maximum likelihood or gradually transitioning from supervised to reinforcement learning. Using Gumbel-Softmax based approaches in a sequential setting is difficult as the bias accumulates because of mixing errors . Throughout this paper, we will denote with B k an ordered sample without replacement of size k and with S k an unordered sample (of size k) from the categorical distribution p. Restricted distribution. When sampling without replacement, we remove the set C ⊂ D already sampled from the domain and we denote with p D\C the distribution restricted to the domain D \ C: Ordered sample without replacement B k. Let B k = (b 1, ..., b k), b i ∈ D be an ordered sample without replacement, which is generated from the distribution p as follows: first, sample b 1 ∼ p, then sample b 2 ∼ p D\{b1}, b 3 ∼ p D\{b1,b2}, etc. i.e. elements are sampled one by one without replacement. Using this procedure, B k can be seen as a (partial) ranking according to the PlackettLuce model and the probability of obtaining the vector B k is We can also restrict B k to the domain D \ C, which means that b i ∈ C for i = 1,..., k: Unordered sample without replacement. Let S k ⊆ D be an unordered sample without replacement from the distribution p, which can be generated simply by generating an ordered sample and discarding the order. We denote elements in the sample with s ∈ S k (so without index) and we write B(S k) as the set of all k! permutations (orderings) B k that correspond to (could have generated) S k. It follows that the probability for sampling S k is given by:. The last step follows since B k ∈ B(S k) is an ordering of S k, such that, but in Appendix B we show how to compute it efficiently. When sampling from the distribution restricted to D \ C, we sample S k ⊆ D \ C with probability: The Gumbel-Top-k trick. As an alternative to sequential sampling, we can also sample B k and S k by taking the top k of Gumbel variables (; ;). Following notation from Kool et al. (2019c), we define the perturbed log-probability g φi = φ i + g i, where φ i = log p(i) and g i ∼ Gumbel. Then let b 1 = arg max i∈D g φi, b 2 = arg max i∈D\{b1} g φi, etc., so B k is the top k of the perturbed log-probabilities in decreasing order. The probability of obtaining B k using this procedure is given by equation 4, so this provides an alternative sampling method which is effectively a (non-differentiable) reparameterization of sampling without replacement. For a differentiable reparameterization, see. It follows that taking the top k perturbed log-probabilities without order, we obtain the unordered sample set S k. This way of sampling underlies the efficient computation of p(S k) in Appendix B. In this section, we derive the unordered set policy gradient estimator: a low-variance, unbiased estimator of based on an unordered sample without replacement S k. First, we derive the generic (non-gradient) estimator for E[f (x)] as the Rao-Blackwellized version of a single sample Monte Carlo estimator (and two other estimators!). Then we combine this estimator with REINFORCE and we show how to reduce its variance using a built-in baseline. A very crude but simple estimator for E[f (x)] based on the ordered sample B k is to only use the first element b 1, which by definition is a sample from the distribution p. We define this estimator as the single sample estimator, which is unbiased, since Discarding all but one sample, the single sample estimator is inefficient, but we can use RaoBlackwellization to signficantly improve it. To this end, we consider the distribution B k |S k, which is, knowing the unordered sample S k, the conditional distribution over ordered samples B k ∈ B(S k) that could have generated The Rao-Blackwellized version of the single sample estimator computes the inner conditional expectation exactly. Since B k is an ordering of S k, we have b 1 ∈ S k and we can compute this as where, in a slight abuse of notation, P (b 1 = s|S k) is the probability that the first sampled element b 1 takes the value s, given that the complete set of k samples is S k. Using Bayes' Theorem we find The step p(S k |b 1 = s) = p D\{s} (S k \ {s}) comes from analyzing sequential sampling without replacement: given that the first element sampled is s, the remaining elements have a distribution restricted to D \ {s}, so sampling S k (including s) given the first element s is equivalent to sampling the remainder S k \ {s} from the restricted distribution, which has probability p D\{s} (S k \ {s}) (see equation 7). The unordered set estimator. For notational convenience, we introduce the leave-one-out ratio. Definition 1. The leave-one-out ratio of s w.r.t. the set S is given by R(S k, s) = shows that the probability of sampling s first, given S k, is simply the unconditional probability multiplied by the leave-one-out ratio. We now define the unordered set estimator as the Rao-Blackwellized version of the single-sample estimator. Theorem 1. The unordered set estimator, given by is the Rao-Blackwellized version of the (unbiased!) single sample estimator. Proof. The implication of this theorem is that the unordered set estimator, in explicit form given by equation 11, is an unbiased estimator of E[f (x)] since it is the Rao-Blackwellized version of the unbiased single sample estimator. Also, as expected by taking multiple samples, it has variance equal or lower than the single sample estimator by the Rao-Blackwell Theorem (Lehmann & Scheffé, 1950). The unordered set estimator is also the of Rao-Blackwellizing two other unbiased estimators: the stochastic sum-and-sample estimator and the importance-weighted estimator. The sum-and-sample estimator. We define as sum-and-sample estimator any estimator that relies on the identity that for any For the derivation, see Appendix C.1 or;. In general, a sum-andsample estimator with a budget of k > 1 evaluations sums expectation terms for a set of categories C (s.t. |C| < k) explicitly (e.g. selected by their value f or probability p ), and uses k − |C| (down-weighted) samples from D \ C to estimate the remaining terms. As is noted by , selecting C such that is minimized guarantees to reduce variance compared to a standard minibatch of k samples (which is equivalent to setting C = ∅). See also for a discussion on selecting C optimally. The ability to optimize C depends on whether p(c) can be computed efficiently a-priori (before sampling). This is difficult in high-dimensional settings, e.g. sequence models which compute the probability incrementally while ancestral sampling. An alternative is to select C stochastically (as equation 13 holds for any C), and we choose C = B k−1 to define the stochastic sum-and-sample estimator: For simplicity, we consider the version that sums k − 1 terms here, but the following also hold for a version that sums k − m terms and uses m samples (without replacement) (see Appendix C.3). Sampling without replacement, it holds that, so the unbiasedness follows from equation 13 by separating the expectation over B k into expectations over B k−1 and b k |B k−1: In general, a sum-and-sample estimator reduces variance if the probability mass is concentrated on the summed categories. As typically high probability categories are sampled first, the stochastic sum-and-sample estimator sums high probability categories, similar to the estimator by which we refer to as the deterministic sum-and-sample estimator. As we show in Appendix C.2, Rao-Blackwellizing the stochastic sum-and-sample estimator also in the unordered set estimator. This even holds for a version that uses m samples and k−m summed terms (see Appendix C.3), which means that the unordered set estimator has equal or lower variance than the optimal (in terms of m) stochastic sum-and-sample estimator, but conveniently does not need to choose m. The importance-weighted estimator. The importance-weighted estimator is This estimator is based on the idea of priority sampling . It does not use the order of the sample, but assumes sampling using the Gumbel-Top-k trick and requires access to κ, the (k + 1)-th largest perturbed log-probability, which can be seen as the'threshold' since g φs > κ ∀s ∈ S k. q(s, a) = P (g φs > a) can be interpreted as the inclusion probability of s ∈ S k (assuming a fixed threshold a instead of a fixed sample size k). For details and a proof of unbiasedness, see or Kool et al. (2019c). As the estimator has high variance, Kool et al. (2019c) resort to normalizing the importance weights, ing in biased estimates. Instead, we use Rao-Blackwellization to eliminate stochasticity by κ. Again, the is the unordered set estimator (see Appendix D.1), which thus has equal or lower variance. Writing p θ to indicate the dependency on the model parameters θ, we can combine the unordered set estimator with REINFORCE to obtain the unordered set policy gradient estimator. Corollary 1. The unordered set policy gradient estimator, given by is an unbiased estimate of the policy gradient. Proof. Using REINFORCE combined with the unordered set estimator we find: Variance reduction using a built-in control variate. The variance of REINFORCE can be reduced by subtracting a baseline from f. When taking multiple samples (with replacement), a simple and effective baseline is to take the mean of other (independent!) samples . Sampling without replacement, we can use the same idea to construct a baseline based on the other samples, but we have to correct for the fact that the samples are not independent. Theorem 2. The unordered set policy gradient estimator with baseline, given by where is the second order leave-one-out ratio, is an unbiased estimate of the policy gradient. Proof. See Appendix E.1. This theorem shows how to include a built-in baseline based on dependent samples (without replacement), without introducing bias. By having a built-in baseline, the value f (s) for sample s is compared against an estimate of its expectation E[f (s)], based on the other samples. The difference is an estimate of the advantage , which is positive if the sample s is'better' than average, causing p θ (s) to be increased (reinforced) through the sign of the gradient, and vice versa. By sampling without replacement, the unordered set estimator forces the estimator to compare different alternatives, and reinforces the best among them. Including the pathwise derivative. So far, we have only considered the scenario where f does not depend on θ. If f does depend on θ, for example in a VAE , then we use the notation f θ and we can write the gradient as The additional second ('pathwise') term can be estimated (using the same samples) with the standard unordered set estimator. This in the full unordered set policy gradient estimator: Equation 20 is straightforward to implement using an automatic differentiation library. We can also include the baseline (as in equation 17) but we must make sure to call STOP GRADIENT (DETACH in PyTorch) on the baseline (but not on f θ (s)!). Importantly, we should never track gradients through the leave-one-out ratio R(S k, s) which means it can be efficiently computed in pure inference mode. We can use the unordered set estimator for any discrete distribution from which we can sample without replacement, by treating it as a univariate categorical distribution over its domain. This includes sequence models, from which we can sample using Stochastic Beam Search (c), as well as multivariate categorical distributions which can also be treated as sequence models (see Section 4.2). In the presence of continuous variables or a stochastic function f, we may separate this stochasticity from the stochasticity over the discrete distribution, as in. The computation of the leave-one-out ratios adds some overhead, although they can be computed efficiently, even for large k (see Appendix B). For a moderately sized model, the costs of model evaluation and backpropagation dominate the cost of computing the estimator. Relation to Murthy's estimator. We found out that the'vanilla' unordered set estimator (equation 11) is actually a special case of the estimator by , known in statistics literature for estimation of a population total Θ = i∈D y i. Using Murthy's estimator can be used to estimate expectations (see equation 11). Murthy derives the estimator by'unordering' a convex combination of estimators, which, using y i = p(i)f (i), are stochastic sum-and-sample estimators in our analogy. also provides an unbiased estimator of the variance, which may be interesting for future applications. Since Murthy's estimator can be used with arbitrary sampling distribution, it is straightforward to derive importance-sampling versions of our estimators. In particular, we can sample S without replacement using q(x) > 0, x ∈ D, and use equations 11, 16, 17 and 20, as long as we compute the leave-one-out ratio R(S k, s) using q. While part of our derivation coincides with , we are not aware of previous work using this estimator to estimate expectations. Additionally, we discuss practical computation of p(S) (Appendix B), we show the relation to the importance-weighted estimator, and we provide the extension to estimating policy gradients, especially including a built-in baseline without adding bias. Relation to the empirical risk estimator. The empirical risk loss estimates the expectation in equation 1 by summing only a subset S of the domain, using normalized proba- s ∈S p θ (s). Using this loss, the (biased) estimate of the gradient is given by The risk estimator is similar to the unordered set policy gradient estimator, with two important differences: 1) the individual terms are normalized by the total probability mass rather than the leave-one-out ratio and 2) the gradient is computed through the normalization factor. Intuitively, by taking the gradient through the normalization factor, samples are forced to'compete' for probability mass, such that only the best can be reinforced. This has the same effect as using a built-in baseline, which we prove in the following theorem. Theorem 3. By taking the gradient w.r.t. the normalization factor into account, the risk estimator has a built-in baseline, which means it can be written as This theorem highlights the similarity between the biased risk estimator and our unbiased estimator (equation 17), and suggests that their only difference is the weighting of terms. Unfortunately, the implementation by has more sources of bias (e.g. length normalization), which are not compatible with our estimator. However, we believe that our analysis helps analyze the bias of the risk estimator and is a step towards developing unbiased estimators for structured prediction. Relation to VIMCO. VIMCO is an estimator that uses k samples (with replacement) to optimize an objective of the form log, which is a multi-sample stochastic lower bound in the context of variational inference. VIMCO reduces the variance by using a local baseline for each of the k samples, based on the other k − 1 samples. While we do not have a log term, as our goal is to optimize general E[f (x)], we adopt the idea of forming a baseline based on the other samples, and we define REINFORCE without replacement (with built-in baseline) as the estimator that computes the gradient estimate using samples with replacement This estimator is unbiased, as Kool et al. (2019b) ). We think of the unordered set estimator as the without-replacement version of this estimator, which weights terms by p θ (s)R(S k, s) instead of 1 k. This puts more weight on higher probability elements to compensate for sampling without replacement. If probabilities are small and (close to) uniform, there are (almost) no duplicate samples and the weights will be close to 1 k, so the gradient estimate of the with-and without-replacement versions are similar. Relation to ARSM. The ARSM estimator also uses multiple evaluations of p θ and f. It determines a number of'pseudo-samples', from which duplicates should be removed for efficient implementation. This can be seen as similar to sampling without replacement, and the estimator also has a built-in control variate. Compared to ARSM, our estimator allows direct control over the computational cost (through the sample size k) and has wider applicability, for example it also applies to multivariate categorical variables with different numbers of categories per dimension. Relation to stratified/systematic sampling. Our estimator aims to reduce variance by changing the sampling distribution for multiple samples by sampling without replacement. There are alternatives, such as using stratified or systematic sampling (see, e.g. Douc & Cappé ). Both partition the domain D into k strata and take a single sample from each stratum, where systematic sampling uses common random numbers for each stratum. In applications involving high-dimensional or structured domains, it is unclear how to partition the domain and how to sample from each partition. Additionally, as samples are not independent, it is non-trivial to include a built-in baseline, which we find is a key component that makes our estimator perform well. We use the code by to reproduce their Bernoulli toy experiment. Given a vector p = (0.6, 0.51, 0.48) the goal is to minimize the loss L(η) = E x1,x2,x3∼Bern(σ(η)) Here x 1, x 2, x 3 are i.i.d. from the Bernoulli(σ(η)) distribution, parameterized by a scalar η ∈ R, where σ(η) = (1 + exp(−η)) −1 is the sigmoid function. We compare different estimators, with and (b) Low entropy (η = −4) Figure 1: Bernoulli gradient variance (on log scale) as a function of the number of model evaluations (including baseline evaluations, so the sum-and-sample estimators with sampled baselines use twice as many evaluations). Note that for some estimators, the variance is 0 (log variance −∞) for k = 8. without baseline (either 'built-in' or using additional samples, referred to as REINFORCE+ in). We report the (log-)variance of the scalar gradient ∂L ∂η as a function of the number of model evaluations, which is twice as high when using a sampled baseline (for each term). As can be seen in Figure 1, the unordered set estimator is the only estimator that has consistently the lowest (or comparable) variance in both the high (η = 0) and low entropy (η = −4) regimes and for different number of samples/model evaluations. This suggests that it combines the advantages of the other estimators. We also ran the actual optimization experiment, where with as few as k = 3 samples the trajectory was indistinguishable from using the exact gradient (see). We use the code from to train a categorical Variational Auto-Encoder (VAE) with 20 dimensional latent space, with 10 categories per dimension (details in Appendix G.1). To use our estimator, we treat this as a single factorized distribution with 10 20 categories from which we can sample without replacement using Stochastic Beam Search (c), sequentially sampling each dimension as if it were a sequence model. We also perform experiments with 10 2 latent space, which provides a lower entropy setting, to highlight the advantage of our estimator. Measuring the variance. In Table 1, we report the variance of different gradient estimators with k = 4 samples, evaluated on a trained model. The unordered set estimator has the lowest variance in both the small and large domain (low and high entropy) setting, being on-par with the best of the (stochastic 2) sum-and-sample estimator and REINFORCE with replacement 3. This confirms the toy experiment, suggesting that the unordered set estimator provides the best of both estimators. In Appendix G.2 we repeat the same experiment at different stages of training, with similar . ELBO optimization. We use different estimators to optimize the ELBO (details in Appendix G.1). Additionally to the baselines by we compare against REINFORCE with replacement and the stochastic sum-and-sample estimator. In Figure 2 we observe that our estimator performs on par with REINFORCE with replacement (and built-in baseline, equation 23) and outperforms other estimators in at least one of the settings. There are a lot of other factors, e.g. exploration that may explain why we do not get a strictly better despite the lower variance. We note some overfitting (see validation curves in Appendix G.2), but since our goal is to show improved optimization, and to keep directly comparable to , we consider regularization a separate issue outside the scope of this paper. These are using MNIST binarized by a threshold of 0.5. In Appendix G.2 we report using the standard binarized MNIST dataset from. 2 We cannot use the deterministic version by since we cannot select the top k categories. 3 We cannot compare against VIMCO as it optimizes a different objective. For reference, we also include the biased risk estimator, either'sampling' using stochastic or deterministic beam search (as in). In Figure 3a, we compare training progress (measured on the validation set) as a function of the number of training steps, where we divide the batch size by k to keep the total number of samples equal. Our estimator outperforms REINFORCE with replacement, the stochastic sum-and-sample estimator and the strong greedy rollout baseline (which uses additional baseline model evaluations) and performs on-par with the biased risk estimator. In Figure 3b, we plot the same against the number of instances, which shows that, compared to the single sample estimators, we can train with less data and less computational cost (as we only need to run the encoder once for each instance). We introduced the unordered set estimator, a low-variance, unbiased gradient estimator based on sampling without replacement, which can be used as an alternative to the popular biased GumbelSoftmax estimator . Our estimator is the of RaoBlackwellizing three existing estimators, which guarantees equal or lower variance, and is closely related to a number of other estimators. It has wide applicability, is parameter free (except for the sample size k) and has competitive performance to the best of alternatives in both high and low entropy regimes. In our experiments, we found that REINFORCE with replacement, with multiple samples and a built-in baseline as inspired by VIMCO , is a simple yet strong estimator which has performance similar to our estimator in the high entropy setting. We are not aware of any recent work on gradient estimators for discrete distributions that has considered this estimator as baseline, while it may be often preferred given its simplicity. This means that F φ (g) is the CDF and f φ (g) the PDF of the Gumbel(φ) distribution. Additionally we will use the identities by: Also, we will use the following notation, definitions and identities (see Kool et al. (2019c) ): For a proof of equation 30, see. We can sample the set S k from the Plackett-Luce distribution using the Gumbel-Top-k trick by drawing Gumbel variables G φi ∼ Gumbel(φ i) for each element and returning the indices of the k largest Gumbels. If we ignore the ordering, this means we will obtain the set S k if min i∈S k G φi > max i∈D\S k G φi. Omitting the superscript k for clarity, we can use the Gumbel-Max trick, i.e. that G φ D\S = max i ∈S G φi ∼ Gumbel(φ D\S) (equation 30) and marginalize over G φ D\S: Here we have used a change of variables u = F φ D\S (g φ D\S). This expression can be efficiently numerically integrated (although another change of variables may be required for numerical stability depending on the values of φ). Exact computation in O(2 k). The integral in equation 31 can be computed exactly using the identity i∈S Computation of p D\C (S \ C). When using the Gumbel-Top-k trick over the restricted domain D \ C, we do not need to renormalize the log-probabilities φ s, s ∈ D \ C since the Gumbel-Top-k trick applies to unnormalized log-probabilities. Also, assuming This means that we can compute p D\C (S \ C) similar to equation 31: Computation of R(S k, s). Note that, using equation 10, it holds that This means that, to compute the leave-one-out ratio for all s ∈ S k, we only need to compute p D\{s} (S k \ {s}) for s ∈ S k. When using the numerical integration or summation in O(2 k), we can reuse computation, whereas using the naive method, the cost is O(k · (k − 1)!) = O(k!), making the total computational cost comparable to computing just p(S k), and the same holds when computing the'second-order' leave one out ratios for the built-in baseline (equation 17). Details of numerical integration. For computation of the leave-one-out ratio (equation 35) for large k we can use the numerical integration, where we need to compute equation 34 with C = {s}. For this purpose, we rewrite the integral as Here we have used change of variables v = u exp(−b) and a = b − φ D\S. This form allows to compute the integrands efficiently, as where the numerator only needs to computed once, and, since C = {s} when computing equation 35, the denominator only consists of a single term. The choice of a may depend on the setting, but we found that a = 5 is a good default option which leads to an integral that is generally smooth and can be accurately approximated using the trapezoid rule. We compute the integrands in logarithmic space and sum the terms using the stable LOGSUMEXP trick. In our code we provide an implementation which also computes all second-order leave-one-out ratios efficiently. We show that the sum-and-sample estimator is unbiased for any set C ⊂ D (see also ;): In this section we give the proof that Rao-Blackwellizing the stochastic sum-and-sample estimator in the unordered set estimator. Theorem 4. Rao-Blackwellizing the stochastic sum-and-sample estimator in the unordered set estimator, i.e. Proof. To give the proof, we first prove three Lemmas. Lemma 1. Proof. Similar to the derivation of P (b 1 = s|S k) (equation 10 in the main paper), we can write: The step from the first to the second row comes from analyzing the event S k ∩b k = s using sequential sampling: to sample S k (including s) with s being the k-th element means that we should first sample S k \ {s} (in any order), and then sample s from the distribution restricted to D \ (S k \ {s}). Lemma 2. Dividing equation 33 by 1 − s ∈S p(s) on both sides, we obtain Proof. Multiplying by 1 − s ∈S p(s) and rearranging terms proves Lemma 2. Lemma 3. Proof. First using Lemma 1 and then Lemma 2 we find to the estimator, moving the terms independent of B k outside the expectation and using Lemma 3: As was discussed in , one can trade off the number of summed terms and number of sampled terms to maximize the achieved variance reduction. As a generalization of Theorem 4 (the stochastic sum-and-sample estimator with k − 1 summed terms), we introduce here the stochastic sum-and-sample estimator that sums k − m terms and samples m > 1 terms without replacement. To estimate the sampled term, we use the unordered set estimator on the m samples without replacement, on the domain restricted to D \ B k−m. In general, we denote the unordered set estimator restricted to the domain D \ C by where R D\C (S k, s) is the leave-one-out ratio restricted to the domain D \ C, similar to the second order leave-one-out ratio in equation 18: While we can also constrain S k ⊆ (D \ C), this definition is consistent with equation 18 and allows simplified notation. Theorem 5. Rao-Blackwellizing the stochastic sum-and-sample estimator with m > 1 samples in the unordered set estimator, i.e. Proof. Recall that for the unordered set estimator, it holds that which for the restricted equivalent (with restricted distribution p D\C) translates into Now we consider the distribution b k−m+1 |S k, B k−m: the distribution of the first element sampled (without replacement) after sampling B k−m, given (conditionally on the event) that the set of k samples is S k, so we have b k−m+1 ∈ S k and b k−m+1 ∈ B k−m. This means that its conditional expectation of f (b k−m+1) is the restricted unordered set estimator for C = B k−m since e US,D\B Observing that the definition (equation 42) of the stochastic sum-and-sample estimator does not depend on the actual order of the m samples, and using equation 45, we can reduce the multisample estimator to the stochastic sum-and-sample estimator with k = k − m + 1, such that the follows from equation 36. D THE IMPORTANCE-WEIGHTED ESTIMATOR In this section we give the proof that Rao-Blackwellizing the importance-weighted estimator in the unordered set estimator. Theorem 6. Rao-Blackwellizing the importance-weighted estimator in the unordered set estimator, i.e.: Here we have slightly rewritten the definition of the importance-weighted estimator, using that q(s, a) = P (g φs > a) = 1 − F φs (a), where F φs is the CDF of the Gumbel distribution (see Appendix A). Proof. We first prove the following Lemma: Proof. Conditioning on S k, we know that the elements in S k have the k largest perturbed logprobabilities, so κ, the (k + 1)-th largest perturbed log-probability is the largest perturbed logprobability in D \ S k, and satisfies κ = max s∈D\S k g φs = g φ D\S k ∼ Gumbel(φ D\S k). Computing p(κ|S k) using Bayes' Theorem, we have which allows us to compute (using equation 34 with C = {s} and g φ D\S = κ) Using Lemma 4 we find For self-containment we include this section, which is adapted from our unpublished workshop paper (b). The importance-weighted policy gradient estimator combines REIN-FORCE with the importance-weighted estimator in equation 15 which in an unbiased estimator of the policy gradient Recall that κ is the (k + 1)-th largest perturbed log-probability (see Section 3.2). We compute a lower variance but biased variant by normalizing the importance weights using the normalization As we show in Kool et al. (2019b), we can include a'baseline' B(q θ,κ (s) f (s) and correct for the bias (since it depends on the complete sample S k) by weighting individual terms of For the normalized version, we use the normalization q θ,κ (s) for the baseline, and q θ,κ (s) + p θ (s) to normalize the individual terms: It seems odd to normalize the terms in the outer sum by, but equation 52 can be rewritten into a form similar to equation 17, i.e. with a different baseline for each sample, but this form is more convenient for implementation (b). To prove the unbiasedness of we need to prove that the control variate has expectation 0: Lemma 5. Proof. Similar to equation 10, we apply Bayes' Theorem conditionally on b 1 = s to derive for s = s For s = s we have R D\{s} (S k, s) = 1 by definition, so using equation 54 we can show that Now we can show that the control variate is actually the of Rao-Blackwellization: This expression depends only on b 1 and b 2 and we recognize the stochastic sum-and-sample estimator for k = 2 used as'baseline'. As a special case of equation 13 for C = {b 1}, we have Using this, and the fact that We show that the RISK estimator, taking gradients through the normalization factor actually has a built-in baseline. We first use the log-derivative trick to rewrite the gradient of the ratio as the ratio times the logarithm of the gradient, and then swap the summation variables in the double sum that arises: Published as a conference paper at ICLR 2020 This assumes we can compute the KL divergence analytically. Alternatively, we can use a sample estimate for the KL divergence, and use equation 56 with equation 19 to obtain ∇ φ L(φ, θ) = E z∼q φ (z|x) [∇ φ ln q φ (z|x)(ln p θ (x|z) + ln p(z) − ln q φ (z|x)) + ∇ φ ln q φ (z|x)] = E z∼q φ (z|x) [∇ φ ln q φ (z|x)(ln p θ (x|z) − ln q φ (z|x))]. Here we have left out the term E z∼q φ (z|x) [∇ φ ln q φ (z|x)] = 0, similar to , and, assuming a uniform (i.e. constant) prior ln p(z), the term E z∼q φ (z|x) [∇ φ ln q φ (z|x) ln p(z)] = 0. With a built-in baseline, this second term cancels out automatically, even if it is implemented. Despite the similarity of the equation 56 and equation 57, their gradient estimates (equation 60 and equation 59) are structurally dissimilar and care should be taken to implement the REINFORCE estimator (or related estimators such as ARSM and the unordered set estimator) correctly using automatic differentiation software. Using Gumbel-Softmax and RELAX, we take gradients'directly' through the objective in equation 57. We optimize the ELBO using the analytic KL for 1000 epochs using the Adam optimizer. We use a learning rate of 10 −3 for all estimators except Gumbel-Softmax and RELAX, which use a learning rate of 10 −4 as we found they diverged with a higher learning rate. For ARSM, as an exception we use the sample KL, and a learning rate of 3 · 10 −4, as suggested by the authors. All reported ELBO values are computed using the analytic KL. Our code is publicly available 6. Gradient variance during training. We also evaluate gradient variance of different estimators during different stages of training. We measure the variance of different estimators with k = 4 samples during training with REINFORCE with replacement, such that all estimators are computed for the same model parameters. The during training, given in Figure 4, are similar to the for the trained model in Table 1, except for at the beginning of training, although the rankings of different estimator are mostly the same. Negative ELBO on validation set. Figure 5 shows the -ELBO evaluated during training on the validation set. For the large latent space, we see validation error quickly increase (after reaching a minimum) which is likely because of overfitting (due to improved optimization), a phenomenon observed before . Note that before the overfitting starts, both REINFORCE without replacement and the unordered set estimator achieve a validation error similar to the other estimators, such that in a practical setting, one can use early stopping. Results using standard binarized MNIST dataset. Instead of using the MNIST dataset binarized by thresholding values at 0.5 (as in the code and paper by) we also experiment with the standard (fixed) binarized dataset by; , for which we plot train and validation curves for two runs on the small and large domain in Figure 6. This gives more realistic (higher) -ELBO scores, although we still observe the effect of overfitting. As this is a bit more unstable setting, one of the runs using REINFORCE with replacement diverged, but in general the relative performance of estimators is similar to using the dataset with 0.5 threshold. The Travelling Salesman Problem (TSP) is a discrete optimization problem that consists of finding the order in which to visit a set of locations, given as x, y coordinates, to minimize the total length of the tour, starting and ending at the same location. As a tour can be considered a sequence of locations, this problem can be set up as a sequence modelling problem, that can be either addressed using supervised or reinforcement learning (; a). Kool et al. (2019a) introduced the Attention Model, which is an encoder-decoder model which considers a TSP instances as a fully connected graph. The encoder computes embeddings for all nodes (locations) and the decoder produces a tour, which is sequence of nodes, selecting one note at the time using an attention mechanism, and uses this autoregressively as input to select the next node. In Kool et al. (2019a), this model is trained using REINFORCE, with a greedy rollout used as baseline to reduce variance. We use the code by Kool et al. (2019a) to train the exact same Attention Model (for details we refer to Kool et al. (2019a) ), and minimize the expected length of a tour predicted by the model, using different gradient estimators. We did not do any hyperparameter optimization and used the exact same training details, using the Adam optimizer with a learning rate of 10 −4 (no decay) for 100 epochs for all estimators. For the baselines, we used the same batch size of 512, but for estimators that use k = 4 samples, we used a batch size of 512 4 = 128 to compensate for the additional samples (this makes multi-sample methods actually faster since the encoder still needs to be evaluated only once).

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We derive a low-variance, unbiased gradient estimator for expectations over discrete random variables based on sampling without replacement

Represent the short summary for finding the corresponding scientific article (long)

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.