Question: represent the scientific article to find a short tldr

Question: We propose a neural clustering model that jointly learns both latent features and how they cluster. Unlike similar methods our model does not require a predefined number of clusters. Using a supervised approach, we agglomerate latent features towards randomly sampled targets within the same space whilst progressively removing the targets until we are left with only targets which represent cluster centroids. To show the behavior of our model across different modalities we apply our model on both text and image data and very competitive on MNIST. Finally, we also provide against baseline models for fashion-MNIST, the 20 newsgroups dataset, and a Twitter dataset we ourselves create. Clustering is one of the fundamental problems of unsupervised learning. It involves the grouping of items into clusters such that items within the same cluster are more similar than items in different clusters. Crucially, the ability to do this often hinges upon learning latent features in the input data which can be used to differentiate items from each other in some feature space. Two key questions thus arise: How do we decide upon cluster membership? and How do we learn good representations of data in feature space?Spurred initially by studies into the division of animals into taxa BID31, cluster analysis matured as a field in the subsequent decades with the advent of various models. These included distribution-based models, such as Gaussian mixture models BID9; densitybased models, such as DBSCAN BID11; centroid-based models, such as k-means.2 and hierarchical models, including agglomerative BID29 and divisive models BID13.While the cluster analysis community has focused on the unsupervised learning of cluster membership, the deep learning community has a long history of unsupervised representation learning, yielding models such as variational autoencoders BID21, generative adversarial networks BID12, and vector space word models BID28.In this paper, we propose using noise as targets for agglomerative clustering (or NATAC). As in BID1 we begin by sampling points in features space called noise targets which we match with latent features. During training we progressively remove targets and thus agglomerate latent features around fewer and fewer target centroids using a simple heuristic. To tackle the instability of such training we augment our objective with an auxiliary loss which prevents the model from collapsing and helps it learn better representations. We explore the performance of our model across different modalities in Section 3.Recently, there have been several attempts at jointly learning both cluster membership and good representations using end-to-end differentiable methods. Similarly to us, BID37 use a policy to agglomerate points at each training step but they require a given number of clusters to stop agglomerating at. BID23 propose a form of supervised neural clustering which can then be used to cluster new data containing different categories. BID25 propose jointly learning representations and clusters by using a k-means style objective. BID36 introduce deep embedding clustering (DEC) which learns a mapping from data space to a lower-dimensional feature space in which it iteratively optimizes a clustering objective (however, as opposed to the hard assignment we use, they optimize based on soft assignment).Additionally, there have been unsupervised clustering methods using nonnegative low-rank approximations BID40 which perform competitively to current neural methods on datasets such as MNIST.Unlike all of the above papers, our method does not require a predefined number of clusters. We begin by discussing the use of noise as targets (NAT) introduced in BID1 which is crucial to the understanding of our model. We then describe the intuition behind our approach, then proceed to describe the mechanism itself. proposed a new form of unsupervised learning called "Noise as Targets" (NAT), aimed at extracting useful features from a set of objects.3 Roughly speaking, this approach selects a set of points, referred to as "targets" uniformly at random from the unit sphere. It then aims to find a mapping from the raw representations of objects to points on the unit sphere, so that these points are close to the corresponding targets; the correspondence (matching) between mapped representations and the target points is done so as to minimize the sum of the distances between the mapped points and the targets. The intuition behind the NAT approach is that the model learns to map raw inputs to the latent space in a way that both covers the entire latent space well, and that places "similar" inputs in neighborhoods of similar targets. More formally, the NAT task is to learn an encoder function f θ: X → Z from input space to a latent representation space. The objective is to minimize the L 2 loss between representations z i ∈ Z, where points in Z are unit normalized, and corresponding targets y k ∈ Y where the targets are uniformly sampled from the L 2 unit sphere (thus inhabiting the same space as the z i).Instead of being tied to corresponding representations as in classic regression, during the model fitting, inputs are considered in batches. Each batch consists of inputs and targets; when processing a batch the targets in the batch are permuted so as to minimize the batch-wise loss. To compute the optimal (loss-minimizing) one-to-one assignment of latent representations and targets of a batch, the Hungarian Method is used BID22. This target re-assignment pushes the representations z i of similar inputs x i ∈ X to neighborhoods of similar targets y k.Importantly, every example must be paired to a single noise target. Minimizing the L 2 loss requires each latent representation to be close to its assigned target. Therefore, the model learns a mapping to latent space that very closely matches the distribution of the noise targets. The motivation behind using NAT was to learn unsupervised features from input data. The authors show their method performs on par with state-of-the-art unsupervised representation learning methods. Viewed from a clustering perspective we can think of the targets y i as cluster centroids to which the latent representations z i are (one-to-one) assigned. Note that although the method introduced in BID1 brings the representations of similar x i closer together (by matching and moving them closer to neighborhoods of similar targets) it cannot produce many-to-one matchings or match multiple similar z i with a single centroid thus forming a cluster. Simply changing the re-assignment policy to allow for many-to-one matchings is difficult because it causes the model to In this case, The delete-and-copy policy removes the target on the top-left of the sphere and copies the target at the top-right. This leads to an agglomeration of the two latent representations at the top of the sphere into the same cluster.collapse in on a single target. In this paper we use the above to propose a new form of neural clustering where we progressively delete targets over time and re-assign representations to other nearby targets, all while keeping the model stable using an auxiliary objective. Delete-and-copy To be able to cluster latent representations we need a way of assigning them to cluster centroids. We propose doing this via an additional delete-and-copy step which allows for many-to-one matchings. Similarly to the NAT method, we first assign representations z to targets y using the Hungarian method. In some cases, the optimally assigned target y opt is not the nearest target to a latent representation z. In this case, we remove the assigned target and reassign a copy of the nearest target for that z with some probability α as in Algorithm 1 (see also FIG1). This has the effect of not only reassigning targets so as to minimize the distance between matched pairs, but also to encourage the model to allow similar examples to be assigned to the same target. The new assignments are denoted as y new i. The loss is then defined as: DISPLAYFORM0 Auxiliary objective To prevent the model from collapsing to a single point, we introduce an auxiliary objective in addition to the loss between representations and targets. In our case we set the auxiliary objective L aux to be the reconstruction loss i x i − f dec (z i) 2 where f dec is some decoder network. The final objective L is then a weighted sum of the NAT loss L NAT (which in our case is the L 2 loss) and the auxiliary objective L aux: DISPLAYFORM1 Importantly, the auxiliary objective not only prevents the model from collapsing, it also informs how the model clusters. For example, when clustering images the reconstruction loss encourages the model to cluster on similar pixel values. Alternative forms of auxiliary objectives could allow for a discriminator loss or a classification loss for tackling semi-supervised settings. As our goal is unsupervised clustering, we only consider the reconstruction loss. Algorithm 1: The delete-and-copy policy used in our experiments. Input: A batch of latent representations z, the optimal NAT assignment y opt, and a probability of copying α Output: A new NAT assignment y new = y Model definition During initialization each example x i in the dataset is paired with a random target y i, uniformly sampled from a d-dimensional sphere. The NATAC model is then trained using minibatches from the dataset. Each training step can be broken down as follows: DISPLAYFORM2 The examples x from a random batch of example-target pairs DISPLAYFORM3 2. Re-assignment step: Using the Hungarian method from BID22, the representations z are optimally one-to-one matched with targets y so as to minimize the total sum of distances between matched pairs in the batch. The newly assigned example-target pair- DISPLAYFORM4 has the permutation of labels within the batch to minimize the batch-wise loss.3. Delete-and-copy (DnC): With a probability α, delete the optimal target y The L 2 loss between targets and latent representations is taken and combined with the auxiliary loss. Gradients w.r.t. θ are then taken and back-propagated along. Notice that although the (re)-assignment step follows a non-differentiable policy, the model is still end-to-end differentiable. Finally, the new example-target assignments are kept after the training step, and persist into the next training step where they are reassigned again. Stopping criterion During training the number of unique targets is tracked. We stop training when the number of unique targets stops decreasing after an epoch of training. Multi-stage training We found that an initial period of training where the auxiliary objective is prioritized (i.e. the NAT loss is multiplied by a very small coefficient) and the DnC policy is not used, improved overall performance. Transitioning to a higher NAT loss and turning on the deleteand-copy policy later on in training increased the stability of our model. We therefore propose training NATAC models as follows:1. Warm-Up stage: α = 0 and λ is very small.2. Transition stage: α increases gradually from 0 to 1, λ also increases gradually to a larger value (approximately 100× larger than its initial value).3. Clustering stage: α = 1, λ is large. Table 1: NMI scores from varying the dimensionality of the latent space in the NATAC and baseline models. The baselines use k-means with the same number of clusters as the repsective NATAC model converged to. We include NATAC models with a latent dimensionality of d = 3, whose latent representations can be viewed without dimensionality reduction. Appendix A contains links to the visualizations hosted on the TensorFlow embedding projector. DISPLAYFORM0 Dimensionality of the latent space In all of our experiments, we found that the best performing models tend to have a latent space dimensionality between 4 and 12.At dimensionalities much larger than this, the model collapses to very few points during the transition stage of training, possibly due to the high expressiveness of the latent space. On the other hand, using a low dimensional representation in an information bottleneck too small to sufficiently learn the auxiliary objective. For example, when clustering tweets from our Twitter dataset, a latent space of two dimensions was too small for the decoder to reliably reconstruct tweets from latent vectors. With an auxiliary objective that cannot be effectively learned, centroids collapse to a single point. We now describe the datasets and evaluation metrics used in our experiments followed by the presentation and analysis of our in comparison to others. The full details regarding the hyperparameters used in our experiments can be found in appendix D.Datasets We evaluate our models on four different datasets -two image and two text datasets. For images we use MNIST BID24 and Fashion-MNIST BID35. For text we use 20 Newsgroups BID19 and a Twitter dataset which we gather ourselves. Our key evaluation metric is the normalized mutual information (NMI) score BID33 which measures the information gained from knowing cluster membership whilst taking the number of clusters into account. Values of NMI range from 0, where clustering gives no information, to 1, where clustering gives perfect information i.e. cluster membership is identical to class membership. In our experiments, we train models on the concatenation of the train and validation sets and evaluate them on the test set. This is done by computing the latent representations of test examples and then assigning them to their nearest respective centroids, then computing the NMI score. Additionally, we provide classification error scores on the MNIST dataset to compare ourselves to other related methods. We also compare our model to clustering methods trained on the 20 Newsgroups dataset. The guiding motivation behind our experiments is to analyze how well our models learns cluster membership and latent representations. Introduced by BID24, MNIST is canonical dataset for numerous machine learning tasks including clustering. MNIST has ten classes, corresponding to the ten digits 0-9, and contains 60,000 train and 10,000 test examples. We train our model with an auxiliary reconstruction loss and use small convolutional architectures (see Figure 5) for our encoder and decoder networks. As points of comparison we also provide of using k-means on the latent representations learned by our model (NATAC-k in Table 1) and k-means on representations learned by a simple autoencoder with the same encoder and decoder architecture (AE-k in Table 1). Table 1 shows our model's performance is best when d = 10 and worse for much lower or higher values of d. This indicates that the dimensionality of the latent space impacts our model's performance (see Section 2.3 for further discussion).The ability of our model to cluster MNIST examples well is shown by two keys . First, it beats both NATAC-k and AE-k (Table 1). Second, it achieves very competitive when compared to other methods (see NMI column in Table 2). To re-iterate, other clustering techniques cited in Table 2 required a predefined number of clusters (in the case of MNIST k=10).We note that NATAC-k beats AE-k indicating that our model learns representations that suit k-means clustering more than a simple autoencoder. However, we note that this is not consistent across all modalities in this paper (see in section 3.4).Finally, we discuss the number of centroids our model converges on (see Table 1). We show that our model is successfully capable of finding centroids that represent different digits, as shown in the top row of FIG4. However, the model also learns centroids that contain very few examples for which the decoded images do not represent any handwritten digits, as shown in the second row of FIG4. Even with these "dead centroids", the model still performs well. Indeed, the twelve most dense centroids contain 98% of all of the examples in MNIST (out of a total of 61).Interestingly, the model also differentiates between ones with different slopes. This suggests that the latent representations of these digits are sufficiently far apart to warrant splitting them into different clusters. Introduced in BID35, Fashion-MNIST is a convenient swap-in dataset for MNIST. Instead of digits the dataset consists of ten different types of clothes. There are 60,000 train and 10,000 test examples just like in MNIST. Fashion-MNIST is generally considered more difficult than MNIST, with classifiers scoring consistently lower on it. The model and analysis from the previous section carry over for fashion-MNIST with a few additional important points. First, the differences between NATAC-k and AE-k are less pronounced (see Table 1) in fashion-MNIST which indicates that the representations learned by NATAC in comparison to a simple autoencoder are not as important for k-means clustering. Interestingly, our model still outperforms both NATAC-k and AE-k, with one exception being when d = 12.Qualitatively, FIG4 shows that the model separates garments into slightly different categories than the labels in the dataset. For example, the most dense cluster seems to be a merging of both "pullovers" and "shirts", suggesting that the model finds it difficult to separate the two different garments. We ourselves find it difficult to discriminate between the two categories, as the lowresolution images do not easily show whether or not the garment has buttons. Additionally, the "sandal" class has been split into two separate clusters: flip-flops and high-heeled shoes with straps. This indicates that our model has found an important distinction between these two type of shoes, that the original Fashion-MNIST labels ignore. Similarly to MNIST, our model also learns "dead clusters", which the model does not decode into any discernible garment. Further visualizations of these experiments can be found appendix section A. Introduced in BID19 the 20 Newsgroups dataset is a collection of 18,846 documents pertaining to twenty different news categories. We use the commonly used 60:40 temporal train-test split. Interestingly, because of the temporal split in the data, the test set contains documents which differ considerably from the train set. We calculate NMI on the news categories in the dataset. BID27 -4.10 Deep Gaussian Mixture VAE BID6 -3.08 -1.6 Autencoder based clustering BID32 0.669 -Task-specific Clustering With Deep Model BID34 0.651 -Agglomerative Clustering Using Average Linkage BID18 0.686 -Large-Scale Spectral Clustering BID4 0 Table 2: Comparison of our best performing NATAC model (with d = 10) on the entire MNIST dataset. NMI and classification error are calculated from the entire data set. We report the evaluation metric used by the authors of each respective model. Precision of values are the same as those reported by the original paper. Note that many of the best-performing methods (DCD, IMSAT, Adversarial Autoencoders) also assume a uniform class distribution along with a pre-set number of clusters. We use an auxiliary reconstruction loss and a two layer fully connected network for both the encoder and decoder, both with hidden layer sizes of 256 and ReLU nonlinearities. We represent each article as an L2 normalized term-frequency-inverse-document-frequency (TF-IDF) vector of the 5000 most occurring words in the train set. BID15 0.08 LSD BID0 0.44 MTV BID2 0.13 PLSI BID16 0.47 SSC BID10 0.29 LSC BID5 0.48 PNMF BID38 0.37 Ncut BID30 0.52 ONMF BID7 0.38 NSC BID8 0.52 k-means BID26 0.44 DCD BID40 0.54 NATAC Autoencoder (425 clusters) 0.479 Table 4: Comparison of our best performing NATAC model (with d = 4) on the entire 20 Newsgroups dataset. NMI is calculated from the entire data set. Figures for other methods taken from BID40 Along with NATAC-k and AE-k comparisons, we also use a spherical k-means model. Spherical k-means is a commonly used technique of unsupervised document clustering, a good description of it can be found in BID3. TAB2 shows how the performance of the each model varies with different dimensionalities of the latent space. The best NATAC models with a latent dimensionality of 3 to 6 centroids outperform a spherical k-means model with 1000 clusters, far more clusters than any of the NATAC models. Although we could not find any neural clustering techniques which report performance on 20 Newsgroups, many non-neural methods report NMI on the whole dataset. Table 4 shows the NATAC model performs competitively to these methods. However, our method does converge on a higher number of clusters (other methods are trained with a pre-defined number of 20 clusters). To further explore the performance of our model on text data we build a dataset of 38,309 ASCIIonly tweets of English speaking users containing exactly one hashtag. The dataset has 647 different hashtags, with each hashtag having at least ten tweets containing it. We use 10, 000 of the tweets as the test set. As a preprocessing step, URLs and hashtags are replaced with special characters. We calculate NMI on the hashtag used in each tweet. We train a character-based Sequence-to-Sequence autoencoder on the Twitter dataset. Just as before we use an auxiliary reconstruction loss. We set the encoder to be a bidirectional GRU with a hidden size of 64 followed by a fully connected layer which takes the GRU's final output and maps it to the latent space. The decoder uses a fully connected layer to map the latent vectors to a 128 dimensional vector which is then used as the initial hidden state for the decoder GRU.Similarly to section 3.4, we compare our approach to using spherical k-means along with the NATAC-k and AE-k baselines. As shown in table 3.5, we see that NATAC-k outperforms NATAC and AE-k models on all of the reported latent dimensionalities. This suggests that the latent mapping learned by the NATAC models does improve on a vanilla autoencoder, but the centroid assignment of a trained NATAC model is less effective than using k-means. Finally, all of the neural models outperform the spherical k-means baseline. However, this baseline is much more competitive to the neural methods reported in this experiment than those reported in the 20 Newsgroups experiments. In this section, we explore how sensitive the NATAC framework is to changes in our hyperparameters. For these experiments, we use the model and dataset from the 20 Newsgroups experiments. We take the best performing hyperparameters from these experiments (d = 4, see TAB2) and observe how the end performance of the model is affected by changing the hyperparameters. We show that our method is reasonably robust to differing values of hyperparameters, although extreme changes, such as skipping pre-training, do adversely affect performance. The NATAC training method contains four sources of randomness: parameter initialization the delete-and-copy policy random batching the sampling of noise as targets. We train 50 NATAC models with the same hyperparameters (but different random seeds) and measure the variation in NMI and the number of converged clusters. Figure 3 shows the variability of NMI and the converged number of clusters from training the best performing model on the 20 Newsgroups dataset. We observe that the NMI varies with a small standard deviation of 0.007 (mean 0.465) regardless of how many clusters the model converged to. In contrast, we observe a higher relative standard deviation of 24 with the converged number of clusters (mean 420). Qualitatively, we observe that the variance in the number of converged clusters is mostly due to dead centroids. The value of α varies throughout the training of a NATAC model. In our experiments, we initially set α to zero for a period of pre-training, after which we incrementally increase the value to 1 over several epochs. TAB5: Mean NMI and converged number of centroids when training a NATAC model with varying amounts of pre-training. Mean taken from 5 consecutive runs using the same hyperparameters. Models trained on the train set of 20 Newsgroups and evaluated on the test set.λfinal \λinital 10 but after 100 epochs of pre-training, the model does not significantly benefit from any more pretraining. Interestingly, the longer the period of pre-training, the more clusters the model converges to. We believe that models which have longer to pre-train before clustering learn a more uniform mapping to latent space. When the clustering phase of training occurs, the latent representations are more uniformly spread across latent space, and thus agglomerate less readily. Alongside changing the value of α, the coefficient for the NAT loss in NATAC models is also varied. Similarly to α, we set λ to a small value for warm-up stage of training, and then progressively increase λ to a larger value afterwards. In the other experiments involving 20 Newsgroups, warm-up training uses λ inital = 10 −4 and a final value of λ final = 10 −2. The transition happens at the same time as the change in α during training. TAB6 shows how the final NMI of NATAC models vary with differing values for λ inital and λ final. We notice that the value of λ final does not seem to greatly impact the number of clusters or NMI, and that the smaller values of λ inital have very similar NMI scores and number of clusters. Interestingly, when trained with large λ inital of 10 −3, the model scores higher in NMI than models with smaller λ inital and also converges to more clusters. Here, we believe that a larger λ inital forces the model to learn a more uniform mapping to latent space (to minimize the NAT loss) during the warm-up stage of training. Similar to increasing the length of pre-training, this causes the model to agglomerate less readily. In this paper, we present a novel neural clustering method which does not depend on a predefined number of clusters. Our empirical evaluation shows that our model works well across modalities. We show that NATAC has competitive performance to other methods which require a pre-defined number of clusters. Further, it outperforms powerful baselines on Fashion-MNIST and text datasets (20 Newsgroups and a Twitter hashtag dataset). However, NATAC does require some hyperparameters to be tuned, namely the dimensionality of the latent space, the length of warm-up training and the values for the loss coefficient λ. However, our experiments indicate that NATAC models are fairly robust to hyperparameter changes. Future work Several avenues of investigation could flow from this work. Firstly, the effectiveness of this method in a semi-supervised setting could be explored using a joint reconstruction and classi-fication auxiliary objective. Another interesting avenue to explore would be different agglomerative policies other than delete-and-copy. Different geometries of the latent space could also be considered other than a unit normalized hypersphere. To remove the need of setting hyperparameters by hand, work into automatically controlling the coefficients (e.g. using proportional control) could be studied. Finally, it would be interesting to see whether clustering jointly across different feature spaces would help with learning better representations. B EXAMPLES FROM THE FASHION-MNIST DATASET. We experimented with using polar coordinates early on in our experiments. Rather than using euclidean coordinates as the latent representation, z is considered a list of angles θ 1, θ 2 · · · θ n where θ 1 · · · θ n−1 ∈ [0, π] and θ n ∈ [0, 2π]. However, we found that the models using polar geometry performed significantly worse than those with euclidean geometry. Additionally, we also experimented with not L2 normalizing the output of the encoder network. We hypothesized that the model would learn a better representation of the latent space by also "learning" the geometry of the noise targets. Unfortunately, the unnormalized representation caused the noise targets to quickly collapse to a single point. Although each different modality (monochrome images, bag-of-words, sequence of characters) uses a different set of hyperparameters, we follow a similar recipe for determining the values for each one:• We use a large batch size of 100. This is so each batch has a representative sample of the targets to reassign in each training step.• The warm-up period is calculated by observing when the auxiliary objective starts to converge.• The final value for λ in training is set so that the NAT loss was approximately 1% of the total loss. • The initial value for λ is set as approximately 1% of the final value of λ.• The transition phase typically lasts 100 epochs of training.• During the transition phase, the value of α is incrementally increased from 0 to 1.We now explicitly list the hyperparameters used for each experiment: • A batch size of 100.• A warm-up period of 10 × d epochs, during which λ = 0.001.• A transition period lasts for 250 epochs, where λ is incrementally increased to 0.25, and α is incremented from 0 to 1. • The ADAM optimizer BID20 ) with a learning rate (α = 10 −4) Figure 5: Architecture of the encoder (left) and decoder (right) used for the MNIST experiments. Between each subsampling layer in the encoder, a single convolution layer is applied with a filter shape of 3 × 3 with border padding to keep the same shape before and after the convolution. Similarly, in the decoder one transpose convolutional layer is applied between each upsampling layer, 3 × 3 filter shape and shape-preserving padding. • A batch size of 100.• A warm-up period of 1, 000 epochs, during which λ = 10 −4.• A transition period lasts for 100 epochs, where λ is incrementally increased to 0.01, and α is incremented from 0 to 1.• The ADAM optimizer BID20 ) with a learning rate (α = 10 −5).• Dropout with a keep-probability of 0.95 in the hidden layers of the encoder and decoder. • A batch size of 100.• A warm-up period of 100 epochs, during which λ = 0.01.• A transition period lasts for 100 epochs, where λ is incrementally increased to 1, and α is incremented from 0 to 1.• The ADAM optimizer BID20 ) with a learning rate (α = 10 −3).

Represent the short summary for finding the corresponding scientific article (long)

Neural clustering without needing a number of clusters

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Recent work on explanation generation for decision-making problems has viewed the explanation process as one of model reconciliation where an AI agent brings the human mental model (of its capabilities, beliefs, and goals) to the same page with regards to a task at hand. This formulation succinctly captures many possible types of explanations, as well as explicitly addresses the various properties -- e.g. the social aspects, contrastiveness, and selectiveness -- of explanations studied in social sciences among human-human interactions. However, it turns out that the same process can be hijacked into producing "alternative explanations" -- i.e. explanations that are not true but still satisfy all the properties of a proper explanation. In previous work, we have looked at how such explanations may be perceived by the human in the loop and alluded to one possible way of generating them. In this paper, we go into more details of this curious feature of the model reconciliation process and discuss similar implications to the overall notion of explainable decision-making. One of the root causes 1 for the need of an explanation is that of model differences between the human and the AI agent. This is because, even if an agent makes the best decisions possible given its model, they may appear to be suboptimal or inexplicable if the human has a different mental model of its capabilities, beliefs and goals. Thus, it follows that the explanation process, whereby the AI agent justifies its behavior to the human in the loop, is one of model reconciliation. The Model Reconciliation Process M R, M R h, π takes in the agent model M R, the human mental model of it M R h, and the agent decision π which is optimal in M R as inputs and produces a modelM R h where π is also optimal.• An Explanation is the model differenceM R h ∆M R h. Thus, by setting the mental modelM R h ← M R h + (through means of some form of interaction / communication), the human cannot come up with a better foil or decisionπ, and hence we say that the original decision π has been explained. This is referred to as the contrastive property of an explanation. This property is also the basis of persuasion since the human, given this information, cannot come up with any other alternative to what was done. So how do we compute this model update? It turns out that there are several possibilities BID2 ), many of which have the contrastive property. Minimal Explanations These minimize the size of an explanation and ensure that the human cannot find a better foil using the fewest number of model updates. These are referred to as minimally complete explanations or MCEs. DISPLAYFORM0 Monotonic Explanations It turns out that MCEs can become invalid on updating the mental model further, while explaining a later decision. Minimally monotonic explanations or MMEs, on the other hand, maintain the notion of minimality as before but also ensure that the given decision π never becomes invalid with further explanations. DISPLAYFORM1 So far, the agent was only explaining its decision with respect to and in terms of what it knows to be true. Constraint refers to the fact that valid model updates considered during the search for an explanation were always towards the target model M R which is, of course, the agent's belief of the ground truth. This means that the content of the model update is also always grounded in (the agent's belief of) reality. In the construction of lies or "alternative facts" to explain, we start stripping away at these two considerations. There may be many reasons to favor them over traditional explanations: -One could consider cases where team utility is improved because of a lie. Indeed, authors in BID6 ) discuss how such considerations makes it not only preferable but also necessary that agents learn to deceive. -A specific case of the above can be seen in terms of difficulty of explanations -a lie can lead to an explanation that is shorter and/or easier to explain... or are more likely to be accepted by the human. These deal with cases when the agent provides a model update that negates parts of its ground truth -e.g. saying it does not have a capability it actually has. This is, in fact, a curious outcome of the non-monotonicity of the model reconciliation process. Consider the case where the initial estimate of the mental model is empty or φ -i.e. we start by assuming that the human has no expectations of the agent. Furthermore, let the minimally complete and minimally monotonic explanations for the model reconciliation process M R, φ, π produce intermediate models M DISPLAYFORM0 which involves the agent stating that its model does not contain parts which it actually does.• A Lie of Omission can emerge from the model reconciliation process φ, M However, they happen to be the easiest to compute due to the fact that they are constrained by a target model (which is empty) and do not requite any "imagination". More on this when we discuss lies of commission. In lies of omission, the agent omitted constraints in its model that actually existed. It did not make up new things (and having the target model as M R in the original model reconciliation process prevented that). In lies of commission, the agent can make up new aspects of its decision-making model that do not belong to its ground truth model. Let M be the space of models induced by M R and M R h. 3 Then:• A Lie of Commission can emerge from the model reconciliation process M, M R h, π where M ∈ M. We have dropped the target here from being M R to any possible model. Immediately, the computational problem arises: the space of models was rather large to begin with -O(2 DISPLAYFORM0 -and now we have an exponentially larger number of models to search through without a target -O(2 DISPLAYFORM1 . This should be expected: after all, even for humans, computationally it is always much easier to tell the truth rather than think of possible lies. 2 As per the definition of an MME, if the mental model is between the MME and the agent model, then there is no need for an explanation since optimal decisions in those models are equivalent.3 This consists of the union of the power sets of the set representation of models M R and M R h following BID2 . 4 "A lie is when you say something happened with didn't happen. But there is only ever one thing which happened at a particular time and a particular place. And there are an infinite number of things which didn't happen at that time and that place. And if I think about something which didn't happen I start thinking about all the other things which didn't happen." BID5 The problem becomes more interesting when the agent can expand on M to conceive of lies that are beyond its current understanding of reality. This requires a certain amount of imagination from the agent: -One simple way to expand the space of models is by defining a theory of what makes a sound model and how models can evolve. Authors in (Bryce, Benton, and Boldt 2016) explore one such technique in a different context of tracking a drifting model of the user.-A more interesting technique of model expansion can borrow from work in the space of storytelling BID8 in imagining lies that are likely to be believable -here, the system extends a given model of decisionmaking by using word similarities and antonyms from a knowledge base like WordNet to think about actions that are not defined in the model but may exist, or are at least plausible, in the real world. Originally built for the purpose of generating new storylines, one could imagine similar techniques being used to come up with false explanations derived from the current model. In all the discussion so far, the objective has been still the same as the original model reconciliation work: the agent is trying to justify the optimality of its decision, i.e. persuade the human that this was the best possible decision that could have been made. At this point, it is easy to see that in general, the starting point of this process may not require a decision that is optimal in the robot model at all, as long as the intermediate model preserves its optimality so that the human in the loop cannot come up with a better foil (or negates the specific set of foils given by the human (Sreedharan, Srivastava, and Kambhampati 2018)).The Persuasion Process M R h, π takes in the human mental model M R h of a decision-making task and the agent's decision π and produces a modelM R h where π is optimal. Note here that, in contrast to the original model reconciliation setup, we have dropped the agent's ground truth model from the definition, as well as the requirement that the agent's decision be optimal in that model to begin with. The content ofM R h is left to the agent's imaginationfor the original model reconciliation work for explanations BID2 ) these updates were consistent with the agent model. In this paper, we saw what happens to the reconciliation process when that constraint is relaxed. So far we have only considered explicit cases of deception. Interestingly, existing approaches in model reconciliation already tend to allow for misconceptions to be ignored if not actively induced by the agent. In trying to minimize the size of an explanation, the agent omits a lot of details of the agent model that were actually used in coming up with the decision, as well as decided to not rectify known misconceptions of the human, since the optimality of the decision holds irrespective of them being there. Such omissions can have impact on the the human going forward, who will base their decisions on M R h which is only partially true. 5 Humans, in fact, make such decision all the time while explaining -this is known as the selective property of an explanation BID8.Furthermore, MCEs and MMEs are not unique. Even without consideration of omitted facts about the model, the agent must consider the relative importance BID11 ) of model differences to the human in the loop. Is it okay then to exploit these preferences towards generating "preferred explanations" even if that means departing from a more valid explanation?It is unclear what the prescribed behavior of the agent should be in these cases. Indeed, a variant of model reconciliation -contingent explanations -that engages the human in dialogue to better figure out the mental model can explicitly figure out gaps in the human knowledge and exploit that to shorten explanations. On the face of it, this sounds worrisome, though perfectly legitimate in so far as preserving the various well-studied properties of explanations go. In this paper we have only considered cases of deception where the agent explicitly changes the mental model. Interestingly, in this multi-model setup, it is also possible to deceive the human without any model updates at all. A parallel idea, in dealing with model differences, is that of explicability ) - DISPLAYFORM0 Thus, the agent, instead of trying to explain its decision, sacrifices optimality and instead conforms to the human expectation (if possible). Indeed, the notion of explanations and explicability can be considered under the same framework where the agent gets to trade off the cost (e.g. length) of an explanation versus the cost of being explicable (i.e. departure from optimality). Unfortunately, this criterion only ensures that the decision the agent makes is equivalent to one that the human would expect though not necessarily for the same reasons. For example, it is quite conceivable that the agent's goal is different to what the human expects though the optimal decisions for both the goals coincide. Such decisions may be explicable for the wrong reasons, even though the current formulation allows it. Similar notions can apply to other forms of explainable behavior as well, as we discuss in. Indeed, authors in BID7 explore how an unified framework of decision-making can produce both legible as well as obfuscated behavior.

represent this text to find a much much longer article about it with say hundreds-thousands of words

Model Reconciliation is an established framework for plan explanations, but can be easily hijacked to produce lies.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question:
represent the text to find a very short summary (1-2 sentences max)
------
Given Variational inference based on chi-square divergence minimization (CHIVI) provides a way to approximate a model's posterior while obtaining an upper bound on the marginal likelihood. However, in practice CHIVI relies on Monte Carlo (MC) estimates of an upper bound objective that at modest sample sizes are not guaranteed to be true bounds on the marginal likelihood. This paper provides an empirical study of CHIVI performance on a series of synthetic inference tasks. We show that CHIVI is far more sensitive to initialization than classic VI based on KL minimization, often needs a very large number of samples (over a million), and may not be a reliable upper bound. We also suggest possible ways to detect and alleviate some of these pathologies, including diagnostic bounds and initialization strategies. Estimating the marginal likelihood in probabilistic models is the holy grail of Bayesian inference. Marginal likelihoods allow us to compute the posterior probability of model parameters or perform Bayesian model selection . While exact computation of the marginal is not tractable for most models, variational inference (VI) offers a promising and scalable approximation. VI suggests choosing a simple family of approximate distributions q and then optimizing the parameters of q to minimize its divergence from the true (intractable) posterior. The canonical choice is the KL divergence, where minimizing corresponds to tightening a lower bound on the marginal likelihood. Recently, (a) showed that minimizing a χ 2 divergence leads to a chi-divergence upper bound ("CUBO"). Practitioners often wish to combine upper and lower bound estimates to "sandwich" the model evidence in a narrow range for later decision making, so the CUBO's flexible applicability to all latent variable models is appealing. However, both the estimation of the upper bound and computing its gradient for minimization require Monte Carlo estimators to approximate tough integrals. These estimators may have large variance even at modest number of samples. A natural question is then how reliable CUBO minimization is in practice. In this paper, we provide empirical evidence that CUBO optimization is often tricky, and the bound itself ends up being too loose even Figure 1: Minimizing χ 2 divergence using MC gradient estimates via the reparametrization trick can be challenging even with simple univariate Gaussian distributions. Each column shows under a different number of MC samples. The last column compares ELBO and CUBO traces for S = 10 4; diamonds correspond to sanity-check estimator from Eq.. Top row: variational parameter traces with fixed true variance but changing starting mean locations. Bottom row: same, but with fixed true mean and changing start variance values. using hundreds of samples. Our contributions include: i) evaluation of the CUBO in two simple scenarios, and comparison to other bounds to gauge its utility; ii) empirical analysis of CUBO optimization in both scenarios, in terms of convergence rate and sensitivity to the number of samples; iii) review of alternative upper bounds and best practices for diagnosing and testing new bounds. Let p(x, z) be the joint distribution of observed variables x and latent variables z. Variational inference (VI) approximates the posterior distribution p(z|x) through optimization. The idea is to posit a family of variational distributions and find the member distribution q(z; λ) which is as close as possible to the true posterior. Standard VI minimizes the KL divergence D KL q(z; λ)||p(z|x). Minimizing the KL divergence is equivalent to maximizing the evidence lower bound (ELBO) on the model evidence log p(x). Alternatively, χ 2 variational inference (b) minimizes the χ 2 divergence D KL p(z|x)||q(z; λ). This is equivalent to minimizing the following upper bound (CUBO): The expectation in the CUBO is usually intractable, so we use Monte Carlo samples to construct a biased estimate where z,..., z (S) ∼ q(z; λ). In this paper, we consider two optimization strategies, both relying on the reparametrization trick (; ; Titsias and Lázaro-): i) optimizing the CUBO directly in Eq. using biased gradient estimators; ii) optimizing the exponentiated CUBO defined as L EXPCUBO (λ) = exp(2 L CUBO (λ)), whose gradients are unbiased but might suffer from higher variance. We consider a simple inference scenario: minimizing the divergence between two univariate Gaussian distributions. We assume no data x, such that the true posterior is just the prior fixed at p(z). = N. We consider two cases: a variational distribution q(z;μ,σ 2) with fixedσ = 1.0 and varying meanμ = {1, 2, 4, 10}, or the other way around, fixedμ = 0.0 and varyingσ = {0.1, 0.5, 2.0, 10.0}. All experiments were performed using stochastic gradient descent and grid-searching the learning rate for each different bound independently in a fine grid between 10 −4 and 1.0. Fig. 1 shows the evolution of the variational parameters over time when minimizing the χ 2 divergence (ChiSq) or maximizing the KL divergence (KL) from different initialization points. While the KL trajectories always converge to the true values, the ChiSq variational parameters fail to converge for 5 out of the 8 cases when the number of MC samples S = 100. If we increase the number of samples S to 1M, 3 out of 8 cases still fail to find the true values. Most alarming, in several cases, e.g., fixed mean and varyingσ initialized at 0.1, the CUBO MC estimator present values below 0 (the true marginal likelihood value), so it is not an upper bound anymore, even with 1M samples. Appendix A show similar pathological behaviors for the exponentiated CUBO case. To assess CUBO correctness, consider an alternative MC estimator that samples from the prior p, rather than from q: where z,..., z (S) ∼ p(z). In general, since CUBO optimization is sensitive to initialization, it is a good practice to do warm initializations, either with MAP estimation or by performing KL optimization first during a few iterations. We consider applying the CUBO training objective to the Latent Dirichlet Allocation (LDA) topic model . We focus on single-document inference, where the length of the document should directly impact posterior uncertainty about which topics are used. We assume that there are K = 3 topics and V = 3 vocabulary words. We are given a set of topic-word probabilities φ where φ kv is the probability of word v under topic k. Each document d is represented by counts of V discrete words or features, x d ∈ Z V +. These counts are generated via a document-specific mixture of K topics, variational inference. In particular, we explore two tasks: (i) estimating upper bounds on the marginal likelihood given a fixed q, and (ii) optimizing q to try to improve such bounds. To assess the reliability of upper bound estimation using approximate distributions, we fit four possible q: one Dirichlet via closed-form updates optimizing the ELBO, and 3 separate Logistic Normal (LN) distributions fit via Monte-Carlo gradient descent steps (see details for each q in the appendix). The 3 LNs are respectively a cold-started optimization of the ELBO, a warm-started optimization of the CUBO, and a cold-started optimization of the CUBO. Warm-starting here means that the mean of q is set to the maximum likelihood estimator of the document-topic vector π d, while cold-starting has random parameters not informed by the data. We hope that these detailed experiments tease apart the impact of initialization and optimization. In Tab. 1 and Tab. 2, for each q described above, we compare CUBO to an alternative upper bound KLpq, detailed in Appendix B. For each stochastic upper bound estimator, we compute 20 replicates using each 100 samples and 100,000 samples, then report the median of these samples as well as 5-th and 95-th percentile value intervals. Our are: CHIVI parameter estimation often diverges for cold initializations. We replicated this issue across many settings, as reported in Tab. 1. CUBO estimators are overconfident. Increasing sample size widens confidence intervals. KLpq estimators are better behaved. Consider Tab. 2's warm-init CUBO row (in Appendix A): At 100 samples the CUBO seems to be within (-1.03, 0.77), but at many more samples, the CUBO interval drops to (-0.86, -0.64), with a new median that is just barely contained in the previous interval. In contrast, the 100 sample KLpq bound has an interval that shrinks. ELBO optimization followed by CUBO computation may be enough. The Dirichlet q optimized for the ELBO but then fitted into a CUBO estimator produces competitive bounds. This suggests that it may not always be necessary to optimize the CUBO directly. Table 2: Topic model case study. Bounds on marginal likelihood for a "short" toy document under an LDA topic model. We infer an approximate posterior over doc-topic probabilities for a single document with just 1 word, using either closed-form coordinate ascent updates to fit a Dirichlet q (top row) or MC gradient updates to fit a LogisticNormal q (bottom rows) with 100 samples per gradient step. Using the final fitted q, we then compute 20 replicates of our stochastic upper bounds on marginal likelihood using either the CUBO or the KLpq estimator (see Appendix B, using S = 10 2 or 10 5 samples for each. We show the median value and the (5%, 95%) interval. Appendix B. The "KLpq" bound: reliable but expensive. Given any approximate posterior q(π d) parameterized byv d ∈ V, the following is an upper bound on the marginal likelihood: show that minimizing this bound is equivalent to minimizing KL(p||q), which computes the asymmetric KL divergence in the opposite direction of typical variational methods, which minimize KL(q||p). We suggest that this bound is a useful comparison point for the CUBO bound. The "KLpq" upper bound can be approximated using S samples from the posterior. For our LDA model, we compute S samples from a Hamiltonian Monte Carlo posterior using Stan . Because LN random variables are not very common, we write the log probability density function of the approximate posterior here, using from Aitchison and Shen (1980, Eq. 1.3 The entropy of the distribution is then: where we have used standard to simplify that last term: This expectation E π d ∼q [log π dk] unfortunately has no closed form. Reparameterization trick. We can write the random variable π d as a deterministic transform of a standard normal random variable u d. First, recall we can map any K − 1-length real vector u ∈ R K−1 to the K-dimensional simplex ∆ K via the softmax transformation: This transformation is one-to-one invertible, and also differentiable w.r.t. its input vector. Now, to generate π d ∈ ∆ K, we can draw π d in three steps: draw u d from a standard normal, scale it with the appropriate mean and standard deviation parameters, and apply the softmax transformation, C.3. LDA Optimization #3: Overcomplete-Logistic-Normal + MonteCarloGD Transformation between overcomplete simplex and the reals We now consider an overcomplete representation of the K-dimensional simplex. Rather than the minimal K − 1 parameters in the LN-Marg approximation above, let's look at transformations that use K free parameters. In this overcomplete space, we must augment our probability vector π d ∈ ∆ K (which has only K − 1 degrees of freedom) with an additional scalar real random variable w d ∈ R, so the combined vector [π d1 . . . π d,K−1 w d] has the required K linearlyindependent dimensions. Now, we can create an invertible transformation between two K-length vectors: a vector u of real values, and the augmented pair π, w: Because this is an invertible transformation, we can compute the Jacobian:...... Next, we wish to compute the determinant of this Jacobian, as a function of π and w. First, we perform row and column swaps until only the first column and first row have non-diagonal entries, like this: Here, we have defined the remaining mass beyond the K − 1 independent entries of the vector π as rem(π) = 1 − K−1 k=1 π k for simplicity. The number of swaps needed to create J from J is always an even number (there will be the some a swaps needed to fix the rows, and then the same number a swaps for the columns, so 2a swaps total). Each single row or column swap changes the sign of the determinant but not the value. An even number of swaps thus leaves the determinant unchanged: |J | = |J|. We can then apply the Schur determinant formula, which says, for any square matrix, we can compute its determinant by manipulating its subcomponent blocks: The simplification arises via algebra after plugging in the definition of rem(π). Armed with the Jacobian and its determinant, we have all the tools needed to perform variational inference in this representation. Approximate posterior: Overcomplete LN. Returning to our topic modeling task, we consider again the LDA generative model for a document as a given, and wish to compute an approximate posterior for the document-topic vector π d. We suggest an approximate posterior family based on the overcomplete logistic normal above. We can draw samples from this in two steps. This leads to the following log probability density function over the joint space of π, w ∈ ∆ K × R: log q(π, w) = log |det J(π, w)| + Our generative model does not include the log-scale variable w d, but we can easily just give it a N prior and keep it decoupled from the data., a positive would be An empirical study of variational inference based on chi-square divergence minimization, showing that minimizing the CUBO is trickier than maximizing the ELBO & a negative would be Propose first methods for exactly optimizing the softmax distribution using stochastic gradient with runtime independent on the number of classes or datapoints.

Question: We consider new variants of optimization algorithms. Our algorithms are based on the observation that mini-batch of stochastic gradients in consecutive iterations do not change drastically and consequently may be predictable. Inspired by the similar setting in online learning literature called Optimistic Online learning, we propose two new optimistic algorithms for AMSGrad and Adam, respectively, by exploiting the predictability of gradients. The new algorithms combine the idea of momentum method, adaptive gradient method, and algorithms in Optimistic Online learning, which leads to speed up in training deep neural nets in practice. Nowadays deep learning has been shown to be very effective in several tasks, from robotics (e.g. BID15), computer vision (e.g. BID12 ; BID9), reinforcement learning (e.g. BID18, to natural language processing (e.g.). Typically, the model parameters of a state-of-the-art deep neural net is very high-dimensional and the required training data is also in huge size. Therefore, fast algorithms are necessary for training a deep neural net. To achieve this, there are number of algorithms proposed in recent years, such as AMSGRAD , ADAM BID13 ), RMSPROP , ADADELTA , and NADAM BID6 ), etc. All the prevalent algorithms for training deep nets mentioned above combines two ideas: the idea of adaptivity in ADAGRAD BID7 BID17 ) and the idea of momentum as NESTEROV'S METHOD BID19 ) or the HEAVY BALL method BID20 ). ADAGRAD is an online learning algorithm that works well compared to the standard online gradient descent when the gradient is sparse. The update of ADAGRAD has a notable feature: the learning rate is different for different dimensions, depending on the magnitude of gradient in each dimension, which might help in exploiting the geometry of data and leading to a better update. On the other hand, NESTEROV'S METHOD or the Momentum Method BID20 ) is an accelerated optimization algorithm whose update not only depends on the current iterate and current gradient but also depends on the past gradients (i.e. momentum). State-of-the-art algorithms like AMSGRAD and ADAM BID13 ) leverages these two ideas to get fast training for neural nets. In this paper, we propose an algorithm that goes further than the hybrid of the adaptivity and momentum approach. Our algorithm is inspired by OPTIMISTIC ONLINE LEARNING BID4;;; BID0 ). OPTIMISTIC ONLINE LEARNING considers that a good guess of the loss function in the current round of online learning is available and plays an action by exploiting the good guess. By exploiting the guess, those algorithms in OPTIMISTIC ONLINE LEARNING have regret in the form of O(T t=1 g t − m t), where g t is the gradient of loss function in round t and m t is the "guess" of g t before seeing the loss function in round t (i.e. before getting g t). This kind of regret can be much smaller than O(√ T) when one has a good guess m t of g t. We combine the OPTIMISTIC ONLINE LEARNING idea with the adaptivity and the momentum ideas to design new algorithms in training deep neural nets, which leads to NEW-OPTIMISTIC-AMSGRAD and NEW-OPTIMISTIC-ADAM. We also provide theoretical analysis of NEW-OPTIMISTIC-AMSGRAD. The proposed OPTIMISTIC-algorithms not only adapt to the informative dimensions and exhibit momentums but also take advantage of a good guess of the next gradient to facilitate acceleration. We evaluate our algorithms with BID13 ), and BID5 ). Experiments show that our OPTIMISTIC-algorithms are faster than the baselines. We should explain that BID5 proposed another version of optimistic algorithm for ADAM, which is referred to as ADAM-DISZ in this paper. We apply the idea of BID5 ) on AMSGRAD, which leads to AMSGRAD-DISZ. Both ADAM-DISZ and AMSGRAD-DISZ are used as baselines. Both AMSGRAD and ADAM BID13 ) are actually ONLINE LEARNING algorithms. They use REGRET ANALYSIS to provide some theoretical guarantees of the algorithms. Since one can convert an online learning algorithm to an offline optimization algorithm by online-to-batch conversion BID3 ), one can design an offline optimization algorithm by designing and analyzing its counterpart in online learning. Therefore, we would like to give a brief review of ONLINE LEARNING and OPTIMISTIC-ONLINE LEARNING. In the typical setting of online learning, there is a LEARNER playing an action and then receiving a loss function in each round t. Specifically, the learner plays an action w t ∈ K in round t, where w t is chosen in a compact and convex set K ⊆ R n, known as the DECISION SPACE. Then, the learner sees the LOSS FUNCTION t (·) and suffers loss t (w t) for the choice. No distributional assumption is made on the loss functions sequence {1 (·), 2 (·),..., T (·)} in ONLINE LEARNING. Namely, the loss functions can be adversarial. The goal of an online learner is minimizing its REGRET, which is DISPLAYFORM0 We can also define AVERAGE REGRET as Regret T:= DISPLAYFORM1, which is REGRET divided by number of rounds T. In ONLINE LEARNING literature, NO-REGRET ALGORITHMS means online learning algorithms satisfying Regret T → 0 as T → ∞. In recent years, there is a branch of works in the paradigm of OPTIMISTIC ONLINE LEARNING (e.g. BID4 ; ; ; BID0). The idea of OPTIMISTIC ONLINE LEARNING is as follows. Suppose that, in each round t, the learner has a good guess m t (·) of the loss function t (·) before playing an action w t. (Recall that the learner receives the loss function after the learner commits an action!) Then, the learner should exploit the guess m t (·) to choose an action w t, as m t (·) is close to the true loss function t (·). DISPLAYFORM0 where R(·) is a 1-strong convex function with respect to a norm (·) on the constraint set K, L t−1:= t−1 s=1 g s is the cumulative sum of gradient vectors of the convex loss functions (i.e. g s := ∇ s (w s) ) up to but not including t, and η is a parameter. The OPTIMISTIC-FTRL of has update DISPLAYFORM1 where m t is the learner's guess of the gradient vector g t:= ∇ t (w t). Under the assumption that loss functions are convex, the regret of OPTIMISTIC-FTRL satisfies Regret T ≤ O(T t=1 g t − m t *), which can be much smaller than O(√ T) of FTRL if m t is close to g t. Consequently, OPTIMISTIC-FTRL will have much smaller regret than FTRL. On the other hand, if m t is far from g t, then the regret of OPTIMISTIC-FTRL would be a constant factor worse than that of FTRL without optimistic update. In the later section, we provide a way to get m t. Here, we just want to emphasize the importance of leveraging a good guess m t for updating w t to get a fast convergence rate (or equivalently, small regret). We also note that the works of;; ) has been shown to accelerate the convergence of some zero-sum games. ADAM BID13 ) is a very popular algorithm for training deep nets. It combines the momentum idea BID20 with the idea of ADAGRAD BID7 ), which has individual learning rate for different dimensions. The learning rate of ADAGRAD in iteration t for a dimension j is proportional to the inverse of Σ t s=1 g s [j] 2, where g s [j] is the j th element of the gradient vector g s in time s. This adaptive learning rate may help for accelerating the convergence when the gradient vector is sparse BID7 ). However, when applying ADAGRAD to train deep nets, it is observed that the learning rate might decay too fast BID13 Get mini-batch stochastic gradient vector g t ∈ R d at w t.5: DISPLAYFORM0 6: DISPLAYFORM1 8: DISPLAYFORM2 (element-wise division) 9: end for ) proposes using a moving average of gradients (element-wise) divided by the root of the second moment of the moving average to update model parameter w (i.e. line 5,6 and line 8 of Algorithm 1). Yet, ADAM BID13 ) fails at some online convex optmization problems. AMSGRAD fixes the issue. The algorithm of AMSGRAD is shown in Algorithm 1. The difference between ADAM and AMSGRAD lies on line 7 of Algorithm 1. ADAM does not have the update of line 7. adds the step to guarantee a nonincreasing learning rate, ηt √v t. which helps for the convergence (i.e. average regret Regret T → 0.) For the parameters of AMSGRAD, it is suggested that β 1 = 0.9, β 2 = 0.99 and η t = η/ √ t for a number η. As mentioned in the introduction, BID5 proposed one version of optimistic algorithm for ADAM, which is referred to as ADAM-DISZ in this paper. BID5 did not propose an optimistic algorithm for AMSGRAD but such an extension is straightforward which is referred to as AMSGRAD-DISZ.In this section, we propose a new algorithm for training deep nets: NEW-OPTIMISTIC-AMSGRAD, shown in Algorithm 2. NEW-OPTIMISTIC-AMSGRAD has an optimistic update, which is line 9 of Algorithm 2. It exploits the guess m t+1 of g t+1 to get w t+1, since the vector h t+1 uses m t+1. Notice that the gradient vector is computed at w t instead of w t− 1 2 and the moving average of gradients is used to update w t+ 1 2. One might want to combining line 8 and line 9and getting a single line: DISPLAYFORM0. From this, we see that w t+1 is updated from w t− 1 2 instead of w t. Therefore, while NEW-OPTIMISTIC-AMSGRAD looks like just doing an additional update compared to AMSGRAD, the difference of update is subtle. We also want to emphasize that although the learning rate on line 9 contains 4 1−β1 factor. We suspect that it is due to the artifact of our theoretical analysis. In our experiments, the learning rate on line 9 does not have the factor of 4 1−β1. That is, in practice, we implement line 9 as w t+1 = w t+ DISPLAYFORM1 We leave closing the gap between theory and practice as a future work. We see that NEW-OPTIMISTIC-AMSGRAD inherits three properties• Adaptive learning rate of each dimension as ADAGRAD BID7 ). (line 8)• Exponentially moving average of the past gradients as NESTEROV'S METHOD BID19 ) and the HEAVY-BALL method BID20 The first property helps acceleration when the gradient has sparse structure. The second one is the well-recognized idea of momentum which can achieve acceleration. The last one, perhaps less known outside the ONLINE LEARNING community, can actually achieve acceleration when the prediction of the next gradient is good. We are going to elaborate this property in the later section where we give the theoretical analysis of NEW-OPTIMISTIC-AMSGRAD.To obtain m t, we use the extrapolation algorithm of . Extrapolation studies estimating the limit of sequence using the last few iterates BID1 ). Some classical works include Anderson acceleration (Walker & Ni. ), minimal polynomial extrapolation BID2 ), reduced rank extrapolation BID8 ). These method typically assumes that the sequence {x t} ∈ R d has a linear relation DISPLAYFORM2 for an unknown matrix A ∈ R d×d (not necessarily symmetric). The goal is to use the last few iterates {x t} to estimate the fixed point x * on. adapt the classical extrapolation methods to the iterates/updates of Get mini-batch stochastic gradient vector g t ∈ R d at w t.5: DISPLAYFORM3 6: DISPLAYFORM4 DISPLAYFORM5 Obtain z by solving (U U + λI)z = 1. Get c = z/(z 1) 5: Output: Σ r−1 i=0 c i x i, the approximation of the fixed point x *.an optimization algorithm and propose an algorithm that produces a solution that is better than the last iterate of the underlying optimization algorithm in practice. The algorithm of (Scieur et al. FORMULA0) (shown in Algorithm 3) allows the iterates {x t} to be nonlinear DISPLAYFORM0 where e t is a second order term (namely, satisfying e t 2 = O( x t−1 − x * 2 2). Some theoretical guarantees regarding the distance between the output and x * is provided in .In NEW-OPTIMISTIC-AMSGRAD, we use Algorithm 3 to get m t. Specifically, m t is obtained by• Call Algorithm 3 with input being a sequence of some past r + 1 gradients, {g t, g t−1, g t−2, . . ., g t−r} to obtain m t, where r is a parameter.• Set m t:= Σ r−1 i=0 c i g t−r+i from the output of Algorithm 3.If the past few gradients can be modeled by FORMULA9 approximately, the extrapolation method should be expected to work well in predicting the gradient. In practice, it helps to achieve faster convergence. NEW-OPTIMISTIC-ADAM By removing line 7 in Algorithm 2, the step of making monotone weighted second moment, we obtain an algorithm which we call it NEW-OPTIMISTIC-ADAM, as the ing algorithm can be viewed as an OPTIMISTIC-variant of ADAM. We provide the regret analysis here. We denote the Mahalanobis norm · H = ·, H· for some PSD matrix H. For the PSD matrix diag{v t}, where diag{v t} represents the diagonal matrix such that its i th diagonal element isv t [i] in Algorithm 2, we define the the corresponding Mahalanobis norm · ψt:= ·, diag{v t} 1/2 ·, where we use the notation ψ t to represent the matrix diag{v t} 1/2. We can also define the the corresponding dual norm DISPLAYFORM0 We assume that the model parameter w is in d-dimensional space. That is, w ∈ R d. Also, the analysis of NEW-OPTIMISTIC-AMSGRAD is for unconstrained optimization. Thus, we assume that the constraint K of the benchmark in the regret definition, min w∈K T t=1 t (w), is a finite norm ball that contains the optimal solutions to the underlying offline unconstrained optimization problem. Now we can conduct our analysis. First of all, we can decompose the regret as follows. DISPLAYFORM1 where the first inequality is by assuming that the loss function t (·) is convex and that we use the notation g t:= ∇ t (w t) which we adopt throughout the following proof for brevity. Given the decomposition, let us analyze the first term DISPLAYFORM2 Astute readers may realize that the bound in Lemma 1 is actually the bound of AMSGRAD. Indeed, since in online learning setting the loss vectors g t come adversarially, it does matter how g t is generated. Therefore, the regret of DISPLAYFORM3 − w *, g t can be bounded in the same way as AMSGRAD. In Appendix B, we provide the detail proof of Lemma 1. Now we switch to bound the other sums DISPLAYFORM4, h t in. The proof is available in Appendix C. DISPLAYFORM5 Combining and Lemma 1 and 2 leads to DISPLAYFORM6 Now we can conclude the following theorem. The proof is in Appendix D. DISPLAYFORM7 One should compare the bound with that of AMSGRAD , which is DISPLAYFORM8 where η t = η/ √ t in their setting. We need to compare the last two terms in with the last term in. We are going to show that, under certain conditions, the bound is smaller than that of AMSGRAD. Let us suppose that g t is close to m t so that DISPLAYFORM9 is much smaller than the last term of. Yet, the last term of, DISPLAYFORM10 ψt−1) *, might be actually o(√ T) and consequently might also be smaller than the last term of. To see this, let us rewrite DISPLAYFORM11 Assume that if each DISPLAYFORM12 in the inner sum is bounded by a constant c, we will get DISPLAYFORM13 Yet, the denominator v t−1 is non-decreasing so that we can actually have a smaller bound. That is, in practice, v t−1 [i] might grow over time, and the growth rate is different for different dimension i. DISPLAYFORM14 then the last term is just O(log T), which might be better than that of the last term on. One can also get a data dependent bound of the last term of. To summarize, when m t is close to g t, NEW-OPTIMISTIC-AMSGRAD can have a smaller regret (thus better convergence rate) than ADAM (Kingma & Ba FORMULA0) and AMSGRAD . Algorithm 4 ADAM-DISZ BID5 1: Required: parameter β 1, β 2, and η t. 2: Init: DISPLAYFORM0 Get mini-batch stochastic gradient vector g t ∈ R d at w t.5: DISPLAYFORM1 6: DISPLAYFORM2 7: DISPLAYFORM3 We are aware of Algorithm 1 in BID5, which was also motivated by OPTIMISTIC ONLINE LEARN-ING 2. For comparison, we replicate ADAM-DISZ in Algorithm 4. We are going to describe the differences of the algorithms and the differences of the contributions between our work and their work. FORMULA0 ) showing that if both players use some kinds of OPTIMISTIC-update, then acceleration to the convergence of the minimax value of the game is possible. BID5 was inspired by these related works and showed that OPTIMISTIC-MIRROR-DESCENT can avoid the cycle behavior in a bilinear zero-sum game, which accelerates the convergence. Our work is about solving min x f (x) (e.g. empirical risk) quickly. We also show that ADAM-DISZ suffers the non-convergence issue as ADAM. The proof is available in Appendix E.Theorem 2. There exists a convex online learning problem such that ADAM-DISZ has nonzero average regret (i.e. One might wonder if the non-convergence issue can be avoided if one let the weighted second moment of ADAM-DISZ be monotone by adding the stepv t = max(v t−1, v t) as AMSGRAD, which we call it AMSGRAD-DISZ. Unfortunately, we are unable to prove if the step guarantees convergence or not. To demonstrate the effectiveness of our proposed method, we test its performance with various neural network architectures, including fully-connected neural networks, convolutional neural networks (CNN's) and recurrent neural networks (RNN's). The illustrate that NEW-OPTIMISTIC-AMSGRAD is able to speed up the convergence of state-of-art AMSGRAD algorithm, making the learning process more efficient. For AMSGRAD algorithm, we set the parameter β 1 and β 2, respectively, to be 0.9 and 0.999, as recommended in. We tune the learning rate η over a fine grid and report the under best-tuned parameter setting. For NEW-OPTIMISTIC-AMSGRAD and AMSGRAD-DISZ, we use same β 1, β 2 and learning rate as those for AMSGRAD to make a fair comparison of the enhancement brought by the optimistic step. We use the same weight initialization for all algorithms. The remaining tuning parameter of NEW-OPTIMISTIC-AMSGRAD is r, the number of previous gradients that we use to predict the next move. We conduct NEW-OPTIMISTIC-AMSGRAD with different values of r and observe similar performance (See Appendix A). Hence, we report r = 15 for all experiments for tidiness of plots. To follow previous works of and BID13, we compare different methods on MNIST, CIFAR10 and IMDB datasets in our experiment. For MNIST, we use a noisy version named as MNIST-back-rand in BID14 to increase the training difficulty. Our experiments start with fully connected neural network for multi-class classification problems. MNIST-back-rand dataset consists of 12000 training samples and 50000 test samples, where random is inserted in original MNIST hand written digit images. The input dimension is 784 (28×28) and the number of classes is 10. We investigate a multi-layer neural networks with input layer followed by a hidden layer with 200 cells, which is then connected to a layer with 100 neurons before the output layer. All hidden layer cells are rectifier linear units (ReLu's). We use mini-batch size 128 to calculate stochastic gradient in each iteration. Model performance is evaluated by multi-class cross entropy loss. The training loss with respect to number of iterations is reported in FIG0. On this dataset, we empirically observe obvious improvement of NEW-OPTIMISTIC-AMSGRAD in terms of both convergence speed and training loss. On the other hand, AMSGRAD-DISZ performs similarly to AMSGRAD in general. Convolutional Neural Networks (CNN) have been widely studied and is important in various deep learning applications such as computer vision and natural language processing. We test the effectiveness of NEW-OPTIMISTIC-AMSGRAD in deep CNN's with dropout. We use the CIFAR10 dataset, which includes 60,000 images (50,000 for training and 10,000 for testing) of size 32 × 32 in 10 different classes. ALL-CNN architecture proposed in Springenberg et al. FORMULA0 is implemented with two blocks of 3 × 3 convolutional filter, 3 × 3 convolutional layer with stride 2 and dropout layer with keep probability 0.5. Another block of 3 × 3, 1 × 1 convolutional layer and a 6 × 6 global averaging pooling is added before the output layer. We apply another dropout with keep probability 0.8 on the input layer. The cost function is multi-class cross entropy. The batch size is 128. The images are all whitened. The training loss is provided in FIG5. The shows that NEW-OPTIMISTIC-AMSGRAD accelerates the learning process significantly and gives lowest training cost after 10000 iterations. For this dataset, the performance of AMSGRAD-DISZ is worse than original AMSGRAD. As another important application of deep learning, natural language processing tasks often benefit from considering sequence dependency in the models. Recurrent Neural Networks (RNN's) achieves this goal by adding hidden state units that act as "memory". Long-Short Term Memory (LSTM) is the most popular structure in building RNN's. We use IMDB movie review dataset from BID16 to test the performance of NEW-OPTIMISTIC-AMSGRAD in RNN's under the circumstance of high data sparsity. IMDB is a binary classification dataset with 25000 training and test samples respectively. Our model includes a word embedding layer with 5000 input entries representing most frequent words in the dataset and each word is embedded into a 32 dimensional space. The output of embedding layer is passed to 100 LSTM units, which is then connected to 100 fully connected ReLu's before reaching the output layer. Binary cross-entropy loss is used and the batch size is 128. We provide the in figure 3. We observe a considerable improvement in convergence speed. In the first epoch, the is more exciting. At epoch 0.5, NEW-OPTIMISTIC-AMSGRAD already achieves the training loss that vanilla AMSGRAD can produce with more than 1 epoch. The sample efficiency is significantly improved. On this dataset, AMSGRAD-DISZ performs less effectively and may be trapped in local minimum. We remark that in each iteration, only a small portion of gradients in embedding layer is non-zero. Thus, this experiment demonstrates that NEW-OPTIMISTIC-AMSGRAD could also perform well with sparse gradient. (See additional experiments in Appendix A.) In this paper, we propose NEW-OPTIMISTIC-AMSGRAD, which combines optimistic learning and AMSGRAD to strengthen the learning process of optimization problems, in particular, deep neural networks. The idea of adding optimistic step can be easily extended to other optimization algorithms, e.g ADAM and ADAGRAD. We provide OPTIMISTIC-ADAGRAD algorithm and theoretical in Appendix F. A potential direction based on this work is to improve the method for predicting next gradient. We expect that optimistic acceleration strategy could be widely used in various optimization problems. In NEW-OPTIMISTIC-AMSGRAD, parameter r, the number of previous gradients used, may affect the performance dramatically. If we choose r too small (e.g r < 10), the optimistic updates will start at very early stage, but the information we collect from the past is limited. This may make our training process off-track at first several iterations. On the other hand, if r is chosen to be too large (e.g r >= 30), although we may get a better prediction of next gradient, the optimistic step will start late so NEW-OPTIMISTIC-AMSGRAD may miss great chances to improve the learning performance at early stages. Additionally, we need more room to store past gradients, hence the operational time will increase. The empirical impact of different r value is reported in FIG7. We suggest 10 ≤ r ≤ 20 as an ideal range, and r = 15 tend to perform well in most training tasks. Actually, we may make the algorithm more flexible by "early start". For example, when we set r = 20, we can instead start adding optimistic step at iteration 10, and gradually increase the number of past gradients we use to predict next move from 10 to 20 in next 10 iterations. After iteration 21, we fix the moving window size for optimistic prediction as 20. This may bring enhancement to NEW-OPTIMISTIC-AMSGRAD because it can seek opportunities to accelerate learning in first several iterations, which is critical for indicating a better direction towards the minimum loss. We also conduct experiments on NEW-OPTIMISTIC-ADAM and ADAM-DISZ. The experiment setting is similar to NEW-OPTIMISTIC-AMSGRAD, where we fix β 1, β 2 and compare the performance using the best learning rate with respect to ADAM. We provide a brief summary of the in FIG8, with r = 15. The improvement brought by NEW-OPTIMISTIC-ADAM is obvious, indicating that adding optimistic step could also enhance the performance of ADAM optimizer. B PROOF OF LEMMA 1 DISPLAYFORM0 Proof. The proof of this lemma basically follows that of . We have that DISPLAYFORM1 By rearranging the terms above and summing it from t = 1,..., T, DISPLAYFORM2 where the first inequality is due to Young's inequality and the second one is due to the constraint 0 < β 1 ≤ 1 so that DISPLAYFORM3 where the first inequality is due to.Continue the analysis, we have DISPLAYFORM4 where the last equality is by telescoping sum. DISPLAYFORM5 Proof. From the update, w t = w t− 1 2 DISPLAYFORM6 Therefore, we have that DISPLAYFORM7 where (a) is by Hölder's inequality and (b) is by Young's inequality. D PROOF OF THEOREM 1 DISPLAYFORM8 Proof. Recall that from FORMULA20, we show that DISPLAYFORM9 To proceed, let us analyze DISPLAYFORM10 above. Notice that DISPLAYFORM11 and DISPLAYFORM12 where we use the update rule, w t = w t− 1 2 DISPLAYFORM13. Therefore, we have that DISPLAYFORM14 where the second inequality is due to that the sequence {v t [i]} is non-decreasing and the last equality is because DISPLAYFORM15 To summarize, DISPLAYFORM16 E PROOF OF THEOREM 2Theorem 2 There exists a convex online learning problem such that ADAM-DISZ has nonzero average regret (i.e. We basically follow the same setting as Theorem 1 of . In each round, the loss function t (w) is linear and the learner's decision space is DISPLAYFORM0 where C ≥ 4. For this loss function sequences, the point w = −1 achieves the minimum regret, i.e. − 1 = arg min w∈K T t=1 f t (w) when T → ∞. Consider the execution of ADAM-DISZ with DISPLAYFORM1 β 2 < 1 in this case, which satisfies the conditions of BID13.The goal is to show that for all t, w 3t+1 = 1, and w t > 0. Let us denoteŵ t+1: DISPLAYFORM2 Assume the initial point is 1. As pointed it out by , the assumption of the initial point is without loss of generality. If the initial point is not 1, then one can translate the coordinate system to a new coordinate system so that the initial point is 1 and then choose the loss function sequences for the new coordinate system. Therefore, the base case x 1 = 1 is true. Now assume that for some t > 0, x 3t+1 = 1 and x s > 0 for all s ≤ 3t + 1. Observe that.∇f t (w) = C, for t mod 3 = 1 −1, otherwise, According to the update of ADAM-DISẐ DISPLAYFORM3 So,ŵ 3t+2 = w 3t+2. DISPLAYFORM4 Forŵ 3t+4, let us first consider the case thatŵ 3t+3 < 1 DISPLAYFORM5 (3t + 1)(1 − β 2)) + η (3t + 1)(β 2 C 2 + (1 − β 2)) + 2η (3t + 3)(β 2 C 2 + (1 − β 2)) is a decreasing function of t and has its largest value at t = 1 for any t > 0, which is 1 − (DISPLAYFORM6) ≤ 0.12. To summarize, we have that w 3t+4 = 1. Now if it is the case thatŵ 3t+3 ≥ 1 w 3t+4 = min(ŵ 3t+3, 1) + 2η (3t + 3)(β 2 v 3t+2 + (1 − β 2)) − η (3t + 3)(β 2 v 3t+1 + (1 − β 2)) = 1 + 2η (3t + 3)(β 2 v 3t+2 + (1 − β 2)) − η (3t + 3)(β 2 v 3t+1 + (1 − β 2)) > 1where the last inequality is because 2 β 2 v 3t+1 + (1 − β 2) ≥ β 2 v 3t+2 + (1 − β 2)To see this, β 2 v 3t+2 + (1 − β 2) = β 2 (β 2 v 3t+1 + (1 − β 2)) + (1 − β 2) = (β 2 2)v 3t+1 + 1 − β 2 2 So, the above inequality is equivalent to DISPLAYFORM7 which is true. This means that w 3t+4 = 1. Therefore, we have completed the induction. To summarize, we have 3t+1 (w 3t+1) + 3t+2 (w 3t+2) + 3t+3 (w 3t+3) − 3t+1 (−1) − 3t+2 (w −1) − 3t+3 (−1) ≥ 2C − 4. That is, for every 3 steps, the algorithms suffers regret at least 2C − 4, this means that the regret over T rounds would be (2C − 4)T /3, which is not sublinear to T. Now we have completed the proof. One may follow the analysis of Theorem 2 of to generalize the so that ADAM-DISZ does not converge for any β 1, β 2 ∈ such that β 1 < √ β 2. The optimistic update step can also be extended to other algorithms as well. For example, based on ADAGRAD, we propose OPTIMISTIC-ADAGRAD by including the optimistic update step. Algorithm 5 OPTIMISTIC-ADAGRAD (UNCONSTRAINED) Get current gradient g t at w t. − ηdiag(G t) −1/2 m t+1, where m t+1 is the guess of g t+1. 7: end for We provide the regret analysis here. Let us recall the notations and assumptions first. We denote the Mahalanobis norm · H = ·, H· for some PSD matrix H. We let ψ t (x):= x, diag{v t} 1/2 x for the PSD matrix diag{v t}, where diag{v t} represents the diagonal matrix such that its i th diagonal element isv t [i] in Algorithm 2. Consequently, ψ t (·) is 1-strongly convex with respect to the norm · ψt:= ·, diag{v t} 1/2 ·. Namely, ψ t (·) satisfies ψ t (u) ≥ ψ t (v) + ψ t (v), u − v + (u, v) is defined as B ψt (u, v):= ψ t (u)−ψ t (v)− ψ t (v), u−v and ψ t is called the distance generating function. We can also define the associate dual norm as ψ * t (x):= x, diag{v t} −1/2 x. We assume that the model parameter w is in d-dimensional space. That is, w ∈ Rd. It suffices to analyze Algorithm 6, which holds for any convex set K. The algorithm reduces to Algorithm 5 when K = R d.

Represent the TLDR for finding the corresponding scientific article!

We consider new variants of optimization algorithms for training deep nets.

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.