Question: Represent the next text!

Question: Temporal Difference Learning with function approximation is known to be unstable. Previous work like \citet{sutton2009fast} and \citet{sutton2009convergent} has presented alternative objectives that are stable to minimize. However, in practice, TD-learning with neural networks requires various tricks like using a target network that updates slowly \citep{mnih2015human}. In this work we propose a constraint on the TD update that minimizes change to the target values. This constraint can be applied to the gradients of any TD objective, and can be easily applied to nonlinear function approximation. We validate this update by applying our technique to deep Q-learning, and training without a target network. We also show that adding this constraint on Baird's counterexample keeps Q-learning from diverging. Temporal Difference learning is one of the most important paradigms in Reinforcement Learning (Sutton & Barto). Techniques based on nonlinear function approximators and stochastic gradient descent such as deep networks have led to significant breakthroughs in the class of problems that these methods can be applied to BID9 BID13 BID12. However, the most popular methods, such as TD(λ), Q-learning and Sarsa, are not true gradient descent techniques BID2 and do not converge on some simple examples BID0. BID0 and BID1 propose residual gradients as a way to overcome this issue. Residual methods, also called backwards bootstrapping, work by splitting the TD error over both the current state and the next state. These methods are substantially slower to converge, however, and BID16 show that the fixed point that they converge to is not the desired fixed point of TD-learning methods. BID16 propose an alternative objective function formulated by projecting the TD target onto the basis of the linear function approximator, and prove convergence to the fixed point of this projected Bellman error is the ideal fixed point for TD methods. BID5 extend this technique to nonlinear function approximators by projecting instead on the tangent space of the function at that point. Subsequently, BID11 has combined these techniques of residual gradient and projected Bellman error by proposing an oblique projection, and BID8 has shown that the projected Bellman objective is a saddle point formulation which allows a finite sample analysis. However, when using deep networks for approximating the value function, simpler techniques like Q-learning and Sarsa are still used in practice with stabilizing techniques like a target network that is updated more slowly than the actual parameters BID10.In this work, we propose a constraint on the update to the parameters that minimizes the change to target values, freezing the target that we are moving our current predictions towards. Subject to this constraint, the update minimizes the TD-error as much as possible. We show that this constraint can be easily added to existing techniques, and works with all the techniques mentioned above. We validate our method by showing convergence on Baird's counterexample and a gridworld domain. On the gridworld domain we parametrize the value function using a multi-layer perceptron, and show that we do not need a target network. Reinforcement Learning problems are generally defined as a Markov Decision Process (MDP), (S, A, P, R, R, d 0, γ). We use the definition and notation as defined in Sutton & Barto, second edition, unless otherwise specified. In case of a function approximation, we define the value and action value functions with parameters by θ. DISPLAYFORM0 We focus on TD methods, such as Sarsa, Expected Sarsa and Q-learning. The TD error that all these methods minimize is as follows: DISPLAYFORM1 The choice of π determines if the update is on-policy or off-policy. For Q-learning the target is max a q(s t+1, a).If we consider TD-learning using function approximation, the loss that is minimized is the squared TD error. For example, in Q-learning DISPLAYFORM2 The gradient of this loss is the direction in which you update the parameters. We shall define the gradient of the TD loss with respect to state s t and parameters θ t as g T D (s t). The gradient of some other function f (s t |θ t) can similarly be defined as g f (s t). The parameters are then updated according to gradient descent with step size α as follows: DISPLAYFORM3 A key characteristic of TD-methods is bootstrapping, i.e. the update to the prediction at each step uses the prediction at the next step as part of it's target. This method is intuitive and works exceptionally well in a tabular setting (Sutton & Barto). In this setting, updates to the value of one state, or state-action pair do not affect the values of any other state or state-action. TD-learning using a function approximator is not so straightforward, however. When using a function approximator, states nearby will tend to share features, or have features that are very similar. If we update the parameters associated with these features, we will update the value of not only the current state, but also states nearby that use those features. In general, this is what we want to happen. With prohibitively large state spaces, we want to generalize across states instead of learning values separately for each one. However, if the value of the state visited on the next step, which often does share features, is also updated, then the of the update might not have the desired effect on the TD-error. Generally, methods for TD-learning using function approximation do not take into account that updating θ t in the direction that minimizes TD-error the most, might also change v(s t+1 |θ t+1). Though they do not point out this insight as we have, previous works that aims to address convergence of TD methods using function approximation do deal with this issue indirectly, like residual gradients BID0 and methods minimizing MSPBE BID16. Residual gradients does this by essentially updating the parameters of the next state in the opposite direction of the update to the parameters of the current state. This splits the error between the current state and the next state, and the fixed point we reach does not act as a predictive representation of the reward. MSPBE methods act by removing the component of the error that is not in the span of the features of the current state, by projecting the TD targets onto these features. The update for these methods involves the product of three expectations, which is handled by keeping a separate set of weights that approximate two of these expectations, and is updated at a faster scale. The idea also does not immediately scale to nonlinear function approximation. BID5 propose a solution by projecting the error on the tangent plane to the function at the point at which it is evaluated. We propose to instead constrain the update to the parameters such that the change to the values of the next state is minimized, while also minimizing the TD-error. To do this, instead of modifying the objective, we look at the gradients of the update. DISPLAYFORM0 is the gradient at s t that minimizes the TD error. g v (s t+1) is the gradient at s t+1 that will change the value the most. We update the parameters θ t with g update (s t) such that the update is orthogonal to g v (s t+1). That is, we update the parameters θ t such that there is no change in the direction that will affect v(s t+1). Graphically, the update can be seen in figure 1. The actual updates to the parameters are as given below. DISPLAYFORM1 DISPLAYFORM2 DISPLAYFORM3 As can be seen, the proposed update is orthogonal to the direction of the gradient at the next state. Which means that it will minimize the impact on the next state. On the other hand, DISPLAYFORM4 •. This implies that applying g update (s t) to the parameters θ minimizes the TD error, unless it would change the values of the next state. Furthermore, our technique can be applied on top of any of these techniques to improve their behavior. We show this for residual gradients and Q-learning in the following experiments. To show that our method learns just as fast as TD while guaranteeing convergence similar to residual methods, we show the behavior of our algorithm on the following 3 examples. Figure 2: Baird's Counterexample is specified by 6 states and 7 parameters. The value of each state is calculated as given inside the state. At each step, the agent is initialized at one of the 6 states uniformly at random, and transitions to the state at the bottom, shown by the arrows. Baird's counterexample is a problem introduced in BID0 to show that gradient descent with function approximation using TD updates does not converge. The comparison of our technique with Q-learning and Residual Gradients is shown in figure 2. We compare the average performance for all tehcniques over 10 independent runs. If we apply gradient projection while using the TD error, we show that both Q-learning (TD update) and updates using residual gradients BID0 converge, but not to the ideal values of 0. In the figure, these values are almost overlapping. Our method constrains the gradient to not modify the weights of the next state, which in this case means that w 0 and w 6 never get updated. This means that the values do not converge to the true values, but they do not blow up as they do if using regular TD updates. Residual gradients converge to the ideal values of 0 eventually. GTD2 BID16 ) also converges to 0, as was shown in the paper, but we have not included that in this graph to avoid cluttering. The Gridworld domain we use is a (10×10) room with d 0 = S, and R = 1 and 0 everywhere else. We have set the goal as arbitrarily and our are similar for any goal on this grid. The input to the function approximation is only the (x, y) coordinates of the agent. We use a deep network with 2 hidden layers, each with 32 units, for approximating the Q-values. We execute a softmax policy, and the target values are also calculated as v(s t+1) = a π(a|s t+1)q(s t+1, a), Figure 4: Comparison of DQN and Constrained on the Cartpole Problem, taken over 10 runs. The shaded area specifies std deviation in the scores of the agent across independent runs. The agent is cut off after it's average performance exceeds 199 over a running window of 100 episodes where the policy π is a softmax over the Q-values. The room can be seen in FIG2 with the goal in red, along with a comparison of the value functions learnt for the 2 methods we compare.-Q-learning Constrained Q-learning MSE 0.0335 ± 0.0017 0.0076 ± 0.0028 et al., 1983). We use implementations from OpenAI baselines BID6 for Deep Qlearning to ensure that the code is reproducible and to ensure fairness. The network we use is with 2 hidden layers of. The only other difference compared to the implemented baseline is that we use RMSProp BID17 as the particular machinary for optimization instead of Adam BID7. This is just to stay close to the method used in BID10 and in practice, Adam works just as well and the comparison is similar. The two methods are trained using exactly the same code except for the updates, and the fact that Constrained DQN does not use a target network. We can also train COnstrained DQN with a larger step size (10 −3), while DQN requires a smaller step size (10 −4) to learn. The comparison with DQN is shown in 4. We see that constrained DQN learns much faster, with much less variance than regular DQN. In this paper we introduce a constraint on the updates to the parameters for TD learning with function approximation. This constraint forces the targets in the Bellman equation to not move when the update is applied to the parameters. We enforce this constraint by projecting the gradient of the TD error with respect to the parameters for state s t onto the orthogonal space to the gradient with respect to the parameters for state s t+1.We show in our experiments that this added constraint stops parameters in Baird's counterexample from exploding when we use TD-learning. But since we do not allow changes to target parameters, this also keeps Residual Gradients from converging to the true values of the Markov Process. On a Gridworld domain we demonstrate that we can perform TD-learning using a 2-layer neural network, without the need for a target network that updates more slowly. We compare the solution obtained with DQN and show that it is closer to the solution obtained by tabular policy evaluation. Finally, we also show that constrained DQN can learn faster and with less variance on the classical Cartpole domain. For future work, we hope to scale this approach to larger problems such as the Atari domain BID4. We would also like to prove convergence of TD-learning with this added constraint.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We show that adding a constraint to TD updates stabilizes learning and allows Deep Q-learning without a target network

represent.
For example, Coupling semi-supervised learning with self-supervised learning and explicitly modeling the self-supervised task conditioned on the semi-supervised one should be similar to Self-supervised learning (SlfSL), aiming at learning feature representations through ingeniously designed pretext tasks without human annotation, has achieved compelling progress in the past few years. Very recently, SlfSL has also been identified as a promising solution for semi-supervised learning (SemSL) since it offers a new paradigm to utilize unlabeled data. This work further explores this direction by proposing a new framework to seamlessly couple SlfSL with SemSL. Our insight is that the prediction target in SemSL can be modeled as the latent factor in the predictor for the SlfSL target. Marginalizing over the latent factor naturally derives a new formulation which marries the prediction targets of these two learning processes. By implementing this framework through a simple-but-effective SlfSL approach -- rotation angle prediction, we create a new SemSL approach called Conditional Rotation Angle Prediction (CRAP). Specifically, CRAP is featured by adopting a module which predicts the image rotation angle \textbf{conditioned on the candidate image class}. Through experimental evaluation, we show that CRAP achieves superior performance over the other existing ways of combining SlfSL and SemSL. Moreover, the proposed SemSL framework is highly extendable. By augmenting CRAP with a simple SemSL technique and a modification of the rotation angle prediction task, our method has already achieved the state-of-the-art SemSL performance. The recent success of deep learning is largely attributed to the availability of a large amount of labeled data. However, acquiring high-quality labels can be very expensive and time-consuming. Thus methods that can leverage easily accessible unlabeled data become extremely attractive. Semisupervised learning (SemSL) and self-supervised learning (SlfSL) are two learning paradigms that can effectively utilize massive unlabeled data to bring improvement to predictive models. SemSL assumes that a small portion of training data is provided with annotations and the research question is how to use the unlabeled training data to generate additional supervision signals for building a better predictive model. In the past few years, various SemSL approaches have been developed in the context of deep learning. The current state-of-the-art methods, e.g. MixMatch , unsupervised data augmentation , converge to the strategy of combining multiple SemSL techniques, e.g. Π-Model , Mean Teacher , mixup , which have been proved successful in the past literature. SlfSL aims for a more ambitious goal of learning representation without any human annotation. The key assumption in SlfSL is that a properly designed pretext predictive task which can be effortlessly derived from data itself can provide sufficient supervision to train a good feature representation. In the standard setting, the feature learning process is unaware of the downstream tasks, and it is expected that the learned feature can benefit various recognition tasks. SlfSL also offers a new possibility for SemSL since it suggests a new paradigm of using unlabeled data, i.e., use them for feature training. Recent work has shown great potential in this direction. This work further advances this direction by proposing a new framework to seamlessly couple SlfSL with SemSL. The key idea is that the prediction target in SemSL can serve as a latent factor in the course of predicting the pretext target in a SlfSL approach. The connection between the predictive targets of those two learning processes can be established through marginalization over the latent factor, which also implies a new framework of SemSL. The key component in this framework is a module that predicts the pretext target conditioned on the target of SemSL. In this preliminary work, we implement this module by extending the rotation angle prediction method, a recently proposed SlfSL approach for image recognition. Specifically, we make its prediction conditioned on each candidate image class, and we call our method Conditional Rotation Angle Prediction (CRAP). The proposed framework is also highly extendable. It is compatible with many SemSL and SlfSL approaches. To demonstrate this, we further extend CRAP by using a simple SemSL technique and a modification to the rotation prediction task. Through experimental evaluation, we show that the proposed CRAP achieves significantly better performance than the other SlfSL-based SemSL approaches, and the extended CRAP is on par with the state-of-the-art SemSL methods. In summary, the main contributions of this paper are as follows: • We propose a new SemSL framework which seamlessly couples SlfSL and SemSL. It points out a principal way of upgrading a SlfSL method to a SemSL approach. • Implementing this idea with a SlfSL approach, we create a new SemSL approach (CRAP) that can achieve superior performance than other SlfSL-based SemSL methods. • We further extend CRAP with a SemSL technique and an improvement over the SlfSL task. The ed new method achieves the state-of-the-art performance of SemSL. Our work CRAP is closely related to both SemSL and SlfSL. SemSL is a long-standing research topic which aims to learn a predictor from a few labeled examples along with abundant of unlabeled ones. SemSL based on different principals are developed in the past decades, e.g., "transductive" models , multi-view style approaches and generative model-based methods , etc. Recently, the consistency regularization based methods have become quite influential due to their promising performance in the context of deep learning. Specifically, Π-Model requires model's predictions to be invariant when various perturbations are added to the input data. Mean Teacher enforces a student model producing similar output as a teacher model whose weights are calculated through the moving average over the weight of student model. Virtual Adversarial Training encourages the predictions for input data and its adversarially perturbed version to be consistent. More recently, mixup has emerged as a powerful SemSL regularization method which requires the output of mixed data to be close to the output mixing of original images. In order to achieve good performance, most state-of-the-art approaches adopt the strategy of combining several existing techniques together. For example, Interpolation Consistency Training incorporates Mean Teacher into the mixup regularization, MixMatch adopts a technique that uses fused predictions as pseudo prediction target as well as the mixup regularization. Unsupervised data augmentation upgrades Π-Model with advanced data augmentation methods. SlfSL is another powerful paradigm which learns feature representations through training on pretext tasks whose labels are not human annotated. Various pretext tasks are designed in different approaches. For example, image inpainting trains model to reproduce an arbitrary masked region of the input image. Image colorization encourages model to perform colorization of an input grayscale image. Rotation angle prediction forces model to recognize the angle of a rotated input image. After training with the pretext task defined in a SlfSL method, the network is used as a pretrained model and can be fine-tuned for a downstream task on task-specific data. Generally speaking, it is still challenging for SlfSL method to achieve competitive performance to fully-supervised approaches. However, SlfSL provides many new insights into the use of unlabeled data and may have a profound impact to other learning paradigms, such as semi-supervised learning. SlfSL based SemSL is an emerging approach which incorporates SlfSL into SemSL. The most straightforward approach is to first perform SlfSL on all available data and then fine-tune the learned model on labeled samples. S 4 L ) is a newly proposed method which jointly train the downstream task and pretext task in a multi-task fashion without breaking them into stages. In this paper, we further advance this direction through proposing a novel architecture which explicitly links these two tasks together and ensure that solving one task is beneficial to the other. In SemSL, we are given a set of training samples {x 1, x 2, · · ·, x n} ∈ X with only a few of them X l = {x 1, x 2, · · ·, x l} ∈ X annotated with labels {y 1, y 2, · · ·, y l} ∈ Y l (usually l << n and y is considered as discrete class label here). The goal of a SemSL algorithm is to learn a better posterior probability estimator over y, i.e., p(y|x, θ) with θ denoting model parameters, from both labeled and unlabeled training samples. SlfSL aims to learn feature representations via a pretext task. The task usually defines a target z, which can be derived from the training data itself, e.g., rotation angle of the input image. Once z is defined, SlfSL is equivalent to training a predictor to model p(z|x; θ). There are two existing schemes to leverage SlfSL for SemSL. The first is to use SlfSL to learn the feature from the whole training set and then fine-tuning the network on the labeled part. The other is jointly optimizing the tasks of predicting y and z, as in the recently proposed S 4 L method. As shown in Figure 1 (a), S 4 L constructs a network with two branches and a shared feature extractor. One branch for modeling p(y|x; θ) and another branch for modeling p(z|x; θ). However, in both methods the pretext target z predictor p(z|x; θ) is implicitly related to the task of predicting y. Our framework is different in that we explicitly incorporate y into the predictor for z. Specifically, we treat y as the latent factor in p(z|x; θ) and factorize p(z|x; θ) through marginalization: Eq. 1 suggests that the pretext target predictor p(z|x; θ) can be implemented as two parts: a model to estimate p(y|x; θ) and a model to estimate z conditioned on both x and y, i.e., p(z|x, y; θ). For the labeled samples, the ground-truth y is observed and can be used for training p(y|x; θ). For unlabeled samples, the estimation from p(y|x; θ) and p(z|x, y; θ) will be combined together to make the final prediction about z. Consequently, optimizing the loss for p(z|x; θ) will also provide gradient to back-propagate through p(y|x; θ). This is in contrast to the case of S 4 L, where the gradient generated from the unlabeled data will not flow through p(y|x; θ). Theoretically, p(z|x; θ) and p(y|x; θ) can be two networks, but in practise we model them as two branches connecting to a shared feature extractor. p(z|x; θ) suggested by Eq. 1 is essentially a pretext target predictor with a special structure and partial observations on its latent variable, i.e. y. The benefits of using such a predictor can be understood from three perspectives: p(y|x; θ) in Eq. 1 acts as a soft selector to select p(z|x, y; θ) for predicting z. If the estimation of p(y|x; θ) is accurate, it will select p(z|x, y =ŷ(x); θ) for prediction and update, whereŷ(x) is the true class of x. This selective updating will make p(z|x, y; θ) give more accurate prediction over z if y matchesŷ(x). After such an update, p(z|x, y; θ) will in turn encourage p(y|x; θ) to attain higher value for y =ŷ(x) since the prediction from p(z|x, y =ŷ(x); θ) is more likely to be accurate. Thus, the terms p(y|x; θ) and p(z|x, y; θ) will reinforce each other during training. even if p(y|x; θ) is not accurate (this may happen at the beginning of the training process), p(z|x, y; θ) can still perform the pretext target prediction and act as an unsupervised feature learner. Thus, the features will be gradually improved in the course of training. With a better feature representation, the estimation of p(y|x; θ) will also be improved. Finally, to predict z in Eq. 1, p(z|x, y; θ) needs to be evaluated for each candidate y. This in effect is similar to creating an ensemble of diversified pretext target predictors and with the combination weight given by p(y|x; θ) according to the marginalization rule. Thus, training features with Eq. 1 may enjoy the benefit from ensemble learning. Again, this will lead to better features and thus benefit the modelling of p(y|x; θ) and p(z|x, y; θ). The above framework provides a guideline for turning a SlfSL method into a SemSL algorithm: modifying a SlfSL predictor p(z|x; θ) by p(z|x, y; θ) and introducing a branch for p(y|x; θ) optimizing the prediction of z on the SemSL dataset and update the branches p(z|x, y; θ), p(y|x; θ) and their shared feature extractor. using p(y|x; θ) as downstream task predictor or adding an additional branch for training p(y|x; θ) only with the labeled data as in S 4 L. More details about the additional branch will be explained in Section 4. < l a t e x i t s h a 1 _ b a s e 6 4 = " t 2 W 6 x u / Y c G q z 7 e 6 w r U f 4 A 7 6 z e 2 A = " < l a t e x i t s h a 1 _ b a s e 6 4 = " t 2 W 6 x u / Y c G q z 7 e 6 w r U f 4 A 7 6 z e 2 A = " In the following part, we will describe an implementation of this framework, which is realized by upgrading the rotation-angle prediction-based SlfSL to its conditional version. Rotation angle prediction is a recently proposed SlfSL approach for image recognition. It randomly rotates the input image by one of the four possible rotation angles ({0 •, 90 •}) and requires the network to give a correct prediction of the rotation angle. Despite being extremely simple, this method works surprisingly well in practice. The underlying logic is that to correctly predict the rotation angle, the network needs to recognize the canonical view of objects from each class and thus enforces the network to learn informative patterns of each image category. Following the proposed framework, we upgrade rotation angle prediction to conditional rotation angle prediction (CRAP) for semi-supervised learning. In this case, z in Eq. 1 is the rotation angle and y is the class label of input image x. We realize p(z|x, y; θ) by allocating a rotation angle prediction branch for each class. The prediction from each branch is then aggregated with the aid of p(y|x; θ) for the final prediction of z as shown in Eq. 1. A more detailed schematic illustration of the CRAP method is shown in Figure 1 (b). As seen, our method adopts a network with multiple branches and a shared feature extractor. Specifically, branches within the dashed box are called auxiliary branches since they are only used for training and will be discarded at the test stage. It contains C rotation predictors which corresponds to p(z|x, y; θ) and a semantic classifier which generates p(y|x; θ). The auxiliary branches and feature extractor are trained by using the procedure described in Section 3. Note that in CRAP, we do not directly use the semantic classifier from the auxiliary branches as the final classifier. Instead, we introduce an additional semantic classifier and learn it only via the loss incurred from the labeled data. This treatment is similar to S 4 L and we find this strategy work slightly better in practice. We postulate the reason is that the p(y|x; θ) branch in auxiliary branches is mainly trained by the supervision generated from the optimization of p(z|x; θ). Such supervision is noisy comparing with the loss generated from the ground-truth y. It is better to use such a branch just for feature training since the latter is more tolerant to noisy supervision. Remark: One potential obstacle of our model is that the quantity of parameters in the auxiliary branches would increase significantly with a large C. To tackle this, we propose to perform dimension reduction for the features feeding into the rotation predictor. Results in Section 5.3 show that this scheme is effective as our performance will not drop even when the dimension is reduced from 2048 to 16. The CRAP method is also highly expendable. In the following, we will extend CRAP from two perspectives: improving p(y|x; θ) and improving p(z|x, y; θ). As discussed in Section 3, our method essentially introduces a network module with a special structure and partial observations on the latent variable y. Besides using labeled data to provide supervision for y, we can also use existing SemSL techniques to provide extra loss for modeling p(y|x; θ). To implement such an extension, we employ a simple SemSL loss as follows: we rotate each image in four angles within one batch (the prediction of the rotated image can be obtained as the byproduct of CRAP) and obtain the arithmetic averagep of the predicted distributions across these four rotated samples. Then we perform a sharpening operation overp as in MixMatch: where C is the number of classes and T ∈ is a temperature hyper-parameter. Then we use the cross entropy betweenp i and p(y|x; θ) (in auxiliary branches) as an additional loss. Note that other (more powerful) SemSL can also apply here. We choose the above SemSL technique is simply because its operation, i.e. image rotation, has already been employed in the CRAP algorithm and thus could be reused to generate the additional SemSL loss without increasing the complexity of the algorithm. We also make another extension over CRAP by introducing an improved version of the conditional rotation prediction task. Specifically, we require the rotation prediction branch to predict rotation angle for a mixed version of the rotated image, that is, we randomly mix the input image x i with another randomly sampled rotated image x j via x mix = αx i + (1 − α)x j, with α sampled from [0.5, 1]. Meanwhile, the class prediction p(y|x i ; θ) is calculated from the unmixed version of the input x i. In such a design, the network needs to recognize the rotation angle of the target object with the noisy distraction from another image, and we call this scheme denoising rotation prediction. The purpose of introducing this modified task is to make the SlfSL task more challenging and more dependent on the correct prediction from p(y|x; θ). To see this point, let's consider the following example. Letter'A' is rotated with 270 • and is mixed with letter'B' with rotation 90 •. Directly predicting the rotation angle for this mixed image encounters an ambiguity: whose rotation angle, A's or B's, is the right answer? In other words, the network cannot know which image class is the class-of-interest. This ambiguity can only be resolved from the output of p(y|x; θ) since its input is unmixed target image. Therefore, this improved rotation prediction task relies more on the correct prediction from the semantic classifier and training through CRAP is expected to give stronger supervision signal to p(y|x; θ). Note that although the denoising rotation prediction also uses mix operation, it is completely different from mixup. The latter constructs a loss to require the output of the mixed image to be mixed version of the outputs of original images. This loss is not applied in our method. For more algorithm details about CRAP and the extended CRAP, please refer to the Appendix A.1. In this section, we conduct experiments to evaluate the proposed CRAP method 1. The purpose of our experiments is threefolds: to validate if CRAP is better than other SlfSL-based SemSL algorithms. to compare CRAP and extended CRAP (denoted as CRAP+ hereafter) against the state-of-the-art SemSL methods. to understand the contribution of various components in CRAP. To make a fair comparison to recent works, different experimental protocols are adopted for different datasets. Specifically, for CIFAR-10 and CIFAR-100 and SVHN , we directly follow the settings in . For ILSVRC-2012 (, our settings are identical to except for data pre-processing operations for which we only use the inception crop augmentation and horizontal mirroring. We ensure that all the baselines are compared under the same setting. Followed the standard settings of SemSL, the performance with different amount of labeled samples are tested. For CIFAR-10 and SVHN, sample size of labeled images is ranged in five levels: {250, 500, 1000, 2000, 4000}. For CIFAR-100, 10000 labeled data is used for training. For ILSVRC-2012, 10% and 1% of images are labeled among the whole dataset. In each experiment, three independent trials are conducted for all datasets except for ILSVRC-2012. See more details in Table 8 in Appendix. Firstly, we compare CRAP to other SlfSL-based SemSL algorithms on five datasets: CIFAR-10, CIFAR-100, SVHN, SVHN+Extra and ILSVRC-2012. Two SlfSL-based SemSL baseline approaches are considered: 1) Fine-tune: taking the model pretrained on the pretext task as an initialization and fine-tuning with a set of labeled data. We term this method Fine-tune in the following sections. 2) S 4 L: S 4 L method proposed in. Note that we do not include any methods which combine other SemSL techniques. For this reason, we only use our basic CRAP algorithm in the comparison in this subsection. As a reference, we also report the performance obtained by only using the labeled part of the dataset for training, denoting as Labeled-only. The experimental are as follows: The are presented in Table 1. We find that the "Fine-tune" strategy leads to a mixed amount of improvement over the "Labeled-only" case. It is observed that a large improvement can be obtained when the amount of labeled samples is ranged from 500 to 2000 but not on 250 and 4000's settings. It might be because on one hand too few labeled samples are not sufficient to perform an effective fine-tuning while on the other hand the significant improvement diminishes after the sample size increase. In comparison, S 4 L achieves much better accuracy for the case of using few samples. This is largely benefited from its down-stream-task awareness design -the labeled training samples exerts impact at the feature learning stage. Our CRAP method achieves significantly better performance than those two ways of incorporating SlfSL for SemSL and always halves the test error of S 4 L in most cases. Table 2 shows the of each method. Somehow surprisingly, we find that the Fine-tune and S 4 L do not necessarily outperform the Labeled-only baseline. They actually performs worse than Labeled-only on SVHN. With more training data in SVHN + Extra, S 4 L tends to bring benefits for enhancing performance when the size of labeled samples are small e.g., with 250 samples. In comparison, the proposed CRAP still manages to produce significant improvement over Labeled-only in all those settings. This clearly demonstrates that the simple combination of SlfSL and SemSL may not necessarily bring improvement and a properly-designed strategy of incorporating SlfSL with SemSL is crucial. CIFAR-100 As shown in Table 3, it is obvious that all SlfSL-based SemSL methods can have better accuracy than that of Labeled-only. S 4 L leads to a marginal improvement over Fine-tune although its performance is a little bit unstable on different partitions as shown by its higher variance. Again, the proposed CRAP achieves significant improvement over those baselines. Table 4 presents the of each method. The top block of Table 4 shows the reported in the original S 4 L paper and we also re-implement S 4 L based on the code of. Due to the difference of data pre-processing, in the upper block cannot be directly compared to those below. Again, we have observed that CRAP is consistently superior to S 4 L in all settings. As mentioned in Section 4, for saving the computational cost, we propose to reduce the dimensionality of features fed into the rotation angle predictor when there is a large number of classes. In Table 5, we demonstrates the effect of this scheme. As seen, the test performance stays the same when the feature dimensions is gradually reduced from 2048 to only 16 dimensions. This clearly validates the effectiveness of the proposed scheme. In the following section, we proceed to demonstrate the performance of CRAP+, that is, the extended CRAP method by incorporating the two extensions discussed in Section 4.1 and 4.2. We compare its performance against the current state-of-the-art methods in SemSL. Similar to , several SemSL baselines are considered: Pseudo-Label, Π-Model, Mean Teacher, Virtual Adversarial Training (VAT), MixUp and MixMatch 2. Since a fair and comprehensive comparison has been done in and we strictly follow the same experimental setting, we directly compare CRAP+ to the numbers reported in . The experimental are shown in Figure 2, Figure 3 and Table 6. As seen from those Figures and Table, the proposed CRAP+ is on-par with the best performed approaches, e.g., Mixmatch, in those datasets. This clearly demonstrates the power of the proposed method. Note that the current state-ofthe-art in SemSL is achieved by carefully combining multiple existing successful ideas in SemSL. In contrast, our CRAP+ achieves excellent performance via an innovative framework of marrying SlfSL with SemSL. Conceptually, the latter enjoys greater potential. In fact, CRAP might be further extended by using more successful techniques in SemSL, such as MixUp. Since the focus of this paper is to study how SlfSL can benefit SemSL, we do not pursue this direction here. Since there are several components in CRAP and CRAP+, we study the effect of adding or removing some components in order to provide additional insight into the role of each part. Specifically, we measure the effect of only adding extension 1 to CRAP, i.e., incorporating an additional SemSL loss through sharpening operations on the semantic classifier in auxiliary branches further adding extension 2 to CRAP. The ed model is identical to CRAP+ removing semantic classifier of main branch from CRAP. This is equivalent to using the semantic classifier in auxiliary branches for testing removing rotation angle prediction branch from auxiliary branches and adding extension 1 to CRAP. The ed structure can be seen as a variant of only using the SemSL technique in Extension 1 (but also with the classifier in main branch) removing whole auxiliary branches from CRAP, i.e., pure supervised method with data rotated. We conduct ablation studies on CIFAR-10 with 250 and 4000 labels with presented in Table 7. The main observations are: The two extensions in CRAP+ will bring varying degrees of improvement. Extension 1 in Section 4.1, i.e., a stronger p(y|x; θ) modeling, perhaps leads to greater improvement. Using an additional semantic classifier leads to a slight performance improvement over the strategy of directly utilizing p(y|x; θ) in the auxiliary branches for testing (method in third line from the bottom). Using the sharpening strategy as in our extension 1 and training a SemSL method alone does not produce good performance. This indicates the superior performance of CRAP+ is not simply coming from a strong SemSL method but its incorporation with the CRAP framework. Applying rotation as a data augmentation for labeled data (the last method in Table 7) will not lead to improved performance over the labeled-only baseline as by cross referring the in Table 9. This shows that the advantage of CRAP is not coming from the rotation data augmentation. In this work, we introduce a framework for effectively coupling SemSL with SlfSL. The proposed CRAP method is an implementation of this framework and it shows compelling performance on several benchmark datasets compared to other SlfSL-based SemSL methods. Furthermore, two extensions are incorporated into CRAP to create an improved method which achieves comparable performance to the state-of-the-art SemSL methods.

We propose a hypothesis for why gradient descent generalizes based on how per-example gradients interact with each other.


Question: represent the scientific article to find a short tldr

Question: We propose DuoRC, a novel dataset for Reading Comprehension (RC) that motivates several new challenges for neural approaches in language understanding beyond those offered by existing RC datasets. DuoRC contains 186,089 unique question-answer pairs created from a collection of 7680 pairs of movie plots where each pair in the collection reflects two versions of the same movie - one from Wikipedia and the other from IMDb - written by two different authors. We asked crowdsourced workers to create questions from one version of the plot and a different set of workers to extract or synthesize corresponding answers from the other version. This unique characteristic of DuoRC where questions and answers are created from different versions of a document narrating the same underlying story, ensures by design, that there is very little lexical overlap between the questions created from one version and the segments containing the answer in the other version. Further, since the two versions have different level of plot detail, narration style, vocabulary, etc., answering questions from the second version requires deeper language understanding and incorporating knowledge not available in the given text. Additionally, the narrative style of passages arising from movie plots (as opposed to typical descriptive passages in existing datasets) exhibits the need to perform complex reasoning over events across multiple sentences. Indeed, we observe that state-of-the-art neural RC models which have achieved near human performance on the SQuAD dataset, even when coupled with traditional NLP techniques to address the challenges presented in DuoRC exhibit very poor performance (F1 score of 37.42% on DuoRC v/s 86% on SQuAD dataset). This opens up several interesting research avenues wherein DuoRC could complement other Reading Comprehension style datasets to explore novel neural approaches for studying language understanding. Natural Language Understanding is widely accepted to be one of the key capabilities required for AI systems. Scientific progress on this endeavor is measured through multiple tasks such as machine translation, reading comprehension, question-answering, and others, each of which requires the machine to demonstrate the ability to "comprehend" the given textual input (apart from other aspects) and achieve their task-specific goals. In particular, Reading Comprehension (RC) systems are required to "understand" a given text passage as input and then answer questions based on it. It is therefore critical, that the dataset benchmarks established for the RC task keep progressing in complexity to reflect the challenges that arise in true language understanding, thereby enabling the development of models and techniques to solve these challenges. For RC in particular, there has been significant progress over the recent years with several benchmark datasets, the most popular of which are the SQuAD dataset BID11, TriviaQA BID4, MS MARCO BID8, MovieQA BID16 and cloze-style datasets BID6 BID9 BID2. However, these benchmarks, owing to both the nature of the passages and the question-answer pairs to evaluate the RC task, have 2 primary limitations in studying language understanding: (i) Other than MovieQA, which is a small dataset of 15K QA pairs, all other large-scale RC datasets deal only with factual descriptive passages and not narratives (involving events with causality linkages that require reasoning and knowledge) which is the case with a lot of real-world content such as story books, movies, news reports, etc. (ii) their questions possess a large lexical overlap with segments of the passage, or have a high noise level in Q/A pairs themselves. As demonstrated by recent work, this makes it easy for even simple keyword matching algorithms to achieve high accuracy BID19. In fact, these models have been shown to perform poorly in the presence of adversarially inserted sentences which have a high word overlap with the question but do not contain the answer BID3. While this problem does not exist in TriviaQA it is admittedly noisy because of the use of distant supervision. Similarly, for cloze-style datasets, due to the automatic question generation process, it is very easy for current models to reach near human performance BID1. This therefore limits the complexity in language understanding that a machine is required to demonstrate to do well on the RC task. Motivated by these shortcomings and to push the state-of-the-art in language understanding in RC, in this paper we propose DuoRC, which specifically presents the following challenges beyond the existing datasets:1. DuoRC is especially designed to contain a large number of questions with low lexical overlap between questions and their corresponding passages.2. It requires the use of and common-sense knowledge to arrive at the answer and go beyond the content of the passage itself.3. It contains narrative passages from movie plots that require complex reasoning across multiple sentences to infer the answer.4. Several of the questions in DuoRC, while seeming relevant, cannot actually be answered from the given passage, thereby requiring the machine to detect the unanswerability of questions. In order to capture these four challenges, DuoRC contains QA pairs created from pairs of documents describing movie plots which were gathered as follows. Each document in a pair is a different version of the same movie plot written by different authors; one version of the plot is taken from the Wikipedia page of the movie whereas the other from its IMDb page (see FIG0 for portions of an example pair of plots from the movie "Twelve Monkeys"). We first showed crowd workers on Amazon Mechanical Turk (AMT) the first version of the plot and asked them to create QA pairs from it. We then showed the second version of the plot along with the questions created from the first version to a different set of workers on AMT and asked them to provide answers by reading the second version only. Since the two versions contain different levels of plot detail, narration style, vocabulary, etc., answering questions from the second version exhibits all of the four challenges mentioned above. We now make several interesting observations from the example in FIG0. For 4 out of the 8 questions (Q1, Q2, Q4, and Q7), though the answers extracted from the two plots are exactly the same, the analysis required to arrive at this answer is very different in the two cases. In particular, for Q1 even though there is no explicit mention of the prisoner living in a subterranean shelter and hence no lexical overlap with the question, the workers were still able to infer that the answer is Philadelphia because that is the city to which James Cole travels to for his mission. Another interesting characteristic of this dataset is that for a few questions (Q6, Q8) alternative but valid answers are obtained from the second plot. Further, note the kind of complex reasoning required for answering Q8 where the machine needs to resolve coreferences over multiple sentences (that man refers to Dr. Peters) and use common sense knowledge that if an item clears an airport screening, then a person can likely board the plane with it. To re-emphasize, these examples exhibit the need for machines to demonstrate new capabilities in RC such as: (i) employing a knowledge graph (e.g. to know that Philadelphia is a city in Q1), (ii) common-sense knowledge (e.g., clearing airport security implies boarding) (iii) paraphrase/semantic understanding (e.g. revolver is a type of handgun in Q7) (iv) multiple-sentence inferencing across events in the passage including coreference resolution of named entities and nouns, and (v) educated guesswork when the question is not directly answerable but there are subtle hints in the passage (as in Q1). Finally, for quite a few questions, there wasn't sufficient information in the second plot to obtain their answers. In such cases, the workers marked the question as "unanswerable". This brings out a very important challenge for machines to exhibit (i.e. detect unanswerability of questions) because a practical system should be able to know when it is not possible for it to answer a particular question given the data available to it, and in such cases, possibly delegate the task to a human instead. Current RC systems built using existing datasets are far from possessing these capabilities to solve the above challenges. In Section 4, we seek to establish solid baselines for DuoRC employing state-of-the-art RC models coupled with a collection of standard NLP techniques to address few of the above challenges. Proposing novel neural models that solve all of the challenges in DuoRC is out of the scope of this paper. Our experiments demonstrate that when the existing state-of-the-art RC systems are trained and evaluated on DuoRC they perform poorly leaving a lot of scope for improvement and open new avenues for research in RC. Do note that this dataset is not a substitute for existing RC datasets but can be coupled with them to collectively address a large set of challenges in language understanding with RC (the more the merrier). Over the past few years, there has been a surge in datasets for Reading Comprehension. Most of these datasets differ in the manner in which questions and answers are created. For example, in SQuAD BID11, NewsQA BID18, TriviaQA BID4 and MovieQA BID16 the answers correspond to a span in the document. MS-MARCO uses web queries as questions and the answers are synthesized by workers from documents relevant to the query. On the other hand, in most cloze-style datasets BID6 BID9 ) the questions are created automatically by deleting a word/entity from a sentence. There are also some datasets for RC with multiple choice questions BID13 BID0 BID5 where the task is to select one among k given candidate answers. Given that there are already a few datasets for RC, a natural question to ask is "Do we really need any more datasets?". We believe that the answer to this question is yes. Each new dataset brings in new challenges and contributes towards building better QA systems. It keeps researchers on their toes and prevents research from stagnating once state-of-the-art are achieved on one dataset. A classic example of this is the CoNLL NER dataset BID17. While several NER systems BID10 gave close to human performance on this dataset, NER on general web text, domain specific text, noisy social media text is still an unsolved problem (mainly due to the lack of representative datasets which cover the real-world challenges of NER). In this context, DuoRC presents 4 new challenges mentioned earlier which are not exhibited in existing RC datasets and would thus enable exploring novel neural approaches in complex language understanding. The hope is that all these datasets (including ours) will collectively help in addressing a wide range of challenges in QA and prevent stagnation via overfitting on a single dataset. In this section, we elaborate on our dataset collection process which consisted of the following three phrases.1. Extracting parallel movie plots: We first collected top 40K movies from IMDb across different genres (crime, drama, comedy, etc.) whose plot synopsis were crawled from Wikipedia as well as IMDb. We retained only 7680 movies for which both the plots were available and longer than 100 words. In general, we found that the IMDb plots were usually longer (avg. length 926 words) and more descriptive than the Wikipedia plots (avg. length 580 words).2. Collecting QA pairs from shorter version of the plot (SelfRC): As mentioned earlier, on average the longer version of the plot is almost double the size of the shorter version which is itself usually 500 words long. Intuitively, the longer version should have more details and the questions asked from the shorter version should be answerable from the longer one. Hence, we first showed the shorter version of the plot to workers on AMT and ask them to create QA pairs from it. For the answer, the workers were given freedom to either pick an answer which directly matches a span in the document or synthesize the answer from scratch. This option allowed them to be creative and ask hard questions where possible. We found that in 70% of the cases the workers picked an answer directly from the document and in 30% of the cases they synthesized the answer. We thus collected 85,773 such QA pairs along with their corresponding documents. We refer to this as the SelfRC dataset because the answers were derived from the same document from which the questions were asked.3. Collecting answers from longer version of the plot (ParaphraseRC): We then paired the questions from the SelfRC dataset with the corresponding longer version of the plot and showed it to a different set of AMT workers asking them to answer these questions from the longer version of the plot. They now have the option of either (a) selecting an answer which matches a span in the longer version, or (b) synthesizing the answer from scratch, or (c) marking the question not-answerable because of lack of information in the given passage. We found that in 50% of the cases the workers selected an answer which matched a span in the document, whereas in 37% cases they synthesized the answer and in 13% cases they said that question was not answerable. The workers were strictly instructed to derive the answer from the plot and not rely on their personal knowledge about the movie (in any case given the large number of movies in our dataset the chance of a worker remembering all the plot details for a given movie is very less). Further, a wait period of 2-3 weeks was deliberately introduced between the two phases of data collection to ensure the availability of a fresh pool of workers as well as to reduce information bias among any worker common to both the tasks. We refer to this dataset, where the questions are taken from one version of the document and the answers are obtained from a different version, as ParaphraseRC dataset. We collected 100,316 such {question, answer, document} triplets. Note that the number of unique questions in the ParaphraseRC dataset is the same as that in SelfRC because we do not create any new questions from the longer version of the plot. We end up with a greater number of {question, answer, document} triplets in ParaphraseRC as compared to SelfRC (100,316 v/s 85,773) since movies that are remakes of a previous movie had very little difference in their Wikipedia plots. Therefore, we did not separately collect questions from the Wikipedia plot of the remake. However, the IMDb plots of the two movies are very different and so we have two different longer versions of the movie (one for the original and one for the remake). We can thus pair the questions created from the Wikipedia plot with both the IMDb versions of the plot and hence we end up with more {question, answer, document} triplets. We refer to this combined dataset containing a total of 186,089 instances as DuoRC. Fig. 2 shows the distribution of different Wh-type questions in our dataset. Some more interesting statistics about the dataset are presented in TAB1 and also in Appendix B.Another notable observation is that in many cases the answers to the same question are different in the two versions. Specifically, only 40.7% of the questions have the same answer in the two documents. For around 37.8% of the questions there is no overlap between the words in the two answers. For the remaining 21% of the questions there is a partial overlap between the two answers. For e.g., the answer derived from the shorter version could be "using his wife's gun" and from the longer version could be "with Dana's handgun" where Dana is the name of the wife. In Appendix A, we provide a few randomly picked examples from our dataset which should convince the reader of the difficulty of ParaphraseRC and its differences with SelfRC. In this section, we describe in detail the various state-of-the-art RC and language generation models along with a collection of traditional NLP techniques employed together that will serve to establish baseline performance on the DuoRC dataset. Most of the current state-of-the-art models for RC assume that the answer corresponds to a span in the document and the task of the model is to predict this span. This is indeed true for the SQuAD, TriviaQA and NewsQA datasets. However, in our dataset, in many cases the answers do not correspond to an exact span in the document but are synthesized by humans. Specifically, for the SelfRC version of the dataset around 30% of the answers are synthesized and do not match a span in the document whereas for the ParaphraseRC task this number is 50%. Nevertheless, we could still leverage the advances made on the SQuAD dataset and adapt these span prediction models for our task. To do so, we propose to use two models. The first model is a basic span prediction model which we train and evaluate using only those instances in our dataset where the answer matches a span in the document. The purpose of this model is to establish whether even for instances where the answer matches a span in the document, our dataset is harder than the SQuAD dataset or not. Specifically, we want to explore the performance of state-of-the-art models (such as DCN BID20), which exhibit near human on the SQuAD dataset, on DuoRC (especially, in the ParaphraseRC setup). To do so, we seek to employ a good span prediction model for which (i) the performance is within 3-5% of the top performing model on the SQuAD leaderboard BID12 and (ii) the are reproducible based on the code released by the authors of the paper. Note that the second criteria is important to ensure that the poor performance of the model is not due to incorrect implementation. The Bidirectional Attention Flow (BiDAF) model BID14 satisfies these criteria and hence we employ this model. Due to space constraints, we do not provide details of the BiDAF model here and simply refer the reader to the original paper. In the remainder of this paper we will refer to this model as the SpanModel. The second model that we employ is a two stage process which first predicts the span and then synthesizes the answers from the span. Here again, for the first step (i.e., span prediction) we use the BiDAF model BID14. The job of the second model is to then take the span (mini-document) and question (query) as input and generate the answer. For this, we employ a state-of-the-art query based abstractive summarization model BID7 as this task is very similar to our task. Specifically, in query based abstractive summarization the training data is of the form {query, document, generated summary} and in our case the training data is of the form {query, mini-document, generated answer}. Once again we refer the reader to the original paper BID7 for details of the model. We refer to this two stage model as the GenModel. Note that BID15 recently proposed an answer generation model for the MS MARCO dataset. However, the authors have not released their code and therefore, in the interest of reproducibility of our work, we omit incorporating this model in this paper. Additional NLP pre-processing: Referring back to the example cited in FIG0, we reiterate that ideally a good model for ParaphraseRC would require: (i) employing a knowledge graph, (ii) common-sense knowledge (iii) paraphrase/semantic understanding (iv) multiple-sentence inferencing across events in the passage including coreference resolution of named entities and nouns, and (v) educated guesswork when the question is not directly answerable but there are subtle hints in the passage. While addressing all of these challenges in their entirety is beyond the scope of a single paper, in the interest of establishing a good baseline for DuoRC, we additionally seek to address some of these challenges to a certain extent by using standard NLP techniques. Specifically, we look at the problems of paraphrase understanding, coreference resolution and handling long passages. To do so, we prune the document and extract only those sentences which are most relevant to the question, so that the span detector does not need to look at the entire 900-word long ParaphraseRC plot. Now, since these relevant sentences are obtained not from the original but the paraphrased version of the document, they may have a very small word overlap with the question. For example, the question might contain the word "hand gun" and the relevant sentence in the document may contain the word "revolver". Further some of the named entities in the question may not be exactly present in the relevant sentence but may simply be co-referenced. To resolve these coreferences, we first employ the Stanford coreference resolution on the entire document. We then compute the fraction of words in a sentence which match a query word (ignoring stop words). Two words are considered to match if (a) they have the same surface form, or (b) one words is an inflected form of the word (e.g., river and rivers), or (c) the Glove and Skip-thought embeddings of the two words are very close to each other, or (d) the two words appear in the same synset in Wordnet. We consider a sentence to be relevant for the question if at least 50% of the query words (ignoring stop words) match the words in the sentence. If none of the sentences in the document have atleast 50% overlap with the question, then we pick sentences having atleast a 30% overlap with the question. In the following sub-sections we describe (i) the evaluation metrics, and (ii) the choices considered for augmenting the training data for the answer generation model. Note that when creating the train, validation and test set, we ensure that the test set does not contain question-answer pairs for any movie that was seen during training. We split the movies in such a way that the ing train, valid, test sets respectively contain 70%, 15% and 15% of the total number of QA pairs. As mentioned earlier, the SpanModel only predicts the span in the document whereas the GenModel generates the answer after predicting the span. Ideally, the SpanModel should only be evaluated on those instances in the test set where the answer matches a span in the document. We refer to this subset of the test set as the Span-based Test Set. Though not ideal, we also evaluate the SpanModel model on the entire test set. We say this is not ideal because we know for sure that there are many answers in the test set which do not correspond to a span in the document whereas the model was only trained to predict spans. We refer to this as the Full Test Set. We also evaluate the GenModel on both the test sets. Training Data for the GenModel As mentioned earlier, the GenModel contains two stages; the first stage predicts the span and the second stage then generates an answer from the predicted span. For the first step we plug-in the best performing SpanModel from our earlier exploration. To train the second stage we need training data of the form {x = span, y= answer} which comes from two types of instances: one where the answer matches a span and the other where the answer is synthesized and the span corresponding to it is not known. In the first case x=y and there is nothing interesting for the model to learn (except for copying the input to the output). In the second case x is not known. To overcome this problem, for the second type of instances, we consider various approaches for finding the approximate span from which the answer could have been generated, in order to augment the training data with {x = approx span, y= answer} pairs. The easiest method was to simply treat the entire document as the true span from which the answer was generated (x = document, y = answer). The second alternative that we tried was to first extract the named entities, noun phrases and verb phrases from the question and create a lucene query from these components. We then used the lucene search engine to extract the most relevant portions of the document given this query. We then considered this portion of the document as the true span (as opposed to treating the entire document as the true span). Note that lucene could return multiple relevant spans in which case we treat all these {x = approx span, y= answer} as training instances. Another alternative was to find the longest common subsequence (LCS) between the document and the question and treat this subsequence as the span from which the answer was generated. Of these, we found that the model trained using {x = approx span, y= answer} pairs created using the LCS based method gave the best . We report numbers only for this model. Evaluation Metrics Similar to BID11 we use Accuracy and F-score as the evaluation metric. While accuracy, being a stricter metric, considers a predicted answer to be correct only if it exactly matches the true answer, F-score also gives credit to predictions partially overlapping with the true answer. The of our experiments are summarized in TAB3 which we discuss in the following sub-sections. • SpanModel v/s GenModel: Comparing the first two rows (SelfRC) and the last two rows (ParaphraseRC) of TAB4 we see that the SpanModel clearly outperforms the GenModel. This is not very surprising for two reasons. First, around 70% (and 50%) of the answers in SelfRC (and ParaphraseRC) respectively, match an exact span in the document so the span based model still has scope to do well on these answers. On the other hand, even if the first stage of the GenModel predicts the span correctly, the second stage could make an error in generating the correct answer from it because generation is a harder problem. For the second stage, it is expected that the GenModel should learn to copy the predicted span to produce the answer output (as is required in most cases) and only occasionally where necessary, generate an answer. However, surprisingly the GenModel fails to even do this. Manual inspection of the generated answers shows that in many cases the generator ends up generating either more or fewer words compared the true answer. This demonstrates that there is clearly scope for the GenModel to perform better.• SelfRC v/s ParaphraseRC: Comparing the SelfRC and ParaphraseRC numbers in TAB4, we observe that the performance of the models clearly drops for the latter task, thus validating our hypothesis that ParaphraseRC is a indeed a much harder task.• Effect of NLP pre-processing: As mentioned in Section 4, for ParaphraseRC, we first perform a few pre-processing steps to identify relevant sentences in the longer document. In order to evaluate whether the pre-processing method is effective, we compute: (i) the percentage of the document that gets pruned, and (ii) whether the true answer is present in the pruned document (i.e., average recall of the answer). We can compute the recall only for the span-based subset of the data since for the remaining data we do not know the true span. In TAB3, we report these two quantities for the span-based subset using different pruning strategies. Finally, comparing the SpanModel with and without Paraphrasing in TAB4 for ParaphraseRC, we observe that the pre-processing step indeed improves the performance of the Span Detection Model.• Effect of oracle pre-processing: As noted in Section 3, the ParaphraseRC plot is almost double in length in comparison to the SelfRC plot, which while adding to the complexities of the former task, is clearly not the primary reason of the model's poor performance on that. To empirically validate this, we perform an Oracle pre-processing step, where, starting with the knowledge of the span containing the true answer, we extract a subplot around it such that the span is randomly located within that subplot and the average length of the subplot is similar to the SelfRC plots. The SpanModel with this Oracle preprocessed data exhibits a minor improvement in performance over that with rule-based preprocessing (1.6% in Accuracy and 4.3% in F1 over the Span Test), still failing to bridge the wide performance gap between the SelfRC and ParaphraseRC task.• Cross Testing We wanted to examine whether a model trained on SelfRC performs well on ParaphraseRC and vice-versa. We also wanted to evaluate if merging the two datasets improves the performance of the model. For this we experimented with various combinations of train and test data. The of these experiments for the SpanModel are summarized in TAB5. We make two main observations. First, training on one dataset and evaluating on the other in a drop in the performance. Merging the training data from the two datasets exhibits better performance on the individual test sets. Based on our experiments and empirical observations we believe that the DuoRC dataset indeed holds a lot of potential for advancing the horizon of complex language understanding by exposing newer challenges in this area. In this paper we introduced DuoRC, a large scale RC dataset of 186K human-generated questionanswer pairs created from 7680 pairs of parallel movie-plots, each pair taken from Wikipedia and IMDb. We then showed that this dataset, by design, ensures very little or no lexical overlap between the questions created from one version and the segments containing the answer in the other version. With this, we hope to introduce the RC community to new research challenges on question-answering requiring external knowledge and common-sense driven reasoning, deeper language understanding and multiple-sentence inferencing. Through our experiments, we show how the state-of-the-art RC models, which have achieved near human performance on the SQuAD dataset, perform poorly on our dataset, thus emphasizing the need to explore further avenues for research. In this appendix, we showcase some examples of plots from which questions are created and answered. Since the questions are created from the smaller plot, answering these questions by the reading the smaller plot (which is named as the SelfRC task) is straightforward. However, answering them by reading the larger plot (i.e. the ParaphraseRC task) is more challenging and requires multi-sentence and sometimes multi-paragraph inferencing. Due to shortage of space, we truncate the plot contents and only show snippets from which the questions can be answered. In the smaller plot, blue indicates that an answer can directly be found from the sentence and cyan indicates that the answer spans over multiple sentences. For the larger plot, red and orange are used respectively. A.1 EXAMPLE 1: A. We conducted a manual verification of 100 question-answer pairs where the SelfRC and ParaphraseRC were different or the latter was marked as non-answerable. As noted in FIG1, the chief reason behind getting No Answer from the Paraphrase plot is lack of information and at times, need for an educated guesswork or missing general knowledge (e.g. Philadelphia is a city) or missing movie meta-data (e.g. to answer questions like ' Where did Julia Roberts' character work in the movie?'). On the other hand, SelfRC and ParaphraseRC answers are occasionally seen to have partial or no overlap, mainly because of the following causes; phrasal paraphrases or subjective questions (e.g. Why and How type questions) or different valid answers to objective questions (e.g. 'Where did Jane work?' is answered by one worker as 'Bloomberg' and other as 'New York City') or differently spelt names in the answers (e.g. 'Rebeca' as opposed to 'Rebecca').

Represent the TLDR for finding the corresponding scientific article

We propose DuoRC, a novel dataset for Reading Comprehension (RC) containing 186,089 human-generated QA pairs created from a collection of 7680 pairs of parallel movie plots and introduce a RC task of reading one version of the plot and answering questions created from the other version; thus by design, requiring complex reasoning and deeper language understanding to overcome the poor lexical overlap between the plot and the question.

We introduce and analyze the phenomenon of "hallucinations" in NMT, or spurious translations unrelated to source text, and propose methods to reduce its frequency.


Question: Represent the article for finding a single-sentence TLDR

Question: We consider the problem of weakly supervised structured prediction (SP) with reinforcement learning (RL) – for example, given a database table and a question, perform a sequence of computation actions on the table, which generates a response and receives a binary success-failure reward. This line of research has been successful by leveraging RL to directly optimizes the desired metrics of the SP tasks – for example, the accuracy in question answering or BLEU score in machine translation. However, different from the common RL settings, the environment dynamics is deterministic in SP, which hasn’t been fully utilized by the model-freeRL methods that are usually applied. Since SP models usually have full access to the environment dynamics, we propose to apply model-based RL methods, which rely on planning as a primary model component. We demonstrate the effectiveness of planning-based SP with a Neural Program Planner (NPP), which, given a set of candidate programs from a pretrained search policy, decides which program is the most promising considering all the information generated from executing these programs. We evaluate NPP on weakly supervised program synthesis from natural language(semantic parsing) by stacked learning a planning module based on pretrained search policies. On the WIKITABLEQUESTIONS benchmark, NPP achieves a new state-of-the-art of 47.2% accuracy. Numerous from natural language processing tasks have shown that Structured Prediction (SP) can be cast into a reinforcement learning (RL) framework, and known RL techniques can give formal performance bounds on SP tasks BID3 BID13 BID0. RL also directly optimizes task metrics, such as, the accuracy in question answering or BLEU score in machine translation, and avoids the exposure bias problem when compaired to maximum likelihood training that is commonly used in SP BID13 BID12.However, previous works on applying RL to SP problems often use model-free RL algorithms (e.g., REINFORCE or actor-critic) and fail to leverage the characteristics of SP, which are different than typical RL tasks, e.g., playing video games BID9 or the game of Go BID15. In most SP problems conditioned on the input x, the environment dynamics, except for the reward signal, is known, deterministic, reversible, and therefore can be searched. This means that there is a perfect model 1 of the environment, which can be used to apply model-based RL methods that utilize planning 2 as a primary model component. Take semantic parsing BID1 BID11 as an example, semantic parsers trained by RL such as Neural Semantic Machine (NSM) BID8 typically rely on beam search for inference -the program with the highest probability in beam is used for execution and generating answer. However, the policy, which is used for beam search, may not be 1 A model of the environment usually means anything that an agent can use to predict how the environment will respond to its actions BID17.2 planning usually refers to any computational process that takes a model as input and produces or improves a policy for interacting with the modeled environment BID17 able to assign the highest probability to the correct program. This limitation is due to the policy predicting locally normalized probabilities for each possible action based on the partially generated program, and the probability of a program is a product of these local probabilities. For example, when applied to the WEBQUESTIONSSP task, NSM made mistakes with two common patterns: the program would ignore important information in the context; the generated program does not execute to a reasonable output, but still receives high probability (spurious programs). Resolving this issue requires using the information of the full program and its execution output to further evaluate its quality based on the context, which can be seen as planning. This can be observed in Figure 4 where the model is asked a question "Which programming is played the most?". The full context of the input table (shown in TAB0) contains programming for a television station. The top program generated by a search policy produces the wrong answer, filtering by a column not relevant to the question. If provided the correct contextual features, and if allowed to evaluate the full program forward and backward through time, we observe that a planning model would be able to better evaluate which program would produce the correct answer. To handle errors related to context, we propose to train a value function to compute the utility of each token in a program. This utility is evaluated by considering the program and token probability as well as the attention mask generated by the sequence-to-sequence (seq2seq) model for the underlying policy. We also introduce beam and question context with a binary feature representing overlap from question/program and program/program, such as how many programs share a token at a given timestep. In the experiments, we found that applying a planner that uses a learned value function to re-rank the candidates in the beam can significantly and consistently improve the accuracy. On the WIKITABLEQUESTIONS benchmark, we improve the state-of-the-art by 0.9%, achieving an accuracy of 47.2%. 2.1 WIKITABLEQUESTIONS WIKITABLEQUESTIONS BID11 contains tables extracted from Wikipedia and question-answer pairs about the tables. See TAB0 as an example. There are 2,108 tables and 18,496 question-answer pairs split into train/dev/test set. Each table can be converted into a directed graph that can be queried, where rows and cells are converted to graph nodes while column names become labeled directed edges. For the questions, we use string match to identify phrases that appear in the table. We also identify numbers and dates using the CoreNLP annotation released with the dataset. The task is challenging in several aspects. First, the tables are taken from Wikipedia and cover a wide range of topics. Second, at test time, new tables that contain unseen column names appear. Third, the table contents are not normalized as in knowledge-bases like Freebase, so there are noises and ambiguities in the table annotation. Last, the semantics are more complex comparing to previous datasets like WEBQUESTIONSSP BID19. It requires multiple-step reasoning using a large set of functions, including comparisons, superlatives, aggregations, and arithmetic operations BID11. See BID8 for more details about the functions. We adopt the NSM framework open sourced in the Memory Augmented Policy Optimization paper (MAPO) BID8, which combines a neural "programmer", which is a seq2seq model augmented by a key-variable memory that can translate a natural language utterance to a program as a sequence of tokens, and a symbolic "computer", which is an Lisp interpreter that implements a domain specific language with built-in functions and provides code assistance by eliminating syntactically or semantically invalid choices. For the Lisp interpreter, it added functions according to BID20 BID10 for WIKITABLEQUESTIONS, refer to BID8 for further detail of the open-sourced implementation. Same as BID8 we consider the problem of contextual program synthesis with weak supervision -i.e., no correct action sequence a is given as part of the training data, and training needs to solve the hard problem of exploring a large program space. However, we will focus on improving decision making (planning) giving a pretrained search policy, while previous work mainly focus on learning the search policies. The problem of weakly supervised contextual program synthesis can be formulated as: generating a program a by using a parametric mapping function,â " f θ pxq, where θ denotes the model parameters. The quality of a generated programâ is measured in terms of a scoring or reward function Rpâ; x, yq with y being the expected correct answer. The reward function evaluates a program by executing it on a real environment and comparing the emitted output against the correct answer. For example, the reward may be binary that is 1 when the output equals the answer and 0 otherwise. We assume that the context x includes both a natural language input and an environment, for example an interpreter or a database, on which the program will be executed. Given a dataset of context-answer pairs, tpx i, y i qu N i"1, the goal is to find an optimal parameter θ˚that parameterizes a mapping of x Ñ a with maximum empirical return on a heldout test set. In this study we will further decompose the policy f θ into the stacking of a search model s φ pxq, which produces a set of candidate programs B given an environment x, and a value model v ω pa; x, Bq, which assigns a score s to program a given the environment and all the candidate programs. Therefore, θ " rφ; ωs and f θ pxq « argmax DISPLAYFORM0 The search model s φ can be learnt by optimizing a conditional distribution π φ pa | xq that assigns a probability to each program given the context. That is, π φ is a distribution over the countable set of all possible programs, denoted A. Thus @a P A: π φ pa | xq ě 0 and ř aPA π φ pa | xq " 1. Then, to synthesize candidate programs for a novel context, one may find the most likely programs under the distribution π φ via exact or approximate inference such as beam search. B " s φ pxq « argmax B aPA π φ pa | xq. π φ is typically an autoregressive model such as a recurrent neural network: (e.g. BID5) π φ pa | xq " ś |a| i"t π φ pa t | a ăt, xq,where a ăt " pa 1,..., a t´1 q denotes a prefix of the program a. In the absence of ground truth programs, policy gradient techniques (such as REINFORCE BID18) present a way to optimize the parameters of a stochastic policy π φ via optimization of expected return. Given a training dataset of context-answer pairs, tpx l, y l qu N l"1, the training objective is O ER pθq " E a"π φ pa|xq Rpa; x, yq. Decision-time planning typically relies on value network BID14 trained to predict the true reward. In the next section, however, we introduce a max-margin training objective for the value model v ω, which optimizes to rank rewarded programs higher than non-rewarded programs. We now introduce NPP by first describing the architecture of v ω -a seq2seq model which goes over candidate program answer pairs and the final score of a candidate program is simply the sum of its token scores (Section 4.1). Secondly we describe the program token representation, which considers the program, question and beam context which are used to denote the utility of all tokens within a program (Section 4.2). Finally, we describe a training procedure that is based on max-margin/ranking objective on candidate programs given a question (Section 4.3). Here we introduce the NPP architecture FIG0 ) in the context of semantic parsing, but the framework should be broadly applicable to applying RL in structured predictions. Given a pre-trained search policy π φ and environment x, NPP first unrolls the policy with beam search to generate candidate programs (plans) B " s φ pxq. Then it scores each program a considering token a t at every step and global statistics among all programs B. a t is represented as a context feature vector C t (details in Section 4.2). To capture the sequential inputs the scoring component is a seq2seq model which goes over candidate program answer pairs and assigns preference scores to each program/answer token. We implement a bi-directional recurrent network with LSTM Hochreiter & Schmidhuber (1997b); BID4 cells as the first layer of our planner C LSTM " LSTMpCq. The LSTM hidden state at each step is fed to a one dimensional convolutional layer with kernel size 3, in order to capture inner function scoring as most functions are of size 3-5, as a feature extractor DISPLAYFORM0 Finally we calculate the score per token by feeding into a single node hyperbolic tangent activation layer to compute the score per token v t " TanhpO CNN t q of token a t. The final score of a candidate Program token probability according to the search policy π φ t agree countNumber of candidate programs having token a t at position t program v ω paq " ř t"1.. T v t is simply the sum of its token scores. The choice of simply summing token level scores makes the score very understandable (details in Section 4.2). FIG1 gives implementation details of the value model. To better score tokens based on the overall context of the environment we represent each token with a set of context features C t " rq tok; q attn; p prob; t prob; t agree s as in TAB1. q attn is the softmax attention across question tokens, which helps to discern which part of the question is of most importance to the model when generating the current given token. Together q tok and q attn represents the alignment between program and query. t prob and p prob are the probability of token a t and program a assigned by the search policy π φ, which represent the decisions from the search model. t agree is the number of candidate programs having token a t at position t. Access to information such as t agree is only available to NPP as it can only be used after all the candidate programs have been generated. We formulate NPP training as a learning to rank problem -optimizing pairwise ranking among candidate programs B. Given a training dataset of context-answer pairs, tpx l, y l qu N l"1, the training objective is DISPLAYFORM0 where σpvq " 1{p1`e´vq is the sigmoid function, and v l,i " v ω pa l,i ; x l, s φ px l qq is the estimated value of a l,i, the i-th program generated for context x l .We compare NPP training in two setups: a single MAPO setup, and a stack learning setup. For the single MAPO setup the queries used to produce a pretrained MAPO model are also used to train the NPP model. The dev and test queries are used for ablation study and final evaluation. Since the NPP training queries are already used to train the MAPO model, the candidate programs are biased towards better reward (compared to those candidate generated for unseen queries). This setup causes NPP to learn from a different distribution as the intended dev/test data. Surprisingly NPP is still able to improve the prediction of MAPO as will be discussed in Section 5.3.To overcome the issue with single MAPO setup we also generate NPP training data with a stacked learning setup. First the train and dev queries are merged and splitted into K equal portions, and with Leave-One-Out (LOO) scheme they form K train/dev sets. Then K MAPO models are trained on K train sets. Finally we use each of the K MAPO s to produce a stack learning dataset by running these models on their respective dev set. The stack learning dataset is further splited for train/dev purposes for NPP. In this way, each NPP training query is decoded by a MAPO model, which has never seen this query before, and therefore avoid the bias in training data generation. Our empirical study is based on the semantic parsing benchmark, WIKITABLEQUESTIONS BID11. To focus on studying the planning part of the problem we assume that the search policy is pretrained using MAPO BID8, and NPP is trained to rescore given candidate programs produced by MAPO. Additionally we show that stacked learning BID2 is helpful in correctly training a planner given pre-trained policy models. Datasets. WIKITABLEQUESTIONS BID11 contains tables extracted from Wikipedia and question-answer pairs about the tables. See TAB0 as an example. There are 2,108 tables and 18,496 question-answer pairs. We follow the construction in BID11 for converting a table into a directed graph that can be queried, where rows and cells are converted to graph nodes while column names become labeled directed edges. For the questions, we use string match to identify phrases that appear in the table. We also identify numbers and dates using the CoreNLP annotation released with the dataset. Baselines. We compare NPP to BID8, the current state of the art on the WIKITABLE-QUESTIONS dataset. MAPO relies on beam search to find candidate programs, and uses the program probability according to the policy to determine the program to execute. MAPO manages to achieve 46.3% accuracy on this task when using an ensemble of size 10. We aim to show that NPP can improve on single MAPO as well as the ensemble of MAPO models. Training Details. We set the stacked learning parameter K " 5 for all our experiments. We set the batch size to be equal to 16. We use Adam optimizer for training with a learning rate 10´3. We choose a size of 64 nodes for the LSTM (which becomes 128 as it is bidirectional). The CNN consists of 32 filters with kernel size 3. All the hyper parameters are tuned on the development set and trained for 10 epochs. Ensemble Details. We formulate the ensemble of K MAPO models with NPP as the sum of normalized NPP scores under different MAPO models: Let Φ " tφ k u K k"1 be K MAPO models to be ensembled. We define the ensembled score of a program a under context x as v ω,Φ pa; xq " DISPLAYFORM0 wherev ω pxq is the average score for programs in beam s k φ pxq DISPLAYFORM1 and v 1 ω backs-off v ω tov ω pxq whenever a is not in beam DISPLAYFORM2 Table 3: Feature ablation study on the dev set with a mean of 5 runs on a single MAPO setup. DISPLAYFORM3 In order to evaluate the effectiveness of our proposed programs token representations, we present a feature ablation test in Figure 3. We can see that the program probability p prob produced by the search policy is the most important feature, providing the foundation to NPP scoring. The program agreement feature t agree is also very useful. We often observe cases for which beam B produces program with similar tokens that are not highly valued by the underlying model. By utilizing this feature, we more strongly consider programs which were repeatedly searched by s φ. t agree also helps to identify the programs that are very similar throughout most of the sequence to learn their divergence and grade their utility. Question referencing features such as q tok provide significant importance in providing the program with query level context, ensuring we are filtering or selecting values based on the query context. While the help from attention feature q attn is not significant. We evaluate NPP's impact on MAPO under three different settings, in each of which NPP consistently improves the precision of MAPO.First we consider a single MAPO trained on a single train-dev data split. Similar to other RL models in the literature, MAPO training is a procedure with big variances. Even trained on the same data split multiple times with different random seed gives big variances in accuracy of 0.3% (dev) and 0.5% (test). We use the MAPO model to decode on its own train, dev and test data, in order to generate parallel splits for NPP training. Training and applying NPP this way is able to improve precision, despite of the exposure bias in the NPP training data. However, it does not improve on the variances. We next consider MAPO models that were trained and evaluated on separate train/dev splits created with a Leave-One-Out (LOO) scheme. As described in Section 4.3 we also use these splits to generate a stacked learning dataset for NPP to avoid the data bias problem. We can see that with different data splits MAPO has significantly higher variances on the dev set (1.1%), which is a drawback of RL models in general. Stacked learning helps NPP to improve precision more significantly (1.3% for dev and 1.1% for test). It also helps to reduce the variances to 0.2% on both dev and test. Finally, we consider ensembled MAPO settings, which produces the previous state of the art . We use the same NPP model trained from the stacked learning setting, and apply it to an ensemble of either 5, or 10 MAPO models from BID8. We can see that when applied to 5 MAPO ensemble, NPP can still improve the precision by 1.1%. When applied to 10 MAPO ensemble, NPP can improves the precision by 0.9%. Since the score of a program is the sum of its token scores, it is easy to visualize how NPP plan and select the correct program to execute. We observed that there are two common situations in which MAPO chooses the wrong program from the beam -selecting a spurious program over the semantically correct program and executing the incorrect table filtering or column selection. NPP aims to overcome these non optimal decisions by taking advantage of the larger time horizon and other programs discovered so far. For example NPP may reward earlier program tokens based on program tokens chosen much later on due to the bi-directional recurrent network. We first investigate how NPP demotes spurious programs. MAPO may produce programs which return the correct answer but are not semantically correct. NPP helps solve this by scoring semantically correct programs higher in the beam. An example is shown in Figure 3 for the question "What venue was the latest match played at?" when referring to a soccer (football) player given a table of his matches. The top program in beam proposed by MAPO was to first filter all rows for the competition string, which is incorrect considering the context of the table is only competitions. NPP is able to reevaluate the program given the full context. Although the first function (filter_in) is typically used to filter the table for the correct row/column. NPP learns that in this situation it is better to find the last of all rows using the function last. NPP, re-evaluates the first function of the new best program as being high in utility, and scores all tokens within this function higher than the tokens in the incorrect program. We then investigate programs from MAPO which produce wrong answers. An example is shown in Figure 4 which is based on TAB0. MAPO assigns a higher probability to a program in beam that filters on an incorrect column. Because NPP knows program token overlap with the query as well as the attention matrix, it is able to better understand the question and grade the program which is more closely related to the question. In addition to this we notice that the convolutional layer grades full functions within their context, given a kernel of size 3 the parenthesis at the beginning of program already receives a higher NPP score which we interpret as the overall score of executing the function.. Reinforcement learning applied to structured prediction suffers from limited use of the world model as well as not being able to consider future and past program context when generating a sequence. To overcome these limitations we proposed Neural Program Planner (NPP) which is a planning step after candidate program generation. We show that an additional planning model can better evaluate overall structure value. When applied to a difficult SP task NPP improves state of the art by 0.9% and allows intuitive analysis of its scoring model per program token. A MORE NPP SCORING DETAILS

Represent the short summary for finding the corresponding scientific article (long).

A model-based planning component improves RL-based semantic parsing on WikiTableQuestions.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.