Question: Represent this text

Question: The field of Deep Reinforcement Learning (DRL) has recently seen a surge in the popularity of maximum entropy reinforcement learning algorithms. Their popularity stems from the intuitive interpretation of the maximum entropy objective and their superior sample efficiency on standard benchmarks. In this paper, we seek to understand the primary contribution of the entropy term to the performance of maximum entropy algorithms. For the Mujoco benchmark, we demonstrate that the entropy term in Soft Actor Critic (SAC) principally addresses the bounded nature of the action spaces. With this insight, we propose a simple normalization scheme which allows a streamlined algorithm without entropy maximization match the performance of SAC. Our experimental demonstrate a need to revisit the benefits of entropy regularization in DRL. We also propose a simple non-uniform sampling method for selecting transitions from the replay buffer during training. We further show that the streamlined algorithm with the simple non-uniform sampling scheme outperforms SAC and achieves state-of-the-art performance on challenging continuous control tasks. Off-policy Deep Reinforcement Learning (RL) algorithms aim to improve sample efficiency by reusing past experience. Recently a number of new off-policy Deep Reinforcement Learning algorithms have been proposed for control tasks with continuous state and action spaces, including Deep Deterministic Policy Gradient (DDPG) and Twin Delayed DDPG (TD3) . TD3, which introduced clipped double-Q learning, delayed policy updates and target policy smoothing, has been shown to be significantly more sample efficient than popular on-policy methods for a wide range of Mujoco benchmarks. The field of Deep Reinforcement Learning (DRL) has also recently seen a surge in the popularity of maximum entropy RL algorithms. Their popularity stems from the intuitive interpretation of the maximum entropy objective and their superior sample efficiency on standard benchmarks. In particular, Soft Actor Critic (SAC), which combines off-policy learning with maximum-entropy RL, not only has many attractive theoretical properties, but can also give superior performance on a wide-range of Mujoco environments, including on the high-dimensional environment Humanoid for which both DDPG and TD3 perform poorly (a; b;). SAC has a similar structure to TD3, but also employs maximum entropy reinforcement learning. In this paper, we first seek to understand the primary contribution of the entropy term to the performance of maximum entropy algorithms. For the Mujoco benchmark, we demonstrate that when using the standard objective without entropy along with standard additive noise exploration, there is often insufficient exploration due to the bounded nature of the action spaces. Specifically, the outputs of the policy network are often way outside the bounds of the action space, so that they need to be squashed to fit within the action space. The squashing in actions persistently taking on their maximal values, so that there is insufficient exploration. In contrast, the entropy term in the SAC objective forces the outputs to have sensible values, so that even with squashing, exploration is maintained. We conclude that the entropy term in the objective for Soft Actor Critic principally addresses the bounded nature of the action spaces in the Mujoco environments. With this insight, we propose Streamlined Off Policy (SOP), a streamlined algorithm using the standard objective without the entropy term. SOP employs a simple normalization scheme to address the bounded nature of the action spaces, allowing satisfactory exploration throughout training. We also consider replacing the aforementioned normalization scheme with inverting gradients (IG) The contributions of this paper are thus threefold. First, we uncover the primary contribution of the entropy term of maximum entropy RL algorithms when the environments have bounded action spaces. Second, we propose a streamlined algorithm which do not employ entropy maximization but nevertheless matches the sampling efficiency and robustness performance of SAC for the Mujoco benchmarks. And third, we combine our streamlined algorithms with a simple non-uniform sampling scheme to achieve state-of-the art performance for the Mujoco benchmarks. We provide anonymized code for reproducibility 1. We represent an environment as a Markov Decision Process (MDP) which is defined by the tuple (S, A, r, p, γ), where S and A are continuous multi-dimensional state and action spaces, r(s, a) is a bounded reward function, p(s |s, a) is a transition function, and γ is the discount factor. Let s(t) and a(t) respectively denote the state of the environment and the action chosen at time t. Let π = π(a|s), s ∈ S, a ∈ A denote the policy. We further denote K for the dimension of the action space, and write a k for the kth component of an action a ∈ A, that is, a = (a 1, . . ., a K). The expected discounted return for policy π beginning in state s is given by: γ t r(s(t), a(t))|s = s] Standard MDP and RL problem formulations seek to maximize V π (s) over policies π. For finite state and action spaces, under suitable conditions for continuous state and action spaces, there exists an optimal policy that is deterministic . In RL with unknown environment, exploration is required to learn a suitable policy. In DRL with continuous action spaces, typically the policy is modeled by a parameterized policy network which takes as input a state s and outputs a value µ(s; θ), where θ represents the current parameters of the policy network (; ; ;). During training, typically additive random noise is added for exploration, so that the actual action taken when in state s takes the form a = µ(s; θ) + where is a K-dimensional Gaussian random vector with each component having zero mean and variance σ. During testing, is set to zero. Maximum entropy reinforcement learning takes a different approach than by optimizing policies to maximize both the expected return and the expected entropy of the policy (; ; ; ; ; ; ; ; 2018a; b). In particular, with maximization entropy RL, the objective is to maximize where H(π(·|s)) is the entropy of the policy when in state s, and the temperature parameter λ determines the relative importance of the entropy term against the reward. For entropy maximization DRL, when given state s the policy network will typically output a Kdimensional vector σ(s; θ) in addition to the vector µ(s; θ). The action selected when in state s is then modeled as µ(s; θ) + where ∼ N (0, σ(s; θ)). Maximum entropy RL has been touted to have a number of conceptual and practical advantages for DRL (a; b). For example, it has been argued that the policy is incentivized to explore more widely, while giving up on clearly unpromising avenues. It has also been argued that the policy can capture multiple modes of near-optimal behavior, that is, in problem settings where multiple actions seem equally attractive, the policy will commit equal probability mass to those actions. In this paper, we show for the Mujoco benchmarks that the standard additive noise exploration suffices and can achieve the same performance as maximum entropy RL. 3 THE SQUASHING EXPLORATION PROBLEM. When selecting an action, the action needs to be selected within these bounds before the action can be taken. DRL algorithms often handle this by squashing the action so that it fits within the bounds. For example, if along any one dimension the value µ(s; θ) + exceeds a max, the action is set (clipped) to a max. Alternatively, a smooth form of squashing can be employed. For example, suppose a min k = −M and a max k = +M for some positive number M, then a smooth form of squashing could use a = M tanh(µ(s; θ) + ) in which tanh is being applied to each component of the K-dimensional vector. DDPG and TD3 use clipping, and SAC (a; b) uses smooth squashing with the tanh function. For concreteness, henceforth we will assume that smooth squashing with the tanh is employed. We note that an environment may actually allow the agent to input actions that are outside the bounds. In this case, the environment will typically first clip the actions internally before passing them on to the "actual" environment . We now make a simple but crucial observation: squashing actions to fit into a bounded action space can have a disastrous effect on additive-noise exploration strategies. To see this, let the output of the policy network be µ(s) = (µ 1 (s),..., µ K (s)). Consider an action taken along one dimension k, and suppose µ k (s) >> 1 and | k | is relatively small compared to µ k (s). Then the action a k = M tanh(µ k (s)+ k ) will be very close (essentially equal) to M. If the condition µ k (s) >> 1 persists over many consecutive states, then a k will remain close to 1 for all these states, and consequently there will be essentially no exploration along the kth dimension. We will refer to this problem as the squashing exploration problem. A similar observation was made in. We will argue that algorithms such as DDPG and TD3 based on the standard objective with additive noise exploration can be greatly impaired by squashing exploration. SAC is a maximum-entropy based off-policy DRL algorithm which provides good performance across all of the Mujuco benchmark environments. To the best of our knowledge, it currently provides state of the art performance for the Mujoco benchmark. In this section, we argue that the principal contribution of the entropy term in the SAC objective is to resolve the squashing exploration problem, thereby maintaining sufficient exploration when facing bounded action spaces. To argue this, we consider two DRL algorithms: SAC with adaptive temperature (b), and SAC with entropy removed altogether (temperature set to zero) but everything else the same. We refer to them as SAC and as SAC without entropy. For SAC without entropy, for exploration we use additive zero-mean Gaussian noise with σ fixed at 0.3. Both algorithms use tanh squashing. We compare these two algorithms on two Mujoco environments: Humanoid-v2 and Walker-v2. Figure 1 shows the performance of the two algorithms with 10 seeds. For Humanoid, SAC performs much better than SAC without entropy. However, for Walker, SAC without entropy performs nearly as well as SAC, implying maximum entropy RL is not as critical for this environment. To understand why entropy maximization is important for one environment but less so for another, we examine the actions selected when training these two algorithms. Humanoid and Walker have action dimensions K = 17 and K = 6, respectively. Here we show representative for one dimension for both environments, and provide the full in the Appendix. The top and bottom rows of Figure 2 shows for Humanoid and Walker, respectively. The first column shows the µ k values for an interval of 1,000 consecutive time steps, namely, for time steps 599,000 to 600,000. The second column shows the actual action values passed to the environment for these time steps. The third and fourth columns show a concatenation of 10 such intervals of 1000 time steps, with each interval coming from a larger interval of 100,000 time steps. The top and bottom rows of Figure 2 are strikingly different. For Humanoid using SAC with entropy, the |µ k | values are small, mostly in the range [-1.5,1.5], and fluctuate significantly. This allows the action values to also fluctuate significantly, providing exploration in the action space. On the other hand, for SAC without entropy the |µ k | values are typically huge, most of which are well outside the interval [-10,10]. This causes the actions a k to be persistently clustered at either M or -M, leading to essentially no exploration along that dimension. As shown in the Appendix, this property (lack of exploration for SAC without entropy maximization) holds for all 17 action dimensions. For Walker, we see that for both algorithms, the µ k values are sensible, mostly in the range [-1,1] and therefore the actions chosen by both algorithms exhibit exploration. In , the principle benefit of maximum entropy RL in SAC for the Mujuco environments is that it resolves the squashing exploration problem. For some environments (such as Walker), the outputs of the policy network take on sensible values, so that sufficient exploration is maintained and overall good performance is achieved without the need for entropy maximization. For other environments (such as Humanoid), entropy maximization is needed to reduce the magnitudes of the outputs so that exploration is maintained and overall good performance is achieved. Given the observations in the previous section, a natural question is: is it possible to design a streamlined off policy algorithm that does not employ entropy maximization but offers performance comparable to SAC (which has entropy maximization)? As we observed in the previous section, without entropy maximization, in some environments the policy network output values |µ k |, k = 1,..., K can become persistently huge, which leads to insufficient exploration due to the squashing. A simple solution is to modify the outputs of the policy network by normalizing the output values when they collectively (across the action dimensions) become too large. To this end, let µ = (µ 1, . . ., µ K) be the output of the original policy network, and let G = k |µ k |/K. The G is simply the average of the magnitudes of the components of µ. The otherwise, we leave µ unchanged. With this simple normalization, we are assured that the average of the normalized magnitudes is never greater than one. Henceforth we assume the policy network has been modified with the simple normalization scheme just described. Our Streamlined Off Policy (SOP) algorithm is described in Algorithm 1. The algorithm is essentially DDPG plus the normalization described above, plus clipped double Q-learning and target policy smoothing . Another way of looking at it is as TD3 plus the normalization described above, minus the delayed policy updates and the target policy parameters. SOP also uses tanh squashing instead of clipping, since tanh gives somewhat better performance in our experiments. The SOP algorithm is "streamlined" as it has no entropy terms, temperature adaptation, target policy parameters or delayed policy updates. In our experiments, we also consider TD3 plus the simple normalization, and also another streamlined algorithm in which we replace the simple normalization scheme described above with the inverting gradients (IG) scheme as described in. The basic idea is: when gradients suggest increasing the action magnitudes, gradients will be downscaled if actions are within the boundaries, and inverted entirely if actions are outside the boundaries. More implementation details can be found in the Appendix. Algorithm 1 Streamlined Off-Policy 1: Input: initial policy parameters θ, Q-function parameters φ 1, φ 2, empty replay buffer D 2: Set target parameters equal to main parameters φ targ i ← φ i for i = 1, 2 3: repeat Generate an episode using actions a = M tanh(µ θ (s) + ) where ∼ N (0, σ 1). for j in range(however many updates) do Randomly sample a batch of transitions, B = {(s, a, r, s)} from D Compute targets for Q functions: Update Q-functions by one step of gradient descent using Update policy by one step of gradient ascent using Update target networks with Figure 3 compares SAC (with temperature adaptation (a; b) ) with SOP, TD3+ (that is, TD3 plus the simple normalization), and inverting gradients (IG) for five of the most chal-lenging Mujuco environments. Using the same baseline code, we train with ten different random seeds for each of the two algorithms. Each algorithm performs five evaluation rollouts every 5000 environment steps. The solid curves correspond to the mean, and the shaded region to the standard deviation of the returns over the ten seeds. Results show that SOP, SAC and IG have similar sample-efficiency performance and robustness across all environments. TD3+ has slightly weaker asymptotic performance for Walker and Humanoid. IG initially learns slowly for Humanoid with high variance across random seeds, but gives similar asymptotic performance. This confirms that with a simple output normalization scheme in the policy network, the performance of SAC can be achieved without maximum entropy RL. In the Appendix we provide an ablation study for SOP, which shows a major performance drop when removing either double Q-learning or normalization, whereas removing target policy smoothing in only a small performance drop in some environments. We now show how a small change in the sampling scheme for SOP can achieve state of the art performance for the Mujoco benchmark. We call this sampling scheme Emphasizing Recent Experience (ERE). ERE has 3 core features: (i) It is a general method applicable to any off-policy algorithm; (ii) It requires no special data structure, is very simple to implement, and has near-zero computational overhead; (iii) It only introduces one additional important hyper-parameter. The basic idea is: during the parameter update phase, the first mini-batch is sampled from the entire buffer, then for each subsequent mini-batch we gradually reduce our range of sampling to sample more aggressively from more recent data. Specifically, assume that in the current update phase we are to make 1000 mini-batch updates. Let N be the max size of the buffer. Then for the k th update, we sample uniformly from the most recent c k data points, where c k = N · η k and η ∈ is a hyper-parameter that determines how much emphasis we put on recent data. η = 1 is uniform sampling. When η < 1, c k decreases as we perform each update. η can made to adapt to the learning speed of the agent so that we do not have to tune it for each environment. The effect of such a sampling formulation is twofold. The first is recent data have a higher chance of being sampled. The second is that we do this in an ordered way: we first sample from all the data in the buffer, and gradually shrink the range of sampling to only sample from the most recent data. This scheme reduces the chance of over-writing parameter changes made by new data with parameter changes made by old data (; ; ; ;). This process allows us to quickly obtain new information from recent data, and better approximate the value functions near recently-visited states, while still maintaining an acceptable approximation near states visited in the more distant past. What is the effect of replacing uniform sampling with ERE? First note if we do uniform sampling on a fixed buffer, the expected number of times a data point is sampled is the same for all data points. Now consider a scenario where we have a buffer of size 1000 (FIFO queue), we collect one data at a time, and then perform one update with mini-batch size of one. If we start with an empty buffer and sample uniformly, as data fills the buffer, each data point gets less and less chance of being sampled. Specifically, over a period of 1000 updates, the expected number of times the tth data is sampled is: 1/t + 1/(t + 1) + · · · + 1/T. Figure 4f shows the expected number of times a data is sampled as a function of its position in the buffer. We see that older data are expected to get sampled much more than newer data. This is undesirable because when the agent is improving and exploring new areas of the state space; new data points may contain more interesting information than the old ones, which have already been updated many times. When we apply the ERE scheme, we effectively skew the curve towards assigning higher expected number of samples for the newer data, allowing the newer data to be frequently sampled soon after being collected, which can accelerate the learning process. In the Appendix, we provide further algorithmic detail and analysis on ERE, and compare ERE to two other sampling schemes: an exponential sampling scheme and Prioritized Experience Replay. Figure 4 compares the performance of SOP, SOP+ERE, SAC and SAC+ERE. With ERE, both SAC and SOP gain a significant performance improvement in all environments. SOP+ERE learns faster than SAC and vanilla SOP in all Mujoco environments. SOP+ERE also greatly improves overall performance for the two most challenging environments, Ant and Humanoid, and has the best performance for Humanoid. In table 1, we show the mean test episode return and std across 10 random seeds at 1M timesteps for all environments. The last column displays the percentage improvement of SOP+ERE over SAC, showing that SOP+ERE achieves state of the art performance. In Ant and Humanoid, SOP+ERE improves performance by 21% and 24% over SAC at 1 million timesteps, respectively. As for the std, SOP+ERE gives lower values, and for Humanoid a higher value. ERE allows new data to be sampled many times soon after being collected. In recent years, there has been significant progress in improving the sample efficiency of DRL for continuous robotic locomotion tasks with off-policy algorithms (; ; a; b). There is also a significant body of research on maximum entropy RL methods (; ; ; ; ; ; ; ; 2018a; b). Uniform sampling is the most common way to sample from a replay buffer. One of the most wellknown alternatives is prioritized experience replay (PER). PER uses the absolute TD-error of a data point as the measure for priority, and data points with higher priority will have a higher chance of being sampled. This method has been tested on DQN and double DQN (DDQN) with significant improvement and applied successfully in other algorithms (; ; ;) and can be implemented in a distributed manner . There are other methods proposed to make better use of the replay buffer. The ACER algorithm has an on-policy part and an off-policy part, with a hyper-parameter controlling the ratio of off-policy to on-policy updates . The RACER algorithm selectively removes data points from the buffer, based on the degree of "off-policyness", bringing improvement to DDPG , NAF and PPO . , replay buffers of different sizes were tested, showing large buffer with data diversity can lead to better performance. Finally, with Hindsight Experience Replay , priority can be given to trajectories with lower density estimation to tackle multi-goal, sparse reward environments. In this paper we first showed that the primary role of maximum entropy RL for the Mujoco benchmark is to maintain satisfactory exploration in the presence of bounded action spaces. We then developed a new streamlined algorithm which does not employ entropy maximization but nevertheless matches the sampling efficiency and robustness performance of SAC for the Mujoco benchmarks. Our experimental demonstrate a need to revisit the benefits of entropy regularization in DRL. Finally, we combined our streamlined algorithm with a simple non-uniform sampling scheme to achieve state-of-the art performance for the Mujoco benchmark. In this ablation study we separately examine the importance of (i) the normalization at the output of the policy network; (ii) the double Q networks; (iii) and randomization used in the line 8 of the SOP algorithm (that is, target policy smoothing ). Figure 5 shows the for the five environments considered in this paper. In Figure 5, "no normalization" is SOP without the normalization of the outputs of the policy network; "single Q" is SOP with one Q-network instead of two; and "no smoothing" is SOP without the randomness in line 8 of the algorithm. Figure 5 confirms that double Q-networks are critical for obtaining good performance (; ; a). Figure 5 also shows that output normalization is also critical. Without output normalization, performance fluctuates wildly, and average performance can decrease dramatically, particularly for Humanoid and HalfCheetah. Target policy smoothing improves performance by a relatively small amount. Table 2 shows hyperparameters used for SOP, SOP+ERE and SOP+PER. For adaptive SAC, we use our own PyTorch implementation for the comparisons. Our implementation uses the same hyperparameters as used in the original paper (b). Our implementation of SOP variants and adaptive SAC share most of the code base. For TD3, our implementation uses the same hyperparamters as used in the authors' implementation, which is different from the ones in the original paper . They claimed that the new set of hyperparamters can improve performance for TD3. We now discuss hyperparameter search for better clarity, fairness and reproducibility (; ;). For the η value in the ERE scheme, in our early experiments we tried the values (0.993, 0.994, 0.995, 0.996, 0.997, 0.998) on the Ant and found 0.995 to work well. This initial range of values was decided by computing the ERE sampling range for the oldest data. We found that for smaller values, the range would simply be too small. For the PER scheme, we did some informal preliminary search, then searched on Ant for β 1 in (0, 0.4, 0.6, 0.8), β 2 in (0, 0.4, 0.5, 0.6, 1), and learning rate in (1e-4, 2e-4, 3e-4, 5e-4, 8e-4, 1e-3), we decided to search these values because the original paper used β 1 = 0.6, β 2 = 0.4 and with reduced learning rate. For the exponential sampling scheme, we searched the λ value in (3e-7, 1e-6, 3e-6, 5e-6, 1e-5, 3e-5, 5e-5, 1e-4) in Ant, this search range was decided by plotting out the probabilities of sampling, and then pick a set of values that are not too extreme. For σ in SOP, in some of our early experiments with SAC, we accidentally found that σ = 0.3 gives good performance for SAC without entropy and with Gaussian noise. We searched values (for HalfCheetah-v2) SOP gaussian noise std σ = σ 1 = σ 2 0.29 TD3 gaussian noise std for data collection σ 0.1 * action limit guassian noise std for target policy smoothingσ 0.2 TD3+ gaussian noise std for data collection σ 0.15 guassian noise std for target policy smoothingσ 0.2 ERE ERE initial η 0 0.995 Algorithm 2 SOP with Emphasizing Recent Experience 1: Input: initial policy parameters θ, Q-function parameters φ 1, φ 2, empty replay buffer D of size N, initial η 0, recent and max performance improvement I recent = I max = 0. 2: Set target parameters equal to main parameters φ targ,i ← φ i for i = 1, 2 3: repeat 4: Generate an episode using actions a = M tanh(µ θ (s) + ) where ∼ N (0, σ 1). update I recent, I max with training episode returns, let K = length of episode 6: for j in range(K) do 8: Sample a batch of transitions, B = {(s, a, r, s)} from most recent c k data in D 10: Compute targets for Q functions: Update Q-functions by one step of gradient descent using Update policy by one step of gradient ascent using Update target networks with In this section we discuss the details of the Inverting Gradient method. discussed three different methods for bounded parameter space learning: Zeroing Gradients, Squashing Gradients and Inverting Gradients, they analyzed and tested the three methods and found that Inverting Gradients method can achieve much stronger performance than the other two. In our implementation, we remove the tanh function from SOP and use Inverting Gradients instead to bound the actions. Let p indicate the output of the last layer of the policy network. During exploration p will be the mean of a normal distribution that we sample actions from, the IG approach can be summarized by the following equation : Where ∇ p is the gradient of the policy loss w.r.t to p. During a policy network update, we first backpropagate the gradients from the outputs of the Q network to the output of the policy network for each data point in the batch, we then compute the ratio for each p value (each action dimension), depending on the sign of the gradient. We then backpropagate from the output of the policy network to parameters of the policy network, and we modify the gradients in the policy network according to the ratios we computed. We made an efficient implementation and further discuss the computation efficiency of IG in the implementation details section. We also investigate the effect of other interesting sampling schemes. We also implement the proportional variant of Prioritized Experience Replay with SOP. Since SOP has two Q-networks, we redefine the absolute TD error |δ| of a transition (s, a, r, s) to be the average absolute TD error in the Q network update: Within the sum, the first term y q (r, is simply the target for the Q network, and the term Q θ,l (s, a) is the current estimate of the l th Q network. For the i th data point, the definition of the priority value p i is p i = |δ i | +. The probability of sampling a data point P (i) is computed as: where β 1 is a hyperparameter that controls how much the priority value affects the sampling probability, which is denoted by α in, but to avoid confusion with the α in SAC, we denote it as β 1. The importance sampling (IS) weight w i for a data point is computed as: where β 2 is denoted as β in. Based on the SOP algorithm, we change the sampling method from uniform sampling to sampling using the probabilities P (i), and for the Q updates we apply the IS weight w i. This gives SOP with Prioritized Experience Replay (SOP+PER). We note that as compared with SOP+PER, ERE does not require a special data structure and has negligible extra cost, while PER uses a sum-tree structure with some additional computational cost. We also tried several variants of SOP+PER, but preliminary show that it is unclear whether there is improvement in performance, so we kept the algorithm simple. The ERE scheme is similar to an exponential sampling scheme where we assign the probability of sampling according to the probability density function of an exponential distribution. Essentially, in such a sampling scheme, the more recent data points get exponentially more probability of being sampled compared to older data. For the i th most recent data point, the probability of sampling a data point P (i) is computed as: We apply this sampling scheme to SOP and refer to this variant as SOP+EXP. Figure 6 shows a performance comparison of SOP, SOP+ERE, SOP+EXP and SOP+PER. Results show that the exponential sampling scheme gives a boost to the performance of SOP, and especially in the Humanoid environment, although not as good as ERE. Surprisingly, SOP+PER does not give a significant performance boost to SOP (if any boost at all). We also found that it is difficult to find hyperparameter settings for SOP+PER that work well for all environments. Some of the other hyperparameter settings actually reduce performance. It is unclear why PER does not work so well for SOP. A similar has been found in another recent paper , showing that PER can significantly reduce performance on TD3. Further research is needed to understand how PER can be successfully adapted to environments with continuous action spaces and dense reward structure. Figure 7 shows, for fixed η, how η affects the data sampling process, under the ERE sampling scheme. Recent data points have a much higher probability of being sampled compared to older data, and a smaller η value gives more emphasis to recent data. Different η values are desirable depending on how fast the agent is learning and how fast the past experiences become obsolete. So to make ERE work well in different environments with different reward scales and learning progress, we adapt η to the the speed of learning. To this end, define performance to be the training episode return. Define I recent to be how much performance improved from N/2 timesteps ago, and I max to be the maximum improvement throughout training, where N is the buffer size. Let the hyperparameter η 0 be the initial η value. We then adapt η according to the formula: η = η 0 · I recent /I max + 1 − (I recent /I max). Under such an adaptive scheme, when the agent learns quickly, the η value is low in order to learn quickly from new data. When progress is slow, η is higher to make use of the stabilizing effect of uniform sampling from the whole buffer. Figure 7b plots the expected number of times a data point in the buffer is sampled, with the data points ordered from most to least recent. In this section we discuss some programming details. These details are not necessary for understanding the algorithm, but they might help with reproducibility. In the ERE scheme, the sampling range always starts with the entire buffer (1M data) and then gradually shrinks. This is true even when the buffer is not full. So even if there are not many data points in the buffer, we compute c k based as if there are 1M data points in the buffer. One can also modify the design slightly to obtain a variant that uses the current amount of data points to compute c k. In addition to the reported scheme, we also tried shrinking the sampling range linearly, but it gives less performance gain. In our implementation we set the number of updates after an episode to be the same as the number of timesteps in that episode. Since environments do not always end at 1000 timesteps, we can give a more general formula for c k. Let K be the number of mini-batch updates, let N be the max size of the replay buffer, then: With this formulation, the range of sampling shrinks in more or less the same way with varying number of mini-batch updates. We always do uniform sampling in the first update, and we always have η When η is small, c k can also become small for some of the mini-batches. To prevent getting a minibatch with too many repeating data points, we set the minimum value for c k to 5000. We did not find this value to be too important and did not find the need to tune it. It also does not have any effect for any η ≥ 0.995 since the sampling range cannot be lower than 6000. In the adaptive scheme with buffer of size 1M, the recent performance improvement is computed as the difference of the current episode return compared to the episode return 500,000 timesteps earlier. Before we reach 500,000 timesteps, we simply use η 0. The exact way of computing performance improvement does not have a significant effect on performance as long as it is reasonable. In this section we give analysis on the additional programming and computation complexity brought by ERE and PER. In terms of programming complexity, ERE is a clear winner since it only requires a small adjustment to how we sample mini-batches. It does not modify how the buffer stores the data, and does not require a special data structure to make it work efficiently. Thus the implementation difficulty is minimal. PER (proportional variant) requires a sum-tree data structure to make it run efficiently. The implementation is not too complicated, but compared to ERE it is a lot more work. The exponential sampling scheme is very easy to implement, although a naive implementation will incur a significant computation overhead when sampling from a large buffer. To improve its computation efficiency, we instead uses an approximate sampling method. We first sample data indexes from segments of size 100 from the replay buffer, and then for each segment sampled, we sample one data point uniformly from that segment. In terms of computation complexity (not sample efficiency), and wall-clock time, ERE's extra computation is negligible. In practice we observe no difference in computation time between SOP and SOP+ERE. PER needs to update the priority of its data points constantly and compute sampling probabilities for all the data points. The complexity for sampling and updates is O(log(N)), and the rank-based variant is similar. Although this is not too bad, it does impose a significant overhead on SOP: SOP+PER runs twice as long as SOP. Also note that this overhead grows linearly with the size of the mini-batch. The overhead for the Mujoco environments is higher compared to Atari, possibly because the Mujoco environments have a smaller state space dimension while a larger batch size is used, making PER take up a larger portion of computation cost. For the exponential sampling scheme, the extra computation is also close to negligible when using the approximate sampling method. In terms of the proposed normalization scheme and the Inverting Gradients (IG) method, the normalization is very simple and can be easily implemented and has negligible computation overhead. IG has a simple idea, but its implementation is slightly more complicated than the normalization scheme. When implemented naively, IG can have a large computation overhead, but it can be largely avoided by making sure the gradient computation is still done in a batch-manner. We have made a very efficient implementation and our code is publicly available so that interested reader can easily reproduce it. In Figure 8 we show additional on applying ERE to SOP+IG. The shows that after applying the ERE scheme, SOP and IG both get a performance boost. The performance of the SOP+ERE and IG+ERE are similar. In figure 9, we show additional comparing TD3 with TD3 plus our normalization scheme, which we refer as TD3+. The show that after applying our normalization scheme, TD3+ has a significant performance boost in Humanoid, while in other environments, both algorithms achieve similar performance. To understand why entropy maximization is important for one environment but less so for another, we examine the actions selected when training SAC with and without entropy. Humanoid and Walker2d have action dimensions K = 17 and K = 6, respectively. In addition to the representative shown for one dimension for both environments in Section 3.2, the for all the dimensions are provided here in Figures 10 and 11. From Figure 10, we see that for Humanoid using SAC (which uses entropy maximization), the |µ k | values are small and fluctuate significantly for all 17 dimensions. On the other hand, for SAC without entropy the |µ k | values are typically huge, again for all 17 dimensions. This causes the actions a k to be persistently clustered at either M or -M. As for Walker, the |µ k | values are sensible for both algorithms for all 6 dimensions, as shown in figure 11.

represent the short summary for finding the corresponding scientific article (long).

We propose a new DRL off-policy algorithm achieving state-of-the-art performance.

Represent the TLDR for finding the corresponding scientific article!

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent text

Question: Very recently, it comes to be a popular approach for answering open-domain questions by first searching question-related passages, then applying reading comprehension models to extract answers. Existing works usually extract answers from single passages independently, thus not fully make use of the multiple searched passages, especially for the some questions requiring several evidences, which can appear in different passages, to be answered. The above observations raise the problem of evidence aggregation from multiple passages. In this paper, we deal with this problem as answer re-ranking. Specifically, based on the answer candidates generated from the existing state-of-the-art QA model, we propose two different re-ranking methods, strength-based and coverage-based re-rankers, which make use of the aggregated evidences from different passages to help entail the ground-truth answer for the question. Our model achieved state-of-the-arts on three public open-domain QA datasets, Quasar-T, SearchQA and the open-domain version of TriviaQA, with about 8\% improvement on the former two datasets. Open-domain question answering (QA) aims to answer questions from a broad range of domains by effectively marshalling evidence from large open-domain knowledge sources. Such resources can be Wikipedia, the whole web BID12, structured knowledge bases BID2 or combinations of the above (Baudiš &Šedivỳ, 2015).Recent work on open-domain QA has focused on using unstructured text retrieved from the web to build machine comprehension models BID9. These studies adopt a two-step process: an information retrieval (IR) model to coarsely select passages relevant to a question, followed by a reading comprehension (RC) model BID26 to infer an answer from the passages. These studies have made progress in bringing together evidence from large data sources, but they predict an answer to the question with only a single retrieved passage at a time. However, answer accuracy can often be improved by using multiple passages. In some cases, the answer can only be determined by combining multiple passages. In this paper, we propose a method to improve open-domain QA by explicitly aggregating evidence from across multiple passages. Our method is inspired by two notable observations from previous open-domain QA analysis:• First, compared with incorrect answers, the correct answer is often suggested by more passages repeatedly. For example, in FIG0 (a), the correct answer "danny boy" has more passages providing evidence relevant to the question compared to the incorrect one. This observation can be seen as multiple passages collaboratively enhancing the evidence for the correct answer.• Second, sometimes the question covers multiple answer aspects, which spreads over multiple passages. In order to infer the correct answer, one has to find ways to aggregate those multiple passages in an effective yet sensible way to try to cover all aspects. In FIG0 the correct answer "Galileo Galilei" at the bottom has passages P1, "Galileo was a physicist..." and P2, "Galileo discovered the first 4 moons of Jupiter", mentioning two pieces of evidence to match the question. In this case, the aggregation of these two pieces of evidence can help entail the ground-truth answer "Galileo Galilei". In comparison, the incorrect answer "Isaac Newton" has passages providing partial evidence on only "physicist, mathematician and astronomer". This observation illustrates the way in which multiple passages may provide complementary evidence to better infer the correct answer to a question. To provide more accurate answers for open-domain QA, we hope to make better use of multiple passages for the same question by aggregating both the strengthened and the complementary evidence from all the passages. We formulate the above evidence aggregation as an answer re-ranking problem. Re-ranking has been commonly used in NLP problems, such as in parsing and translation, in order to make use of high-order or global features that are too expensive for decoding algorithms BID6 BID27 BID16 BID11. Here we apply the idea of re-ranking; for each answer candidate, we efficiently incorporate global information from multiple pieces of textual evidence without significantly increasing the complexity of the prediction of the RC model. Specifically, we first collect the top-K candidate answers based on their probabilities computed by a standard RC/QA system, and then we use two proposed re-rankers to re-score the answer candidates by aggregating each candidate's evidence in different ways. The re-rankers are:• A strength-based re-ranker, which ranks the answer candidates according to how often their evidence occurs in different passages. The re-ranker is based on the first observation if an answer candidate has multiple pieces of evidence, and each passage containing some evidence tends to predict the answer with a relatively high score (although it may not be the top score), then the candidate is more likely to be correct. The passage count of each candidate, and the aggregated probabilities for the candidate, reflect how strong its evidence is, and thus in turn suggest how likely the candidate is the corrected answer.• A coverage-based re-ranker, which aims to rank an answer candidate higher if the union of all its contexts in different passages could cover more aspects included in the question. To achieve this, for each answer we concatenate all the passages that contain the answer together. The is a new context that aggregates all the evidence necessary to entail the answer for the question. We then treat the new context as one sequence to represent the answer, and build an attention-based match-LSTM model between the sequence and the question to measure how well the new aggregated context could entail the question. Overall, our contributions are as follows: 1) We propose a re-ranking-based framework to make use of the evidence from multiple passages in open-domain QA, and two re-rankers, namely, a strengthbased re-ranker and a coverage-based re-ranker, to perform evidence aggregation in existing opendomain QA datasets. We find the second re-ranker performs better than the first one on two of the three public datasets. 2) Our proposed approach leads to the state-of-the-art on three different datasets (Quasar-T BID9, SearchQA BID10 and TriviaQA BID17) and outperforms previous state of the art by large margins. In particular, we achieved up to 8% improvement on F1 on both Quasar-T and SearchQA compared to the previous best . Given a question q, we are trying to find the correct answer a g to q using information retrieved from the web. Our method proceeds in two phases. First, we run an IR model (with the help of a search engine such as google or bing) to find the top-N web passages p 1, p 2,..., p N most related to the question. Then a reading comprehension (RC) model is used to extract the answer from these passages. This setting is different from standard reading comprehension tasks (e.g. BID24), where a single fixed passage is given, from which the answer is to be extracted. When developing a reading comprehension system, we can use the specific positions of the answer sequence in the given passage for training. By contrast, in the open-domain setting, the RC models are usually trained under distant supervision BID9 BID17. Specifically, since the training data does not have labels indicating the positions of the answer spans in the passages, during the training stage, the RC model will match all passages that contain the ground-truth answer with the question one by one. In this paper we apply an existing RC model called R 3 to extract these candidate answers. After the candidate answers are extracted, we aggregate evidence from multiple passages by reranking the answer candidates. Given a question q, suppose we have a baseline open-domain QA system that can generate the top-K answer candidates a 1,..., a K, each being a text span in some passage p i. The goal of the re-ranker is to rank this list of candidates so that the top-ranked candidates are more likely to be the correct answer a g. With access to these additional features, the re-ranking step has the potential to prioritize answers not easily discoverable by the base system alone. We investigate two re-ranking strategies based on evidence strength and evidence coverage. An overview of our method is shown in FIG1. In open-domain QA, unlike the standard RC setting, we have more passages retrieved by the IR model and the ground-truth answer may appear in different passages, which means different answer spans may correspond to the same answer. To exploit this property, we provide two features to further re-rank the top-K answers generated by the RC model. This method is based on the hypothesis that the more passages that entail a particular answer, the stronger the evidence for that answer and the higher it should be ranked. To implement this we count the number of occurrences of each answer in the top-K answer spans generated by the baseline QA model and return the answer with the highest count. Measuring Strength by Probability Since we can get the probability of each answer span in a passages based on the RC model, we can also sum up the probabilities of the answer spans that are referring to the same answer. In this method, the answer with the highest probability is the final prediction 1. In the re-ranking scenario, it is not necessary to exhaustively consider all the probabilities of all the spans in the passages, as there may be a large number of different answer spans and most of them are irrelevant to the ground-truth answer. Remark: Note that neither of the above methods require any training. Both just take the candidate predictions from the baseline QA system and perform counting or probability calculations. At test time, the time complexity of strength-based re-ranking is negligible. Consider FIG0 where the two answer candidates both have evidence matching the first half of the question. Note that only the correct answer has evidence that could also match the second half. In this case, the strength-based re-ranker will treat both answer candidates the same due to the equal amount of supporting evidence, while the second answer has complementary evidence satisfying all aspects of the question. To handle this case, we propose a coverage-based re-ranker that ranks the answer candidates according to how well the union of their evidence from different passages covers the question. In order to take the union of evidence into consideration, we first concatenate the passages containing the answer into a single "pseudo passage" then measure how well this passage entails the answer for the question. As in the examples shown in FIG0 (b), we hope the textual entailment model will reflect (i) how each aspect of the question is matched by the union of multiple passages; and (ii) whether all the aspects of the question can be matched by the union of multiple passages. In our implementation an "aspect" of the question is a hidden state of a bi-directional LSTM BID15. The match-LSTM BID31 model is one way to achieve the above effect in entailment. Therefore we build our coverage-based re-ranker on top of the concatenated pseudo passages using the match-LSTM. The detailed method is described below. Passage Aggregation We consider the top-K answers, a 1,..., a K, provided by the baseline QA system. For each answer a k, k ∈ [1, K], we concatenate all the passages that contain a k, {p n |a k ∈ p n, n ∈ [1, N]}, to form the union passagep k. Our further model is to identify which union passage, e.g.,p k, could better entail its answer, e.g., a k, for the question. Measuring Aspect(Word)-Level Matching As discussed earlier, the first mission of the coverage-based re-ranker is to measure how each aspect of the question is matched by the union of multiple passages. We achieve this with word-by-word attention followed by a comparison module. First, we write the answer candidate a, question q and the union passagep of a as matrices A, Q,P, with each column being the embedding of a word in the sequence. We then feed them to the bidirectional LSTM as follows: DISPLAYFORM0 where H a ∈ R l×A, H q ∈ R l×Q and H p ∈ R l×P are the hidden states for the answer candidate, question and passage respectively; l is the dimension of the hidden states, and A, Q and P are the length of the three sequences, respectively. Next, we enhance the question representation H q with H a: DISPLAYFORM1 where [·; ·] is the concatenation of two matrices in row and H aq ∈ R l×(A+Q). As most of the answer candidates do not appear in the question, this is for better matching with the passage and finding more answer-related information from the passage.2 Now we can view each aspect of the question as a column vector (i.e. a hidden state at each word position in the answer-question concatenation) in the enhanced question representation H aq. Then the task becomes to measure how well each column vector can be matched by the union passage; and we achieve this by computing the attention vector Parikh et al. FORMULA0 for each hidden state of sequences a and q as follows: DISPLAYFORM2 where α ∈ R P ×(A+Q) is the attention weight matrix which is normalized in column through softmax. H aq ∈ R l×(A+Q) are the attention vectors for each word of the answer and the question by weighted summing all the hidden states of the passagep. Now in order to see whether the aspects in the question can be matched by the union passage, we use the following matching function: DISPLAYFORM3 where · ⊗ e (A+Q) is to repeat the vector (or scalar) on the left A + Q times; (· ·) and (· − ·) are the element-wise operations for checking whether the word in the answer and question can be matched by the evidence in the passage. We also concatenate these matching representations with the hidden state representations H aq and H aq, so that the lexical matching representations are also integrated into the the final aspect-level matching representations 3 M ∈ R 2l×(A+Q), which is computed through the non-linear transformation on four different representations with parameters DISPLAYFORM4 Measuring the Entire Question Matching Next, in order to measure how the entire question is matched by the union passagep by taking into consideration of the matching at each aspect, we add another bi-directional LSTM on top of it to aggregate the aspect-level matching information 4: DISPLAYFORM5 where H m ∈ R l×(A+Q) is to denote all the hidden states and h s ∈ R l, the max of pooling of each dimension of H m, is the entire matching representation which reflects how well the evidences in questions could be matched by the union passage.2 Besides concatenating H q with H a, there are other ways to make the matching process be aware of an answer's positions in the passage, e.g. replacing the answer spans in the passage to a special token like in. We tried this approach, which gives similar but no better , so we keep the concatenation in this paper. We leave the study of the better usage of answer position information for future work. 4 Note that we use LSTM here to capture the conjunction information (the dependency) among aspects, i.e. how all the aspects are jointly matched. In comparison simple pooling methods will treat the aspects independently. Low-rank tensor inspired neural architectures (e.g., BID21) could be another choice and we will investigate them in future work. Re-ranking Objective Function Our re-ranking is based on the entire matching representation. For each candidate answer a k, k ∈ [1, K], we can get a matching representation h s k between the answer a k, question q and the union passagep k through Eqn.. Then we transform all representations into scalar values followed by a normalization process for ranking: DISPLAYFORM6 where we concatenate the match representations for each answer in row through [·; ·], and do a non-linear transformation by parameters W r ∈ R l×l and b r ∈ R l to get hidden representation R ∈ R l×K. Finally, we map the transformed matching representations into scalar values through parameters w o ∈ R l and w o ∈ R. o ∈ R K is the normalized probability for the candidate answers to be ground-truth. Due to the aliases of the ground-truth answer, there may be multiple answers in the candidates are ground-truth, we use KL distance as our objective function: DISPLAYFORM7 where y k indicates whether a k the ground-truth answer or not and is normalized by K k=1 y k and o k is the ranking output of our model for a k. Although the coverage-based re-ranker tries to deal with more difficult cases compared to the strength-based re-ranker, the strength-based re-ranker works on more common cases according to the distributions of most open-domain QA datasets. We can try to get the best of both worlds by combining the two approaches. The full re-ranker is a weighted combination of the outputs of the above different re-rankers without further training. Specifically, we first use softmax to re-normalize the top-5 answer scores provided by the two strength-based rankers and the one coverage-based reranker; we then weighted sum up the scores for the same answer and select the answer with the largest score as the final prediction. We conduct experiments on three publicly available open-domain QA datasets, namely, Quasar-T BID9, SearchQA BID10 and TriviaQA BID17. These datasets contain passages retrieved for all questions using a search engine such as Google or Bing. We do not retrieve more passages but use the provided passages only. The statistics of the three datasets are shown in Table 1. 5 BID9 ) is based on a trivia question set. The data set makes use of the "Lucene index" on the ClueWeb09 corpus. For each question, 100 unique sentence-level passages were collected. The human performance is evaluated in an open-book setting, i.e., the human subjects had access to the same passages retrieved by the IR model and tried to find the answers from the passages. 6 BID10 ) is based on Jeopardy! questions and uses Google to collect about 50 web page snippets as passages for each question. The human performance is evaluated in a similar way to the Quasar-T dataset. 7 BID17 Our baseline models 9 include the following: GA BID8 b), a reading comprehension model with gated-attention; BiDAF BID26 ), a RC model with bidirectional attention flow; AQA BID4, a reinforced system learning to aggregate the answers generated by the re-written questions; R 3 ), a reinforced model making use of a ranker for selecting passages to train the RC model. As R 3 is the first step of our system for generating candidate answers, the improvement of our re-ranking methods can be directly compared to this baseline. TriviaQA does not provide the leaderboard under the open-domain setting. As a , there is no public baselines in this setting and we only compare with the R 3 baseline. We first use a pre-trained R 3 model, which gets the state-of-the-art performance on the three public datasets we consider, to generate the top 50 candidate spans for the training, development and test datasets, and we use them for further ranking. During training, if the groundtruth answer does not appear in the answer candidates, we will manually add it into the answer candidate list. For the coverage-based re-ranker, we use Adam BID19 to optimize the model. Word embeddings are initialized by GloVe BID23 and are not updated during training. We set all the words beyond Glove as zero vectors. We set l to 300, batch size to 30, learning rate to 0.002. We tune the dropout probability from 0 to 0.5 and the number of candidate answers for re-ranking (K) in 11. In this section, we present and analysis of our different re-ranking methods on the three different public datasets.8 Despite the open-domain QA data provided, the leaderboard of TriviaQA focuses on evaluation of RC models over filtered passages that is guaranteed to contain the correct answers (i.e. more like closed-domain setting). The evaluation is also passage-wise, different from the open-domain QA setting.9 Most of the of different models come from the public paper. While we re-run model R 3 based on the authors' source code and extend the model to the datasets of SearchQA and TriviaQA datasets. 10 To demonstrate that R 3 serves as a strong baseline on the TriviaQA data, we generate the R 3 following the leaderboard setting. The showed that R 3 achieved F1 56.0, EM 50.9 on Wiki domain and F1 68.5, EM 63.0 on Web domain, which is competitive to the state-of-the-arts. This confirms that R 3 is a competitive baseline when extending the TriviaQA questions to open-domain setting.11 Our code will be released under https://github.com/shuohangwang/mprc. DISPLAYFORM0 GA BID8 26.4 26.4 ----BiDAF BID26 25.9 28.5 28.6 34.6 --AQA BID4 The performance of our models is shown in TAB3. We use F1 score and Exact Match (EM) as our evaluation metrics 12. From the , we can clearly see that the full re-ranker, the combination of different re-rankers, significantly outperforms the previous best performance by a large margin, especially on Quasar-T and SearchQA. Moreover, our model is much better than the human performance on the SearchQA dataset. In addition, we see that our coverage-based re-ranker achieves consistently good performance on the three datasets, even though its performance is marginally lower than the strength-based re-ranker on the SearchQA dataset. In this subsection, we analyze the benefits of our re-ranking models. BM25 as an alternative coverage-based re-ranker We use the classical BM25 retrieval model BID25 ) to re-rank the aggregated passages the same way as the coveragebased re-ranker, where the IDF values are first computed from the raw passages before aggregation. From the in TAB3, we see that the BM25-based re-ranker improves the F1 scores compared with the R 3 model, but it is still lower than our coverage-based re-ranker with neural network models. Moreover, with respect to EM scores, the BM25-based re-ranker sometimes gives lower performance. We hypothesize that there are two reasons behind the relatively poor performance of BM25. First, because BM25 relies on a bag-of-words representation, context information is not taken into consideration and it cannot model the phrase similarities. Second, shorter answers tend to be preferred by BM25. For example, in our method of constructing pseudo-passages, when an answer sequence A is a subsequence of another answer sequence B, the pseudo passage of A is always a superset of the pseudo passage of B that could better cover the question. Therefore the F1 score could be improved but the EM score sometimes becomes worse. Re-ranking performance versus answer lengths and question types FIG3 decomposes the performance according to the length of the ground truth answers and the types of questions on TriviaQA and Quasar-T. We do not include the analysis on SearchQA because, for the Jeopardy! style questions, it is more difficult to distinguish the questions types, and the range of answer lengths is narrower. Our show that the coverage-based re-ranker outperforms the baseline in different lengths of answers and different types of questions. The strength-based re-ranker (counting) also gives improvement but is less stable across different datasets, while the strength-based re-ranker (probability) Table 3: The upper bound (recall) of the Top-K answer candidates generated by the baseline R 3 system (on dev set), which indicates the potential of the coverage-based re-ranker.tends to have and trends that are close to the baseline curves, which is probably because the method is dominated by the probabilities predicted by the baseline. The coverage-based re-ranker and the strength-based re-ranker (counting) have similar trends on most of the question types. The only exception is that the strength-based re-ranker performs significantly worse compared to the coverage-based re-ranker on the "why" questions. This is possibly because those questions usually have non-factoid answers, which are less likely to have exactly the same text spans predicted on different passages by the baseline. Potential improvement of re-rankers Table 3 shows the percentage of times the correct answer is included in the top-K answer predictions of the baseline R 3 method. More concretely, the scores are computed by selecting the answer from the top-K predictions with the best EM/F1 score. Therefore the final top-K EM and F1 can be viewed as the recall or an upper bound of the top-K predictions. From the , we can see that although the top-1 prediction of R 3 is not very accurate, there is high probability that a top-K list with small K could cover the correct answer. This explains why our re-ranking approach achieves large improvement. Also by comparing the upper bound performance of top-5 and our re-ranking performance in TAB3, we can see there is still a clear gap of about 10% on both datasets and on both F1 and EM, showing the great potential improvement for the re-ranking model in future work. Effect of the selection of K for the coverage-based re-ranker As shown in Table 3, as K ranges from 1 to 10, the recall of top-K predictions from the baseline R 3 system increases significantly. Table 5: Results of running strength-based re-ranker (counting) on different number of top-K answer candidates on Quasar-T (dev set).Ideally, if we use a larger K, then the candidate lists will be more likely to contain good answers. At the same time, the lists to be ranked are longer thus the re-ranking problem is harder. Therefore, there is a trade-off between the coverage of rank lists and the difficulty of re-ranking; and selecting an appropriate K becomes important. Table 4 shows the effects of K on the performance of coveragebased re-ranker. We train and test the coverage-based re-ranker on the top-K predictions from the baseline, where K ∈ {3, 5, 10}. The upper bound are the same ones from Table 3. The show that when K is small, like K=3, the performance is not very good due to the low coverage (thus low upper bound) of the candidate list. With the increase of K, the performance becomes better, but the top-5 and top-10 are on par with each other. This is because the higher upper bound of top-10 counteracts the harder problem of re-ranking longer lists. Since there is no significant advantage of the usage of K=10 while the computation cost is higher, we report all testing with K=5.Effect of the selection of K for the strength-based re-ranker Similar to Table 4, we conduct experiments to show the effects of K on the performance of the strength-based re-ranker. We run the strength-based re-ranker (counting) on the top-K predictions from the baseline, where K ∈ {10, 50, 100, 200}. We also evaluate the upper bound for these Ks. Note that the strength-based re-ranker is very fast and the different values of K do not affect the computation speed significantly compared to the other QA components. The are shown in Table 5, where we achieve the best when K=50. The performance drops significantly when K increases to 200. This is because the ratio of incorrect answers increases notably, making incorrect answers also likely to have high counts. When K is smaller, such incorrect answers appear less because statistically they have lower prediction scores. We report all testing with K=50.Examples Table 6 shows an example from Quasar-T where the re-ranker successfully corrected the wrong answer predicted by the baseline. This is a case where the coverage-based re-ranker helped: the correct answer "Sesame Street" has evidence from different passages that covers the aspects "Emmy Award" and "children's television shows". Although it still does not fully match all the facts in the question, it still helps to rank the correct answer higher than the top-1 prediction "Great Dane" from the R 3 baseline, which only has evidence covering "TV" and "1969" in the question. Table 6: An example from Quasar-T dataset. The ground-truth answer is "Sesame Street". Q: question, A: answer, P: passages containing corresponding answer. Open Domain Question Answering The task of open domain question answering dates back to as early as BID13 and was popularized by TREC-8 BID30. The task is to produce the answer to a question by exploiting resources such as documents BID30, webpages BID20 or structured knowledge bases BID2 BID3.Recent efforts BID10 BID9 benefit from the advances of machine reading comprehension (RC) and follow the search-and-read QA direction. These deep learning based methods usually rely on a document retrieval module to retrieve a list of passages for RC models to extract answers. As there is no passage-level annotation about which passages entail the answer, the model has to find proper ways to handle the noise introduced in the IR step. uses bi-gram passage index to improve the retrieval step; BID10; BID9 propose to reduce the length of the retrieved passages. focus more on noise reduction in the passage ranking step, in which a ranker module is jointly trained with the RC model with reinforcement learning. To the best of our knowledge, our work is the first to improve neural open-domain QA systems by using multiple passages for evidence aggregation. Moreover, we focus on the novel problem of "text evidence aggregation", where the problem is essentially modeling the relationship between the question and multiple passages (i.e. text evidence). In contrast, previous answer re-ranking research did not address the above problem: traditional QA systems like BID12 have similar passage retrieval process with answer candidates added to the queries. The retrieved passages were used for extracting answer scoring features, but the features were all extracted from single-passages thus did not utilize the information of union/co-occurrence of multiple passages. KB-QA systems BID0; BID34 sometimes use text evidence to enhance answer re-ranking, where the features are also extracted on the pair of question and a single-passage but ignored the union information among multiple passages. We are the first to introduce re-ranking methods to neural open-domain QA and multi-passage RC. Meanwhile, our two-step approach shares some similarities to the previous multi-step approaches proposed for standard single-passage RC, in terms of the purposes of either using additional information or re-fining answer predictions that are not easily handled by the standard answer extraction models for RC.On cloze-test tasks BID14, relates to our work in the sense that it is a two-step extractor-reasoner model, which first extracts K most probable singletoken answer candidates and then constructs a hypothesis by combining each answer candidate to the question and compares the hypothesis with all the sentences in the passage. Their model differs from ours in several aspects: (i) Epireader matches a hypothesis to every single sentence, including all the "noisy" ones that does not contain the answer, that makes the model inappropriate for open-domain QA setting; (ii) The sentence matching is based on the sentence embedding vectors computed by a convolutional neural network, which makes it hard to distinguish redundant and complementary evidence in aggregation; (iii) Epireader passes the probabilities predicted by the extractor to the reasoner directly to sustain differentiability, which cannot be easily adapted to our problem to handle phrases as answers or to use part of passages. Similarly, BID7 ) also combined answer candidates to the question to form hypotheses, and then explicitly use language models trained on documents to re-rank the hypotheses. This method benefits from the consistency between the documents and gold hypotheses (which are titles of the documents) in cloze-test datasets, but does not handle multiple evidence aggregation like our work. S-Net BID28 proposes a two-step approach for generative QA. The model first extracts an text span as the answer clue and then generates the answer according to the question, passage and the text span. Besides the different goal on answer generation instead of re-ranking like this work, their approach also differs from ours on that it extracts only one text span from a single selected passage. We have observed that open-domain QA can be improved by explicitly combining evidence from multiple retrieved passages. We experimented with two types of re-rankers, one for the case where evidence is consistent and another when evidence is complementary. Both re-rankers helped to significantly improve our individually, and even more together. Our considerably advance the state-of-the-art on three open-domain QA datasets. Although our proposed methods achieved some successes in modeling the union or co-occurrence of multiple passages, there are still much harder problems in open-domain QA that require reasoning and commonsense inference abilities. In future work, we will explore the above directions, and we believe that our proposed approach could be potentially generalized to these more difficult multipassage reasoning scenarios. This work was partially supported by DSO grant DSOCL15223.We thank Mandar Joshi for testing our model on the unfiltered TriviaQA hidden test dataset.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a method that can make use of the multiple passages information for open-domain QA.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.