Question: Represent the article for finding a single-sentence TLDR

Question: Deep reinforcement learning algorithms can learn complex behavioral skills, but real-world application of these methods requires a considerable amount of experience to be collected by the agent. In practical settings, such as robotics, this involves repeatedly attempting a task, resetting the environment between each attempt. However, not all tasks are easily or automatically reversible. In practice, this learning process requires considerable human intervention. In this work, we propose an autonomous method for safe and efficient reinforcement learning that simultaneously learns a forward and backward policy, with the backward policy resetting the environment for a subsequent attempt. By learning a value function for the backward policy, we can automatically determine when the forward policy is about to enter a non-reversible state, providing for uncertainty-aware safety aborts. Our experiments illustrate that proper use of the backward policy can greatly reduce the number of manual resets required to learn a task and can reduce the number of unsafe actions that lead to non-reversible states. Deep reinforcement learning (RL) algorithms have the potential to automate acquisition of complex behaviors in a variety of real-world settings. Recent have shown success on games BID16 ), locomotion BID22 ), and a variety of robotic manipulation skills;; BID8 ). However, the complexity of tasks achieved with deep RL in simulation still exceeds the complexity of the tasks learned in the real world. Why have real-world lagged behind the simulated accomplishments of deep RL algorithms?One challenge with real-world application of deep RL is the scaffolding required for learning: a bad policy can easily put the system into an unrecoverable state from which no further learning is possible. For example, an autonomous car might collide at high speed, and a robot learning to clean glasses might break them. Even in cases where failures are not catastrophic, some degree of human intervention is often required to reset the environment between attempts (e.g., BID2).Most RL algorithms require sampling from the initial state distribution at the start of each episode. On real-world tasks, this operation often corresponds to a manual reset of the environment after every episode, an expensive solution for complex environments. Even when tasks are designed so that these resets are easy (e.g., and BID8), manual resets are necessary when the robot or environment breaks (e.g., BID7). The bottleneck for learning many real-world tasks is not that the agent collects data too slowly, but rather that data collection stops entirely when the agent is waiting for a manual reset. To avoid manual resets caused by the environment breaking, task designers often add negative rewards to dangerous states and intervene to prevent agents from taking dangerous actions. While this works well for simple tasks, scaling to more complex environments requires writing large numbers of rules for types of actions the robot should avoid. For example, a robot should avoid hitting itself, except when clapping. One interpretation of our method is as automatically learning these safety rules. Decreasing the number of manual resets required to learn to a task is important for scaling up RL experiments outside simulation, allowing researchers to run longer experiments on more agents for more hours. We propose to address these challenges by forcing our agent to "leave no trace." The goal is to learn not only how to do the task at hand, but also how to undo it. The intuition is that the sequences of actions that are reversible are safe; it is always possible to undo them to get back to the original state. This property is also desirable for continual learning of agents, as it removes the requirements for manual resets. In this work, we learn two policies that alternate between attempting the task and resetting the environment. By learning how to reset the environment at the end of each episode, the agent we learn requires significantly fewer manual resets. Critically, our value-based reset policy restricts the agent to only visit states from which it can return, intervening to prevent the forward policy from taking potentially irreversible actions. Using the reset policy to regularize the forward policy encodes the assumption that whether our learned reset policy can reset is a good proxy for whether any reset policy can reset. The algorithm we propose can be applied to both deterministic and stochastic MDPs. For stochastic MDPs we say that an action is reversible if the probability that an oracle reset policy can successfully reset from the next state is greater than some safety threshold. The set of states from which the agent knows how to return grows over time, allowing the agent to explore more parts of the environment as soon as it is safe to do so. The main contribution of our work is a framework for continually and jointly learning a reset policy in concert with a forward task policy. We show that this reset policy not only automates resetting the environment between episodes, but also helps ensure safety by reducing how frequently the forward policy enters unrecoverable states. Incorporating uncertainty into the value functions of both the forward and reset policy further allows us to make this process risk-aware, balancing exploration against safety. Our experiments illustrate that this approach reduces the number of "hard" manual resets required during learning of a variety of simulated robotic skills. Our method builds off previous work in areas of safe exploration, multiple policies, and automatic curriculum generation. Previous work has examined safe exploration in small MDPs. Moldovan & Abbeel (2012a) examine risk-sensitive objectives for MDPs, and propose a new objective of which minmax and expectation optimization are both special cases. Moldovan & Abbeel (2012b) consider safety using ergodicity, where an action is safe if it is still possible to reach every other state after having taken that action. These methods are limited to small, discrete MDPs where exact planning is straightforward. Our work includes a similar notion of safety, but can be applied to solve complex, high-dimensional tasks. Thomas et al. (2015a; b) prove high confidence bounds for off policy evaluation and policy improvement. While these works look at safety as guaranteeing some reward, our work defines safety as guaranteeing that an agent can reset. Previous work has also used multiple policies for safety and for learning complex tasks. BID9 learn a sequence of forward and reset policies to complete a complex manipulation task. Similar to BID9, our work learns a reset policy to undo the actions of the forward policy. While BID9 engage the reset policy when the forward policy fails, we preemptively predict whether the forward policy will fail, and engage the reset policy before allowing the forward policy to fail. Similar to our approach, BID21 also propose to use a safety policy that can trigger an "abort" to prevent a dangerous situation. However, in contrast to our approach, BID21 use a heuristic, hand-engineered reset policy, while our reset policy is learned simultaneously with the forward policy. BID11 uses uncertainty estimation via bootstrap to provide for safety. Our approach also uses bootstrap for uncertainty estimation, but unlike our method, BID11 does not learn a reset or safety policy. Learning a reset policy is related to curriculum generation: the reset controller is engaged in increasingly distant states, naturally providing a curriculum for the reset policy. Prior methods have studied curriculum generation by maintaining a separate goal setting policy or network BID25 BID14. In contrast to these methods, we do not set explicit goals, but only allow the reset policy to abort an episode. When learning the forward and reset policies jointly, the training dynamics of our reset policy resemble those of reverse curriculum generation, but in reverse. In particular, reverse curriculum learning can be viewed as a special case of our method: our reset policy is analogous to the learner in the reverse curriculum, while the forward policy plays a role similar to the initial state selector. However, reverse curriculum generation requires that the agent can be reset to any state (e.g., in a simulator), while our method is specifically aimed at streamlining real-world learning, through the use of uncertainty estimation and early aborts. In this section, we discuss the episodic RL problem setup, which motivates our proposed joint learning of forward and reset policies. RL considers decision-making problems that consist of a state space S, action space A, transition dynamics P (s | s, a), an initial state distribution p 0 (s), and a scalar reward function r(s, a). In episodic, finite horizon tasks, the objective is to find the optimal policy π * (a | s) that maximizes the expected sum of γ-discounted returns, E π T t=0 γ t r(s t, a t), where DISPLAYFORM0 Typical RL training routines involve iteratively sampling new episodes; at the end of each episode, a new starting state s 0 is sampled from a given initial state distribution p 0. In practical applications, such as robotics, this procedure involves a hard-coded reset policy or a human intervention to manually reset the agent. Our work is aimed at avoiding these manual resets by learning an additional reset policy that satisfies the following property: when the reset policy is executed from any state reached by the forward policy, the distribution over final states is close to the initial state distribution p 0. If we learn such a reset policy, then the agent never requires querying the black-box distribution p 0 and can continually learn on its own. Our method for continual learning relies on jointly learning a forward policy and reset policy, using early aborts to avoid manual resets. The forward policy aims to maximize the task reward, while the reset policy takes actions to reset the environment. Both have the same state and action spaces, but are given different reward objectives. The forward policy reward r f (s, a) is the usual task reward given by the environment. The reset policy reward r r (s) is designed to approximate the initial state distribution. In practice, we found that a very simple design worked well for our experiments. We used the negative distance to some start state, plus any reward shaping included in the forward reward. To make this set-up applicable for solving the task, we make two assumptions on the task environment. First, we make the weak assumption that there exists a policy that can reset from at least one of the reachable states with maximum reward in the environment. This assumption ensures that it is possible to solve the task without any manual resets. Many manipulation and locomotion tasks in robotics satisfy this assumption. As a counterexample, the Atari game Ms. Pacman violates this assumption because transitioning from one level to the next level is not reversible; the agent cannot transition from level 3 back to level 1. Second, we assume that the initial state distribution is unimodal and has narrow support. This assumption ensures that the distribution over the reset policy's final state is close to the initial state distribution p 0. If the initial state distribution were multi-modal, the reset policy might only learn to return to one of these modes. Detecting whether an environment violates this second assumption is straightforward. A mismatch between p 0 and the reset policy's final state distribution will cause the forward policy to earn a small reward when the initial state is sampled from p 0 and a larger reward when the initial state is the final state of the reset policy. We choose off-policy actor-critic as the base RL algorithm BID24 BID13, since its off-policy learning allows sharing of the experience between the forward and reset policies. Additionally, the Q-functions can be used to signal early aborts. Our method can also be used directly with any other Q-learning method (; BID16 BID8 BID0 BID15 . The reset policy learns how to transition from the forward policy's final state back to an initial state. In challenging domains where the reset policy is unable to reset from some states or would take prohibitively long to reset, a costly manual reset is required. The reset policy offers a natural mechanism for reducing these manual resets. We observe that, for states from which we cannot quickly reset, the value function of the reset policy will be low. We can therefore use this value function (or, specifically, its Q-function) as a metric to determine when to terminate the forward policy, performing an early abort. Before an action proposed by the forward policy is executed in the environment, it must be "approved" by the reset policy. In particular, if the reset policy's Q-value for the proposed action is too small, then an early abort is performed: the proposed action is not taken and the reset policy takes control. Formally, early aborts restrict exploration to a'safe' subspace of the MDP. Let E ⊆ S × A be the set of (possibly stochastic) transitions, and let Q reset (s, a) be the Q-value of our reset policy at state s taking action a. The subset of transitions E * ∈ E allowed by our algorithm is DISPLAYFORM0 Noting that V (s) max a∈A Q(s, a), we see that given access to the true Q-values, Leave No Trace only visits safe states: DISPLAYFORM1 In Appendix A, we prove that if we learn the true Q-values for the reset policy, then early aborts restrict the forward policy to visiting states that are safe in expectation at convergence. Early aborts can be interpreted as a learned, dynamic, safety constraint, and a viable alternative for the manual constraints that are typically used for real-world RL experiments. Early aborts promote safety by preventing the agent from taking actions from which it cannot recover. These aborts are dynamic because the states at which they occur change throughout training as more states are considered safe. Early aborts can make learning the forward policy easier by preventing the agent from entering unsafe states. We experimentally analyze early aborts in Section 6.3, and discuss how our approach handles over/under-estimates of Q-values in Appendix B. A hard reset is an action that resamples that state from the initial state distribution. Hard resets are available to an external agent (e.g., a human) but not the learned agent. Early aborts decrease the requirement for "hard" resets, but do not eliminate them, since an imperfect reset policy might still miss a dangerous state early in the training process. It is challenging to identify whether any policy can reset from the current state. Formally, we define a set of states S reset that give a reward greater than r min to the reset policy: DISPLAYFORM0 We say that we are in an irreversible state if we have not visited a state in S reset within the past N episodes, where N is a hyperparameter. This is a necessary but not sufficient condition, as the reset policy may have not yet learned to reset from a safe state. Increasing N decreases the number of hard resets. However, when we are in an irreversible state, increasing N means that we remain in that state (learning nothing) for more episodes. Section 6.4 empirically examines this trade-off. In practice, the setting of this parameter should depend on the cost of hard resets. Our full algorithm (Algorithm 1) consists of alternately running a forward policy and reset policy. When running the forward policy, we perform an early abort if the Q-value for the reset policy is less than Q min. Only if the reset policy fails to reset after N episodes do we do a manual reset. The accuracy of the Q-value estimates directly affects task reward and indirectly affects safety (through early aborts). Our Q-values may not be good estimates of the true value function for (s, r) ← ENVIRONMENT. STEP(a) 7:Update the forward policy.8:for max steps per episode do 9:a ← RESET AGENT.CHOOSE ACTION(s) 10:(s, r) ← ENVIRONMENT.STEP (a) 11:Update the reset policy.12:Let S N reset be the final states from the last N reset episodes. DISPLAYFORM0 s ← ENVIRONMENT.RESET Hard Reset previously-unseen states. To address this, we train Q-functions for both the forward and reset policies that provide uncertainty estimates. Several prior works have explored how uncertainty estimates can be obtained in such settings BID6 BID19. In our method, we train an ensemble of Q-functions, each with a different random initialization. This technique has been established in the literature as a principled way to provides a distribution over Q-values at each state given the observed data BID19 BID3.Given this distribution over Q-values, we can propose three strategies for early aborts: Optimistic Aborts: Perform an early abort only if all the Q-values are less than Q min. Equivalently, do an early abort if max θ Q θ reset (s, a) < Q min. Realist Aborts: Perform an early abort if the mean Q-value is less than Q min. Pessimistic Aborts: Perform an early abort if any of the Q-values are less than Q min. Equivalently, do an early abort if min θ Q θ reset (s, a) < Q min. We expect that optimistic aborts will provide better exploration at the cost of more hard resets, while pessimistic aborts should decrease hard resets, but may be unable to effectively explore. We empirically test this hypothesis in Appendix C. We first present a small didactic example to illustrate how our forward and reset policies interact and how cautious exploration reduces the number of hard resets. We first discuss the gridworld in FIG0. The states with red borders are absorbing, meaning that the agent cannot leave them and must use a hard reset. The agent receives a reward of 1 for reaching the goal state, and 0 otherwise. States are colored based on the number of early aborts triggered in each state. Note that most aborts occur next to the initial state, when the forward policy attempts to enter the absorbing state South-East of the start state, but is blocked by the reset policy. In FIG1, we present a harder environment, where the task can be successfully completed by reaching one of the two goals, exactly one of which is reversible. The forward policy has no preference for which goal is better, but the reset policy successfully prevents the forward policy from entering the absorbing goal state, as indicated by the much larger early abort count in the blue-colored state next to the absorbing goal. 78% without increasing the number of steps to solve the task. In a real-world setting, this might produce a substantial gain in efficiency, as time spend waiting for a hard reset could be better spent collecting more experience. Thus, for some real-world experiments, increasing Q min can decrease training time even if it requires more steps to learn. Pusher Cliff Cheetah Cliff Walker Peg InsertionIn this section, we use the five complex, continuous control environments shown above to answer questions about our approach. While ball in cup and peg insertion are completely reversible, the other environments are not: the pusher can knock the puck outside its workspace and the cheetah and walker can jump off a cliff. Crucially, reaching the goal states or these irreversible states does not terminate the episode, so the agent remains in the irreversible state until it calls for a hard reset. To ensure fair evaluation of all approaches, we use a different procedure for evaluation than for training. We evaluate the performance of a policy by creating a copy of the policy in a separate thread, running the forward policy for a fixed number of steps, and computing the average per-step reward. All approaches observe the same amount of data during training. We visualize the training dynamics and provide additional plots and experimental details are in the Appendix. Figure 5: We compare our method to a nonepisodic ("forward-only") approach on ball in cup. Although neither uses hard resets, only our method learns to catch the ball. As an upper bound, we also show the "status quo" approach that performs a hard reset after episode, which is often impractical outside simulation. One proposal for learning without resets is to run the forward policy until the task is learned. This "forwardonly" approach corresponds to the standard, fully online, non-episodic lifelong RL setting, commonly studied in the context of temporal difference learning BID26 ). We show that this approach fails, even on reversible environments where safety is not a concern. We benchmarked the forward-only approach and our method on ball in cup, using no hard resets for either. Figure 5 shows that our approach solves the task while the "forward-only" approach fails to learn how to catch the ball when initialized below the cup. Note that the x axis includes steps taken by the reset policy. Once the forward-only approach catches the ball, it gets maximum reward by keeping the ball in the cup. In contrast, our method learns to solve this task by automatically resetting the environment after each attempt, so the forward policy can practice catching the ball without hard resets. As an upper bound, we show policy reward for the "status quo" approach, which performs a hard reset after every attempt. Note that the dependence on hard resets makes this third method impractical outside simulation. Pusher Cliff CheetahFigure 6: Our method achieves equal or better rewards than the status quo with fewer manual resets. Our first goal is to reduce the number of hard resets during learning. In this section, we compare our algorithm to the standard, episodic learning setup ("status quo"), which only learns a forward policy. As shown in Figure 6 (left), the conventional approach learns the pusher task somewhat faster than ours, but our approach eventually achieves the same reward with half the number of hard resets. In the cliff cheetah task (Figure 6 (right)), not only does our approach use an order of magnitude fewer hard resets, but the final reward of our method is substantially higher. This suggests that, besides reducing the number of resets, the early aborts can actually aid learning by preventing the forward policy from wasting exploration time waiting for resets in irreversible states. Pusher Cliff Cheetah Figure 7: Early abort threshold: Increasing the early abort threshold to act more cautiously avoids many hard resets, indicating that early aborts help avoid irreversible states. To test whether early aborts prevent hard resets, we can see if the number of hard resets increases when we lower the early abort threshold. Figure 7 shows the effect of three values for Q min while learning the pusher and cliff cheetah. In both environments, decreasing the early abort threshold increased the number of hard resets, supporting our hypothesis that early aborts prevent hard resets. On pusher, increasing Q min to 80 allowed the agent to learn a policy that achieved nearly the same reward using 33% fewer hard resets. The cliff cheetah task has lower rewards than pusher, even an early abort threshold of 10 is enough to prevent 69% of the total early aborts that the status quo would have performed. While early aborts help avoid hard resets, our algorithm includes a mechanism for requesting a manual reset if the agent reaches an unresettable state. As described in Section 4.2, we only perform a hard reset if the reset agent fails to reset in N consecutive episodes. Figure 8 shows how the number of reset attempts, N, affects hard resets and reward. On the pusher task, when our algorithm was given a single reset attempt, it used 64% fewer hard resets than the status quo approach would have. Increasing the number of reset attempts to 4 ed in another 2.5x reduction in hard resets, while decreasing the reward by less than 25%. On the cliff cheetah task, increasing the number of reset attempts brought the number of resets down to nearly zero, without changing the reward. Surprisingly, these indicate that for some tasks, it is possible to learn an equally good policy with significantly fewer hard resets. Our approach uses an ensemble of value functions to trigger early aborts. Our hypothesis was that our algorithm would be sensitive to bias in the value function if we used a single Q network. To test this hypothesis, we varied the ensemble size from 1 to 50. Figure 9 shows the effect on learning the pushing task. An ensemble with one network failed to learn, but still required many hard resets. Increasing the ensemble size slightly decreased the number of hard resets without affecting the reward. Our method can automatically produce a curriculum in settings where the desired skill is performed by the reset policy, rather than the forward policy. As an example, we evaluate our method on a peg insertion FIG0: Our method automatically induces a curriculum, allowing the agent to solve peg insertion with sparse rewards.task, where the reset policy inserts the peg and the forward policy removes it. The reward for a successful peg insertion is provided only when the peg is in the hole, making this task challenging to learn with random exploration. Hard resets provide illustrations of what a successful outcome looks like, but do not show how to achieve it. Our algorithm starts with the peg in the hole and runs the forward (peg removal) policy until an early abort occurs. As the reset (peg insertion) policy improves, early aborts occur further and further from the hole. Thus, the initial state distribution for the reset (peg insertion) policy moves further and further from the hole, increasing the difficulty of the task as the policy improves. We compare our approach to an "insert-only" baseline that only learns the peg insertion policy -we manually remove the peg from the hole after every episode. For evaluation, both approaches start outside the hole. FIG0 shows that only our method solves the task. The number of resets required by our method plateaus after one million steps, indicating that it has solved the task and no longer requires hard resets at the end of the episode. In contrast, the "insert-only" baseline fails to solve the task, never improving its reward. Thus, even if reducing manual resets is not important, the curriculum automatically created by Leave No Trace can enable agents to learn policies they otherwise would be unable to solve. In this paper, we presented a framework for automating reinforcement learning based on two principles: automated resets between trials, and early aborts to avoid unrecoverable states. Our method simultaneously learns a forward and reset policy, with the value functions of the two policies used to balance exploration against recoverability. Experiments in this paper demonstrate that our algorithm not only reduces the number of manual resets required to learn a task, but also learns to avoid unsafe states and automatically induces a curriculum. Our algorithm can be applied to a wide range of tasks, only requiring a few manual resets to learn some tasks. During the early stages of learning we cannot accurately predict the consequences of our actions. We cannot learn to avoid a dangerous state until we have visited that state (or a similar state) and experienced a manual reset. Nonetheless, reducing the number of manual resets during learning will enable researchers to run experiments for longer on more agents. A second limitation of our work is that we treat all manual resets as equally bad. In practice, some manual resets are more costly than others. For example, it is more costly for a grasping robot to break a wine glass than to push a block out of its workspace. An approach not studied in this paper for handling these cases would be to specify costs associated with each type of manual reset, and incorporate these reset costs into the learning algorithm. While the experiments for this paper were done in simulation, where manual resets are inexpensive, the next step is to apply our algorithm to real robots, where manual resets are costly. A challenge introduced when switching to the real world is automatically identifying when the agent has reset. In simulation we can access the state of the environment directly to compute the distance between the current state and initial state. In the real world, we must infer states from noisy sensor observations to deduce if they are the same. In this section, we prove that if we indeed learn the true Q-values for the reset policy, then the abort condition stipulated by our method will keep the forward policy safe (able to reset) for deterministic infinite-horizon discounted reward MDPs. For stochastic MDPs, the abort condition will keep the forward policy safe in expectation. Thus, the abort condition is effective at convergence. Before convergence, the reliability of this abort condition depends on the accuracy of the learned Q-values. In practice, we partially mitigate issues with imperfect Q functions by means of the Q-function ensemble (Section 4.4). In the proofs that follow, we make the following assumptions:1. The reward function for the reset policy depends only on the current state. We use r r (s) as the reset reward received for arriving at state s throughout this proof. 2. From every state s t, there exists an action a t such that the expected reset reward at the next state is at least as large as the reset reward at the current state: DISPLAYFORM0 For example, if the reset reward is uniform over S reset and zero everywhere else, then this assumption requires that for every state S reset, there exists an action that deterministically transitions to another state in S reset. As a counterexample, a modified cliff cheetah environment where the cheetah is initialized a meter above the ground does not satisfy this assumption. If the cheetah in S reset (above the ground), there are no actions it can take to stop itself from falling to the ground and leaving S reset. In the proofs below, we assume a stochastic MDP. For a deterministic MDP, we can remove the expectations over s t+1 (Lemma 4). To begin, we use the two assumptions to establish a lower bound for the value of any state for the reset policy. Lemma 1. For every state s ∈ S, the expected cumulative discounted reward for the reset agent is greater than a term that depends on the discount γ and the reward of the current state r r (s): DISPLAYFORM0 Proof. As a consequence of the assumptions, a reset policy at state s that acts optimally is guaranteed to receive an expected reset reward of at least r r (s) in every future time step. Thus, its expected cumulative discounted reward is at least DISPLAYFORM1 Next, we show that the reset policy can choose actions so that the Q-values do not decrease in expectation. Theorem 1. For any state s t ∈ S and action a t ∈ A, there exists another action a * t+1 ∈ A such that DISPLAYFORM2 In the proof that follows, note that the next state s t+1 is an unknown random variable. Functions of s t+1 are also random variables. Proof. Let state s t and action a t be given and let s t+1 be a random variable indicating the next state following a possibly stochastic transition. Let a * t+1 to be the action with largest Q-value at the next state: DISPLAYFORM3 We want to bound the expected difference between Q reset (s t, a t) and Q reset (s t+1, a * t+1), where the expectation is with respect to the unknown next state s t+1. We begin by unrolling the first term of the Q-value. Because a * t+1 is defined to be the action with largest Q-value, we can replace the first Q-value expression with the value function. We then apply the bound from Lemma 1. For brevity, we omit the subscript reset and omit that the expectation is over s t+1. DISPLAYFORM4 Next, we want to show that if one state is safe, the next state will also be safe in expectation. As a reminder, we say transitions (s, a) in E * are safe and states s in S * are safe (Eq. 1 and 3): DISPLAYFORM5 Lemma 2. Let safe state s t ∈ S * be given and choose an action a t such that (s t, a t) ∈ E *. Then the following state s t+1 is also safe in expectation: DISPLAYFORM6 Proof. By our assumption that (s t, a t) ∈ E *, we know Q reset (s t, a t) > Q min. Combining with Theorem 1, we get Proof. Proof by induction. We assumed that the initial state s 0 is safe. Lemma 2 shows that safety is a preserved invariant. Thus, each future state s t is also safe in expectation. DISPLAYFORM7 Leave No Trace being safe in expectation means that if we look an arbitrary number of steps into the future, the expected Q-value for that state is at least Q min. Equivalently, the probability that the state we arrive at is safe is greater than 50%.Deterministic MDPs are a special case for which we can prove that Leave No Trace only visits safe states (not in expectation).Lemma 4. For deterministic MDPs, if the initial state s 0 is safe, then Leave No Trace only visits states that are also safe (not in expectation.)Proof. When the next state s t+1 is a deterministic function of the current state s t and action a t, we can remove the expectation over s t+1 from Theorem 1 and Lemma 2. Thus, if the initial state s 0 is safe, we are guaranteed that every future state is also safe. In practice, Leave No Trace does visit unsafe states (though significantly less frequently than existing approaches). First, the proofs above only show that each state is safe in expectation. We do not prove that every state is safe with high probability. Second, we do not have access to the true Q-values. Our learned Q-function may overestimate the Q-value of some action, leading us to take an unsafe action. Empirically, we found that using an ensemble of Q-functions helped mitigate this problem, decreasing the number of unsafe actions taken as compared to using a single Q-function (Section 6.5). We introduced early aborts in Section 4.1 and analyzed them in Appendix A under the assumption that we had access to the true Q-values. In practice, our learned Q-values may over/under-estimate the Q-value for the reset policy. First, consider the case that the Q-function overestimates the reset Q-value for state s u, so the agent mistakenly thinks that unsafe state s u is safe. The agent will visit state s u and discover that it cannot reset. When the reset Q-function is updated with this experience, it will decrease its predicted reset Q-value for state s u. Second, consider the case that the Q-function underestimates the reset Q-value for state s s, so the agent mistakenly thinks that safe state s s is unsafe. For continuous tasks, once the agent learns to reset from a nearby safe state, generalization of the Q-function across states will lead the reset policy to assume that it can also reset from the state s s. For discrete state tasks where the Q-function does not generalize across states, we act optimistically in the face of uncertainty by acting based on the largest predicted Q-value from our ensemble, helping to avoid this second case (see Appendix C). We benchmarked three methods for combining our ensemble of values functions (optimistic, realistic, and pessimistic, as discussed in Section 4.4). FIG0 compares the three methods on gridworld on the gridworld environment from Section 5. Only the optimistic agent efficiently explored. As expected, the realistic and pessimistic agents, which are more conservative in letting the forward policy continue, fail to explore when Q min is too large. Interestingly, for the continuous control environments, the ensembling method makes relatively little difference for the number of resets or final performance, as shown in FIG0. This suggests that much of the benefit of ensemble comes from its ability to produce less biased abort predictions in novel states, rather than the particular risk-sensitive rule that is used. This also indicates that no Q-function in the ensemble significantly overestimates or underestimates the value function -such a Q-function would in bogus Q-value estimates when the ensemble was combined by taking the max or min (respectively). In this section, we provide some intuition for the training dynamics of our algorithm. In particular, we visualize the number of steps taken by the forward policy before an early abort occurs. FIG0 shows this quantity (the episode length for the forward policy) as a function of training iteration. Note that we stop the forward policy after a fixed number of steps (500 steps for cliff cheetah and cliff walker, 100 steps for pusher) if an early abort has not already occurred. For all tasks, initially the reset policy is unable to reset from any state, so early aborts happen almost immediately. As the reset policy improves, early aborts occur further and further from the initial state distribution, corresponding to longer forward episode lengths. In all tasks, increasing the safety threshold Q min caused early aborts to occur sooner, especially early in training. For the cliff cheetah, another curious pattern emerges when Q min is 10 and 20. After 200 thousand steps, the agent had learned rudimentary policies for running forwards and backwards. As the agent learns to run forwards faster, it reaches the cliff sooner and does an early abort, so the forward episode length actually decreases. For cliff walker, we do not see the same pattern because the forward task is more difficult, so the agent only reaches the cliff near the end of training. Both the cliff walker and pusher environments highlight the sensitivity of our method to Q min. If Q min is too small, early aborts will never occur. Automatically tuning the safety threshhold based on the real-world cost of hard resets is an exciting direction for future research. Cliff Cheetah We show the number of steps taken before an early abort for cliff cheetah (top row), cliff walker (middle row), and pusher (bottom row). Increasing the safety threshold causes early aborts to occur earlier, causing the agent to explore more cautiously. These plots are the average across 5 random seeds. For each experiment in the main paper, we chose one or two demonstrative environments. Below, we show all experiments run on cliff cheetah, cliff walker, and pusher. This experiment, described in Section 6.2, compared our method to the status quo approach (resetting after every episode). FIG0 shows plots for all environments. This experiment, described in Section 6.3, shows the effect of varying the early abort threshold. FIG0 shows plots for all environments. To generate FIG0, we averaged early abort counts across 10 random seeds. For Figure 3 we took the median across 10 random seeds. Both gridworld experiments used 5 models in the ensemble. In this section, we provide additional information on the continuous control environments in our experiments. We use the ball in cup environment implemented in. The cliff cheetah and cliff walker environments are modified versions of the cheetah and walker environments in. The pusher environment is a modified version of Pusher-v0 environment in BID1. Finally, the peg insertion environment is based on BID4.

We propose an autonomous method for safe and efficient reinforcement learning that simultaneously learns a forward and backward policy, with the backward policy resetting the environment for a subsequent attempt.

represent the tldr for finding the corresponding scientific article

Adaptation of an RL agent in a target environment with unknown dynamics is fast and safe when we transfer prior experience in a variety of environments and then select risk-averse actions during adaptation.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: It has been argued that current machine learning models do not have commonsense, and therefore must be hard-coded with prior knowledge . Here we show surprising evidence that language models can already learn to capture certain common sense knowledge. Our key observation is that a language model can compute the probability of any statement, and this probability can be used to evaluate the truthfulness of that statement. On the Winograd Schema Challenge , language models are 11% higher in accuracy than previous state-of-the-art supervised methods. Language models can also be fine-tuned for the task of Mining Commonsense Knowledge on ConceptNet to achieve an F1 score of 0.912 and 0.824, outperforming previous best (Jastrzebskiet al., 2018). Further analysis demonstrates that language models can discover unique features of Winograd Schema contexts that decide the correct answers without explicit supervision. It has been argued that current machine learning models do not have common sense BID4 BID15. For example, even best machine learning models perform poorly on commonsense reasoning tasks such as Winograd Schema Challenge BID11 BID14. This argument is often combined with another important criticism of supervised learning that it only works well on problems that have a lot of labeled data. The Winograd Schema Challenge (WSC) is an opposite of such problems because its labeled set size is only on the order of a few hundreds examples, with no official training data. Based on this argument, it is suggested that machine learning models must be integrated with prior knowledge BID15 BID10.As an example, consider the following question from the WSC dataset:"The trophy doesn't fit in the suitcase because it is too big. " What is "it"? Answer 0: the trophy. Answer 1: the suitcase. The main point of this dataset is that no machine learning model today can do a good job at answering this type of questions. In this paper, we present surprising evidence that language models do capture certain common sense knowledge and this knowledge can be easily extracted. Key to our method is the use of language models (LMs), trained on a large amount of unlabeled data, to score multiple choice questions posed by the challenge and similar datasets. In the above example, we will first substitute the pronoun ("it") with the candidates ("the trophy" and "the suitcase"), and then use an LM to compute the probability of the two ing sentences ("The trophy doesn't fit in the suitcase because the trophy is too big. " and "The trophy doesn't fit in the suitcase because the suitcase is too big. "). The substitution that in a more probable sentence will be the chosen answer. Using this simple method, we are able to achieve 63.7% accuracy, 11% above that of the previous state-of-the-art 1.To demonstrate a practical impact of this work, we show that the trained LMs can be used to enrich human-annotated knowledge bases, which are known to be low in coverage and expensive to expand. For example, "Suitcase is a type of container", a relevant knowledge to the above Winograd Schema example, does not present in the ConceptNet knowledge base BID13. The goal of this task is to add such new facts to the knowledge base at a cheaper cost than human annotation, in our case using LM scoring. We followed the Commonsense Knowledge Mining task formulation from BID0 BID12 BID8, which posed the task as a classification problem of unseen facts and non-facts. Without an additional classification layer, LMs are fine-tuned to give different scores to facts and non-facts tuples from ConceptNet. Results obtained by this method outperform all previous , despite the small training data size (100K instances). On the full test set, LMs can identify commonsense facts with 0.912 F1 score, which is 0.02 better than supervised trained networks BID8. Previous attempts at solving Winograd Schema Challenge usually involve heavy utilization of annotated knowledge bases, rule-based reasoning, or hand-crafted features BID21 BID1 BID27. BID29 rely on a semantic parser to understand the question, query Google Search, and perform rule-based reasoning. BID27 formalizes the knowledge-graph data structure and a reasoning process based on cognitive linguistics theories. BID1 introduce a mathematical reasoning framework with knowledge bases as axioms. BID24 is an early empirical work towards WSC making use of learning. Their SVM, however, utilizes nearly 70K hand-crafted features and additional supervised training data, while being tested on a less restricted version of WSC. Concurrent work from BID23 attempts WSC by fine-tuning pretrained Transformer LMs on supervised training data, but did not produce better than previous methods. In contrast, we make use of LSTMs, which are shown to be qualitatively different BID30 ) and obtain significant improvements without fine-tuning. The previous best method on WSC makes use of the skip-gram model to learn word representations BID14. Their model, however, also includes supervised neural networks and three knowledge bases. Our work uses the same intuition that unsupervised learning from texts such as a skip-gram model can capture some aspect of commonsense. For example, BID16 BID10 show that by learning to predict adjacent words in a sentence, word vectors can be used to answer analogy questions such as Man:King::Woman:?. The difference is that WSC requires more contextual information, and hence we use LMs instead of just word vectors. By training LMs on very large text corpora, we obtain good without any supervised learning nor the aid of knowledge bases. Closely related to our substitution method on Winograd Schema Challenge are Cloze type reading comprehension tasks such as LAMBADA BID20 or Store Cloze Test BID19, where LM scoring also reported great successes BID2 BID28. On a broader impact, neural LMs have been applied to improve downstream applications BID3 BID26 BID22 BID7 BID23 by providing better sentence or paragraph vector representations. Knowledge bases constructed by human are high in precision, but low in coverage. Since increasing the coverage by more human annotation is expensive, automated methods have been proposed. Previous attempts using deep neural networks are known to produce limited success on the ConceptNet knowledge base, where training data is limited. BID12 shows that a supervised LSTM is outperformed by a simpler model in scoring unseen facts from ConceptNet. Furthermore, BID8 find Deep Neural Network's performance degrades significantly on a selected subset of most novel test instances in comparison to training data. In Section 5.2, we demonstrate that our trained LMs do not suffer from this phenomenon and outperform all previous methods on both test criteria. In this section, we introduce a simple and straightforward application of pretrained language models on Winograd Schema Challenge. Our method is based on the observation that a language model can compute the probability of any given statement. We use this probability to judge the truthfulness of the statement. We first substitute the pronoun in the original sentence with each of the candidate choices. The problem of coreference resolution then reduces to identifying which substitution in a more probable sentence. Language modeling subsequently becomes a natural solution by its definition. Namely, language models are trained on text corpora, which encodes human knowledge in the form of natural language. During inference, LMs are able to assign probability to any given text based on what they have learned from training data. An overview of our method is shown in Figure 1.Figure 1: Overview of our method and analysis. We consider the test "The trophy doesn't fit in the suitcase because it is too big." Our method first substitutes two candidate references trophy and suitcase into the pronoun position. We then use an LM to score the ing two substitutions. By looking at the probability ratio at every word position, we are able to detect "big" as the main contributor to trophy being the chosen answer. When "big" is switched to "small", the answer changes to suitcase. This switching behaviour is an important feature characterizing the Winograd Schema Challenge. Consider a sentence S consisting of n consecutive words has its pronoun to be resolved specified at the k th position: 2 S = {w 1, .., w k−1, w k ≡ p, w k+1, .., w n}. We make use of a trained language model P θ (w t |w 1, w 2, .., w t−1), which defines the probability of word w t conditioned on the previous words w 1,..., w t−1. The substitution of a candidate reference c in to the pronoun position k in a new sentence S w k ←c (we use notation w k ← c to mean that word w k is substituted by candidate c). We consider two different ways of scoring the substitution:• Score f ull (w k ← c) = P θ (w 1, w 2, ..., w k−1, c, w k+1, ..., w n) which scores how probable the ing full sentence is, and DISPLAYFORM0 which scores how probable the part of the ing sentence following c is, given its antecedent. In other words, it only scores a part of S w k ←c conditioned on the rest of the substituted sentence. An example of these two scores is shown in Table 1. In our experiments, we find that the partial scoring strategy is generally better than the naive full scoring strategy. More comparison and analysis on scoring type is done in Section 6.3. Table 1: Example of full and partial scoring for the test "The trophy doesn't fit in the suitcase because it is too big." with two reference choices "the suitcase" and "the trophy". c = the suitcase Score f ull (w k ← "the suitcase") = P (The trophy doesn't fit in the suitcase because the suitcase is too big) Score partial (w k ← "the suitcase") = P (is too big| The trophy doesn't fit in the suitcase because the suitcase) c = the trophy Score f ull (w k ← "the trophy") = P (The trophy doesn't fit in the suitcase because the trophy is too big) Score partial (w k ← "the trophy") = P (is too big| The trophy doesn't fit in suitcase because the trophy) We consider two types of Recurrent language models, one processes word inputs and the other processes character inputs. All output layers are constructed to only produce word outputs, allowing both types of input processing to join in ensembles where the conditional probability at each word position is averaged over all ensemble members. To handle human names in Winograd Schema Challenge, we simply make use of a very large vocabulary (approximately 800K tokens). We follow architectural design and training scheme in BID9, with additional modifications to create more LM variants. More details about our LMs can be found in Appendix A. In this section we describe training text corpora used in our experiments. We also detail tests for commonsense reasoning and commonsense knowledge mining. Training text corpora. We perform experiments on several different text corpora to examine the effect of training data type on test accuracy. Namely, we consider LM-1-Billion, CommonCrawl, 3SQuAD and Gutenberg Books. For SQuAD, we collect context passages from the Stanford QuestionAnswering Dataset BID25 to form its training and validation set accordingly. Commonsense Reasoning Tests. We consider two tests: Pronoun Disambiguation Problems and Winograd Schema Challenge. The first consists of 60 pronoun disambiguation questions (PDP-60). The latter consists of 273 questions and is designed to work against techniques such as traditional linguistic restrictions, common heuristics or simple statistical tests . also built a Winograd Schema-like dataset but relaxed some criteria, allowing the context wording to reveal information about the correct answer. 6 We also found instances of incorrect annotation and ambiguous tests in their training and test sets (see Appendix C). In this work, therefore, we focus on the official Winograd Schema Challenge test set. Commonsense Knowledge Mining test. Following BID0 BID12, we use the same data split on the ConceptNet knowledge base, which in training, validation and test sets having sizes of 100K, 1200, and 2400 respectively. With one half of the validation and test sets being non-facts, the commonsense knowledge mining task is posed as performing classification between facts and non-facts on these sets. Another test set in included which consists of 800 instances with highest novelty measurement computed against the training set . We first train our LMs on all text corpora and test them on the two Commonsense Reasoning tests. The LMs are then finetuned for mining novel commonsense knowledge on ConceptNet. We first examine PDP-60 with unsupervised single-model resolvers by training one word-level LM on the Gutenberg corpus. In TAB0, this resolver outperforms the previous best by more than 11% in accuracy. Next, we compare against systems that make use of both supervised and unsupervised training data. As can be seen in TAB0, the single-model LM can still produce better when its competing system includes either supervised deep neural network or knowledge bases. By training more LMs for ensembling, we are able to reach 70% accuracy, outperforming the previous state-of-the-art of 66.7%. For this task, we found full scoring gives better than partial scoring. In Section 6.3, we provide more comparison between these two types of scoring. On the harder task WSC-273 where questions are designed to exclude relevant knowledge in their wording, incorporating supervised learning and knowledge base to USSM BID14 provides insignificant gain this time (+3%), compared to the large gain on PDP-60 (+19%). On the other hand, our single-model resolver can still outperform the other methods by a large margin as shown in TAB1. By ensembling predictions from multiple LMs, we obtain nearly 10% of absolute accuracy improvement compared to the previous state-of-the-art. We note that BID29 also attempted WSC but their approach is only applicable to 53 out of 273 test cases, therefore not comparable to our . Customizing training data for Winograd Schema Challenge As previous systems collect relevant data from knowledge bases after observing questions during evaluation BID24 BID29, we also explored using this option. Namely, we build a customized text corpus based on questions in commonsense reasoning tasks. It is important to note that this does not include the answers and therefore does not provide supervision to our resolvers. In particular, we aggregate documents from the CommonCrawl dataset that have the most overlapping n-grams with the questions. The score for each document is a weighted sum of F 1 (n) scores when counting overlapping n-grams: DISPLAYFORM0 The top 0.1% highest ranked documents are chosen as our new training corpus. This procedure ed in nearly 1,000,000 documents, with the highest ranking document having a score of 8×10 −2, still relatively small compared to a perfect score of 1.0. We name this dataset Stories since most of the constituent documents take the form of a story with long chain of coherent events. More statistics on Stories can be found in Appendix B.We train four different LMs on Stories and add them to the previous ensemble of 10 LMs, ing in an accuracy of 63.7% in the final system. Remarkably, single models trained on this corpus are already extremely strong, with one word-level LM achieving 62.6% accuracy. In the previous sections, we show that unsupervised LMs can outperform other methods equipped with additional knowledge bases on two Commonsense Reasoning tests. In this section, we demonstrate how these trained LMs can help expand the coverage of these human-annotated knowledge bases. To make LM scoring applicable, knowledge tuples of the form Relation(head, tail) from ConceptNet are first converted to a form that resembles natural language sentences. For example, UsedFor(post office, mail letter) is converted to "Post office is used for mail letter." by simply concatenating its head, relation, and tail phrases in order. Although this simple procedure in ungrammatical sentences, 7 we find our LMs can still adapt to this new data distribution and generalize extremely well to test instances. For fine tuning, each commonsense fact in the training set is accompanied by a negative example, generated by replacing its tail phrase by another random phrase (for example "Post office is used for dance."). Instead of adding a classification layer, we add to the original LM objective a term that encourages perplexity discrepancy between the pair of positive and negative examples. Loss new = Loss LM + max(0, log(Perp positive) − log(Perp negative) + α), where α indicates how much of a discrepancy is needed beyond which no loss is added. P erp is the perplexity evaluated on the tail phrase, given the corresponding head and relation phrases. During evaluation, a threshold is used to classify low-perplexity and high-perlexity instances as fact and non-fact. We found a word-level LM with α = 0.5 perform best on the validation set. As shown in TAB2, our fine-tuned LM outperforms other methods on both tests. Unlike DNN BID12, LM ranking is robust to the novelty-based test instances, while supervised DNN performance degrades significantly on this test despite good performance on the full test. We suggest that this happened because supervised trained DNNs tend to overfit easily when training data is limited. On the other hand, by leveraging a massive amount of unsupervised training data, LM does not overfit to the limited training data for this task (100K instances) despite its large size of approximately 2 billion parameters. In this section, we perform analysis on both correct and incorrect predictions made by LM scoring on the Winograd Schema, and the influence of training data types on test performance. We introduce a method to detect keywords from the context at which our proposed resolvers make the decision between the two candidates c correct and c incorrect. We then demonstrate that these detected keywords surprisingly match the annotated features in each Winograd Schema question that play the role of identifying the correct answer. Namely, we look at the following ratio: DISPLAYFORM0 where 1 ≤ t ≤ n for full scoring, and k + 1 ≤ t ≤ n for partial scoring. It follows that the choice between c correct or c incorrect is made by whether the value of Q = t q t is bigger than 1.0 or not. By looking at the value of each individual q t, it is possible to retrieve words with the largest values of q t and hence most responsible for the final value of Q.We visualize the probability ratios q t to have more insights into the decisions of our resolvers. Figure 2 displays a sample of incorrect decisions made by full scoring which are corrected by partial scoring. Figure 2: A sample of questions from WSC-273 predicted incorrectly by full scoring, but corrected by partial scoring. Here we mark the correct prediction by an asterisk and display the normalized probability ratioq t by coloring its corresponding word. It can be seen that the wrong predictions are made mainly due to q t at the pronoun position, where the LM has not observed the full sentence. Partial scoring shifts the attention to later words and places highest q values on the special keywords, marked by a squared bracket. These keywords characterize the Winograd Schema Challenge, as they uniquely decide the correct answer. In the last question, since the special keyword appear before the pronoun, our resolver instead chose "upset", as a reasonable switch word could be "annoying".Interestingly, we found q t with large values coincides with the special keyword of each Winograd Schema in several cases. Intuitively, this means our LMs assigned very low probability for the keyword after observing the wrong substitution. It follows that we can predict the keyword in each Winograd Schema question by selecting the word positions with the highest value of q t. For questions with keywords appearing before the reference, we detect them by backward-scoring models. Namely, we ensemble 6 LMs, each trained on one text corpus with word order reversed. Overall, we are able to discover a significant number of special keywords (115 out of 178 correctly answered questions) as shown in TAB3. This strongly indicates a correct understanding of the context and a good grasp of commonsense knowledge in the resolver's decision process. In the original proposal of the Winograd Schema Challenge, BID11 argue that by careful wording of the context, no relevant knowledge is revealed about the correct answer. For example, "big" is not a property exclusive to either "trophy" or "suitcase". This forces the system to resort to the context for correct answer, as opposed to exploiting simple statistical correlation in the training data to cheat the test. We use the trained LMs to expose such correlation in the used training data by gradually ablating the context from a WSC question. For example, at 100% ablation, the scoring reduces to comparing only "the trophy is too big" versus "the suitcase is too big". FIG0 -left shows that there is indeed such bias. For some LMs the bias made them perform worse than random at 100% ablation, while for others they perform better than random without any context. Note that this bias, however, does not necessarily affect the corresponding LM-scoring when context is included. As more context is included, all LMs improve and reach the best performance at 0% ablation, indicating the critical role of context in their scoring.6.3 SCORING TYPE AND EFFECT OF TRAINING DATA.We look at incorrect predictions from a word-level LM. With full scoring strategy, we observe that q t at the pronoun position is most responsible for a very large percentage of incorrect decisions as shown in Figure 2. For example, with the test "The trophy cannot fit in the suitcase because it is too big.", the system might return c incorrect ="suitcase" simply because c correct = "trophy" is a very rare word in its training corpus and therefore is assigned a very low probability, overpowering subsequent q t values. To verify this observation, we apply a simple fix to full scoring by normalizing its score with the unigram count of c: Score f ull normalized = Score f ull /Count(c). This normalization indeed fixes full scoring in 9 out of 10 tested LMs on PDP-60. On WSC-273, the observation is again confirmed as partial scoring, which ignores c altogether, strongly outperforms the other two scorings in all cases as shown in FIG0 -middle. We therefore attribute the different behaviour observed on PDP-60 as an atypical case due to its very small size. Next, we examine the effect of training data on commonsense reasoning test performance. An ensemble of 10 LMs is trained on each of the five corpora: LM-1-Billion, CommonCrawl, SQuAD, Gutenberg Books, and Stories. A held-out dataset from each text corpus is used for early stopping on the corresponding training data.8 FIG0 -right shows that among single training text corpora, test performance improves as the training text contains longer documents (LM-1-Billion is a set of mostly independent sentences, while Gutenberg or Stories are full books or very-long documents). Finally, the ensemble trained on a mix of different datasets perform best, highlighting the important role of diversity in training data for commonsense reasoning accuracy of the final system. We introduced a simple method to apply pretrained language models to tasks that require commonsense knowledge. Key to our method is the insight that large LMs trained on massive text corpora can capture certain aspect of human knowledge, and therefore can be used to score textual statements. On the Winograd Schema Challenge, LMs are able to achieve 11 points of accuracy above the best previously reported . On mining novel commonsense facts from ConceptNet knowledge base, LM scoring also outperforms previous methods on two different test criteria. We analyse the trained language models and observe that key features of the context that identify the correct answer are discovered and used in their predictions. Traditional approaches to capturing common sense usually involve expensive human annotation to build knowledge bases. This work demonstrates that commonsense knowledge can alternatively be learned and stored in the form of distributed representations. At the moment, we consider language modeling for learning from texts as this supplies virtually unlimited data. It remains an open question for unsupervised learning to capture commonsense from other modalities such as images or videos. The base model consists of two layers of Long-Short Term Memory (LSTM) BID6 with 8192 hidden units. The output gate of each LSTM uses peepholes and a projection layer to reduce its output dimensionality to 1024. We perform drop-out on LSTM's outputs with probability 0.25. For word inputs, we use an embedding lookup of 800000 words, each with dimension 1024. For character inputs, we use an embedding lookup of 256 characters, each with dimension 16. We concatenate all characters in each word into a tensor of shape (word length, 16) and add to its two ends the <begin of word> and <end of word> tokens. The ing concatenation is zero-padded to produce a fixed size tensor of shape. This tensor is then processed by eight different 1-D convolution (Conv) kernels of different sizes and number of output channels, listed in TAB4, each followed by a ReLU acitvation. The output of all CNNs are then concatenated and processed by two other fully-connected layers with highway connection that persist the input dimensionality. The ing tensor is projected down to a 1024-feature vector. For both word input and character input, we perform dropout on the tensors that go into LSTM layers with probability 0.25. We use a single fully-connected layer followed by a Sof tmax operator to process the LSTM's output and produce a distribution over word vocabulary of size 800K. During training, LM loss is evaluated using importance sampling with negative sample size of 8192. This loss is minimized using the AdaGrad BID5 algorithm with a learning rate of 0.2. All gradients on LSTM parameters and Character Embedding parameters are clipped by their global norm at 1.0. To avoid storing large matrices in memory, we shard them into 32 equal-sized smaller pieces. In our experiments, we used 8 different variants of this base model as listed in TAB5.In TAB6, we listed all LMs and their training text corpora used in each of the experiments in Section 5. FIG2 shows a histogram of similarity score introduced in Section 5.1. Inspecting an excerpt from the highest ranking document reveals many complex references from pronouns, within long chains of events. We hypothesize that this allows LM trained on this corpus to learn disambiguating pronouns to make correct predictions. One day when John and I had been out on somebusiness of our master's, and were returning gently on a long, straight road, at some distance we saw a boy trying to leap a pony over a gate; the pony would not take the leap, -and the boy cut him with the whip, but he only turned off on one side. He whipped him again, but the pony turned off on the other side. Then the boy got off and gave him a hard thrashing, and knocked him about the head... On a non-exhaustive inspection of the dataset constructed by BID24, we found some instances of incorrect or ambiguous annotation. 9 Below we list two cases with our comment.• A series of injections are used to battle a type of cancer in patients because they have a special type of drug which counteracts this sickness. Label: patients. Comment: Found in training set, incorrect label.• John attacked Tim because he was a communist. Label: Tim. Comment: Found in test set, there is no clear answer to this question as communists can also attack their enemy. Using the similarity scoring technique in section 5.1, we observe a large amount of low quality training text on the lower end of the ranking. Namely, these are documents whose content are mostly unintelligible or unrecognized by our vocabulary. Training LMs for commonsense reasoning tasks on full CommonCrawl, therefore, might not be ideal. On the other hand, we detected and removed a portion of PDP-122 questions presented as an extremely high ranked document.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We present evidence that LMs do capture common sense with state-of-the-art results on both Winograd Schema Challenge and Commonsense Knowledge Mining.

Represent the TLDR for finding the corresponding scientific article

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In many real-world learning scenarios, features are only acquirable at a cost constrained under a budget. In this paper, we propose a novel approach for cost-sensitive feature acquisition at the prediction-time. The suggested method acquires features incrementally based on a context-aware feature-value function. We formulate the problem in the reinforcement learning paradigm, and introduce a reward function based on the utility of each feature. Specifically, MC dropout sampling is used to measure expected variations of the model uncertainty which is used as a feature-value function. Furthermore, we suggest sharing representations between the class predictor and value function estimator networks. The suggested approach is completely online and is readily applicable to stream learning setups. The solution is evaluated on three different datasets including the well-known MNIST dataset as a benchmark as well as two cost-sensitive datasets: Yahoo Learning to Rank and a dataset in the medical domain for diabetes classification. According to the , the proposed method is able to efficiently acquire features and make accurate predictions. In traditional machine learning settings, it is usually assumed that a training dataset is freely available and the objective is to train models that generalize well. In this paradigm, the feature set is fixed, and we are dealing with complete feature vectors accompanied by class labels that are provided for training. However, in many real-world scenarios, there are certain costs for acquiring features as well as budgets limiting the total expenditure. Here, the notation of cost is more general than financial cost and it also refers to other concepts such as computational cost, privacy impacts, energy consumption, patient discomfort in medical tests, and so forth BID22. Take the example of the disease diagnosis based on medical tests. Creating a complete feature vector from all the relevant information is synonymous with conducting many tests such as MRI scan, blood test, etc. which would not be practical. On the other hand, a physician approaches the problem by asking a set of basic easy-to-acquire features, and then incrementally prescribes other tests based on the current known information (i.e., context) until a reliable diagnosis can be made. Furthermore, in many real-world use-cases, due to the volume of data or necessity of prompt decisions, learning and prediction should take place in an online and stream-based fashion. In the medical diagnosis example, it is consistent with the fact that the latency of diagnosis is vital (e.g., urgency of specific cases and diagnosis), and it is often impossible to defer the decisions. Here, by online we mean processing samples one at a time as they are being received. Various approaches were suggested in the literature for cost-sensitive feature acquisition. To begin with, traditional feature selection methods suggested to limit the set of features being used for training BID11 BID17. For instance, L1 regularization for linear classifiers in models that effectively use a subset of features BID9. Note that these methods focus on finding a fixed subset of features to be used (i.e., feature selection), while a more optimal solution would be making feature acquisition decisions based on the sample at hand and at the prediction-time. More recently, probabilistic methods were suggested that measure the value of each feature based on the current evidence BID5. However, these methods are usually applicable to Bayesian networks or similar probabilistic models and make limiting assumptions such as having binary features and binary classes. Furthermore, these probabilistic methods are computationally expensive and intractable in large scale problems BID5.Motivated by the success of discriminative learning, cascade and tree based classifiers suggested as an intuitive way to incorporate feature costs BID20 BID3. Nevertheless, these methods are basically limited to the modeling capability of tree classifiers and are limited to fixed predetermined structures. A recent work by BID27 suggested a gating method that employs adaptive linear or tree-based classifiers, alternating between low-cost models for easy-to-handle instances and higher-cost models to handle more complicated cases. While this method outperforms many of the previous work on the tree-based and cascade cost-sensitive classifiers, the low-cost model being used is limited to simple linear classifiers or pruned random forests. As an alternative approach, sensitivity analysis of trained predictors is suggested to measure the importance of each feature given a context BID7 BID18. These approaches either require an exhaustive measurement of sensitivities or rely on approximations of sensitivity. These methods are easy to use as they work without any significant modification to the predictor models being trained. However, theoretically, finding the global sensitivity is a difficult and computationally expensive problem. Therefore, frequently, approximate or local sensitivities are being used in these methods which may cause not optimal solutions. Another approach that is suggested in the literature is modeling the feature acquisition problem as a learning problem in the imitation learning BID13 or reinforcement learning BID14 BID29 BID15 domain. These approaches are promising in terms of performance and scalability. However, the value functions used in these methods are usually not intuitive and require tuning hyper-parameters to balance the cost vs. accuracy trade-off. More specifically, they often rely on one or more hyper-parameters to adjust the average cost at which these models operate. On the other hand, in many real-world scenarios it is desirable to adjust the trade-off at the prediction-time rather than the training-time. For instance, it might be desirable to spend more for a certain instance or continue the feature acquisition until a desired level of prediction confidence is achieved. This paper presents a novel method based on deep Q-networks for cost-sensitive feature acquisition. The proposed solution employs uncertainty analysis in neural network classifiers as a measure for finding the value of each feature given a context. Specifically, we use variations in the certainty of predictions as a reward function to measure the value per unit of the cost given the current context. In contrast to the recent feature acquisition methods that use reinforcement learning ideas BID14 BID29 BID15, the suggested reward function does not require any hyper-parameter tuning to balance cost versus performance trade-off. Here, features are acquired incrementally, while maintaining a certain budget or a stopping criterion. Moreover, in contrast to many other work in the literature that assume an initial complete dataset BID13 BID5 BID8 BID27, the proposed solution is stream-based and online which learns and optimizes acquisition costs during the training and the prediction. This might be beneficial as, in many real-world use cases, it might be prohibitively expensive to collect all features for all training data. Furthermore, this paper suggests a method for sharing the representations between the class predictor and action-value models that increases the training efficiency. In this paper, we consider the general scenario of having a stream of samples as input (S i). Each sample S i corresponds to a data point of a certain class in R d, where there is a cost for acquiring each feature (c j ; 1 ≤ j ≤ d). For each sample, initially, we do not know the value of any feature. Subsequently, at each time step t, we only have access to a partial realization of the feature vector denoted by x t i that consists of features that are acquired so far. There is a maximum feature acquisition budget (B) that is available for each sample. Note that the acquisition may also be terminated before reaching the maximum budget, based on any other termination condition such as reaching a certain prediction confidence. Furthermore, for each S i, there is a ground truth target labelỹ i. It is also worth noting that we consider the online stream processing task in which acquiring features is only possible for the current sample being processed. In other words, any decision should take place in an online fashion. In this setting, the goal of an Opportunistic Learning (OL) solution is to make accurate predictions for each sample by acquiring as many features as necessary. At the same time, learning should take place by updating the model while maintaining the budgets. Please note that, in this setup, we are assuming that the feature acquisition algorithm is processing a stream of input samples and there are no distinct training or test samples. However, we assume that ground truth labels are only available to us after the prediction and for a subset of samples. More formally, we define a mask vector k DISPLAYFORM0 d where each element of k indicates if the corresponding feature is available in x t i. Using this notation, the total feature acquisition cost at each time step can be represented as DISPLAYFORM1 Furthermore, we define the feature query operator (q) as DISPLAYFORM2 In Section 3, we use these primitive operations and notations for presenting the suggested solution. As prediction certainty is used extensively throughout this paper, we devote this section to certainty measurement. The softmax output layer in neural networks are traditionally used as a measure of prediction certainty. However, interpreting softmax values as probabilities is an ad hoc approach prone to errors and inaccurate certainty estimates BID30. In order to mitigate this issue, we follow the idea of Bayesian neural networks and Monte Carlo dropout (MC dropout) BID31 BID10. Here we consider the distribution of model parameters at each layer l in an L layer neural network as:ω DISPLAYFORM0 whereω l is a realization of layer parameters from the probability distribution of p(ω l). In this setting, a probability estimate conditioned on the input and stochastic model parameters is represented as: DISPLAYFORM1 where fω D is the output activation of a neural network with parametersω trained on dataset D. In order to find the uncertainty of final predictions with respect to inputs, we integrate equation 4 with respect to ω: DISPLAYFORM2 Finally, MC dropout suggests interpreting the dropout forward path evaluations as Monte Carlo samples (ω t) from the ω distribution and approximating the prediction probability as: DISPLAYFORM3 With reasonable dropout probability and number of samples, the MC dropout estimate can be considered as an accurate estimate of the prediction uncertainty. Readers are referred to BID10 for a more detailed discussion. In this paper, we denote the certainty of prediction for a given sample (Cert(x t i)) as a vector providing the probability of the sample belonging to each class in equation 6. We formulate the problem at hand as a generic reinforcement learning problem. Each episode is basically consisting of a sequence of interactions between the suggested algorithm and a single data instance (i.e., sample). At each point, the current state is defined as the current realization of the feature vector (i.e., x t i) for a given instance. At each state, the set of valid actions consists of acquiring any feature that is not acquired yet (i.e., A t i = {j = 1 . . . d|k t i,j = 0}). In this setting, each action along with the state transition as well as a reward, defined in the following, is characterizing an experience. We suggest incremental feature acquisition based on the value per unit cost of each feature. Here, the value of acquiring a feature is defined as the expected amount of change in the prediction uncertainty that acquiring the feature causes. Specifically, we define the value of each unknown feature as: DISPLAYFORM0 where r t i,j is the value of acquiring feature j for sample i at time step t. It can be interpreted as the expected change of the hypothesis due to acquiring each feature per unit of the cost. Other reinforcement learning based feature acquisition methods in the literature usually use the final prediction accuracy and feature acquisition costs as components of reward function BID14 BID29 BID15. However, the reward function of equation 7 is modeling the weighted changes of hypothesis after acquiring each feature. Consequently, it in an incremental solution which is selecting the most informative feature to be acquired at each point. As it is demonstrated in our experiments, this property is particularly beneficial when a single model is to be used under a budget determined at the prediction-time or any other, not predefined, termination condition. While it is possible to directly use the measure introduced in equation 7 to find features to be acquired at each time, it would be computationally expensive; because it requires exhaustively measuring the value function for all features at each time. Instead, in addition to a predictor model, we train an action value (i.e., feature value) function which estimates the gain of acquiring each feature based on the current context. For this purpose, we follow the idea of the deep Q-network (DQN) BID26. Briefly, DQN suggests end-to-end learning of the action-value function. It is achieved by exploring the space through taking actions using an -greedy policy, storing experiences in a replay memory, and gradually updating the value function used for exploration. Due to space limitations, readers are referred to BID26 for a more detailed discussion. FIG0 presents the network architecture of the proposed method for prediction and feature acquisition. In this architecture, a predictor network (P-Network) is trained jointly with an action value network (Q-Network). The P-Network is responsible for making prediction and consists of dropout layers that are sampled in order to find the prediction uncertainty. The Q-Network estimates the value of each unknown feature being acquired. Here, we suggest sharing the representations learned from the P-Network with the Q-Network. Specifically, the activations of each layer in the P-Network serve as input to the adjacent layers of the Q-Network (see FIG0 . Note that, in order to increase model stability during the training, we do not allow back-propagation from Q-Network outputs to P-Network weights. We also explored other architectures and sharing methods including using fully-shared layers between P-and Q-Networks that are trained jointly or only sharing the first few layers. According to our experiments, the suggested sharing method of FIG0 is reasonably efficient, while introducing a minimal burden on the prediction performance. Algorithm 1 summarizes the procedures for cost-sensitive feature acquisition and training the networks. This algorithm is designed to operate on a stream of input instances, actively acquire features, make predictions, and optimize the models. In this algorithm, if any features are available for free we include them in the initial feature vector; otherwise, we start with all features not being available initially. Here, the feature acquisition is terminated when either a maximum budget is exceeded, a user-defined stopping function decides to stop, or there is no unknown feature left to acquire. It is worth noting that, in Algorithm 1, to simplify the presentation, we assumed that ground-truth labels are available at the beginning of each episode. However, in the actual implementation, we store experiences within an episode in a temporary buffer, excluding the label. At last, after the termination of the feature acquisition procedure, a prediction is being made and upon the availability of label for that sample, the temporary experiences along with the ground-truth label are pushed to the experience replay memory. In our experiments, for the simplicity of presentation, we assume that all features are independently acquirable at a certain cost, while in many scenarios, features are bundled and acquired together (e.g., certain clinical measurements). However, it should be noted that the current formulation presented in this paper allows for having bundled feature sets. In this case, each action would be acquiring each bundle and the reward function is evaluated for the acquisition of the bundle by measuring the variations of uncertainty before and after acquiring the bundle. In this paper, PyTorch numerical computational library BID28 is used for the implementation of the proposed method. The experiments took between a few hours to a couple days on a GPU server, depending on the experiment. Here, we explored fully connected multi-layer neural network architectures; however, the approach taken can be readily applied to other neural network and deep learning architectures. We normalize features prior to our experiments statistically (µ = 0, σ = 1) and impute missing features with zeros. Note that, in our implementation, for efficiency reasons, we use NaN (not a number) values to represent features that are not available and impute them with zeros during the forward/backward computation. Cross-entropy and mean squared error (MSE) loss functions were used as the objective functions for the P and Q networks, respectively. Furthermore, the Adam optimization algorithm BID21 was used throughout this work for training the networks. We used dropout with the probability of 0.5 for all hidden layers of the P-Network and no dropout for the Q-Network. The target Q-Network was updated softly with the rate of 0.001. We update P, Q, and target Q networks every 1 + n f e 100 experiences, where n f e is the total number of features in an experiment. In addition, the replay memory size is set to store 1000 × n f e most recent experiences. The random exploration probability is decayed such that eventually it reaches the probability of 0.1. We determined these hyper-parameters using the validation set. Based on our experiments, the suggested solution is not much sensitive to these values and any reasonable setting, given enough training iterations, would in reasonable a performance. A more detailed explanation of implementation details for each specific experiment is provided in Section 4. We evaluated the proposed method on three different datasets: MNIST handwritten digits BID24, Yahoo Learning to Rank (LTRC) BID2, and a health informatics dataset. The MNIST dataset is used as it is a widely used benchmark. For this dataset, we assume equal feature acquisition cost of 1 for all features. It is worth noting that we are considering the permutation invariant setup for MNIST where each pixel is a feature discarding the spatial information. Regarding the LTRC dataset, we use feature acquisition costs provided by Yahoo! that corresponding to the computational cost of each feature. Furthermore, we evaluated our method using a real-world health dataset for diabetes classification where feature acquisition costs and budgets are natural and essential to be considered. The national health and nutrition examination survey (NAHNES) data (nha, 2018) was used for this purpose. A feature set including: (i) demographic information (age, gender, ethnicity, etc.), (ii) lab (total cholesterol, triglyceride, etc.), (iii) examination data (weight, height, etc.), and (iv) questionnaire answers (smoking, alcohol, sleep habits, etc.) is used here. An expert with experience in medical studies is asked to suggest costs for each feature based on the overall financial burden, patient privacy, and patient inconvenience. Finally, the fasting glucose values were used to define three classes: normal, pre-diabetes, and diabetes based on standard threshold values. The final dataset consists of 92062 samples of 45 features. BID6, GreedyMiser, and a recent work based on reinforcement learning (RL-Based) BID15., Cronus BID3, and Early Exit approaches. In the current study, we use reinforcement learning as an optimization algorithm, while processing data in a sequential manner. Throughout all experiments, we used fully-connected neural networks with ReLU non-linearity and dropout applied to hidden layers. We apply MC dropout sampling using 1000 evaluations of the predictor network for confidence measurements and finding the reward values. Meanwhile, 100 evaluations are used for prediction at test-time. We selected these value for our experiments as it showed stable prediction and uncertainty estimates. Each dataset was randomly splitted to 15% for test, 15% for validation, and the rest for train. During the training and validation phase, we use the random exploration mechanism. For comparison of the with other work in the literature, as they are all offline methods, the random exploration is not used during the feature acquisition. However, intuitively we believe in datasets with non-stationary distributions, it may be helpful to use random exploration as it helps to capture concept drift. Furthermore, we do model training multiple time for each experiment and average the outcomes. It is also worth noting that, as the proposed method is incremental, we continued feature acquisition until all features were acquired and reported the average accuracy corresponding to each feature acquisition budget. TAB0 presents a summary of datasets and network architectures used throughout the experiments. In this table, we report the number of hidden neurons at each network layer of the P and Q networks. For the Q-Network architecture, the number of neurons in each hidden layer is reported as the number of shared neurons from the P-Network plus the number of neurons specific to the Q-Network. FIG2 presents the accuracy versus acquisition cost curve for the MNIST dataset. Here, we compared of the proposed method (OL) with a feature acquisition method based on recurrent neural networks (RADIN) BID6, a tree-based feature acquisition method (GreedyMiser), and a recent work using reinforcement learning ideas (RL-Based) BID15. As it can be seen from this figure, our cost-sensitive feature acquisition method achieves higher accuracies at a lower cost compared to other competitors. Regarding the RL-Based method, BID15, to make a fair comparison, we used the similar network sizes and learning algorithms as with the OL method. Also, it is worth mentioning that the RL-based curve is the of training many models with different cost-accuracy trade-off hyper-parameter values, while training the OL model gives us a complete curve. Accordingly, evaluating the method of BID15 took more than 10 times compared to OL. FIG3 presents the accuracy versus acquisition cost curve for the LTRC dataset. As LTRC is a ranking dataset, in order to have a fair comparison with other work in the literature, we have used the normalized discounted cumulative gain (NDCG) BID16 performance measure. In short, NDCG is the ratio of the discounted relevance achieved using a suggested ranking method to the discounted relevance achieved using the ideal ranking. Inferring from FIG3, an exhaustive sensitivity-based method (Exhaustive) BID7, the method suggested by BID15 (RL-Based), and using gating functions and adaptively trained random forests BID27 ) (Adapt-GBRT). proposed method is able to achieve higher NDCG values using a much lower acquisition budget compared to tree-based approaches in the literature including CSTC, Cronus BID3, and Early Exit . FIG4 shows a visualization of the OL feature acquisition on the diabetes dataset. In this figure, the y-axis corresponds to 50 random test samples and the x-axis corresponds to each feature. Here, warmer colors represent features that were acquired with more priority and colder colors represent less acquisition priority. It can be observed from this figure that OL acquires features based on the available context rather than having a static feature importance and ordering. It can also be seen that OL gives more priority to less costly and yet informative features such as demographics and examinations. Furthermore, FIG4 demonstrates the accuracy versus acquisition cost for the diabetes classification. As it can be observed from this figure, OL achieves a superior accuracy with a lower acquisition cost compared to other approaches. Here, we used the exhaustive feature query method as suggested by BID7 using sensitivity as the utility function, the method suggested by BID15 (RL-Based), as well a recent paper using gating functions and adaptively trained random forests BID27 ) (Adapt-GBRT). In this section we show the effectiveness of three ideas suggested by this paper i.e, using model uncertainty as a feature-value measure, representation sharing between the P and Q networks, and using MC-dropout as a measure of prediction uncertainty. Additionally, we study the influence of the available budget on the performance of the algorithm. In these experiments, we used the diabetes dataset. A comparison between the suggested feature-value function (OL) in this paper with a traditional feature-value function (RL-Based) was presented in FIG2 and FIG4. We implemented the RL-Based method such that it is using a similar architecture and learning algorithm as the OL, while the reward function is simply the the negative of feature costs for acquiring each feature and a positive value for making correct predictions. As it can be seen from the comparison of these approaches, the reward function suggested in this paper in a more efficient feature acquisition. In order to demonstrate the importance of MC-dropout, we measured the average of accuracy at each certainty value. Statistically, confidence values indicate the average accuracy of predictions BID12. For instance, if we measure the certainty of prediction for a group of samples to be 90%, we expect to correctly classify samples of that group 90% of the time. Figure 5 shows the average prediction accuracy versus the certainty of samples reported using the MC-dropout method (using 1000 samples) and directly using the softmax output values. As it can be inferred from this figure, MC-dropout estimates are highly accurate, while softmax estimates are mostly over-confident and inaccurate. Note that the accuracy of certainty estimates are crucially important to us as any inaccuracy in these values in having inaccurate reward values. Figure 5b shows the accuracy versus cost curves that the suggested architecture achieves using the accurate MC-dropout certainty estimates and using the inaccurate softmax estimates. It can be seen from this figure that more accurate MC-dropout estimates are essential. Figure 6 demonstrates the speed of convergence using the suggested sharing between the P and Q networks (W/ Sharing) as well as not using the sharing architecture (W/O Sharing). Here, we use the normalized area under the accuracy-cost curve (AUACC) as measure of acquisition performance at each episode. Please note that we adjust the number of hidden neurons such that the number of Q-Network parameters is the same for each corresponding layer between the two cases. As it can be seen from this figure, the suggested representation sharing between the P and Q networks increases the speed of convergence. Figure 7 shows the performance of the OL method having various limited budgets during the operation. Here, we report the accuracy-cost curves for 25%, 50%, 75%, and 100% of the budget required to acquire all features. As it can be inferred from this figure, the suggested method is able to efficiently operate at different enforced budget constraints. Figure 8a and 8b demonstrate the validation accuracy and AUACC values measured during the processing of the data stream at each episode for the MNIST and Diabetes datasets, respectively. As it can be seen from this figure, as the algorithm observes more data samples, it achieves higher validation accuracy/AUACC values, and it eventually converges after a certain number of episodes. It should be noted that, in general, convergence in reinforcement learning setups is dependent on the training algorithm and parameters used. For instance, the random exploration strategy, the update condition, and the update strategy for the target Q network would influence the overall time behavior of the algorithm. In this paper, we use conservative and reasonable strategies as reported in Section 3.2 that in stable across a wide range of experiments. In this paper, we proposed an approach for cost-sensitive learning in stream-based settings. We demonstrated that certainty estimation in neural network classifiers can be used as a viable measure for the value of features. Specifically, variations of the model certainty per unit of the cost is used as measure of feature value. In this paradigm, a reinforcement learning solution is suggested which is efficient to train using a shared representation. The introduced method is evaluated on three different real-world datasets representing different applications: MNIST digits recognition, Yahoo LTRC web ranking dataset, and diabetes prediction using health records. Based on the , the suggested method is able to learn from data streams, make accurate predictions, and effectively reduce the prediction-time feature acquisition cost.

represent the tldr for finding the corresponding scientific article

An online algorithm for cost-aware feature acquisition and prediction

The paper designs two algorithms for the stochastic AUC maximization problem with state-of-the-art complexities when using deep neural network as predictive model, which are also verified by empirical studies.