Question: Represent text

Question: A reinforcement learning agent that needs to pursue different goals across episodes requires a goal-conditional policy. In addition to their potential to generalize desirable behavior to unseen goals, such policies may also enable higher-level planning based on subgoals. In sparse-reward environments, the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended appears crucial to enable sample efficient learning. However, reinforcement learning agents have only recently been endowed with such capacity for hindsight. In this paper, we demonstrate how hindsight can be introduced to policy gradient methods, generalizing this idea to a broad class of successful algorithms. Our experiments on a diverse selection of sparse-reward environments show that hindsight leads to a remarkable increase in sample efficiency. In a traditional reinforcement learning setting, an agent interacts with an environment in a sequence of episodes, observing states and acting according to a policy that ideally maximizes expected cumulative reward. If an agent is required to pursue different goals across episodes, its goal-conditional policy may be represented by a probability distribution over actions for every combination of state and goal. This distinction between states and goals is particularly useful when the probability of a state transition given an action is independent of the goal pursued by the agent. Learning such goal-conditional behavior has received significant attention in machine learning and robotics, especially because a goal-conditional policy may generalize desirable behavior to goals that were never encountered by the agent BID17 BID3;; BID16 BID29;;; BID11. Consequently, developing goal-based curricula to facilitate learning has also attracted considerable interest (; ; BID20 BID19 . In hierarchical reinforcement learning, goal-conditional policies may enable agents to plan using subgoals, which abstracts the details involved in lower-level decisions BID10 BID26 ;).In a typical sparse-reward environment, an agent receives a non-zero reward only upon reaching a goal state. Besides being natural, this task formulation avoids the potentially difficult problem of reward shaping, which often biases the learning process towards suboptimal behavior BID9. Unfortunately, sparse-reward environments remain particularly challenging for traditional reinforcement learning algorithms BID0 ). For example, consider an agent tasked with traveling between cities. In a sparse-reward formulation, if reaching a desired destination by chance is unlikely, a learning agent will rarely obtain reward signals. At the same time, it seems natural to expect that an agent will learn how to reach the cities it visited regardless of its desired destinations. In this context, the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended is called hindsight. This capacity was recently introduced by BID0 to off-policy reinforcement learning algorithms that rely on experience replay . In earlier work, introduced hindsight to policy search based on Bayesian optimization BID5.In this paper, we demonstrate how hindsight can be introduced to policy gradient methods BID27 BID28 BID22, generalizing this idea to a successful class of reinforcement learning algorithms BID13 ).In contrast to previous work on hindsight, our approach relies on importance sampling BID2. In reinforcement learning, importance sampling has been traditionally employed in order to efficiently reuse information obtained by earlier policies during learning BID15 BID12; BID7. In comparison, our approach attempts to efficiently learn about different goals using information obtained by the current policy for a specific goal. This approach leads to multiple formulations of a hindsight policy gradient that relate to well-known policy gradient . In comparison to conventional (goal-conditional) policy gradient estimators, our proposed estimators lead to remarkable sample efficiency on a diverse selection of sparse-reward environments. We denote random variables by upper case letters and assignments to these variables by corresponding lower case letters. We let Val(X) denote the set of valid assignments to a random variable X. We also omit the subscript that typically relates a probability function to random variables when there is no risk of ambiguity. For instance, we may use p(x) to denote p X (x) and p(y) to denote p Y (y).Consider an agent that interacts with its environment in a sequence of episodes, each of which lasts for exactly T time steps. The agent receives a goal g ∈ Val(G) at the beginning of each episode. At every time step t, the agent observes a state s t ∈ Val(S t), receives a reward r(s t, g) ∈ R, and chooses an action a t ∈ Val(A t). For simplicity of notation, suppose that Val(G), Val(S t), and Val(A t) are finite for every t. In our setting, a goal-conditional policy defines a probability distribution over actions for every combination of state and goal. The same policy is used to make decisions at every time step. Let τ = s 1, a 1, s 2, a 2,..., s T −1, a T −1, s T denote a trajectory. We assume that the probability p(τ | g, θ) of trajectory τ given goal g and a policy parameterized by θ ∈ Val(Θ) is given by p(τ | g, θ) = p(s 1) T −1 t=1 p(a t | s t, g, θ)p(s t+1 | s t, a t).(In contrast to a Markov decision process, this formulation allows the probability of a state transition given an action to change across time steps within an episode. More importantly, it implicitly states that the probability of a state transition given an action is independent of the goal pursued by the agent, which we denote by S t+1 ⊥ ⊥ G | S t, A t . For every τ, g, and θ, we also assume that p(τ | g, θ) is non-zero and differentiable with respect to θ. Assuming that G ⊥ ⊥ Θ, the expected return η(θ) of a policy parameterized by θ is given by DISPLAYFORM0 T t=1 r(s t, g).The action-value function is given by Q This section presents for goal-conditional policies that are analogous to well-known for conventional policies BID13. They establish the foundation for the presented in the next section. The corresponding proofs are included in Appendix A for completeness. The objective of policy gradient methods is finding policy parameters that achieve maximum expected return. When combined with Monte Carlo techniques BID2, the following allows pursuing this objective using gradient-based optimization. Theorem 3.1 (Goal-conditional policy gradient). The gradient ∇η(θ) of the expected return with respect to θ is given by DISPLAYFORM0 The following allows employing a baseline to reduce the variance of the gradient estimator. Theorem 3.2 (Goal-conditional policy gradient, baseline formulation). For every t, θ, and associated real-valued (baseline) function b θ t, the gradient ∇η(θ) of the expected return with respect to θ is given by DISPLAYFORM1 Appendix A.7 presents the constant baselines that minimize the (elementwise) variance of the corresponding estimator. However, such baselines are usually impractical to compute (or estimate), and the variance of the estimator may be reduced further by a baseline function that depends on state and goal. Although generally suboptimal, it is typical to let the baseline function b θ t approximate the value function V θ t . Lastly, actor-critic methods may rely on the following for goal-conditional policies. Theorem 3.3 (Goal-conditional policy gradient, advantage formulation). The gradient ∇η(θ) of the expected return with respect to θ is given by DISPLAYFORM2 This section presents the novel ideas that introduce hindsight to policy gradient methods. The corresponding proofs can be found in Appendix B.Suppose that the reward r(s, g) is known for every combination of state s and goal g, as in previous work on hindsight BID0 ). In that case, it is possible to evaluate a trajectory obtained while trying to achieve an original goal g for an alternative goal g. Using importance sampling, this information can be exploited using the following central . Theorem 4.1 (Every-decision hindsight policy gradient). For an arbitrary (original) goal g, the gradient ∇η(θ) of the expected return with respect to θ is given by DISPLAYFORM0 In the formulation presented above, every reward is multiplied by the ratio between the likelihood of the corresponding trajectory under an alternative goal and the likelihood under the original goal (see Eq. 1). Intuitively, every reward should instead be multiplied by a likelihood ratio that only considers the corresponding trajectory up to the previous action. This intuition underlies the following important , named after an analogous for action-value functions by BID15.Theorem 4.2 (Per-decision hindsight policy gradient). For an arbitrary (original) goal g, the gradient ∇η(θ) of the expected return with respect to θ is given by DISPLAYFORM1 This section details gradient estimation based on the presented in the previous section. The corresponding proofs can be found in Appendix C. DISPLAYFORM0 where each trajectory τ (i) is obtained using a policy parameterized by θ in an attempt to achieve a goal g (i) chosen by the environment. The following points to a straightforward estimator based on Theorem 4.2. Theorem 5.1. The per-decision hindsight policy gradient estimator, given by DISPLAYFORM1 is a consistent and unbiased estimator of the gradient ∇η(θ) of the expected return. In preliminary experiments, we found that this estimator leads to unstable learning progress, which is probably due to its potential high variance. The following , inspired by weighted importance sampling BID2, represents our attempt to trade variance for bias. Theorem 5.2. The weighted per-decision hindsight policy gradient estimator, given by DISPLAYFORM2 is a consistent estimator of the gradient ∇η(θ) of the expected return. In simple terms, the likelihood ratio for every combination of trajectory, (alternative) goal, and time step is normalized across trajectories by this estimator. In Appendix C.3, we present a that enables the corresponding consistency-preserving weighted baseline. t, g) = 0} composed of so-called active goals during the i-th episode. The feasibility of the proposed estimators relies on the fact that only active goals correspond to non-zero terms inside the expectation over goals in Expressions 11 and 12. In many natural sparse-reward environments, active goals will correspond directly to states visited during episodes (for instance, the cities visited while trying to reach other cities), which enables computing said expectation exactly when the goal distribution is known. The proposed estimators have remarkable properties that differentiate them from previous (weighted) importance sampling estimators for off-policy learning. For instance, although a trajectory is often more likely under the original goal than under an alternative goal, in policies with strong optimal substructure, a high probability of a trajectory between the state a and the goal (state) c that goes through the state b may naturally allow for a high probability of the corresponding (sub)trajectory between the state a and the goal (state) b. In other cases, the (unnormalized) likelihood ratios may become very small for some (alternative) goals after a few time steps across all trajectories. After normalization, in the worst case, this may even lead to equivalent ratios for such goals for a given time step across all trajectories. In any case, it is important to note that only likelihood ratios associated to active goals for a given episode will affect the gradient estimate. Additionally, an original goal will always have (unnormalized) likelihood ratios equal to one for the corresponding episode. Under mild additional assumptions, the proposed estimators also allow using a dataset containing goals chosen arbitrarily (instead of goals drawn from the goal distribution). Although this feature is not required by our experiments, we believe that it may be useful to circumvent catastrophic forgetting during curriculum learning BID4 ). This section reports of an empirical comparison between goal-conditional policy gradient estimators and hindsight policy gradient estimators.1 Because there are no well-established sparsereward environments intended to test agents under multiple goals, this comparison focuses on our own selection of environments. These environments are diverse in terms of stochasticity, state space dimensionality and size, relationship between goals and states, and number of actions. In every one of these environments, the agent receives the remaining number of time steps plus one as a reward for reaching the goal state, which also ends the episode. In every other situation, the agent receives no reward. Importantly, the weighted per-decision hindsight policy gradient estimator used in our experiments (HPG) does not precisely correspond to Expression 12. Firstly, the original estimator requires a constant number of time steps T, which would often require the agent to act after the end of an episode in the environments that we consider. Secondly, although it is feasible to compute Expression 12 exactly when the goal distribution is known (as explained in Sec. 5), we sometimes subsample the sets of active goals per episode. Furthermore, when including a baseline that approximates the value function, we again consider only active goals, which by itself generally in an inconsistent estimator (HPG+B). As will become evident in the following sections, these compromised estimators still lead to remarkable sample efficiency. We assess sample efficiency through learning curves and average performance scores, which are obtained as follows. After collecting a number of batches (composed of trajectories and goals), each of which enables one step of gradient ascent, an agent undergoes evaluation. During evaluation, the agent interacts with the environment for a number of episodes, selecting actions with maximum probability according to its policy. A learning curve shows the average return obtained during each evaluation step, averaged across multiple runs (independent learning procedures). The curves presented in this text also include a 95% bootstrapped confidence interval. The average performance is given by the average return across evaluation steps, averaged across runs. During both training and evaluation, goals are drawn uniformly at random. Note that there is no held-out set of goals for evaluation, since we are interested in evaluating sample efficiency instead of generalization. For every combination of environment and batch size, grid search is used to select hyperparameters for each estimator according to average performance scores (after the corresponding standard deviation across runs is subtracted, as suggested by). Definitive are obtained by using the best hyperparameters found for each estimator in additional runs. In this section, we discuss definitive for small and medium batch sizes. More details about our experiments can be found in Appendices E.1 and E.2. Appendix E.3 contains unabridged , a supplementary empirical study of likelihood ratios (Appendix E.3.6), and an empirical comparison with hindsight experience replay (Appendix E.3.7). In a bit flipping environment, the agent starts every episode in the same state (0, represented by k bits), and its goal is to reach a randomly chosen state. The actions allow the agent to toggle (flip) each bit individually. The maximum number of time steps is k + 1. Despite its apparent simplicity, this environment is an ideal testbed for reinforcement learning algorithms intended to deal with sparse rewards, since obtaining a reward by chance is unlikely even for a relatively small k. BID0 employed a similar environment to evaluate their hindsight approach. Figure 4 presents the learning curves for k = 8. Goal-conditional policy gradient estimators with and without an approximate value function baseline (GCPG+B and GCPG, respectively) obtain excellent policies and lead to comparable sample efficiency. HPG+B obtains excellent policies more than 400 batches earlier than these estimators, but its policies degrade upon additional training. Additional experiments strongly suggest that the main cause of this issue is the fact that the value function baseline is still very poorly fit by the time that the policy exhibits desirable behavior. In comparison, HPG obtains excellent policies as early as HPG+B, but its policies remain remarkably stable upon additional training. The learning curves for k = 16 are presented in Figure 5. Clearly, both GCPG and GCPG+B are unable to obtain policies that perform better than chance, which is explained by the fact that they rarely incorporate reward signals during training. Confirming the importance of hindsight, HPG leads to stable and sample efficient learning. Although HPG+B also obtains excellent policies, they deteriorate upon additional training. Similar can be observed for a small batch size (see App. E.3.3). The average performance documented in Appendix E.3.5 confirm that HPG leads to remarkable sample efficiency. Importantly, Appendices E.3.1 and E.3.2 present hyperparameter sensitivity graphs suggesting that HPG is less sensitive to hyperparameter settings than the other estimators. The same two appendices also document an ablation study where the likelihood ratios are removed from HPG, which notably promotes increased hyperparameter sensitivity. This study confirms the usefulness of the correction prescribed by importance sampling. In the grid world environments that we consider, the agent starts every episode in a (possibly random) position on an 11 × 11 grid, and its goal is to reach a randomly chosen (non-initial) position. Some of the positions on the grid may contain impassable obstacles (walls). The actions allow the agent to move in the four cardinal directions. Moving towards walls causes the agent to remain in its current position. A state or goal is represented by a pair of integers between 0 and 10. The maximum number of time steps is 32. In the empty room environment, the agent starts every episode in the upper left corner of the grid, and there are no walls. In the four rooms environment BID23, the agent starts every episode in one of the four corners of the grid (see FIG0). There are walls that partition the grid into four rooms, such that each room provides access to two other rooms through single openings (doors). With probability 0.2, the action chosen by the agent is ignored and replaced by a random action. Figure 6 shows the learning curves for the empty room environment. Clearly, every estimator obtains excellent policies, although HPG and HPG+B improve sample efficiency by at least 200 batches. The learning curves for the four rooms environment are presented in Figure 7. In this surprisingly challenging environment, every estimator obtains unsatisfactory policies. However, it is still clear that HPG and HPG+B improve sample efficiency. In contrast to the experiments presented in the previous section, HPG+B does not give rise to instability, which we attribute to easier value function estimation. Similar can be observed for a small batch size (see App. E.3.3). HPG achieves the best average performance in every grid world experiment except for a single case, where the best average performance is achieved by HPG+B (see App. E.3.5). The hyperparameter sensitivity graphs presented in Appendices E.3.1 and E.3.2 once again suggest that HPG is less sensitive to hyperparameter choices, and that ignoring likelihood ratios promotes increased sensitivity (at least in the four rooms environment). The Ms. Pac-man environment is a variant of the homonymous game for ATARI 2600 (see FIG1). The agent starts every episode close to the center of the map, and its goal is to reach a randomly chosen (non-initial) position on a 14 × 19 grid defined on the game screen. The actions allow the agent to move in the four cardinal directions for 13 game ticks. A state is represented by the of preprocessing a sequence of game screens (images) as described in Appendix E.1. A goal is represented by a pair of integers. The maximum number of time steps is 28, although an episode will also end if the agent is captured by an enemy. In comparison to the grid world environments considered in the previous section, this environment is additionally challenging due to its high-dimensional states and the presence of enemies. Figure 8 presents the learning curves for a medium batch size. Approximate value function baselines are excluded from this experiment due to the significant cost of systematic hyperparameter search. Although HPG obtains better policies during early training, GCPG obtains better final policies. However, for such a medium batch size, only 3 active goals per episode (out of potentially 28) are subsampled for HPG. Although this harsh subsampling brings computational efficiency, it also appears to handicap the estimator. This hypothesis is supported by the fact that HPG outperforms GCPG for a small batch size, when all active goals are used (see Apps. E.3.3 and E.3.5). Policies obtained using each estimator are illustrated by videos included on the project website. The FetchPush environment is a variant of the environment recently proposed by BID14 to assess goal-conditional policy learning algorithms in a challenging task of practical interest (see FIG2). In a simulation, a robotic arm with seven degrees of freedom is required to push a randomly placed object (block) towards a randomly chosen position. The arm starts every episode in the same configuration. In contrast to the original environment, the actions in our variant allow increasing the desired velocity of the gripper along each of two orthogonal directions by ±0.1 or ±1, leading to a total of eight actions. A state is represented by a 28-dimensional real vector that contains the following information: positions of the gripper and block; rotational and positional velocities of the gripper and block; relative position of the block with respect to the gripper; state of the gripper; and current desired velocity of the gripper along each direction. A goal is represented by three coordinates. The maximum number of time steps is 50. Figure 9 presents the learning curves for a medium batch size. HPG obtains good policies after a reasonable number of batches, in sharp contrast to GCPG. For such a medium batch size, only 3 active goals per episode (out of potentially 50) are subsampled for HPG, showing that subsampling is a viable alternative to reduce the computational cost of hindsight. Similar are observed for a small batch size, when all active goals are used (see Apps. E.3.3 and E.3.5). Policies obtained using each estimator are illustrated by videos included on the project website. We introduced techniques that enable learning goal-conditional policies using hindsight. In this context, hindsight refers to the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended. Prior to our work, hindsight has been limited to off-policy reinforcement learning algorithms that rely on experience replay BID0 and policy search based on Bayesian optimization .In addition to the fundamental hindsight policy gradient, our technical include its baseline and advantage formulations. These are based on a self-contained goal-conditional policy framework that is also introduced in this text. Besides the straightforward estimator built upon the per-decision hindsight policy gradient, we also presented a consistent estimator inspired by weighted importance sampling, together with the corresponding baseline formulation. A variant of this estimator leads to remarkable comparative sample efficiency on a diverse selection of sparsereward environments, especially in cases where direct reward signals are extremely difficult to obtain. This crucial feature allows natural task formulations that require just trivial reward shaping. The main drawback of hindsight policy gradient estimators appears to be their computational cost, which is directly related to the number of active goals in a batch. This issue may be mitigated by subsampling active goals, which generally leads to inconsistent estimators. Fortunately, our experiments suggest that this is a viable alternative. Note that the success of hindsight experience replay also depends on an active goal subsampling heuristic (, Sec. 4.5).The inconsistent hindsight policy gradient estimator with a value function baseline employed in our experiments sometimes leads to unstable learning, which is likely related to the difficulty of fitting such a value function without hindsight. This hypothesis is consistent with the fact that such instability is observed only in the most extreme examples of sparse-reward environments. Although our preliminary experiments in using hindsight to fit a value function baseline have been successful, this may be accomplished in several ways, and requires a careful study of its own. Further experiments are also required to evaluate hindsight on dense-reward environments. There are many possibilities for future work besides integrating hindsight policy gradients into systems that rely on goal-conditional policies: deriving additional estimators; implementing and evaluating hindsight (advantage) actor-critic methods; assessing whether hindsight policy gradients can successfully circumvent catastrophic forgetting during curriculum learning of goal-conditional policies; approximating the reward function to reduce required supervision; analysing the variance of the proposed estimators; studying the impact of active goal subsampling; and evaluating every technique on continuous action spaces. Theorem A.1. The gradient ∇η(θ) of the expected return with respect to θ is given by DISPLAYFORM0 Proof. The partial derivative ∂η(θ)/∂θ j of the expected return η(θ) with respect to θ j is given by DISPLAYFORM1 The likelihood-ratio trick allows rewriting the previous equation as DISPLAYFORM2 Note that DISPLAYFORM3 Therefore, DISPLAYFORM4 A.2 THEOREM 3.1Theorem 3.1 (Goal-conditional policy gradient). The gradient ∇η(θ) of the expected return with respect to θ is given by DISPLAYFORM5 Proof. Starting from Eq. 17, the partial derivative ∂η(θ)/∂θ j of η(θ) with respect to θ j is given by DISPLAYFORM6 The previous equation can be rewritten as DISPLAYFORM7 Let c denote an expectation inside Eq. 19 for t ≥ t. In that case, A t ⊥ ⊥ S t | S t, G, Θ, and so DISPLAYFORM8 Reversing the likelihood-ratio trick, DISPLAYFORM9 Therefore, the terms where t ≥ t can be dismissed from Eq. 19, leading to DISPLAYFORM10 The previous equation can be conveniently rewritten as DISPLAYFORM11 A.3 LEMMA A.1Lemma A.1. For every j, t, θ, and associated real-valued (baseline) function b DISPLAYFORM12 Proof. Letting c denote an expectation inside Eq. 24, DISPLAYFORM13 Reversing the likelihood-ratio trick, DISPLAYFORM14 A.4 THEOREM 3.2 Theorem 3.2 (Goal-conditional policy gradient, baseline formulation). For every t, θ, and associated real-valued (baseline) function b θ t, the gradient ∇η(θ) of the expected return with respect to θ is given by DISPLAYFORM15 Proof. The is obtained by subtracting Eq. 24 from Eq. 23. Importantly, for every combination of θ and t, it would also be possible to have a distinct baseline function for each parameter in θ. A.5 LEMMA A.2 Lemma A.2. The gradient ∇η(θ) of the expected return with respect to θ is given by DISPLAYFORM16 Proof. Starting from Eq. 23 and rearranging terms, DISPLAYFORM17 By the definition of action-value function, DISPLAYFORM18 A.6 THEOREM 3.3Theorem 3.3 (Goal-conditional policy gradient, advantage formulation). The gradient ∇η(θ) of the expected return with respect to θ is given by DISPLAYFORM19 Proof. The is obtained by choosing b θ t = V θ t and subtracting Eq. 24 from Eq. 29.A.7 THEOREM A.2For arbitrary j and θ, consider the following definitions of f and h. DISPLAYFORM20 DISPLAYFORM21 For every b j ∈ R, using Theorem 3.1 and the fact that DISPLAYFORM22 Proof. The is an application of Lemma D.4. The following theorem relies on importance sampling, a traditional technique used to obtain estimates related to a random variable X ∼ p using samples from an arbitrary positive distribution q. This technique relies on the following equalities: Theorem 4.1 (Every-decision hindsight policy gradient). For an arbitrary (original) goal g, the gradient ∇η(θ) of the expected return with respect to θ is given by DISPLAYFORM0 Proof. Starting from Theorem 3.1, importance sampling allows rewriting the partial derivative ∂η(θ)/∂θ j as DISPLAYFORM1 B.2 THEOREM 4.2 Theorem 4.2 (Per-decision hindsight policy gradient). For an arbitrary (original) goal g, the gradient ∇η(θ) of the expected return with respect to θ is given by DISPLAYFORM2 Proof. Starting from Eq. 36, the partial derivative ∂η(θ)/∂θ j can be rewritten as DISPLAYFORM3 If we split every trajectory into states and actions before and after t, then ∂η(θ)/∂θ j is given by g p(g) DISPLAYFORM4 where z is defined by DISPLAYFORM5 Using Lemma D.2 and canceling terms, DISPLAYFORM0 Using Lemma D.2 once again, DISPLAYFORM1 Using the fact that DISPLAYFORM2 Substituting z into Expression 38 and returning to an expectation over trajectories, DISPLAYFORM3 B.3 LEMMA 4.1Lemma 4.1. For every g, t, θ, and associated real-valued (baseline) function b DISPLAYFORM4 Proof. Let c denote the j-th element of the vector in the left-hand side of Eq. 8, such that DISPLAYFORM5 Using Lemma D.1 and writing the expectations explicitly, DISPLAYFORM6 Canceling terms, using Lemma D.1 once again, and reversing the likelihood-ratio trick, DISPLAYFORM7 Pushing constants outside the summation over actions at time step t, DISPLAYFORM8 Theorem B.1 (Hindsight policy gradient, baseline formulation). For every g, t, θ, and associated real-valued (baseline) function b θ t, the gradient ∇η(θ) of the expected return with respect to θ is given by DISPLAYFORM9 where DISPLAYFORM10 Proof. The is obtained by subtracting Eq. 8 from Eq. 7. Importantly, for every combination of θ and t, it would also be possible to have a distinct baseline function for each parameter in θ. Lemma B.1 (Hindsight policy gradient, action-value formulation). For an arbitrary goal g, the gradient ∇η(θ) of the expected return with respect to θ is given by DISPLAYFORM0 Proof. Starting from Eq. 29, the partial derivative ∂η(θ)/∂θ j can be written as DISPLAYFORM1 Using importance sampling, for an arbitrary goal g, DISPLAYFORM2 Using Lemma D.1 and rewriting the previous equation using expectations, DISPLAYFORM3 B.6 THEOREM 4.3 Theorem 4.3 (Hindsight policy gradient, advantage formulation). For an arbitrary (original) goal g, the gradient ∇η(θ) of the expected return with respect to θ is given by DISPLAYFORM4 Proof. The is obtained by choosing b θ t = V θ t and subtracting Eq. 44 from Eq. 53. For arbitrary g, j, and θ, consider the following definitions of f and h. DISPLAYFORM0 DISPLAYFORM1 For every b j ∈ R, using Theorem 4.2 and the fact that E [h(T) | g, θ] = 0 by Lemma 4.1, DISPLAYFORM2 is given by DISPLAYFORM3.Proof. The is an application of Lemma D.4. This appendix contains proofs related to the estimators presented in Section 5: Theorem 5.1 (App. C.1) and Theorem 5.2 (App. C.2). Appendix C.3 presents a that enables a consistency-preserving weighted baseline. In this appendix, we will consider a dataset DISPLAYFORM0 where each trajectory τ (i) is obtained using a policy parameterized by θ in an attempt to achieve a goal g (i) chosen by the environment. Because D is an iid dataset given Θ, DISPLAYFORM1 C.1 THEOREM 5.1Theorem 5.1. The per-decision hindsight policy gradient estimator, given by DISPLAYFORM2 is a consistent and unbiased estimator of the gradient ∇η(θ) of the expected return. Proof. Let I (N) j denote the j-th element of the estimator, which can be written as DISPLAYFORM3 where DISPLAYFORM4 Using Theorem 4.2, the expected value E I (N) j | θ is given by DISPLAYFORM5 Therefore, I(N) j is an unbiased estimator of ∂η(θ)/∂θ j.Conditionally on Θ, the random variable DISPLAYFORM6 is an average of iid random variables with expected value ∂η(θ)/∂θ j (see Eq. 61). By the strong law of large numbers BID18, Theorem 2.3.13), DISPLAYFORM7 Therefore, I(N) j is a consistent estimator of ∂η(θ)/∂θ j. Theorem 5.2. The weighted per-decision hindsight policy gradient estimator, given by DISPLAYFORM0, FORMULA2 is a consistent estimator of the gradient ∇η(θ) of the expected return. Proof. Let W (N) j denote the j-th element of the estimator, which can be written as DISPLAYFORM1 where X(g, t, t) DISPLAYFORM2 DISPLAYFORM3 DISPLAYFORM4 DISPLAYFORM5 Consider the expected value DISPLAYFORM6 Using the fact that t > t, Lemma D.1, and canceling terms, E Xi can be written as DISPLAYFORM0 Because S t ⊥ ⊥ G | S 1:t −1, A 1:t −1, Θ, DISPLAYFORM1 Conditionally on Θ, the variable X(g, t, t) DISPLAYFORM2 is an average of iid random variables with expected value E Xi. By the strong law of large numbers BID18, Theorem 2.3.13), X(g, t, t) DISPLAYFORM3 Conditionally on Θ, the variable Y (g, t, t) DISPLAYFORM4 is an average of iid random variables with expected value 1. By the strong law of large numbers, Y (g, t, t) (N) j a.s.− − → 1. and Y (g, t, t) (N) j converge almost surely to real numbers (, Ch. 3, Property 2), DISPLAYFORM0 By Theorem 3.1 and the fact that W (N) j is a linear combination of terms X(g, t, t) DISPLAYFORM1 C.3 THEOREM C.1Theorem C.1. The weighted baseline estimator, given by DISPLAYFORM2, converges almost surely to zero. Proof. Let B (N) j denote the j-th element of the estimator, which can be written as DISPLAYFORM3 where DISPLAYFORM4 DISPLAYFORM5 DISPLAYFORM6 DISPLAYFORM7 Using Eqs. 44 and 47, the expected value DISPLAYFORM8 Conditionally on Θ, the variable X(g, t) DISPLAYFORM9 is an average of iid random variables with expected value zero. By the strong law of large numbers BID18, Theorem 2.3.13), X(g, t) DISPLAYFORM10 The fact that Y (g, t)(N) j a.s.− − → 1 is already established in the proof of Theorem 5.2. Because both DISPLAYFORM11 and Y (g, t)(N) j converge almost surely to real numbers (, Ch. 3, Property 2), DISPLAYFORM12 Because B(N) j is a linear combination of terms X(g, t) DISPLAYFORM13 is a consistent estimator of a some quantity given θ, then so is E (N) − B DISPLAYFORM14 Proof. In order to employ backward induction, consider the case t = T − 1. By marginalization, DISPLAYFORM15 which completes the proof of the base case. Assuming the inductive hypothesis is true for a given 2 ≤ t ≤ T − 1 and considering the case t − 1, DISPLAYFORM16 Lemma D.2. For every τ, g, θ, and 1 ≤ t ≤ T, DISPLAYFORM17 Proof. The case t = 1 can be inspected easily. Consider 2 ≤ t ≤ T. By definition, DISPLAYFORM18 Using Lemma D.1, DISPLAYFORM19 DISPLAYFORM20 DISPLAYFORM21 Proof. From the definition of action-value function and using the fact that DISPLAYFORM22 Let z denote the second term in the right-hand side of the previous equation, which can also be written as DISPLAYFORM23 Consider the following three independence properties: DISPLAYFORM24 DISPLAYFORM25 DISPLAYFORM26 Together, these properties can be used to demonstrate that DISPLAYFORM27 From the definition of value function, DISPLAYFORM28 Theorem 4.4. For every t and θ, the advantage function DISPLAYFORM29 Proof. The follows from the definition of advantage function and Lemma D.3. DISPLAYFORM30 Consider a discrete random variable X and real-valued functions f and h. Suppose also that DISPLAYFORM31 Proof. Let v = Var [f (X) − bh(X)]. Using our assumptions and the definition of variance, DISPLAYFORM32 The first and second derivatives of v with respect to b are given by dv DISPLAYFORM33 Our assumptions guarantee that E h(X) 2 > 0. Therefore, by Fermat's theorem, if b is a local minimum, then dv/db = 0, leading to the desired equality. By the second derivative test, b must be a local minimum. This appendix contains additional information about the experiments introduced in Section 6. Appendix E.1 details policy and baseline representations. Appendix E.2 documents experimental settings. Appendix E.3 presents unabridged . In every experiment, a policy is represented by a feedforward neural network with a softmax output layer. The input to such a policy is a pair composed of state and goal. A baseline function is represented by a feedforward neural network with a single (linear) output neuron. The input to such a baseline function is a triple composed of state, goal, and time step. The baseline function is trained to approximate the value function using the mean squared (one-step) temporal difference error BID21. Parameters are updated using Adam . The networks are given by the following. Bit flipping environments and grid world environments. Both policy and baseline networks have two hidden layers, each with 256 hyperbolic tangent units. Every weight is initially drawn from a Gaussian distribution with mean 0 and standard deviation 0.01 (and redrawn if far from the mean by two standard deviations), and every bias is initially zero. Ms. Pac-man environment. The policy network is represented by a convolutional neural network. The network architecture is given by a convolutional layer with 32 filters (8×8, stride 4); convolutional layer with 64 filters (4 × 4, stride 2); convolutional layer with 64 filters (3 × 3, stride 1); and three fully-connected layers, each with 256 units. Every unit uses a hyperbolic tangent activation function. Every weight is initially set using variance scaling , and every bias is initially zero. These design decisions are similar to the ones made by BID6.A sequence of images obtained from the Arcade Learning Environment BID1 ) is preprocessed as follows. Individually for each color channel, an elementwise maximum operation is employed between two consecutive images to reduce rendering artifacts. Such 210 × 160 × 3 preprocessed image is converted to grayscale, cropped, and rescaled into an 84 × 84 image x t. A sequence of images x t−12, x t−8, x t−4, x t obtained in this way is stacked into an 84 × 84 × 4 image, which is an input to the policy network (recall that each action is repeated for 13 game ticks). The goal information is concatenated with the flattened output of the last convolutional layer. FetchPush environment. The policy network has three hidden layers, each with 256 hyperbolic tangent units. Every weight is initially set using variance scaling , and every bias is initially zero. Tables 1 and 2 document the experimental settings. The number of runs, training batches, and batches between evaluations are reported separately for hyperparameter search and definitive runs. The number of training batches is adapted according to how soon each estimator leads to apparent convergence. Note that it is very difficult to establish this setting before hyperparameter search. The number of batches between evaluations is adapted so that there are 100 evaluation steps in total. Other settings include the sets of policy and baseline learning rates under consideration for hyperparameter search, and the number of active goals subsampled per episode. In Tables 1 and 2, R 1 = {α×10 −k | α ∈ {1, 5} and k ∈ {2, 3, 4, 5}} and R 2 = {β ×10 −5 | β ∈ {1, 2.5, 5, 7.5, 10}}.As already mentioned in Section 6, the definitive runs use the best combination of hyperparameters (learning rates) found for each estimator. Every setting was carefully chosen during preliminary experiments to ensure that the best for each estimator is representative. In particular, the best performing learning rates rarely lie on the extrema of the corresponding search range. In the single case where the best performing learning rate found by hyperparameter search for a goal-conditional policy gradient estimator was such an extreme value (FetchPush, for a small batch size), evaluating one additional learning rate lead to decreased average performance. This appendix contains unabridged experimental . Appendices E.3.1 and E.3.2 present hyperparameter sensitivity plots for every combination of environment and batch size. A hyperparameter sensitivity plot displays the average performance achieved by each hyperparameter setting (sorted from best to worst along the horizontal axis). Appendices E.3.3 and E.3.4 present learning curves for every combination of environment and batch size. Appendix E.3.5 presents average performance . Appendix E.3.6 presents an empirical study of likelihood ratios. Appendix E.3.7 presents an empirical comparison with hindsight experience replay BID0 ). hyperparameter setting (best to worst) This study is conveyed through plots that encode the distribution of active likelihood ratios computed during training, individually for each time step within an episode. Each plot corresponds to an agent that employs HPG and obtains the highest definitive average performance for a given environment FIG2. Note that the length of the largest bar for a given time step is fixed to aid visualization. The most important insight provided by these plots is that likelihood ratios behave very differently across environments, even for equivalent time steps (for instance, compare bit flipping environments to grid world environments). In contrast, after the first time step, the behavior of likelihood ratios changes slowly across time steps within the same environment. In any case, alternative goals have a significant effect on gradient estimates, which agrees with the presented in Section 6. This appendix documents an empirical comparison between goal-conditional policy gradients (GCPG), hindsight policy gradients (HPG), deep Q-networks (, DQN), and a combination of DQN and hindsight experience replay (, DQN+HER).Experience replay. Our implementations of both DQN and DQN+HER are based on OpenAI Baselines , and use mostly the same hyperparameters that BID0 used in their experiments on environments with discrete action spaces, all of which resemble our bit flipping environments. The only notable differences in our implementations are the lack of both Polyak-averaging and temporal difference target clipping. Concretely, a cycle begins when an agent collects a number of episodes FORMULA6 by following an -greedy policy derived from its deep Q-network (= 0.2). The corresponding transitions are included in a replay buffer, which contains at most 10 6 transitions. In the case of DQN+HER, hindsight transitions derived from a final strategy are also included in this replay buffer. Completing the cycle, for a total of 40 different batches, a batch composed of 128 transitions chosen at random from the replay buffer is used to define a loss function and allow one step of gradient-based minimization. The targets required to define these loss functions are computed using a copy of the deep Q-network from the start of the corresponding cycle. Parameters are updated using Adam . A discount factor of γ = 0.98 is used, and seems necessary to improve the stability of both DQN and DQN+HER.Network architectures. In every experiment, the deep Q-network is implemented by a feedforward neural network with a linear output neuron corresponding to each action. The input to such a network is a triple composed of state, goal, and time step. The network architectures are the same as those described in Appendix E.1, except that every weight is initially set using variance scaling , and all hidden layers use rectified linear units BID8. For the Ms. Pac-man environment, the time step information is concatenated with the flattened output of the last convolutional layer (together with the goal information). In comparison to the architecture employed by BID0 for environments with discrete action spaces, our architectures have one or two additional hidden layers (besides the convolutional architecture used for Ms. Pac-man).Experimental protocol. The experimental protocol employed in our comparison is very similar to the one described in Section 6. Each agent is evaluated periodically, after a number of cycles that depends on the environment. During this evaluation, the agent collects a number of episodes by following a greedy policy derived from its deep Q-network. For each environment, grid search is used to select the learning rates for both DQN and DQN+HER according to average performance scores (after the corresponding standard deviation across runs is subtracted, as in Section 6). The candidate sets of learning rates are the following. Bit flipping and grid world environments: {α × 10 −k | α ∈ {1, 5} and k ∈ {2, 3, 4, 5}}, FetchPush: {10 −2, 5 × 10 −3, 10 −3, 5 × 10 −4, 10 −4}, Ms. Pac-man: {10 −3, 5 × 10 −4, 10 −4, 5 × 10 −5, 10 −5}. These sets were carefully chosen such that the best performing learning rates do not lie on their extrema. Definitive for a given environment are obtained by using the best hyperparameters found for each method in additional runs. These definitive are directly comparable to our previous for GCPG and HPG (batch size 16), since every method will have interacted with the environment for the same number of episodes before each evaluation step. For each environment, the number of runs, the number of training batches (cycles), the number of batches (cycles) between evaluations, and the number of episodes per evaluation step are the same as those listed in Tables 1 and 2.Results. The definitive for the different environments are represented by learning curves Pg. 38). In the bit flipping environment for k = 8 (Figure 40), HPG and DQN+HER lead to equivalent sample efficiency, while GCPG lags far behind and DQN is completely unable to learn. In the bit flipping environment for k = 16 FIG0 ), HPG surpasses DQN+HER in sample efficiency by a small margin, while both GCPG and DQN are completely unable to learn. In the empty room environment FIG1 ), HPG is arguably the most sample efficient method, although DQN+HER is more stable upon obtaining a good policy. GCPG eventually obtains a good policy, whereas DQN exhibits instability. In the four rooms environment FIG2 ), DQN+HER outperforms all other methods by a large margin. Although DQN takes much longer to obtain good policies, it would likely surpass both HPG and GCPG given additional training cycles. In the Ms. Pac-man environment (Figure 44), DQN+HER once again outperforms all other methods, which achieve equivalent sample efficiency (although DQN appears unstable by the end of training). In the FetchPush environment FIG3, HPG dramatically outperforms all other methods. Both DQN+HER and DQN are completely unable to learn, while GCPG appears to start learning by the end of the training process. Note that active goals are harshly subsampled to increase the computational efficiency of HPG for both Ms. Pac-man and FetchPush (see Sec. 6.3 and Sec. 6.4).Discussion. Our suggest that the decision between applying HPG or DQN+HER in a particular sparse-reward environment requires experimentation. In contrast, the decision to apply hindsight was always successful. Note that we have not employed heuristics that are known to sometimes increase the performance of policy gradient methods (such as entropy bonuses, reward scaling, learning rate annealing, and simple statistical baselines) to avoid introducing confounding factors. We believe that such heuristics would allow both GCPG and HPG to achieve good in both the four rooms environment and Ms. Pac-man. Furthermore, whereas hindsight experience replay is directly applicable to state-of-the-art techniques, our work can probably benefit from being extended to state-of-the-art policy gradient approaches, which we intend to explore in future work. Similarly, we believe that additional heuristics and careful hyperparameter settings would allow DQN+HER to achieve good in the FetchPush environment. This is evidenced by the fact that BID0 achieve good using the deep deterministic policy gradient (, DDPG) in a similar environment (with a continuous action space and a different reward function). The empirical comparisons between either GCPG and HPG or DQN and DQN+HER are comparatively more conclusive, since the similarities between the methods minimize confounding factors. Regardless of these empirical , policy gradient approaches constitute one of the most important classes of model-free reinforcement learning methods, which by itself warrants studying how they can benefit from hindsight. Our approach is also complementary to previous work, since it is entirely possible to combine a critic trained by hindsight experience replay with an actor that employs hindsight policy gradients. Although hindsight experience replay does not require a correction analogous to importance sampling, indiscriminately adding hindsight transitions to the replay buffer is problematic, which has mostly been tackled by heuristics (, Sec. 4.5). In contrast, our approach seems to benefit from incorporating all available information about goals at every update, which also avoids the need for a replay buffer.

Represent the TLDR for finding the corresponding scientific article

We introduce the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended to policy gradient methods.

We propose Dual Actor-Critic algorithm, which is derived in a principled way from the Lagrangian dual form of the Bellman optimality equation. The algorithm achieves the state-of-the-art performances across several benchmarks.


Question: Represent the article for finding a single-sentence TLDR

Question: Computer vision has undergone a dramatic revolution in performance, driven in large part through deep features trained on large-scale supervised datasets. However, much of these improvements have focused on static image analysis; video understanding has seen rather modest improvements. Even though new datasets and spatiotemporal models have been proposed, simple frame-by-frame classification methods often still remain competitive. We posit that current video datasets are plagued with implicit biases over scene and object structure that can dwarf variations in temporal structure. In this work, we build a video dataset with fully observable and controllable object and scene bias, and which truly requires spatiotemporal understanding in order to be solved. Our dataset, named CATER, is rendered synthetically using a library of standard 3D objects, and tests the ability to recognize compositions of object movements that require long-term reasoning. In addition to being a challenging dataset, CATER also provides a plethora of diagnostic tools to analyze modern spatiotemporal video architectures by being completely observable and controllable. Using CATER, we provide insights into some of the most recent state of the art deep video architectures. While deep features have revolutionized static image analysis, video descriptors have struggled to outperform classic hand-crafted descriptors . Though recent works have shown improvements by merging image and video models by inflating 2D models to 3D , simpler 2D models (b) still routinely appear among top performers in video benchmarks such as the Kinetics Challenge at CVPR'17. This raises the natural question: are videos trivially understandable by simply averaging the predictions over a sampled set of frames? At some level, the answer must be no. Reasoning about high-level cognitive concepts such as intentions, goals, and causal relations requires reasoning over long-term temporal structure and order . Consider, for example, the movie clip in Fig. 1 (a), where an actor leaves the table, grabs a firearm from another room, and returns. Even though no gun is visible in the final frames, an observer can easily infer that the actor is surreptitiously carrying the gun. Needless to say, any single frame from the video seems incapable of supporting that inference, and one needs to reason over space and time in order to reach that . As a simpler instance of the problem, consider the cup-and-balls magic routine 1, or the gamblingbased shell game 2, as shown in Fig. 1 (b). In these games, an operator puts a target object (ball) under one of multiple container objects (cups), and moves them about, possibly revealing the target at various times and recursively containing cups within other cups. The task at the end is to tell which of the cups is covering the ball. Even in its simplest instantiation, one can expect any human or computer system that solves this task to require the ability to model state of the world over long temporal horizons, reason about occlusion, understand the spatiotemporal implications of containment, etc. An important aspect of both our motivating examples is the adversarial nature of the task, where the operator in control is trying to make the observer fail. Needless to say, a frame by frame prediction model would be incapable of solving such tasks. Figure 1: Real world video understanding. Consider this iconic movie scene from The Godfather in (a), where the protagonist leaves the table, goes to the bathroom to extract a hidden firearm, and returns to the table presumably with the intentions of shooting a person. While the gun itself is visible in only a few frames of the whole clip, it is trivial for us to realize that the protagonist has it in the last frame. An even simpler instantiation of such a reasoning task could be the cup-and-ball shell game in (b), where the task is to determine which of the cups contain the ball at the end of the trick. Can we design similarly hard tasks for computers? Given these motivating examples, why don't spatiotemporal models dramatically outperform their static counterparts for video understanding? We posit that this is due to limitations of existing video benchmarks. Even though video datasets have evolved from the small regime with tens of labels (; ;) to large with hundreds of labels , tasks have remained highly correlated to the scene and object context. For example, it is trivial to recognize a swimming action given a swimming pool in the (b). This is further reinforced by the fact that state of the art pose-based action recognition models are outperformed by simpler frame-level models (b) on the Kinetics benchmark, with a difference of nearly 45% in accuracy! Sigurdsson et al. also found similar for their Charades benchmark, where adding ground truth object information gave the largest boosts to action recognition performance . In this work, we take an alternate approach to developing a video understanding dataset. Inspired by the recent CLEVR dataset (that explores spatial reasoning in tabletop scenes) and inspired by the adversarial parlor games above (that require temporal reasoning), we introduce CATER, a diagnostic dataset for Compositional Actions and TEmporal Reasoning in dynamic tabletop scenes. We define three tasks on the dataset, each with an increasingly higher level of complexity, but set up as classification problems in order to be comparable to existing benchmarks for easy transfer of existing models and approaches. Specifically, we consider primitive action recognition, compositional action recognition, and adversarial target tracking under occlusion and containment. However, note that this does not limit the usability of our dataset to these tasks, and we provide full metadata with the rendered videos that can be used for more complex, structured prediction tasks like detection, tracking, forecasting, and so on. Our dataset does not model an operator (or hand) moving the tabletop objects, though this could be simulated as well in future variants, as in . Being synthetic, CATER can easily be scaled up in size and complexity. It also allows for detailed model diagnostics by controlling various dataset generation parameters. We use CATER to benchmark state-of-the-art video understanding models ), and show even the best models struggle on our dataset. We also uncover some insights into the behavior of these models by changing parameters such as the temporal duration of an occlusion, the degree of camera motion, etc., which are difficult to both tune and label in real-world video data. Spatiotemporal networks: Video understanding for action recognition has evolved from iconic hand-designed models (; ;) to sophisticated spatiotemporal deep networks (; ; ; ; ; . While similar developments Dataset Size Len Task #cls TO STR LTR CSB UCF101 13K 7s cls 101 HMDB51 5K 4s cls 51 Kinetics 300K 10s cls 400 AVA 430 15m det 80 VLOGs 114K 10s cls 30 DAHLIA 51 39m det 7 TACoS in the image domain have lead to large improvements on tasks like classification (; a; and localization , video models have struggled to out-perform previous hand-crafted descriptors . Even within the set of deep video architectures, models capable of temporal modeling, such as RNNs and 3D convolutions (; a) have not shown significantly better performance than much simpler, per-frame prediction models, such as variants of two-stream architectures (b;). Though some recent works have shown improvements by merging image and video models by inflating 2D models to 3D , simple 2D models (b) were still among the top performers in the Kinetics Challenge at CVPR'17. Video action understanding datasets: There has been significant effort put forth to collecting video benchmarks. One line of attack employs human actors to perform scripted actions. This is typically done in controlled environments (; ;), but recent work has pursued online crowd sourcing . Another direction collects videos from movies and online sharing platforms. Many popular video benchmarks follow this route for diverse, in-the-wild videos, such as UCF-101 , HMDB-51 and more recently Kinetics and VLOGs . As discussed earlier, such datasets struggle with the strong bias of actions with scenes and objects. Our underlying thesis is that the field of video understanding is hampered by such biases because they favor image-based baselines. One might argue that since such biases are common in the visual world, video benchmarks should reflect them. We take the view that a diverse set of benchmarks are needed to enable comprehensive diagnostics and validation of the state-of-affairs in video understanding. Table 1 shows that CATER fills a missing gap in the benchmark landscape, most notably because of its size, label distribution, and relative resilience to object and scene bias. Synthetic data in computer vision: Our work, being synthetically generated, is also closely related to other works in using synthetic data for computer vision applications. There has been a large body of work in this direction, with the major focus on using synthetic training data for real world applications. This includes semantic scene understanding (; ;), 3D scene understanding (; ; ;, human understanding (b;), optical flow and navigation, RL or embodied learning;; ). Our work, on the other hand, attempts to develop a benchmark for video based action understanding. Similar attempts have been made for scene understanding through abstract scenes , with more recently focusing on building a complex reasoning benchmark, CLEVR . In the video domain, Long et al. proposed Flash-MNIST, with the task of recognizing all the digits that appear. We build upon CLEVR and extend it for spatiotemporal reasoning in videos, defining tasks that truly require spatiotemporal reasoning to be solved.: CATER dataset and tasks. Sampled frames from a random video from CATER. We show some of the actions afforded by objects in this video, as labeled on the top using arrows. We define three tasks on these videos. Task 1 requires identifying all active actions in the video. Task 2 requires identifying all active compositional actions. Task 3 requires quantized spatial localization of the snitch object at the end of the video. Note that, as in this case, the snitch may be occluded or'contained' by another object, and hence models would require spatiotemporal understanding to complete the task. Please refer to the supplementary video for more example videos. Object tracking: Detecting and tracking objects has typically been used as an initial representation for long-term video and activity understanding (; ;). Extensions include adversarial tracking, where the objects are designed to be hidden from plain view. It has typically been used for tasks such as determining if humans are carrying an object or abandoned / exchanging objects . We embrace this direction of work and include state-of-the-art deep trackers in our benchmark evaluation. CATER provides a video understanding dataset that requires long term temporal reasoning to be solved. Additionally, it provides diagnostic tools that can evaluate video models in specific scenarios, such as with or without camera motion, with varying number of objects and so on. This control over the dataset parameters is achieved by synthetically rendering the data. These videos come with a ground truth structure that can be used to design various different video understanding tasks, including but not limited to object localization and spatiotemporal action composition. Unlike existing video understanding benchmarks, this dataset is free of object or scene bias, as the same set of simple objects are used to render the videos. Fig. 2 describes the dataset and the associated tasks. We provide sample videos from the dataset in the supplementary video. The CATER universe is built upon CLEVR , inheriting most of the standard object shapes, sizes, colors and materials present in it. This includes three object shapes (cube, sphere, cylinder), in three sizes (small, medium, large), two materials (shiny metal and matte rubber) and eight colors, as well as a large "table" plane on which all objects are placed. In addition to these objects, we add two new object shapes: inverted cones and a special object called a'snitch'. Cones also come in the same set of sizes, materials and colors. The'snitch' is a special object shaped like three intertwined toruses in metallic gold color. Actions: We define four atomic actions:'rotate','pick-place','slide' and'contain'; a subset of which is afforded by each object. The'rotate' action means that the object rotates by 90°about its Y (or horizontal) axis, and is afforded by cubes, cylinders and the snitch. The'pick-place' action means the object is picked up into the air along the Y axis, moved to a new position, and placed down. This is afforded by all objects. The'slide' action means the object is moved to a new location by sliding along the bottom surface, and is also afforded by all objects. Finally,'contain' is a special operation, only afforded by the cones, in which a cone is pick-placed on top of another object, which may be a sphere, a snitch or even a smaller cone. This allows for recursive containment, as a cone can contain a smaller cone that contains another object. Once a cone'contains' an object, it is constrained to only'slide' actions and effectively slides all objects contained within the cone. This Figure 3: Allen's temporal algebra. Exhaustive list of temporal relations between intervals, as defined by Allen's algebra . For simplicity, we group them into three broad relations to define classes for composite actions, although in principle we could use all thirteen. Figure courtesy of (Alspaugh). holds until the top-most cone is pick-placed to another location, effectively ending the containment for that top-most cone. Animation process: We start with an initial setup similar to CLEVR. A random number (N) of objects with random parameters are spawned at random locations at the beginning of the video. They exist on a 6 × 6 portion of a 2D plane with the global origin in the center. In addition to the random objects, we ensure that every video has a snitch and a cone. For the purposes of this work, we render 300-frame 320x240px videos, at 24 FPS, making it comparable to standard benchmarks (; ;). We split the video into 30-frame slots, and each action is contained within these slots. At the beginning of each slot, we iterate through up to K objects in a random order and attempt to add an action afforded by that object one by one without colliding with another object. As we describe later, we use K = 2 for our initial tasks and K = N for the final task. For each action, we pick a random start and end time from within the 30-frame slot. To further add to the diagnostic ability of this dataset, we render an additional set of videos with camera motion, with all other aspects of the data similarly distributed as the static camera case. For this, the camera is always kept pointed towards the global origin, and moved randomly between a predefined set of 3D coordinates. These coordinates include X and Y ∈ {−10, 10} and Z ∈ {8, 10, 12}. Every 30 frames, we randomly pick a new location from the Cartesian product of X, Y, Z, and move the camera to that location over the next 30 frames. However, we do constrain the camera to not change both X and Y coordinates at the same time, as that causes a jarring viewpoint shift as the camera passes over the (0, 0, Z) point. Also, we ensure all the camera motion videos start from the same viewpoint, to make it easy to register the axes locations for localization task. Spatiotemporal compositions: We wish to label our animations with the atomic actions present, as well as their compositions. Atomic actions have a well-defined spatiotemporal footprint, and so we can define composites using spatial relations ("a cylinder is rotating behind a sliding red ball"), similar to CLEVR. Unique to CATER is the ability to designate temporal relationships ("a cylinder rotates before a ball is picked-and-placed"). Because atomic actions occupy a well-defined temporal extent, we need temporal logic that reasons about relations between intervals rather than instantaneous events. While the latter can be dealt with timestamps, the former can be described with Allen's interval algebra with thirteen basic relations (Figure 3) along with composition operations. For simplicity, we group those into three broad relations. However, our dataset contains examples of all such interval relations and can be used to explore fine-grained temporal relationships. Given this CATER universe with videos, ground truth objects and their actions at any time point, we can define arbitrarily complex tasks for a video understanding system. Our choice of tasks is informed by two of the main goals of video understanding: 1) Recognizing the states of the actor, including spatiotemporal compositions of those atomic actions. For example, a spatiotemporal composition of atomic human body movements can be described as an exercise or dance routine. And 2) Recognizing the effect of those actions on the state of the world. For example, an action involving picking and placing a cup would change the position of the cup and any constituent objects contained within it, and understanding this change in the world state would implicitly require understanding the action itself. Given these two goals, we define three tasks on CATER. Each has progressively higher complexity, and tests for a higher level reasoning ability. To be consistent with existing popular benchmarks (; ; ;), we stick to standard single or multi-label classification setup, with standard evaluation metrics, as described next. For each of these tasks, we start by rendering 5500 total videos, to be comparable in size with existing popular benchmarks . Since tasks 1 and 2 (defined next) explicitly require recognizing individual actions, we use K = 2 for the videos rendered to keep the number of actions happening in any given video small. For task 3, we set K = N as the task is to recognize the end effect of actions, and not necessarily the actions themselves. We split the data randomly in 70:30 ratio into a training and test set. We similarly render a same size dataset with camera motion, and define tasks and splits in the same way as for the static camera. Task 1: Atomic action recognition. This first task on CATER is primarily designed as a simple debugging task, which should be easy for contemporary models to solve. Given the combinations of object shapes and actions afforded by them, we define 14 classes such as'slide(cone)','rotate(cube)' and so on. Since each video can have multiple actions, we define it as a multi-label classification problem. The task is to produce 14 probability values, denoting the likelihood of that action happening in the video. The performance is evaluated using average precision per-class. Final dataset-level performance is computed by mean over all classes, to get mean average precision (mAP). This is a popular metric used in other multi-label action classification datasets . Task 2: Compositional action recognition. While recognizing individual objects and motions is important, it is clearly not enough. Real world actions tend to be composite in nature, and humans have no difficulty recognizing them in whole or in parts. To that end, we construct a compositional action recognition task through spatiotemporal composition of the basic actions used in Task 1. For simplicity, we limit composites to pairs of 14 atomic actions, where the temporal relation is grouped into broad categories of'before','during' and'after' as shown in Figure 3. Combining all possible atomic actions with the three possible relations, we get a total of 14 × 14 × 3 = 588 classes, and removing duplicates (such as 'X after Y' is a duplicate of 'Y before X'), leaves 301 classes. Similar to task 1, multiple compositions can be active in any given video, so we set it up as a multi-label classification problem, evaluated using mAP. If certain compositions never occur in the dataset, those are ignored for the final evaluation. Task 3: Snitch localization. The final, and the flagship task in CATER, tests models' ability to recognize the effect of actions on the environment. Just as in the case of cup-and-ball trick, the ability of a model to recognize location of objects after some activity can be thought of as an implicit evaluation of its ability to understand the activity itself. The task now is to predict the location of the special object introduced above, the Snitch. While it may seem trivial to localize it from the last frame, it may not always be possible to do that due to occlusions and recursive containments. The snitch can be contained by other objects (cones), which can further be contained by other larger cones. All objects move together until'uncontained', so the final location of the snitch would require long range reasoning about these interactions. For simplicity, we pose this as a classification problem by quantizing the 6 × 6 grid into 36 cells and asking which cell the snitch is in, at the end of the video. We ablate the grid size in experiments. Since the snitch can only be at a single location at the end of the video, we setup the problem as a single label classification, and evaluate it using standard percentage accuracy metrics such as top-1 and top-5 accuracy. However, one issue with this metric is that is would penalize predictions where the snitch is slightly over the cell boundaries. While the top-5 metric is somewhat robust to this issue, we also report mean L 1 distance of predicted grid cell from the ground truth, as a metric that is congnizant of the grid structure in this task. Hence, it would penalize confusion between adjacent cells less than those between distant cells. The data is also amenable to a purely regression-style evaluation, though we leave that to future work. We now experiment with CATER using recently introduced state of the art video understanding and temporal reasoning models . I3D , called R3D when implemented using a ResNet (a) in, brings the best of image models to video domain by inflating it into 3D for spatiotemporal feature learning. Non-local networks further build upon that to add a spatiotemporal interaction layer that gives strong improvements and out-performs many multi-stream architectures (that use audio, flow etc) on Kinetics and Charades benchmarks. For our main task, snitch localization, we also experiment with a 2D-conv based approach, Temporal Segment Networks (TSN) (b), which another top performing method on standard benchmarks . This approaches uses both RGB and flow modalities. All these architectures learn a model for individual frames or short clips, and at test time aggregate the predictions by averaging over those clips. While simple averaging works well enough on most recent datasets (; ;), it clearly loses all temporal information and may not be well suited to our set of tasks. Hence, we also experiment with a learned aggregation strategy: specifically using an LSTM for aggregation, which is the tool of choice for temporal modelling in various domains including language and audio. We use a common LSTM implementation for aggregating either (b) or ) that operates on the last layer features (before logits). We extract these features for subclips from train and test videos, and train a 2-layer LSTM with 512 hidden units in each layer on the train subclips. The LSTM produces an output at each clip it sees, and we enforce a classification loss at the end, once the model has seen all the clips. At test time we take the prediction from the last clip as the aggregated prediction. We report the LSTM performance averaged over three runs to control for random variation. It is worth noting that LSTMs have been previously used for action recognition in videos , however with only marginal success over simple average pooling. As we show later, LSTMs actually perform significantly better on CATER, indicating the importance of temporal reasoning. For task 3, we also experiment with a state-of-the-art visual tracking method . We start by using the GT information of the starting position of snitch, and project it to screen coordinates using the render camera parameters. We defined a fixed size box around it to initialize the tracker, and run it until the end of the video. At the last frame, we project the center point of the tracked box to the 3D plane (and eventually, the class label) by using a homography transformation between the image and the 3D plane. This provides a more traditional, symbolic reasoning baseline for our dataset, and as we show in , is also not enough to solve the task. Finally, we do note that many other video models have been proposed in literature involving 2.5D convolutions , VLAD-style aggregation and other multi-modal architectures (a;). We focus on the most popular and best performing models, and leave a more comprehensive study to future work. A random baseline is also provided for all tasks, computed as the average performance of random scores passed into the evaluation functions. Implementation details for all baselines are provided in the supplementary and code will be released. Task 1: Atomic action recognition: Table 2 (a) shows the performance of R3D with and without the non-local (NL) blocks, using different number of frames in the clips. We use a fixed sampling rate of 8, but experiment with different clip sizes. Adding more frames helps significantly in this case. Given the ease of the task, R3D obtains fairly strong performance for static camera, but not so much for moving camera, suggesting potential future work in building models agnostic to camera motion. Task 2: Compositional action recognition: Next we experiment with the compositional action recognition task. The training and testing is done in the same way as Task 1, except this predicts confidence over 301 classes. As evident from Table 2 (b), this task is harder for the existing models, presumably as recognizing objects and simple motions would no longer solve it, and models need to reason about spatiotemporal compositions as well. It is interesting to note that non-local blocks now add to the final performance, which was not the case for Task 1, suggesting modeling spatiotemporal relations is more useful for this task. LSTM aggregation also helps quite a bit as the model can learn to reason about long-range temporal compositions. As expected, moving camera makes the problem harder. Task 3: Snitch localization: Finally we turn to the localization task. Since this is setup as a single label classification, we use softmax cross entropy loss to train and classification accuracy for evaluation. For tracking, no training is required as we use the pre-trained model from and run it on the validation videos. Table 3: Long term reasoning. Comparing the best reported performance of standard models on existing datasets and CATER (task 3). Unlike previous benchmarks, temporal modeling using LSTM helps and local temporal cues (flow) are not effective by itself on CATER. 2S here refers to'Two Stream'. TSN performance from (; 2016). evaluated at different clip lengths and frame rates. For this task we also experiment with TSN (b), though it ends up performing significantly worse than R3D. Note that this contrasts with standard video datasets , where it tends to perform similar to R3D . We also experiment with the flow modality and observe it obtains even lower performance, which is expected as this task requires recognizing objects which is much harder from flow. Again, note that flow models obtain similar if not better performance as RGB on standard datasets . We also note higher performance on considering longer clips with higher sample rate. This is not surprising as a task like this would require long term temporal reasoning, which is aided by looking at longer videos. This is also reinforced by the observation that using LSTM for aggregation leads to a major improvement in performance for most models. Finally, the tracking approach also only solves about a third of the videos, as even the state of the art tracker ends up drifting due to occlusions and contain operations. In Table 4, we ablate the performance with respect to the underlying grid granularity, with 6 × 6 being the default used in Table 2 (c). We observe tracking is a stronger baseline as the localization task gets more fine-grained. Finally in Table 3 we compare performance of some of these models on existing benchmarks and CATER. Analysis: Having close control over the dataset generation process enables us to perform diagnostics impossible with any previous dataset. We use the R3D+NL, 32-frame, static camera model with average (or LSTM, when specified) pooling for all following visualizations. We first analyze aggregate performance of our model over multiple bins in Figure 4, and observe some interesting phenomena. (a) Performance drops if the snitch keeps moving until the end. This makes sense: if the snitch reaches its final position early in the video, models have a lot more frames to reinforce their hypothesis of its final location. Between LSTM and avg-pooling, LSTM is much better able to handle the motion of the snitch, as expected. Perhaps not surprisingly, the tracker is much less effected by snitch movement, indicating the power of such classic computational pipelines for longterm spatiotemporal understanding. (b) Drops if the snitch is contained in the end. Being contained in the final frame makes the snitch harder to spot and track (just like the cups and ball game!), hence the lower performance.: Diagnostic analysis of localization performance. We bin the test set using certain parameters. For each, we show the test set distribution with the bar graph, the performance over that bin using the line plot, and performance of that model on the full val set with the dotted line. We find that localization performance, (a) Drops significantly if the snitch is kept moving till the end. This is possibly because for cases when snitch only moves in the beginning and is static after, the models have a lot more evidence to predict the correct location from. Interestingly the tracker is much less affected by this, as it tracks the snitch until the very end; (b) Drops if the snitch is'contain'-ed by another object in the end, and the tracker is the worst affected by it; (c) Drops initially with increasing displacement of the snitch from its start position, but is stable after that; and (d) Is relatively stable with different number of objects in the scene. Next, we visualize the videos that our models gets right or wrong. We sort all validation videos based on the softmax confidence score for the ground truth class, and visualize the top and bottom six in Figure 5 (full video in supplementary). We find that the easiest videos for avg-pooled model tend to be ones with little snitch motion, i.e. the object stays at the position it starts off in. On the other hand, the LSTM-aggregated model fares better with snitch motion, as long as it happens early in the video. The hardest videos for both tend to be ones with sudden motion of the snitch towards the end of the video, as shown by the bright golden trail denoting the motion towards the end (better viewed in supplementary video). These observations are supported by the quantitative plots in Figure 4 (a) and (c). We use CATER to analyze several leading network designs on hard spatiotemporal tasks. We find most models struggle on our proposed dataset, especially on the snitch localization task which requires long term reasoning. Interestingly, average pooling clip predictions or short temporal cues (optical flow) perform rather poorly on CATER, unlike most previous benchmarks. Such temporal reasoning challenges are common in the real world (eg. Fig. 1 (a) ), and solving those would be the cornerstone of the next improvements in machine video understanding. We believe CATER would serve as an intermediary in building systems that will reason over space and time to understand actions. That said, CATER is, by no means, a complete solution to the video understanding problem. Like any other synthetic or simulated dataset, it should be considered in addition to real world benchmarks. While we have focused on classification tasks for simplicity, our fully-annotated dataset can be used for much richer parsing tasks such as spacetime action localization. One of our findings is that while high-level semantic tasks such as activity recognition may be addressable with current architectures given a richly labeled dataset, "mid-level" tasks such as tracking still pose tremendous challenges, particularly under long-term occlusions and containment. We believe addressing such challenges will enable broader temporal reasoning tasks that capture intentions, goals, and causal behavior. We analyze the top most confident a) correct and b) incorrect predictions on the test videos for localization task. For each video, we show the last frame, followed by a top-down view of the 6 × 6 grid. The grid is further overlayed with: 1) the ground truth positions of the snitch over time, shown as the golden trail, which fades in color over time =⇒ brighter yellow depicts later positions; and 2) the softmax prediction confidence scores for each location (black is low, white is high). The model has easiest time classifying the location when the snitch does not move much or moves early on in the video. Full video in supplementary. We use the provided implementation for ResNet-3D (R3D) and non-local (NL) block from, and temporal segment networks (TSN) from (b) for all our experiments. For, all the models are based on ResNet-50 base architecture, and trained with hyperparameters scaled down from Kinetics as per CATER size. For non-local (NL) experiments, we replace the conv3 and conv4 blocks in ResNet with the NL blocks. All models are trained with classification loss implemented using sigmoid cross-entropy for Task 1 and 2 (multi-label classification task), and softmax cross-entropy for task 3. At test time, we split the video into 10 temporal clips and 3 spatial clips. When aggregating using average pooling, we average the predictions from all 30-clips. For LSTM, we train and test on the 10 center clips. We experiment with varying the number of frames (#frames) and sampling rate (SR). For TSN (b), the model is based on BN-inception , with hyperparamters following their implementation on HMDB given its similar size and setup to our dataset. For optical flow we use the TVL1 (Sánchez Pérez et al., 2013) implementation. At test time we aggregate the predictions over 250 frames uniformly sampled from the video, either by averaging or using LSTM. While CATER videos look different from real world, we found the networks much easier to optimize with the ImageNet initialization for both approaches. This is consistent with prior work (b) that finds ImageNet initialization is useful even when training diverse modalities (such as optical flow). We will make the code, generated data and models available for more implementation details. Figure 6 shows the data distribution over classes for each of the tassk. The supplementary video 3 visualizes: 1. Sample videos from the dataset (with and without camera motion). 2. Easiest and hardest videos for task 3. We rank all validation videos for task 3 based on their softmax probability for the correct class. We show the top-6 (easiest) and bottom-6 (hardest) for 32-frame stride-8 non-local + LSTM model. We observe the hardest ones involve sudden motion towards the end of the video. This reinforces the observation made in Figure 5 (a) in the main paper, that videos where snitch keeps moving till the end are the hardest. If the snitch stops moving earlier, models have more evidence for the final location of the snitch, making the task easier. 3. Tracking . We visualize the of tracking the snitch over the video as one approach to solving task 3. We observe that while it works in the simple scenarios, it fails when there is a lot of occlusion or complex contain operations. 4. Model bottom-up attention. We visualize where does the model look for Task 3. As suggested in , we visualize the l 2 -norm of the last layer features from our 32-frame stride-8 non-local model on the center video crop. The deep red color denotes large norm value at that spatiotemporal location. We find that the model automatically learns to focus on the snitch towards the end of clips, which makes sense as that is the most important object for solving the localization task. 35 30 23 1 31 13 3 20 33 16 17 26 27 18 9 32 6 28 24 11 2 14 15 4 34 19 21 7 22 12 10 8 Histograms of training and validation data distribution for different tasks we define on the dataset. (a) requires the model to recognize atomic actions, such as'a sphere slides'. We defined 13 such classes. (b) requires recognizing spatiotemporal compositions of actions, such as'a sphere slides while a cube rotates'. Since there are a total of 588 combinations, we omit the labels here for ease of visualization. Finally (c) evaluates the snitch localization task, where the model needs to answer where the snitch is on the board, quantized into a 6 × 6 grid, at the end of the video. This is defined as a 36-way classification problem.

We propose a new video understanding benchmark, with tasks that by-design require temporal reasoning to be solved, unlike most existing video datasets.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We can identify prototypical and outlier examples in machine learning that are quantifiably very different, and make use of them to improve many aspects of neural networks.