{'fraction_non_alphanumeric': 0.030870116613566766, 'fraction_numerical': 0.01073339725942652, 'mean_word_length': 4.380262938924946, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
PoseAgent: Budget-Constrained 6D Object Pose Estimation via Reinforcement Learning

State-of-the-art computer vision algorithms often achieve efficiency by making discrete choices about which hypotheses to explore next. This allows allocation of computational resources to promising candidates, however, such decisions are non-differentiable. As a result, these algorithms are hard to train in an end-to-end fashion. In this work we propose to learn an efficient algorithm for the task of 6D object pose estimation. Our system optimizes the parameters of an existing state-of-the art pose estimation system using reinforcement learning, where the pose estimation system now becomes the stochastic policy, parametrized by a CNN. Additionally, we present an efficient training algorithm that dramatically reduces computation time. We show empirically that our learned pose estimation procedure makes better use of limited resources and improves upon the state-of-the-art on challenging datasets. Our approach enables differentiable end-to-end training of complex algorithmic pipelines and learns to make optimal use of a given computational budget.


Introduction
Many tasks in computer vision involve learning a function, usually learning to predict a desired output label given an input image. Advances in deep learning have led to huge progress in solving such tasks. In particular, convolutional neural networks (CNNs) work well when trained over large training sets using gradient descent methods to minimize the expected loss between the predictions and the ground truth labels.
However, important computer vision algorithms take the form of algorithms rather than being a simple differentiable function: example sliding window search, superpixel partioning, particle filters, and classification cascades are algorithms realizing complex non-continuous functions.
The algorithmic approach is especially useful in situa-tions where computational budget is limited: an algorithm can dynamically assign its budget to solving different aspects of the problem, for example, to take shortcuts in order to spend computation on more promising solutions at the expense of less promising ones. We would like to learn the algorithm. Unfortunately, the hard decisions taken in most algorithmic approaches are non-differentiable, and this means that the structure and parameters of these efficient algorithm cannot be easily learned from data.
Reinforcement learning (RL) [22] offers a possible solution to learning algorithms. We view the algorithm as the policy of an RL agent, i.e. a description of dynamic sequential behaviour. Then RL provides a framework to learn the parameters of such behaviour with the goal of maximizing an expected reward, for example, the accuracy of the algorithm output.
We apply this perspective on algorithmic computer vision approaches. In particular, we address the problem of 6D object pose estimation and use RL to learn the parameters of a deep algorithmic pipeline to provide the best possible accuracy given a limited computational budget.
Object pose estimation is the task of estimating from an image the 3D translation (position) and 3D rotation (orientation) of a specific object relative to its environment. This task is important in many applications such as robotics and augmented reality where the efficient use of a limited computation budget is an important requirement.
State-of-the-art pose systems such as the system of Krull et al. [12] generate a pool of pose hypotheses, then score each hypothesis using a pre-trained CNN. The subset of high-scoring hypotheses get refined and ultimately the highest-scoring hypothesis is returned as the answer. Computationally the refinement step is the most expensive, and there is a trade-off between the number of refinements allowed and the expected quality of the result.
Ideally, one would train such state-of-the-art system endto-end in order to learn how to use the optimal number of refinements to maximize the expected success of pose estimation. Unfortunately, treating the system as a black box with parameters to optimize is impossible for two reasons: (i) each selection process is non-differentiable with respect to the scoring function; and (ii) the loss used to determine whether an estimated pose is correct is also nondifferentiable.
We recast pose estimation as an RL problem in order to overcome these difficulties. We model the pose inference process as an RL agent which we call PoseAgent. PoseAgent is granted more flexibility than the original system: it is given a fixed budget of refinement steps, and is allowed to manipulate its hypothesis pool by selecting individual poses for refinement, until the budget is spent. In our PoseAgent model each agent decision follows a probability distribution over possible actions. This distribution is called the policy and we can differentiate and optimize this continuous policy through the stochastic policy gradient approach [23]. As a result of this stochastic approach the final pose estimate becomes a random variable, and each run of PoseAgent will produce a slightly different result.
This policy gradient approach is very general and does not require differentiability of the used loss function. As a consequence we can directly take the gradient with respect to the expected loss of interest, i.e. the number of correctly estimated poses.
Training in policy gradient methods can be difficult due to the additional variance of estimated gradients [7,23], because the additional randomness leads to a bigger variance in the estimated gradients. To overcome this problem we propose an efficient training algorithm that radically reduces the variance during training compared to a naïve technique.
We compare our approach to the state-of-the-art [12] and achieve substantial improvements in accuracy, while using the same or smaller average budget of refinement steps compared to [12].
In summary our contributions are: • To the best of our knowledge, we are the first to apply a policy gradient approach to the object pose estimation problem. • Our approach allows the use of a non-differentiable reward function corresponding to the original evaluation criterion. • We present an efficient training algorithm that dramatically reduces the variance during training. • We improve significantly upon the best published results on the dataset.

Related Work
Below, we first discuss approaches for 6D pose estimation, focusing in particular on object coordinate prediction methods, and then provide a short review of RL methods used in a setting similar to ours.

Pose Estimation
There is a large variety of approaches for 6D object pose estimation. Traditionally, approaches based on sparse features [14,15] have been successful, but work well only for textured objects. Other approaches include template-based methods [9,19], voting schemes [6,10], and CNN-based direct pose regression [8].
We focus on the line of work called object coordinate regression [3], which provides the basic framework for our approach. Object coordinate regression was originally proposed for human body pose estimation [24] and camera localization [20]. In [3] a random forest provides a dense pixel-wise prediction for 6D object pose prediction. At each pixel, the forest predicts whether and where the pixel is located on the surface of the object. One can then efficiently generate pose hypotheses by sampling a small set of pixels and combining the forest predictions with depth information from an RGB-D camera.
The object coordinate regression methods in [3,12,17] score these hypotheses by comparing rendered and observed image patches. While [3,17] use a simple pixel wise distance function, [12] propose a learned comparison: a CNN compares rendered and observed images and outputs an energy value representing a parameter of the posterior distribution in pose space. Despite their differences in the particular scoring functions, [3,17,12] use the same inference technique to arrive at the final pose estimate: they all refine the best hypotheses, re-score them, and output the best one as their final choice. Our PoseAgent approach can be seen as a generalization of this algorithm, in which the agent selects the hypotheses for refinement repeatedly, each time being able to make a more informed choice.
The work of Krull et al. [12] is the most closely related to our work. We use a similar CNN construction as Krull et al., feeding both rendered and observed image patches into to our CNN. However, we use the output of the CNN as a parameter of the stochastic policy that controls the behaviour of our pose agent. Moreover, while the training process in [12] is seen as learning the posterior distribution, which is then maximized during testing using the fixed inference procedure, our training process instead modifies the behaviour of the agent directly in order to maximize the number of correctly estimated poses.

Reinforcement Learning in Similar Tasks
RL has traditionally been successful in areas like robotics [21], control [1], advertising, network routing, or playing games. While the application of RL seems natural for such cases where real agents and environments are involved, RL is increasingly being successfully applied in  [3], the image is processed by a random forest. c) The forest outputs dense predictions of object probabilities (top) and object coordinates (bottom). The object coordinates are mapped to the RGB cube for visualization. d) We use the predictions together with the depth information to sample a pool of pose hypotheses H 0 . e) An RL agent manipulates the hypotheses pool by repeatedly selecting individual hypotheses to refine. This is the focus of this paper. f) The agent outputs a final pose estimateH.
computer vision systems where the interpretation of the system as an agent interacting with an environment is not always so intuitive. While we are to our knowledge the first to apply RL for 6D object pose estimation, there are several recent papers that apply RL for 2D object detection and recognition [18,5,16,2]. In [18,5], an agent shifts its area of attention over the image until it makes a final decision. Instead of moving a single 2D area of attention over search space like [18,5], we work with a pool of multiple 6D pose hypotheses. The agent in [16] focuses its attention by moving a 2D fixation point, though operates on a set of precomputed image regions to gather information and make a final decision. Our agent instead manipulates its hypothesis pool by refining individual hypotheses.
Caicedo et al. [5] use Q-learning, in which the CNN predicts the quality of the available state-action pairs. Mnih et al. [18] and Mathe et al. [16] use a different RL approach based on stochastic policy gradient, in which the behaviour of the agent is directly learned to maximize an expected reward. We follow [18,5] in using stochastic policy gradient, which allows us to use a non-differentiable reward function, directly corresponding to the final success criterion used during evaluation.

Method
In this section, we first define the pose estimation task and briefly review the pose estimation pipeline from [3,12,4]. We then continue to describe PoseAgent, our reinforcement learning agent, designed to solve the same problem. Finally, we discuss how to train our agent, introducing our new, efficient training algorithm.

Pose Estimation Pipeline
We begin by describing the object pose estimation task. Given an RGB-D image x we are interested in localizing a specific, known, rigid object ( Fig. 1a). We assume that exactly one object instance is present in the scene. Our goal is to estimate the true pose H * of the instance, i.e. its position in space as well as its orientation. The pose has a total of six degrees of freedom, three for translation and three for rotation. We define the pose as the rigid body transformation that maps a point from the local coordinate system of the object to the coordinate system of the camera.
Our method is based on the work of Krull et al. [12]. As in [12], we use an intermediate image representation called object coordinates. By looking at small patches of the RGB-D input image, a random forest (Fig. 1b) provides two predictions for every pixel i. Each tree predicts an object probability p i ∈ [0, 1] as well as a set of object coordinates y i (Fig. 1c). The object probability p i describes whether the pixel is believed to be part of the object or not. The object coordinates y i represent the predicted position of the pixel on the surface of the object, i.e. its 3D coordinates in the local coordinate system of the object.
Again following [12], we use these forest predictions in a RANSAC-inspired sampling scheme to generate pose hypotheses. We repeatedly sample three pixels from the image according to the object probabilities p i . By combining the predicted object coordinate y i with the camera coordinates of the pixels (calculated from the depth channel of the input image), we obtain three 3D-3D correspondences. We calculate a pose hypothesis from these correspondences using the Kabsch algorithm [11]. We sample a fixed number N of hypotheses, which are combined in hypothesis pool Fig. 1d). The upper index denotes time steps which we will use later in our algorithm.
Krull et al. [12] proposed the following rigid scheme for pose optimization. All hypotheses are scored and the 25 top-scoring hypotheses are refined. Then, the refined hypotheses are scored again, and the best scoring hypothesis is returned as the final pose estimate of the algorithm. The agent selects a hypotheses H t a t by sampling from the policy π a t |S t ; θ . c) The selected hypothesis is refined. d) If refinement budget is left, the refinement phase continues. If the budget is exhausted a final selection is made. e) The final selection is made by sampling from the policy π a T |S T ; θ . f) The selected pose H T a T is output as final pose estimate.
Our paper focuses on improving the process by which the correct pose is found, starting from the same initial hypothesis pool. We propose to use an RL agent (Fig. 1e) to dynamically decide which hypothesis to refine next, in order to make most efficient use of a given computational budget. When its budget is exhausted, the agent selects a final pose estimate (Fig. 1f).

PoseAgent
We now describe our RL agent, PoseAgent, and how it performs inference. An overview of the process can be found in Fig. 3. The agent operates in two phases: (i) the refinement phase, in which the agent chooses individual hypotheses to undergo the expensive refinement step; and (ii) the final decision phase, in which it has to decide which pose should be selected as final output. In the following, we discuss both phases in detail.
Inference begins with the refinement phase. The pose agent starts with a pool H 0 = (H 0 1 . . . H 0 N ) of hypotheses which have been generated as described in Sec. 3.1, and a fixed budget B 0 of possible refinement steps.
At each time step t, the agent chooses one hypothesis index a t , which we call an action. The chosen hypothesis is refined and the next time step begins. We limit the maximum number of times the agent may choose the same hypothesis for refinement to τ max . Hence, over time, the pool of actions (resp. hypotheses) the agent may choose for refinement decreases. We denote the set of possible actions A t = {a ∈ {1, . . . , N }|τ t a < τ max }, where τ t a denotes how many times hypothesis a has been refined before time t. Subsequently, the agent modifies the hypothesis pool by refining hypothesis H t+1 is the refinement function. All other hypotheses remain unchanged H t+1 a = H t a ∀a = a t . We perform refinement as follows (see also [12]). We render the object in pose H t a t . Each pixel within the rendered mask is tested for being an inlier. 1 All inlier pixels are used to re-calculate the pose with the Kabsch algorithm. We repeat this procedure multiple times for the single, chosen hypothesis until the number of inlier pixels stops increasing or until the number m t of executed refinement steps exceeds a maximum m max . The budget is decreased by the number of refinement steps performed, B t+1 = B t −m t . The agent proceeds choosing refinement actions until B t < m max , in which case further refinement may exceed the total budget B 0 of refinement steps.
When this point has been reached, the refinement phase terminates, and the agent enters the final decision phase in which the agent chooses a hypothesis as the final output. We denote the final action as a T ∈ {1 . . . N } and the final pose estimate asH = H T a T . The agent receives a reward of r = 1 in case the pose is correct or a negative reward of r = −1 otherwise. We use the pose correctness criterion from [9].
In the following, we describe how the agent makes its decisions. During both, the refinement phase and the final decision phase, the agent chooses from the hypothesis pool. We describe the agent behaviour by a probability distribution π (a t |S t ; θ) referred to as "policy". Given the current state S t , which contains information about the hypothesis pool and the input image x, the agent selects a hypothesis by drawing a sample from the policy. The vector θ of learnable parameters consists of CNN weights (described in Sec. 3.2.3). We will first give details on the state space S t before explaining policy π (a t |S t ; θ).

State Space
We model our state space in a way that allows us to use our new, efficient training algorithm, described in Sec. 3.3.1. We assume that the current state S t of the hypothesis pool decomposes as S t = (s t 1 , . . . s t N ), where s t a will be called the state of hypothesis H t a . The state of an hypothesis contains the original input image x, the forest prediction z for the image, the pose hypothesis H t a , as well as a vector f t a of additional context features of the hypothesis (see Sec. 3.2.3). In summary, this gives s t a = (x, z, H t a , f t a ).

Policy
Our agent makes its decisions using a softmax policy. The probability of choosing a particular action a t during the re-1 A pixel i is tested for being an inlier for pose H by transforming its predicted object coordinates y i to camera space using H. If the Euclidean distance between resulting camera coordinates and the observed coordinates at the pixel is below a threshold the pixel is considered an inlier. finement phase is given by where E(s t a ; θ) will be called the energy of the state s t a . We will abbreviate it as E t a = E(s t a ; θ). The energy of a state in the softmax policy is a measure of how desirable it is for the agent to refine the hypothesis. We use the same policy in the final decision phase, but with a different energy E (s t a ; θ) abbreviated by E t a . We use a CNN to predict both energies, E t a and E t a . In the next section, we discuss the CNN architecture and how it governs the behaviour of the agent.

CNN Architecture
We give an overview of the CNN architecture used in this work in Fig. 3. As in [12], the CNN compares rendered and observed images. We use the same six input channels as in [12], namely: the rendered depth channel, the observed depth channel, a rendered segmentation channel, the object probability channel, a depth mask, and a single channel holding the difference between object coordinates.
There are however two major differences in our CNN compared to the one used in [12], highlighted in Fig. 3. Firstly, while Krull et al. predict a single energy value of a pose, we jointly predict two separate energy values: one energy E t a for the refinement phase and one energy E t a for the final decision phase.
Secondly, we input additional features to the network by concatenating them to the first fully connected layer. The features are: The number of times the hypothesis has already been selected for refinement. The distance the hypothesis has moved during its last refinement. The average distance of the hypothesis before refinement to all other hypotheses in the original pool.

Policy Gradient Training
We will now discuss the training procedure for our PoseAgent. First, we will give a general introduction of policy gradient training, and then apply the approach to the PoseAgent. Finally, we will introduce an efficient algorithm that greatly reduces variance during training and makes training feasible.
The goal of the training is the maximization of the expected reward E [R]. This expected value depends on the environment as well as on the policy of our agent. In stochastic policy gradient methods one attempts to approximate the gradient with respect to the policy parameters θ. Note that since we are dealing with the expected value it becomes possible calculate derivatives, even if the reward function itself is non differentiable. We can write the derivative of the expected reward with respect to each parameter θ j in θ as where p(s 1:T , a 1:T ; θ) is the probability of a particular sequence of states s 1:T = (s 1 . . . s T ) and actions a 1:T = (a 1 . . . a T ) to occur. Because of the Markov property of the environment, it is possible to decompose the probability and rewrite it as Let us now consider the PoseAgent, which uses the softmax policy. In this case, we can calculate the derivative where the derivative ∂ ∂θj E t a of the energy can be calculated using the standard back propagation algorithm.

Efficient Training
We will now introduce our training algorithm (Algorithm 1), to dramatically reduce the variance of estimated gradients training. The basic idea is to make use of the special decomposable structure of the state space and our policy. The advantage of our algorithm compared to the naïve implementation is illustrated in Fig. 4.
To motivate the algorithm consider the following. Starting from a hypothesis pool H 0 = (H 0 1 . . . H 0 N )), only a finite number of different hypothesis states s τ a |τ ∈ {0, . . . τ max } can occur during a run of our PoseAgent. Here, s τ a = (H τ a , f τ a ) shall denote the state of hypothesis a after it has been refined τ times.
The algorithm pre-computes all possibly occurring hypothesis states s τ a , and predicts the corresponding energy values E τ a using the CNN. While this comes with some computational expense, it allows us to rapidly sample large numbers of sequences without having to re-evaluate the energy function.
To illustrate why this is possible, let us now reconsider Eq. (5). We can also write it as ln π a t |S t ; θ , (7) If we combine Eq. (6) and Eq. (4) we can rewrite the derivative of our expected reward as ln π a t |S t k ; θ r k .
Here, 1 s t a,k = s τ a is the indicator function. It has the value 1 whenever the sampled hypothesis state s t a,k corresponds to the hypothesis state s τ a , and the value 0 in any other case.
Our algorithm works by first calculating the inner sums over k and t in Eq (8) for all τ and a. We compute these sums with a single iteration over all sequences k and all time steps t by accumulating the accordant summands in the entries D(τ t k , a) of table D. The accumulation of these values is computationally cheap, because it does not not require any rendering or involvement of the CNN.
This structure allows us to increase the number of sampled sequences M without much computational cost. The algorithm can process an arbitrary amount of sequences using only a single back propagation pass of the CNN for each possible hypothesis state s t a . In a naive implementation, the number of required forward-backward passes would increase linearly with the number of sampled sequences.
Let us look at the algorithm in detail. It consists of three parts: During the Initialization Phase we generated the original hypothesis pool as described in 3.1. Then, we refine all of hypotheses τ max times and predict the energy values E τ a and E τ a for all of them. During the Sampling Phase we sample sequences (s 1:T k , a 1:T k ) using the precomputed energies. We observe the reward r k for each sequence and accumulate ∂ ∂E τ a ln π (a t |S t k ; θ) r k . This corresponds to the inner sums in Eq. (8) and has a very low computational cost per sequence.
During the final Gradient Update Phase we once more process each of the hypothesis states s τ a with the CNN and use the accumulated derivatives from the previous phase to update the derivatives of the CNN parameters θ. This corresponds to the outer sums in Eq. (8).

Experiments
In the following we will describe the experiments to compare our method to the baseline system from [12]. Our experiments confirm, that our learned inference procedure is able to use its budget in a more efficient way, compared to the baseline from [12]. It outperforms the baseline system, while using on average a smaller number refinement steps.
Additionally we will describe an experiment regarding the efficiency of our training algorithm compared to a naïve implementation of the REINFORCE algorithm. We find, that our algorithm can dramatically reduce the gradient variance during training.
We conducted our experiments on the dataset introduced in [13]. It features six RGB-D sequences of hand held sometimes strongly occluded objects.
We use the same division suggested by [12], reserving the sequences samurai 1, cat 1 and toolbox 1 for training and validation, while testing on the remaining samurai 2, cat 2 and toolbox 2 sequences.

Training and Validation Procedure
We train our system on the samurai 1 sequence. As [12] we omit the first 400 frames in the training sequence to achieve a higher percentage of occluded images. We train our system with two different parameter settings: Using a hypothesis pool size of N = 210, which is the setting used in [12], and a larger pool size of N = 420.
To determine the adequate size of the budget B of refinement steps, we ran the system from [12] on our validation set and determined the average number of refinement steps it used. We set our budget during training to the resulting number B = 77.
During training we allow a hypothesis to be chosen τ max = 3 times for refinement. We set the maximum number of refinement steps per iteration to m max = 10.
Using stochastic gradient descent, we go through our training images in random order and run Algorithm 1 to approximate the gradient. We sample M = 50k sequences for every image, and use an additional 50k to estimates a baseline b. We perform a parameter update after every image. Starting with an initial learning rate of λ 0 = 25 · 10 −4 , we reduce it according to λ l = λ 0 /(1 + lν), with ν = 0.01. We use a fixed momentum of 0.9.
We skip all images in which none of the hypotheses from the pool leads to a correct pose after being refined τ max times. The reason for this is that such an image will always produce a reward of r = −1 and cannot provide meaningful information.
We also skip all images in which more than 10% of the hypotheses from the pool lead to a correct pose. Such images contribute little, because they are easily solved. We augment our training data, by randomly deciding for every image whether to mirror the CNN input along the x-and y-axis.
We run the training procedure for 96 hours on an Intel E5-2450 2.10GHz with Nvidia Tesla K20x GPU and save a snapshot every 50 training images. To avoid over-fitting, we test these saved snapshots on our validation set and choose the model with the highest accuracy.
In order to reduce the computational time during validation, we considered only images in which the object was at least 5% occluded 2 .

Testing Conditions
We compared both versions of our model, using N = 210 and N = 420, against the system of [12] using the corresponding pool size. In all experiments with the baseline method [12], we use the identical CNN with the original weights trained by Krull et al. in [12] on the samurai 2 sequence. This is the network that [12] reports the best results for.
Apart from the pool size, we used the identical testing conditions as in [12], including the same random forest originally trained in [3]. To classify a pose in correct or false we use the same point-distance-based criterion used in [12]. A pose is considered correct, when the average distance between the vertices of the 3D model in the ground truth pose and the evaluated pose is below a threshold.
While the number of refinement steps in our setting is restricted, the method of [12] does not provide any guarantees on how many refinement steps are used. To ensure a fair comparison, we first ran the method from [12] and recorded the average number of refinement steps that it required on each test sequence. When running our method, we set the budget for each sequence to this recorded value, making sure that PoseAgent could never use more refinement steps than [12]. The total average number of refinement steps required by both methods can be found Table 1 and 2.
We evaluate our method using different parameters for τ max and m max , so that τ max · m max ≈ 30. Meaning that a each pose can have an approximate maximum of 30 refinement steps. A higher value of τ max (and lower value of m max ) means that PoseAgent can make more fine grained decisions on where to spend its budget. We use the following combinations for the two values (τ max =3,m max = 10), (τ max = 5, m max = 6), (τ max = 6, m max = 5), (τ max = 7, m max = 4).

Results
The results of our experiments can be seen in Table 1 and 2. PoseAgent is able to improve the best published results on the dataset by a total of 10.56%. When we compare our method only to the baseline working on the same hypothesis pool size we are still able to outperform it. With the original pool size of N = 210 by 2.12% and the increased pool size of N = 420 by 2.59%.
Note, that the budget is set in a way, that is extremely restrictive, ensuring that PoseAgent can never use more refinement steps than [12] uses on average.
In both settings (N = 210 and N = 420) there appears to be a trend, that an increase of τ max , which corresponds to a more fine grained control of PoseAgent, leads to an improvement in accuracy. The only exception here is τ max = 7 in the N = 210 setting. It should be noted that PoseAgent was trained with a different setting of τ max = 3 and was able to generalize to the different settings used during testing.

Additional Experiments on the Efficiency of the Training Algorithm
In order to investigate the efficiency of out training algorithm compared to a naïve implementation of the REIN-FORCE algorithm, we conducted the following experiment: We ran our training algorithm as well as the naïve implementation 100 times on a single training image without updating the network.
To estimate the variance of the gradient, we calculated the standard deviation of 1000 randomly selected elements from the resulting gradient vector of the CNN and averaged them. We recorded the required computation time to process the image on an Intel E5-2450 2.10GHz with Nvidia Tesla K20x GPU.
The To keep the computation time in reasonable limits we used a reduced setting with a hypothesis pool size of N = 21 for both methods.
As can be seen in Fig. 4 our algorithm allows us to reduce variance greatly with almost no increase in computation time.

Conclusion
We have demonstrated a method learn the algorithmic inference procedure in a pose estimation system using a policy gradient method. Our system learns to make efficient  use of a given budget and is able to outperform the original system, while using on average less computational resources. We have presented an efficient algorithm for the gradient approximation during training. The algorithm is able to sharply reduce gradient variance, without a significant increase in computation time. We see multiple interesting future directions of research in the context of our system. (i) One could investigate a soft version of PoseAgent, which is not working with a fixed budget, but can instead decide what is the appropriate time to stop. In such systems the used computational budget can be part of the reward function. (ii) The sequential structure of the current system does not allow simple parallelization, but a PoseAgent that learns to do inference making use of multiple computational cores could be conceived.