Question: Represent the scientific article to find a short TLDR

Question: Policy gradient methods have enjoyed great success in deep reinforcement learning but suffer from high variance of gradient estimates. The high variance problem is particularly exasperated in problems with long horizons or high-dimensional action spaces. To mitigate this issue, we derive a bias-free action-dependent baseline for variance reduction which fully exploits the structural form of the stochastic policy itself and does not make any additional assumptions about the MDP. We demonstrate and quantify the benefit of the action-dependent baseline through both theoretical analysis as well as numerical , including an analysis of the suboptimality of the optimal state-dependent baseline. The is a computationally efficient policy gradient algorithm, which scales to high-dimensional control problems, as demonstrated by a synthetic 2000-dimensional target matching task. Our experimental indicate that action-dependent baselines allow for faster learning on standard reinforcement learning benchmarks and high-dimensional hand manipulation and synthetic tasks. Finally, we show that the general idea of including additional information in baselines for improved variance reduction can be extended to partially observed and multi-agent tasks. Deep reinforcement learning has achieved impressive in recent years in domains such as video games from raw visual inputs BID10, board games, simulated control tasks BID16, and robotics ). An important class of methods behind many of these success stories are policy gradient methods BID28 BID22 BID5 BID18 BID11, which directly optimize parameters of a stochastic policy through local gradient information obtained by interacting with the environment using the current policy. Policy gradient methods operate by increasing the log probability of actions proportional to the future rewards influenced by these actions. On average, actions which perform better will acquire higher probability, and the policy's expected performance improves. A critical challenge of policy gradient methods is the high variance of the gradient estimator. This high variance is caused in part due to difficulty in credit assignment to the actions which affected the future rewards. Such issues are further exacerbated in long horizon problems, where assigning credits properly becomes even more challenging. To reduce variance, a "baseline" is often employed, which allows us to increase or decrease the log probability of actions based on whether they perform better or worse than the average performance when starting from the same state. This is particularly useful in long horizon problems, since the baseline helps with temporal credit assignment by removing the influence of future actions from the total reward. A better baseline, which predicts the average performance more accurately, will lead to lower variance of the gradient estimator. The key insight of this paper is that when the individual actions produced by the policy can be decomposed into multiple factors, we can incorporate this additional information into the baseline to further reduce variance. In particular, when these factors are conditionally independent given the current state, we can compute a separate baseline for each factor, whose value can depend on all quantities of interest except that factor. This serves to further help credit assignment by removing the influence of other factors on the rewards, thereby reducing variance. In other words, information about the other factors can provide a better evaluation of how well a specific factor performs. Such factorized policies are very common, with some examples listed below.• In continuous control and robotics tasks, multivariate Gaussian policies with a diagonal covariance matrix are often used. In such cases, each action coordinate can be considered a factor. Similarly, factorized categorical policies are used in game domains like board games and Atari.• In multi-agent and distributed systems, each agent deploys its own policy, and thus the actions of each agent can be considered a factor of the union of all actions (by all agents). This is particularly useful in the recent emerging paradigm of centralized learning and decentralized execution BID2 BID9. In contrast to the previous example, where factorized policies are a common design choice, in these problems they are dictated by the problem setting. We demonstrate that action-dependent baselines consistently improve the performance compared to baselines that use only state information. The relative performance gain is task-specific, but in certain tasks, we observe significant speed-up in the learning process. We evaluate our proposed method on standard benchmark continuous control tasks, as well as on a high-dimensional door opening task with a five-fingered hand, a synthetic high-dimensional target matching task, on a blind peg insertion POMDP task, and a multi-agent communication task. We believe that our method will facilitate further applications of reinforcement learning methods in domains with extremely highdimensional actions, including multi-agent systems. Videos and additional of the paper are available at https://sites.google.com/view/ad-baselines. Three main classes of methods for reinforcement learning include value-based methods BID26, policy-based methods BID28 BID5 BID18, and actor-critic methods BID6 BID14 BID11. Valuebased and actor-critic methods usually compute a gradient of the objective through the use of critics, which are often biased, unless strict compatibility conditions are met BID22 BID6. Such conditions are rarely satisfied in practice due to the use of stochastic gradient methods and powerful function approximators. In comparison, policy gradient methods are able to compute an unbiased gradient, but suffer from high variance. Policy gradient methods are therefore usually less sample efficient, but can be more stable than critic-based methods BID1.A large body of work has investigated variance reduction techniques for policy gradient methods. One effective method to reduce variance without introducing bias is through using a baseline, which has been widely studied BID21 BID27 BID3. However, fully exploiting the factorizability of the policy probability distribution to further reduce variance has not been studied. Recently, methods like Q-Prop BID4 ) make use of an action-dependent control variate, a technique commonly used in Monte Carlo methods and recently adopted for RL. Since Q-Prop utilizes off-policy data, it has the potential to be more sample efficient than pure on-policy methods. However, Q-prop is significantly more computationally expensive, since it needs to perform a large number of gradient updates on the critic using the off-policy data, thus not suitable with fast simulators. In contrast, our formulation of action-dependent baselines has little computational overhead, and improves the sample efficiency compared to on-policy methods with state-only baseline. The idea of using additional information in the baseline or critic has also been studied in other contexts. Methods such as Guided Policy Search BID13 and variants train policies that act on high-dimensional observations like images, but use a low dimensional encoding of the problem like joint positions during the training process. Recent efforts in multi-agent systems BID2 BID9 ) also use additional information in the centralized training phase to speed-up learning. However, using the structure in the policy parameterization itself to enhance the learning speed, as we do in this work, has not been explored. In this section, we establish the notations used throughout this paper, as well as basic for policy gradient methods, and variance reduction via baselines. This paper assumes a discrete-time Markov decision process (MDP), defined by (S, A, P, r, ρ 0, γ), in which S ⊆ R n is an n-dimensional state space, A ⊆ R m an m-dimensional action space, P: S × A × S → R + a transition probability function, r: S × A → R a bounded reward function, ρ 0: S → R + an initial state distribution, and γ ∈ a discount factor. The presented models are based on the optimization of a stochastic policy π θ: S × A → R + parameterized by θ. Let η(π θ) denote its expected return: DISPLAYFORM0, where τ = (s 0, a 0, . . .) denotes the whole trajectory, s 0 ∼ ρ 0 (s 0), a t ∼ π θ (a t |s t), and s t+1 ∼ P(s t+1 |s t, a t) for all t. Our goal is to find the optimal policy arg max θ η(π θ). We will useQ(s t, a t) to describe samples of cumulative discounted return, and Q(a t, s t) to describe a function approximation ofQ(s t, a t). We will use "Q-function" when describing an abstract action-value function. For a partially observable Markov decision process (POMDP), two more components are required, namely Ω, a set of observations, and O: S × Ω → R ≥0, the observation probability distribution. In the fully observable case, Ω ≡ S. Though the analysis in this article is written for policies over states, the same analysis can be done for policies over observations. An important technique used in the derivation of the policy gradient is known as the score function (SF) estimator BID28, which also comes up in the justification of baselines. Suppose that we want to estimate ∇ θ E x [f (x)] where x ∼ p θ (x), and the family of distributions {p θ (x): θ ∈ Θ} has common support. Further suppose that log p θ (x) is continuous in θ. In this case we have DISPLAYFORM0 The Policy Gradient Theorem BID22 states that DISPLAYFORM0 For convenience, define ρ π (s) = ∞ t=0 γ t p(s t = s) as the state visitation frequency, and DISPLAYFORM1 We can rewrite the above equation (with abuse of notation) as DISPLAYFORM2 It is further shown that we can reduce the variance of this gradient estimator without introducing bias by subtracting off a quantity dependent on s t fromQ(s t, a t) BID28 BID3. See Appendix A for a derivation of the optimal state-dependent baseline. DISPLAYFORM3 This is valid because, applying the SF estimator in the opposite direction, we have DISPLAYFORM4 In practice there can be rich internal structure in the policy parameterization. For example, for continuous control tasks, a very common parameterization is to make π θ (a t |s t) a multivariate Gaussian with diagonal variance, in which case each dimension a i t of the action a t is conditionally independent of other dimensions, given the current state s t. Another example is when the policy outputs a tuple of discrete actions with factorized categorical distributions. In the following subsections, we show that such structure can be exploited to further reduce the variance of the gradient estimator without introducing bias by changing the form of the baseline. Then, we derive the optimal action-dependent baseline for a class of problems and analyze the suboptimality of non-optimal baselines in terms of variance reduction. We then propose several practical baselines for implementation purposes. We conclude the section with the overall policy gradient algorithm with action-dependent baselines for factorized policies. We provide an exposition for situations when the conditional independence assumption does not hold, such as for stochastic policies with general covariance structures, in Appendix E, and for compatibility with other variance reduction techniques in Appendix F. In the following, we analyze action-dependent baselines for policies with conditionally independent factors. For example, multivariate Gaussian policies with a diagonal covariance structure are commonly used in continuous control tasks. Assuming an m-dimensional action space, we have DISPLAYFORM0 In this case, we can set b i, the baseline for the ith factor, to depend on all other actions in addition to the state. Let a t denote all dimensions other than i in a t and denote the ith baseline by b i (s t, a −i t). Due to conditional independence and the score function estimator, we have DISPLAYFORM0 Hence we can use the following gradient estimator DISPLAYFORM1 This is compatible with advantage function form of the policy gradient: DISPLAYFORM2 DISPLAYFORM3 Note that the policy gradient now comprises of m component policy gradient terms, each with a different advantage term. In Appendix E, we show that the methodology also applies to general policy structures (for example, a Gaussian policy with a general covariance structure), where the conditional independence assumption does not hold. The is bias-free albeit different baselines. In this section, we derive the optimal action-dependent baseline and show that it is better than the state-only baseline. We seek the optimal baseline to minimize the variance of the policy gradient estimate. First, we write out the variance of the policy gradient under any action-dependent baseline. Let us define z i:= ∇ θ log π θ (a which translates to meaning that different subsets of parameters strongly influence different action dimensions or factors. We note that this assumption is primarily for the theoretical analysis to be clean, and is not required to run the algorithm in practice. In particular, even without this assumption, the proposed baseline is bias-free. When the assumption holds, the optimal actiondependent baseline can be analyzed thoroughly. Some examples where these assumptions do hold include multi-agent settings where the policies are conditionally independent by construction, cases where the policy acts based on independent components BID0 of the observation space, and cases where different function approximators are used to control different actions or synergies BID23 BID24 without weight sharing. The optimal action-dependent baseline is then derived to be: DISPLAYFORM0 See Appendix B for the full derivation. Since the optimal action-dependent baseline is different for different action coordinates, it is outside the family of state-dependent baselines barring pathological cases. How much do we reduce variance over a traditional baseline that only depends on state? We use the following notation: DISPLAYFORM0 DISPLAYFORM1 Then, using Equation (Appendix C), we show the following improvement with the optimal action-dependent baseline: DISPLAYFORM2 See Appendices C and D for the full derivation. We conclude that the optimal action-dependent baseline does not degenerate into the optimal state-dependent baseline. Equation FORMULA1 states that the variance difference is a weighted sum of the deviation of the per-component score-weighted marginalized Q (denoted Y i) from the component weight (based on score only, not Q) of the overall aggregated marginalized Q values (denoted j Y j). This suggests that the difference is particularly large when the Q function is highly sensitive to the actions, especially along those directions that influence the gradient the most. Our empirical in Section 5 additionally demonstrate the benefit of action-dependent over state-only baselines. Using the previous theory, we now consider various baselines that could be used in practice and their associated computational cost. Marginalized Q baseline Even though the optimal state-only baseline is known, it is rarely used in practice BID1. Rather, for both computational and conceptual benefit, the choice of b(DISPLAYFORM0 which is the action-dependent analogue. In particular, when log probability of each policy factor is loosely correlated with the action-value function, then the proposed baseline is close to the optimal baseline. DISPLAYFORM1 This has the added benefit of requiring learning only one function approximator, for estimating Q(s t, a t), and implicitly using it to obtain the baselines for each action coordinate. That is, Q(s t, a t) is a function approximating samplesQ(s t, a t).Monte Carlo marginalized Q baseline After fitting Q π θ (s t, a t) we can obtain the baselines through Monte Carlo estimates: DISPLAYFORM2 where α j ∼ π θ (a i t |s t) are samples of the action coordinate i. In general, any function may be used to aggregate the samples, so long as it does not depend on the sample value a i t. For instance, for discrete action dimensions, the sample max can be computed instead of the mean. Mean marginalized Q baseline Though we reduced the computational burden from learning m functions to one function, the use of Monte Carlo samples can still be computationally expensive. In particular, when using deep neural networks to approximate the Q-function, forward propagation through the network can be even more computationally expensive than stepping through a fast simulator (e.g. MuJoCo). In such settings, we further propose the following more computationally practical baseline: DISPLAYFORM3 DISPLAYFORM4 t is the average action for coordinate i. The final practical algorithm for fully factorized policies is as follows. Require: number of iterations N, batch size B, initial policy parameters θ Initialize action-value function estimate Q π θ (s t, a t) ≡ 0 and policy DISPLAYFORM0, ∀t Perform a policy update step on θ usingÂ i (s t, a t) [Equation FORMULA10] Update action-value function approximation with current batch: Q π θ (s t, a t) end forComputing the baseline can be done with either proposed technique in Section 4.4. A similar algorithm can be written for general policies (Appendix E), which makes no assumptions on the conditional independence across action dimensions. Continuous control benchmarks Firstly, we present the of the proposed action-dependent baselines on popular benchmark tasks. These tasks have been widely studied in the deep reinforcement learning community BID1 BID4 BID17. The studied tasks include the hopper, half-cheetah, and ant locomotion tasks simulated in MuJoCo BID25.1 In addition to these tasks, we also consider a door opening task with a high-dimensional multi-fingered hand, introduced in BID16 to study the effectiveness of the proposed approach in high-dimensional tasks. FIG0 presents the learning curves on these tasks. We compare the action-dependent baseline with a baseline that uses only information about the states, which is the most common approach in the literature. We observe that the action-dependent baselines perform consistently better. A popular baseline parameterization choice is a linear function on a small number of non-linear features of the state BID1, especially for policy gradient methods. In this work, to enable a fair comparison, we use a Random Fourier Feature representation for the baseline BID15 BID17. The features are constructed as: y(x) = sin(1 ν P x + φ) where P is a matrix with each element independently drawn from the standard normal distribution, φ is a random phase shift in [−π, π) and, and ν is a bandwidth parameter. These features approximate the RKHS features under an RBF kernel. Using these features, the baseline is parameterized as b = w T y(x) where x are the appropriate inputs to the baseline, and w are trainable parameters. P and φ are not trained in this parameterization. Such a representation was chosen for two reasons: (a) we wish to have the same number of trainable parameters for all the baseline architectures, and not have more parameters in the action-dependent case (which has a larger number of inputs to the baseline); (b) since the final representation is linear, it is possible to accurately estimate the optimal parameters with a Newton step, thereby alleviating the from confounding optimization issues. For policy optimization, we use a variant of the natural policy gradient method as described in BID17. See Appendix G for further experimental details. Choice of action-dependent baseline form Next, we study the influence of computing the baseline by using empirical averages sampled from the Q-function versus using the mean-action of the action-coordinate for computing the baseline (both described in 4.4). In our experiments, as shown in Figure 2 we find that the two variants perform comparably, with the latter performing slightly better towards the end of the learning process. This suggests that though sampling from the Q-function might provide a better estimate of the conditional expectation in theory, function approximation from finite samples injects errors that may degrade the quality of estimates. In particular, sub-sampling from the Q-function is likely to produce better if the learned Q-function is accurate for a large fraction of the action space, but getting such high quality approximations might be hard in practice. High-dimensional action spaces Intuitively, the benefit of the action-dependent baseline can be greater for higher dimensional problems. We show this effect on a simple synthetic example called m-DimTargetMatching. The example is a one-step MDP comprising of a single state, S = {0}, an m-dimensional action space, A = R m, and a fixed vector c ∈ R m. The reward is given as the negative squared 2 loss of the action vector, r(s, a) = − a − c 2 2. The optimal action is thus to match Figure 2: Variants of the action-dependent baseline that use: (i) sampling from the Q-function to estimate the conditional expectation; (ii) Using the mean action to form a linear approximation to the conditional expectation. We find that both variants perform comparably, with the latter being more computationally efficient. the given vector by selecting a = c. The for the demonstrative example are shown in Table 1, which shows that the action-dependent baseline successfully improves convergence more for higher dimensional problems than lower dimensional problems. Due to the lack of state information, the linear baseline reduces to whitening the returns. The action-dependent baseline, on the other hand, allows the learning algorithm to assess the advantage of each individual action dimension by utilizing information from all other action dimensions. Additionally, this experiment demonstrates that our algorithm scales well computationally to high-dimensional problems. Table 1: Shown are the for the synthetic high-dimensional target matching task (5 seeds), for 12 to 2000 dimensional action spaces. At high dimensions, the linear feature action-dependent baseline provides notable and consistent variance reduction, as compared to a linear feature baseline, ing in around 10% faster convergence. For the corresponding learning curves, see Appendix G.Partially observable and multi-agent tasks Finally, we also consider the extension of the core idea of using global information, by studying a POMDP task and a multi-agent task. We use the blind peg-insertion task which is widely studied in the robot learning literature BID12. The task requires the robot to insert the peg into the hole (slot), but the robot is blind to the location of the hole. Thus, we expect a searching behavior to emerge from the robot, where it learns that the hole is present on the table and performs appropriate sweeping motions till it is able to find the hole. In this case, we consider a baseline that is given access to the location of the hole. We observe that a baseline with this additional information enables faster learning. For the multi-agent setting, we analyze a two-agent particle environment task in which the goal is for each agent to reach their goal, where their goal is known by the other agent and they have a continuous communication channel. Similar training procedures have been employed in recent related works BID9;. FIG2 shows that including the inclusion of information from other agents into the action-dependent baseline improves the training performance, indicating that variance reduction may be key for multi-agent reinforcement learning. An action-dependent baseline enables using additional signals beyond the state to achieve bias-free variance reduction. In this work, we consider both conditionally independent policies and general policies, and derive an optimal action-dependent baseline. We provide analysis of the variance DISPLAYFORM0 (a) Success percentage on the blind peg insertion task. The policy still acts on the observations and does not know the hole location. However, the baseline has access to this goal information, in addition to the observations and action, and helps to speed up the learning. By comparison, in blue, the baseline has access only to the observations and actions. reduction improvement over non-optimal baselines, including the traditional optimal baseline that only depends on state. We additionally propose several practical action-dependent baselines which perform well on a variety of continuous control tasks and synthetic high-dimensional action problems. The use of additional signals beyond the local state generalizes to other problem settings, for instance in POMDP and multi-agent tasks. In future work, we propose to investigate related methods in such settings on large-scale problems. We provide a derivation of the optimal state-dependent baseline, which minimizes the variance of the policy gradient estimate, and is based in (, Theorem 8). More precisely, we minimize the trace of the covariance of the policy gradient; that is, the sum of the variance of the components of the vectors. Recall the policy gradient expression with a state-dependent baseline: DISPLAYFORM0 Denote g to be the associated random variable, that is, ∇ θ η(π θ) = E ρπ,π [g]: DISPLAYFORM1 The variance of the policy gradient is: DISPLAYFORM2 Note that E [η(π θ)]) contains a bias-free term, by the score function argument, which then does not affect the minimizer. Terms which do not depend on b(s t) also do not affect the minimizer. DISPLAYFORM3 We derive the optimal action-dependent baseline, which minimizes the variance of the policy gradient estimate. First, we write out the variance of the policy gradient under any action-dependent baseline. Recall the following notations: we define z i:= ∇ θ log π θ (a i t |s t) and the component policy gradient: DISPLAYFORM0 Denote g i to be the associated random variables: DISPLAYFORM1 such that DISPLAYFORM2 Recall the following assumption: DISPLAYFORM3 which translates to meaning that different subsets of parameters strongly influence different action dimensions or factors. This is true in case of distributed systems by construction, and also true in a single agent system if different action coordinates are strongly influenced by different policy network channels. Under this assumption, we have: DISPLAYFORM4 DISPLAYFORM5 DISPLAYFORM6 where we denote the mean correction term DISPLAYFORM7, and thus does not affect the optimal value. The overall variance is minimized when each component variance is minimized. We now derive the optimal baselines b * i (s t, a −i t) which minimize each respective component. DISPLAYFORM8 + E ρπ,a DISPLAYFORM9 Having written down the expression for variance under any action-dependent baseline, we seek the optimal baseline that would minimize this variance. DISPLAYFORM10 The optimal action-dependent baseline is: DISPLAYFORM11 We now turn to quantifying the reduction in variance of the policy gradient estimate under the optimal baseline derived above. Let Var * (i g i) denote the variance ing from the optimal action-dependent baseline, and let Var(i g i) denote the variance ing from another baseline DISPLAYFORM0, which may be suboptimal or action-independent. Recall the notation: DISPLAYFORM1 DISPLAYFORM2 DISPLAYFORM3 Finally, define the variance improvement DISPLAYFORM4 Using these definitions, the variance can be re-written as: DISPLAYFORM5 Furthermore, the variance of the gradient with the optimal baseline can be written as DISPLAYFORM6 The difference in variance can be calculated as: DISPLAYFORM7 DISPLAYFORM8 Using the notation from Appendix C and working off of Equation FORMULA1, we have: DISPLAYFORM0 DISPLAYFORM1 E BASELINES FOR GENERAL ACTIONSIn the preceding derivations, we have assumed policy actions are conditionally independent across dimensions. In the more general case, we only assume that there are m factors a which altogether forms the action a t. Conditioned on s t, the different factors form a certain directed acyclic graphical model (including the fully dependent case). Without loss of generality, we assume that the following factorization holds: DISPLAYFORM2 where f (i) denotes the indices of the parents of the ith factor. Let D(i) denote the indices of descendants of i in the graphical model (including i itself). In this case, we can set the ith baseline to be b i (s t, a DISPLAYFORM3), where [m] = {1, 2, . . ., m}. In other words, the ith baseline can depend on all other factors which the ith factor does not influence. The overall gradient estimator is given by DISPLAYFORM4 In the most general case without any conditional independence assumptions, we have f (i) = {1, 2, . . ., i − 1}, and D(i) = {i, i + 1, . . ., m}. The above equation reduces to DISPLAYFORM5 The above analysis for optimal baselines and variance suboptimality transfers also to the case of general actions. The applicability of our techniques to general action spaces may be of crucial importance for many application domains where the conditional independence assumption does not hold up, such as language tasks and other compositional domains. Even in continuous control tasks, such as hand manipulation, and many other tasks where it is common practice to use conditionally independent factorized policies, it is reasonable to expect training improvement from policies without a full conditionally independence structure. Computing action-dependent baselines for general actions The marginalization presented in Section 4.4 does not apply for the general action setting. Instead, m individual baselines can be trained according to the factorization, and each of them can be fitted from data collected from the previous iteration. In the general case, this means fitting m functions b i (s t, a 1 t, . . ., a i−1 t), for i ∈ {1, . . ., m}. The ing method is described in Algorithm 2.There may also exist special cases like conditional independent actions, for which more efficient baseline constructions exist. A closely related example to the conditionally independent case is the case of block diagonal covariance structures (e.g. in multi-agent settings), where we may wish to instead learn an overall Q function and marginalize over block factors. Another interesting example to explore is sparse covariance structures. Algorithm 2 Policy gradient for general factorization policies using action-dependent baselines Require: number of iterations N, batch size B, initial policy parameters θ Initialize baselines allow us to smoothly interpolate between high-bias, low-variance estimates and low-bias, high-variance estimates of the policy gradient. These methods are based on the idea of being able to predict future returns, thereby bootstrapping the learning procedure. In particular, when using the value function as a baseline, we have DISPLAYFORM6 DISPLAYFORM7 is an unbiased estimator for V (s). GAE uses an exponential averaging of such temporal difference terms over a trajectory to significantly reduce the variance of the advantage at the cost of a small bias (it allows us to pick where we want to be on the bias-variance curve DISPLAYFORM8 Thus, the temporal difference error with the action dependent baselines is an unbiased estimator for the advantage function as well. This allows us to use the GAE procedure to further reduce variance at the cost of some bias. The following study shows that action-dependent baselines are consistent with TD procedures with their temporal differences being estimates of the advantage function. Our summarized in FIG3 suggests that slightly biasing the gradient to reduce variance produces the best , while high-bias estimates perform poorly. Prior work with baselines that utilize global information BID2 employ the high-bias variant. The here suggest that there is potential to further improve upon those by carefully studying the bias-variance trade-off. G HIGH-DIMENSIONAL ACTION SPACES: TRAINING CURVES Figure 5 shows the ing training curves for a synthetic high-dimensional target matching task, as described in Section 5. For higher dimensional action spaces (100 dimensions or greater), the action-dependent baseline consistently converges to the optimal solution 10% faster than the stateonly baseline. For reference, Figure 6 shows the of the original high-dimensional action space experiment. Due to a discovered issue in the TensorFlow version of rllab, which in training instability, both methods (action-dependent and state-dependent baselines) under-performed relative to the revised experiment (Figure 5), which uses a clean implementation based on the implementation referenced inRajeswaran et al. (2017b). The regression in training is most evident by the number of iterations required to solve the task; for instance, the old experiment could take as long as five times more iterations to solve the same task, even for a 12-dimensional task. Parameters: Unless otherwise stated, the following parameters are used in the experiments in this work: γ = 0.995, λ GAE = 0.97, kl desired = 0.025. Policies: The policies used are 2-layer fully connected networks with hidden sizes=.Initialization: the policy is initialized with Xavier initialization except final layer weights are scaled down (by a factor of 100x). Note that since the baseline is linear (with RBF features) and estimated with a Newton step, the initialization is inconsequential. Per-experiment configuration: The following parameters in TAB5 are for both state-only and action-dependent versions of the experiments. The m-DimTargetMatching experiments use a linear feature baseline.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Action-dependent baselines can be bias-free and yield greater variance reduction than state-only dependent baselines for policy gradient methods.

We propose a model-based method called "Search with Amortized Value Estimates" (SAVE) which leverages both real and planned experience by combining Q-learning with Monte-Carlo Tree Search, achieving strong performance with very small search budgets.


Question: The cost of annotating training data has traditionally been a bottleneck for supervised learning approaches. The problem is further exacerbated when supervised learning is applied to a number of correlated tasks simultaneously since the amount of labels required scales with the number of tasks. To mitigate this concern, we propose an active multitask learning algorithm that achieves knowledge transfer between tasks. The approach forms a so-called committee for each task that jointly makes decisions and directly shares data across similar tasks. Our approach reduces the number of queries needed during training while maintaining high accuracy on test data. Empirical on benchmark datasets show significant improvements on both accuracy and number of query requests. A triumph of machine learning is the ability to predict with high accuracy. However, for the dominant paradigm, which is supervised learning, the main bottleneck is the need to annotate data, namely, to obtain labeled training examples. The problem becomes more pronounced in applications and systems which require a high level of personalization, such as music recommenders, spam filters, etc. Several thousand labeled emails are usually sufficient for training a good spam filter for a particular user. However, in real world email systems, the number of registered users is potentially in the millions, and it might not be feasible to learn a highly personalized spam filter for each of them by getting several thousand labeled data points for each user. One method to relieve the need of the prohibitively large amount of labeled data is to leverage the relationship between the tasks, especially by transferring relevant knowledge from information-rich tasks to information-poor ones, which is called multitask learning in the literature. We consider multitask learning in an online setting where the learner sees the data sequentially, which is more practical in real world applications. In this setting, the learner receives an example at each time round, along with its task identifier, and then predicts its true label. Afterwards, the learner queries the true label and updates the model(s) accordingly. The online multitask setting has received increasing attention in the machine learning community in recent years BID6 BID0 BID7 BID9 BID4 BID13 BID11. However, they make the assumption that the true label is readily available to be queried, which is impractical in many applications. Also, querying blindly can be inefficient when annotation is costly. Active learning further reduces the work of the annotator by selectively requesting true labels from the oracles. Most approaches in active learning for sequential and streambased problems adopt a measure of uncertainty / confidence of the learner in the current example BID5 BID3 BID12 BID8 BID1.The recent work by BID10 combines active learning with online multitask learning using peers or related tasks. When the classifier of the current task is not confident, it first queries its similar tasks before requesting a true label from the oracle, incurring a lower cost. Their learner gives priority to the current task by always checking its confidence first. In the case when the current task is confident, the opinions of its peers are ignored. This paper proposes an active multitask learning framework which is more humble, in a sense that both the current task and its peers' predictions are considered simultaneously using a weighted sum. We have a committee which makes joint decisions for each task. In addition, after the true label of a training sample is obtained, this sample is shared directly to similar tasks, which makes training more efficient. The problem formulation and setup are similar to BID11 BID10. Suppose we are given K tasks and the k-th task is associated with N k training samples. We consider each task to be a linear binary classification problem, but the extensions to multiclass or non-linear cases are straightforward. We use the good-old perceptron-based update rule in which the model for a given task is only updated when the prediction for that training example is in error. The data for task k is {x DISPLAYFORM0 D is the i-th instance from the k-th task, yk ∈ {−1, +1} is the corresponding label and D is the dimension of features. When the notation is clear from the context, we drop task index k and simply write ((x (i), k), y (i) ). We consider the online setting where the training example ((x (t), k), y (t) ) comes at round t. Denote {w DISPLAYFORM1 the set of weights learned for the K binary classifiers at round t. Also denote w ∈ R K×D the weight matrix whose k-th row is w k . The labelŷ (t) is predicted based on the sign of the output value from the model. Then the hinge loss of task k on the sample (( DISPLAYFORM2 at round t is given by DISPLAYFORM3 In addition, we also consider the losses of its peer tasks m (m = k) as DISPLAYFORM4 km indicates the loss incurred by using task m's knowledge / classifier to predict the label of task k's training sample. DISPLAYFORM5 km plays an important role in learning the similarities among tasks and hence the committee weights. Intuitively, two tasks should be more similar if one task's training samples can be correctly predicted using the other task's classifier. The goal of this paper is to achieve a high accuracy on the test data, and at the same time to issue as small a number of queries to the oracle as possible during training, by efficiently sharing and transferring knowledge among similar tasks. In this section we introduce our algorithm Active Multitask Learning with Committees (AMLC) as shown in Algorithm 1. This algorithm provides an efficient way for online multitask learning. Each task uses not only its own knowledge but also knowledge from other tasks, and shares training examples across similar tasks when necessary. The two main components of Algorithm 1 are described in Section 3.1 and 3.2. In Section 3.3, we compare AMLC with the state-of-the-art online multitask learning algorithm. We maintain and update a relationship matrix τ ∈ R K×K through the learning process. The k-th row of τ, denoted τ k, is the committee weight vector for task k, also referred to as committee for brevity. Element τ ij of the relationship matrix indicates the closeness or similarity between task i and task if P (t) = 1 then 10: Query true label y (t) and set DISPLAYFORM0 12: Update τ: DISPLAYFORM1 13: 20: end function j, and also the importance of task j in task i's committee in predicting. Given a sample ((x (t), k), y (t) ) at round t, the confidence of task k is jointly decided by its committee; namely, a weighted sum of confidences of all tasks, DISPLAYFORM2 DISPLAYFORM3 Each confidence is just the common confidence measure for perceptron, using distance from the decision boundary BID5. The prediction is done by taking the sign of the confidence value. The learner then makes use of this confidence value by drawing a sample P (t) from a Bernoulli distribution, to decide whether to query the true label of this sample. The larger p is, the more likely for P (t) to be 0, signifying greater confidence. The hyperparameter b controls the level of confidence that the current task has to have to not request the true label. The learner only queries the true label when the current task's committee turns out to be unconfident. Another binary variable M (t) is set to be 1 if task k makes a mistake. Subsequently, its weight vector is updated following the conventional perceptron scheme. The learner then updates the relationship matrix following a similar policy as in BID11 BID10 Table 1. Accuracy on test set and total number of queries during training over 10 random shuffles of the training examples. The 95% confidence level is provided after the average accuracy. The best performance is highlighted in bold. On Spam Detection, both PEER+Share and AMLC are highlighted because AMLC has a lower mean but also smaller variance. DISPLAYFORM4 km /λ). The hyperparameter C decides how much decrease happens on the weight given non-zero loss, and λ = K m=1 l (t) km. These new weights are then normalized to sum to 1. To further encourage data sharing and information transfer between similar tasks, after the true label is obtained, the learner also shares the data with similar tasks of task k, so that peer tasks can learn from this sample as well. Similar tasks are identified by having a larger weight than the current task in the committee. We set S (t) m = 1 to indicate task m is a similar task to k and thus the data is shared with it. The most related work to ours is active learning from peers (PEER) BID10. In this section we discuss the main difference between our method and theirs with some intuition. Firstly, we do not treat the task itself and its peer tasks separately. Instead, the final confidence of the current task is jointly decided using the confidences of all tasks, weighted by the committee weight vector. It is humble in a sense that it always considers its peer tasks' advice when making a decision. There are two main advantages of our approach. 1) For PEER, no updates happen and no knowledge is transferred when the current task itself is confident. This can in difficulties for the learner to recover from being blindly confident. Blind confidence happens when the classifier makes mistakes on training examples but with high confidence, especially in early stage of training when data are not enough. 2) Our method updates the committee weight vector while keeping m∈ [K] τ km = 1 instead of m∈ [K],m =k τ km = 1. It then becomes possible that the current task itself has an equal or lower influence than other tasks on the final prediction. This is more desirable because identical tasks should have equal weights, and informationpoor tasks should rely more on their information-rich peers when making predictions. Secondly, our algorithm enables the sharing of training data across similar tasks directly, after acquiring the true label of this data. Querying can be costly, and the best way to make use of the expensive label information is to share it. Assuming that all tasks are identical, the most productive algorithm would merge all data to learn a single classifier. PEER is not able to achieve this because each task is still trained independently, since tasks only have access to their own data. Though PEER indirectly accesses others' data through querying peer tasks, this sharing mechanism can be insufficient when tasks are highly similar. In the case that all tasks are identical, our algorithm converges to a relationship matrix with identical elements and eventually all tasks are trained on every example that has been queried. In this section, we evaluate our proposed algorithm on three benchmark datasets for multitask learning, and compare our performance with many baseline models. We set b = 1 for all the experiments and tune the value of C from 20 values using 10-fold cross validation. Unless otherwise specified, all other model parameters are chosen via 10-fold cross validation. Landmine Detection 1 consists of 19 tasks collected from different landmine fields. Each task is a binary classification problem: landmines (+) or clutter (-), and each example consists of 9 features. Spam Detection 2 consists of labeled training data: spam (+) or non-spam (-) from the inboxes of 15 users, and each user is considered as a single task. Sentiment Analysis 3 (Blitzer et al.) consists of product re- views from Amazon containing reviews from 22 domains. We consider each domain as a binary classification task: positive review (+) and negative review (-). Details about our training and test sets are shown in Appendix A. We compare the performance of 5 different models. Random does not use any measure of confidence. Namely, the probability of querying or not querying true label are equal. Independent uses the confidence which is purely computed form the weight vector of the current task. Obviously both Random and Independent have no knowledge transfer among tasks. PEER is the algorithm from BID10. AMLC (Active Multitask Learning with Committees) is our proposed method as shown in Algorithm 1. In addition, we also show the performance of PEER+Share, in which we simply add to PEER the data sharing mechanism as illustrated in section 3.2. Table 1 shows the accuracy on test set and the total number of queries (label requests) to oracles during training of five models. Each value is the average of 10 random shuffles of the training set. The 95% confidence level is also shown. Notice that our re-implementation of PEER achieves similar performance on the Landmine and Spam datasets but seems to perform worse on Sentiment. The reason is that we are using a different representation of the training examples. We use the default bag-of-words representation coming with the dataset and there are approximately 2.9M features. The highlighted values illustrate the best performance across all models. On Spam Detection, AMLC is also highlighted because it is more confident about its accuracy even though the actual value is slightly lower than PEER+Share. It can be seen that our proposed methods (PEER+Share and AMLC) significantly outperform the the others. PEER has better performance compared to Random and Independent but still behaves worse than PEER+Share and AMLC. It can be shown that simply adding data sharing can improve both accuracy and number of queries used during training. The only exception is on Landmine Detection, where PEER+Share requests more queries than PEER. Though simply adding data sharing in improvement, after learning with joint decisions in AMLC, we observe further drastic decrease on the number of queries, while maintaining a high accuracy. Another goal of active multitask learning is to efficiently make use of the labels. In order to evaluate this, we give each model a fixed number of query budget and the training process is ended after the budget is exhausted. We show three plots (one for each dataset) in FIG1. Based on the difficulty of learning from each dataset, we choose different budgets to evaluate (up to 10%, 30% and 30% of the total training examples for Landmine, Spam and Sentiment respectively). We can see that given a limited number of query budgets, AMLC outperforms all models on all three datasets, as a of it encouraging more knowledge transfer among tasks. It is worth noting that the Landmine dataset is quite unbalanced (high proportion of negative labels), and PEER+Share and AMLC can achieve high accuracy with extremely limited number of queries. However, the classifier learned by PEER+Share is unconfident and thus it keeps requesting true labels in the following training process. We propose a new active multitask learning algorithm that encourages more knowledge transfer among tasks compared to the state-of-the-art models, by using joint decision / prediction and directly sharing training examples with true labels among similar tasks. Our proposed methods achieve both higher accuracy and lower number of queries on three benchmark datasets for multitask learning problems. Future work includes theoretical analysis of the error bound and comparison with those of the baseline models. Another interesting direction is to handle unbalanced task data. In other words, one task has much more / less training data than the others.

represent the tldr for finding the corresponding scientific article

We propose an active multitask learning algorithm that achieves knowledge transfer between tasks.

Represent the TLDR for finding the corresponding scientific article

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.


Question: Represent the scientific article to find a short TLDR

Question: Detection of photo manipulation relies on subtle statistical traces, notoriously removed by aggressive lossy compression employed online. We demonstrate that end-to-end modeling of complex photo dissemination channels allows for codec optimization with explicit provenance objectives. We design a lightweight trainable lossy image codec, that delivers competitive rate-distortion performance, on par with best hand-engineered alternatives, but has lower computational footprint on modern GPU-enabled platforms. Our show that significant improvements in manipulation detection accuracy are possible at fractional costs in bandwidth/storage. Our codec improved the accuracy from 37% to 86% even at very low bit-rates, well below the practicality of JPEG (QF 20). dard JPEG compression, and we extended it to support trainable codecs. We show a generic version of the updated model in Fig. 1 with highlighted potentially trainable elements. In this study, we fixed the camera model, and jointly optimize the FAN and a deep compression network (DCN). We describe the design of our DCN codec, and its pre-training protocol below. Our DCN model follows the general auto-encoder architecture proposed by , but uses different quantization, entropy estimation and entropy coding schemes (Section 3.2). The model is fully convolutional, and consists of 3 sub-sampling (stride-2) convolutional layers, and 3 residual blocks in between (Fig. 2). We do not use any normalization layers (such as GDN), and rely solely on a single trainable scaling factor. Distribution shaping occurs organically thanks to entropy regularization (see Fig. A.3b in the appendix). The decoder mirrors the encoder, and implements up-sampling using sub-pixel convolutions (combination of convolutional and depth-to-space layers). We experimented with different variants of latent representation quantization, eventually converging on soft-quantization with a fixed codebook of integers with a given maximal number of bits per feature (bpf). We used a 5-bpf uniform codebook (M = 32 values from -15 to 16). We show the impact of codebook size in the appendix (Fig. A.3a). The model is trained to minimize distortion between the input and reconstructed images regularized by entropy of the latent representation: where X is the input image, and E, Q, and D denote the encoder, quantization, and decoder, respectively. We used a simple L 2 loss in the RGB domain as the distortion measure d(·, ·), a differentiable soft estimate of entropy H (Section 3.2), and SSIM as the validation metric. We developed our own quantization and entropy estimation mechanism, because we found existing approaches unnecessarily complicated and/or lacking in accuracy. Some of the most recent solutions include: addition of uniform random noise to quantized samples and non-parametric entropy modeling by a fitted piece-wise linear model (Ballé et al., 2016); differentiable entropy upper bound with a uniform random noise component ; regularization by penalizing norm of quantized coefficients and differences between spatial neighbors ; PixelCNN for entropy estimation and context modeling . Our approach builds upon the soft quantization used by , but is extended to address numerical stability problems, and allow for accurate entropy estimation. Let z be a vectorized latent representation Z of N images, i.e.: z k = z n,i,j,c where n, i, j, c advance sequentially along an arbitrary memory layout (here image, width, height, channel). quantization asz = Wc. Hard quantization replaces an input value with the closest available codeword, and corresponds to a rounding operation performed by the image codec. Soft quantization is a differentiable relaxation, which uses a linear combination of all code-words -as specified by the weight matrix. A detailed comparison of both quantization modes, along with an illustration of potential numerical pitfalls, can be observed in the top row of Fig. A.1 in the appendix. The hard and soft quantization are used in the forward and backward passes, respectively. In Tensorflow, this can be implemented as z = tf.stop gradient(ẑ -z) +z. The weights for individual code-words in the mixture are computed by applying a kernel κ to the distances between the values and the code-words, which can be organized into a distance matrix D: The most commonly used implementations use a Gaussian kernel: which suffers from numerical problems for edge cases overflowing the codebook range (see Fig. A .1 top row in the 4-th and 5-th columns). To alleviate these problems, we adopt a t-Student kernel: which behaves much better in practice. We do not normalize the kernels, and ensure correct proportions of the weights by numerically normalizing rows of the weight matrix. We estimate entropy of the quantized values by summing the weight matrix along the sample dimension, which yields an estimate of the histogram w.r.t. codebook entries (comparison with an actual histogram is shown in Fig. A. 3):h This allows to estimate the entropy of the latent representation by simply: We assess the quality of the estimate both for synthetic random numbers (1,000 numbers sampled from Laplace distributions of various scales) and an actual latent representation of 128 × 128 px RGB image patches sampled from the clic test set (see Section 3.5 and examples in Fig. 4a). For the random sample, we fixed the quantization codebook to integers from −5 to 5, and performed the experiment numerically. For the real patches, we fed the images through a pre-trained DCN model (a medium-quality model with 32 feature channels; 32-C) and used the codebook embedded in the model (integers from −15 to 16). Fig. 3 shows the entropy estimation error (both absolute and relative) and scatter plots of real entropies vs. their soft estimates using the Gaussian and t-Student kernels. It can be observed that the t-Student kernel consistently outperforms the commonly used Gaussian. The impact of the kernels' hyperparameters on the relative estimation error is shown in Fig. A.2. The best combination of kernel parameters (v = 50, γ = 25) is highlighted in red and used in all subsequent experiments. Published as a conference paper at ICLR 2020 Figure 4: Example images from the considered clic, kodak and raw test sets (512×512 px). We used a state-of-the-art entropy coder based on asymmetric numeral systems and its finite state entropy (FSE) implementation . For simplicity and computational efficiency, we did not employ a context model 1 and instead encode individual feature channels (channel EC). Bitrate savings w.r.t. global entropy coding (global EC) vary based on the model, image size and content. For 512 × 512 px images, we observed average savings of ≈ 12%, but for very small patches (e.g., 128 px), it may actually in overhead (Tab. A.2). This can be easily addressed with a flag that switches between different compression modes, but we leave practical design of the format container for future work. We use a simple structure of the bit-stream, which enables variable-length, per-channel entropy coding with random channel access (Tab. A.1). Such an approach offers flexibility and scalability benefits, e.g.: it allows for rapid analysis of selected feature channels ; it enables trivial parallel processing of the channels to speed up encoding/decoding on modern multi-core platforms. We pre-trained the DCN model in isolation and minimize the entropy-regularized L 2 loss (equation 1) on mixed natural images (MNI) from 6 sources: native camera output from the RAISE and MIT-5k datasets ; photos from the Waterloo exploration database ; HDR images ; computer game footage ; city scapes ; and the training sub-set of the CLIC professional dataset . In total, we collected 32,000 square crops ranging from 512 × 512 to 1024 × 1024 px, which were subsequently down-sampled to 256 × 256 px and randomly split into training and validation subsets. We used three augmentation strategies: we trained on 128 × 128 px patches randomly sampled in each step; we flip the patches vertically and/or horizontally with probability 0.5; and we apply random gamma correction with probability 0.5. This allowed for reduction of the training set size, to ≈10k images where the performance saturates. We used batches of 50 images, and learning rate starting at 10 −4 and decaying by a factor of 0.5 every 1,000 epochs. The optimization algorithm was Adam with default settings (as of Tensorflow 1.12). We train by minimizing MSE (L 2 loss) until convergence of SSIM on a validation set with 1,000 images. We control image quality by changing the number of feature channels. We consider three configurations for low, medium, and high quality with 16, 32, and 64 channels, respectively. Standard Codecs: As hand-crafted baselines, we consider three codecs: JPEG from the libJPEG library via the imageio interface, JPEG2000 from the OpenJPEG library via the Glymur interface, and BPG from its reference implementation . Since our model uses full-resolution RGB channels as input, we also use full-resolution chrominance channels whenever possible (JPEG and BPG). To make the comparison as fair as possible, we measure effective payload of the codecs. For the JPEG codec, we manually seek byte markers and include only the Huffman tables and Huffman-coded image data. For JPEG2000, we add up lengths of tile-parts, as reported by jpylyzer. For BPG, we seek the picture data length marker. Rate-distortion Trade-off: We used 3 datasets for the final evaluation (Fig. 4): (raw) 39 images with native camera output from 4 different cameras ; (clic) 39 images from the professional validation subset of; (kodak) 24 images from the standard Kodak dataset. All test images are of size 512 × 512px, and were obtained either by cropping directly without re-sampling (raw, kodak) or by resizing a central square crop (clic). We measured PSNR and SSIM using the scikit-image package and MS-SSIM using sewar. We used the default settings and only set the data range to. The values are computed as the average over RGB channels. The bit-rate for hand-crafted codecs was computed using the effective payload, as explained above. For the DCN codec, we completely encoded and decoded the images (Section A.1). Fig. 5 shows rate-distortion curves (SSIM vs. effective bpp) for the clic and raw datasets (see appendix for additional ). We show 4 individual images (Fig. 4) and averages over the respective datasets. Since standard quality control (e.g., quality level in JPEG, or number of channels in DCN) leads to a wide variety of bpps, we fit individual images to a parametric curve f (x) = (1 + e −αx β +γ) −1 − δ and show the averaged fits. It can be observed that our DCN model delivers significantly better than JPEG and JPEG2000, and approaches BPG. Processing Time: We collected DCN processing times for various platforms (Table 1), including desktops, servers, and low-power edge AI. We report network inference and complete encoding/decoding times on 512 × 512 px and 1920 × 1080 px images, averaged over the clic dataset (separate runs with batch size 1) and obtained using an unoptimized Python 3 implementation 2. On GPUenabled platforms, the inference time becomes negligible (over 100 fps for 512 × 512 px images, and over 20 fps for 1920 × 1080 px images on GeForce 1080 with a 2.6 GHz Xeon CPU), and entropy coding starts to become the bottleneck (down to 13 and 5 fps, respectively). We emphasize that the adopted FSE codec is one of the fastest available, and significantly outperforms commonly used arithmetic coding . If needed, channel EC can be easily parallelized, and the ANS codec could be re-implemented to run on GPU as well (Weißenberger &). As a reference, we measured the processing times of 1920 × 1080 px images for the standard codecs on the i7-7700 CPU @ 3.60GHz processor. JPEG coding with 1 thread takes between 0.061 s (Q=30) and 0.075 s (Q=90) (inclusive of writing time to RAM disk; using the pillow library). JPEG 2000 with 1 thread takes 0.61 s regardless of the quality level (inclusive of writing time to RAM disk; glymur library). BPG with 4 parallel threads takes 2.4 s (Q=1), 1.25 s (Q=20) and 0.72 s (Q=30) (inclusive of PNG reading time from RAM disk; bpgenc command line tool). While not directly comparable and also not optimized, some state-of-the-art deep learned codecs require minutes to process even small images, e.g., 5-6 min for 768 × 512 px images from the Kodak dataset reported by. The fastest state-of-the-art learned codec is reported to run at ≈100 fps on images of that size on a GPU-enabled desktop computer . We consider the standard photo manipulation detection setup where an adversary uses contentpreserving post-processing, and a forensic analysis network (FAN) needs to identify the applied operation or confirm that the image is unprocessed. We use a challenging real-world setup, where the FAN can analyze images only after transmission through a lossy dissemination channel (Fig. 1). In such conditions, forensic analysis is known to fail . We consider several versions of the channel, including: standard JPEG compression, pre-trained DCN codecs, and trainable DCN codecs jointly optimized along with the FAN. We analyze 128 × 128 px patches, and don't use down-sampling to isolate the impact of the codec. We consider 6 benign post-processing operations which preserve image content, but change lowlevel traces that can reveal a forgery. Such operations are commonly used either during photo manipulation or as a masking step afterwards. We include: (a) sharpening -implemented as an unsharp mask operator applied to the luminance channel in the HSV color space; (b) resampling involving successive down-sampling and up-sampling using bilinear interpolation and scaling factors 1:2 and 2:1; (c) Gaussian filtering with a 5 × 5 filter and standard deviation 0.83; (d) JPEG compression using a differentiable dJPEG model with quality level 80; (e) AWGN noise with standard deviation 0.02; and (f) median filtering with a 3 × 3 kernel. The operations are difficult to distinguish visually from native camera output -even without lossy compression (Fig. 7). The FAN is a state-of-the-art image forensics CNN with a constrained residual layer . We used the model provided in the toolbox , and optimize for classification (native camera output + 6 post-processing classes) of RGB image patches. In total, the model has 1.3 million parameters. We jointly train the entire workflow and optimize both the FAN and DCN models. Let M c denote the c-th manipulation (including identity for native camera output), and F denote the output of the FAN with F c being the probability of the corresponding manipulation class c. Let also C denote the adopted lossy compression model, e.g., D • Q • E for the DCN. We denote sRGB images rendered by the camera ISP as X. The FAN model is trained to minimize a cross-entropy loss: and the DCN to minimize its combination with the original fidelity/entropy loss (equation 1): where λ c is used to control the balance between the objectives (we consider values from 10 −3 to 1). We start from pre-trained DCN models (Section 3.4). The FAN model is trained from scratch. When JPEG compression was used in the channel, we used the differentiable dJPEG model from the original study , but modified it to use hard-quantization in the forward pass to ensure equivalent to libJPEG. We used quality levels from 10 to 95 with step 5. We followed the same training protocol as , and trained on native camera output (NCO). We used the DNet pipeline for Nikon D90, and randomly sampled 128 × 128 px RGB patches from 120 full-resolution images. The remaining 30 images were used for validation (we sampled 4 patches per image to increase diversity). We used batches of 20 images, and trained for 2,500 epochs with learning rate starting at 10 −4 and decaying by 10% every 100 epochs. For each training configuration, we repeated the experiment 3-5 times to validate training stability. We summarize the obtained in Fig. 8 which shows the trade-off between effective bpp (rate), SSIM (distortion), and manipulation detection accuracy. The figure compares standard JPEG compression (diamond markers), pre-trained basic DCN models (connected circles with black border), and fine-tuned DCN models for various regularization strenghts λ c (loose circles with gray border). Fine-tuned models are labeled with a delta in the auxiliary metric (also encoded as marker size and color), and the text is colored in red/green to indicate deterioration or improvement. Fig. 8a shows how the manipulation detection capability changes with effective bitrate of the codec. We can make the following observations. Firstly, JPEG delivers the worst trade-off and exhibits irregular behavior. The performance gap may be attributed to: (a) better image fidelity for the DCN codec, which retains more information at any bitrate budget; (b) presence of JPEG compression as one of the manipulations. The latter factor also explains irregular drops in accuracy, which coincide with the quality level of the manipulation and unfavorable multiples of the quantization tables (see also Fig. B.1). Secondly, we observe that fine-tuning the DCN model leads to a sudden increase in payload requirements, minor improvement in quality, and gradual increase in manipulation detection accuracy (as λ c → 0). We obtain significant improvements in accuracy even for the lowest JPG JPG JPG JPG JPG (70 JPG JPG JPG JPG (quality level (37% → 85%; at such low bitrates JPEG stays below 30%). Interestingly, we don't see major differences in payload between the fine-tuned models, which suggests that qualitative differences in encoding may be expected beyond simple inclusion of more information. Fig. 8b shows the same from a different perspective, and depicts the standard rate-distortion trade-off supplemented with auxiliary accuracy information. We can observe that DCN fine-tuning moves the model to a different point (greater payload, better quality), but doesn't seem to visibly deteriorate the rate-distortion trade-off (with the exception of the smallest regularization λ c = 0.001 which consistently shows better accuracy and worse fidelity). To explain the behavior of the models, we examine frequency attenuation patterns. We compute FFT spectra of the compressed images, and divide them by the corresponding spectra of uncompressed images. We repeat this procedure for all images in our raw test set, and average them to show consistent trends. The are shown in Fig. 9 for the pre-trained DCN model (1st column) and fine-tuned models with decreasing λ c (increasing emphasis on accuracy). The plots are calibrated to show unaffected frequencies as gray, and attenuated/emphasized frequencies as dark/bright areas. The pre-trained models reveal clear and gradual attenuation of high frequencies. Once the models are plugged in to the dissemination workflow, high frequencies start to be retained, but it does not suffice to improve manipulation detection. Increasing importance of the cross-entropy loss gradually changes the attenuation patterns. Selection of frequencies becomes more irregular, and some bands are actually emphasized by the codec. The right-most column shows the most extreme configuration where the trend is clearly visible (the outlier identified in quantitative analysis in Section 4.3). The changes in codec behavior generally do not introduce visible differences in compressed images (as long as the data distribution is similar, see Section 5). We show an example image (raw test set), its compressed variants (low-quality, 16-C), and their corresponding spectra in Fig. 10. The spectra follow the identified attenuation pattern (Fig. 9). The compressed images do not reveal any obious artifacts, and the most visible change seems to be the jump in entropy, as predicted in Section 4.3. While the proposed approach can successfully facilitate pre-screening of photographs shared online, further research is needed to improve model generalization. We observed that the fine-tuning procedure tends bias the DCN/FAN models towards the secondary image dataset, in our case the native camera output (NCO). The baseline DCN was pre-trained on mixed natural images (MNI) with extensive augmentation, leading to competitive on all test sets. However, fine-tuning was performed on NCO only. Characteristic pixel correlations, e.g., due to color interpolation, bias the codec and lead to occasional artifacts in MNIs (mostly in the clic test set; see Appendix B), and deterioration of the rate-distortion trade-off. The problem is present regardless of λ c, which suggests issues with the fine-tuning protocol (data diversity) and not the forensic optimization objective. We ran additional experiments by skipping photo acquisition and fine-tuning directly on MNI from the original training set (subset of 2,500 RGB images). We observed the same behavior (see Appendix C), and the optimized codec was artifact-free on all test sets. (Although, due to a smaller training set, the model loses some of its performance; cf. MNI in Fig. A.6 .) However, the FANs generalized well only to clic and kodak images. The originally trained FANs generalized reasonably well to different NCO images (including images from other 3 cameras) but not to clic or kodak. This confirms that existing forensics models are sensitive to data distribution, and that further work will be needed to establish more universal training protocols (see detailed discussion in Appendix D). Short fine-tuning is known to help , and we leave this aspect for future work. We are also planning to explore new transfer learning protocols . Generalization should also consider other forensic tasks. We optimized for manipulation detection, which serves as a building block for more complex problems, like processing history analysis or tampering localization (; ; ; a). However, additional pre-screening may also be needed, e.g., analysis of sensor fingerprints , or identification of computer graphics or synthetic content (b). Our study shows that lossy image codecs can be explicitly optimized to retain subtle low-level traces that are useful for photo manipulation detection. Interestingly, simple inclusion of high frequencies in the signal is insufficient, and the models learns more complex frequency attenuation/amplification patterns. This allows for reliable authentication even at very low bit-rates, where standard JPEG compression is no longer practical, e.g., at bit-rates around 0.4 bpp where our DCN codec with lowquality settings improved manipulation detection accuracy from 37% to 86%. We believe the proposed approach is particularly valuable for online media platforms (e.g., Truepic, or Facebook), who need to pre-screen content upon reception, but need to aggressively optimize bandwidth/storage. The standard soft quantization with a Gaussian kernel works well for rounding to arbitrary integers, but leads to numerical issues for smaller codebooks. Values significantly exceeding codebook endpoints have zero affinity to any of the entries, and collapse to the mean (i.e., ≈ 0 in our case; Fig. A.1a). Such issues can be addressed by increasing numerical precision, sacrificing accuracy (due to larger kernel bandwidth), or adding explicit conditional statements in the code. The latter approach is inelegant and cumbersome in graph-based machine learning frameworks like Tensorflow. We used a t-Student kernel instead and increased precision of the computation to 64-bits. This doesn't solve the problem entirely, but successfully eliminated all issues that we came across in our experiments, and further improved our entropy estimation accuracy. Fig. A.2 shows entropy estimation error for Laplace-distributed random values, and different hyper-parameters of the kernels. We observed the best for a t-Student kernel with 50 degrees of freedom and bandwidth γ = 25 (marked in red). This kernel is used in all subsequent experiments. We experimented with different codebooks and entropy regularization strengths. Fig. A.3a shows how the quantized latent representation (QLR) changes with the size of the codebook. The figures also compare the actual histogram with its soft estimate (equation 6). We observed that the binary codebook is sub-optimal and significantly limits the achievable image quality, especially as the number of feature channels grows. Adding more entries steadily improved quality and the codebook with M = 32 entires (values from -15 to 16) seemed to be the point of diminishing returns. Our entropy-based regularization turned out to be very effective at shaping the QLR (Fig. A.3b) and dispensed with the need to use other normalization techniques (e.g., GDN). We used only a single scalar multiplication factor responsible for scaling the distribution. All baseline and finetuned models use λ H = 250 (last column). Fig. A.4 visually compares the QLRs of our baseline low-quality codec (16 feature channels) with weak and strong regularization. We used a rudimentary bit-stream structure with the essential meta-data and markers that allow for successful decoding (Tab. A.1). Feature channels are entropy-coded independently (we refer to this approach as channel EC), and can be accessed randomly after decoding a lookup Figure A.6: Rate-distortion performance for standard codecs and all DCN versions including the pre-trained baselines (b) and 3 fine-tuned models with the the weakest (λ c = 1) and the strongest emphasis on manipulation detection (λ c = 0.005 and 0.001). The latter incure only fractional cost in payload/quality but bring significant benefits for manipulation detection. The top (bottom) rows show for SSIM (MS-SSIM), respectively, and include DCN models fine tuned on native camera output (raw) and mixed natural images (clic). Fig. 8 visualizes the trade-offs in image compression and forensic analysis performance. Here we show direct impact of image compression and fine-tuning settings on the achievable manipulation detection accuracy and its variations (Fig. B.1). For the JPEG codec, we observe nearly perfect manipulation detection for the highest quality level (QF=95), and a steady decline starting immediately below. The sudden drop in accuracy corresponds to the quality level of JPEG as one of the manipulations (QF=80). For DCN models, we clearly see steady improvement of fine-tuning w.r.t. the baseline models (on the right). Interestingly, the high-quality model shows a slight decline at first. Qualitative Analysis: The learned frequency attenuation/amplification patterns for all of the considered quality levels are shown in Fig. B.2. The visualizations were computed in the FFT domain and show the relative magnitude of individual frequencies w.r.t. original uncompressed images (averaged over all test images). In all cases, we observe complex behavior beyond simple inclusion of high-frequency content. The pattern seem to have a stable trajectory, despite independent runs of the experiment with different regularization strengths λ c. The same patterns will also be visible in individual image spectra (Fig. B.4 -Fig. B.6). Generalization and Imaging Artifacts: While our baseline DCN models were pre-trained on a large and diverse training set, the fine-tuning procedure relied on the complete model of photo acquisition and dissemination. Photo acquisition with digital cameras yields characteristic imaging artifacts, e.g., due to color filtering and interpolation. This leads to a characteristic distribution of native camera output (NCO), and ultimately biases the codec. Fig. B.3 shows the differences in SSIM between the baseline models and the models fine-tuned with a very weak cross-entropy objective (leading to no improvement in manipulation detection accuracy). For NCO (raw test set), we observe improvement of image quality (and corresponding increase in bitrate). For the kodak set, the quality remains mostly unaffected (with an increased bitrate). On the clic set, we observe minor quality loss, and occasional artifacts (see examples in Fig. B.4). In Fig. B.4 -Fig. B.6 we collected example images from all test sets (clic, kodak, and raw) and compress them with baseline and fine-tuned models. The images are ordered by SSIM deterioration due to weak fine-tuning (quality loss without gains in accuracy; Fig. B. 3) -the worst cases are shown at the top. (Note that some of the artifacts are caused by JPEG encoding of the images embedded in the PDF, and some geometric distortions were introduced by imperfect scaling in matplotlib.) In the first clic image (1st row in Fig. B.4), we can see color artifacts along the high-contrast wires. In the second image, we see distortions in the door blinds, and a subtle shift in the hue of the bike. For the remaining two images, SSIM remains the same or better and we do not see any artifacts. In the kodak set, the worst image quality was observed for kodim05 (1st row in Fig. B .5), but we don't see any artifacts. Figure C.1: Visualization of the rate-distortion-accuracy trade-off on the clic dataset after fine-tuning on mixed natural images. As discussed in Section 5, we ran additional experiments by skipping photo acquisition and finetuning directly on mixed natural images (MNI) -a subset of the original DCN training set (2,500 images). Images in this dataset tend to have more details and depict objects at a coarser scale, since they were all down-sampled to 256 × 256 px (from various original sizes). This required adjusting manipulation strength to maintain visual similarity between photo variations. In particular, we used weaker sharpening, Gaussian filtering with a smaller standard deviation (0.5), down&up-sampling via 75% of the image size (instead of 50%), Gaussian noise with standard deviation 0.012, and JPEG quality level 90. We fine-tuned for 600 epochs. We summarize the obtained in Fig C. 1 using images from the clic test set. In this experiment, the gap in manipulation detection accuracy between JPEG and baseline DCN has disappeared, except for remaining sudden drops at selected JPEG quality levels (corresponding to the manipulation quality factor 90). We still observe significant improvement for fine-tuned DCN models, but here it tends to saturate around 86%, which might explain negligible improvement of the high-quality 64-C model. By inspecting confusion matrices, we see most of the confusion between native, sharpen and awgn classes where the differences are extremely subtle. The fine-tuned DCN models remain close to the baseline rate-distortion behavior. Interestingly, except for the weakest regularization (λ c = 0.001), all fine-tuned models seem to be equivalent (w.r.t. the trade-off). We did not observe any obvious artifacts, even in the most aggressive models. The only image with deteriorated SSIM is the alejandro-escamilla-6 image from clic, which was consistently the most affected outlier in nearly all fine tuned models for both NCO and MNI (1st row in Fig. C.2). In some replications it actually managed to improve, e.g., for the shown cases with λ c = 0.005 and 0.001. However, we don't see any major differences between these variations. Visualization of frequency attenuation patterns (Fig. C.3) shows analogous behavior, but the changes are more subtle on MNI. We included additional difference plots w.r.t. baseline and weakly finetuned models, where the changes are better visible. On NCO, due to less intrinsic high-frequency content, the behavior is still very clear (cf. bottom part of Fig. C.3). Published as a conference paper at ICLR 2020 In this study, we considered two broad classes of images: native camera output (NCO) and mixed natural images (MNI) which exhibit significant differences in pixel distribution. For DCN pretraining, we relied on a large MNI dataset with images down-sampled to 256 × 256 px (Section 3.4). Fine-tuning was then performed on either NCO from a single camera (Nikon D90; Section 4) or a smaller sub-set of the original training MNI (2,500 images; Appendix C). Finally, we considered three test sets: raw with NCO from 4 different cameras; clic and kodak with MNI. We observed that the FAN models tend to have limited generalization capabilities to images with a different pixel distribution. We ran additional experiments to quantify this phenomenon, and show the obtained in Fig. D.1 where we compare test vs validation accuracy for various test sets (we also included a version of the clic set resized to 256×256 px). In the top row, we show for models trained on NCO from a single camera. We can see imperfect, but reasonable generalization to the output of various cameras (red markers). Once the data distribution changes, the performance drops significantly. Analogously, FAN models trained on MNI generalize well to other MNI datasets (clic and kodak), but fail on NCO. We see an additional bias towards images down-sampled to the same resolution as the training data (compare clic vs clic-256 images), but here the difference is small and consistent -between 5.2 -6% based on a linear fit to the data.

Represent the short summary for finding the corresponding scientific article (long).

We learn an efficient lossy image codec that can be optimized to facilitate reliable photo manipulation detection at fractional cost in payload/quality and even at low bitrates.

We propose accelerating Batch Normalization (BN) through sampling less correlated data for reduction operations  with regular execution pattern, which achieves up to 2x and 20% speedup for BN itself and the overall training, respectively.