Question: represent the scientific article to find a short tldr

Question: Deep Reinforcement Learning (DRL) has led to many recent breakthroughs on complex control tasks, such as defeating the best human player in the game of Go. However, decisions made by the DRL agent are not explainable, hindering its applicability in safety-critical settings. Viper, a recently proposed technique, constructs a decision tree policy by mimicking the DRL agent. Decision trees are interpretable as each action made can be traced back to the decision rule path that lead to it. However, one global decision tree approximating the DRL policy has significant limitations with respect to the geometry of decision boundaries. We propose MoET, a more expressive, yet still interpretable model based on Mixture of Experts, consisting of a gating function that partitions the state space, and multiple decision tree experts that specialize on different partitions. We propose a training procedure to support non-differentiable decision tree experts and integrate it into imitation learning procedure of Viper. We evaluate our algorithm on four OpenAI gym environments, and show that the policy constructed in such a way is more performant and better mimics the DRL agent by lowering mispredictions and increasing the reward. We also show that MoET policies are amenable for verification using off-the-shelf automated theorem provers such as Z3. Deep Reinforcement Learning (DRL) has achieved many recent breakthroughs in challenging domains such as Go . While using neural networks for encoding state representations allow DRL agents to learn policies for tasks with large state spaces, the learned policies are not interpretable, which hinders their use in safety-critical applications. Some recent works leverage programs and decision trees as representations for interpreting the learned agent policies. PIRL uses program synthesis to generate a program in a Domain-Specific Language (DSL) that is close to the DRL agent policy. The design of the DSL with desired operators is a tedious manual effort and the enumerative search for synthesis is difficult to scale for larger programs. In contrast, Viper learns a Decision Tree (DT) policy by mimicking the DRL agent, which not only allows for a general representation for different policies, but also allows for verification of these policies using integer linear programming solvers. Viper uses the DAGGER imitation learning approach to collect state action pairs for training the student DT policy given the teacher DRL policy. It modifies the DAGGER algorithm to use the Q-function of teacher policy to prioritize states of critical importance during learning. However, learning a single DT for the complete policy leads to some key shortcomings such as i) less faithful representation of original agent policy measured by the number of mispredictions, ii) lower overall performance (reward), and iii) larger DT sizes that make them harder to interpret. In this paper, we present MOËT (Mixture of Expert Trees), a technique based on Mixture of Experts (MOE) (; ;), and reformulate its learning procedure to support DT experts. MOE models can typically use any expert as long as it is a differentiable function of model parameters, which unfortunately does not hold for DTs. Similar to MOE training with Expectation-Maximization (EM) algorithm, we first observe that MOËT can be trained by interchangeably optimizing the weighted log likelihood for experts (independently from one another) and optimizing the gating function with respect to the obtained experts. Then, we propose a procedure for DT learning in the specific context of MOE. To the best of our knowledge we are first to combine standard non-differentiable DT experts, which are interpretable, with MOE model. Existing combinations which rely on differentiable tree or treelike models, such as soft decision trees and hierarchical mixture of experts are not interpretable. We adapt the imitation learning technique of Viper to use MOËT policies instead of DTs. MOËT creates multiple local DTs that specialize on different regions of the input space, allowing for simpler (shallower) DTs that more accurately mimic the DRL agent policy within their regions, and combines the local trees into a global policy using a gating function. We use a simple and interpretable linear model with softmax function as the gating function, which returns a distribution over DT experts for each point in the input space. While standard MOE uses this distribution to average predictions of DTs, we also consider selecting just one most likely expert tree to improve interpretability. While decision boundaries of Viper DT policies must be axis-perpendicular, the softmax gating function supports boundaries with hyperplanes of arbitrary orientations, allowing MOËT to more faithfully represent the original policy. We evaluate our technique on four different environments: CartPole, Pong, Acrobot, and Mountaincar. We show that MOËT achieves significantly better rewards and lower misprediction rates with shallower trees. We also visualize the Viper and MOËT policies for Mountaincar, demonstrating the differences in their learning capabilities. Finally, we demonstrate how a MOËT policy can be translated into an SMT formula for verifying properties for CartPole game using the Z3 theorem prover (De Moura and Bjørner, 2008) under similar assumptions made in Viper. In summary, this paper makes the following key contributions: 1) We propose MOËT, a technique based on MOE to learn mixture of expert decision trees and present a learning algorithm to train MOËT models. 2) We use MOËT models with a softmax gating function for interpreting DRL policies and adapt the imitation learning approach used in Viper to learn MOËT models. 3) We evaluate MOËT on different environments and show that it leads to smaller, more faithful, and performant representations of DRL agent policies compared to Viper while preserving verifiability. Interpretable Machine Learning: In numerous contexts, it is important to understand and interpret the decision making process of a machine learning model. However, interpretability does not have a unique definition that is widely accepted. Accoding to Lipton , there are several properties which might be meant by this word and we adopt the one which Lipton names transparency which is further decomposed to simulability, decomposability, and algorithmic transparency. A model is simulable if a person can in reasonable time compute the outputs from given inputs and in that way simulate the model's inner workings. That holds for small linear models and small decision trees . A model is decomposable if each part of a models admits an intuitive explanation, which is again the case for simple linear models and decision trees . Algorithmic transparency is related to our understanding of the workings of the training algorithm. For instance, in case of linear models the shape of the error surface and properties of its unique minimum towards which the algorithm converges are well understood . MOËT models focus on transparency (as we discuss at the end of Section 5). Explainable Machine Learning: There has been a lot of recent interest in explaining decisions of black-box models (a;). For image classification, activation maximization techniques can be used to sample representative input patterns . TCAV uses human-friendly high-level concepts to associate their importance to the decision. Some recent works also generate contrastive robust explanations to help users understand a classifier decision based on a family of neighboring inputs . LORE (b) explains behavior of a blackbox model around an input of interest by sampling the black-box model around the neighborhood of the input, and training a local DT over the sampled points. Our model presents an approach that combines local trees into a global policy. Tree-Structured Models: Irsoy et al. propose a a novel decision tree architecture with soft decisions at the internal nodes where both children are chosen with probabilities given by a sigmoid gating function. Similarly, binary tree-structured hierarchical routing mixture of experts (HRME) model, which has classifiers as non-leaf node experts and simple regression models as leaf node experts, were proposed in . Both models are unfortunately not interpretable. Knowledge Distillation and Model Compression: We rely on ideas already explored in fields of model compression and knowledge distillation . The idea is to use a complex well performing model to facilitate training of a simpler model which might have some other desirable properties (e.g., interpretability). Such practices have been applied to approximate decision tree ensemble by a single tree , but this is different from our case, since we approximate a neural network. In a similar fashion a neural network can be used to train another neural network , but neural networks are hard to interpret and even harder to formally verify, so this is also different from our case. Such practices have also been applied in the field of reinforcement learning in knowledge and policy distillation (; ;), which are similar in spirit to our work, and imitation learning (; ; ;), which provide a foundation for our work. We now present a simple motivating example to showcase some of the key differences between Viper and MOËT approaches. Consider the N × N Gridworld problem shown in Figure 1a (for N = 5). The agent is placed at a random position in a grid (except the walls denoted by filled rectangles) and should find its way out. To move through the grid the agent can choose to go up, left, right or down at each time step. If it hits the wall it stays in the same position (state). State is represented using two integer values (x, y coordinates) which range from-bottom left to (N − 1, N − 1)-top right. The grid can be escaped through either left doors (left of the first column), or right doors (right of the last column). A negative reward of −0.1 is received for each agent action (negative reward encourages the agent to find the exit as fast as possible). An episode finishes as soon as an exit is reached or if 100 steps are made whichever comes first. The optimal policy (π *) for this problem consists of taking the left (right resp.) action for each state below (above resp.) the diagonal. We used π * as a teacher and used imitation learning approach of Viper to train an interpretable DT policy that mimics π *. The ing DT policy is shown in Figure 1b. The DT partitions the state space (grid) using lines perpendicular to x and y axes, until it separates all states above diagonal from those below. This in a DT of depth 3 with 9 nodes. On the other hand, the policy learned by MOËT is shown in Figure 1c. The MOËT model with 2 experts learns to partition the space using the line defined by a linear function 1.06x + 1.11y = 4 (roughly the diagonal of the grid). Points on the different sides of the line correspond to two different experts which are themselves DTs of depth 0 always choosing to go left (below) or right (above). We notice that DT policy needs much larger depth to represent π * while MOËT can represent it as only one decision step. Furthermore, with increasing N (size of the grid), complexity of DT will grow, while MOËT complexity stays the same; we empirically confirm this for N =. 5, 6, 7, 8, 9, 10 DT depths are 3, 4, 4, 4, 4, 5 and number of nodes are 9, 11, 13, 15, 17, 21 respectively. In contrast, MOËT models of same complexity and structure as the one shown in Figure 1c are learned for all values of N (models differ in the learned partitioning linear function). In this section we provide description of two relevant methods we build upon: Viper, an approach for interpretable imitation learning, and MOE learning framework. Viper. Viper algorithm (included in appendix) is an instance of DAGGER imitation learning approach, adapted to prioritize critical states based on Q-values. Inputs to the Viper training algorithm are environment e which is an finite horizon (T -step) Markov Decision Process (MDP) (S, A, P, R) with states S, actions A, transition probabilities P: S × A × S →, and rewards R: S → R; teacher policy π t: S → A; its Q-function Q πt: S × A → R and number of training iterations N. Distribution of states after T steps in environment e using a policy π is d (π) (e) (assuming randomly chosen initial state). Viper uses the teacher as an oracle to label the data (states with actions). It initially uses teacher policy to sample trajectories (states) to train a student (DT) policy. It then uses the student policy to generate more trajectories. Viper samples training points from the collected dataset D giving priority to states s having higher importance I(s), where. This sampling of states leads to faster learning and shallower DTs. The process of sampling trajectories and training students is repeated for number of iterations N, and the best student policy is chosen using reward as the criterion. MOE is an ensemble model (; ;) that consists of expert networks and a gating function. Gating function divides the input (feature) space into regions for which different experts are specialized and responsible. MOE is flexible with respect to the choice of expert models as long as they are differentiable functions of model parameters (which is not the case for DTs). In MOE framework, probability of outputting y ∈ IR m given an input x ∈ IR n is given by: where E is the number of experts, g i (x, θ g) is the probability of choosing the expert i (given input x), P (y|x, θ i) is the probability of expert i producing output y (given input x). Learnable parameters are θ = (θ g, θ e), where θ g are parameters of the gating function and θ e = (θ 1, θ 2, ..., θ E) are parameters of the experts. Gating function can be modeled using a softmax function over a set of linear models. Let θ g consist of parameter vectors (θ g1, . . ., θ gE), then the gating function can be defined as. In the case of classification, an expert i outputs a vector y i of length C, where C is the number of classes. Expert i associates a probability to each output class c (given by y ic) using the gating function. Final probability of a class c is a gate weighted sum of y ic for all experts i ∈ 1, 2,..., E. This creates a probability vector y = (y 1, y 2, ..., y C), and the output of MOE is arg max i y i. MOE is commonly trained using EM algorithm, where instead of direct optimization of the likelihood one performs optimization of an auxiliary functionL defined in a following way. Let z denote the expert chosen for instance x. Then joint likelihood of x and z can be considered. Since z is not observed in the data, log likelihood of samples (x, z, y) cannot be computed, but instead expected log likelihood can be considered, where expectation is taken over z. Since the expectation has to rely on some distribution of z, in the iterative process, the distribution with respect to the current estimate of parameters θ is used. More precisely functionL is defined by : where θ (k) is the estimate of parameters θ in iteration k. Then, for a specific sample D = {(x i, y i) | i = 1,..., N }, the following formula can be derived : where it holds In this section we explain the adaptation of original MOE model to mixture of decision trees, and present both training and inference algorithms. Considering that coefficients h (k) ij (Eq. 4) are fixed with respect to θ and that in Eq. 3 the gating part (first double sum) and each expert part depend on disjoint subsets of parameters θ, training can be carried out by interchangeably optimizing the weighted log likelihood for experts (independently from one another) and optimizing the gating function with respect to the obtained experts. The training procedure for MOËT, described by Algorithm 1, is based on this observation. First, the parameters of the gating function are randomly initialized (line 2). Then the experts are trained one by one. Each expert j is trained on a dataset D w of instances weighted by coefficients h (k) ij (line 5), by applying specific DT learning algorithm (line 6) that we adapted for MOE context (described below). After the experts are trained, an optimization step is performed (line 7) in order to increase the gating part of Eq. 3. At the end, the parameters are returned (line 8). Our tree learning procedure is as follows. Our technique modifies original MOE algorithm in that it uses DTs as experts. The fundamental difference with respect to traditional model comes from the fact that DTs do not rely on explicit and differentiable loss function which can be trained by gradient descent or Newton's methods. Instead, due to their discrete structure, they rely on a specific greedy training procedure. Therefore, the training of DTs has to be modified in order to take into account the attribution of instances to the experts given by coefficients h (k) ij, sometimes called responsibility of expert j for instance i. If these responsibilities were hard, meaning that each instance is assigned to strictly one expert, they would in partitioning the feature space into disjoint regions belonging to different experts. On the other hand, soft responsibilities are fractionally distributing each instance to different experts. The higher the responsibility of an expert j for an instance i, the higher the influence of that instance on that expert's training. In order to formulate this principle, we consider which way the instance influences construction of a tree. First, it affects the impurity measure computed when splitting the nodes and second, it influences probability estimates in the leaves of the tree. We address these two issues next. A commonly used impurity measure to determine splits in the tree is the Gini index. Let U be a set of indices of instances assigned to the node for which the split is being computed and D U set of corresponding instances. Let categorical outcomes of y be 1,..., C and for l = 1,..., C denote p l fraction of assigned instances for which it holds y = l. More formally, where I denotes indicator function of its argument expression and equals 1 if the expression is true. Then the Gini index G of the set D U is defined by: Considering that the assignment of instances to experts are fractional as defined by responsibility coefficients h (k) ij (which are provided to tree fitting function as weights of instances computed in line 5 of the algorithm), this definition has to be modified in that the instances assigned to the node should not be counted, but instead, their weights should be summed. Hence, we propose the following definition: and compute the Gini index for the set D U as G(p 1, . . .,p C). Similar modification can be performed for other impurity measures relying on distribution of outcomes of a categorical variable, like entropy. Note that while the instance assignments to experts are soft, instance assignments to nodes within an expert are hard, meaning sets of instances assigned to different nodes are disjoint. Probability estimate for y in the leaf node is usually performed by computing fractions of instances belonging to each class. In our case, the modification is the same as the one presented by Eq. 5. That way, estimates of probabilities P (y|x, θ j) needed by MOE are defined. In Algorithm 1, function f it tree performs decision tree training using the above modifications. We consider two ways to perform inference with respect to the obtained model. First one which we call MOËT, is performed by maximizing P (y|x, θ) with respect to y where this probability is defined by Eq. 1. The second way, which we call MOËT h, performs inference as arg max y P (y|x, θ arg max j gj (x,θg) ), meaning that we only rely on the most probable expert. Algorithm 1 MOËT training. for e ← 1 to N E do 4: Adaptation of MOËT to imitation learning. We integrate MOËT model into imitation learning approach of Viper by substituting training of DT with the MOËT training procedure. Expressiveness. Standard decision trees make their decisions by partitioning the feature space into regions which have borders perpendicular to coordinate axes. To approximate borders that are not perpendicular to coordinate axes very deep trees are usually necessary. MOËT h mitigates this shortcoming by exploiting hard softmax partitioning of the feature space using borders which are still hyperplanes, but need not be perpendicular to coordinate axes (see Section 3). This improves the expressiveness of the model. A MOËT h model is a combination of a linear model and several decision tree models. For interpretability which is preserved in Lipton's sense of transparency, it is important that a single DT is used for each prediction (instead of weighted average). Simultability of MOËT h consisting of DT and linear models is preserved because our models are small (2 ≤ depth ≤ 10) and we do not use high dimensional features , so a person can easily simulate the model. Similarly, decomposability is preserved because simple linear models without heavily engineered features and decision trees are decomposable and MOËT h is a simple combination of the two. Finally, algorithmic transparency is achieved because MOËT training relies on DT training for the experts and linear model training for the gate, both of which are well understood. However, the alternating refinement of initial feature space partitioning and experts makes the procedure more complicated, so our algorithmic transparency is partially achieved. Importantly, we define a well-founded translation of MOËT h models to SMT formulas, which opens a new range of possibilities for interpreting and validating the model using automated reasoning tools. SMT formulas provide a rich means of logical reasoning, where a user can ask the solver questions such as: "On which inputs do the two models differ?", or "What is the closest input to the given input on which model makes a different prediction?", or "Are the two models equivalent?", or "Are the two models equivalent in respect to the output class C?". Answers to these and similar questions can help better understand and compare models in a rigorous way. Also note that our symbolic reasoning of the gating function and decision trees allows us to construct SMT formulas that are readily handled by off-the-shelf tools, whereas direct SMT encodings of neural networks do not scale for any reasonably sized network because of the need for non-linear arithmetic reasoning. We now compare MOËT and Viper on four OpenAI Gym environments: CartPole, Pong, Acrobot and Mountaincar. For DRL agents, we use policy gradient model in CartPole, in other environments we use a DQN (training parameters provided in appendix). The rewards obtained by the agents on CartPole, Pong, Acrobot and Mountaincar are 200.00, 21.00, −68.60 and −105.27, respectively (higher reward is better). Rewards are averaged across 100 runs (250 in CartPole). Comparison of MOËT, MOËT h, and Viper policies. For CartPole, Acrobot, and Mountaincar environments, we train Viper DTs with maximum depths of {1, 2, 3, 4, 5}, while in the case of Pong we use maximum depths of {4, 6, 8, 10} as the problem is more complex and requires deeper trees. For experts in MOËT policies we use the same maximum depths as in Viper (except for Pong for which we use depths 1 to 9) and we train the policies for 2 to 8 experts (in case of Pong we train for {2, 4, 8} experts). We train all policies using 40 iterations of Viper algorithm, and choose the best performing policy in terms of rewards (and lower misprediction rate in case of equal rewards). We use two criteria to compare policies: rewards and mispredictions (number of times the student performs an action different from what a teacher would do). High reward indicates that the student learned more crucial parts of the teacher's policy, while a low misprediction rate indicates that in most cases student performs the same action as the teacher. In order to measure mispredictions, we run the student for number of runs, and compare actions it took to the actions teacher would perform. To ensure comparable depths for evaluating Viper and MOËT models while accounting for the different number of experts in MOËT, we introduce the notion of effective depth of a MOËT model as log 2 (E) + D, where E denotes the number of experts and D denotes the depth of each expert. Table 1 compares the performance of Viper, MOËT and MOËT h. The first column shows the depth of Viper decision trees and the corresponding effective depth for MOËT, rewards and mispredictions are shown in R and M columns resp. We show of the best performing MOËT configuration for a given effective depth chosen based on average for rewards and mispredictions, where e.g. E3:D2 denotes 3 experts with DTs of depth 2. All shown are averaged across 10 runs 1. For CartPole, Viper, MOËT and MOËT h all achieve perfect reward with depths of 2 and greater. More interestingly, for depth 2 MOËT and MOËT h obtain significantly lower average misprediction rates of 0.84% and 0.91% respectively compared to 16.65% for Viper. Even for larger depths, the misprediction rates for MOËT and MOËT h remain significantly lower. For Pong, we observe that MOËT and MOËT h consistently outperform Viper for all depths in terms of rewards and mispredictions, whereas MOËT and MOËT h have similar performance. For Acrobot, we similarly notice that both MOËT and MOËT h achieve consistently better rewards compared to Viper for all depths. Moreover, the misprediction rates are also significantly lower for MOËT and MOËT h in majority of the cases. Finally, for Mountaincar as well, we observe that MOËT and MOËT h both consistently outperform Viper with significantly higher rewards and lower misprediction rates. Moreover, in both of these environments, we observe that both MOËT and MOËT h achieve comparable reward and misprediction rates. Additional are presented in appendix. Analyzing the learned Policies. We analyze the learned student policies (Viper and MOËT h) by visualizing their state-action space, the differences between them, and differences with the teacher policy. We use the Mountaincar environment for this analysis because of the ease of visualizing its 2-dimensional state space comprising of car position (p) and car velocity (v) features, and 3 allowed actions left, right, and neutral. We visualize DRL, Viper and MOËT h policies in Figure 2, showing the actions taken in different parts of the state space (additional visualizations are in appendix). The state space is defined by feature bounds p ∈ [−1.2, 0.6] and v ∈ [−0.07, 0.07], which represent sets of allowed feature values in Mountaincar. We sample the space uniformly with a resolution 200 × 200. The actions left, neutral, and right are colored in green, yellow, and blue, respectively. Recall that MOËT h can cover regions whose borders are hyperplanes of arbitrary orientation, while Viper, i.e. DT can only cover regions whose borders are perpendicular to coordinate axes. This manifests in MOËT h policy containing slanted borders in yellow and green regions to capture more precisely the geometry of DRL policy, while the Viper policy only contains straight borders. Furthermore, we visualize mispredictions for Viper and MOËT h policies. While in previous section we calculated mispredictions by using student policy for playing the game, in this analysis we visualize mispredictions across the whole state space by sampling. Note that in some states (critical states) it is more important to get the action right, while in other states choosing non-optimal action does not affect the overall score much. Viper authors make use of this observation to weight states by their importance, and they use difference between Q values of optimal and non-optimal actions as a proxy for calculating how important (critical) state is. Importance score is calculated as follows: I(s) = max a∈A Q(s, a)−min a∈A Q(s, a), where Q(s, a) denotes the Q value of action a in state s, and A is a set of all possible actions. Using I function we weight mispredictions by their importance. We create a vector i consisting of importance scores for sampled points, and normalize it to range. We also create a binary vector z which is 1 in the case of misprediction (student policy decision is different from DRL decision) and 0 otherwise. We visualize m = z i (element-wise multiplication), where higher value indicates misprediction of higher importance and is denoted by a red color of higher intensity. Such normalized mispredictions (m) for Viper and MOËT h policies are shown in Figure 2d and Figure 2e respectively. We can observe that the MOËT h policy has fewer high intensity regions leading to fewer overall mispredictions. To provide a quantitative difference between the mispredictions of two policies, we compute M = (j m j / j i j) · 100, which is measure in bounds such that its value is 0 in the case of no mispredictions, and 100 in the case of all mispredictions. For the policies shown in Figure 2d and Figure 2e, we obtain M = 15.51 for Viper and M = 11.78 for MOËT h policies. We also show differences in mispredictions between Viper and MOËT h (Figure 2f) Translating MOËT to SMT. We now show the translation of MOËT policy to SMT constraints for verifying policy properties. We present an example translation of MOËT policy on CartPole environment with the same property specification that was proposed for verifying Viper policies . The goal in CartPole is to keep the pole upright, which can be encoded as a formula: where s i represents state after i steps, φ is the deviation of pole from the upright position. In order to encode this formula it is necessary to encode the transition function f t (s, a) which models environment dynamics: given a state and action it returns the next state of the environment. Also, it is necessary to encode the policy function π(s) that for a given state returns action to perform. There are two issues with verifying ψ: infinite time horizon; and the nonlinear transition function f t. To solve this problem, use a finite time horizon T max = 10 and linear approximation of the dynamics and we make the same assumptions. To encode π(s) we need to translate both the gating function and DT experts to logical formulas. Since the gating function in MOËT h uses exponential function, it is difficult to encode the function directly in Z3 as SMT solvers do not have efficient decision procedures to solve non-linear arithmetic. The direct encoding of exponentiation therefore leads to prohibitively complex Z3 formulas. We exploit the following simplification of gating function that is sound when hard prediction is used: First simplification is possible since the denominators for gatings of all experts are same, and second simplification is due to the monotonicity of the exponential function. For encoding DTs we use the same encoding as in Viper. To verify that ψ holds we need to show that ¬ψ is unsatisfiable. We run the verification with our MOËT h policies and show that ¬ψ is indeed unsatisfiable. To better understand the scalability of our verification procedure, we report on the verification times needed to verify policies for different number of experts and different expert depths in Figure 3. We observe that while MOËT h policies with 2 experts take from 2.6s to 8s for verification, the verification times for 8 experts can go up to as much as 319s. This directly corresponds to the complexity of the logical formula obtained with an increase in the number of experts. We introduced MOËT, a technique based on MOE with expert decision trees and presented a learning algorithm to train MOËT models. We then used MOËT models for interpreting DRL agent policies, where different local DTs specialize on different regions of input space and are combined into a global policy using a gating function. We showed that MOËT models lead to smaller, more faithful and performant representation of DRL agents compared to previous state-of-the-art approaches like Viper while still maintaining interpretability and verifiability. Algorithm 2 Viper training 1: procedure VIPER (MDP e, TEACHER π t, Q-FUNCTION Q πt, ITERATIONS N) 2: Initialize dataset and student: D ← ∅, π s0 ← π t 3: Sample trajectories and aggregate: Sample dataset using Q values: Train decision tree: return Best policy π s ∈ {π s1, ..., π s N}. Viper algorithm is shown in Algorithm 2. Here we present parameters we used to train DRL agents for different environments. For CartPole, we use policy gradient model as used in Viper. While we use the same model, we had to retrain it from scratch as the trained Viper agent was not available. For Pong, we use a deep Q-network (DQN) network , and we use the same model as in Viper, which originates from OpenAI baselines (OpenAI Baselines). For Acrobot and Mountaincar, we implement our own version of dueling DQN network following . We use 3 hidden layers with 15 neurons in each layer. We set the learning rate to 0.001, batch size to 30, step size to 10000 and number of epochs to 80000. We checkpoint a model every 5000 steps and pick the best performing one in terms of achieved reward. In this section we provide a brief description of environments we used in our experiments. We used four environments from OpenAI Gym: CartPole, Pong, Acrobot and Mountaincar. This environment consists of a cart and a rigid pole hinged to the cart, based on the system presented by Barto et al. . At the beginning pole is upright, and the goal is to prevent it from falling over. Cart is allowed to move horizontally within predefined bounds, and controller chooses to apply either left or right force to the cart. State is defined with four variables: x (cart position),ẋ (cart velocity), θ (pole angle), andθ (pole angular velocity). Game is terminated when the absolute value of pole angle exceeds 12 •, cart position is more than 2.4 units away from the center, or after 200 successful steps; whichever comes first. In each step reward of +1 is given, and the game is considered solved when the average reward is over 195 in over 100 consecutive trials. This is a classical Atari game of table tennis with two players. Minimum possible score is −21 and maximum is 21. This environment is analogous to a gymnast swinging on a horizontal bar, and consists of a two links and two joins, where the joint between the links is actuated. The environment is based on the system presented by Sutton . Initially both links are pointing downwards, and the goal is to swing the end-point (feet) above the bar for at least the length of one link. The state consists of six variables, four variables consisting of sin and cos values of the joint angles, and two variables for angular velocities of the joints. The action is either applying negative, neutral, or positive torque on the joint. At each time step reward of −1 is received, and episode is terminated upon successful reaching the height, or after 200 steps, whichever comes first. Acrobot is an unsolved environment in that there is no reward limit under which is considered solved, but the goal is to achieve high reward. This environment consists of a car positioned between two hills, with a goal of reaching the hill in front of the car. The environment is based on the system presented by Moore . Car can move in a one-dimensional track, but does not have enough power to reach the hill in one go, thus it needs to build momentum going back and forth to finally reach the hill. Controller can choose left, right or neutral action to apply left, right or no force to the car. State is defined by two variables, describing car position and car velocity. In each step reward of −1 is received, and episode is terminated upon reaching the hill, or after 200 steps, whichever comes first. The game is considered solved if average reward over 100 consecutive trials is no less than −110. In this section we provide visualization of a gating function. Figure 4 shows how gating function partitions the state space for which different experts specialize. Gatings of MOËT h policy with 4 experts and depth 1 are shown. E ADDITIONAL TABLES Table 2 shows similar to Table 1, but here in addition to averaging across multiple trained models, it averages across multiple MOËT configurations that have the same effective depth. Table 3 shows the of best performing DRL, MOËT and MOËT h models on the evaluation subjects.

represent the tldr for finding the corresponding scientific article

Explainable reinforcement learning model using novel combination of mixture of experts with non-differentiable decision tree experts.

Represent the TLDR for finding the corresponding scientific article:

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.


Question: Represent the scientific article to find a short TLDR

Question: We consider the problem of unconstrained minimization of a smooth objective function in $\mathbb{R}^d$ in setting where only function evaluations are possible. We propose and analyze stochastic zeroth-order method with heavy ball momentum. In particular, we propose, SMTP, a momentum version of the stochastic three-point method (STP). We show new complexity for non-convex, convex and strongly convex functions. We test our method on a collection of learning to continuous control tasks on several environments with varying difficulty and compare against STP, other state-of-the-art derivative-free optimization algorithms and against policy gradient methods. SMTP significantly outperforms STP and all other methods that we considered in our numerical experiments. Our second contribution is SMTP with importance sampling which we call SMTP_IS. We provide convergence analysis of this method for non-convex, convex and strongly convex objectives. In this paper, we consider the following minimization problem where f: R d → R is "smooth" but not necessarily a convex function in a Derivative-Free Optimization (DFO) setting where only function evaluations are possible. The function f is bounded from below by f (x *) where x * is a minimizer. Lastly and throughout the paper, we assume that f is L-smooth. DFO. In DFO setting; , the derivatives of the objective function f are not accessible. That is they are either impractical to evaluate, noisy (function f is noisy) or they are simply not available at all. In standard applications of DFO, evaluations of f are only accessible through simulations of black-box engine or software as in reinforcement learning and continuous control environments. This setting of optimization problems appears also in applications from computational medicine and fluid dynamics; Haslinger & Mäckinen; to localization Marsden et al. (2004; 2007) and continuous control; to name a few. The literature on DFO for solving is long and rich. The first approaches were based on deterministic direct search (DDS) and they span half a century of work;;. However, for DDS methods complexity bounds have only been established recently by the work of Vicente and coauthors;. In particular, the work of showed the first complexity on non-convex f and the were extended to better complexities when f is convex. However, there have been several variants of DDS, including randomized approaches; Karmanov (1974a; b);;;. Only very recently, complexity bounds have also been derived for randomized methods;;;;. For instance, the work of; imposes a decrease condition on whether to accept or reject a step of a set of random directions. derived new complexity bounds when the random directions are normally distributed vectors for both smooth and non-smooth f. They proposed both accelerated and non-accelerated zero-order (ZO) methods. Accelerated derivative-free methods in the case of inexact oracle information was proposed in. An extension of for non-Euclidean proximal setup was proposed by for the smooth stochastic convex optimization with inexact oracle. More recently and closely related to our work, proposed a new randomized direct search method called Stochastic Three Points (STP). At each iteration k STP generates a random search direction s k according to a certain probability law and compares the objective function at three points: current iterate x k, a point in the direction of s k and a point in the direction of −s k with a certain step size α k. The method then chooses the best of these three points as the new iterate: The key properties of STP are its simplicity, generality and practicality. Indeed, the update rule for STP makes it extremely simple to implement, the proofs of convergence for STP are short and clear and assumptions on random search directions cover a lot of strategies of choosing decent direction and even some of first-order methods fit the STP scheme which makes it a very flexible in comparison with other zeroth-order methods (e.g. two-point evaluations methods like in , , , that try to approximate directional derivatives along random direction at each iteration). Motivated by these properties of STP we focus on further developing of this method. 1 is a special technique introduced by Polyak in 1964 Polyak (1964 to get faster convergence to the optimum for the first-order methods. In the original paper, Polyak proved that his method converges locally with O L /µ log 1 /ε rate for twice continuously differentiable µ-strongly convex and L-smooth functions. Despite the long history of this approach, there is still an open question whether heavy ball method converges to the optimum globally with accelerated rate when the objective function is twice continuous differentiable, L-smooth and µ-strongly convex. For this class of functions, only non-accelerated global convergence was proved and for the special case of quadratic strongly convex and L-smooth functions Lessard et. al. recently proved asymptotic accelerated global convergence. However, heavy ball method performs well in practice and, therefore, is widely used. One can find more detailed survey of the literature about heavy ball momentum in Loizou & Richtárik. Importance Sampling. Importance sampling has been celebrated and extensively studied in stochastic gradient based methods or in coordinate based methods Richtárik & Takáč. Only very recently, proposed, STP_IS, the first DFO algorithm with importance sampling. In particular, under coordinate-wise smooth function, they show that sampling coordinate directions, can be generalized to arbitrary directions, with probabilities proportional to the function coordinate smoothness constants, improves the leading constant by the same factor typically gained in gradient based methods. Contributions. Our contributions can be summarized into three folds. • First ZO method with heavy ball momentum. Motivated by practical effectiveness of first-order momentum heavy ball method, we introduce momentum into STP method and propose new DFO algorithm with heavy ball momentum (SMTP). We summarized the method in Algorithm 1, with theoretical guarantees for non-convex, convex and strongly convex functions under generic sampling directions D. We emphasize that the SMTP with momentum is not a straightforward generalization of STP and Polyak's method and requires insights from virtual iterates analysis from. To the best of our knowledge it is the first analysis of derivative-free method with heavy ball momentum, i.e. we show that the same momentum trick that works for the first order method could be applied for zeroth-order methods as well. • First ZO method with both heavy ball momentum and importance sampling. In order to get more gain from momentum in the case when the sampling directions are coordinate directions and the objective function is coordinate-wise L-smooth (see Assumption 4.1), we consider importance sampling to the above method. In fact, we propose the first zeroth-order Algorithm 1 SMTP: Stochastic Momentum Three Points Require: learning rates {γ where we propose an importance sampling that improves the leading constant marked in red. Note that r 0 = f (x 0) − f (x *) and that all assumptions listed are in addition to Assumption 2.1. Complexity means number of iterations in order to guarantee E ∇f (z K) D ≤ ε for the non-convex case, E f (z K) − f (x *) ≤ ε for convex and strongly convex cases. R 0 < ∞ is the radius in · * D -norm of a bounded level set where the exact definition is given in Assumption 3.2. We notice that for STP_IS · D = · 1 and · * D = · ∞ in non-convex and convex cases and · D = · 2 in the strongly convex case. momentum method with importance sampling (SMTP_IS) summarized in Algorithm 2 with theoretical guarantees for non-convex, convex and strongly convex functions. The details and proofs are left for Section 4 and Appendix E. • Practicality. We conduct extensive experiments on continuous control tasks from the MuJoCo suite following recent success of DFO compared to modelfree reinforcement learning;. We achieve with SMTP_IS the state-of-the-art on across all tested environments on the continuous control outperforming and policy gradient methods;. We provide more detailed comparison of SMTP and SMTP_IS in Section E.4 of the Appendix. We use · p to define p -norm of the vector for p ≥ 1 and As we mention in the introduction we assume throughout the paper 2 that the objective function f is L-smooth. Assumption 2.1. (L-smoothness) We say that f is L-smooth if From this definition one can obtain and if additionally f is convex, i.e. f (y) ≥ f (x) + ∇f (x), y − x, we have Our analysis of SMTP is based on the following key assumption. Assumption 3.1. The probability distribution D on R d satisfies the following properties: Some examples of distributions that meet above assumption are described in Lemma 3.4 from. For convenience we provide the statement of the lemma in the Appendix (see Lemma F.1). Recall that one possible view on is as following. If we substitute gradient ∇f (x k) in the update rule for the gradient descent by ±s k where s k is sampled from distribution D satisfied Assumption 3.1 and then select x k+1 as the best point in terms of functional value among we will get exactly STP method. However, gradient descent is not the best algorithm to solve unconstrained smooth minimization problems and the natural idea is to try to perform the same substitution-trick with more efficient first-order methods than gradient descent. We put our attention on Polyak's heavy ball method where the update rule could be written in the following form: By definition of z k+1, we get that the sequence {f (z k)} k≥0 is monotone: Now, we establish the key which will be used to prove the main complexity and remaining theorems in this section. Lemma 3.1. Assume that f is L-smooth and D satisfies Assumption 3.1. Then for the iterates of SMTP the following inequalities hold: and 3.1 NON-CONVEX CASE In this section, we show our complexity for Algorithm 1 in the case when f is allowed to be non-convex. In particular, we show that SMTP in Algorithm 1 guarantees complexity bounds with the same order as classical bounds, i.e. 1/ √ K where K is the number of iterations, in the literature. We notice that query complexity (i.e. number of oracle calls) of SMTP coincides with its iteration complexity up to numerical constant factor. For clarity and completeness, proofs are left for the appendix. Theorem 3.1. Let Assumptions 2.1 and 3.1 be satisfied. Let SMTP with γ k ≡ γ > 0 produce points {z 0, z 1, . . ., z K−1} and z K is chosen uniformly at random among them. Then Moreover, if we choose γ = γ0 √ K the complexity reduces to minimizes the right-hand side of and for this choice we have In other words, the above theorem states that SMTP converges no worse than STP for non-convex problems to the stationary point. In the next sections we also show that theoretical convergence guarantees for SMTP are not worse than for STP for convex and strongly convex problems. However, in practice SMTP significantly outperforms STP. So, the relationship between SMTP and STP correlates with the known in the literature relationship between Polyak's heavy ball method and gradient descent. In this section, we present our complexity for Algorithm 1 when f is convex. In particular, we show that this method guarantees complexity bounds with the same order as classical bounds, i.e. 1/K, in the literature. We will need the following additional assumption in the sequel. Assumption 3.2. We assume that f is convex, has a minimizer x * and has bounded level set at x 0: where From the above assumption and Cauchy-Schwartz inequality we get the following implication: Theorem 3.2 (Constant stepsize). Let Assumptions 2.1, 3.1 and 3.2 be satisfied. If we set γ k ≡ γ <, then for the iterates of SMTP method the following inequality holds: If we choose γ = and run SMTP for k = K iterations where In order to get rid of factor ln in the complexity we consider decreasing stepsizes. (1−β)R0 and θ ≥ 2 α, then for the iterates of SMTP method the following inequality holds: where (1−β)R0 and run SMTP for k = K iterations where We notice that if we choose β sufficiently close to 1, we will obtain from the formula that In this section we present our complexity for Algorithm 1 when f is µ-strongly convex. Assumption 3.3. We assume that f is µ-strongly convex with respect to the norm · * D: It is well known that strong convexity implies Theorem 3.4 (Solution-dependent stepsizes). Let Assumptions 2.1, 3.1 and 3.3 be satisfied. If we set, then for the iterates of SMTP, the following inequality holds: Then, If we run SMTP for k = K iterations where where is the condition number of the objective, we will get E f (z Note that the previous uses stepsizes that depends on the optimal solution f (x *) which is often not known in practice. The next theorem removes this drawback without spoiling the convergence rate. However, we need an additional assumption on the distribution D and one extra function evaluation. Assumption 3.4. We assume that for all s ∼ D we have s 2 = 1. Theorem 3.5 (Solution-free stepsizes). Let Assumptions 2.1, 3.1, 3.3 and 3.4 be satisfied. If additionally we compute, then for the iterates of SMTP the following inequality holds: Moreover, for any ε > 0 if we set t such that and run SMTP for k = K iterations where where In this section we consider another assumption, in a similar spirit to , on the objective. Assumption 4.1 (Coordinate-wise L-smoothness). We assume that the objective f has coordinatewise Lipschitz gradient, with Lipschitz constants where ∇ i f (x) is i-th partial derivative of f at the point x. For this kind of problems we modify SMTP and present STMP_IS method in Algorithm 2. In general, the idea behind methods with importance sampling and, in particular, behind SMTP_IS is to adjust probabilities of sampling in such a way that gives better convergence guarantees. In the case when f satisfies coordinate-wise L-smoothness and Lipschitz constants L i are known it is natural to sample direction s k = e i with probability depending on L i (e.g. proportional to L i). One can find more detailed discussion of the importance sampling in and Richtárik & Takáč. Now, we establish the key which will be used to prove the main complexity of STMP_IS. Lemma 4.1. Assume that f satisfies Assumption 4.1. Then for the iterates of SMTP_IS the following inequalities hold: and Due to the page limitation, we provide the complexity of SMTP_IS in the Appendix. Require: stepsize parameters w 1,..., w n > 0, probabilities p 1,..., p n > 0 summing to 1, starting point Select i k = i with probability p i > 0 Choose stepsize γ k i proportional to Experimental Setup. We conduct extensive experiments 3 on challenging non-convex problems on the continuous control task from the MuJoCO suit. In particular, we address the problem of model-free control of a dynamical system. Policy gradient methods for model-free reinforcement learning algorithms provide an off-the-shelf model-free approach to learn how to control a dynamical system and are often benchmarked in a simulator. We compare our proposed momentum stochastic three points method SMTP and the momentum with importance sampling version SMTP_IS against state-of-art DFO based methods as STP_IS and. Moreover, we also compare against classical policy gradient methods as and. We conduct experiments on several environments with varying difficulty Swimmer-v1, Hopper-v1, HalfCheetah-v1, Ant-v1, and Humanoid-v1. Note that due to the stochastic nature of problem where f is stochastic, we use the mean of the function values of f (, see Algorithm 1, over K observations. Similar to the work in , we use K = 2 for Swimmer-v1, K = 4 for both Hopper-v1 and HalfCheetah-v1, K = 40 for Ant-v1 and Humanoid-v1. Similar to , these values were chosen based on the validation performance over the grid that is K ∈ {1, 2, 4, 8, 16} for the smaller dimensional problems Swimmer-v1, Hopper-v1, HalfCheetah-v1 and K ∈ {20, 40, 80, 120} for larger dimensional problems Ant-v1, and Humanoid-v1. As for the momentum term, for SMTP we set β = 0.5. For SMTP_IS, as the smoothness constants are not available for continuous control, we use the coordinate smoothness constants of a θ parameterized smooth functionf θ (multi-layer perceptron) that estimates f. In particular, consider running any DFO for n steps; with the queried sampled 2. for further implementation details as we follow the same experimental procedure. In contrast to STP_IS, our method (SMTP) does not required sampling from directions in the canonical basis; hence, we use directions from standard Normal distribution in each iteration. For SMTP_IS, we follow a similar procedure as and sample from columns of a random matrix B. Similar to the standard practice, we perform all experiments with 5 different initialization and measure the average reward, in continuous control we are maximizing the reward function f, and best and worst run per iteration. We compare algorithms in terms of reward vs. sample complexity. Comparison Against STP. Our method improves sample complexity of STP and STP_IS significantly. Especially for high dimensional problems like Ant-v1 and Humanoid-v1, sample efficiency of SMTP is at least as twice as the STP. Moreover, SMTP_IS helps in some experiments by Table 2 to demonstrate complexity of each method. improving over SMTP. However, this is not consistent in all environments. We believe this is largely due to the fact that SMTP_IS can only handle sampling from canonical basis similar to STP_IS. Comparison Against State-of-The-Art. We compare our method with state-of-the-art DFO and policy gradient algorithms. For the environments, Swimmer-v1, Hopper-v1, HalfCheetah-v1 and Ant-v1, our method outperforms the state-of-the-art . Whereas for Humanoid-v1, our methods in a comparable sample complexity. We have proposed, SMTP, the first heavy ball momentum DFO based algorithm with convergence rates for non-convex, convex and strongly convex functions under generic sampling direction. We specialize the sampling to the set of coordinate bases and further improve rates by proposing a momentum and importance sampling version SMPT_IS with new convergence rates for non-convex, convex and strongly convex functions too. We conduct large number of experiments on the task of controlling dynamical systems. We outperform two different policy gradient methods and achieve comparable or better performance to the best DFO algorithm (ARS) on the respective environments. Assumption A.2. The probability distribution D on R d satisfies the following properties: 2 is positive and finite. 2. There is a constant µ D > 0 and norm We establish the key lemma which will be used to prove the theorems stated in the paper. Lemma A.1. Assume that f is L-smooth and D satisfies Assumption A.2. Then for the iterates of SMTP the following inequalities hold: and Proof. By induction one can show that That is, for k = 0 this recurrence holds and update rules for z k, x k and v k−1 do not brake it. From this we get Similarly, Unifying these two inequalities we get which proves. Finally, taking the expectation E s k ∼D of both sides of the previous inequality and invoking Assumption A.2, we obtain Theorem B.1. Let Assumptions A.1 and A.2 be satisfied. Let SMTP with γ k ≡ γ > 0 produce points {z 0, z 1, . . ., z K−1} and z K is chosen uniformly at random among them. Then Moreover, if we choose γ = γ0 √ K the complexity reduces to minimizes the right-hand side of and for this choice we have Proof. Taking full expectation from both sides of inequality we get Further, summing up the for k = 0, 1,..., K −1, dividing both sides of the obtained inequality by K and using tower property of the mathematical expectation we get The last part where γ = γ0 √ K is straightforward. Assumption C.1. We assume that f is convex, has a minimizer x * and has bounded level set at x 0: where Theorem C.1 (Constant stepsize). Let Assumptions A.1, A.2 and C.1 be satisfied. If we set γ k ≡ γ <, then for the iterates of SMTP method the following inequality holds: If we choose γ = and run SMTP for k = K iterations where then we will get E f (z Proof. From the and monotonicity of {f (z k)} k≥0 we have Taking full expectation, subtracting f (x *) from the both sides of the previous inequality and using the tower property of mathematical expectation we get Since γ < (1−β)R0 is positive and we can unroll the recurrence: Lastly, putting γ = in we have Next we use technical lemma from. We provide the original proof for completeness. Lemma C.1 (Lemma 6 from). Let a sequence {a k} k≥0 satisfy inequality and take C such that N ≤ αθ 4 C and a 0 ≤ C. Then, it holds Proof. We will show the inequality for a k by induction. Since inequality a 0 ≤ C is one of our assumptions, we have the initial step of the induction. To prove the inductive step, consider To show that the right-hand side is upper bounded by θC α(k+1)+θ, one needs to have, after multiplying both sides by (αk + θ)(αk + α + θ)(θC) which is equivalent to The last inequality is trivially satisfied for all k ≥ 0. (1−β)R0 and θ ≥ 2 α, then for the iterates of SMTP method the following inequality holds: where (1−β)R0 and run SMTP for k = K iterations where Proof. In we proved that Having that, we can apply Lemma C.1 to the sequence E f (z k) − f (x *). The constants for the lemma are: α 2 k+2 is equivalent to the choice θ = 2 α. In this case, we have αθ = 2,. Putting these parameters and K from in the we get the . Assumption D.1. We assume that f is µ-strongly convex with respect to the norm · * D: It is well known that strong convexity implies Theorem D.1 (Solution-dependent stepsizes). Let Assumptions A.1, A.2 and D.1 be satisfied. If we set, then for the iterates of SMTP the following inequality holds: If we run SMTP for k = K iterations where where κ def = L µ is the condition number of the objective, we will get E f (z Proof. From and Using θ = inf and taking the full expectation from the previous inequality we get Lastly, from we have ≤ ε., then for the iterates of SMTP the following inequality holds: Moreover, for any ε > 0 if we set t such that and run SMTP for k = K iterations where where Proof. Recall that from we have If we minimize the right hand side of the previous inequality as a function of γ k, we will get that the optimal choice in this sense is γ. However, this stepsize is impractical for derivative-free optimization, since it requires to know ∇f (z k). The natural way to handle this is to and that is what we do. We choose Next we estimate |δ k |: It implies that + Lt 2 8 and after taking full expectation from the both sides of the obtained inequality we get Note that from the tower property of mathematical expectation and Jensen's inequality we have Putting all together we get Lastly, from we have Again by definition of z k+1 we get that the sequence {f (z k)} k≥0 is monotone: Lemma E.1. Assume that f satisfies Assumption 4.1. Then for the iterates of SMTP_IS the following inequalities hold: and Proof. In the similar way as in Lemma A.1 one can show that and It implies that Unifying these two inequalities we get which proves. Finally, taking the expectation E[· | z k] conditioned on z k from the both sides of the previous inequality we obtain Theorem E.1. Assume that f satisfies Assumption 4.1. Let SMTP_IS with γ k i = γ wi k for some γ > 0 produce points {z 0, z 1, . . ., z K−1} and z K is chosen uniformly at random among them. Then Moreover, if we choose γ = in order to minimize right-hand side of, we will get Note that for Li with w i = L i we have that the rates improves to Proof. Recall that from we have Putting it in and taking full expectation from the both sides of obtained inequality we get Summing up previous inequality for k = 0, 1,..., K − 1 and dividing both sides of the by K, we get It remains to notice that As for SMTP to tackle convex problems by SMTP_IS we use Assumption 3.2 with Theorem E.2 (Constant stepsize). Let Assumptions 3.2 and 4.1 be satisfied. If we set γ, then for the iterates of SMTP_IS method the following inequality holds: and run SMTP_IS for k = K iterations where we will get E f (z Li with w i = L i, the rate improves to Proof. Recall that from we have and Putting it in and taking full expectation from the both sides of obtained inequality we get pi wi we have that the factor 1 − pi wi is nonnegative and, therefore, pi wi and k = K from in we have (1−β)R0 and θ ≥ 2 α, then for the iterates of SMTP_IS method the following inequality holds: where (1−β)R0 and run SMTP_IS for k = K iterations where Proof. In we proved that Under review as a conference paper at ICLR 2020 Having that, we can apply Lemma C.1 to the sequence E f (z k) − f (x *). The constants for the lemma are: α. In this case we have αθ = 2 and C = max f (x 0) − f (x *),. Putting these parameters and K from in the we get the . Theorem E.4 (Solution-dependent stepsizes). Let Assumptions 3.3 (with · D = · 1) and 4.1 be satisfied. If we set γ, then for the iterates of SMTP_IS method the following inequality holds: If we run SMTP_IS for k = K iterations where we will get E f (z Proof. Recall that from we have ≤ ε. The previous based on the choice of γ k which depends on the f (z k) − f (x *) which is often unknown in practice. The next theorem does not have this drawback and makes it possible to obtain the same rate of convergence as in the previous theorem using one extra function evaluation. Theorem E.5 (Solution-free stepsizes). Let Assumptions 3.3 (with · D = · 2) and 4.1 be satisfied. Li k t for t > 0, then for the iterates of SMTP_IS method the following inequality holds: Moreover, for any ε > 0 if we set t such that and run SMTP_IS for k = K iterations where we will get E f (z Proof. Recall that from we have 2(1 − β) 2. If we minimize the right hand side of the previous inequality as a function of γ k i, we will get that the optimal choice in this sense is γ. However, this stepsize is impractical for derivative-free optimization, since it requires to know ∇ i k f (z k). The natural way to handle this is to and that is what we do. We choose γ. From this we get Next we estimate |δ k i |: It implies that 8 and after taking expectation E · | z k conditioned on z k from the both sides of the obtained inequality we get Note that, and Putting all together we get Taking full expectation from the previous inequality we get Since µ ≤ L i for all i = 1,..., d we have Lastly, from we have ≤ ε 2 + ε 2 = ε. Here we compare SMTP when D is normal distribution with zero mean and Table 3 summarizes complexities in this case. We notice that for SMTP we have · D = · 2. That is why one needs to compare SMTP with SMTP_IS accurately. At the first glance, Table 3 says that for non-convex and convex cases we get an extra d factor in the complexity of SMTP_IS when L 1 =... = L d = L. However, it is natural since we use different norms for SMTP and SMTP_IS. In the non-convex case for SMTP we give number of iterations in order to guarantee E ∇f (z K) 2 ≤ ε while for SMTP_IS we provide number of iterations in order to guarantee E ∇f (z K) 1 ≤ ε. From Holder's inequality · 1 ≤ √ d · 2 and, therefore, in order to have E ∇f (z K) 1 ≤ ε for SMTP we need to ensure. That is, to guarantee E ∇f (z K) 1 ≤ ε SMTP for aforementioned distribution needs to perform Analogously, in the convex case using Cauchy-Schwartz inequality · 2 ≤ √ d · ∞ we have that R 0, 2 ≤ √ dR 0, ∞. Typically this inequality is tight and if we assume that R 0, ∞ ≥ C

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We develop and analyze a new derivative free optimization algorithm with momentum and importance sampling with applications to continuous control.

For complex constraints in which it is not easy to estimate the gradient, we use the discounted penalty as a guiding signal. We prove that under certain assumptions it converges to a feasible solution.


Question: Represent the scientific article to find a short TLDR:

Question: Using class labels to represent class similarity is a typical approach to training deep hashing systems for retrieval; samples from the same or different classes take binary 1 or 0 similarity values. This similarity does not model the full rich knowledge of semantic relations that may be present between data points. In this work we build upon the idea of using semantic hierarchies to form distance metrics between all available sample labels; for example cat to dog has a smaller distance than cat to guitar. We combine this type of semantic distance into a loss function to promote similar distances between the deep neural network embeddings. We also introduce an empirical Kullback-Leibler divergence loss term to promote binarization and uniformity of the embeddings. We test the ing SHREWD method and demonstrate improvements in hierarchical retrieval scores using compact, binary hash codes instead of real valued ones, and show that in a weakly supervised hashing setting we are able to learn competitively without explicitly relying on class labels, but instead on similarities between labels. Content-Based Image Retrieval (CBIR) on very large datasets typically relies on hashing for efficient approximate nearest neighbor search; see e.g. BID12 for a review. Early methods such as (LSH) BID5 were data-independent, but Data-dependent methods (either supervised or unsupervised) have shown better performance. Recently, Deep hashing methods using CNNs have had much success over traditional methods, see e.g. Hashnet BID1, DADH. Most supervised hashing techniques rely on a pairwise binary similarity matrix S = {s ij}, whereby s ij = 1 for images i and j taken from the same class, and 0 otherwise. A richer set of affinity is possible using semantic relations, for example in the form of class hierarchies. BID13 consider the semantic hierarchy for non-deep hashing, minimizing inner product distance of hash codes from the distance in the semantic hierarchy. In the SHDH method, the pairwise similarity matrix is defined from such a hierarchy according to a weighted sum of weighted Hamming distances. In Unsupervised Semantic Deep Hashing , semantic relations are obtained by looking at embeddings on a pre-trained VGG model on Imagenet. The goal of the semantic loss here is simply to minimize the distance between binarized hash codes and their pre-trained embeddings, i.e. neighbors in hashing space are neighbors in pre-trained feature space. This is somewhat similar to our notion of semantic similarity except for using a pre-trained embedding instead of a pre-labeled semantic hierarchy of relations. BID14 consider class-wise Deep hashing, in which a clustering-like operation is used to form a loss between samples both from the same class and different levels from the hierarchy. Recently BID0 explored image retrieval using semantic hierarchies to design an embedding space, in a two step process. Firstly they directly find embedding vectors of the class labels on a unit hypersphere, using a linear algebra based approach, such that the distances of these embeddings are similar to the supplied hierarchical similarity. In the second stage, they train a standard CNN encoder model to regress images towards these embedding vectors. They do not consider hashing in their work. We also make use of hierarchical relational distances in a similar way to constrain our embeddings. However compared to our work, BID0 consider continuous representations and require the embedding dimension to equal the number of classes, whereas we learn compact quantized hash codes of arbitrary length, which are more practical for real world retrieval performance. Moreover, we do not directly find fixed target embeddings for the classes, but instead require that the neural network embeddings will be learned in conjunction with the network weights, to best match the similarities derived from the labels. And unlike BID14, in our weakly supervised SHREWD method, we do not require explicit class membership, only relative semantic distances to be supplied. Let (x, y) denote a training example pair consisting of an image and some (possibly weakly) supervised target y, which can be a label, tags, captions etc. The embeddings are defined asẑ = f θ (x) for a deep neural network f parameterized by weights θ. Instead of learning to predict the target y, we assume that there exists an estimate of similarity between targets, d(y, y). The task of the network is then to learn this similarity by attempting to match ẑ −ẑ with d(y, y) under some predefined norm in the embedding space. While in this work we use class hierarchies to implicitly inform our loss function via the similarity metric d, in general our formulation is weakly supervised in the sense that these labels themselves are not directly required as targets. We could equally well replace this target metric space with any other metric based on for instance web-mined noisy tag distances in a word embedding space such as GloVe or word2vec, as in BID4, or ranked image similarities according to recorded user preferences. In addition to learning similarities between images, it is important to try to fully utilize the available hashing space in order to facilitate efficient retrieval by using the Hamming distance to rank most similar images to a given query image. Consider for example a perfect ImageNet classifier. We could trivially map all 1000 class predictions to a 10-bit hash code, which would yield a perfect mAP score. The retrieval performance of such a "mAP-miner" model would however be poor, because the model is unable to rank examples both within a given class and between different classes BID3. We therefore introduce an empirical Kullback-Leibler (KL) divergence term between the embedding distribution and a (near-)binary target distribution, which we add as an additional loss term. The KL loss serves an additional purpose in driving the embeddings close to binary values in order to reduce the information loss due to binarizing the embeddings. We next describe the loss function, L(θ), that we minimize in order to train our CNN model. We break down our approach into the following 3 parts: DISPLAYFORM0 L cls represents a traditional categorical cross-entropy loss on top of a linear layer with softmax placed on the non-binarized latent codes. The meaning and use of each of the other two terms are described in more detail below. Similar to BID0 we consider variants with and without the L cls, giving variants of the algorithm we term SHREWD (weakly supervised, no explicit class labels needed) and SHRED (fully supervised). In order to weakly supervise using a semantic similarity metric, we seek to find affinity between the normalized distances in the learned embedding space and normalized distances in the semantic space. Therefore we define DISPLAYFORM0 where B is a minibatch size,... M denotes Manhattan distance (because in the end we will measure similarity in the binary space by Hamming distance), d (y b, y b) is the given ground truth similarity and w bb is an additional weight, which is used to give more weight to similar example pairs (e.g. cat-dog) than distant ones (e.g. cat-moon). τ z and τ y are normalizing scale factors estimated Note that while L cls performs best on supervised classification, L sim allows for better retrieval performance, however this is degraded unless L KL is also included to regularize towards binary embeddings. For measuring classification accuracy on methods that don't include L cls, we measure using a linear classifier with the same structure as in L cls trained on the output of the first network.from the current batch. We use a slowly decaying form for the weight, DISPLAYFORM1 with parameter values γ = 0.1 and ρ = 2. Our empirical loss for minimizing the KL divergence KL(p||q). = dzp(z) log(p(z)/q(z)) between the sample embedding distribution p(z) and a target distribution q(z) is based on the Kozachenko-Leonenko estimator of entropy BID7, and can be defined as DISPLAYFORM0 where ν(ẑ b ; z) denotes the distance ofẑ b to a nearest vector z b, where z is a sample (of e.g. size B) of vectors from a target distribution. We employ the beta distribution with parameters α = β = 0.1 as this target distribution, which is thus moderately concentrated to binary values in the embedding space. The is that our embedding vectors will be regularized towards uniform binary values, whilst still enabling continuous backpropagation though the network and giving some flexibility in allowing the distance matching loss to perform its job. When quantized, the ing embeddings are likely to be similar to their continuous values, meaning that the binary codes will have distances more similar to their corresponding semantic distances. Metrics As discussed in section 2, the mAP score can be a misleading metric for retrieval performance when using class information only. Similarly to other works such as BID2 BID0, we focus on measuring the retrieval performance taking semantic hierarchical relations into account by the mean Average Hierarchical Precision (mAHP). However more in line with other hashing works, we use the hamming distance of the binary codes for ranking the retrieved . We first test on CIFAR-100 BID8 using the same semantic hierarchy and Resnet-110w architecture as in BID0, where only the top fully connected layer is replaced to return embeddings at the size of the desired hash length. See TAB2 for comparisons with previous methods, an ablation study, and effects of hash code length. ILSVRC 2012 We also evaluate on the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2012 dataset. For similarity labels, we use the same tree-structured WordNet hierarchy as in BID0. We use a standard Resnet-50 architecture with a fully connected hashing layer as before. Retrieval are summarized in TAB3. We compare the ing Hierarchical Precision scores with and without L KL, for binarized and continuous values in FIG1. We see that our improve on the previously reported hierarchical retrieval whilst using quantized embeddings, enabling efficient retrieval. We see a substantial drop in the precision after binarization when not using the KL loss. Also binarization does not cause as severe a drop in precision when using the KL loss. We approached Deep Hashing for retrieval, introducing novel combined loss functions that balance code binarization with equivalent distance matching from hierarchical semantic relations. We have demonstrated new state of the art for semantic hierarchy based image retrieval (mAHP scores) on CIFAR and ImageNet with both our fully supervised (SHRED) and weakly-supervised (SHREWD) methods.

We propose a new method for training deep hashing for image retrieval using only a relational distance metric between samples

We generate examples to explain a classifier desicion via interpolations in latent space. The variational auto encoder cost is extended with a functional of the classifier over the generated example path in data space.