Question: Represent

Question: We propose a new approach, known as the iterative regularized dual averaging (iRDA), to improve the efficiency of convolutional neural networks (CNN) by significantly reducing the redundancy of the model without reducing its accuracy. The method has been tested for various data sets, and proven to be significantly more efficient than most existing compressing techniques in the deep learning literature. For many popular data sets such as MNIST and CIFAR-10, more than 95% of the weights can be zeroed out without losing accuracy. In particular, we are able to make ResNet18 with 95% sparsity to have an accuracy that is comparable to that of a much larger model ResNet50 with the best 60% sparsity as reported in the literature. In recent decades, deep neural network models have achieved unprecedented success and state-ofthe-art performance in various tasks of machine learning or artificial intelligence, such as computer vision, natural language processing and reinforcement learning BID11. Deep learning models usually involve a huge number of parameters to fit variant kinds of datasets, and the number of data may be much less than the amount of parameters BID9. This may implicate that deep learning models have too much redundancy. This can be validated by the literatures from the general pruning methods BID18 to the compressing models BID6.While compressed sensing techniques have been successfully applied in many other problems, few reports could be found in the literature for their application in deep learning. The idea of sparsifying machine learning models has attracted much attention in the last ten years in machine learning BID2; BID22. When considering the memory and computing cost for some certain applications such as Apps in mobile, the sparsity of parameters plays a very important role in model compression BID6; BID0. The topic of computing sparse neural networks can be included in the bigger topic on the compression of neural networks, which usually further involves the speedup of computing the compressed models. There are many sparse methods in machine learning models such as FOBOS method BID3, also known as proximal stochastic gradient descent (prox-SGD) methods BID16, proposed for general regularized convex optimization problem, where 1 is a common regularization term. One drawback of prox-SGD is that the thresholding parameters will decay in the training process, which in unsatisfactory sparsity BID22. Apart from that, the regularized dual averaging (RDA) method BID22, proposed to obtain better sparsity, has been proven to be convergent with specific parameters in convex optimization problem, but has not been applied in deep learning fields. In this paper, we analyze the relation between simple dual averaging (SDA) method BID17 and the stochastic gradient descent (SGD) method BID19, as well as the relation between SDA and RDA. It is well-known that SGD and its variants work quite well in deep learning problems. However, there are few literatures in applying pure training algorithms to deep CNNs for model sparsification. We propose an iterative RDA (iRDA) method for training sparse CNN models, and prove the convergence under convex conditions. Numerically, we compare prox-SGD with iRDA, where the latter can achieve better sparsity while keeping satisfactory accuracy on MNIST, CIFAR-10 and CIFAR-100. We also show iRDA works for different CNN models such as VGG BID21 and BID9. Finally, we compare the performance of iRDA with some other state-of-the-art compression methods. BID0 reviews the work on compressing neural network models, and categorizes the related methods into four schemes: parameter pruning and sharing, low-rank factorization, transfered/compact convolutional filters and knowledge distillation. Among them, BID14 uses sparse decomposition on the convolutional filters to get sparse neural networks, which could be classified to the second scheme. Apart from that, BID7 prunes redundant connections by learning only the important parts. BID15 starts from a Bayesian point of view, and removes large parts of the network through sparsity inducing priors. BID23 BID10 combines reinforcement learning methods to compression. BID13 considers deep learning as a discrete-time optimal control problem, and obtains sparse weights on ternary networks. Recently, BID4 applies RDA to fully-connected neural network models on MNIST. Let z = (x, y) be an input-output pair of data, such as a picture and its corresponding label in a classification problem, and f (w, z) be the loss function of neural networks, i.e. a scalar function that is differentiable w.r.t. weights w. We are interested in the expected risk minimization problem DISPLAYFORM0 The empirical risk minimization DISPLAYFORM1 is an approximation of based on some finite given samples {z 1, z 2, . . ., z T}, where T is the size of the sample set. Regularization is a useful technique in deep learning. In general, the regularized expected risk minimization has the form DISPLAYFORM2 where Ψ(w) is a regularization term with certain effect. For example, Ψ(w) = w 2 2 may improve the generalization ability, and an 1 -norm of w can give sparse solutions. The corresponding regularized empirical risk minimization we concern takes the form DISPLAYFORM3 SDA method is a special case of primal-dual subgradient method first proposed in BID17. BID22 proposes RDA for online convex and stochastic optimization. RDA not only keeps the same convergence rate as Prox-SGD, but also achieves more sparsity in practice. In next sections, we will discuss the connections between SDA and SGD, as well as RDA and Prox-SGD. We then propose iRDA for 1 regularized problem of deep neural networks. As a solution of, SDA takes the form DISPLAYFORM0 The first term t τ =1 g τ, w is a linear function obtained by averaging all previous stochastic gradient. g t is the subgradient of f t. The second term h(w) is a strongly convex function, and {β t} is a nonnegative and nondecreasing sequence which determines the convergence rate. As g τ (w τ), τ = 1,..., t − 1 is constant in current iteration, we use g τ instead for simplicity in the following. Since subproblem equation 5 is strongly convex, it has a unique optimal solution w t+1.Let w 0 be the initial point, and h(w) = 1 2 w − w 0 2 2, the iteration scheme of SDA can be written as DISPLAYFORM1 DISPLAYFORM2. Let β t = γt α, SDA can be rewritten recursively as DISPLAYFORM3 where 1 − 1 − For the regularized problem, we recall the well-known Prox-SGD and RDA method first. At each iteration, Prox-SGD solves the subproblem DISPLAYFORM0 Specifically, α t = 1 γ √ t obtains the best convergence rate. The first two terms are an approximation of the original objective function. Note that without the regularization term Ψ, equation 8 is equivalent to SGD. It can be written in forward-backward splitting (FOBOS) scheme DISPLAYFORM1 DISPLAYFORM2 where the forward step is equivalent to SGD, and the backward step is a soft-thresholding operator working on w t+ DISPLAYFORM3 with the soft-thresholding parameter α t.Different from Prox-SGD, each iteration of RDA takes the form DISPLAYFORM4 Similarly, taking h(w) = 1 2 w − w 0 2 2, RDA can be written as DISPLAYFORM5 = arg min DISPLAYFORM6 or equivalently, DISPLAYFORM7 w t+1 = arg min DISPLAYFORM8 where β t = γ √ t to obtain the best convergence rate. From equation 14, one can see that the forward step is actually SDA and the backward step is the soft-thresholding operator, with the parameter t/β t.3.3 1 REGULARIZATION AND THE SPARSITY Set Ψ(w) = λ w 1. The problem then becomes DISPLAYFORM9 where λ is a hyper-parameter that determines sparsity. In this case, from Xiao's analysis of , the expected cost Eφ(w t) − φ associated with the random variablew t converges with rate O(DISPLAYFORM10 This convergence rate is consistent with . However, both assume f to be a convex function, which can not be guaranteed in deep learning. Nevertheless, we can still verify that RDA is a powerful sparse optimization method for deep neural networks. We conclude the closed form solutions of Prox-SGD and RDA for equation 16 as follows. has the closed form solution DISPLAYFORM0 2. The subproblem of RDA DISPLAYFORM1 has the closed form solution DISPLAYFORM2 3. The √ t-proximal stochastic gradient method has the form DISPLAYFORM3 The difference between √ t-Prox-SGD and Prox-SGD is the soft-thresholding parameter chosen to be √ t. It has the closed form solution DISPLAYFORM4 It is equivalent to DISPLAYFORM5 where the objective function is actually an approximation of DISPLAYFORM6 We can easily conclude that this iteration will converge to w = 0 if DISPLAYFORM7 Now compare the threshold λ P G = α t λ of PG and the threshold λ RDA = t βt λ of RDA. With DISPLAYFORM8 and β t = γ √ t, we have λ P G → 0 and λ RDA → ∞ as t → 0. It is clear that RDA uses a much more aggressive threshold, which guarantees to generate significantly more sparse solutions. Note that when Ψ = λ w 1, RDA requires w 1 = w 0 = 0. However, this will make deep neural network a constant function, with which the parameters can be very hard to update. Thus, in Algorithm 1, we modify the RDA method as Step 1, where w 1 can be chosen not equal to 0, and add an extra Step 2 to improve the performance. We also prove the convergence rate of Step 1 for convex problem is O( DISPLAYFORM0 Theorem 3.1 Assume there exists an optimal solution w to the problem with Ψ(w) = λ w 1 that satisfies h(w) ≤ D 2 for some D > 0, and let φ = φ(w). Let the sequences {w t} t≥1 be generated by Step 1 in iRDA, and assume g t * ≤ G for some constant G. Then the expected cost Eφ(w t) converges to φ with rate O(DISPLAYFORM1 See Appendix A for the proof. To apply iRDA, the weights of a neural network should be initialized differently from that in a normal optimization method such as SGD or its variants. Our initialization is based on BID12, BID5 and BID8, with an additional re-scaling. Let s be a scalar, the mean and the standard deviation of the uniform distribution for iRDA is zero and DISPLAYFORM0 respectively, where c is the number of channels, and k is the spatial filter size of the layer (see BID8).Choosing a suitable s is important when applying iRDA. As shown in TAB2 and TAB3 in Appendix B, if s is too small or too large, the training process could be slowed down and the generalization ability may be affected. Moreover, a small s usually requires much better initial weights, which in too many samplings in initialization process. In our experiments, a good s for iRDA is usually much larger than √ 2, and unsuitable for SGD algorithms. Iterative retraining is a method that only updates the non-zero parameters at each iteration. A trained model can be further updated with retraining, thus both the accuracies and sparsity can be improved. See Table 4 for comparisons on CIFAR-10. The iterative RDA method for 1 regularized DNNs Input:• A strongly convex function h(w) = w 2 2.• A nonnegative and nondescreasing sequence β t = γ √ t. Step 1: RDA with proper initialization Initialize: set w 0 = 0,ḡ 0 = 0 and randomly choose w 1 with methods explained in section 3.5. for t=1,2,..., T do Given the sample z it and corresponding loss function f it.Compute the stochastic gradient g t = ∇f it (w t).Update the average gradient:ḡ DISPLAYFORM0 Compute the next weight vector: DISPLAYFORM1 Step 2: iterative retraining for t=T+1,T+2,T+3,... do Given the sample z it and corresponding loss function f it.Compute the stochastic gradient DISPLAYFORM2 Set (g t) j = 0 if (w t) j = 0 for every j. Update the average gradient:ḡ DISPLAYFORM3 Compute the next weight vector: DISPLAYFORM4 In this section, σ denotes the sparsity of a model, i.e. σ = quantity of zero parameters quantity of all parameters.All neural networks are trained with mini-batch size 128. We provide a test on different hyper-parameters, so as to give an overview of their effects on performance, as shown in TAB4. We also show that the sparsity and the accuracy can be balanced with iRDA by adjusting the parameters λ and γ, as shown in TAB6. Both tables are put in Appendix C. We compare iRDA with several methods including prox-SGD, √ t−SGD and normal SGD, on different datasets including MNIST, CIFAR-10, CIFAR-100 and ImageNet(ILSVRC2012). The main are shown in TAB0. Table 2 shows the performance of iRDA on different architectures including ResNet18, VGG16 and VGG19. TAB1 shows the performance of iRDA on different Figure 1: The first 120 epochs of loss curves corresponding to TAB0, and the sparsity curve for another , where the top-1 validation accuracy is 91.34%, and σ = 0.87. datasets including MNIST, CIFAR-10, CIFAR-100 and ImageNet(ILSVRC2012). In all tables, SGD denotes stochastic gradient methods with momentum. Currently, many compression methods include human experts involvement. Some methods try to combine other structures in training process to automatize the compression process. For example, BID10 combines reinforcement learning. iRDA, as an algorithm, requires no extra structure. As shown above, iRDA can achieve good sparsity while keeping accuracy automatically, with carefully chosen parameters. For CIFAR-10, we compare the performance of iRDA with some other state-of-art compression methods in Table 4. Due to different standards, σ is referred to directly or computed from the original papers approximately. In comparison with many existing rule-based heuristic approaches, the new approach is based on a careful and iterative combination of 1 regularization and some specialized training algorithms. We find that the commonly used training algorithms such as SGD methods are not effective. We thus develop iRDA method that can be used to achieve much better sparsity. iRDA is a variant of RDA methods that have been used for some special types of online convex optimization problems in the literature. New elements in the iRDA mainly consist of judicious initialization and iterative retraining. In addition, iRDA method is carefully analyzed on its convergence for convex objective functions. Many deep neural networks trained by iRDA can achieve good sparsity while keeping the same validation accuracy as those trained by SGD with momentum on many popular datasets. This shows iRDA is a powerful sparse optimization method for image classification problems in deep learning fields. One of the differences between and iRDA is that the former one takes w 1 = arg min w h(w) whereas the latter one chooses w 1 randomly. In the following, we will prove the convergence of iRDA Step 1 for convex problem. The proofs use Lemma 9, Lemma 10, Lemma 11 directly and modify Theorem 1 and Theorem 2 in BID22. For clarity, we have some general assumptions:• The regularization term Ψ(w) is a closed convex function with convexity parameter σ and domΨ is closed.• For each t ≥ 1, f t (w) is convex and subdifferentiable on domΨ.• h(w) is strongly convex on domΨ and subdifferentiable on rint(domΨ) and also satisfies DISPLAYFORM0 Without loss of generality, assume h(w) has convexity parameter 1 and min w h(w) = 0.• There exist a constant G such that DISPLAYFORM1 • Require {β} t≥1 be a nonnegative and nondecreasing sequence and DISPLAYFORM2 Moreover, we could always choose β 1 ≥ σ such that β 0 = β 1.• For a random choosing w 1, we assume DISPLAYFORM3 First of all, we define two functions: DISPLAYFORM4 DISPLAYFORM5 The maximum in is always achieved because F D = {w ∈ domΨ|h(w) ≤ D 2 } is a nonempty compact set. Because of, we have σt+β t ≥ β 0 > 0 for all t ≥ 0, which means tΨ(w)+β t h(w) are all strongly convex, therefore the maximum in is always achieved and unique. As a , we have domU t = domV t = E * for all t ≥ 0. Moreover, by the assumption, both of the functions are nonnegative. Let s t denote the sum of the subgradients obtained up to time t in iRDA Step 1, that is DISPLAYFORM6 and π t (s) denotes the unique maximizer in the definition of V t (s) DISPLAYFORM7 which then gives DISPLAYFORM8 Lemma A.1 For any s ∈ E * and t ≥ 0, we have DISPLAYFORM9 For a proof, see Lemma 9 in.Lemma A.2 The function V t is convex and differentiable. Its gradient is given by DISPLAYFORM10 and the gradient Lipschitz continuous with constant 1/(σt + β t), that is DISPLAYFORM11 Moreover, the following inequality holds: DISPLAYFORM12 The are from Lemma 10 in BID22.Lemma A.3 For each t ≥ 1, we have DISPLAYFORM13 Since h(w t+1) ≥ 0 and the sequence {β t} t≥1 is nondecreasing, we have DISPLAYFORM14 DISPLAYFORM15 To prove this lemma, we refer to the Lemma 11 in. What's more, from the assumption 35, we could always choose β 1 ≥ σ such that β 1 = β 0 and DISPLAYFORM16 The learner's regret of online learning is the difference between his cumulative loss and the cumulative loss of the optimal fixed hypothesis, which is defined by DISPLAYFORM17 and bounded by DISPLAYFORM18 Lemma A.4 Let the sequence {w t} t≥1 and {g t} t≥1 be generated by iRDA Step 1, and assume FORMULA2 and FORMULA2 hold. Then for any t ≥ 1 and any DISPLAYFORM19 Proof First, we define the following gap sequence which measures the quality of the solutions w 1,.., w t: DISPLAYFORM20 and δ t is an upper bound on the regret R t (w) for all w ∈ F D, to see this, we use the convexity of f t (w) in the following: DISPLAYFORM21 Then, We are going to derive an upper bound on δ t. For this purpose, we subtract t τ =1 g τ, w 0 in, which leads to DISPLAYFORM22 the maximization term in is in fact U t (−s t), therefore, by applying Lemma A.1, we have DISPLAYFORM23 Next, we show that ∆ t is an upper bound for the right-hand side of inequality. We consider τ ≥ 2 and τ = 1 respectively. For any τ ≥ 2, we have DISPLAYFORM24 where FORMULA3, FORMULA2, FORMULA3 and FORMULA2 are used. Therefore, we have DISPLAYFORM25, ∀τ ≥ 2.For τ = 1, we have a similar inequality by using DISPLAYFORM26 Summing the above inequalities for τ = 1,..., t and noting that V 0 (−s 0) = V 0 = 0, we arrive at DISPLAYFORM27 Since Ψ(w t+1) ≥ 0, we subtract it from the left hand side and add Ψ(w 1) to both sides of the above inequality yields DISPLAYFORM28 Combing FORMULA3, FORMULA4, and using assumption andwe conclude DISPLAYFORM29 Lemma A.5 Assume there exists an optimal solution w to the problem that satisfies h(w) ≤ D 2 for some D > 0, and let φ = φ(w). Let the sequences {w t} t≥1 be generated by iRDA Step 1, and assume g t * ≤ G for some constant G. Then for any t ≥ 1, the expected cost associated with the random variablew t is bounded as DISPLAYFORM30 Proof First, from the definition, we have the regret at w DISPLAYFORM31 Let z[t] denote the collection of i.i.d. random variables (z, ..., z t). We note that the random variable w τ, where 1 ≤ w ≥ t, is a function of (z 1, ..., z τ −1) and is independent of (z τ, ..., z t). Therefore DISPLAYFORM32 and DISPLAYFORM33 Since φ = φ(w) = min w φ(w), we have the expected regret DISPLAYFORM34 Then, by convexity of φ, we have DISPLAYFORM35 Finally, from FORMULA4 and FORMULA4, we have DISPLAYFORM36 Then the desired follows from that of Lemma A.4. Proof of Theorem 3.1 From Lemma A.5, the expected cost associated with the random variablew t is bounded as DISPLAYFORM37 Here, we consider 1 regularization function Ψ(w) = λ w 1 and it is a convex but not strongly convex function, which means σ = 0. Now, we consider how to choose β t for t ≥ 1 and β 0 = β 1. First if β t = γt, we have 1 t · γtD 2 = γD 2, which means the expected cost does not converge. Then assume β t = γt α, α > 0 and α = 1, the right hand side of the inequality becomes DISPLAYFORM38 From above, we see that if 0 < α < 1, the expected cost converges and the optimal convergence rate O(t We have shown why prox-SGD will give poor sparsity, and although √ t-prox-SGD may introduce greater sparsity, it is not convergent. Finally, iRDA gives the best , on both the top-1 accuracy and the sparsity. iRDA (

represent the tldr for finding the corresponding scientific article

A sparse optimization algorithm for deep CNN models.

Represent the short summary for finding the corresponding scientific article (long)!

General analysis of sign-based methods (e.g. signSGD) for non-convex optimization, built on intuitive bounds on success probabilities.


Question: Represent the scientific article to find a short TLDR

Question: Learning to imitate expert behavior from demonstrations can be challenging, especially in environments with high-dimensional, continuous observations and unknown dynamics. Supervised learning methods based on behavioral cloning (BC) suffer from distribution shift: because the agent greedily imitates demonstrated actions, it can drift away from demonstrated states due to error accumulation. Recent methods based on reinforcement learning (RL), such as inverse RL and generative adversarial imitation learning (GAIL), overcome this issue by training an RL agent to match the demonstrations over a long horizon. Since the true reward function for the task is unknown, these methods learn a reward function from the demonstrations, often using complex and brittle approximation techniques that involve adversarial training. We propose a simple alternative that still uses RL, but does not require learning a reward function. The key idea is to provide the agent with an incentive to match the demonstrations over a long horizon, by encouraging it to return to demonstrated states upon encountering new, out-of-distribution states. We accomplish this by giving the agent a constant reward of r=+1 for matching the demonstrated action in a demonstrated state, and a constant reward of r=0 for all other behavior. Our method, which we call soft Q imitation learning (SQIL), can be implemented with a handful of minor modifications to any standard Q-learning or off-policy actor-critic algorithm. Theoretically, we show that SQIL can be interpreted as a regularized variant of BC that uses a sparsity prior to encourage long-horizon imitation. Empirically, we show that SQIL outperforms BC and achieves competitive compared to GAIL, on a variety of image-based and low-dimensional tasks in Box2D, Atari, and MuJoCo. This paper is a proof of concept that illustrates how a simple imitation method based on RL with constant rewards can be as effective as more complex methods that use learned rewards. Many sequential decision-making problems can be tackled by imitation learning: an expert demonstrates near-optimal behavior to an agent, and the agent attempts to replicate that behavior in novel situations . This paper considers the problem of training an agent to imitate an expert, given expert action demonstrations and the ability to interact with the environment. The agent does not observe a reward signal or query the expert, and does not know the state transition dynamics. Standard approaches based on behavioral cloning (BC) use supervised learning to greedily imitate demonstrated actions, without reasoning about the consequences of actions . As a , compounding errors cause the agent to drift away from the demonstrated states . The problem with BC is that, when the agent drifts and encounters out-of-distribution states, the agent does not know how to return to the demonstrated states. Recent methods based on inverse reinforcement learning (IRL) overcome this issue by training an RL agent not only to imitate demonstrated actions, but also to visit demonstrated states (; ; b;). This is also the core idea behind generative adversarial imitation learning (GAIL) , which implements IRL using generative adversarial networks (; a). Since the true reward function for the task is unknown, these methods construct a reward signal from the demonstrations through adversarial training, making them difficult to implement and use in practice . The main idea in this paper is that the effectiveness of adversarial imitation methods can be achieved by a much simpler approach that does not require adversarial training, or indeed learning a reward function at all. Intuitively, adversarial methods encourage long-horizon imitation by providing the agent with an incentive to imitate the demonstrated actions in demonstrated states, and an incentive to take actions that lead it back to demonstrated states when it encounters new, out-ofdistribution states. One of the reasons why adversarial methods outperform greedy methods, such as BC, is that greedy methods only do, while adversarial methods do both and. Our approach is intended to do both and without adversarial training, by using constant rewards instead of learned rewards. The key idea is that, instead of using a learned reward function to provide a reward signal to the agent, we can simply give the agent a constant reward of r = +1 for matching the demonstrated action in a demonstrated state, and a constant reward of r = 0 for all other behavior. We motivate this approach theoretically, by showing that it implements a regularized variant of BC that learns long-horizon imitation by (a) imposing a sparsity prior on the reward function implied by the imitation policy, and (b) incorporating information about the state transition dynamics into the imitation policy. Intuitively, our method accomplishes (a) by training the agent using an extremely sparse reward function -+1 for demonstrations, 0 everywhere else -and accomplishes (b) by training the agent with RL instead of supervised learning. We instantiate our approach with soft Q-learning by initializing the agent's experience replay buffer with expert demonstrations, setting the rewards to a constant r = +1 in the demonstration experiences, and setting rewards to a constant r = 0 in all of the new experiences the agent collects while interacting with the environment. Since soft Q-learning is an off-policy algorithm, the agent does not necessarily have to visit the demonstrated states in order to experience positive rewards. Instead, the agent replays the demonstrations that were initially added to its buffer. Thus, our method can be applied in environments with stochastic dynamics and continuous states, where the demonstrated states are not necessarily reachable by the agent. We call this method soft Q imitation learning (SQIL). The main contribution of this paper is SQIL: a simple and general imitation learning algorithm that is effective in MDPs with high-dimensional, continuous observations and unknown dynamics. We run experiments in four image-based environments -Car Racing, Pong, Breakout, and Space Invadersand three low-dimensional environments -Humanoid, HalfCheetah, and Lunar Lander -to compare SQIL to two prior methods: BC and GAIL. We find that SQIL outperforms BC and achieves competitive compared to GAIL. Our experiments illustrate two key benefits of SQIL: that it can overcome the state distribution shift problem of BC without adversarial training or learning a reward function, which makes it easier to use, e.g., with images, and that it is simple to implement using existing Q-learning or off-policy actor-critic algorithms. SQIL performs soft Q-learning with three small, but important, modifications: it initially fills the agent's experience replay buffer with demonstrations, where the rewards are set to a constant r = +1; as the agent interacts with the world and accumulates new experiences, it adds them to the replay buffer, and sets the rewards for these new experiences to a constant r = 0; and it balances the number of demonstration experiences and new experiences (50% each) in each sample from the replay buffer. 1 These three modifications are motivated theoretically in Section 3, via an equivalence to a regularized variant of BC. Intuitively, these modifications create a simple reward structure that gives the agent an incentive to imitate the expert in demonstrated states, and to take actions that lead it back to demonstrated states when it strays from the demonstrations. Algorithm 1 Soft Q Imitation Learning (SQIL) 1: Require λsamp ∈ R ≥0, Ddemo 2: Initialize Dsamp ← ∅ 3: while Q θ not converged do 4: Sample transition (s, a, s) with imitation policy π(a|s) ∝ exp (Q θ (s, a)) 6: Dsamp ← Dsamp ∪ {(s, a, s)} 7: end while Crucially, since soft Q-learning is an off-policy algorithm, the agent does not necessarily have to visit the demonstrated states in order to experience positive rewards. Instead, the agent replays the demonstrations that were initially added to its buffer. Thus, SQIL can be used in stochastic environments with high-dimensional, continuous states, where the demonstrated states may never actually be encountered by the agent. SQIL is summarized in Algorithm 1, where Q θ is the soft Q function, D demo are demonstrations, δ 2 is the squared soft Bellman error, and r ∈ {0, 1} is a constant reward. 2 The experiments in Section 4 use a convolutional neural network or multi-layer perceptron to model Q θ, where θ are the weights of the neural network. Section A.3 in the appendix contains additional implementation details, including values for the hyperparameter λ samp; note that the simple default value of λ samp = 1 works well across a variety of environments. As the imitation policy in line 5 of Algorithm 1 learns to behave more like the expert, a growing number of expert-like transitions get added to the buffer D samp with an assigned reward of zero. This causes the effective reward for mimicking the expert to decay over time. Balancing the number of demonstration experiences and new experiences (50% each) sampled for the gradient step in line 4 ensures that this effective reward remains at least 1/(1 + λ samp), instead of decaying to zero. In practice, we find that this reward decay does not degrade performance if SQIL is halted once the squared soft Bellman error loss converges to a minimum (e.g., see Figure 8 in the appendix). Note that prior methods also require similar techniques: both GAIL and adversarial IRL (AIRL) balance the number of positive and negative examples in the training set of the discriminator, and AIRL tends to require early stopping to avoid overfitting. To understand why SQIL works, we sketch a surprising theoretical : SQIL is equivalent to a variant of behavioral cloning (BC) that uses regularization to overcome state distribution shift. BC is a simple approach that seeks to imitate the expert's actions using supervised learning -in particular, greedily maximizing the conditional likelihood of the demonstrated actions given the demonstrated states, without reasoning about the consequences of actions. Thus, when the agent makes small mistakes and enters states that are slightly different from those in the demonstrations, the distribution mismatch between the states in the demonstrations and those actually encountered by the agent leads to compounding errors . We show that SQIL is equivalent to augmenting BC with a regularization term that incorporates information about the state transition dynamics into the imitation policy, and thus enables long-horizon imitation. Maximum entropy model of expert behavior. SQIL is built on soft Q-learning, which assumes that expert behavior follows the maximum entropy model . In an infinite-horizon Markov Decision Process (MDP) with a continuous state space S and discrete action space A, 3 the expert is assumed to follow a policy π that maximizes reward R(s, a). The policy π forms a Boltzmann distribution over actions, where Q is the soft Q function. The soft Q values are a function of the rewards and dynamics, given by the soft Bellman equation, In our imitation setting, the rewards and dynamics are unknown. The expert generates a fixed set of demonstrations D demo, by rolling out their policy π in the environment and generating state transitions (s, a, s) ∈ D demo. Training an imitation policy with standard BC corresponds to fitting a parametric model π θ that minimizes the negative log-likelihood loss, In our setting, instead of explicitly modeling the policy π θ, we can represent the policy π in terms of a soft Q function Q θ via Equation 2: Using this representation of the policy, we can train Q θ via the maximum-likelihood objective in Equation 4: However, optimizing the BC loss in Equation 6 does not in general yield a valid soft Q function Q θ -i.e., a soft Q function that satisfies the soft Bellman equation (Equation 3) with respect to the dynamics and some reward function. The problem is that the BC loss does not incorporate any information about the dynamics into the learning objective, so Q θ learns to greedily assign high values to demonstrated actions, without considering the state transitions that occur as a consequence of actions. As a , Q θ may output arbitrary values in states that are off-distribution from the demonstrations D demo. In Section 3.2, we describe a regularized BC algorithm that adds constraints to ensure that Q θ is a valid soft Q function with respect to some implicitly-represented reward function, and further regularizes the implicit rewards with a sparsity prior. In Section 3.3, we show that this approach recovers an algorithm similar to SQIL. Under the maximum entropy model described in Section 3.1, expert behavior is driven by a reward function, a soft Q function that computes expected future returns, and a policy that takes actions with high soft Q values. In the previous section, we used these assumptions to represent the imitation policy in terms of a model of the soft Q function Q θ (Equation 5). In this section, we represent the reward function implicitly in terms of Q θ, as shown in Equation 7. This allows us to derive SQIL as a variant of BC that imposes a sparsity prior on the implicitly-represented rewards. Sparsity regularization. The issue with BC is that, when the agent encounters states that are outof-distribution with respect to D demo, Q θ may output arbitrary values. One solution from prior work is to regularize Q θ with a sparsity prior on the implied rewards -in particular, a penalty on the magnitude of the rewards s∈S,a∈A |R q (s, a)| implied by Q θ via the soft Bellman equation (Equation 3), where Note that the reward function R q is not explicitly modeled in this method. Instead, we directly minimize the magnitude of the right-hand side of Equation 7, which is equivalent to minimizing |R q (s, a)|. The purpose of the penalty on |R q (s, a)| is two-fold: it imposes a sparsity prior motivated by prior work , and it incorporates information about the state transition dynamics into the imitation learning objective, since R q (s, a) is a function of an expectation over next state s. is critical for learning long-horizon behavior that imitates the demonstrations, instead of greedy maximization of the action likelihoods in standard BC. For details, see. Approximations for continuous states. Unlike the discrete environments tested in , we assume the continuous state space S cannot be enumerated. Hence, we approximate the penalty s∈S,a∈A |R q (s, a)| by estimating it from samples: transitions (s, a, s) observed in the demonstrations D demo, as well as additional rollouts D samp periodically sampled during training using the latest imitation policy. This approximation, which follows the standard approach to constraint sampling , ensures that the penalty covers the state distribution actually encountered by the agent, instead of only the demonstrations. To make the penalty continuously differentiable, we introduce an additional approximation: instead of penalizing the absolute value |R q (s, a)|, we penalize the squared value (R q (s, a)) 2. Note that since the reward function R q is not explicitly modeled, but instead defined via Q θ in Equation 7, the squared penalty (R q (s, a)) 2 is equivalent to the squared soft Bellman error Regularized BC algorithm. Formally, we define the regularized BC loss function adapted from as where λ ∈ R ≥0 is a constant hyperparameter, and δ 2 denotes the squared soft Bellman error defined in Equation 1. The BC loss encourages Q θ to output high values for demonstrated actions at demonstrated states, and the penalty term propagates those high values to nearby states. In other words, Q θ outputs high values for actions that lead to states from which the demonstrated states are reachable. Hence, when the agent finds itself far from the demonstrated states, it takes actions that lead it back to the demonstrated states. The RBC algorithm follows the same procedure as Algorithm 1, except that in line 4, RBC takes a gradient step on the RBC loss from Equation 8 instead of the SQIL loss. The gradient of the RBC loss in Equation 8 is proportional to the gradient of the SQIL loss in line 4 of Algorithm 1, plus an additional term that penalizes the soft value of the initial state s 0 (full derivation in Section A.1 of the appendix): In other words, SQIL solves a similar optimization problem to RBC. The reward function in SQIL also has a clear connection to the sparsity prior in RBC: SQIL imposes the sparsity prior from RBC, by training the agent with an extremely sparse reward function -r = +1 at the demonstrations, and r = 0 everywhere else. Thus, SQIL can be motivated as a practical way to implement the ideas for regularizing BC proposed in. The main benefit of using SQIL instead of RBC is that SQIL is trivial to implement, since it only requires a few small changes to any standard Q-learning implementation (see Section 2). Extending SQIL to MDPs with a continuous action space is also easy, since we can simply replace Q-learning with an off-policy actor-critic method (see Section 4.3). Given the difficulty of implementing deep RL algorithms correctly , this flexibility makes SQIL more practical to use, since it can be built on top of existing implementations of deep RL algorithms. Furthermore, the ablation study in Section 4.4 suggests that SQIL actually performs better than RBC. Our experiments aim to compare SQIL to existing imitation learning methods on a variety of tasks with high-dimensional, continuous observations, such as images, and unknown dynamics. We benchmark SQIL against BC and GAIL 4 on four image-based games -Car Racing, Pong, Breakout, and Space Invaders -and three state-based tasks -Humanoid, HalfCheetah, and Lunar Lander (; ;). We also investigate which components of SQIL contribute most to its performance via an ablation study on the Lunar Lander game. Section A.3 in the appendix contains additional experimental details. The goal of this experiment is to study not only how well each method can mimic the expert demonstrations, but also how well they can acquire policies that generalize to new states that are not seen in the demonstrations. To do so, we train the imitation agents in an environment with a different initial state distribution S train 0 than that of the expert demonstrations S demo 0, allowing us to systematically control the mismatch between the distribution of states in the demonstrations and the states actually encountered by the agent. We run experiments on the Car Racing game from OpenAI Gym. To create S train 0, the car is rotated 90 degrees so that it begins perpendicular to the track, instead of parallel to the track as in S demo 0. This intervention presents a significant generalization challenge to the imitation learner, since the expert demonstrations do not contain any examples of states where the car is perpendicular to the road, or even significantly off the road axis. The agent must learn to make a tight turn to get back on the road, then stabilize its orientation so that it is parallel to the road, and only then proceed forward to mimic the expert demonstrations. The in Figure 1 show that SQIL and BC perform equally well when there is no variation in the initial state. The task is easy enough that even BC achieves a high reward. Note that, in the unperturbed condition (right column), BC substantially outperforms GAIL, despite the wellknown shortcomings of BC. This indicates that the adversarial optimization in GAIL can substantially hinder learning, even in settings where standard BC is sufficient. SQIL performs much better than BC when starting from S train 0, showing that SQIL is capable of generalizing to a new initial state distribution, while BC is not. SQIL learns to make a tight turn that takes the car through the grass and back onto the road, then stabilizes the car's orientation so that it is parallel to the track, and then proceeds forward like the expert does in the demonstrations. BC tends to drive straight ahead into the grass instead of turning back onto the road. 4 For all the image-based tasks, we implement a version of GAIL that uses deep Q-learning (GAIL-DQL) instead of TRPO as in the original GAIL paper , since Q-learning performs better than TRPO in these environments, and because this allows for a head-to-head comparison of SQIL and GAIL: both algorithms use the same underlying RL algorithm, but provide the agent with different rewards -SQIL provides constant rewards, while GAIL provides learned rewards. We use the standard GAIL-TRPO method as a baseline for all the low-dimensional tasks, since TRPO performs better than Q-learning in these environments. The original GAIL method implicitly encodes prior knowledge -namely, that terminating an episode is either always desirable or always undesirable. As pointed out in , this makes comparisons to alternative methods unfair. We implement the unbiased version of GAIL proposed by , and use this in all of the experiments. Comparisons to the biased version with implicit termination knowledge are included in Section A.2 in the appendix. SQIL outperforms GAIL in both conditions. Since SQIL and GAIL both use deep Q-learning for RL in this experiment, the gap between them may be attributed to the difference in the reward functions they use to train the agent. SQIL benefits from providing a constant reward that does not require fitting a discriminator, while GAIL struggles to train a discriminator to provide learned rewards directly from images. The in Figure 2 show that SQIL outperforms BC on Pong, Breakout, and Space Invaders -additional evidence that BC suffers from compounding errors, while SQIL does not. SQIL also outperforms GAIL on all three games, illustrating the difficulty of using GAIL to train an imagebased discriminator, as in Section 4.1. The experiments in the previous sections evaluate SQIL on MDPs with a discrete action space. This section illustrates how SQIL can be adapted to continuous actions. We instantiate SQIL using soft actor-critic (SAC) -an off-policy RL algorithm that can solve continuous control tasks . In particular, SAC is modified in the following ways: the agent's experience replay buffer is initially filled with expert demonstrations, where rewards are set to r = +1, when taking gradient steps to fit the agent's soft Q function, a balanced number of demonstration experiences and new experiences (50% each) are sampled from the replay buffer, and the agent observes rewards of r = 0 during its interactions with the environment, instead of an extrinsic reward signal that specifies the desired task. This instantiation of SQIL is compared to GAIL on the Humanoid (17 DoF) and HalfCheetah (6 DoF) tasks from MuJoCo. The show that SQIL outperforms BC and performs comparably to GAIL on both tasks, demonstrating that SQIL can be successfully deployed on problems with continuous actions, and that SQIL can perform well even with a small number of demonstrations. This experiment also illustrates how SQIL can be run on top of SAC or any other off-policy value-based RL algorithm. We hypothesize that SQIL works well because it combines information about the expert's policy from demonstrations with information about the environment dynamics from rollouts of the imitation policy periodically sampled during training. We also expect RBC to perform comparably to SQIL, since their objectives are similar. To test these hypotheses, we conduct an ablation study using the Lunar Lander game from OpenAI Gym. As in Section 4.1, we control the mismatch between the distribution of states in the demonstrations and the states encountered by the agent by manipulating the initial state distribution. To create S train 0, the agent is placed in a starting position never visited in the demonstrations. In the first variant of SQIL, λ samp is set to zero, to prevent SQIL from using additional samples drawn from the environment (see line 4 of Algorithm 1). This comparison tests if SQIL really needs to interact with the environment, or if it can rely solely on the demonstrations. In the second condition, γ is set to zero to prevent SQIL from accessing information about state transitions (see Equation 1 and line 4 of Algorithm 1). This comparison tests if SQIL is actually extracting information about the dynamics from the samples, or if it can perform just as well with a naïve regularizer (setting γ to zero effectively imposes a penalty on the L2-norm of the soft Q values instead of the squared soft Bellman error). In the third condition, a uniform random policy is used to sample additional rollouts, instead of the imitation policy π θ (see line 6 of Algorithm 1). This comparison tests how important it is that the samples cover the states encountered by the agent during training. In the fourth condition, we use RBC to optimize the loss in Equation 8 The in Figure 4 show that all methods perform well when there is no variation in the initial state. When the initial state is varied, SQIL performs significantly better than BC, GAIL, and the ablated variants of SQIL. This confirms our hypothesis that SQIL needs to sample from the environment using the imitation policy, and relies on information about the dynamics encoded in the samples. Surprisingly, SQIL outperforms RBC by a large margin, suggesting that the penalty on the soft value of the initial state V (s 0), which is present in RBC but not in SQIL (see Equation 9), degrades performance. Related work. Concurrently with SQIL, two other imitation learning algorithms that use constant rewards instead of a learned reward function were developed . We see our paper as contributing additional evidence to support this core idea, rather than proposing a competing method. First, SQIL is derived from sparsity-regularized BC, while the prior methods are derived from an alternative formulation of the IRL objective and from support estimation methods , showing that different theoretical approaches independently lead to using RL with constant rewards as an alternative to adversarial training -a sign that this idea may be a promising direction for future work. Second, SQIL is shown to outperform BC and GAIL in domains that were not evaluated in or -in particular, tasks with image observations and significant shift in the state distribution between the demonstrations and the training environment. Summary. We contribute the SQIL algorithm: a general method for learning to imitate an expert given action demonstrations and access to the environment. Simulation experiments on tasks with high-dimensional, continuous observations and unknown dynamics show that our method outperforms BC and achieves competitive compared to GAIL, while being simple to implement on top of existing off-policy RL algorithms. Limitations and future work. We have not yet proven that SQIL matches the expert's state occupancy measure in the limit of infinite demonstrations. One direction for future work would be to rigorously show whether or not SQIL has this property. Another direction would be to extend SQIL to recover not just the expert's policy, but also their reward function; e.g., by using a parameterized reward function to model rewards in the soft Bellman error terms, instead of using constant rewards. This could provide a simpler alternative to existing adversarial IRL algorithms. (s, a) ). Splitting up the squared soft Bellman error terms for D demo and D samp in Equation 8, Setting γ 1 turns the inner sum in the first term into a telescoping sum: Since s T is assumed to be absorbing, V (s T) is zero. Thus, In our experiments, we have that all the demonstration rollouts start at the same initial state s 0. Thus, where λ samp ∈ R ≥0 is a constant hyperparameter. As discussed in Section 4, to correct the original GAIL method's biased handling of rewards at absorbing states, we implement the suggested changes to GAIL in Section 4.2 of: adding a transition to an absorbing state and a self-loop at the absorbing state to the end of each rollout sampled from the environment, and adding a binary feature to the observations indicating whether or not a state is absorbing. This enables GAIL to learn a non-zero reward for absorbing states. We refer to the original, biased GAIL method as GAIL-DQL-B and GAIL-TRPO-B, and the unbiased version as GAIL-DQL-U and GAIL-TRPO-U. The mechanism for learning terminal rewards proposed in does not apply to SQIL, since SQIL does not learn a reward function. SQIL implicitly assumes a reward of zero at absorbing states in demonstrations. This is the case in all our experiments, which include some environments where terminating the episode is always undesirable (e.g., walking without falling down) and other environments where success requires terminating the episode (e.g., landing at a target), suggesting that SQIL is not sensitive to the choice of termination reward, and neither significantly benefits nor is significantly harmed by setting the termination reward to zero. Car Racing. The in Figure 5 show that both the biased (GAIL-DQL-B) and unbiased (GAIL-DQL-U) versions of GAIL perform equally poorly. The problem of training an image-based discriminator for this task may be difficult enough that even with an unfair bias toward avoiding crashes that terminate the episode, GAIL-DQL-B does not perform better than GAIL-DQL-U. Figure 6 show that SQIL outperforms both variants of GAIL on Pong and the unbiased version of GAIL (GAIL-DQL-U) on Breakout and Space Invaders, but performs comparably to the biased version of GAIL (GAIL-DQL-B) on Space Invaders and worse than it on Breakout. This may be due to the fact that in Breakout and Space Invaders, the agent has multiple livesfive in Breakout, and three in Space Invaders -and receives a termination signal that the episode has ended after losing each life. Thus, the agent experiences many more episode terminations than in Pong, exacerbating the bias in the way the original GAIL method handles rewards at absorbing states. Our implementation of GAIL-DQL-B in this experiment provides a learned reward of r(s, a) = − log (1 − D(s, a) ), where D is the discriminator (see Section A.3 in the appendix for details). The learned reward is always positive, while the implicit reward at an absorbing state is zero. Thus, the agent is inadvertently encouraged to avoid terminating the episode. For Breakout and Space Invaders, this just happens to be the right incentive, since the objective is to stay alive as long as possible. GAIL-DQL-B outperforms SQIL in Breakout and performs comparably to SQIL in Space Invaders because GAIL-DQL-B is accidentally biased in the right way. Lunar Lander. The in Figure 7 show that when the initial state is varied, SQIL outperforms the unbiased variant of GAIL (GAIL-TRPO-U), but underperforms against the biased version of GAIL (GAIL-TRPO-B). The latter is likely due to the fact that the implementation of GAIL-TRPO-B we used in this experiment provides a learned reward of r(s, a) = log (D (s, a) ), where D is the discriminator (see Section A.3 in the appendix for details). The learned reward is always negative, while the implicit reward at an absorbing state is zero. Thus, the agent is inadvertently encouraged to terminate the episode quickly. For the Lunar Lander game, this just happens to be the right incentive, since the objective is to land on the ground and thereby terminate the episode. As in the Atari experiments, GAIL-TRPO-B performs better than SQIL in this experiment because GAIL-TRPO-B is accidentally biased in the right way. To ensure fair comparisons, the same network architectures were used to evaluate SQIL, GAIL, and BC. For Lunar Lander, we used a network architecture with two fully-connected layers containing 128 hidden units each to represent the Q network in SQIL, the policy and discriminator networks in GAIL, and the policy network in BC. For Car Racing, we used four convolutional layers (following ) and two fully-connected layers containing 256 hidden units each. For Humanoid and HalfCheetah, we used two fully-connected layers containing 256 hidden units each. For Atari, we used the convolutional neural network described in to represent the Q network in SQIL, as well as the Q network and discriminator network in GAIL. To ensure fair comparisons, the same demonstration data were used to train SQIL, GAIL, and BC. For Lunar Lander, we collected 100 demonstration rollouts. For Car Racing, Pong, Breakout, and Space Invaders, we collected 20 demonstration rollouts. Expert demonstrations were generated from scratch for Lunar Lander using DQN , and collected from open-source pretrained policies for Car Racing as well as Humanoid and HalfCheetah . The Humanoid demonstrations were generated by a stochastic expert policy, while the HalfCheetah demonstrations were generated by a deterministic expert policy; both experts were trained using TRPO. 6 We used two open-source implementations of GAIL: for Lunar Lander, and for MuJoCo. We adapted the OpenAI Baselines implementation of GAIL to use soft Q-learning for Car Racing and Atari. Expert demonstrations were generated from scratch for Atari using DQN. For Lunar Lander, we set λ samp = 10 −6. For Car Racing, we set λ samp = 0.01. For all other environments, we set λ samp = 1. SQIL was not pre-trained in any of the experiments. GAIL was pre-trained using BC for HalfCheetah, but was not pre-trained in any other experiments. In standard implementations of soft Q-learning and SAC, the agent's experience replay buffer typically has a fixed size, and once the buffer is full, old experiences are deleted to make room for new experiences. In SQIL, we never delete demonstration experiences from the replay buffer, but otherwise follow the standard implementation. We use Adam to take the gradient step in line 4 of Algorithm 1. The BC and GAIL performance metrics in Section 4.3 are taken from . 7 The GAIL and SQIL policies in Section 4.3 are set to be deterministic during the evaluation rollouts used to measure performance.

Represent the short summary for finding the corresponding scientific article (long)

A simple and effective alternative to adversarial imitation learning: initialize experience replay buffer with demonstrations, set their reward to +1, set reward for all other data to 0, run Q-learning or soft actor-critic to train.

Parameter space noise allows reinforcement learning algorithms to explore by perturbing parameters instead of actions, often leading to significantly improved exploration performance.


Question: Represent the scientific article to find a short TLDR

Question: Generating visualizations and interpretations from high-dimensional data is a common problem in many fields. Two key approaches for tackling this problem are clustering and representation learning. There are very performant deep clustering models on the one hand and interpretable representation learning techniques, often relying on latent topological structures such as self-organizing maps, on the other hand. However, current methods do not yet successfully combine these two approaches. We present a new deep architecture for probabilistic clustering, VarPSOM, and its extension to time series data, VarTPSOM, composed of VarPSOM modules connected by LSTM cells. We show that they achieve superior clustering performance compared to current deep clustering methods on static MNIST/Fashion-MNIST data as well as medical time series, while inducing an interpretable representation. Moreover, on the medical time series, VarTPSOM successfully predicts future trajectories in the original data space. Given a set of data samples {x i} i=1,...,n, where x i ∈ R d, the goal is to partition the data into a set 102 of clusters {S i} i=1,...,K, while retaining a topological structure over the cluster centroids. The proposed architecture for static data is presented in Figure 1a. The input vector x i is embedded 104 into a latent representation z i using a VAE. This latent vector is then clustered using PSOM, a 112 DECODER ENCODER (a) VarPSOM architecture for clustering of static data. Data points xi are mapped to a continuous embedding zi using a VAE (parameterized by Φ). The loss function is the sum of a SOMbased clustering loss and the ELBO. (b) VarTPSOM architecture, composed of VarPSOM modules connected by LSTMs across the time axis, which predict the continuous embedding zt+1 of the next time step. This architecture allows to unroll future trajectories in the latent space as well as the original data space by reconstructing the xt using the VAE. A Self-Organizing Map is comprised of K nodes connected to form a grid M ⊆ N 2, where the 114 node m i,j, at position (i, j) of the grid, corresponds to a centroid vector, µ i,j in the input space. The centroids are tied by a neighborhood relation N (µ i,j) = {µ i−1,j, µ i+1,j, µ i,j−1, µ i,j+1}. Given a 116 random initialization of the centroids, the SOM algorithm randomly selects an input x i and updates 117 both its closest centroid µ i,j and its neighbors N (µ i,j) to move them closer to x i. For a complete 118 description of the SOM algorithm, we refer to the appendix (A). The Clustering Assignment Hardening method has been recently introduced by the DEC model and was shown to perform well in the latent space of AEs . Given an embedding function z i = f (x i), it uses a Student's t-distribution (S) as a kernel to measure the similarity between an embedded data point z i, and a centroid µ j: It improves the cluster purity by enforcing the distribution S to approach a target distribution, T: By taking the original distribution to the power of γ and normalizing it, the target distribution puts more emphasis on data points that are assigned a high confidence. We follow in choosing γ=2, which leads to larger gradient contributions of points close to cluster centers, as they show empirically. The ing clustering loss is defined as: Our proposed clustering method, called PSOM, expands Clustering Assignment Hardening to include a SOM neighborhood structure over the centroids. We add an additional loss to to achieve an interpretable representation. This loss term maximizes the similarity between each data point and the neighbors of the closest centroids. For each embedded data point z i and each centroid µ j the loss is defined as the negative sum of all the neighbors of µ j, {e : µ e ∈ N (µ j (x i))}, of the probability that z i is assigned to e, defined as s ie. This sum is weighted by the similarity s ij between z i and the centroid µ j: The complete PSOM clustering loss is then: L PSOM = KL(T S) + βL SOM. We note that for β = 0 it becomes equivalent to Clustering Assignment Hardening. In our method, the nonlinear mapping between the input x i and embedding z i is realized by a VAE. Instead of directly embedding the input x i into a latent embedding z i, the VAE learns a probability distribution q φ (z | x i) parametrized as a multivariate normal distribution whose mean and variance are (µ φ, Σ φ) = f φ (x i). Similarly, it also learns the probability distribution of the reconstructed output given a sampled latent embedding, p θ (x i | z) where (µ θ, Σ θ) = f θ (z i). Both f φ and f θ are neural networks, respectively called encoder and decoder. The ELBO loss is: where p(z) is an isotropic Gaussian prior over the latent embeddings. The second term can be interpreted as a form of regularization, which encourages the latent space to be compact. For each data point x i the latent embedding z i is sampled from q φ (z | x i). Adding the ELBO loss to the PSOM loss from the previous subsection, we get the overall loss function of VarPSOM: To the best of our knowledge, no previous SOM methods attempted to use a VAE to embed the 123 inputs into a latent space. There are many advantages of a VAE over an AE for realizing our goals. Its prior on the latent space encourages structured and disentangled factors To extend our proposed model to time series data, we add a temporal component to the architecture. Given a set of N time series of length T, {x t,i} t=1,...,T;i=1,...,N, the goal is to learn interpretable trajectories on the SOM grid. To do so, the VarPSOM could be used directly but it would treat each time step t of the time series independently, which is undesirable. To exploit temporal information and enforce smoothness in the trajectories, we add an additional loss to: where u it,it+1 = g(z i,t, z i,t+1) is the similarity between z i,t and z i,t+1 using a Student's t- between time points are discouraged. One of the main goals in time series modeling is to predict future data points, or alternatively, future embeddings. This can be achieved by adding a long short-term memory network (LSTM) across the latent embeddings of the time series, as shown in Fig 1b. Each cell of the LSTM takes as input the latent embedding z t at time step t, and predicts a probability distribution over the next latent embedding, p ω (z t+1 | z t). We parametrize this distribution as a Multivariate Normal Distribution whose mean and variance are learnt by the LSTM. The prediction loss is the log-likelihood between the learned distribution and a sample of the next embedding z t+1: The final loss of VarTPSOM, which is trainable in a fully end-to-end fashion, is configurations we refer to the appendix, (B.3). Implementation In implementing our models we focused on retaining a fair comparison with the 158 baselines. Hence we decided to use a standard network structure, with fully connected layers of 159 dimensions d − 500 − 500 − 2000 − l, to implement both the VAE of our models and the AE of the 160 baselines. The latent dimension, l, is set to 100 for the VAE, and to 10 for the AEs. Since the prior 161 in the VAE enforces the latent embeddings to be compact, it also requires more dimensions to learn 162 performance. We suspect this is due to the regularization effect of the SOM's topological structure. Overall, VarPSOM outperforms both DEC and IDEC. Improvement over Training After obtaining the initial configuration of the SOM structure, both 187 clustering and feature extraction using the VAE are trained jointly. To illustrate that our architecture 188 improves clustering performance over the initial configuration, we plotted NMI and Purity against 189 the number of training iterations in Figure 2. We observe that the performance is stable when 190 increasing the number of epochs and no overfitting is visible. as this is the only method among the baselines that is suited for temporal data. We presented two novel methods for interpretable unsupervised clustering, VarPSOM and VarTP- SOM. Both models make use of a VAE and a novel clustering method, PSOM, that extends the 229 classical SOM algorithm to include a centroid-based probability distribution. Our models achieve are tied by a neighborhood relation, here defined as N (µ i,j) = {µ i−1,j, µ i+1,j, µ i,j−1, µ i,j+1}. Given a random initialization of the centroids, the SOM algorithm randomly selects an input x i and 335 updates both its closest centroid µ i,j and its neighbors N (µ i,j) to move them closer to x i. The 336 algorithm then iterates these steps until convergence. Algorithm 1 Self-Organizing Maps At each time t, present an input x(t) and select the winner, Update the weights of the winner and its neighbours, until the map converges score in the next 6 and 12 hours (APACHE-6/12), and the mortality in the next 24 hours. Only those variables from the APACHE score definition which are recorded in the eICU 360 database were taken into account. Each dataset is divided into training, validation and test sets for both our models and the baselines. We evaluate the DEC model for different latent space dimensions. Table S1 shows that the AE, used 364 in the DEC model, performs better when a lower dimensional latent space is used. Figure S3: Randomly sampled VarTPSOM trajectories, from patients expired at the end of the ICU stay, as well as healthily dispatched patients. Superimposed is a heatmap which displays the cluster enrichment in the current APACHE score, from this model run. We observe that trajectories of dying patients are often in different locations of the map as healthy patients, in particular in those regions enriched for high APACHE scores, which corresponds with clinical intuition. assignments of data points to clusters which in a better ability to quantify uncertainty in the 392 data. For visualizing health states in the ICU, this property is very important. In Fig S4 we plot an 393 example patient trajectory, where 6 different time-steps (in temporal order) of the trajectory were 394 chosen. Our model yields a soft centroid-based probability distribution which evolves with time and 395 which allows estimation of likely discrete health states at a given point in time. For each time-step 396 the distribution of probabilities is plotted using a heat-map, whereas the overall trajectory is plotted 397 using a black line. The circle and cross indicate ICU admission and dispatch, respectively. Figure S4: Probabilities over discrete patient health states for 6 different time-steps of the selected time series.

We present a new deep architecture, VarPSOM, and its extension to time series data, VarTPSOM,  which achieve superior clustering performance compared to current deep clustering methods on static and temporal data.

Represent the short summary for finding the corresponding scientific article (long).

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.


Question: Represent the article for finding a single-sentence TLDR.

Question: Many computer vision applications require solving multiple tasks in real-time. A neural network can be trained to solve multiple tasks simultaneously using'multi-task learning'. This saves computation at inference time as only a single network needs to be evaluated. Unfortunately, this often leads to inferior overall performance as task objectives compete, which consequently poses the question: which tasks should and should not be learned together in one network when employing multi-task learning? We systematically study task cooperation and competition and propose a framework for assigning tasks to a few neural networks such that cooperating tasks are computed by the same neural network, while competing tasks are computed by different networks. Our framework offers a time-accuracy trade-off and can produce better accuracy using less inference time than not only a single large multi-task neural network but also many single-task networks. Many applications, especially robotics and autonomous vehicles, are chiefly interested in using multi-task learning to reduce the inference time and computational complexity required to estimate many characteristics of visual input. For example, an autonomous vehicle may need to detect the location of pedestrians, determine a per-pixel depth, and predict objects' trajectories, all within tens of milliseconds. In multi-task learning, multiple tasks are solved at the same time, typically with a single neural network. In addition to reduced inference time, solving a set of tasks jointly rather than independently can, in theory, have other benefits such as improved prediction accuracy, increased data efficiency, and reduced training time. Unfortunately, the quality of predictions are often observed to suffer when a network is tasked with making multiple predictions. This is because learning objectives can have complex and unknown dynamics and may compete. In fact, multi-task performance can suffer so much that smaller independent networks are often superior (as we will see in the experiments section). We refer to any situation in which the competing priorities of the network cause poor task performance as crosstalk. On the other hand, when task objectives do not interfere much with each other, performance on both tasks can be maintained or even improved when jointly trained. Intuitively, this loss or gain of quality seems to depend on the relationship between the jointly trained tasks. Prior work has studied the relationship between tasks for transfer learning . However, we find that transfer relationships are not highly predictive of multi-task relationships. In addition to studying multi-task relationships, we attempt to determine how to produce good prediction accuracy under a limited inference time budget by assigning competing tasks to separate networks and cooperating tasks to the same network. More concretely, this leads to the following problem: Given a set of tasks, T, and a computational budget b (e.g., maximum allowable inference time), what is the optimal way to assign tasks to networks with combined cost ≤ b such that a combined measure of task performances is maximized? To this end, we develop a computational framework for choosing the best tasks to group together in order to have a small number of separate deep neural networks that completely cover the task set and that maximize task performance under a given computational budget. We make the intriguing Figure 1: Given five tasks to solve, there are many ways that they can be split into task groups for multitask learning. How do we find the best one? We propose a computational framework that, for instance, suggests the following grouping to achieve the lowest total loss, using a computational budget of 2.5 units: train network A to solve Semantic Segmentation, Depth Estimation, and Surface Normal Prediction; train network B to solve Keypoint Detection, Edge Detection, and Surface Normal Prediction; train network C with a less computationally expensive encoder to solve Surface Normal Prediction alone; including Surface Normals as an output in the first two networks were found advantageous for improving the other outputs, while the best Normals were predicted by the third network. This task grouping outperforms all other feasible ones, including learning all five tasks in one large network or using five dedicated smaller networks. observation that the inclusion of an additional task in a network can potentially improve the accuracy of the other tasks, even though the performance of the added task might be poor. This can be viewed as regularizing or guiding the loss of one task by adding an additional loss, as often employed in curriculum learning or network regularization. Achieving this, of course, depends on picking the proper regularizing task -our system can take advantage of this phenomenon, as schematically shown in Figure 1. This paper has two main contributions. In Section 3, we outline a framework for systematically assigning tasks to networks in order to achieve the best total prediction accuracy with a limited inference-time budget. We then analyze the ing accuracy and show that selecting the best assignment of tasks to groups is critical for good performance. Secondly, in Section 6, we analyze situations in which multi-task learning helps and when it doesn't, quantify the compatibilities of various task combinations for multi-task learning, compare them to the transfer learning task affinities, and discuss the implications. Moreover, we analyze the factors that influence multi-task affinities. Multi-Task Learning: for a good overview of multi-task learning. The authors identify two clusters of contemporary techniques that we believe cover the space well, hard parameter sharing and soft parameter sharing. In brief, the primary difference between the majority of the existing works and our study is that we wish to understand the relationships between tasks and find compatible groupings of tasks for any given set of tasks, rather than designing a neural network architecture to solve a particular fixed set of tasks well. A known contemporary example of hard parameter sharing in computer vision is UberNet . The authors tackle 7 computer vision problems using hard parameter sharing. The authors focus on reducing the computational cost of training for hard parameter sharing, but experience a rapid degradation in performance as more tasks are added to the network. Hard parameter sharing is also used in many other works such as (; ; ; ; ; ; ; ; ; ; ; d. ; ;). Other works, such as and (Chen et al. (2018b) ), aim to dynamically reweight each task's loss during training. The former work finds weights that provably lead to a Pareto-optimal solution, while the latter attempts to find weights that balance the influence of each task on network weights. Finally, (Bingel & Søgaard ) studies task interaction for NLP. In soft or partial parameter sharing, either there is a separate set of parameters per task, or a significant fraction of the parameters are unshared. The models are tied together either by information sharing or by requiring parameters to be similar. Examples include (; ; ; ; ;). The canonical example of soft parameter sharing can be seen in . The authors are interested in designing a deep dependency parser for languages such as Irish that do not have much treebank data available. They tie the weights of two networks together by adding an L2 distance penalty between corresponding weights and show substantial improvement. Another example of soft parameter sharing is Cross-stitch Networks . Starting with separate networks for two tasks, the authors add'cross-stitch units' between them, which allow each network to peek at the other network's hidden layers. This approach reduces but does not eliminate task interfearence, and the overall performance is less sensitive to the relative loss weights. Unlike our method, none of the aforementioned works attempt to discover good groups of tasks to train together. Also, soft parameter sharing does not reduce inference time, a major goal of ours. Transfer Learning: Transfer learning is similar to multi-task learning in that solutions are learned for multiple tasks. Unlike multi-task learning, however, transfer learning methods often assume that a model for a source task is given and then adapt that model to a target task. Transfer learning methods generally neither seek any benefit for source tasks nor a reduction in inference time as their main objective. Neural Architecture Search (NAS): Many recent works search the space of deep learning architectures to find ones that perform well (; ; ; ; ; ;). This is related to our work as we search the space of task groupings. Just as with NAS, the found task groupings often perform better than human-engineered ones. Task Relationships: Our work is most related to Taskonomy , where the authors studied the relationships between visual tasks for transfer learning and introduced a dataset with over 4 million images and corresponding labels for 26 tasks. This was followed by a number of recent works, which further analyzed task relationships (; Dwivedi & Roig.;; ) for transfer learning. While they extract relationships between these tasks for transfer learning, we are interested in the multi-task learning setting. Interestingly, we find notable differences between transfer task affinity and multi-task affinity. Their method also differs in that they are interested in labeled-data efficiency and not inference-time efficiency. Finally, the transfer quantification approach taken by Taskonomy (readout functions) is only capable of finding relationships between the high-level bottleneck representations developed for each task, whereas structural similarities between tasks at all levels are potentially relevant for multi-task learning. Our goal is to find an assignment of tasks to networks that in the best overall loss. Our strategy is to select from a large set of candidate networks to include in our final solution. We define the problem as follows: We want to minimize the overall loss on a set of tasks T = {t 1, t 2, ..., t k} given a limited inference time budget, b, which is the total amount of time we have to complete all tasks. Each neural network that solves some subset of T and that could potentially be a part of the final solution is denoted by n. It has an associated inference time cost, c n, and a loss for each task, L(n, t i) (which is ∞ for each task the network does not attempt to solve). A solution S is a set of networks that together solve all tasks. The computational cost of a solution is cost(S) = n∈S c n. The loss of a solution on a task, L(S, t i), is the lowest loss on that task among the solution's networks 1, L(S, We want to find the solution with the lowest overall loss and a cost that is under our budget, S b = argmin S:cost(S)≤b L(S). For a given task set T, we wish to determine not just how well each pair of tasks performs when trained together, but also how well each combination of tasks performs together so that we can capture higher-order task relationships. To that end, our candidate set of networks contains all 2 |T | − 1 possible groupings: |T | 1 networks with one task, |T | 2 networks with two tasks, networks with three tasks, etc. For the five tasks we use in our experiments, this is 31 networks, of which five are single-task networks. The size of the networks is another design choice, and to somewhat explore its effects we also include 5 single task networks each with half of the computational cost of a standard network. This brings our total up to 36 networks. Consider the situation in which we have an initial candidate set C 0 = {n 1, n 2, ..., n m} of fullytrained networks that each solve some subset of our task set T. Our goal is to choose a subset of C 0 that solve all the tasks with total inference time under budget b and the lowest overall loss. More formally, we want to find a solution It can be shown that solving this problem is NP-hard in general (reduction from SET-COVER). However, many techniques exist that can optimally solve most reasonably-sized instances of problems like these in acceptable amounts of time. All of these techniques produce the same solutions. We chose to use a branch-and-bound-like algorithm for finding our optimal solutions (shown as Algorithm 1 in the Appendix), but in principle the exact same solutions could be achieved by other optimization methods, such as encoding the problem as a binary integer program (BIP) and solving it in a way similar to Taskonomy . Most contemporary MTL works use fewer than 4 unique task types, but in principal, the NP-hard nature of the optimization problem does limit the number of candidate solutions that can be considered. However, using synthetic inputs, we found that our branch-and-bound like approach requires less time than network training for all 2 |T | − 1 + |T | candidates for fewer than ten tasks. Scaling beyond that would require approximations or stronger optimization techniques. This section describes two techniques for reducing the training time required to obtain a collection of networks as input to the network selection algorithm. Our goal is to produce task groupings with similar to the ones produced by the complete search, but with less training time burden. Both techniques involve predicting the performance of a network without actually training it to convergence. The first technique involves training each of the networks for a short amount of time, and the second involves inferring how networks trained on more than two tasks will perform based on how networks trained on two tasks perform. We found a moderately high correlation (Pearson's r = 0.49) between the validation loss of our neural networks after a pass through just 20% of our data and the final test loss of the fully trained networks. This implies that the task relationship trends stabilize early. We fine that we can get decent by running network selection on the lightly trained networks, and then simply training the chosen networks to convergence. For our setup, this technique reduces the training time burden by about 20x over fully training all candiate networks and would require fewer than 150 GPU hours to execute. This is only 35% training-time overhead. Obviously, this technique does come with a prediction accuracy penalty. Because the correlation between early network performance and final network performance is not perfect, the decisions made by network selection are no longer guaranteed to be optimal once networks are trained to convergence. We call this approximation the Early Stopping Approximation (ESA) and present the of using this technique in Section 5. Do the performances of a network trained with tasks A and B, another trained with tasks A and C, and a third trained with tasks B and C tell us anything about the performance of a network trained on tasks A, B, and C? As it turns out, the answer is yes. Although this ignores complex task interactions and nonlinearities, a simple average of the first-order networks' accuracies was a good indicator of the accuracy of a higher-order network. Experimentally, this prediction strategy has an average max ratio error of only 5.2% on our candidate networks. Using this strategy, we can predict the performance of all networks with three or more tasks using the performance of all of the fully trained two task networks. First, simply train all networks with two or fewer tasks to convergence. Then predict the performance of higher-order networks. Finally, run network selection on both groups. With our setup (see Section 4), this strategy saves training time by only about 50%, compared with 95% for the early stopping approximation, and it still comes with a prediction quality penalty. However, this technique requires only a quadratic number of networks to be trained rather than an exponential number, and would therefore win out when the number of tasks is large. We call this strategy the Higher Order Approximation (HOA), and present its in Section 5. We perform our evaluation using the Taskonomy dataset , which is currently the largest multi-task dataset in vision with diverse tasks. The data was obtained from 3D scans of about 600 buildings. There are 4,076,375 examples, which we divided into 3,974,199 training instances, 52,000 validation instances, and 50,176 test instances. There was no overlap in the buildings that appeared in the training and test sets. All data labels were normalized (x = 0, σ = 1). Our framework is agnostic to the particular set of tasks. We have chosen to perform the study using five tasks in Taskonomy: Semantic Segmentation, Depth Estimation, Surface Normal Prediction, Keypoint Detection, and Edge Detection, so that one semantic task, two 3D tasks, and two 2D tasks are included. These tasks were chosen to be representative of major task categories, but also to have enough overlap in order to test the hypothesis that similar tasks will train well together. Crossentropy loss was used for Semantic Segmentation, while an L1 loss was used for all other tasks. Network Architecture: The proposed framework can work with any network architecture. In our experiments, all of the networks used a standard encoder-decoder architecture with a modified Xception encoder. Our choice of architecture is not critical and was chosen for reasonably fast inference time performance. The Xception network encoder was simplified to have 17 layers and the middle flow layers were reduced to having 512 rather than 728 channels. All maxpooling layers were replaced by 2 × 2 convolution layers with a stride of 2 (similar to Chen et al. (2018a) ). The full-size encoder had about 4 million parameters. All networks had an input image size of 256x256. We measure inference time in units of the time taken to do inference for one of our full-size encoders. We call this a Standard Network Time (SNT). This corresponds to 2.28 billion multiply-adds and about 4 ms/image on a single Nvidia RTX 2080 Ti. Our decoders were designed to be lightweight and have four transposed convolutional layers and four separable convolutional layers . Every decoder has about 116,000 parameters. All training was done using PyTorch with Apex for fp16 acceleration . Trained Networks: As described in Section 3.1, we trained 31 networks with full sized encoders and standard decoders. 26 were multi-task networks and 5 were single task networks. Another five single-task networks were trained, each having a half-size encoder and a standard decoder. These 36 networks were included in network optimization as C 0. 20 smaller, single-task networks of various sizes were also trained to be used in the baselines and the analysis of Section 6, but not used for network selection. In order to produce our smaller models, we shrunk the number of channels in every layer of the encoder such that it had the appropriate number of parameters and flops. The training loss we used was the unweighted mean of the losses for the included tasks. Networks were trained with an initial learning rate of 0.2, which was reduced by half every time the training loss stopped decreasing. Networks were trained until their validation loss stopped improving, typically requiring only 4-8 passes through the dataset. The network with the highest validation loss (checked after each epoch of 20% of our data) was saved. The performance scores used for network selection were calculated on the validation set. We computed solutions for inference time budgets from 1 to 5 at increments of 0.5. Each solution chosen was evaluated on the test set. We compare our with conventional methods, such as five single-task networks and a single network with all tasks trained jointly. We also compare with two multi-task methods in the literature. The first one is. We found that their algorithm under-weighted the Semantic Segmentation task too aggressively, leading to poor performance on the task and poor performance overall compared to a simple sum of task losses. We speculate that this is because semantic segmentation's loss behaves differently from the other losses. Next we compared to GradNorm (Chen et al. (2018b) ). The here were also slightly worse than classical MTL with uniform task weights. In any event, these techniques are orthogonal to ours and can be used in conjunction for situations in which they lead to better solutions than simply summing losses. Finally, we compare our to two control baselines illustrative of the importance of making good choices about which tasks to train together,'Random' and'Pessimal.''Random' is a solution consisting of valid random task groupings that solve our five tasks. The reported values are the average of a thousand random trials.' Pessimal' is a solution in which we choose the networks that lead to the worst overall performance, though the solution's performance on each task is still the best among its networks. Each baseline was evaluated with multiple encoder sizes so that all models' could be compared at many inference time budgets. Figure 2 shows the task groups that were chosen for each technique, and Figure 3 shows the performance of these groups along with those of our baselines. We can see that each of our methods outperforms our traditional baselines for every computational budget. When the computational budget is only 1 SNT, all of our methods must select the same model-a traditional multi-task network with a 1 SNT encoder and five decoders. This strategy outperforms , and individual training. However, solutions that utilize multiple networks outperform this traditional strategy for every budget > 1.5-better performance can always be achieved by grouping tasks according to their compatibility. Table 7. When the computational budget is effectively unlimited (5 SNT), our optimal method picks five networks, each of which is used to make predictions for a separate task. However, three of the networks are trained with three tasks each, while only two are trained with one task each. This shows that the representations learned through multi-task learning were found to be best for three of our tasks (s, d, and e), whereas two of our tasks (n and k) are best solved individually. We also see that our optimal technique using 2.5 SNT and our Higher Order Approximation using 3.5 SNT can both outperform five individual networks (which uses 5 SNT). Total Loss All-in-one (triple-size resnet18) 0.50925 Five Individual (resnet18s .6-size each) 0.53484 nKE, SDn, N (3 standard resnet18's) 0.50658 In order to determine how these task groupings generalize to other architectures, we retrained our best solution for 3 SNT using resnet18 ). The in Table 1 suggest that good task groupings for one architecture are likely to be good in another, though to a lesser extent. Task affinities seem to be somewhat architecture-dependent, so for the very best , task selection must be run for each architecture choice. Figure 4 allows qualitative comparison between our methods and our baselines. We can see clear visual issues with each of our baselines that are not present in our methods. Both of our approximate methods produce predictions similar to the optimal task grouping. The data generated by the above evaluation presents an opportunity to analyze how tasks interact in a multi-task setting, and allows us to compare with some of the vast body of research in transfer learning, such as Taskonomy (Table 4 : The transfer learning affinities between pairs of tasks according to the authors of Taskonomy . Forward and backward transfer affinities are averaged. Transfer Affinity Multi-Task Affinity In order to determine the between task affinity for multi-task learning, we took the average of our first-order relationships matrix (Table 2) and its transpose. The is shown in Table 3. The pair with the highest affinity by this metric are Surface Normal Prediction and 2D Edge Detection. Our two 3D tasks, Depth Estimation and Surface Normal Prediction, do not score highly on this similarity metric. This contrasts with the findings for transfer learning in Taskonomy (Table 4), in which they have the highest affinity. Our two 2D tasks also do not score highly. We speculate that the Normals task naturally preserves edges, while Depth and Normals (for example) don't add much training signal to each other. See Section A.3 for more on factors that influence multi-task affinity. Figure 5 depicts the relationship between transfer learning affinities and multi-task affinities, which surprisingly seem to be negatively correlated in our high-data scenario. This suggests that it might be better to train dissimilar tasks together. This could be because dissimilar tasks are able to provide stronger and more meaningful regularization. More research is necessary to discover when and if this correlation and explanation hold. We describe the problem of task compatibility as it pertains to multi-task learning. We provide an algorithm and computational framework for determining which tasks should be trained jointly and which tasks should be trained separately. Our solution can take advantage of situations in which joint training is beneficial to some tasks but not others in the same group. For many use cases, this framework is sufficient, but it can be costly at training time. Hence, we offer two strategies for coping with this issue and evaluate their performance. Our methods outperform single-task networks, a multi-task network with all tasks trained jointly, as well as other baselines. Finally, we use this opportunity to analyze how particular tasks interact in a multi-task setting and compare that with previous on transfer learning task interactions. A.1 NETWORK SELECTION ALGORITHM Input: C r, a running set of candidate networks, each with an associated cost c ∈ R and a performance score for each task the network solves. Initially, C r = C 0 Input: S r ⊆ C 0, a running solution, initially Ø Input: b r ∈ R, the remaining time budget, initially b Most promising networks first 4: Best ← S r for n ∈ C r do 6: C r ← C r \ n \ is set subtraction. S i ← S r ∪ {n} 8: Child ← GETBESTNETWORKS(C r, S i, b i) Best ← BETTER(Best, Child) return Best 12: function FILTER(C r, S r, b r) Remove networks from C r with c n > b r. 14: Remove networks from C r that cannot improve S r's performance on any task. return S 2 Algorithm 1 chooses the best subset of networks in our collection, subject to the inference time budget constraint. The algorithm recursively explores the space of solutions and prunes branches that cannot lead to optimal solutions. The recursion terminates when the budget is exhausted, at which point C r becomes empty and the loop body does not execute. The sorting step on line 3 requires a heuristic upon which to sort. We found that ranking models based on how much they improve the current solution, S, works well. It should be noted that this algorithm always produces an optimal solution, regardless of which sorting heuristic is used. However, better sorting heuristics reduce the running time because subsequent iterations will more readily detect and prune portions of the search space that cannot contain an optimal solution. In our setup, we tried variants of problems with 5 tasks and 36 networks, and all of them took less than a second to solve. The definition of the BETTER function is application-specific. For our experiments, we prefer networks that have the lowest total loss across all five tasks. Other applications may have hard performance requirements for some of the tasks, and performance on one of these tasks cannot be sacrificed in order to achieve better performance on another task. Such application-specific constraints can be encoded in BETTER. In order to determine how well network selection works for different task sets, we re-ran network selection on all five 4-task subsets of our task set. The performance average of all 5 sets is shown in Figure 6. We see that our techniques generalize at least to subsets of our studied tasks. The finding that Depth and Normals don't cooperate is counter to much of the multitask learning literature such as , , and. However, the majority of these works use training sets with fewer than 100k instances, while we use nearly 4 million training instances. Table 5 shows the loss obtained on our setup when we limit to only 100k training instances. The fact that task affinities can change depending on the amount of available training data demonstrates the necessity of using an empirical approach like ours for finding task affinities and groupings. A. Table 9: The test set performance of our 31 networks on each task that they solve.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We analyze what tasks are best learned together in one network, and which are best to learn separately.

Represent the TLDR for finding the corresponding scientific article

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.