Question: Represent the scientific article to find a short TLDR

Question: Adam is shown not being able to converge to the optimal solution in certain cases. Researchers recently propose several algorithms to avoid the issue of non-convergence of Adam, but their efficiency turns out to be unsatisfactory in practice. In this paper, we provide a new insight into the non-convergence issue of Adam as well as other adaptive learning rate methods. We argue that there exists an inappropriate correlation between gradient $g_t$ and the second moment term $v_t$ in Adam ($t$ is the timestep), which in that a large gradient is likely to have small step size while a small gradient may have a large step size. We demonstrate that such unbalanced step sizes are the fundamental cause of non-convergence of Adam, and we further prove that decorrelating $v_t$ and $g_t$ will lead to unbiased step size for each gradient, thus solving the non-convergence problem of Adam. Finally, we propose AdaShift, a novel adaptive learning rate method that decorrelates $v_t$ and $g_t$ by temporal shifting, i.e., using temporally shifted gradient $g_{t-n}$ to calculate $v_t$. The experiment demonstrate that AdaShift is able to address the non-convergence issue of Adam, while still maintaining a competitive performance with Adam in terms of both training speed and generalization. First-order optimization algorithms with adaptive learning rate play an important role in deep learning due to their efficiency in solving large-scale optimization problems. Denote g t ∈ R n as the gradient of loss function f with respect to its parameters θ ∈ R n at timestep t, then the general updating rule of these algorithms can be written as follows : DISPLAYFORM0 In the above equation, m t φ(g 1, . . ., g t) ∈ R n is a function of the historical gradients; v t ψ(g 1, . . ., g t) ∈ R n + is an n-dimension vector with non-negative elements, which adapts the learning rate for the n elements in g t respectively; α t is the base learning rate; and αt √ vt is the adaptive step size for m t.One common choice of φ(g 1, . . ., g t) is the exponential moving average of the gradients used in Momentum and Adam , which helps alleviate gradient oscillations. The commonly-used ψ(g 1, . . ., g t) in deep learning community is the exponential moving average of squared gradients, such as Adadelta , RMSProp , Adam and Nadam .Adam is a typical adaptive learning rate method, which assembles the idea of using exponential moving average of first and second moments and bias correction. In general, Adam is robust and efficient in both dense and sparse gradient cases, and is popular in deep learning research. However, Adam is shown not being able to converge to optimal solution in certain cases. point out that the key issue in the convergence proof of Adam lies in the quantity DISPLAYFORM1 which is assumed to be positive, but unfortunately, such an assumption does not always hold in Adam. They provide a set of counterexamples and demonstrate that the violation of positiveness of Γ t will lead to undesirable convergence behavior in Adam. then propose two variants, AMSGrad and AdamNC, to address the issue by keeping Γ t positive. Specifically, AMSGrad definesv t as the historical maximum of v t, i.e.,v t = max {v i} t i=1, and replaces v t withv t to keep v t non-decreasing and therefore forces Γ t to be positive; while AdamNC forces v t to have "long-term memory" of past gradients and calculates v t as their average to make it stable. Though these two algorithms solve the non-convergence problem of Adam to a certain extent, they turn out to be inefficient in practice: they have to maintain a very large v t once a large gradient appears, and a large v t decreases the adaptive learning rate αt √ vt and slows down the training process. In this paper, we provide a new insight into adaptive learning rate methods, which brings a new perspective on solving the non-convergence issue of Adam. Specifically, in Section 3, we study the counterexamples provided by via analyzing the accumulated step size of each gradient g t. We observe that in the common adaptive learning rate methods, a large gradient tends to have a relatively small step size, while a small gradient is likely to have a relatively large step size. We show that the unbalanced step sizes stem from the inappropriate positive correlation between v t and g t, and we argue that this is the fundamental cause of the non-convergence issue of Adam. In Section 4, we further prove that decorrelating v t and g t leads to equal and unbiased expected step size for each gradient, thus solving the non-convergence issue of Adam. We subsequently propose AdaShift, a decorrelated variant of adaptive learning rate methods, which achieves decorrelation between v t and g t by calculating v t using temporally shifted gradients. Finally, in Section 5, we study the performance of our proposed AdaShift, and demonstrate that it solves the non-convergence issue of Adam, while still maintaining a decent performance compared with Adam in terms of both training speed and generalization. Adam. In Adam, m t and v t are defined as the exponential moving average of g t and g 2 t: m t = β 1 m t−1 + (1 − β 1)g t and v t = β 2 v t−1 + (1 − β 2)g 2 t,where β 1 ∈ and β 2 ∈ are the exponential decay rates for m t and v t, respectively, with m 0 = 0 and v 0 = 0. They can also be written as: DISPLAYFORM0 To avoid the bias in the estimation of the expected value at the initial timesteps, propose to apply bias correction to m t and v t. Using m t as instance, it works as follows: DISPLAYFORM1 Online optimization problem. An online optimization problem consists of a sequence of cost functions f 1 (θ),..., f t (θ),..., f T (θ), where the optimizer predicts the parameter θ t at each timestep t and evaluate it on an unknown cost function f t (θ). The performance of the optimizer is usually evaluated by regret R(T) DISPLAYFORM2, which is the sum of the difference between the online prediction f t (θ t) and the best fixed-point parameter prediction f t (θ *) for all the previous steps, where θ * = arg min θ∈ϑ T t=1 f t (θ) is the best fixed-point parameter from a feasible set ϑ.Counterexamples. highlight that for any fixed β 1 and β 2, there exists an online optimization problem where Adam has non-zero average regret, i.e., Adam does not converge to optimal solution. The counterexamples in the sequential version are given as follows: DISPLAYFORM3 where C is a relatively large constant and d is the length of an epoch. In Equation 6, most gradients of f t (θ) with respect to θ are −1, but the large positive gradient C at the beginning of each epoch makes the overall gradient of each epoch positive, which means that one should decrease θ t to minimize the loss. However, according to , the accumulated update of θ in Adam under some circumstance is opposite (i.e., θ t is increased), thus Adam cannot converge in such case. argue that the reason of the non-convergence of Adam lies in that the positive assumption of Γ t (√ v t /α t − √ v t−1 /α t−1) does not always hold in Adam. The counterexamples are also extended to stochastic cases in , where a finite set of cost functions appear in a stochastic order. Compared with sequential online optimization counterexample, the stochastic version is more general and closer to the practical situation. For the simplest one dimensional case, at each timestep t, the function f t (θ) is chosen as i.i.d.: DISPLAYFORM4 where δ is a small positive constant that is smaller than C. The expected cost function of the above problem is F (θ) = 1+δ C+1 Cθ − C−δ C+1 θ = δθ, therefore, one should decrease θ to minimize the loss. prove that when C is large enough, the expectation of accumulated parameter update in Adam is positive and in increasing θ. propose maintaining the strict positiveness of Γ t as solution, for example, keeping v t non-decreasing or using increasing β 2. In fact, keeping Γ t positive is not the only way to guarantee the convergence of Adam. Another important observation is that for any fixed sequential online optimization problem with infinitely repeating epochs (e.g., Equation 6), Adam will converge as long as β 1 is large enough. Formally, we have the following theorem: Theorem 1 (The influence of β 1). For any fixed sequential online convex optimization problem with infinitely repeating of finite length epochs (d is the length of an epoch), if ∃G ∈ R such that DISPLAYFORM0 for any fixed β 2 ∈, there exists a β 1 ∈ such that Adam has average regret ≤ 2;The intuition behind Theorem 1 is that, if DISPLAYFORM1 In this section, we study the non-convergence issue by analyzing the counterexamples provided by. We show that the fundamental problem of common adaptive learning rate methods is that: v t is positively correlated to the scale of gradient g t, which in a small step size α t / √ v t for a large gradient, and a large step size for a small gradient. We argue that such an unbalanced step size is the cause of non-convergence. We will first define net update factor for the analysis of the accumulated influence of each gradient g t, then apply the net update factor to study the behaviors of Adam using Equation 6 as an example. The argument will be extended to the stochastic online optimization problem and general cases. When β 1 = 0, due to the exponential moving effect of m t, the influence of g t exists in all of its following timesteps. For timestep i (i ≥ t), the weight of g t is (1 − β 1)β i−t 1. We accordingly define a new tool for our analysis: the net update net(g t) of each gradient g t, which is its accumulated influence on the entire optimization process: DISPLAYFORM0 and we call k(g t) the net update factor of g t, which is the equivalent accumulated step size for gradient g t. Note that k(g t) depends on {v i} It is worth noticing that in Momentum method, v t is equivalently set as 1. Therefore, we have k(g t) = α t and net(g t) = α t g t, which means that the accumulated influence of each gradient g t in Momentum is the same as vanilla SGD (Stochastic Gradient Decent). Hence, the convergence of Momentum is similar to vanilla SGD. However, in adaptive learning rate methods, v t is function over the past gradients, which makes its convergence nontrivial. Note that v t exists in the definition of net update factor (Equation 8). Before further analyzing the convergence of Adam using the net update factor, we first study the pattern of v t in the sequential online optimization problem in Equation 6. Since Equation 6 is deterministic, we can derive the formula of v t as follows:Lemma 2. In the sequential online optimization problem in Equation 6, denote β 1, β 2 ∈ as the decay rates, d ∈ N as the length of an epoch, n ∈ N as the index of epoch, and i ∈ {1, 2, ..., d} as the index of timestep in one epoch. Then the limit of v nd+i when n → ∞ is: DISPLAYFORM0 Given the formula of v t in Equation 9, we now study the net update factor of each gradient. We start with a simple case where β 1 = 0. In this case we have DISPLAYFORM1 Since the limit of v nd+i in each epoch monotonically decreases with the increase of index i according to Equation 9, the limit of k(g nd+i) monotonically increases in each epoch. Specifically, the first gradient g nd+1 = C in epoch n represents the correct updating direction, but its influence is the smallest in this epoch. In contrast, the net update factor of the subsequent gradients −1 are relatively larger, though they indicate a wrong updating direction. We further consider the general case where β 1 = 0. The is presented in the following lemma:Lemma 3. In the sequential online optimization problem in Equation 6, when n → ∞, the limit of net update factor k(g nd+i) of epoch n satisfies: DISPLAYFORM2 and lim DISPLAYFORM3 where k(C) denotes the net update factor for gradient g i = C.Lemma 3 tells us that, in sequential online optimization problem in Equation 6, the net update factors are unbalanced. Specifically, the net update factor for the large gradient C is the smallest in the entire epoch, while all gradients −1 have larger net update factors. Such unbalanced net update factors will possibly lead Adam to a wrong accumulated update direction. Similar also holds in the stochastic online optimization problem in Equation 7. We derive the expectation of the net update factor for each gradient in the following lemma:Lemma 4. In the stochastic online optimization problem in Equation 7, assuming α t = 1, it holds that k(C) < k(−1), where k(C) denote the expectation net update factor for g i = C and k(−1) denote the expectation net update factor for g i = −1.Though the formulas of net update factors in the stochastic case are more complicated than those in deterministic case, the analysis is actually more easier: the gradients with the same scale share the same expected net update factor, so we only need to analyze k(C) and k(−1). From Lemma 4, we can see that in terms of the expectation net update factor, k(C) is smaller than k(−1), which means the accumulated influence of gradient C is smaller than gradient −1. As we have observed in the previous section, a common characteristic of these counterexamples is that the net update factor for the gradient with large magnitude is smaller than these with small magnitude. The above observation can also be interpreted as a direct consequence of inappropriate correlation between v t and g t. Recall that v t = β 2 v t−1 + (1 − β 2)g 2 t. Assuming v t−1 is independent of g t, then: when a new gradient g t arrives, if g t is large, v t is likely to be larger; and if g t is small, v t is also likely to be smaller. If β 1 = 0, then k(g t) = α t / √ v t. As a , a large gradient is likely to have a small net update factor, while a small gradient is likely to have a large net update factor in Adam. When it comes to the scenario where β 1 > 0, the arguments are actually quite similar. Given DISPLAYFORM0 are independent from g t, then: not only does v t positively correlate with the magnitude of g t, but also the entire infinite sequence {v i} ∞ i=t positively correlates with the magnitude of g t. Since the net update factor k(DISPLAYFORM1, it is thus negatively correlated with the magnitude of g t . That is, k(g t) for a large gradient is likely to be smaller, while k(g t) for a small gradient is likely to be larger. The unbalanced net update factors cause the non-convergence problem of Adam as well as all other adaptive learning rate methods where v t correlates with g t. To construct a counterexample, the same pattern is that: the large gradient is along the "correct" direction, while the small gradient is along the opposite direction. Due to the fact that the accumulated influence of a large gradient is small while the accumulated influence of a small gradient is large, Adam may update parameters along the wrong direction. Finally, we would like to emphasize that even if Adam updates parameters along the right direction in general, the unbalanced net update factors are still unfavorable since they slow down the convergence. According to the previous discussion, we conclude that the main cause of the non-convergence of Adam is the inappropriate correlation between v t and g t. Currently we have two possible solutions: making v t act like a constant, which declines the correlation, e.g., using a large β 2 or keep v t nondecreasing ; using a large β 1 (Theorem 1), where the aggressive momentum term helps to mitigate the impact of unbalanced net update factors. However, neither of them solves the problem fundamentally. The dilemma caused by v t enforces us to rethink its role. In adaptive learning rate methods, v t plays the role of estimating the second moments of gradients, which reflects the scale of gradient on average. With the adaptive learning rate α t / √ v t, the update step of g t is scaled down by √ v t and achieves rescaling invariance with respect to the scale of g t, which is practically useful to make the training process easy to control and the training system robust. However, the current scheme of v t, i.e., v t = β 2 v t−1 + (1 − β 2)g 2 t, brings a positive correlation between v t and g t, which in reducing the effect of large gradients and increasing the effect of small gradients, and finally causes the non-convergence problem. Therefore, the key is to let v t be a quantity that reflects the scale of the gradients, while at the same time, be decorrelated with current gradient g t. Formally, we have the following theorem:Theorem 5 (Decorrelation leads to convergence). For any fixed online optimization problem with infinitely repeating of a finite set of cost functions {f 1 (θ),..., f t (θ),... f n (θ)}, assuming β 1 = 0 and α t is fixed, we have, if v t follows a fixed distribution and is independent of the current gradient g t, then the expected net update factor for each gradient is identical. Let P v denote the distribution of v t. In the infinitely repeating online optimization scheme, the expectation of net update factor for each gradient g t is DISPLAYFORM0 Given P v is independent of g t, the expectation of the net update factor E[k(g t)] is independent of g t and remains the same for different gradients. With the expected net update factor being a fixed constant, the convergence of the adaptive learning rate method reduces to vanilla SGD.Momentum can be viewed as setting v t as a constant, which makes v t and g t independent. Furthermore, in our view, using an increasing β 2 (AdamNC) or keepingv t as the largest v t (AMSGrad) is also to make v t almost fixed. However, fixing v t is not a desirable solution, because it damages the adaptability of Adam with respect to the adapting of step size. We next introduce the proposed solution to make v t independent of g t, which is based on temporal independent assumption among gradients. We first introduce the idea of temporal decorrelation, then extend our solution to make use of the spatial information of gradients. Finally, we incorporate first moment estimation. The pseudo code of the proposed algorithm is presented as follows. Algorithm 1 AdaShift: Temporal Shifting with Block-wise Spatial Operation DISPLAYFORM1 5: DISPLAYFORM2 end for 9: end for 10: // We ignore the bias-correction, epsilon and other misc for the sake of clarity In practical setting, f t (θ) usually involves different mini-batches x t, i.e., f t (θ) = f (θ; x t). Given the randomness of mini-batch, we assume that the mini-batch x t is independent of each other and further assume that f (θ; x) keeps unchanged over time, then the gradient g t = ∇f (θ; x t) of each mini-batch is independent of each other. Therefore, we could change the update rule for v t to involve g t−n instead of g t, which makes v t and g t temporally shifted and hence decorrelated: DISPLAYFORM0 Note that in the sequential online optimization problem, the assumption "g t is independent of each other" does not hold. However, in the stochastic online optimization problem and practical neural network settings, our assumption generally holds. Most optimization schemes involve a great many parameters. The dimension of θ is high, thus g t and v t are also of high dimension. However, v t is element-wisely computed in Equation 14. Specifically, we only use the i-th dimension of g t−n to calculate the i-th dimension of v t. In other words, it only makes use of the independence between g t−n [i] and g t [i], where g t [i] denotes the i-th element of g t. Actually, in the case of high-dimensional g t and v t, we can further assume that all elements of gradient g t−n at previous timesteps are independent with the i-th dimension of g t. Therefore, all elements in g t−n can be used to compute v t without introducing correlation. To this end, we propose introducing a function φ over all elements of g 2 t−n, i.e., DISPLAYFORM0 For easy reference, we name the elements of g t−n other than g t−n [i] as the spatial elements of g t−n and name φ the spatial function or spatial operation. There is no restriction on the choice of φ, and we use φ(x) = max i x[i] for most of our experiments, which is shown to be a good choice. The max i x[i] operation has a side effect that turns the adaptive learning rate v t into a shared scalar. An important thing here is that, we no longer interpret v t as the second moment of g t. It is merely a random variable that is independent of g t, while at the same time, reflects the overall gradient scale. We leave further investigations on φ as future work. In practical setting, e.g., deep neural network, θ usually consists of many parameter blocks, e.g., the weight and bias for each layer. In deep neural network, the gradient scales (i.e., the variance) for different layers tend to be different . Different gradient scales make it hard to find a learning rate that is suitable for all layers, when using SGD and Momentum methods. In traditional adaptive learning rate methods, they apply element-wise rescaling for each gradient dimension, which achieves rescaling-invariance and somehow solves the above problem. However, Adam sometimes does not generalize better than SGD , which might relate to the excessive learning rate adaptation in Adam. In our temporal decorrelation with spatial operation scheme, we can solve the "different gradient scales" issue more naturally, by applying φ block-wisely and outputs a shared adaptive learning rate scalar v t [i] for each block: DISPLAYFORM0 It makes the algorithm work like an adaptive learning rate SGD, where each block has an adaptive learning rate α t / v t [i] while the relative gradient scale among in-block elements keep unchanged. As illustrated in Algorithm 1, the parameters θ t including the related g t and v t are divided into M blocks. Every block contains the parameters of the same type or same layer in neural network. First moment estimation, i.e., defining m t as a moving average of g t, is an important technique of modern first order optimization algorithms, which alleviates mini-batch oscillations. In this section, we extend our algorithm to incorporate first moment estimation. We have argued that v t needs to be decorrelated with g t. Analogously, when introducing the first moment estimation, we need to make v t and m t independent to make the expected net update factor unbiased. Based on our assumption of temporal independence, we further keep out the latest n gradients {g t−i} n−1 i=0, and update v t and m t via DISPLAYFORM0 In Equation 17, β 1 ∈ plays the role of decay rate for temporal elements. It can be viewed as a truncated version of exponential moving average that only applied to the latest few elements. Since we use truncating, it is feasible to use large β 1 without taking the risk of using too old gradients. In the extreme case where β 1 = 1, it becomes vanilla averaging. The pseudo code of the algorithm that unifies all proposed techniques is presented in Algorithm 1 and a more detailed version can be found in the Appendix. It has the following parameters: spatial operation φ, n ∈ N +, β 1 ∈, β 2 ∈ and α t. The key difference between Adam and the proposed method is that the latter temporally shifts the gradient g t for n-step, i.e., using g t−n for calculating v t and using the kept-out n gradients to evaluate m t (Equation 17), which makes v t and m t decorrelated and consequently solves the nonconvergence issue. In addition, based on our new perspective on adaptive learning rate methods, v t is not necessarily the second moment and it is valid to further involve the calculation of v t with the spatial elements of previous gradients. We thus proposed to introduce the spatial operation φ that outputs a shared scalar for each block. The ing algorithm turns out to be closely related to SGD, where each block has an overall adaptive learning rate and the relative gradient scale in each block is maintained. We name the proposed method that makes use of temporal-shifting to decorrelated v t and m t AdaShift, which means "ADAptive learning rate method with temporal SHIFTing". In this section, we empirically study the proposed method and compare them with Adam, AMSGrad and SGD, on various tasks in terms of training performance and generalization. Without additional declaration, the reported for each algorithm is the best we have found via parameter grid search. The anonymous code is provided at http://bit.ly/2NDXX6x. Firstly, we verify our analysis on the stochastic online optimization problem in Equation 7, where we set C = 101 and δ = 0.02. We compare Adam, AMSGrad and AdaShift in this experiment. For fair comparison, we set α = 0.001, β 1 = 0 and β 2 = 0.999 for all these methods. The are shown in FIG1. We can see that Adam tends to increase θ, that is, the accumulate update of θ in Adam is along the wrong direction, while AMSGrad and AdaShift update θ in the correct direction. Furthermore, given the same learning rate, AdaShift decreases θ faster than AMSGrad, which validates our argument that AMSGrad has a relatively higher v t that slows down the training. In this experiment, we also verify Theorem 1. As shown in FIG1, Adam is also able to converge to the correct direction with a sufficiently large β 1 and β 2. Note that AdaShift still converges with the fastest speed; a small β 1 (e.g., β 1 = 0.9, the light-blue line in FIG1) does not make Adam converge to the correct direction. We do not conduct the experiments on the sequential online optimization problem in Equation 6, because it does not fit our temporal independence assumption. To make it converge, one can use a large β 1 or β 2, or set v t as a constant. We further compare the proposed method with Adam, AMSGrad and SGD by using Logistic Regression and Multilayer Perceptron on MNIST, where the Multilayer Perceptron has two hidden layers and each has 256 hidden units with no internal activation. The are shown in Figure 2 and Figure 3, respectively. We find that in Logistic Regression, these learning algorithms achieve very similar final in terms of both training speed and generalization. In Multilayer Perceptron, we compare Adam, AMSGrad and AdaShift with reduce-max spatial operation (max-AdaShift) and without spatial operation (non-AdaShift). We observe that max-AdaShift achieves the lowest training loss, while non-AdaShift has mild training loss oscillation and at the same time achieves better generalization. The worse generalization of max-AdaShift may be due to overfitting in this task, and the better generalization of non-AdaShift may stem from the regularization effect of its relatively unstable step size. ResNet and DenseNet are two typical modern neural networks, which are efficient and widely-used. We test our algorithm with ResNet and DenseNet on CIFAR-10 datasets. We use a 18-layer ResNet and 100-layer DenseNet in our experiments. We plot the best of Adam, AMSGrad and AdaShift in Figure 4 and Figure 5 for ResNet and DenseNet, respectively. We can see that AMSGrad is relatively worse in terms of both training speed and generalization. Adam and AdaShift share competitive , while AdaShift is generally slightly better, especially the test accuracy of ResNet and the training loss of DenseNet. We further increase the complexity of dataset, switching from CIFAR-10 to Tiny-ImageNet, and compare the performance of Adam, AMSGrad and AdaShift with DenseNet. The are shown in FIG3, from which we can see that the training curves of Adam and AdaShift are basically overlapped, but AdaShift achieves higher test accuracy than Adam. AMSGrad has relatively higher training loss, and its test accuracy is relatively lower at the initial stage. We also test our algorithm on the training of generative model and recurrent model. We choose WGAN-GP that involves Lipschitz continuity condition (which is hard to optimize), and Neural Machine Translation (NMT) that involves typical recurrent unit LSTM, respectively. In FIG4, we compare the performance of Adam, AMSGrad and AdaShift in the training of WGAN-GP discriminator, given a fixed generator. We notice that AdaShift is significantly better than Adam, while the performance of AMSGrad is relatively unsatisfactory. The test performance in terms of BLEU of NMT is shown in FIG4, where AdaShift achieves a higher BLEU than Adam and AMSGrad. In this paper, we study the non-convergence issue of adaptive learning rate methods from the perspective of the equivalent accumulated step size of each gradient, i.e., the net update factor defined in this paper. We show that there exists an inappropriate correlation between v t and g t, which leads to unbalanced net update factor for each gradient. We demonstrate that such unbalanced step sizes are the fundamental cause of non-convergence of Adam, and we further prove that decorrelating v t and g t will lead to unbiased expected step size for each gradient, thus solving the non-convergence problem of Adam. Finally, we propose AdaShift, a novel adaptive learning rate method that decorrelates v t and g t via calculating v t using temporally shifted gradient g t−n.In addition, based on our new perspective on adaptive learning rate methods, v t is no longer necessarily the second moment of g t, but a random variable that is independent of g t and reflects the overall gradient scale. Thus, it is valid to calculate v t with the spatial elements of previous gradients. We further found that when the spatial operation φ outputs a shared scalar for each block, the ing algorithm turns out to be closely related to SGD, where each block has an overall adaptive learning rate and the relative gradient scale in each block is maintained. The experiment demonstrate that AdaShift is able to solve the non-convergence issue of Adam. In the meantime, AdaShift achieves competitive and even better training and testing performance when compared with Adam. FIG7. It suggests that for a fixed sequential online optimization problem, both of β 1 and β 2 determine the direction and speed of Adam optimization process. Furthermore, we also study the threshold point of C and d, under which Adam will change to the incorrect direction, for each fixed β 1 and β 2 that vary among. To simplify the experiments, we keep d = C such that the overall gradient of each epoch being +1. The is shown in FIG7, which suggests, at the condition of larger β 1 or larger β 2, it needs a larger C to make Adam stride on the opposite direction. In other words, large β 1 and β 2 will make the non-convergence rare to happen. We also conduct the experiment in the stochastic problem to analyze the relation among C, β 1, β 2 and the convergence behavior of Adam. Results are shown in the FIG7 and FIG7 and the observations are similar to the previous: larger C will cause non-convergence more easily and a larger β 1 or β 2 somehow help to resolve non-convergence issue. In this experiment, we set δ = 1.Lemma 6 (Critical condition). In the sequential online optimization problem Equation 6, let α t being fixed, define S(β 1, β 2, C, d) to be the sum of the limits of step updates in a d-step epoch: DISPLAYFORM0 Let S(β 1, β 2, C) = 0, assuming β 2 and C are large enough such that v t 1, we get the equation: DISPLAYFORM1 Equation FORMULA0, though being quite complex, tells that both β 1 and β 2 are closely related to the counterexamples, and there exists a critical condition among these parameters. Algorithm 2 AdaShift: We use a first-in-first-out queue Q to denote the averaging window with the length of n. P ush(Q, g t) denotes pushing vector g t to the tail of Q, while P op(Q) pops and returns the head vector of Q. And W is the weight vector calculated via β 1. DISPLAYFORM0 3: for t = 1 to T do 4: DISPLAYFORM1 if t ≤ n then 6: DISPLAYFORM2 else 8: DISPLAYFORM3 P ush(Q, g t)10: DISPLAYFORM4 p t = p t−1 β 2 for i = 1 to M do 13: DISPLAYFORM0 14: DISPLAYFORM1 DISPLAYFORM2 To verify the temporal correlation between g t [i] and g t−n [i], we range n from 1 to 10 and calculate the average temporal correlation coefficient of all variables i. Results are shown in TAB3. To verify the spatial correlation between g t [i] and g t−n [j], we again range n from 1 to 10 and randomly sample some pairs of i and j and calculate the average spatial correlation coefficient of all the selected pairs. Results are shown in [i] and v t within max-AdaShift, we range the keep number n from 1 to 10 to calculate v t and the average correlation coefficient of all variables i. The is shown in Table 3 and Table 4. With bias correction, the formulation of m t is written as follows DISPLAYFORM3 According to L'Hospitals rule, we can draw the following: DISPLAYFORM4 Thus, DISPLAYFORM5 According to the definition of limitation, let g DISPLAYFORM6 2 So, m t shares the same sign with g * in every dimension. Given it is a convex optimization problem, let the optimal parameter be θ *, and the maximum step size is DISPLAYFORM7 Given ∇f t (θ) ∞ ≤ G, we have f t (θ) − f t (θ *) < 2, which implies the average regret DISPLAYFORM8 E PROOF OF LEMMA 2 DISPLAYFORM9 For a fixed d, as n approach infinity, we get the limit of m nd+i as: DISPLAYFORM10 Similarly, for v nd+i: DISPLAYFORM11 For a fixed d, as n approach infinity, we get the limit of v nd+i as: DISPLAYFORM12 Proof. First, we define V i as: DISPLAYFORM0 where 1 ≤ i ≤ d and i ∈ N. And V i has a period of d. Let t = t − nd, then we can draw: DISPLAYFORM1 Thus, we can get the forward difference of k(g nd+i) as: DISPLAYFORM2 V nd+1i monotonically increases within one period, when 1 ≤ i ≤ d and i ∈ N. And the weigh β j 1 for every difference term V j+i+1 − V i is fixed when i varies. Thus, the weighted summation DISPLAYFORM3 is monotonically decreasing from positive to negative. In other words, the forward difference is monotonically decreasing, such that there exists j, 1 ≤ j ≤ d and lim nd→∞ k(g nd+1) is the maximum among all net updates. Moreover, it is obvious that lim DISPLAYFORM4 is the minimum. Hence, we can draw the : DISPLAYFORM5 and lim DISPLAYFORM6 where K(C) is the net update factor for gradient g i = C. Lemma 7. 1 For a bounded random variable X and a differentiable function f (x), the expectation of f (X) is as follows: DISPLAYFORM0 where D(X) is variance of X, and R 3 is as follows: DISPLAYFORM1 F (x) is the distribution function of X. R 3 is a small quantity under some condition. And c is large enough, such that: for any > 0, DISPLAYFORM2 Proof. (Proof of Lemma 4) In the stochastic online optimization problem equation 7, the gradient subjects the distribution as: DISPLAYFORM3 Then we can get the expectation of g i: DISPLAYFORM4 Meanwhile, under the assumption that gradients are i.i.d., the expectation and variance of v i are as following when nd → ∞: DISPLAYFORM5 Then, for the gradient g i, the net update factor is as follows: DISPLAYFORM6 It should to be clarified that we define t j=1 β t−j 2 g 2 i+j equal to zero when t = 0. Then we define X t as: DISPLAYFORM7 DISPLAYFORM8 DISPLAYFORM9 = β 2(t+1) 2 DISPLAYFORM10 DISPLAYFORM11 f (x) = 3 · x −5/2 8 According to lemma 7, we can the expectation of f (X t) as follows: DISPLAYFORM12 DISPLAYFORM13 where DISPLAYFORM14 Then for gradient C and −1, the net update factor is as follows: DISPLAYFORM15 and DISPLAYFORM16 We can see that each term in the infinite series of k(C) is smaller than the corresponding one in k(−1). Thus, k(C) < k(−1). Proof. From Lemma 2, we can get: DISPLAYFORM0 We sum up all updates in an epoch, and define the summation as S(β 1, β 2, C). DISPLAYFORM1 Assume β 2 and C are large enough such that v t 1, we get the approximation of limit of v nd+i as: DISPLAYFORM2 Then we can draw the expression of S(β 1, β 2, C) as: DISPLAYFORM3 Let S(β 1, β 2, C) = 0, we get the equation about critical condition: DISPLAYFORM4 I HYPER-PARAMETERS INVESTIGATION Here, we list all hyper-parameter setting of all above experiments. In this section, we discuss the learning rate α t sensitivity of AdaShift. We set α t ∈ {0.1, 0.01, 0.001} and let n = 10, β 1 = 0.9 and β 2 = 0.999. The are shown in Figure 9 and FIG1. Empirically, we found that when using the max spatial operation, the best learning rate for AdaShift is around ten times of Adam. In this section, we discuss the β 1 and β 2 sensitivity of AdaShift. We set α = 0.01, n = 10 and let β 1 ∈ {0, 0.9} and β 2 ∈ {0.9, 0.99, 0.999}. The are shown in FIG1 and FIG1. According to the , AdaShift holds a low sensitivity to β 1 and β 2. In some tasks, using the first moment estimation (with β 1 = 0.9 and n = 10) or using a large β 2, e.g., 0.999 can attain better performance. The suggested parameters setting is n = 10, β 1 = 0.9, β 2 = 0.999. In this section, we discuss the n sensitivity of AdaShift. Here we also test a extended version of first moment estimation where it only uses the latest m gradients (m ≤ n): DISPLAYFORM0 We set β 1 = 0.9, β 2 = 0.999. The are shown in FIG1, FIG1 and FIG1. In these experiments, AdaShift is fairly stable when changing n and m. We have not find a clear pattern on the performance change with respect to n and m. J TEMPORAL-ONLY AND SPATIAL-ONLYIn our proposed algorithm, we apply a spatial operation on the temporally shifted gradient g t−n to update v t: DISPLAYFORM1 It is based on the temporal independent assumption, i.e., g t−n is independent of g t. And according to our argument in Section 4.2, one can further assume every element in g t−n is independent of the i-th dimension of g t.We purposely avoid involving the spatial elements of the current gradient g t, where the independence might not holds: when a sample which is rare and has a large gradient appear in the mini-batch x t, the overall scale of gradient g t might increase. However, for the temporally already decorrelation g t−i, further taking the advantage of the spatial irrelevance will not suffer from this problem. We here provide extended experiments on two variants of AdaShift: (i) AdaShift (temporal-only), which only uses the vanilla temporal independent assumption and evaluate v t with: v t = β 2 v t−1 + (1 − β 2)g 2 t−n; (ii) AdaShift (spatial-only), which directly uses the spatial elements without temporal shifting. According to our experiments, AdaShift (temporal-only), i.e., without the spatial operation, is less stable than AdaShift. In some tasks, AdaShift (temporal-only) works just fine; while in some other cases, AdaShift (temporal-only) suffers from explosive gradient and requires a relatively small learning rate. The performance of AdaShift (spatial-only) is close to Adam. More experiments for AdaShift (spatial-only) are included in the next section. In this section, we extend the experiments and add the comparisons with Nadam and AdaShift (spatial-only). The are shown in FIG1, Figure19 and Figure20. According to these experiments, Nadam and AdaShift (spatial-only) share similar performence as Adam. Rahimi & Recht raise the point, at test of time talk at NIPS 2017, that it is suspicious that gradient descent (aka back-propagation) is ultimate solution for optimization. A ill-conditioned quadratic problem with Two Layer Linear Net is showed to be challenging for gradient descent based methods, while alternative solutions, e.g., Levenberg-Marquardt, may converge faster and better. The problem is defined as follows: DISPLAYFORM0 where A is some known badly conditioned matrix (k = 10 20 or 10 5), and W 1 and W 2 are the trainable parameters. We test SGD, Adam and AdaShift with this problem, the are shown in FIG1, Figure 24. It turns out as long as the training goes enough long, SGD, Adam, AdaShift all basically converge in this problem. Though SGD is significantly better than Adam and AdaShift. We would tend to believe this is a general issue of adaptive learning rate method when comparing with vanilla SGD. Because these adaptive learning rate methods generally are scale-invariance, i.e., the step-size in terms of g t /sqrt(v t) is basically around one, which makes it hard to converge very well in such a ill-conditioning quadratic problem. SGD, in contrast, has a step-size g t; as the training converges SGD would have a decreasing step-size, makes it much easier to converge better. The above analysis is confirmed with Figure 22 and Figure 23, with a decreasing learning rate, Adam and AdaShfit both converge very good.

Represent the short summary for finding the corresponding scientific article (long)

We analysis and solve the non-convergence issue of Adam.

Represent the TLDR for finding the corresponding scientific article:

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Represent the article for finding a single-sentence TLDR

Question: Most domain adaptation methods consider the problem of transferring knowledge to the target domain from a single source dataset. However, in practical applications, we typically have access to multiple sources. In this paper we propose the first approach for Multi-Source Domain Adaptation (MSDA) based on Generative Adversarial Networks. Our method is inspired by the observation that the appearance of a given image depends on three factors: the domain, the style (characterized in terms of low-level features variations) and the content. For this reason we propose to project the image features onto a space where only the dependence from the content is kept, and then re-project this invariant representation onto the pixel space using the target domain and style. In this way, new labeled images can be generated which are used to train a final target classifier. We test our approach using common MSDA benchmarks, showing that it outperforms state-of-the-art methods. A well known problem in computer vision is the need to adapt a classifier trained on a given source domain in order to work on another domain, i.e. the target. Since the two domains typically have different marginal feature distributions, the adaptation process needs to align the one to the other in order to reduce the domain shift . In many practical scenarios, the target data are not annotated and Unsupervised Domain Adaptation (UDA) methods are required. While most previous adaptation approaches consider a single source domain, in real world applications we may have access to multiple datasets. In this case, Multi-Source Domain Adaptation (MSDA) (; ; ;) methods may be adopted, in which more than one source dataset is considered in order to make the adaptation process more robust. However, despite more data can be used, MSDA is challenging as multiple domain shift problems need to be simultaneously and coherently solved. In this paper we tackle MSDA (unsupervised) problem and we propose a novel Generative Adversarial Network (GAN) for addressing the domain shift when multiple source domains are available. Our solution is based on generating artificial target samples by transforming images from all the source domains. Then the synthetically generated images are used for training the target classifier. While this strategy has been recently adopted in single-source UDA scenarios (; ; ; ;), we are the first to show how it can be effectively exploited in a MSDA setting. The holy grail of any domain adaptation method is to obtain domain invariant representations. Similarly, in multi-domain image-to-image translation tasks it is very crucial to obtain domain invariant representations in order to reduce the number of learned translations from O(N 2) to O(N), where N is the number of domains. Several domain adaptation methods (; ; ;) achieve domain-invariant representations by aligning only domain specific distributions. However, we postulate that style is the most important latent factor that describe a domain and need to be modelled separately for obtaining optimal domain invariant representation. More precisely, in our work we assume that the appearance of an image depends on three factors: i.e. the content, the domain and the style. The domain models properties that are shared by the elements of a dataset but which may not be shared by other datasets, whereas, the factor style represents a property that is shared among different parts of a single image and describes low-level features which concern a specific image. Our generator obtains the do-main invariant representation in a two-step process, by first obtaining style invariant representations followed by achieving domain invariant representation. In more detail, the proposed translation is implemented using a style-and-domain translation generator. This generator is composed of two main components, an encoder and a decoder. Inspired by in the encoder we embed whitening layers that progressively align the styleand-domain feature distributions in order to obtain a representation of the image content which is invariant to these factors. Then, in the decoder, we project this invariant representation onto a new domain-and-style specific distribution with Whitening and Coloring (W C) ) batch transformations, according to the target data. Importantly, the use of an intermediate, explicit invariant representation, obtained through W C, makes the number of domain transformations which need to be learned linear with the number of domains. In other words, this design choice ensures scalability when the number of domains increases, which is a crucial aspect for an effective MSDA method. Contributions. Our main contributions can be summarized as follows. (i) We propose the first generative model dealing with MSDA. We call our approach TriGAN because it is based on three different factors of the images: the style, the domain and the content. (ii) The proposed style-anddomain translation generator is based on style and domain specific statistics which are first removed from and then added to the source images by means of modified W C layers: Instance Whitening Transform (IW T), Domain Whitening Transform (DW T) , conditional Domain Whitening Transform (cDW T) and Adaptive Instance Whitening Transform (AdaIW T). Notably, the IW T and AdaIW T are novel layers introduced with this paper. (iii) We test our method on two MSDA datasets, Digits-Five and Office-Caltech10 , outperforming state-of-the-art methods. In this section we review previous approaches on UDA, considering both single source and multisource methods. Since, the proposed architecture is also related to deep models used for image-toimage translation, we also discuss related work on this topic. Single Source UDA. Single source UDA approaches assume a single labeled source domain and can be broadly classified under three main categories, depending upon the strategies adopted to cope with the domain-shift problem. The first category utilizes the first and second order statistics to model the source and target feature distributions. For instance, (; ; ;) minimize the Maximum Mean Discrepancy, i.e. the distance between the mean of feature distributions between the two domains. On the other hand,;; ) achieve domain invariance by aligning the second-order statistics through correlation alignment. Differently, (; ;) reduce the domain shift by domain alignment layers derived from batch normalization (BN) . This idea has been recently extended in , where grouped-feature whitening (DWT) is used instead of feature standardization as in BN. Contrarily, in our proposed encoder we use the W C transform, which we adapt to work in a generative network. In addition, we also propose other style and domain dependent batch-based normalizations (i.e., IW T, cDW T and AdaIW T). The second category of methods aim to build domain-agnostic representations by means of an adversarial learning-based approach. For instance, discriminative domain-invariant representations are constructed through a gradient reversal layer in . Similarly, the approach in ) utilizes a domain confusion loss to promote the alignment between the source and the target domain. A third category of methods use adversarial learning in a generative framework (i.e., GANs to reconstruct artificial source and/or target images and perform domain adaptation. Notable approaches are SBADA-GAN , CyCADA ), CoGAN , I2I Adapt and Generate To Adapt (GTA) . While these generative methods have been shown to be very successful in UDA, none of them deals with a multi-source setting. Indeed, extending these approaches to deal with multiple source domains is not trivial, because the construction of O(N 2) one-to-one translation generators and discriminator networks would most likely dramatically increase the number of parameters which need to be trained. Figure 1: An overview of the TriGAN generator. We schematically show 3 domains {T, S 1, S 2} -objects with holes, 3D objects and skewered objects, respectively. The content is represented by the object's shape -square, circle or triangle. The style is represented by the color: each image input to G has a different color and each domain has it own set of styles. First, the encoder E creates a styleinvariant representation using IWT blocks. DWT blocks are then used to obtain a domain-invariant representation. Symmetrically, the decoder D brings back domain-specific information with cDWT blocks (for simplicity we show only a single domain, T). Finally, we apply a reference style. The reference style is extracted using style path and it is applied using Adaptive IWT blocks. Multi-source UDA. deal with multiple-source knowledge transfer by borrowing knowledge from the target k nearest-neighbour sources. Similarly, a distribution-weighed combining rule is proposed in to construct a target hypothesis as a weighted combination of source hypotheses. Recently, Deep Cocktail Network (DCTN) uses the distribution-weighted combining rule in an adversarial setting. A Moment Matching Network (M 3 SDA) is introduced in for reducing the discrepancy between the multiple source and the target domains. Differently from these methods which operate in a discriminative setting, our method relies on a deep generative approach for MSDA. Image-to-image Translation. Image-to-image translation approaches, i.e. the methods that learn how to transform an image from one domain to another, possibly keeping its semantics, are the basis of our method. In ) the pix2pix network translates images under the assumption that paired images in the two domains are available at training time. In contrast, CycleGAN ) can learn to translate images using unpaired training samples. Note that, by design, these methods work with two domains. ComboGAN partially alleviates this issue by using N generators for translations among N domains. Our work is also related to StarGAN which handles unpaired image translation amongst N domains (N ≥ 2) through a single generator. However, StarGAN achieves image translation without explicitly forcing representations to be domain invariant and this may lead to significant reduction of network representation power as the number of domains increases. On the other hand, our goal is to obtain an explicit, intermediate image representation which is style-and-domain independent. We use IWT and DWT to achieve this. We also show that this invariant representation can simplify the re-projection process onto a desired style and target domain. This is achieved through AdaIW T and cDW T which into very realistic translations amongst domains. Very recently, a whitening and colouring based image-to-image translation method was proposed in , where the whitening operation is weight-based. Specifically, the whitening operation is approximated by enforcing the convariance matrix, computed from the intermediate features, to be equal to the identity matrix. Conversely, our whitening layers are data dependent and they use the Cholesky decomposition to compute the whitening matrices from the input samples in a closed form, thereby eliminating the need for additional ad-hoc losses. In this section we describe the proposed approach for MSDA. We first provide an overview of our method and introduce the notation adopted throughout the paper (Sec. 3.1). Then we describe the TriGAN architecture (Sec. 3.2) and our training procedure (Sec.3.3). In the MSDA scenario we have access to N labeled source datasets {S j} N j=1, where, and a target unlabeled dataset T = {x k} nt k=1. All the datasets (target included) share the same categories and each of them is associated to a domain D t, respectively. Our final goal is to build a classifier for the target domain D t exploiting the data in {S j} N j=1 ∪ T. Our method is based on two separate training steps. We initially train a generator G which learns how to change the appearance of a real input image in order to adhere to a desired domain and style. Importantly, our G learns mappings between every possible pair of image domains. Once G is trained, we use it to generate target data having the same content of the source data, thus creating a new, labeled, target dataset, which is finally used to train a target classifier C. In more detail, G is trained using {S j} N j=1 ∪ T, however no class label is involved in this phase and T is treated in the same way as the other domain datasets. As mentioned in Sec. 1, G is composed of an encoder E and a decoder D (Fig. 1). The role of E is to whiten, i.e., to remove, both domain-specific and style-specific aspects of the input image features in order to obtain domain and style invariant representations. Conversely and symmetrically, D needs to progressively project the domain-andstyle invariant features generated by E onto a domain-and-style specific space. At training time, G takes as input a batch of images B = {x 1, ..., x m} with corresponding domain labels L = {l 1, ..., l m}, where x i belongs to the domain D li and, and a batch of style images and has the same style of image x O i. The TriGAN architecture is made of a generator network G and a discriminator D P. As stated above, G comprises an encoder E and decoder D, which will be described in (Sec. 3.2.2-3.2.3). The discriminator D P is based on the Projection Discriminator . Before describing the details of G, we briefly review the W C transform ) (Sec. 3.2.1) which is used as the basic operation in our proposed batch-based feature transformations. h×w×d be the tensor representing the activation values of the convolutional feature maps in a given layer corresponding to the input image x, with d channels and h × w spatial locations. We treat each spatial location as a d-dimensional vector, in this way each image x i contains a set of vectors X i = {v 1, ..., v h×w}. With a slight abuse of the notation, we use.., v h×w×m }, which includes all the spatial locations in all the images in a batch. The W C transform is a multivariate extension of the per-dimension normalization and shiftscaling transform (BN) proposed in and widely adopted in both generative and discriminative networks. W C can be described by: where: In Eq. 2, µ B is the centroid of the elements in B, while W B is such that: B, where Σ B is the covariance matrix computed using B. The of applying Eq. 2 to the elements of B, is a set of whitened featuresB = {v 1, ...,v h×w×m}, which lie in a spherical distribution (i.e., with a covariance matrix equal to the identity matrix). On the other hand, Eq. 1 performs a coloring transform, i.e. projects the elements inB onto a learned multivariate Gaussian distribution. While µ B and W B are computed from the elements in B, Eq. 1 depends on the learnable d dimensional vector β and d × d dimensional matrix Γ. Eq. 1 is a linear operation and can be simply implemented using a convolutional layer with kernel size 1 × 1. In this paper we use the WC transform in our encoder E and decoder D, in order to first obtain a style-and-domain invariant representation for each x i ∈ B, and then transform this representation The encoder E is composed of a sequence of standard Convolution k×k -N ormalization -ReLU -AverageP ooling blocks and some ResBlocks (more details in Appendix B), in which we replace the common BN layers with our proposed normalization modules, which are detailed below. Obtaining Style Invariant Representations. In the first two blocks of E we whiten first and secondorder statistics of the low-level features of each X i ⊆ B, which are mainly responsible for the style of an image . To do so, we propose the Instance Whitening Transform (IW T), where the term instance is inspired by Instance Normalization (IN) and highlights that the proposed transform is applied to a set of features extracted from a single image Eq. 3 implies that whitening is performed using an image-specific feature centroid µ Xi and covariance matrix Σ Xi, which represent the first and second-order statistics of the low-level features of x i. Coloring is based on the parameters β and Γ, which do not depend on x i or l i. The coloring operation is the analogous of the shift-scaling per-dimension transform computed in BN just after feature standardization and is necessary to avoid decreasing the network representation capacity ). Obtaining Domain Invariant Representations. In the subsequent blocks of E we whiten first and second-order statistics which are domain specific. For this operation we adopt the Domain Whitening Transform (DW T) proposed in . Specifically, for each X i ⊆ B, let l i be its domain label (see Sec. 3.1) and let B li ⊆ B be the subset of feature which have been extracted from all those images in B which share the same domain label. Then, for each v j ∈ B li: Similarly to Eq. 3, Eq. 4 performs whitening using a subset of the current feature batch. Specifically, all the features in B are partitioned depending on the domain label of the image they have been extracted from, so obtaining B 1, B 2,..., etc, where all the features in B l belongs to the images from domain D l. Then, first and second order statistics (µ B l, Σ B l) are computed thus effectively projecting each v j ∈ B li onto a domain-invariant spherical distribution. A similar idea was recently proposed in in a discriminative network for single-source UDA. However, differently from , we also use coloring by re-projecting the whitened features onto a new space governed by a learned multivariate distribution. This is done using the learnable parameters β and Γ which do not depend on l i. Our decoder D is functionally and structurally symmetric with respect to E: it takes as input domain and style invariant features computed by E and projects these features onto the desired domain with style extracted from desired image x O i. Similarly to E, D is a sequence of ResBlocks and a few U psampling -N ormalizationReLU -Convolution k×k blocks (more details in Appendix B). Similarly to Sec. 3.2.2, in the N ormalization layers we replace BN with our proposed feature normalization approaches, which are detailed below. Projecting Features onto a Domain-specific Distribution. Apart from the last two blocks of D (see below), all the other blocks are dedicated to project the current set of features onto a domain-specific subspace. This subspace is learned from data using domain-specific coloring parameters (β l, Γ l), where l is the label of the corresponding domain. To this purpose we introduce the conditional Domain Whitening Transform (cDW T), where the term "conditional" specifies that the coloring step is conditioned on the domain label l. In more detail: Similarly to Eq. 4, we first partition B into B 1, B 2,..., etc. However, the membership of v j ∈ B to B l is decided taking into account the desired output domain label l O i for each image rather than its original domain as in case of to Eq. 4. Specifically, let v j ∈ X i and the output domain is given by the label l Once B has been partitioned we define cDW T as follows: Note that, after whitening, and differently from Eq. 4, coloring in Eq. 5 is performed using domainspecific Applying a Specific Style. In order to apply a given style to x i, we extract the style from image x O i using the Style Path (see Fig. 1). Style Path consists of two Convolution k×k -IW T -ReLU -AverageP ooling blocks (which shares the parameters with the first two layers of encoder) and a MultiLayer Perceptron (MLP) F. Following we describe a style using first and second order statistics, which are extracted using the IW T blocks. Then we use F to adapt these statistics to the domain-specific representation obtained as the output of the previous step. In fact, in principle, for each v j ∈ X O i, the W hitening operation inside the IW T transform could be "inverted" using: Indeed, the coloring operation (Eq. 1) is the inverse of whitening (Eq. 2). However, the elements of X i now lie in a feature space different from the output space of Eq. 3, thus the transformation defined by Style Path needs to be adapted. For this reason, we use a MLP (F) which implements this adaptation: Note that, in Eq. 7, has been generated, we use it as the coloring parameters of our Adaptive IWT (AdaIW T): Eq. 8 imposes style-specific first and second order statistics to the features of the last blocks of D in order to mimic the style of x O i. GAN Training. For the sake of clarity, in the rest of the paper we use a simplified notation for G, in which G takes as input only one image instead of a batch. Specifically, ) be the generated image, starting from x i (x i ∈ D li) and with desired output domain l O i and style image x O i. G is trained using the combination of three different losses, with the goal of changing the style and the domain of x i while preserving its content. First, we use an adversarial loss based on the Projection Discriminator (D P), which is conditioned on labels (domain labels, in our case) and uses a hinge loss: The second loss is the Identity loss proposed in ), and which in our framework is implemented as follows: In Eq. 11, G computes an identity transformation, being the input and the output domain and style the same. After that, a pixel-to-pixel L 1 norm is computed. Finally, we propose to use a third loss which is based on the rationale that the generation process should be equivariant with respect to a set of simple transformations which preserve the main content of the images (e.g., the foreground object shape). Specifically, we use the set of the affine transformations {h(x; θ)} of image x which are defined by the parameter θ (θ is a 2D transformation matrix). The affine transformation is implemented by differentiable billinear kernel as in . The Equivariance loss is: In Eq. 12, for a given image x i, we randomly choose a parameter θ i and we apply h(·; θ i) tô Then, using the same θ i, we apply h(·; θ i) to x i and we get x i = h(x i ; θ i), which is input to G in order to generate a second image. The two generated images are finally compared using the L 1 norm. This is a form of self-supervision, in which equivariance to geometric transformations is used to extract semantics. Very recently a similar loss has been proposed in , where equivariance to affine transformations is used for image co-segmentation. The complete loss for G is: Classifier Training. Once G is trained, we use it to artificially create a labeled training dataset for the target domain. Specifically, for each S j and each (x i, y i) ∈ S j, we randomly pick one image x t from T which is used as the style-image reference, and we generate: where N + 1 is fixed and indicates the target domain D t label (see Sec. 3.1). and the process is iterated. Finally, we train a classfier C on T L using the cross-entropy loss: In this section we describe the experimental setup and then we evaluate our approach using MSDA datasets. We also present an ablation study in which we analyse the impact of each of TriGAN component on the classification accuracy. In our experiments we consider two common domain adaptation benchmarks, namely the DigitsFive benchmark and the Office-Caltech . The Digits-Five is composed of five digit-recognition datasets: USPS , MNIST , MNIST-M , SVHN and Synthetic numbers datasets (SYNDIGITS). SVHN contains images from Google Street View of real-world house numbers. Synthetic numbers includes 500K computer-generated digits with different sources of variations (i.e. position, orientation, color, blur). USPS is a dataset of digits scanned from U.S. envelopes, MNIST is a popular benchmark for digit recognition and MNIST-M is its colored counterpart. We adopt the experimental protocol described in : in each domain the train/test split is composed of a subset of 25000 images for training and 9000 for testing. For USPS the entire dataset is used. The Office-Caltech is a domain-adaptation benchmark obtained selecting the subset of 10 categories shared between the Office31 and the Caltech256 datasets. It contains 2533 images, about half of which belong to Caltech256. There are four different domains: Amazon (A), DSLR (D), Webcam (W) and Caltech256 (C). We provide architecture details about our generator G and discriminator D P in the Appendix B. We train TriGAN for 100 epochs using the Adam optimizer with the learning rate set to 1e-4 for the G and 4e-4 for the D P as in . The loss weighing factor λ in Eqn. 13 is set to 10 as in ). All other hyperparameters are chosen by crossvalidating on the MNIST-M, USPS, SVHN, SYNDIGITS → MNIST adaptation setting and are used in all the other settings. For the Digits-Five experiments we use a mini-batch of size 256 for TriGAN training. Due to the difference in image resolution and image channels, the images of all the domains are converted to 32 × 32 RGB. For a fair comparison, for the final target classifier C we use exactly the same network architecture used in . In the Office-Caltech10 experiments we downsample the images to 164 × 164 to accommodate more samples in a mini-batch. We use a mini-batch of size 24 for training with 1 GPU. For the back-bone target classifier C we use the ResNet101 architecture used by. The weights are initialized with a network pre-trained on the ILSVRC-2012 dataset . In our experiments we remove the output layer and we replace it with a randomly initialized fully-connected layer that has 10 logits, one per each class of the OfficeCaltech10 dataset. C is trained with Adam with an initial learning rate of 1e-5 for the randomly initialized last layer and 1e-6 for all other layers. Since there are only a few training data in the T L dataset, we also use {S j} N j=1 for training C. In this section we analyse our proposal using an ablation study and we compare with MSDA stateof-the-art methods. In Appendix A we show our qualitative . In this section we compare our method with previous MSDA approaches. Tab. 1 and Tab. 2 show the on the Digits-Five and Office-Caltech10 datset, respectively. Table 1 shows that TriGAN achieves an average accuracy of 90.08% which is higher than all other methods. M 3 SDA is better in the mm, up, sv, sy → mt and in the mt, mm, sv, sy → up settings, where TriGAN is the second best. In all the other settings, TriGAN outperforms all other approaches. As an example, in the mt, up, sv, sy → mm setting, TriGAN is better than the second best method M 3 SDA by a significant margin of 10.38%. For the StarGAN baseline, synthetic images are generated in the target domain and a target classifier is trained using our protocol described in Sec. 3.3. StarGAN, despite known to work well for aligned face translation, fails drastically when digits are concerned. This shows the importance of a well-designed generator that enforces domain invariant representations in the MSDA setting when there is a significant domain shift. Models Table 1: Classification accuracy (%) on Digits-Five experiments. MNIST-M, MNIST, USPS, SVHN, Synthetic Digits are abbreviated as mm, mt, up, sv and sy respectively. The best value is in bold and the second best is underlined. Finally, we also experimented using the Office-Caltech10, which is considered to be difficult for reconstruction-based GAN methods because of the high-resolution images. Although the dataset is quite saturated, TriGAN achieves a classification accuracy of 97.0%, outperforming all the other methods and beating the previous state-of-the-art approach (M 3 SDA) by a margin of 0.6% on average (see Tab. 2). Table 2: Classification accuracy (%) on Office-Caltech10 dataset. ResNet-101 pre-trained on ImageNet is used as the backbone network. The best value is in bold and the second best is underlined. In this section we analyse the different components of our method and study in isolation their impact on the final accuracy. Specifically, we use the Digits-Five dataset and the following models: i) Model A, which is our full model containing the following components: IWT, DWT, cDWT, AdaIWT and L Eq. ii) Model B, which is similar to Model A except we replace L Eq with the cycle-consistency loss L Cycle of CycleGAN ). iii) Model C, where we replace IWT, DWT, cDWT and AdaIWT of Model A with IN , BN , conditional Batch Normalization (cBN) and Adaptive Instance Normalization (AdaIN) . This comparison highlights the difference between feature whitening and feature standardisation. iv) Model D, which ignores the style factor. Specifically, in Model D, the blocks related to the style factor, i.e., the IWT and the AdaIWT blocks, are replaced by DWT and cDWT blocks, respectively. v) Model E, in which the style path differs from Model A in the way the style is applied to the domain-specific representation. Specifically, we remove the MLP F and we directly apply (µ). vi) Finally, Model F represents no-domain assumption (e.g. the DWT and cDWT blocks are replaced with standard WC blocks). Tab. 3 shows that Model A outperforms all the ablated models. Model B shows that L Cycle is detrimental for the accuracy because G may focus on meaningless information to reconstruct back the image. Conversely, the affine transformations used in case of L Eq, force G to focus on the shape of the content of the images. Model C is outperformed by model A, demonstrating the importance of feature whitening over feature standardisation, corroborating the findings of in a pure-discriminative setting. Moreover, the no-style assumption in Model D hurts the classification accuracy by a margin of 1.76% when compared with Model A. We believe this is due to the fact that, when only domain-specific latent factors are modeled but instance-specific style information is missing in the image translation process, then the diversity of translations decreases, consequently reducing the final accuracy. Model E shows the need of using the proposed style path. Finally, Model F shows that having a separate factor for domain yields better performance Our proposed method can be used for multi-domain image-to-image translation tasks. We conduct experiments on Alps Seasons dataset which consists of images of Alps mountain range belonging to four different domains. Fig. 2 shows some images generated using our generator on the Alps Seasons. For this experiment we compare our generator with StarGAN and report the FID metrics for the generated images. FID measures the realism of generated images and it is desirable to have a lower FID score. The FID is computed considering all the real samples in the target domain and generating equivalent number of synthetic images in the target domain. It can be observed from Tab. 4 that the FID scores of our approach is significantly lower than that of StarGAN. This further highlights the fact that explicit enforcing of domain and style invariant representation is essential for multi-domain translation. 5 In this work we proposed TriGAN, an MSDA framework which is based on data-generation from multiple source domains using a single generator. The underlying principle of our approach to to obtain domain-style invariant representations in order to simplify the generation process. Specifically, our generator progressively removes style and domain specific statistics from the source images and then re-projects the so obtained invariant representation onto the desired target domain and styles. We obtained state-of-the-art on two MSDA datasets, showing the potentiality of our approach. We performed a detailed ablation study which shows the importance of each component of the proposed method. Some sample translations of our G are shown in Fig. 3. For example, in Fig. 3 (a) when the SVHN digit "six" with side-digits is translated to MNIST-M the cDWT blocks re-projects it to MNIST-M domain (i.e., single digit without side-digits) and the AdaIWT block applies the instance-specific style of the digit "three" (i.e., blue digit with red ) to yield a blue "six" with red . Similar trends are also observed in Fig. 3 (b). Adaptive Instance Whitening (AdaIWT) blocks. The AdaIWT blocks are analogous to the IWT blocks except from the IWT which is replaced by the AdaIWT. The AdaIWT block is a sequence: U psampling m×m − Convolution k×k − AdaIW T − ReLU, where m = 2 and k = 3. AdaIWT also takes as input the coloring parameters (Γ, β) (See Sec. 3.2.3) and Fig. 4 (b) ). Two AdaIWT blocks are consecutively used in D. The last AdaIWT block is followed by a Convolution 5×5 layer. Style Path. The Style Path is composed of: (Fig. 4 (c) ). The output of the Style Path is (β 1 Γ 1) and (β 2 Γ 2), which are input to the second and the first AdaIWT blocks, respectively (see Fig. 4 (b) ). The M LP is composed of five fully-connected layers with 256, 128, 128, 256 neurons, with the last fully-connected layer having a number of neurons equal to the cardinality of the coloring parameters (β Γ). Domain Whitening Transform (DWT) blocks. The schematic representation of a DWT block is shown in Fig. 5 (a). For the DWT blocks we adopt a residual-like structure: We also add identity shortcuts in the DWT residual blocks to aid the training process. Conditional Domain Whitening Transform (cDWT) blocks. The proposed cDWT blocks are schematically shown in Fig. 5 (b). Similarly to a DWT block, a cDWT block contains the following layers: cDW T − ReLU − Convolution 3×3 − cDW T − ReLU − Convolution 3×3. Identity shortcuts are also used in the cDWT residual blocks. All the above blocks are assembled to construct G, as shown in Fig. 6. Specifically, G contains two IWT blocks, one DWT block, one cDWT block and two AdaIWT blocks. It also contains the Style Path and 2 Convolution 5×5 (one before the first IWT block and another after the last AdaIWT block), which is omitted in Fig. 6 for the sake of clarity. {Γ 1, β 1, Γ 2, β 2} are computed using the Style Path. For the discriminator D P architecture we use a Projection Discriminator . In D P we use projection shortcuts instead of identity shortcuts. In Fig 7 we schematically show a discriminator block. D P is composed of 2 such blocks. We use spectral normalization in D P. Since, our proposed TriGAN has a generic framework and can handle N -way domain translations, we also conduct experiments for Single-Source UDA scenario where N = 2 and the source domain is grayscale MNIST. We consider the following UDA settings with the digits dataset: C.1 DATASETS MNIST → USPS. The MNIST dataset contains grayscale images of handwritten digits 0 to 9. The pixel resolution of MNIST digits is 28 × 28. The USPS contains similar grayscale handwritten digits except the resolution is 16 × 16. We up-sample images from both domains to 32 × 32 during 91.2 62.0 -ADDA 89.4 --PixelDA 95.9 98.2 -UNIT 95.9 --SBADA-GAN 97.6 99.4 61.1 GenToAdapt 92.5 -36.4 CyCADA 94.8 --I2I Adapt 92.1 --TriGAN (Ours) 98.0 95.7 66.3 Table 5: Classification Accuracy (%) of GAN-based methods on the Single-source UDA setting for Digits Recognition. The best number is in bold and the second best is underlined. In this section we compare our proposed TriGAN with GAN-based state-of-the-art methods, both with adversarial learning based approaches and reconstruction-based approaches. Tab. 5 reports the performance of our TriGAN alongside the obtained from the following baselines: Domain Adversarial Neural Network (DANN), Coupled generative adversarial networks (CoGAN), Adversarial discriminative domain adaptation ) (ADDA), Pixel-level domain adaptation (PixelDA), Unsupervised image-to-image translation networks ) (UNIT), Symmetric bi-directional adaptive gan (SBADA-GAN), Generate to adapt (GenToAdapt), Cycle-consistent adversarial domain adaptation ) (CyCADA) and Image to image translation for domain adaptation (I2I Adapt). As can be seen from Tab. 5 TriGAN does better in two out of three adaptation settings. It is only worse in the MNIST → MNIST-M setting where it is the third best. It is to be noted that TriGAN does significantly well in MNIST → SVHN adaptation which is particularly considered as a hard setting. TriGAN is 5.2% better than the second best method SBADA-GAN for MNIST → SVHN.

In this paper we propose generative method for multisource domain adaptation based on decomposition of content, style and domain factors.

Represent the short summary for finding the corresponding scientific article (long)

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.


Question: Represent the article for finding a single-sentence TLDR

Question: Inferring the most likely configuration for a subset of variables of a joint distribution given the remaining ones – which we refer to as co-generation – is an important challenge that is computationally demanding for all but the simplest settings. This task has received a considerable amount of attention, particularly for classical ways of modeling distributions like structured prediction. In contrast, almost nothing is known about this task when considering recently proposed techniques for modeling high-dimensional distributions, particularly generative adversarial nets (GANs). Therefore, in this paper, we study the occurring challenges for co-generation with GANs. To address those challenges we develop an annealed importance sampling (AIS) based Hamiltonian Monte Carlo (HMC) co-generation algorithm. The presented approach significantly outperforms classical gradient-based methods on synthetic data and on CelebA. While generative adversarial nets (GANs) and variational auto-encoders (VAEs) model a joint probability distribution which implicitly captures the correlations between multiple parts of the output, e.g., pixels in an image, and while those methods permit easy sampling from the entire output space domain, it remains an open question how to sample from part of the domain given the remainder? We refer to this task as co-generation. To enable co-generation for a domain unknown at training time, for GANs, optimization based algorithms have been proposed. Intuitively, they aim at finding that latent sample which accurately matches the observed part. However, successful training of the GAN leads to an increasingly ragged energy landscape, making the search for an appropriate latent variable via backpropagation through the generator harder and harder until it eventually fails. To deal with this ragged energy landscape during co-generation, we develop a method using an annealed importance sampling (AIS) based Hamiltonian Monte Carlo (HMC) algorithm, which is typically used to estimate (ratios of) the partition function. Rather than focus on the partition function, the proposed approach leverages the benefits of AIS, i.e., gradually annealing a complex probability distribution, and HMC, i.e., avoiding a localized random walk. We evaluate the proposed approach on synthetic data and imaging data (CelebA), showing compelling via MSE and MSSIM metrics. For more details and please see our main conference paper. In the following we first motivate the problem of co-generation before we present an overview of our proposed approach and discuss the details of the employed Hamiltonian Monte Carlo method. Assume we are given a well trained generatorx = G θ (z), parameterized by θ, which is able to produce samplesx from an implicitly modeled distribution p G (x|z) via a transformation of embeddings z. Further assume we are given partially observed data x o while the remaining part x h of the data x = (x o, x h) is latent. To reconstruct the latent parts of the data x h from available observations x o, a program can be formulated as follows: where G θ (z) o denotes the restriction of the generated sample G θ (z) to the observed part. Upon solving the program given in Eq., we obtain an estimate for the missing datax However, in practice, Eq. turns out to be extremely hard to address, particularly if the generator G θ (z) is very well trained. To see this, consider as an example a generator operating on a 2-dimensional latent space z = (z 1, z 2) and 2-dimensional data x = (x 1, x 2) (blue points in Fig. 2(a) ). We use h = 1 and let x o = x 2 = 0. In the first row of Fig. 1 we illustrate the loss surface of the objective given in Eq. obtained when using a generator G θ (z) trained on the original 2-dimensional data for 500, 1.5k, 2.5k and 15k iterations (columns in Fig. 1). We observe the latent space to become increasingly ragged, exhibiting folds that clearly separate different data regimes. First (e.g., gradient descent (GD)) or second order optimization techniques cannot cope easily with such a loss landscape and likely get trapped in local optima. We observe GD (red trajectory in Fig. 1 first row and loss in second row) to get stuck in a local optimum as the loss fails to decrease to zero once the generator better captures the data. To prevent those local-optima issues for co-generation, we propose an annealed importance-sampling (AIS) based Hamiltonian Monte Carlo (HMC) method in the following (Alg. 1). In order to reconstruct the hidden portion x h of the data. To obtain samplesẑ following the posterior distribution p(z|x o), we use annealed importance sampling (AIS) to gradually approach the complex and often high-dimensional posterior distribution p(z|x o) by simulating a Markov Chain starting from the prior distribution p(z) = N (z|0, I), a standard normal distribution (zero mean and unit variance). Formally, we define an annealing schedule for the parameter β t from β 0 = 0 to β T = 1. At every time step t ∈ {1, . . ., T} we refine the samples drawn at the previous timestep t − 1 so as to represent the distributionp t (z|x o) = p(z|x o) βt p(z) 1−βt. We use a sigmoid schedule for the parameter β t. To successively refine the samples we use Hamilton Monte Carlo (HMC) sampling because a proposed update can be far from the current sample while still having a high acceptance probability. We use 0.01 as the leapfrog step size and employ 10 leapfrog updates per HMC loop for the synthetic 2D dataset and 20 leapfrog updates for real dataset at first. The acceptance rate is 0.65, as recommended by Neal. Hamilton Monte Carlo (HMC) is capable of traversing folds in an energy landscape. For this, HMC methods trade potential energy U t (z) = − logp t (z|x o) with kinetic energy K t (v). ∀z ∈ Z compute new proposal sample using leapfrog integration on Hamiltonian 8: ∀z ∈ Z use Metropolis Hastings to check whether to accept the proposal and update Z 9: end for 10: end for 11: Return: Z HMC defines a Hamiltonian H(z, v) = U (z) + K(v) or conversely a joint probability distribution log p(z, v) ∝ −H(z, v) and proceeds by iterating three steps M times. In a first step, the Hamiltonian is initialized by randomly sampling the momentum variable v, typically using a standard Gaussian. In the second step, (z *, v *) are proposed via leapfrog integration to move along a hypersurface of the Hamiltonian. In the final third step we decide whether to accept the proposal (z *, v *) computed via leapfrog integration. Formally, we accept the proposal with probability min{1, exp (−H(z If the proposed state (z *, v *) is rejected, the m + 1-th iteration reuses z, otherwise z is replaced with z * in the m + 1-th iteration. This process is shown in Alg. 1 line 6, 7 and 8. Baselines: In the following, we evaluate the proposed approach on synthetic and imaging data. We use two GD baselines, employing different initialization methods. The first one samples a single z randomly. The second picks that one sample z from 5000 points which best matches the objective given in Eq. initially. To illustrate the advantage of our proposed method over the common baseline, we first demonstrate our on 2-dimensional synthetic data. Specifically, the 2-dimensional data x = (x 1, x 2) is drawn from a mixture of five equally weighted Gaussians each with a variance of 0.02, the means of In this experiment, we aim to reconstruct x = (x 1, x 2), given x o = x 2 = 0. The optimal solution for the reconstruction isx =, where the reconstruction error should be 0. However, as discussed in reference to Fig. 1 earlier, we observe that energy barriers in the Z-space complicate optimization. In contrast, our proposed AIS co-generation method only requires one initialization to achieve the desired after 6, 000 AIS loops, as shown in Fig. 2 (15000 (d) ). Specifically, reconstruction with generators trained for a different number of epochs (500, 1.5k and 15k) are shown in the rows. The samples obtained from the generator for the data (blue points in column (a)) are illustrated in column (a) using black color. Using the respective generator to solve the program given in Eq. via GD yields highlighted with yellow color in column (b). The empirical reconstruction error frequency for this baseline is given in column (c). The and the reconstruction error frequency obtained with Alg. 1 are shown in columns (d, e). We observe significantly better and robustness to initialization. In Fig. 3 we show for 100 samples that Alg. 1 moves them across the energy barriers during the annealing procedure, illustrating the benefits of AIS based HMC over GD. To validate our method on real data, we evaluate on CelebA, using MSE and MSSIM metrics. We use the progressive GAN architecture. The size of the input is 512 and the size of the output is 128 × 128. We randomly mask blocks of width and height ranging from 30 to 60. Then we use Alg. 1 for reconstruction with 500 HMC loops. In Fig. 3 (a,b), we observe that Alg. 1 outperforms over both baselines for all GAN training iterations on both MSSIM and MSE metrics. In Fig. 4 we show generated by both baselines and Alg. 1. We propose a co-generation approach, i.e., we complete partially given input data, using annealed importance sampling (AIS) based on the Hamiltonian Monte Carlo (HMC). Different from classical optimization based methods, specifically GD, which get easily trapped in local optima when solving this task, the proposed approach is much more robust. Importantly, the method is able to traverse large energy barriers that occur when training generative adversarial nets. Its robustness is due to AIS gradually annealing a probability distribution and HMC avoiding localized walks. We show additional for real data experiments. We observe our proposed algorithm to recover masked images more accurately than baselines and to generate better high-resolution images given low-resolution images. We show masked CelebA (Fig. 5) and LSUN (Fig. 6) recovery for baselines and our method, given a Progressive GAN generator. Note that our algorithm is pretty robust to the position of the z initialization, since the generated are consistent in Fig. 5. (a) (b) (c)

Represent the short summary for finding the corresponding scientific article (long).

Using annealed importance sampling on the co-generation problem.