Question: Represent the article for finding a single-sentence TLDR

Question: Temporal difference (TD) learning is a popular algorithm for policy evaluation in reinforcement learning, but the vanilla TD can substantially suffer from the inherent optimization variance. A variance reduced TD (VRTD) algorithm was proposed by , which applies the variance reduction technique directly to the online TD learning with Markovian samples. In this work, we first point out the technical errors in the analysis of VRTD in , and then provide a mathematically solid analysis of the non-asymptotic convergence of VRTD and its variance reduction performance. We show that VRTD is guaranteed to converge to a neighborhood of the fixed-point solution of TD at a linear convergence rate. Furthermore, the variance error (for both i.i.d. and Markovian sampling) and the bias error (for Markovian sampling) of VRTD are significantly reduced by the batch size of variance reduction in comparison to those of vanilla TD. In reinforcement learning (RL), policy evaluation aims to obtain the expected long-term reward of a given policy and plays an important role in identifying the optimal policy that achieves the maximal cumulative reward over time;;. The temporal difference (TD) learning algorithm, originally proposed by , is one of the most widely used policy evaluation methods, which uses the Bellman equation to iteratively bootstrap the estimation process and continually update the value function in an incremental way. In practice, if the state space is large or infinite, function approximation is often used to find an approximate value function efficiently. Theoretically, TD with linear function approximation has been shown to converge to the fixed point solution with i.i.d. samples and Markovian samples in;. The finite sample analysis of TD has also been studied in;; Dalal et al. (2018a);. Since each iteration of TD uses one or a mini-batch of samples to estimate the mean of the gradient 1, TD learning usually suffers from the inherent variance, which substantially degrades the convergence accuracy. Although a diminishing stepsize or very small constant stepsize can reduce the variance;; Dalal et al. (2018a), they also slow down the convergence significantly. Two approaches have been proposed to reduce the variance. The first approach is the so-called batch TD, which takes a fixed sample set and transforms the empirical mean square projected Bellman error (MSPBE) into an equivalent convex-concave saddle-point problem. Due to the finite-sample nature of such a problem, stochastic variance reduction techniques for conventional optimization can be directly applied here to reduce the variance. In particular, showed that and can be applied to improve the performance of batch TD algorithms, and proposed two variants of SVRG to further save the computation cost. However, the analysis of batch TD does not take into account the statistical nature of the training samples, which are generated by a MDP. Hence, there is no guarantee of such obtained solutions to be close to the fixed point of TD learning. The second approach is the so-called TD with centering (CTD) algorithm proposed in , which introduces the variance reduction idea to the original TD learning algorithm. For the sake of better reflecting its major feature, we refer to CTD as Variance Reduced TD (VRTD) throughout this paper. Similarly to the SVRG in , VRTD has outer and inner loops. The beginning of each inner-loop (i.e. each epoch) computes a batch of sample gradients so that each subsequent inner loop iteration modifies only one sample gradient in the batch gradient to reduce the variance. The main difference between VRTD and batch TD is that VRTD applies the variance reduction directly to TD learning rather than to a transformed optimization problem in batch TD. empirically verified that VRTD has better convergence accuracy than vanilla TD learning, some technical errors in the analysis in have been pointed out in follow up studies Dalal et al. (2018a); Narayanan and Szepesvári. Furthermore, as we discuss in Section 3, the technical proof in regarding the convergence of VRTD also has technical errors so that their do not correctly characterize the impact of variance reduction on TD learning. Given the recent surge of interest in the finite time analysis of the vanilla;; Dalal et al. (2018a), it becomes imperative to reanalyze the VRTD and accurately understand whether and how variance reduction can help to improve the convergence accuracy over vanilla TD. Towards this end, this paper specifically addresses the following central questions. • For i.i.d. sampling, it has been shown in that vanilla TD converges only to a neighborhood of the fixed point for a constant stepsize and suffers from a constant error term caused by the variance of the stochastic gradient at each iteration. For VRTD, does the variance reduction help to reduce such an error and improve the accuracy of convergence? How does the error depend on the variance reduction parameter, i.e., the batch size for variance reduction? • For Markovian sampling, it has been shown in; that the convergence of vanilla TD further suffers from a bias error due to the correlation among samples in addition to the variance error as in i.i.d. sampling. Does VRTD, which was designed to have reduced variance, also enjoy reduced bias error? If so, how does the bias error depend on the batch size for variance reduction? Our main contributions are summarized in Table 1 and are described as follows. For i.i.d. sampling, we show that a slightly modified version of VRTD (for avoiding bias error) converges linearly to a neighborhood of the fixed point solution for a constant stepsize α, with the variance error at the order of O(α/M), where M is the batch size for variance reduction. This clearly reduces the corresponding variance error O(α) of vanilla TD in. For Markovian sampling, we show that VRTD has the same linear convergence and the same variance error reduction over the vanilla; as i.i.d. sampling. More importantly, the variance reduction in VRTD also attains a substantially reduced bias error at the order of O(1/M) over the vanilla; , where the bias error is at the order of O(α). Therefore, vanilla TD typically needs to decrease the stepsize α in order to reduce the variance and bias errors, which however slows down the convergence. In contrast, VRTD can increase the batch size to reduce both errors while still keeping the stepsize at a desired constant level to maintain fast convergence, as can be observed in our experiments. At the technical level, our analysis of bias error for Markovian sampling takes a different path from the techniques used in;;. Due to the batch average of stochastic gradients adopted by VRTD to reduce the variance, we apply a concentration bound established in Dedecker and Gouëzel for Markovian samples. This shows that the correlation among samples in different epochs is eliminated due to the concentration to a deterministic average, and the correlation among samples within each epoch is implicitly captured by the parameters in the concentration inequality. Such an analysis also explicitly explains why the variance reduction step can also reduce the bias error. On-policy TD learning and variance reduction. On-policy TD learning aims to minimize the Mean Squared Bellman Error (MSBE) when samples are drawn independently from the stationary distribution of the corresponding MDP. The non-asymptotic convergence under i.i.d. sampling has been established in Dalal et al. (2018a) for TD with linear function approximation and for TD with overparameterized neural network approximation. The convergence of averaged linear SA with constant stepsize has been studied in. In the Markovian setting, the non-asymptotic convergence has been studied for on-policy TD in;;;. proposed a variance reduced CTD algorithm (called VRTD in this paper), which directly applies variance reduction technique to the TD algorithm. The analysis of VRTD provided in has technical errors. The aim of this paper is to provide a technically solid analysis for VRTD to characterize the advantage of variance reduction. Variance reduced batch TD learning. algorithms are generally designed for policy evaluation by solving an optimization problem on a fixed dataset. , the empirical MSPBE is first transformed into a quadratic convex-concave saddle-point optimization problem and variance reduction methods of and were then incorporated into a primal-dual batch gradient method. applied two variants of variance reduction methods to solve the same saddle point problems, and showed that those two methods can save gradient computation cost. We note that due to the extensive research in TD learning, we include here only studies that are highly related to our work, and cannot cover many other interesting topics on TD learning such as asymptotic convergence of TD learning Tadić; , off-policy TD learning Sutton et al. (2008; 2009);;; , two time-scale TD algorithms Dalal et al. (2018b); , fitted TD algorithms , etc. The idea of the variance reduction algorithm proposed in as well as the analysis techniques that we develop in this paper can potentially be useful for these algorithms. We describe the problem of value function evaluation over a Markov decision process (MDP) (S, A, P, r, γ), where each component is explained in the sequel. Suppose S ⊂ R d is a compact state space, and A is a finite action set. Consider a stationary policy π, which maps a state s ∈ S to the actions in A via a probability distribution π(·|s). At time-step t, suppose the process is in some state s t ∈ S, and an action a t ∈ A is taken based on the policy π(·|s t). Then the transition kernel P = P(s t+1 |s t, a t) determines the probability of being at state s t+1 ∈ S in the next time-step, and the reward r t = r(s t, a t, s t+1) is received, which is assumed to be bounded by r max. We denote the associated Markov chain by p(s |s) = a∈A p(s |s, a)π(a|s), and assume that it is ergodic. Let µ π be the induced stationary distribution, i.e., s p(s |s)µ π (s) = µ π (s). We define the value function for a policy π as v with φ i (s) for i = 1, 2, · · · d denoting the fixed basis feature functions of state s, and θ ∈ R d is a parameter vector. Let Φ be the |S| × d feature matrix (with rows indexed by the state and columns corresponding to components of θ). The linear function approximation can be written in the vector form asv(θ) = Φθ. Our goal is to find the fixed-point parameter θ The TD learning algorithm performs the following fixed-point iterative update to find such θ *. where α t > 0 is the stepsize, and A xt and b xt are specified below. For i.i.d. samples generated from the distribution µ π, we denote the sample as x t = (s t, r t, s t), and A xt = φ(s t)(γφ(s t) − φ(s t)) and b xt = r(s t)φ(s t). For Markovian samples generated sequentially from a trajectory, we denote the sample as x t = (s t, r t, s t+1), and in this case A xt = φ(s t)(γφ(s t+1) − φ(s t)) and b xt = r(s t)φ(s t). We further define the mean gradient g(θ) = Aθ + b where We call g(θ) as gradient for convenience due to its analogous role as in the gradient descent algorithm. It has been shown that the iteration in eq. converges to the fix point θ * = −A −1 b at a sublinear rate O(1/t) with diminishing stepsize α t = O(1/t) using both Markovian and i.i.d. samples; Dalal et al. (2018a);. Throughout the paper, we make the following standard assumptions;;;;. Assumption 1 (Problem solvability). The matrix A is non-singular. Assumption 3 (Geometric ergodicity). The considered MDP is irreducible and aperiodic, and there exist constants κ > 0 and ρ ∈ such that where d T V (P, Q) denotes the total-variation distance between the probability measures P and Q. Assumption 1 requires the matrix A to be non-singular so that the optimal parameter θ * = −A −1 b is well defined. Assumption 2 can be ensured by normalizing the basis functions. Assumption 3 holds for any time-homogeneous Markov chain with finite state-space and any uniformly ergodic Markov chains with general state space. In this section, we first introduce the variance-reduced TD (VRTD) algorithm proposed in for Markovian sampling and then discuss the technical errors in the analysis of VRTD in. 3.1 Since the standard TD learning takes only one sample in each update as can be seen in eq., it typically suffers from a large variance. This motivates the development of the VRTD algorithm in (named as CTD in). VRTD is formally presented in Algorithm 2, and we briefly introduce the idea below. The algorithm runs in a nested fashion with each inner-loop (i.e., each epoch) consists of M updates. At the beginning of the m-th epoch, a batch of M samples are acquired and a batch gradient g m (θ m−1) is computed based on these samples as an estimator of the mean gradient. Then, each inner-loop update randomly takes one sample from the batch, and updates the corresponding component in g m (θ m−1). Here, Π R θ in Algorithm 2 denotes the projection operator onto a norm ball with the radius R θ. The idea is similar to the SVRG algorithm proposed in for conventional optimization. Since a batch gradient is used at each inner-loop update, the variance of the gradient is expected to be reduced. Input: batch size M, learning rate α and initializatioñ θ0 1: for m = 1, 2,..., S do 2: θm,0 =θm−1 3: Sample a set Bm with M samples indepedently from the distribution µπ 4: Sample xj m,t indepedently from the distribution µπ 7: θm,t+1 = θm,t + α gx j m,t (θm,t) −gx j m,t (θm−1) + gm(θm−1) 9: end for 10: setθm = θm,t for randomly chosen t ∈ {1, 2, ..., M} 11: end for Output:θS Algorithm 2 Variance Reduced TD with Markovian samples Input: batch size M, learning rate α and initializatioñ θ0 1: for m = 1, 2,..., S do 2: θm,0 =θm−1 3: Sample jm,t uniformly at random in {(m − 1)M,..., mM − 1} from trajetory 6: θm,t+1 = ΠR θ θm,t + α gx j m,t (θm,t) −gx j m,t (θm−1) + gm(θm−1) 8: end for 9: setθm = θm,t for randomly chosen t ∈ {1, 2, ..., M} 10: end for Output:θS 3.2 In this subsection, we point out the technical errors in the analysis of VRTD in , which thus fails to provide the correct variance reduction performance for VRTD. At the high level, the batch gradient g m (θ m−1) computed at the beginning of each epoch m should necessarily introduce a non-vanishing variance error for a fixed stepsize, because it cannot exactly equal the mean (i.e. population) gradient g(θ m−1). Furthermore, due to the correlation among samples, the gradient estimator in expectation (with regard to the randomness of the sample trajectory) does not equal to the mean gradient, which should further cause a non-vanishing bias error in the convergence bound. Unfortunately, the convergence bound in indicates an exact convergence to the fixed point, which contradicts the aforementioned general understanding. More specifically, if the batch size M = 1 (with properly chosen λ A defined as λ A := 2|λ max (A + A)|), VRTD reduces to the vanilla TD. However, the exact convergence in Theorem 3 in does not agree with that of vanilla TD characterized in the recent studies; , which has variance and bias errors. In Appendix A, we further provide a counter-example to show that one major technical step for characterizing the convergence bound in does not hold. The goal of this paper is to provide a rigorous analysis of VRTD to characterize its variance reduction performance. As aforementioned, the convergence of VRTD consists of two types of errors: the variance error due to inexact estimation of the mean gradient and the bias error due to Markovian sampling. In this section, we first focus on the first type of error and study the convergence of VRTD under i.i.d. sampling. We then study the Markovian case to further analyze the bias. In both cases, we compare the performance of VRTD to that of the vanilla TD described in eq. to demonstrate its advantage. For i.i.d. samples, it is expected that the bias error due to the time correlation among samples does not exist. However, if we directly apply VRTD (Algorithm 2) originally designed for Markovian samples, there would be a bias term due to the correlation between the batch gradient estimate and every inner-loop updates. Thus, we slightly modify Algorithm 2 to Algorithm 1 to avoid the bias error in the convergence analysis with i.i.d. samples. Namely, at each inner-loop iteration, we draw a new sample from the stationary distribution µ π for the update rather than randomly selecting one from the batch of samples drawn at the beginning of the epoch as in Algorithm 2. In this way, the new independent samples avoid the correlation with the batch gradient evaluated at the beginning of the epoch. Hence, Algorithm 1 does not suffer from an extra bias error. To understand the convergence of Algorithm 1 at the high level, we first note that the sample batch gradient cannot estimate the mean gradient g(θ m−1) exactly due to its population nature. Then, we define e m (θ m) = g m (θ m−1) − g(θ m−1) as such a gradient estimation error, our analysis (see Appendix C) shows that after each epoch update, we have where F m,0 denotes the σ-field that includes all the randomness in sampling and updates before the m-th epoch. The first term in the right-hand side of eq. captures the contraction property of Algorithm 1 and the second term corresponds to the variance of the gradient estimation error. It can be seen that due to such an error term, Algorithm 1 is expected to have guaranteed convergence only to a neighborhood of θ *, when applying eq. iteratively. Our further analysis shows that such an error term can still be well controlled (to be small) by choosing an appropriate value for the batch size M, which captures the advantage of the variance reduction. The following theorem precisely characterizes the non-asymptotic convergence of Algorithm 1. Theorem 1. Consider the VRTD algorithm in Algorithm 1. Suppose Assumptions 1-3 hold. Set a constant stepsize α < λ A 8(1+γ) 2 and the batch size M > where (1+γ) 2 (with C 1 < 1 due to the choices of α and M), and We note that the convergence rate in eq. can be written in a simpler form as Theorem 1 shows that Algorithm 1 converges linearly (under a properly chosen constant stepsize) to a neighborhood of the fixed point solution, and the size of the neighborhood (i.e., the error term) has the order of O(α M), which can be made as small as possible by properly increasing the batch size M. This is in contrast to the convergence of the vanilla TD, which suffers from the constant error term with order O(α) for a fixed stepsize. Thus, a small stepsize α is required in vanilla TD to reduce the variance error, which, however, slows down the practical convergence significantly. In contrast, this is not a problem for VRTD, which can attain a high accuracy solution while still maintaining fast convergence at a desirable stepsize. We further note that if we have access to the mean gradient g(θ m−1) in each epoch m, then the error term ||θ m − θ * || 2 2 becomes zero, and Algorithm 1 converges linearly to the exact fixed point solution, as the iteration number m goes to infinity with respect to the conditional number C 1, which is a positive constant and less than 1. This is similar to the conventional convergence of SVRG for strongly convex optimization. However, the proof here is very different. , the convergence proof relies on the relationship between the gradient and the value of the objective function, but there is not such an objective function in the TD learning problem. Thus, the convergence of the parameter θ needs to be developed by exploiting the structure of the Bellman operator. In this section, we study the VRTD algorithm (i.e., Algorithm 2) with Markovian samples, in which samples are generated from one single MDP path. In such a case, we expect that the convergence of VRTD to have both the variance error due to the gradient estimation (similar to the case with i.i.d. samples) and the bias error due to the correlation among samples. To understand this at the high level, we define the bias at each iteration as ). Then our analysis (see Appendix D) shows that after each epoch update, we have The first term on the right-hand side of eq. captures the epochwise contraction property of Algorithm 2. The second term is due to the variance of the gradient estimation, which captures how well the batch gradient g m (θ *) approximates the mean gradient g(θ *) (note that g(θ *) = 0). Such a variance term can be shown to decay to zero as the batch size gets large similarly to the i.i.d. case. The third term captures the bias introduced by the correlation among samples in the m-th epoch. To quantitatively understand this error term, we provide the following lemma that characterizes how the bias error is controlled by the batch size M. Lemma 1. For any m > 0 and any θ ∈ B θ, which is a ball with the radius R θ, we have where the expectation is over the random trajectory, θ is treated as a fixed variable, and 0 < C 0 < ∞ is a constant depending only on the MDP. In Lemma 1, the constant C 0 depends proportionally on the coupling time and the returning time of the underlying Markov chain. Specifically, the coupling time indicates how fast the Markov chain converges to the stationary distribution, and the returning time, which depends on the nature of the stationary distribution, captures the expected time that the Markov chain returns to the same state. Although it is in general difficult to obtain an explicit form for C 0, the value of C 0 is typically small if the Markov chain has a small mixing time and the stationary distribution is less degenerate. Lemma 1 shows that the bias error diminishes as the batch size M increases and the algorithm approaches to the fixed point θ *. To explain why this happens, the definition of ξ m (θ) immediately yields the following bound: The first term on the right-hand-side of eq. can be bounded by the concentration property for the ergodic process as g m (θ) = → g(θ). As M increases, the randomness due to the gradient estimation is essentially averaged out due to the variance reduction step in VRTD, which implicitly eliminates its correlation from samples in the previous epochs. As a comparison, the bias error in vanilla TD has been shown to be bounded by E[ξ n (θ)] = O(α log(1/α));. In order to reduce the bias and achieve a high convergence accuracy, the stepsize α is required to be small, which causes the algorithm to run very slowly. The advantage of VRTD is that the bias can be reduced by choosing a sufficiently large batch size M so that the stepsize can still be kept at a desirable constant to guarantee fast convergence. Theorem 2. Consider the VRTD algorithm in Algorithm 2. Suppose Assumptions 1-3 hold. Set the constant stepsize α < λ A 12(1+γ) 2 and the batch size M > 1 0.5αλ A −6α 2 (1+γ) 2. Then, we have where 0.5αλ A −3α 2 (1+γ) 2 (with C 1 < 1 due to the choices for α and M),. We note that the convergence rate in eq. can be written in a simpler form as Theorem 2 shows that VRTD (i.e., Algorithm 2) with Markovian samples converges to a neighborhood of θ * at a linear rate, and the size of the neighborhood (i.e., the convergence error) decays sublinearly with the batch size M. More specifically, the first term in the right-hand side of eq. captures the linear convergence of the algorithm, the second term corresponds to the sum of the cumulative gradient estimation error and the cumulative bias error. For the fixed stepsize, the total convergence error is dominated by the sum of those two error terms with the order O(1/M). Therefore, the variance reduction in Algorithm 2 reduces both the variance and the bias of the gradient estimator. In this section, we provide numerical to verify our theoretical . We consider an MDP with γ = 0.95 and |S| = 50. Each transition probability are randomly sampled from and the transitions were normalized to one. The expected reward for each transition is also generated randomly in and the reward on each transition was sampled without noise. Each component of the feature matrix Φ ∈ R 50×4 is randomly and uniformly sampled between 0 and 1. The baseline for comparison is the vanilla TD algorithm, which corresponds to the case with M = 1 in our figure. We conduct two experiments to investigate how the batch size M for variance reduction affects the performance of VRTD with i.i.d. and Markovian samples. In the Markovian setting, we sample the data from a MDP trajectory. In the i.i.d. setting, we sample the data independently from the corresponding stationary distribution. In both experiments, we set the constant stepsize to be α = 0.1 and we run the experiments for five different batch sizes: M = 1, 50, 500, 1000, 2000. Our are reported in Figure 1. All the plots report the square error over 1000 independent runs. In each case, the left figure illustrates the convergence process over the number of gradient computations and the right figure shows the convergence errors averaged over the last 10000 iterations for different batch size values. It can be seen that in both i.i.d. and Markovian settings, the averaged error decreases as the batch size increases, which corroborates both Theorem 1 and Theorem 2. We also observe that increased batch size substantially reduces the error without much slowing down the convergence, demonstrating the desired advantage of variance reduction. Moreover, we observe that the error of VRTD with i.i.d samples is smaller than that of VRTD with Markovian samples under all batch size settings, which indicates that the correlation among Markovian samples introduces additional errors. In this paper, we provided the convergence analysis for VRTD with both i.i.d. and Markovian samples. We developed a novel technique to bound the bias of the VRTD gradient estimator. Our demonstrate the advantage of VRTD over vanilla TD on the reduced variance and bias errors by the batch size. We anticipate that such a variance reduction technique and our analysis tools can be further applied to other RL algorithms. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W.. OpenAI Gym. In this section, we use a counter-example to show that one major technical step for characterizing the convergence bound in does not hold. Consider Step 4 in the proof of Theorem 3 in. For the following defined (θ) where Ψ denotes the stationary distribution of the corresponding Markov chain, claimed that the following inequality holds This is not correct. Consider the following counter-example. Let the batch size M = 3 and the dimension of the feature vector be one, i.e., Φ ∈ R |S|×1. Hence, all variables in eq. and eq. are scalars. Since the steps for proving eq. in do not have specific requirements for the transition kernel, eq. should hold for any distribution of v. Thus, suppose v follows the uniform distribution over [−3, 3]. Further assume that in the n-th epoch, the samples of v are given by {1, 2, −3}. Recall that E(·|F n) is the average over the batch samples in the n-th epoch. We have: Substituting the above values into eq. yields which obviously does not hold in general when θ = θ *. Consequently the second statement in Theorem 3 of , which is critically based on the above erroneous steps, does not hold. Hence, the first statement in the same theorem whose proof is based on the second statement cannot hold either. Lemma 2. For any x i = (s i, r i, s i) (i.i.d. sample) or x i = (s i, r i, s i+1) (Markovian sample), we have A xi 2 ≤ 1 + γ and b xi 2 ≤ r max. Proof. First consider the case when samples are i.i.d. Due to the definition of A xi, we have Following similar steps, we can obtain the same upper bounds for the case with Markovian samples. Lemma 3. Let G = (1 + γ)R θ + r max. Consider Algorithm 2. For any m > 0 and 0 ≤ t ≤ M − 1, Proof. First, we bound g xj m,t (θ m,t) 2 as follows. Following the steps similar to the above, we have g xj m,t (θ m−1) 2 ≤ G. Finally for where eq. follows from the last fact g xj m,t (θ m−1) 2 ≤ G. Proof. Recalling the definition of g xi, and applying Lemma 2, we have Lemma 5. Considering Algorithm 2 with Markovian samples. We have Proof. We first derive Following the steps similar to the above, we can derive Recall that B m is the sample batch drawn at the beginning of each m-th epoch and x i,j denotes the sample picked at the j-th iteration in the i-th epoch in Algorithm 1. We denote σ(θ 0) as a trivial σ-field whenθ 0 is a deterministic vector. Let σ(A ∪ B) indicate the smallest σ-field that contains both A and B. Then, we construct a set of σ-fields in the following incremental way. The proof of Theorem 1 proceeds along the following steps. Step 1: Iteration within the m-th epoch For the m-th epoch, we consider the last update (i.e., the M -th iteration in the epoch), and decompose its error into the following form. First, consider the third term in the right-hand side of eq., we have Then, by taking the expectation conditioned on F m,M −1 on both sides of eq., we have where (i) follows from the fact that and (ii) follows from the inequality E[(X − EX) 2 ] ≤ EX 2 and Lemma 2. Then, taking the For all 1 ≤ i ≤ M, we have Then, arranging terms in eq. and using the above fact yield Finally, dividing eq. by [αλ A − 4α 2 (1 + γ) 2 ]M on both sides yields Step 2: Bounding the variance error where eq. follows from Lemma 4. Step 3: Iteration over m epoches First, we substitute eq. into eq. to obtain where we define Taking the expectation of eq. conditioned on F m−1,0 and following the steps similar to those in step 1 to upper bound E θ m−1 − θ * 2 2 F m−1,0, we obtain Then, by following the above steps for (m − 1) times, we have which yields the desirable . We define σ(S) to be the σ-field of all sample trajectories {x 1, x 2, ...} and recall that j m,t is the index of the sample picked at the t-th iteration in the m-th epoch in Algorithm 2. Then we define a set of σ-fields in the following incremental way: We first prove Lemma 1, which is useful for step 4 in the main proof in Theorem 2 provided in Section D.2. Proof. Recall the definition of the bias term: where and To bound eq. and eq., we apply the concentration inequality over Markov chains developed in Dedecker and Gouëzel. We first introduce such a concentration bound as follows. Theorem 3 (Dedecker and Gouëzel, Theorem 2). Let {X n} be an irreducible aperiodic Markov chain which is geometrically ergodic on a space S. Let π be its stationary distribution. There exists a constant C 0 depending on the Markov chain (see the detailed definition of C 0 in Dedecker and Gouëzel) with the following property. Let n ∈ N. Let K(x 0, · · ·, x n−1) be a function of n variables on S n. Then for all t > 0, where µ is the stationary distribution of the Markov chain and 0 ≤ L i < +∞ is a constant that satisfies: Since the MDP in Algorithm 2 satisfies Assumption 3, it satisfies the assumptions in Theorem 3. Then applying Theorem 3 to each W n,(i,j) and V n,i, we have and where 0 < C 0 < ∞ is a constant depending on the MDP parameters. Then, substituting eq. into eq. and eq. into eq. yield and Then we derive the following two bounds: and Finally, substituting eq. and eq. into eq. yields D.2 PROOF OF THEOREM 2 Step 1: Iteration within the m-th inner loop For the m-th inner loop, we consider the last update (i.e., the M -th iteration in the epoch), and decompose its error into the following form. First, consider the third term in the right-hand side of eq.. Then, by taking the expectation conditioned on F m,(M −1) on both sides of eq., we have where (i) follows from the fact that, and (iii) follows from Lemma 2. We further consider the last term in eq.: Then, taking the expectation conditioned on F m,M −1 on both sides of eq. yields where (i) follows by plugging eq. into its preceding step and from the fact that for θ ∈ R d. Then, by applying eq. iteratively, we have Arranging the terms in eq. yields Then, substituting eq. into eq., we obtain Subtracting 0.5λ 2 |F m,0 ] on both sides of eq. yields Then, dividing eq. by [0.5αλ A − 3α 2 (1 + γ) 2 ]M on both sides, we obtain For simplicity, let 0.5λ A −3α(1+γ) 2. Then we rewrite eq.: Step 2 by following similar steps in the previous steps, we obtained By following the above steps for (m − 1) times, we have Then taking the expectation of σ(S) (which contains the randomness of the entire sample trajectory) on both sides of eq. yields where the second term in the right hand side of eq. corresponds to the bias error and the third term corresponds to the variance error. Without loss of generality, we consider the case when j > i as follows: computations required by VRTD (i.e., Algorithm 1) under i.i.d. sampling to attain such an -accuracy solution is at most Proof. Given the values of α and M in the theorem, it can be easily checked that E||θ m − θ * || 2 ≤ for m = log. Then the total number of gradient computations is given by 2mM that yields the desired order given in the theorem. As a comparison, consider the vanilla TD algorithm studied in Proof. Given the values of α and M in the theorem, it can be easily checked that E||θ m − θ * || 2 ≤ for m = log. Then the total number of gradient computations is given by 2mM that yields the desired order given in the theorem. As a comparison, consider the vanilla TD algorithm studied in gradient computations in total to obtain an -accuracy solution. Hence, in the Markovian setting, VRTD outperforms vanilla TD in terms of the total computational complexity by a factor of log 1. To intuitively explain, we first note that the correlation among data samples in the Markovian case also causes a bias error in addition to the variance error. For VRTD, due to the variance reduction scheme, the bias and variance errors are kept at the same level (with respect to the batch size) so that the bias error does not cause order-level increase in the computational complexity for VRTD. However, for vanilla TD, the bias error dominates the variance error, which turns out to require more iterations to attain an -accurate solution, and yields an additional log 1 factor in the total complexity compared to VRTD. The finite-time convergence rate of vanilla TD under i.i.d. and Markovian sampling has been characterized in;. However, these studies did not provide the overall computational complexity, i.e., the total number of gradient computations to achieve an -accuracy solution. This section provides such an analysis based on their convergence for completeness. only when it reaches the goal and 0 otherwise. Each transition probability is randomly sampled from and normalized to one, and each component of the feature matrix Φ ∈ R 16×4 is also randomly sampled from. Given the feature matrix and the transition probability, the ground truth value of θ * can be calculated, which is used to evaluate the error in the experiments. We set the stepsize to be α = 0.1 and run vanilla TD (M = 1) and VRTD with the batch sizes M = 50, 500, 1000, 2000. Note that M = 1 corresponds to the base line vanilla TD. We compute the squared error over 1000 independent runs. The left plot in Figure 2 shows the convergence process over the number of gradient computations and the right plot in Figure 2 shows the convergence error averaged over the last 10000 iterations. It can be observed that VRTD achieves much smaller error than TD, and increasing the batch size for VRTD substantially reduces the error without much slowing down the convergence. Mountain Car is a game in OpenAI Gym, which is driven by an MDP with an infinite state space and a finite action space. At each time step, an agent randomly chooses an action ∈ {push left, push right, no push}. In this problem, the ground truth value of θ * is not known. In order to quantify the performance of VRTD, we apply the error metric known as the norm of the expected TD update given by NEU= E[δφ] 2 2, where δ is the temporal difference;. The state sample is transformed into a feature vector with the dimension 20 using an approximation of a RBF kernel. The agent follows a random policy in our experiment and we initialize θ 0 = 0. At t = 0, the agent starts from the lowest point, receives a reward of −1 at each time step, and returns to the starting point every time it reaches the goal. We set the stepsize to be α = 0.2 and run vanilla TD (M = 1) and VRTD with batch size M = 1000. After every 10000 gradient computations, learning is paused and the NEU is computed by averaging over 1000 test samples. We conduct 1000 independent runs and the are reported by averaging over these runs. Figure 3 shows the convergence process of the NEU versus the number of gradient computations. It can been seen that VRTD achieves smaller NEU than vanilla TD. Upon the request of one reviewer, we provide an additional experiment to compare the performance of VRTD given in Algorithm 2 (under constant stepsize) with the TD algorithm (under a changing stepsize as suggested by the reviewer). We adopt the same setting of Frozen Lake as in Appendix G.1. Let VRTD take a batch size M = 5000 and stepsize α = 0.1. For a fair comparison, we start TD with the same constant stepsize α = 0.1 and then reduce the stepsize by half whenever the error stops decrease. The comparison is reported in Figure 4, where both curves are averaged over 1000 independent runs. The two algorithms are compared in terms of the squared error versus the total number of gradient computations (equivalently, the total number of samples being used). It can be seen that VRTD reaches the required accuracy much faster than TD.

represent the tldr for finding the corresponding scientific article
For example, 'We propose a novel state space time series model with the capability to capture the structure of change points and anomaly points, so that it has a better forecasting performance when there exist change points and anomalies in the time series.' should be close to 'Time series forecasting plays a crucial role in marketing, finance and many other quantitative fields. A large amount of methodologies has been developed on this topic, including ARIMA, Holt–Winters, etc. However, their performance is easily undermined by the existence of change points and anomaly points, two structures commonly observed in real data, but rarely considered in the aforementioned methods. In this paper, we propose a novel state space time series model, with the capability to capture the structure of change points and anomaly points, as well as trend and seasonality. To infer all the hidden variables, we develop a Bayesian framework, which is able to obtain distributions and forecasting intervals for time series forecasting, with provable theoretical properties. For implementation, an iterative algorithm with Markov chain Monte Carlo (MCMC), Kalman filter and Kalman smoothing is proposed. In both synthetic data and real data applications, our methodology yields a better performance in time series forecasting compared with existing methods, along with more accurate change point detection and anomaly detection. Time series forecasting has a rich and luminous history, and is essentially important in most of business operations nowadays. The main aim of time series forecasting is to carefully collect and rigorously study the past observations of a time series to develop an appropriate model which could describe the inherent structure of the series, in order to generate future values. For instance, the internet companies are interested in the number of daily active users (DAU), say, what is DAU after certain period of time, or when will reach their target DAU goal. Time series forecasting is a fruitful research area with many existing methodologies. The most popular and frequently used time series model might be the Autoregressive Integrated Moving Average (ARIMA) BID4 BID31 BID8 BID16. Taking seasonality into consideration, BID4 proposed the Seasonal ARIMA. The Holt-Winters method BID30 ) is also very popular by using exponential smoothing. State space model BID10 BID24 BID5 also attracts much attention, which is a linear function of an underlying Markov process plus additive noise. Exponential Smoothing State Space Model (ETS) decomposes times series into error, trend, seasonal that change over time. Recently, deep learning is applied for time-series trend learning using LSTM BID26, bidirectional dynamic Boltzmann machine BID23 is applied for time-series long-term dependency learning, and coherent probabilistic forecast BID25 ) is proposed for a hierarchy or an aggregation-level comprising a set of time series. Orthogonal to these works, this paper focuses on robust ways of time series forecasting in presence of change points and anomalies. In Internet time series forecasting, Google develops the Bayesian structure time series (BSTS) model BID5 BID24 to capture the trend, seasonality, and similar components of the target series. Recently, Facebook proposes the Prophet approach BID27 based on a decomposable model with interpretable parameters that can be intuitively adjusted by analyst. However, as in the DAU example, some special events like Christmas Holiday or President Election, newly launched apps or features, may cause short period or long-term change of DAU, leading to weird forecasting of those traditional models. The aforementioned special cases are well known as• Anomaly points. The items, events or observations that don't conform to an expected pattern or other items in the dataset, leading to a sudden spike or decrease in the series.• Change points. A market intervention, such as a new product launch or the onset of an advertising (or ad) campaign, may lead to the level change of the original series. Time series forecasting without change/anomaly point detection and adjustment may also lead to bizarre forecasting since these models might learn the abrupt changes in the past. There are literatures on detecting anomaly or change points individually, examples can be found in BID29; BID22; BID3; BID21 BID29. However, the aforementioned change point detection models could not support detection in the presence of seasonality, while the presence of trend/change point is not handled by the anomaly detection models. Most importantly, there is a discrepancy between anomaly/change points detection and adjustment, and commonly used manually adjustment might be a bit arbitrary. Unfortunately, the forecasting gap caused by abnormal and change points, to the best of our knowledge, has not been given full attention and no good solution has been found so far. This paper is strongly motivated by bridging this gap. In this paper, to overcome the limitations of the most (if not all) current models that the anomaly points and change points are not properly considered, we develop a state space time series forecasting model in the Bayesian framework that can simultaneously detect anomaly and change points and perform forecasting. The learned structure information related to anomaly and change points is automatically incorporated into the forecasting process, which naturally enhances the model prediction based on the feedback of state-space model. To solve the ant optimization problem, an iterative algorithm based on Bayesian approximate inference with Markov chain Monte Carlo (MCMC), Kalman filter and Kalman smoothing is proposed. The novel model could explicitly capture the structure of change points, anomaly points, trend and seasonality, as also provide the distributions and forecasting intervals due to Bayesian forecasting framework. Both synthetic and real data sets show the better performance of proposed model, in comparison with existing baseline. Moreover, our proposed model outperforms state-of-the-art models in identifying anomaly and change points. To summarize, our work has the following contributions.• We proposed a robust 1 Bayesian state-space time series forecasting model that is able to explicitly capture the structures of change points and anomalies (which are generally ignored in most current models), and therefore automatically adapt for forecasting by incorporating the prior information of trend, seasonality, as well as change points and anomalies using state space modeling. Due to the enhancement of model description capability, the of model prediction and abnormal and change points detection are mutually improved.• To solve the ant optimization problem, an effective algorithm based on approximate inference using Markov chain Monte Carlo (MCMC) is proposed with theoretical guaranteed forecasting paths.• Our proposed method outperforms the state-of-the-art methods in time series forecasting in presence of change points and anomalies, and detects change points and anomalies with high accuracy and low false discovery rate on both tasks, outperforming popular change point and anomaly detection methods. Our method is flexible to capture the structure of time series under various scenarios with any component combinations of trend, seasonality, change points and anomalies. Therefore our method can be applied in many settings in practice. State space time series model BID13 has been one of the most popular models in time series analysis. It is capable of fitting complicated time series structure including linear trend and seasonality. However, times series observed in real life are almost all prevailed with outliers. Change points, less in frequency but are still widely observed in real time series analysis. Unfortunately, both structures are ignored in the classic state space time series model. In the section, we aim to address this issue by introducing a novel state space time series model. Let y = (y 1, y 2, . . ., y n) be a sequence of time series observations with length n. The ultimate goal is to forecast (y n+1, y n+2, . . .). The accuracy in forecasting lies in a successful decomposition of y into existing components. Apart from the residuals, we assume the time series is composed by trend, seasonality, change points and anomaly points. In a nutshell, we have an additive model with time series = trend + seasonality + change point + anomaly point + residual. As the classical state space model, we have observation equation and transition equations to model y and hidden variables. We use µ = (µ 1, µ 2, . . ., µ n) to model trend, and use γ = (γ 1, γ 2, . . ., γ n) to model seasonality. We use a binary vector z a = (z a 1, z a 2, . . ., z a n) to indicate anomaly points. Then we have Observation equation: DISPLAYFORM0 The deviation between the observation y t and its "mean" µ t + γ t is modeled by t and o t, depending on the value of z a t. If z a t = 1, then y t is an anomaly point; otherwise it is not. Distinguished from the residues = (1, 2, . . ., n), the anomaly is captured by o = (o 1, o 2, . . ., o n) which has relative large magnitude. The hidden state variable µ and γ have intrinsic structures. There are two transition equations, for trend and seasonality separately Transition Equations: Trend: DISPLAYFORM1 DISPLAYFORM2 In Equation, δ = (δ 1, δ 2, . . ., δ n) can be viewed as the "slope" of the trend, measuring how fast the trend changes over time. The change point component is also incorporated in Equation by a binary vector z c = (z is one of the five additive components along with trend, seasonality, anomaly points and residuals. Here we model the change point directly into the trend component. Though differing in formulation, they are equivalent to each other. We choose to model in as in Equation FORMULA1 due to simplicity, and its similarity with the definition of anomaly points in Equation.The seasonality component is presented in Equation. Here S is the length of one season and w = (w 1, w 2, . . ., w n) is the noise for seasonality. The seasonality component is assumed to have almost zero average in each season. The observation equation and transition equations (i.e., Equation) define how y is generated from all the hidden variables including change points and anomaly points. We continue to explore this new model, under a Bayesian framework. Bayesian methods are widely used in many data analysis fields. It is easy to implement and interpret, and it also has the ability to produce posterior distribution. The Bayesian method on state space time series model has been investigated in BID24 BID5. In this section, we also consider Bayesian framework for our novel state space time series model. We assume all the noises are normally distributed DISPLAYFORM0 where σ, σ o, σ u, σ r, σ v, σ w are parameters for standard deviation. As binary vectors, a natural choice is to model anomaly point indicator z a and change point indicator z c to the model them as Bernoulli random variables DISPLAYFORM1 where p a, p c are probabilities for each point to be an anomaly or change point. For simplicity, we denote α t = (µ t, δ t, γ t, γ t−1, . . ., γ t−(S−2) ) to include the main hidden variables (except z a t and z c t) in the transition equations. All the α t are well defined and can be generated from the previous status, except α 1. We denote a 1 to be the parameter for α 1, which can be interpreted as the "mean" for α 1.With Bayesian framework, we are able to represent our model graphically as in FIG2. As shown in FIG2, the only observations are y and all the others are hidden. In this paper, we assume there is no additional information on all the hidden states. If we have some prior information, for example, some points are more likely to be change points, then our model can be easily modified to incorporate such information, by using proper prior. In FIG2, we use squares and circles to classify unknown variables. Despite all being unknown, they actually behave differently according to their own functionality. For those in squares, they behave like turning parameters. Once they are initialized or given, those in circles behaves like latent variables. We call the former "parameters" and the latter "latent variable", as listed in TAB0. The "mean" for the initial trend and seasonality p = (pa, pc)Probabilities for each point to be anomaly or change point σ = (σ, σo, σu, σr, σv, σw) Standard deviationThe discrepancy between these two categories is clearly captured by the joint likelihood function. From FIG2, the joint distribution (i.e., the likelihood function) can be written down explicitly as DISPLAYFORM2 2 ) is the density function for normal distribution with mean x 1 and standard deviation x 2. Here we slightly abuse the notation by using µ 0, δ 0, γ 0, γ −1,..., γ 2−S, which are actually the corresponding coordinates of a 1.As long with other probabilistic graphical models, our model can also be viewed as a generative model. Given the parameters a 1, p, σ, we are able to generate time series. We present the generative procedure as follows. Algorithm 1: Generative Procedure Input: Parameters a 1, σ = (σ, σ o, σ u, σ r, σ v, σ w) and p a, p c, length of time series to generate m Output: Time series y = (y 1, y 2, . . ., y m) 1 Generate the indexes where anomalies or change points occur DISPLAYFORM3 2 Generate all the noises, o, u, r, v, w as independent normal random variables with mean zero and standard deviation σ, σ o, σ u, σ r, σ v, σ w respectively; 3 Generate {α t} m t=1 sequentially by the transition functions in Equation FORMULA1 This section is about inferring unknown variables from y, given the Bayesian setting described in the previous section. The main framework here is to sequentially update each hidden variable by fixing the remaining ones. As stated in the previous section, there are two different categories of unknown variables. Different update schemes need to be used due to the difference in their functionality. For the latent variables, we implement Markov chain Monte Carlo (MCMC) for inference. Particular, we use Gibbs sampler. We will elaborate the details of updates in the following sections. In this section, we focus on updating α assuming all the other hidden variables are given and fixed. The essence of Gibbs sampler is to obtain posterior distribution p a1,p,σ (α|y, z). This can be achieved by a combination of Kalman filter, Kalman smoothing and the so-called "fake-path" trick. We provide some intuitive explanation here and refer the readers to BID10 for detailed implementation. Kalman filter and Kalman smoothing are classic algorithms in signal processing and pattern recolonization for Bayesian inference. It is well related to other algorithms especially message passing algorithm. Kalman filter collects information forwards to obtain E(α t |y 1, y 2, . . ., y t); while Kalman smoothing distribute information backwards to achieve E(α t |y).However, the combination of Kalman filter and Kalman smoothing is not enough, as it only gives the the expectations of marginal distributions {E(α t |y)} n t=1, instead of the joint distribution required for Gibbs sampler. To address this issue, we can use the "fake-path" trick described in Brodersen et al. FORMULA0; BID10. The main idea underlying this trick lies on the fact that the covariance structure of p(α t |y) is not dependent on the means. If we are able to obtain the covariance by some other way, then we can add it up with {E(α t |y)} n t=1 to obtain a sample from p(α|y). This trick involves three steps. Note that all the other hidden variables z, p, σ are given.1. Pick some vectorã 1, and generate a sequence of time seriesỹ from it by Algorithm 1. In this way, we also observeα. 2. Obtain {E(α t |ỹ)} n t=1 fromỹ by Kalman filter and Kalman smoothing. 3. We use {α t − E(α t |ỹ) + E(α t |y)} n t=1 as our sampling from the conditional distribution. In this section, we update z by Gibbs sampler, assuming α, a 1, p, σ are all given and fixed. We need to obtain the conditional distribution p a1,p,σ (z|y, α). Note that in the graphical model described in Section 2, {z are still independent Bernoulli random variables, but possibly with different success probabilities. Thus, we can take the calculation point by point. For example, for the anomaly detection for the t-th point, we have DISPLAYFORM0 . And the prior on z a t is P(z a t = 1) = p a and P(z a t = 0) = p 1. Let p a t = P(z a t = 1|y, α). Directly calculation leads to DISPLAYFORM1 This equality holds for all t = 1, 2,..., n. Similarly for change point detection, let p As mentioned above, all the coordinates in z are still independent Bernoulli random variables conditioned on y, α. Thus, for Gibbs sampler, we can generate z by sampling independently with DISPLAYFORM2 For change point detection here, we have an additional segment control step. After obtaining {z c t} n t=1 as mentioned above, we need to make sure that the change points detected satisfy some additional requirement on the length of segment among two consecutive change points. This issue arises from the ambiguity between the definitions of change point and anomaly points. For example, consider a time series with value. We can view it with two change points, one increases the trend by 1 and the other decreases it by 1. Alternatively, we can also argue the three 1s in this time series are anomalies, though next to each other. One way to address this ambiguity is by defining the minimum length of segment (denoted as). In this toy example, if we set the minimum length to be 4, then they are anomaly points; if we set it to be 3, then we regard them to be change points. But a more complicated criterion is needed than using minimum length as the time series usually own much more complex structure than this toy example. Consider time series FIG0 ) and the minimum time series parameter = 3. It is reasonable to view it with one change point with increment 1, and the two -1s should be regarded as anomalies. As a combination of all these factors, we propose the following segment control method. A default value for the parameter is the length of seasonality, i.e., = S.Algorithm 2: Segment control on change points Input: change point binary vector z c,trend µ, standard deviation for outliers σ r, change point minimum segment Output: change point binary vector z The parameters σ, a 1 and p need both initialization and update. We have different initializations and update schemes for each of them. For all the standard deviations, once we obtain α and z, we update them by taking the empirical standard deviation correspondingly. For σ δ and σ γ, the calculation is straightforward as they only involve δ and γ respectively. For σ, σ o, σ u and σ r, it is a bit more involved due to z. Nevertheless, we can obtain the following update equations for all of them: DISPLAYFORM0 DISPLAYFORM1 Note that in some iterations, when there is no change point or anomaly detected in z, then the updates above for σ o, σ r are not well-defined. In those cases, we simply let them remain the same. To initialize σ, we let them all equal to the standard deviation of y. For a 1, we initialize it by letting its first coordinate to be equal to the average of y 1, y 2,..., y S, and all the remaining coordinates to be equal to 0. Since a 1 can be interpreted as the mean vector of α 1, in this way the trend is initialized to be matched up with average of the first season, and the slope and seasonality are initialized to be equal to 0. We update a 1 by using information of α. We let the first two coordinates (trend and slope) of a 1 to be equal to those of α 1, and we let the remaining coordinates (seasonality) of a 1 to be equal to those of α S+1. The reason why we do not let a 1 to be equal to α 1 entirely is due to the consideration on convergence and robustness. Since we initialize the seasonality part in a 1 as 0, it will remain 0 if we let a 1 equals α 1 entirely (due to the mechanism how we update α 1 as described in Section 4.1. We can avoid such trouble via using α S+1 .For p, we initialize them to be equal to 1/n. If we have additional information on the number of change points or anomaly points, we can initiate them with different values, for example, 0.1/n, or 10/n. We can update p after obtaining z, but we choose not to, also for the sake of robustness. In the early iterations when the algorithm is far from convergence, it is highly possible that z a or z c may turn out to be all 0. If we update p, say, by taking the proportion of change point or anomaly points in z. Then p a or p c might be 0, and it may get stuck in 0 in the remaining iterations. Once we infer all the latent variables α, z and tune all the parameters p, a 1, σ, we are able to forecast the future time series y future . From the graphical model described in Section 3, the future forecasting only involves α n instead of the whole α. Note that we assume that there exists no change point and anomaly point in the future. This is reasonable as in most cases we have no additional information on the future time series. Given α n and σ we can use our predictive procedures (i.e., Algorithm 1) to generate future time series y future. We can further integrate out α n to have the posterior predictive distribution as p σ (y future |y).The forecasting on future time series is not deterministic. There are two sources for the randomness in y future. One comes from the inference of α n (and also σ) from y. Under the Bayesian framework in Section 3, we have a posterior distribution over α n rather than a single point estimation. The second one comes from the forecasting function itself. The forecasting involves intrinsic noise like t, u t, v t and w t. Thus, the predictive density function p σ (y future |y, α n) will lead to different path even with fixed σ and α n. In this way we are able to obtain distribution and predictive interval for forecasting. We also suggest to take the average of multiple forecasting paths, as the posterior mean for the forecasting. The average of multiple forecasting paths (denoted asȳ future), if the number of paths is large enough, always takes the form as a combination of linear trend and seasonality. This can be observed in both our synthesis data (Section 7) and real data analysis (Section 8). This seems to be surprising at the first glance, but makes some sense intuitively. Under our assumption, we have no information on the future, and thus a reliable way to forecast the future is to use the information collected at the end of observed time series, i.e., trend µ n, slope δ n and seasonality structure. Theorem 1 gives mathematical explanation of the linearity ofȳ future, in both mean and standard deviation. Theorem 1. Let N be the number of future time series paths we generate from Algorithm 1). Let m be the number of points we are going to forecast. Denote {y to be the future paths. Defineȳ future = (ȳ n+1,ȳ n+2, . . .,ȳ n+m) to be the average such that DISPLAYFORM0 Then for all j = 1, 2,..., N, we haveȳ n+j as a normal distribution with mean and variance as DISPLAYFORM1 Consequently, for all j = 1, 2,..., m, E[ȳ n+j] is in a linear form with respect to j, and the standard deviation ofȳ n+j also takes a approximately linear form with respect to j. Proof. Recall that α n, σ are given and fixed, and we assume there is no change point or anomaly in the future time series. The Equation leads to δ n+j = δ n + j l=1 v n+l, which implies that DISPLAYFORM2 For the seasonality part, simple linear algebra together with Equation 3 leads to γ n+j = γ n−S+(j mod S) + j l=1 w n+l. Thus, DISPLAYFORM3 Due to the independence and Gaussian distribution of all the noises,ȳ n+j is also normally distributed and its means and variance can be calculated accordingly. Our proposed method can be divided into three parts: initialization, inference, and forecasting. Section 4 and Section 5 provide detailed explanation and reasoning for each of them. We present a whole picture of our proposed methodology in Algorithm 3.Algorithm 3: Proposed Algorithm Input: Observed time series y = (y 1, y 2, . . ., y n), seasonality length S, length of time series for forecasting m, number of predictive paths N, change point minimum segment l Output: Change point detection z c, anomaly points z a, forecasting y future = (y n+1, y n+1, . . ., y n+m) and its distribution or predictive intervals Part I: Initialization; 1 Initialize σ, σ o, σ u, σ r, σ v, σ w all with the empirical standard deviation of y; 2 Initialize a 1 such that its first coordinate equals to the average of (y 1, y 2, . . ., y S) and all the remaining S coordinates with 0; 3 Initialize p a and p c by 1/n. Then generate z a and z c as independent Bernoulli random variables with success probability p a and p c respectively; DISPLAYFORM0 Infer α by Kalman filter, Kalman smoothing and "fake-path" trick described in Section 4.1; Update z a and z c by sampling from DISPLAYFORM0, where the success probability {p Update σ by Equation FORMULA11 to; Update a 1 such that its first two coordinates equal to the those of α 1 and the remaining (S − 1) coordinates equals to those of α S+1; Calculate the likelihood function L a1,p,σ (y, α, z) given in Equation; end Part III: Forecasting; 10 With a n and σ, use the generate procedure in Algorithm 1 to generate future time series y future with length m. Repeat the generative procedure to obtain multiple future paths y DISPLAYFORM0 future,..., yfuture; 11 Combine all the predictive paths give the distribution for the future time series forecasting. If needed, calculate the point-wise quantile to obtain predictive intervals. Use the point-wise average as our final forecasting . It is worth mentioning that our proposed methodology is downward compatible with many simpler state space time series models. By letting p c = 0, we assume there is no change point in the time series. By letting p a = 0, we assume there is no anomaly point in the time series. If both p c and p a are set to be 0, then our model is reduced to the classic state space time series model. Also, the seasonality and slope can be removed from our model, if we know there exists no such structure in the data. In this section, we study the synthetic data generated from our model. We let S = 7 and provide values for σ and a 1. The change points and anomaly points are randomly generated. We use our generative procedure (Algorithm 1) to generate time series with total length 500 by fixed parameters. The first 350 points will be used as training set and the remaining 150 points will be used to evaluate the performance of forecasting. When generating, we let the time series have weekly seasonality with S = 7. For σ we have σ = 0.1, σ u = 0.1, σ v = 0.0004, σ w = 0.01, σ r = 1, σ o = 4. For α 1 we have value for µ as 20, value for δ as 0, and value for seasonality as (1, 2, 4, −1, −3, −2)/10. For p we have p c = 4/350 and p a = 10/350. Despite that, to make sure that at least one change point is in existence, we force z c 330 = 1 and r 330 = 2. That is, for each time series we generate, its 330th point is a change point with the mean shifted up by 3. Also to be consistence with our assumption, we force z c i = z a i = 0, ∀351 ≤ i ≤ 500 so there exists no change point or anomaly point in the testing part. The top panel of FIG9 shows one example of synthesis data. The blue line marks the separation between training and testing set. The blue dashed line indicates the locations for the change point, while the yellow dots indicate the positions of anomaly points. Also see FIG9 for illustration on the returned by implementing our proposed algorithm on the same dataset. The red line gives the fitting in the first 350 points and forecasting in the last 150 points. The change points detected are marked with vertical red dotted line, and the anomaly detected are flagged with purple squares. FIG9 shows that on this dataset, our proposed algorithm yields perfect detection on both change points and anomaly points. In FIG9, the gray part indicates the 90% predictive interval for forecasting. We run our generative model 100 times to produce 100 different time series, and implement multiply methods on each of them, and aggregate the together for comparison. We include the following methodologies. For time series forecasting, we compare our method against Bayesian Structural Time Series (BSTS) BID24 BID5 ), Seasonal Decomposition of Time Series by Loess (STL) BID7 ), Seasonal ARIMA BID4, Holt-Winters , Exponential Smoothing State Space Model (ETS) ), and the Prophet R package by BID27. We evaluate the performances by mean absolute percentage error (MAPE), mean square error (MSE) and mean absolute error (MAE) on forecasting set. The mathematical definition of these three criterion is given as follows. Let x 1, x 2,..., x n be the true value andx 1,x 2,...,x n be the estimation or predictive values. Then we have DISPLAYFORM0 The comparison of our proposed algorithm and the aforementioned algorithms are included below in TAB1. As we mentioned in Section 6, our algorithm is downward compatible with the cases ignoring the existence of change point or anomaly, by setting p c = 0 or p a = 0. We also run proposed algorithm on the synthetic data with p c = 0 (no change point), or p a = 0 (no anomaly point), or p c = p a = 0 (no change and anomaly point), for the purpose of numeric comparison. From TAB1 it turns out that our proposed algorithm achieves the best performance compared to other existing methods. Our proposed algorithm also performs better compared with the cases ignoring change point or anomaly point. This is a convincing evidence on the importance of incorporating both change point structure and anomaly point structure when modeling, for time series forecasting. We also compare our proposed method with other existing change point detection methods and anomaly detection algorithm with respect to the performance of detections. We evaluate the performance by two criterions: True Positive Rate (TPR) and False Positive (FP). TPR measures the percentage of change points or anomalies to be correctly detected. FP count the number of points wrongly detected as change points or anomaly points. The mathematical definitions of TPR and FP are as follows. Let (z 1, z 2, . . ., z n) be the true binary vector for change points or anomalies, and (ẑ 1,ẑ 2, . . .,ẑ n) are the estimated ones. Then DISPLAYFORM1 From the definition, we can see high TPR and low FP means the algorithm has better performance in detection. The comparison on change point detection is shown in TAB2. We compare our against three popular change point detection methods: Bayesian Change Point (BCP) BID3, Change-Point (CP) BID21 and Breakout (twitter, 2017). From TAB2 our proposed method outperforms the most of the others by both TPR and FP. We have smaller TPR compared to CP, but we are better in FP. In TAB3, we also compare the performance of our algorithm on anomaly detection with three existing common anomaly detection methods: the AnomalyDetection package by BID29, RAD by BID22 and Tsoutlier by BID6. The comparison is listed in TAB3. We can see our method also outperforms most of the others with respect to anomaly detection, by both TPR and FP. RAD has slightly better TPR but its FP is much worse compared with ours. In this section, we implement our proposed method on real-world datasets. We also compare its performance against other existing time series forecasting methodologies. We consider two datasets, one is a public data called Well-log dataset, and the other is an unpublished internet traffic dataset. The bottom panels of FIG10 and FIG11 give the of our proposed algorithms. The blue line separates the training set and testing set. We use red line to show our fitting and forecasting , vertical red dashed line to indicate change points and purple dots to indicate anomaly points. The gray part shows 90% predication interval. This dataset BID11 BID20 was collected when drilling a well. It measures the nuclear magnetic response, which provides geophysical information to analyze the structure of rock surrounding the well. This dataset is public and available online 2. It has 4050 points in total. We split it such that the first 3000 points are used as training set and last 1000 points are used to evaluate the forecasting performance. From FIG10, it is obvious that there exists no seasonality or slope structure in the dataset. This motivates us not to include these two components in our model. We implement our proposed algorithm without seasonality and slope, and compare the forecasting performance with other methods in TAB4. Our method outperforms BSTS, ARIMA, ETS and Prophet. However in TAB4 the performance can be slightly improved if we ignore the existence of anomaly points by letting p a = 0. This may be caused by model mis-specification as the data may not generated in a way not entirely captured by our model. Nevertheless, the performances of our method considering anomaly points or not, are comparable to each other. In this dataset there is no ground-truth of change point and anomaly point on their locations or even existence. However, from bottom panel of FIG10, there are some obvious changes in the sequence and they all successfully captured by our algorithm. Our second real data is an Internet traffic data acquired from a major Tech company (see FIG11). It is a daily traffic data, with seasonality S = 7. We use the first 800 observations as training set and evaluate the performance of forecasting on the remaining 265 points. The bottom panel of FIG11 show the from implementing our algorithm. We also do the comparison of forecasting performance of our proposed algorithm together with other existing methods, shown in TAB5. We can also see that our algorithm outperforms all the other algorithms with respect to MAPE, MSE and MAE. From FIG11 our proposed algorithm identifies one change point (the 576th point, indicated by the vertical red dashed line), which can be confirmed that this is exactly the only one change point existing in this time series caused by the change of counting methods, by some external information. Thus, we give the perfect change point detection in this Internet traffic data. For this Internet traffic dataset, since we have ground-truth for change point, we can compare the performance of change point detection of different methodologies. BCP returns posterior distribution, which peaks in the the 576th point with posterior probability value 0.5. And it also returns with many other points with posterior probability value around 0.1. CP returns 4 change points, where the 576th point (the only true one) is one of them. Breakout returns 8 change points without including the 576th point. To sum up, our proposed method achieves the best change point detection in this real dataset. Compared to the aforementioned models, our work differs in Bayesian modeling which samples posterior to estimate hidden components given the independent Bernoulli priors of changing point and anomalies.'

This paper provides a rigorous study of the variance reduced TD learning and characterizes its advantage over vanilla TD learning

Represent the short summary for finding the corresponding scientific article (long)

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.