Question:
represent the scientific article to find a short tldr

------

Fewshots:
'Convolutional architectures have recently been shown to be competitive on many sequence modelling tasks when compared to the de-facto standard of recurrent neural networks (RNNs) while providing computational and modelling advantages due to inherent parallelism. However, currently, there remains a performance gap to more expressive stochastic RNN variants, especially those with several layers of dependent random variables. In this work, we propose stochastic temporal convolutional networks (STCNs), a novel architecture that combines the computational advantages of temporal convolutional networks (TCN) with the representational power and robustness of stochastic latent spaces. In particular, we propose a hierarchy of stochastic latent variables that captures temporal dependencies at different time-scales. The architecture is modular and flexible due to the decoupling of the deterministic and stochastic layers. We show that the proposed architecture achieves state of the art log-likelihoods across several tasks. Finally, the model is capable of predicting high-quality synthetic samples over a long-range temporal horizon in modelling of handwritten text. Generative modeling of sequence data requires capturing long-term dependencies and learning of correlations between output variables at the same time-step. Recurrent neural networks (RNNs) and its variants have been very successful in a vast number of problem domains which rely on sequential data. Recent work in audio synthesis, language modeling and machine translation tasks BID8 BID9 BID13 has demonstrated that temporal convolutional networks (TCNs) can also achieve at least competitive performance without relying on recurrence, and hence reducing the computational cost for training. Both RNNs and TCNs model the joint probability distribution over sequences by decomposing the distribution over discrete time-steps. In other words, such models are trained to predict the next step, given all previous time-steps. RNNs are able to model long-term dependencies by propagating information through their deterministic hidden state, acting as an internal memory. In contrast, TCNs leverage large receptive fields by stacking many dilated convolutions, allowing them to model even longer time scales up to the entire sequence length. It is noteworthy that there is no explicit temporal dependency between the model outputs and hence the computations can be performed in parallel. The TCN architecture also introduces a temporal hierarchy: the upper layers have access to longer input sub-sequences and learn representations at a larger time scale. The local information from the lower layers is propagated through the hierarchy by means of residual and skip connections BID2.However, while TCN architectures have been shown to perform similar or better than standard recurrent architectures on particular tasks BID2, there currently remains a performance gap to more recent stochastic RNN variants BID3 BID7 BID11 BID12 BID14 BID25. Following a similar approach to stochastic RNNs, BID21 present a significant improvement in the log-likelihood when a TCN model is coupled with latent variables, albeit at the cost of limited receptive field size. The computational graph of generative (left) and inference (right) models of STCN. The approximate posterior q is conditioned on dt and is updated by the prior p which is conditioned on the TCN representations of the previous time-step dt−1. The random latent variables at the upper layers have access to a long history while lower layers receive inputs from more recent time steps. In this work we propose a new approach for augmenting TCNs with random latent variables, that decouples deterministic and stochastic structures yet leverages the increased modeling capacity efficiently. Motivated by the simplicity and computational advantages of TCNs and the robustness and performance of stochastic RNNs, we introduce stochastic temporal convolutional networks (STCN) by incorporating a hierarchy of stochastic latent variables into TCNs which enables learning of representations at many timescales. However, due to the absence of an internal state in TCNs, introducing latent random variables analogously to stochastic RNNs is not feasible. Furthermore, defining conditional random variables across time-steps would in breaking the parallelism of TCNs and is hence undesirable. In STCN the latent random variables are arranged in correspondence to the temporal hierarchy of the TCN blocks, effectively distributing them over the various timescales (see FIG0 . Crucially, our hierarchical latent structure is designed to be a modular add-on for any temporal convolutional network architecture. Separating the deterministic and stochastic layers allows us to build STCNs without requiring modifications to the base TCN architecture, and hence retains the scalability of TCNs with respect to the receptive field. This conditioning of the latent random variables via different timescales is especially effective in the case of TCNs. We show this experimentally by replacing the TCN layers with stacked LSTM cells, leading to reduced performance compared to STCN.We propose two different inference networks. In the canonical configuration, samples from each latent variable are passed down from layer to layer and only one sample from the lowest layer is used to condition the prediction of the output. In the second configuration, called STCN-dense, we take inspiration from recent CNN architectures BID18 and utilize samples from all latent random variables via concatenation before computing the final prediction. Our contributions can thus be summarized as: 1) We present a modular and scalable approach to augment temporal convolutional network models with effective stochastic latent variables. 2) We empirically show that the STCN-dense design prevents the model from ignoring latent variables in the upper layers BID32. 3) We achieve state-of-the-art log-likelihood performance, measured by ELBO, on the IAM-OnDB, Deepwriting, TIMIT and the Blizzard datasets. 4) Finally we show that the quality of the synthetic samples matches the significant quantitative improvements. Auto-regressive models such as RNNs and TCNs factorize the joint probability of a variable-length sequence x = {x 1, . . ., x T} as a product of conditionals as follows: DISPLAYFORM0 where the joint distribution is parametrized by θ. The prediction at each time-step is conditioned on all previous observations. The observation model is frequently chosen to be a Gaussian or Gaussian mixture model (GMM) for real-valued data, and a categorical distribution for discrete-valued data. In TCNs the joint probabilities in Eq. are parametrized by a stack of convolutional layers. Causal convolutions are the central building block of such models and are designed to be asymmetric such that the model has no access to future information. In order to produce outputs of the same size as the input, zero-padding is applied at every layer. In the absence of a state transition function, a large receptive field is crucial in capturing long-range dependencies. To avoid the need for vast numbers of causal convolution layers, typically dilated convolutions are used. Exponentially increasing the dilation factor in an exponential growth of the receptive field size with depth BID31 BID2. In this work, without loss of generality, we use the building blocks of Wavenet as gated activation units have been reported to perform better. A deterministic TCN representation d l t at time-step t and layer l summarizes the input sequence x 1:t: DISPLAYFORM0 where the filter width is 2 and j denotes the dilation step. In our work, the stochastic variables z l, l = 1... L are conditioned on TCN representations d l that are constructed by stacking K Wavenet blocks over the previous d l−1 (for details see FIG6 in Appendix). VAEs BID20 BID24 introduce a latent random variable z to learn the variations in the observed non-sequential data where the generation of the sample x is conditioned on the latent variable z. The joint probability distribution is defined as: DISPLAYFORM0 and parametrized by θ. Optimizing the marginal likelihood is intractable due to the non-linear mappings between z and x and the integration over z. Instead the VAE framework introduces an approximate posterior q φ (z|x) and optimizes a lower-bound on the marginal likelihood: DISPLAYFORM1 where KL denotes the Kullback-Leibler divergence. Typically the prior p θ (z) and the approximate q φ (z|x) are chosen to be in simple parametric form, such as a Gaussian distribution with diagonal covariance, which allows for an analytical calculation of the KL-term in Eq.. An RNN captures temporal dependencies by recursively processing each input, while updating an internal state h t at each time-step via its state-transition function: DISPLAYFORM0 where f (h) is a deterministic transition function such as LSTM BID17 or GRU BID6 cells. The computation has to be sequential because h t depends on h t−1.The VAE framework has been extended for sequential data, where a latent variable z t augments the RNN state h t at each sequence step. The joint distribution p θ (x, z) is modeled via an auto-regressive model which in the following factorization: DISPLAYFORM1 In contrast to the fixed prior of VAEs, N (0, I), sequential variants define prior distributions conditioned on the RNN hidden state h and implicitly on the input sequence x BID7., and the inference model (right), which is shared by both variants. Diamonds represent the outputs of deterministic dilated convolution blocks where the dependence of dt on the past inputs is not shown for clarity (see Eq. FORMULA1). xt and zt are observable inputs and latent random variables, respectively. The generative task is to predict the next step in the sequence, given all past steps. Note that in the STCN-dense variant the next step is conditioned on all latent variables z l t for l = 1... L. The mechanics of STCNs are related to those of VRNNs and LVAEs. Intuitively, the RNN state h t is replaced by temporally independent TCN layers d l t. In the absence of an internal state, we define hierarchical latent variables z l t that are conditioned vertically, i.e., in the same time-step, but independent horizontally, i.e., across time-steps. We follow a similar approach to LVAEs in defining the hierarchy in a top-down fashion and in how we estimate the approximate posterior. The inference network first computes the approximate likelihood, and then this estimate is corrected by the prior, ing in the approximate posterior. The TCN layers d are shared between the inference and generator networks, analogous to VRNNs BID7. To preserve the parallelism of TCNs, we do not introduce an explicit dependency between different time-steps. However, we suggest that conditioning a latent variable z l−1 t on the preceding variable z l t implicitly introduces temporal dependencies. Importantly, the random latent variables in the upper layer have access to a larger receptive field due to its deterministic input d l t−1, whereas latent random variables in lower layers are updated with different, more local information. However, the latent variable z l−1 t may receive longer-range information from z l t. The generative and inference models are jointly trained by optimizing a step-wise variational lower bound on the log-likelihood BID20 BID24. In the following sections we describe these components and build up the lower-bound for a single time-step t. Each sequence step x t is generated from a set of latent variables z t, split into layers as follows: DISPLAYFORM0 where DISPLAYFORM1 Here the prior is modeled by a Gaussian distribution with diagonal covariance, as is common in the VAE framework. The subscript p denotes items of the generative distribution. For the inference distribution we use the subscript q. The distributions are parameterized by a neural network f We propose two variants of the observation model. In the non-sequential scenario, the observations are defined to be conditioned on only the last latent variable in the hierarchy, i.e., p θ (x t |z 1 t), following Sønderby et al. FORMULA0; BID16 and BID24 our STCN variant uses the same observation model, allowing for an efficient optimization. However, latent units are likely to become inactive during training in this configuration BID5 BID4 BID32 ing in a loss of representational power. The latent variables at different layers are conditioned on different contexts due to the inputs d l t. Hence, the latent variables are expected to capture complementary aspects of the temporal context. To propagate the information all the way to the final prediction and to ensure that gradients flow through all layers, we take inspiration from BID18 and directly condition the output probability on samples from all latent variables. We call this variant of our architecture STCN-dense. The final predictions are then computed by the respective observation functions: DISPLAYFORM0 where f (o) corresponds to the output layer constructed by stacking 1D convolutions or Wavenet blocks depending on the dataset. In the original VAE framework the inference model is defined as a bottom-up process, where the latent variables are conditioned on the stochastic layer below. Furthermore, the parameterization of the prior and approximate posterior distributions are computed separately BID5 BID24. In contrast, propose a top-down dependency structure shared across the generative and inference models. From a probabilistic point of view, the approximate Gaussian likelihood, computed bottom-up by the inference model, is combined with the Gaussian prior, computed top-down from the generative model. We follow a similar procedure in computing the approximate posterior. First, the parameters of the approximate likelihood are computed for each stochastic layer l: DISPLAYFORM0 followed by the downward pass, recursively computing the prior and approximate posterior by precision-weighted addition: DISPLAYFORM1 Finally, the approximate posterior has the same decomposition as the prior (see Eq. FORMULA6): DISPLAYFORM2 DISPLAYFORM3 Note that the inference and generative network share the parameters of dilated convolutions Conv (l). The variational lower-bound on the log-likelihood at time-step t can be defined as follows: DISPLAYFORM0 Using the decompositions from Eq. FORMULA6 and FORMULA0, the Kullback-Leibler divergence term becomes: DISPLAYFORM1 The KL term is the same for the STCN and STCN-dense variants. The reconstruction term L Recon t, however, is different. In STCN we only use samples from the lowest layer of the hierarchy, whereas in STCN-dense we use all latent samples in the observation model: DISPLAYFORM2 DISPLAYFORM3 In the dense variant, samples drawn from the latent variables z l t are carried over the dense connections. Similar to, the expectation over z One alternative option to use the latent samples could be to sum individual samples before feeding them into the observation model, i.e., sum([z . We empirically found that this does not work well in STCN-dense. Instead, we concatenate all samples [z DISPLAYFORM4 DISPLAYFORM5 analogously to DenseNet BID18 and BID19 . We evaluate the proposed variants STCN and STCN-dense both quantitatively and qualitatively on modeling of digital handwritten text and speech. We compare with vanilla TCNs, RNNs, VRNNs and state-of-the art models on the corresponding tasks. In our experiments we use two variants of the Wavenet model: the original model proposed in and a variant that we augment with skip connections analogously to STCN-dense. This additional baseline evaluates the benefit of learning multi-scale representations in the deterministic setting. Details of the experimental setup are provided in the Appendix. Our code is available at https://ait.ethz.ch/projects/2019/stcn/.Handwritten text: The IAM-OnDB and Deepwriting datasets consist of digital handwriting sequences where each time-step contains real-valued (x, y) pen coordinates and a binary pen-up event. The IAM-OnDB data is split and pre-processed as done in BID7. BID1 extend this dataset with additional samples and better pre-processing. BID7 26643 7413 1358 528 * VRNN (Normal) BID7 ≈ 30235 ≈ 9516 ≈ 1354 ≥ 495 * VRNN (GMM) BID7 ≈ 29604 ≈ 9392 ≈ 1384 ≥ 673 * SRNN (Normal) BID12 ≥ 60550 ≥ 11991 n/a n/a Z-forcing (Normal) BID14 ≥ 70469 ≥ 15430 n/a n/a Var. Bi-LSTM (Normal) BID25 ≥ 73976 ≥ 17319 n/a n/a SWaveNet (Normal) BID21 ≥ 72463 ≥ 15708 DISPLAYFORM0 ≥ 77438 ≥ 17670 n/a n/a the STCN-dense version. The same relative ordering is maintained on the Deepwriting dataset, indicating that the proposed architecture is robust across datasets. Fig. 3 compares generated handwriting samples. While all models produce consistent style, our model generates more natural looking samples. Note that the spacing between words is clearly visible and most of the letters are distinguishable. Speech modeling: TIMIT and Blizzard are standard benchmark dataset in speech modeling. The models are trained and tested on 200 dimensional real-valued amplitudes. We apply the same pre-processing as BID7. For this task we introduce STCN-dense-large, with increased model capacity. Here we use 512 instead of 256 convolution filters. Note that the total number of model parameters is comparable to SWaveNet and other SOA models. On TIMIT, STCN-dense TAB0 significantly outperforms the vanilla TCN and RNN, and stochastic models. On the Blizzard dataset, our model is marginally better than the Variational Bi-LSTM. Note that the inference models of SRNN BID12, Z-forcing BID14, and Variational Bi-LSTM BID25 receive future information by using backward RNN cells. Similarly, SWaveNet BID21 applies causal convolutions in the backward direction. Hence, the latent variable can be expected to model future dynamics of the sequence. In contrast, our models have only access to information up to the current time-step. These indicate that the STCN variants perform very well on the speech modeling task. Latent Space Analysis: BID32 observe that in hierarchical latent variable models the upper layers have a tendency to become inactive, indicated by a low KL loss BID10. TAB2 shows the KL loss per latent variable and the corresponding log-likelihood measured by ELBO in our models. Across the datasets it can be observed that our models make use of many of the latent variables which may explain the strong performance across tasks in terms of log-likelihoods. Note that STCN uses a standard hierarchical structure. However, individual latent variables have different information context due to the corresponding TCN block's receptive field. This observation suggests that the proposed combination of TCNs and stochastic variables is indeed effective. Furthermore, in STCN we see a similar utilization pattern of the z variables across tasks, whereas STCN-dense may have more flexibility in modeling the temporal dependencies within the data due to its dense connections to the output layer. Replacing TCN with RNN: To better understand potential symergies between dilated CNNs and the proposed latent variable hierarchy, we perform an ablation study, isolating the effect of TCNs and the latent space. To this end the deterministic TCN blocks are replaced with LSTM cells by keeping the latent structure intact. We dub this condition LadderRNN. We use the TIMIT and IAM-OnDB datasets for evaluation. TAB3 summarizes performance measured by the ELBO.The most direct translation of the the STCN architecture into an RNN counterpart has 25 stacked LSTM cells with 256 units each. Similar to STCN, we use 5 stochastic layers (see Appendix 7.1). Note that stacking this many LSTM cells is unusual and ed in instabilities during training. Hence, the performance is similar to vanilla RNNs. The second LadderRNN configuration uses 5 stacked LSTM cells with 512 units and a one-to-one mapping with the stochastic layers. On the TIMIT dataset, all LadderRNN configurations show a significant improvement. We also observe a pattern of improvement with densely connected latent variables. This experiments shows that the proposed modular latent variable design does allow for the usage of different building blocks. Even when attached to LSTM cells, it boosts the log-likelihood performance (see 5x512-LadderRNN), in particular when used with dense connections. However, the empirical suggest that the densely connected latent hierarchy interacts particularly well with dilated CNNs. We suggest this is due to the hierarchical nature on both sides of the architecture. On both datasets STCN models achieved the best performance and significantly improve with dense connections. This supports our contribution of a latent variable hierarchy, which models different aspects of information from the input time-series. DISPLAYFORM1 5 RELATED WORK BID24 propose Deep Latent Gaussian Models (DLGM) and propose the Ladder Variational Autoencoder (LVAE). In both models the latent variables are hierarchically defined and conditioned on the preceding stochastic layer. LVAEs improve upon DLGMs via implementation of a top-down hierarchy both in the generative and inference model. The approximate posterior is computed via a precisionweighted update of the approximate likelihood (i.e., the inference model) and prior (i.e., the generative model). Similarly, the PixelVAE BID16 incorporates a hierarchical latent space decomposition and uses an autoregressive decoder. BID32 show under mild conditions that straightforward stacking of latent variables (as is done e.g. in LVAE and PixelVAE) can be ineffective, because the latent variables that are not directly conditioned on the observation variable become inactive. Due to the nature of the sequential problem domain, our approach differs in the crucial aspects that STCNs use dynamic, i.e., conditional, priors BID7 at every level. Moreover, the hierarchy is not only implicitly defined by the network architecture but also explicitly defined by the information content, i.e., receptive field size. Dieng et al. FORMULA0 both theoretically and empirically show that using skip connections from the latent variable to every layer of the decoder increases mutual information between the latent and observation variables. Similar to BID10 in STCN-dense, we introduce skip connections from all latent variables to the output. In STCN the model is expected to encode and propagate the information through its hierarchy. BID30 suggest using autoregressive TCN decoders to remedy the posterior collapse problem observed in language modeling with LSTM decoders BID4. BID29 and BID9 use TCN decoders conditioned on discrete latent variables to model audio signals. Stochastic RNN architectures mostly vary in the way they employ the latent variable and parametrize the approximate posterior for variational inference. BID7 and BID3 use the latent random variable to capture high-level information causing the variability observed in sequential data. Particularly BID7 shows that using a conditional prior rather than a standard Gaussian distribution is very effective in sequence modeling. In BID12 BID14 BID25 ), the inference model, i.e., the approximate posterior, receives both the past and future summaries of the sequence from the hidden states of forward and backward RNN cells. The KL-divergence term in the objective enforces the model to learn predictive latent variables in order to capture the future states of the sequence. BID21's SWaveNet is most closely related to ours. SWaveNet also introduces latent variables into TCNs. However, in SWaveNet the deterministic and stochastic units are coupled which may prevent stacking of larger numbers of TCN blocks. Since the number of stacked dilated convolutions determines the receptive field size, this directly correlates with the model capacity. For example, the performance of SWaveNet on the IAM-OnDB dataset degrades after stacking more than 3 stochastic layers BID21, limiting the model to a small receptive field. In contrast, we aim to preserve the flexibility of stacking dilated convolutions in the base TCN. In STCNs, the deterministic TCN units do not have any dependency on the stochastic variables (see FIG0) and the ratio of stochastic to deterministic units can be adjusted, depending on the task. In this paper we proposed STCNs, a novel auto-regressive model, combining the computational benefits of convolutional architectures and expressiveness of hierarchical stochastic latent spaces. We have shown the effectivness of the approach across several sequence modelling tasks and datasets. The proposed models are trained via optimization of the ELBO objective. Tighter lower bounds such as IWAE BID5 or FIVO may further improve modeling performance. We leave this for future work. The network architecture of the proposed model is illustrated in FIG6. We make only a small modification to the vanilla Wavenet architecture. Instead of using skip connections from Wavenet blocks, we only use the latent sample zt in order to make a prediction of xt. In STCN-dense configuration, zt is the concatenation of all latent variables in the hierarchy, i.e., zt = [z Output layer f (o): For the IAM-OnDB and Deepwriting datasets we use 1D convolutions with ReLU nonlinearity. We stack 5 of these layers with 256 filters and filter size 1. DISPLAYFORM0 For TIMIT and Blizzard datasets Wavenet blocks in the output layer perform significantly better. We stack 5 Wavenet blocks with dilation size 1. For each convolution operation in the block we use 256 filters. The filter size of the dilated convolution is set to 2. The STCN-dense-large model is constructed by using 512 filters instead of 256. l t: The number of Wavenet blocks is usually determined by the desired receptive field size.• For the handwriting datasets K = 6 and L = 5. In total we have 30 Wavenet blocks where each convolution operation has 256 filters with size 2.• For speech datasets K = 5 and L = 5. In total we have 25 Wavenet blocks where each convolution operation has 256 filters with size 2. The large model configuration uses 512 filters. p and f q: The number of stochastic layers per task is given by L. We used dimensional latent variables for the handwriting tasks. It is for speech datasets. Note that the first entry of the list corresponds to z 1.The mean and sigma parameters of the Normal distributions modeling the latent variables are calculated by the f (l) p and f (l) q networks. We stack 2 1D convolutions with ReLU nonlinearity and filter size 1. The number of filters are the same as the number of Wavenet block filters for the corresponding task. Finally, we clamped the latent sigma predictions between 0.001 and 5. In all STCN experiments we applied KL annealing. In all tasks, the weight of the KL term is initialized with 0 and increased by 1 × e −4 at every step until it reaches 1.The batch size was 20 for all datasets except for Blizzard where it was 128.We use the ADAM optimizer with its default parameters and exponentially decay the learning rate. For the handwriting datasets the learning rate was initialized with 5 × e −4 and followed a decay rate of 0.94 over 1000 decay steps. On the speech datasets it was initialized with 1 × e −3 and decayed with a rate of 0.98. We applied early stopping by measuring the ELBO performance on the validation splits. We implement STCN models in Tensorflow BID0. Our code and models achieving the SOA are available at https://ait.ethz.ch/projects/2019/stcn/. Here we provide the extended table with Normal observation model entries for available models. Table 4: Average log-likelihood per sequence on TIMIT, Blizzard, IAM-OnDB and Deepwriting datasets. (Normal) and (GMM) stand for unimodal Gaussian or multi-modal Gaussian Mixture Model (GMM) as the observation model BID15 BID7. Asterisks * indicate that we used our re-implementation only for the Deepwriting dataset.' == 'We combine the computational advantages of temporal convolutional architectures with the expressiveness of stochastic latent variables.' != 'We investigate a variant of variational autoencoders where there is a superstructure of discrete latent variables on top of the latent features.'

Question: Time series forecasting plays a crucial role in marketing, finance and many other quantitative fields. A large amount of methodologies has been developed on this topic, including ARIMA, Holt–Winters, etc. However, their performance is easily undermined by the existence of change points and anomaly points, two structures commonly observed in real data, but rarely considered in the aforementioned methods. In this paper, we propose a novel state space time series model, with the capability to capture the structure of change points and anomaly points, as well as trend and seasonality. To infer all the hidden variables, we develop a Bayesian framework, which is able to obtain distributions and forecasting intervals for time series forecasting, with provable theoretical properties. For implementation, an iterative algorithm with Markov chain Monte Carlo (MCMC), Kalman filter and Kalman smoothing is proposed. In both synthetic data and real data applications, our methodology yields a better performance in time series forecasting compared with existing methods, along with more accurate change point detection and anomaly detection. Time series forecasting has a rich and luminous history, and is essentially important in most of business operations nowadays. The main aim of time series forecasting is to carefully collect and rigorously study the past observations of a time series to develop an appropriate model which could describe the inherent structure of the series, in order to generate future values. For instance, the internet companies are interested in the number of daily active users (DAU), say, what is DAU after certain period of time, or when will reach their target DAU goal. Time series forecasting is a fruitful research area with many existing methodologies. The most popular and frequently used time series model might be the Autoregressive Integrated Moving Average (ARIMA) BID4 BID31 BID8 BID16. Taking seasonality into consideration, BID4 proposed the Seasonal ARIMA. The Holt-Winters method BID30 ) is also very popular by using exponential smoothing. State space model BID10 BID24 BID5 also attracts much attention, which is a linear function of an underlying Markov process plus additive noise. Exponential Smoothing State Space Model (ETS) decomposes times series into error, trend, seasonal that change over time. Recently, deep learning is applied for time-series trend learning using LSTM BID26, bidirectional dynamic Boltzmann machine BID23 is applied for time-series long-term dependency learning, and coherent probabilistic forecast BID25 ) is proposed for a hierarchy or an aggregation-level comprising a set of time series. Orthogonal to these works, this paper focuses on robust ways of time series forecasting in presence of change points and anomalies. In Internet time series forecasting, Google develops the Bayesian structure time series (BSTS) model BID5 BID24 to capture the trend, seasonality, and similar components of the target series. Recently, Facebook proposes the Prophet approach BID27 based on a decomposable model with interpretable parameters that can be intuitively adjusted by analyst. However, as in the DAU example, some special events like Christmas Holiday or President Election, newly launched apps or features, may cause short period or long-term change of DAU, leading to weird forecasting of those traditional models. The aforementioned special cases are well known as• Anomaly points. The items, events or observations that don't conform to an expected pattern or other items in the dataset, leading to a sudden spike or decrease in the series.• Change points. A market intervention, such as a new product launch or the onset of an advertising (or ad) campaign, may lead to the level change of the original series. Time series forecasting without change/anomaly point detection and adjustment may also lead to bizarre forecasting since these models might learn the abrupt changes in the past. There are literatures on detecting anomaly or change points individually, examples can be found in BID29; BID22; BID3; BID21 BID29. However, the aforementioned change point detection models could not support detection in the presence of seasonality, while the presence of trend/change point is not handled by the anomaly detection models. Most importantly, there is a discrepancy between anomaly/change points detection and adjustment, and commonly used manually adjustment might be a bit arbitrary. Unfortunately, the forecasting gap caused by abnormal and change points, to the best of our knowledge, has not been given full attention and no good solution has been found so far. This paper is strongly motivated by bridging this gap. In this paper, to overcome the limitations of the most (if not all) current models that the anomaly points and change points are not properly considered, we develop a state space time series forecasting model in the Bayesian framework that can simultaneously detect anomaly and change points and perform forecasting. The learned structure information related to anomaly and change points is automatically incorporated into the forecasting process, which naturally enhances the model prediction based on the feedback of state-space model. To solve the ant optimization problem, an iterative algorithm based on Bayesian approximate inference with Markov chain Monte Carlo (MCMC), Kalman filter and Kalman smoothing is proposed. The novel model could explicitly capture the structure of change points, anomaly points, trend and seasonality, as also provide the distributions and forecasting intervals due to Bayesian forecasting framework. Both synthetic and real data sets show the better performance of proposed model, in comparison with existing baseline. Moreover, our proposed model outperforms state-of-the-art models in identifying anomaly and change points. To summarize, our work has the following contributions.• We proposed a robust 1 Bayesian state-space time series forecasting model that is able to explicitly capture the structures of change points and anomalies (which are generally ignored in most current models), and therefore automatically adapt for forecasting by incorporating the prior information of trend, seasonality, as well as change points and anomalies using state space modeling. Due to the enhancement of model description capability, the of model prediction and abnormal and change points detection are mutually improved.• To solve the ant optimization problem, an effective algorithm based on approximate inference using Markov chain Monte Carlo (MCMC) is proposed with theoretical guaranteed forecasting paths.• Our proposed method outperforms the state-of-the-art methods in time series forecasting in presence of change points and anomalies, and detects change points and anomalies with high accuracy and low false discovery rate on both tasks, outperforming popular change point and anomaly detection methods. Our method is flexible to capture the structure of time series under various scenarios with any component combinations of trend, seasonality, change points and anomalies. Therefore our method can be applied in many settings in practice. State space time series model BID13 has been one of the most popular models in time series analysis. It is capable of fitting complicated time series structure including linear trend and seasonality. However, times series observed in real life are almost all prevailed with outliers. Change points, less in frequency but are still widely observed in real time series analysis. Unfortunately, both structures are ignored in the classic state space time series model. In the section, we aim to address this issue by introducing a novel state space time series model. Let y = (y 1, y 2, . . ., y n) be a sequence of time series observations with length n. The ultimate goal is to forecast (y n+1, y n+2, . . .). The accuracy in forecasting lies in a successful decomposition of y into existing components. Apart from the residuals, we assume the time series is composed by trend, seasonality, change points and anomaly points. In a nutshell, we have an additive model with time series = trend + seasonality + change point + anomaly point + residual. As the classical state space model, we have observation equation and transition equations to model y and hidden variables. We use µ = (µ 1, µ 2, . . ., µ n) to model trend, and use γ = (γ 1, γ 2, . . ., γ n) to model seasonality. We use a binary vector z a = (z a 1, z a 2, . . ., z a n) to indicate anomaly points. Then we have Observation equation: DISPLAYFORM0 The deviation between the observation y t and its "mean" µ t + γ t is modeled by t and o t, depending on the value of z a t. If z a t = 1, then y t is an anomaly point; otherwise it is not. Distinguished from the residues = (1, 2, . . ., n), the anomaly is captured by o = (o 1, o 2, . . ., o n) which has relative large magnitude. The hidden state variable µ and γ have intrinsic structures. There are two transition equations, for trend and seasonality separately Transition Equations: Trend: DISPLAYFORM1 DISPLAYFORM2 In Equation, δ = (δ 1, δ 2, . . ., δ n) can be viewed as the "slope" of the trend, measuring how fast the trend changes over time. The change point component is also incorporated in Equation by a binary vector z c = (z is one of the five additive components along with trend, seasonality, anomaly points and residuals. Here we model the change point directly into the trend component. Though differing in formulation, they are equivalent to each other. We choose to model in as in Equation FORMULA1 due to simplicity, and its similarity with the definition of anomaly points in Equation.The seasonality component is presented in Equation. Here S is the length of one season and w = (w 1, w 2, . . ., w n) is the noise for seasonality. The seasonality component is assumed to have almost zero average in each season. The observation equation and transition equations (i.e., Equation) define how y is generated from all the hidden variables including change points and anomaly points. We continue to explore this new model, under a Bayesian framework. Bayesian methods are widely used in many data analysis fields. It is easy to implement and interpret, and it also has the ability to produce posterior distribution. The Bayesian method on state space time series model has been investigated in BID24 BID5. In this section, we also consider Bayesian framework for our novel state space time series model. We assume all the noises are normally distributed DISPLAYFORM0 where σ, σ o, σ u, σ r, σ v, σ w are parameters for standard deviation. As binary vectors, a natural choice is to model anomaly point indicator z a and change point indicator z c to the model them as Bernoulli random variables DISPLAYFORM1 where p a, p c are probabilities for each point to be an anomaly or change point. For simplicity, we denote α t = (µ t, δ t, γ t, γ t−1, . . ., γ t−(S−2) ) to include the main hidden variables (except z a t and z c t) in the transition equations. All the α t are well defined and can be generated from the previous status, except α 1. We denote a 1 to be the parameter for α 1, which can be interpreted as the "mean" for α 1.With Bayesian framework, we are able to represent our model graphically as in FIG2. As shown in FIG2, the only observations are y and all the others are hidden. In this paper, we assume there is no additional information on all the hidden states. If we have some prior information, for example, some points are more likely to be change points, then our model can be easily modified to incorporate such information, by using proper prior. In FIG2, we use squares and circles to classify unknown variables. Despite all being unknown, they actually behave differently according to their own functionality. For those in squares, they behave like turning parameters. Once they are initialized or given, those in circles behaves like latent variables. We call the former "parameters" and the latter "latent variable", as listed in TAB0. The "mean" for the initial trend and seasonality p = (pa, pc)Probabilities for each point to be anomaly or change point σ = (σ, σo, σu, σr, σv, σw) Standard deviationThe discrepancy between these two categories is clearly captured by the joint likelihood function. From FIG2, the joint distribution (i.e., the likelihood function) can be written down explicitly as DISPLAYFORM2 2 ) is the density function for normal distribution with mean x 1 and standard deviation x 2. Here we slightly abuse the notation by using µ 0, δ 0, γ 0, γ −1,..., γ 2−S, which are actually the corresponding coordinates of a 1.As long with other probabilistic graphical models, our model can also be viewed as a generative model. Given the parameters a 1, p, σ, we are able to generate time series. We present the generative procedure as follows. Algorithm 1: Generative Procedure Input: Parameters a 1, σ = (σ, σ o, σ u, σ r, σ v, σ w) and p a, p c, length of time series to generate m Output: Time series y = (y 1, y 2, . . ., y m) 1 Generate the indexes where anomalies or change points occur DISPLAYFORM3 2 Generate all the noises, o, u, r, v, w as independent normal random variables with mean zero and standard deviation σ, σ o, σ u, σ r, σ v, σ w respectively; 3 Generate {α t} m t=1 sequentially by the transition functions in Equation FORMULA1 This section is about inferring unknown variables from y, given the Bayesian setting described in the previous section. The main framework here is to sequentially update each hidden variable by fixing the remaining ones. As stated in the previous section, there are two different categories of unknown variables. Different update schemes need to be used due to the difference in their functionality. For the latent variables, we implement Markov chain Monte Carlo (MCMC) for inference. Particular, we use Gibbs sampler. We will elaborate the details of updates in the following sections. In this section, we focus on updating α assuming all the other hidden variables are given and fixed. The essence of Gibbs sampler is to obtain posterior distribution p a1,p,σ (α|y, z). This can be achieved by a combination of Kalman filter, Kalman smoothing and the so-called "fake-path" trick. We provide some intuitive explanation here and refer the readers to BID10 for detailed implementation. Kalman filter and Kalman smoothing are classic algorithms in signal processing and pattern recolonization for Bayesian inference. It is well related to other algorithms especially message passing algorithm. Kalman filter collects information forwards to obtain E(α t |y 1, y 2, . . ., y t); while Kalman smoothing distribute information backwards to achieve E(α t |y).However, the combination of Kalman filter and Kalman smoothing is not enough, as it only gives the the expectations of marginal distributions {E(α t |y)} n t=1, instead of the joint distribution required for Gibbs sampler. To address this issue, we can use the "fake-path" trick described in Brodersen et al. FORMULA0; BID10. The main idea underlying this trick lies on the fact that the covariance structure of p(α t |y) is not dependent on the means. If we are able to obtain the covariance by some other way, then we can add it up with {E(α t |y)} n t=1 to obtain a sample from p(α|y). This trick involves three steps. Note that all the other hidden variables z, p, σ are given.1. Pick some vectorã 1, and generate a sequence of time seriesỹ from it by Algorithm 1. In this way, we also observeα. 2. Obtain {E(α t |ỹ)} n t=1 fromỹ by Kalman filter and Kalman smoothing. 3. We use {α t − E(α t |ỹ) + E(α t |y)} n t=1 as our sampling from the conditional distribution. In this section, we update z by Gibbs sampler, assuming α, a 1, p, σ are all given and fixed. We need to obtain the conditional distribution p a1,p,σ (z|y, α). Note that in the graphical model described in Section 2, {z are still independent Bernoulli random variables, but possibly with different success probabilities. Thus, we can take the calculation point by point. For example, for the anomaly detection for the t-th point, we have DISPLAYFORM0 . And the prior on z a t is P(z a t = 1) = p a and P(z a t = 0) = p 1. Let p a t = P(z a t = 1|y, α). Directly calculation leads to DISPLAYFORM1 This equality holds for all t = 1, 2,..., n. Similarly for change point detection, let p As mentioned above, all the coordinates in z are still independent Bernoulli random variables conditioned on y, α. Thus, for Gibbs sampler, we can generate z by sampling independently with DISPLAYFORM2 For change point detection here, we have an additional segment control step. After obtaining {z c t} n t=1 as mentioned above, we need to make sure that the change points detected satisfy some additional requirement on the length of segment among two consecutive change points. This issue arises from the ambiguity between the definitions of change point and anomaly points. For example, consider a time series with value. We can view it with two change points, one increases the trend by 1 and the other decreases it by 1. Alternatively, we can also argue the three 1s in this time series are anomalies, though next to each other. One way to address this ambiguity is by defining the minimum length of segment (denoted as). In this toy example, if we set the minimum length to be 4, then they are anomaly points; if we set it to be 3, then we regard them to be change points. But a more complicated criterion is needed than using minimum length as the time series usually own much more complex structure than this toy example. Consider time series FIG0 ) and the minimum time series parameter = 3. It is reasonable to view it with one change point with increment 1, and the two -1s should be regarded as anomalies. As a combination of all these factors, we propose the following segment control method. A default value for the parameter is the length of seasonality, i.e., = S.Algorithm 2: Segment control on change points Input: change point binary vector z c,trend µ, standard deviation for outliers σ r, change point minimum segment Output: change point binary vector z The parameters σ, a 1 and p need both initialization and update. We have different initializations and update schemes for each of them. For all the standard deviations, once we obtain α and z, we update them by taking the empirical standard deviation correspondingly. For σ δ and σ γ, the calculation is straightforward as they only involve δ and γ respectively. For σ, σ o, σ u and σ r, it is a bit more involved due to z. Nevertheless, we can obtain the following update equations for all of them: DISPLAYFORM0 DISPLAYFORM1 Note that in some iterations, when there is no change point or anomaly detected in z, then the updates above for σ o, σ r are not well-defined. In those cases, we simply let them remain the same. To initialize σ, we let them all equal to the standard deviation of y. For a 1, we initialize it by letting its first coordinate to be equal to the average of y 1, y 2,..., y S, and all the remaining coordinates to be equal to 0. Since a 1 can be interpreted as the mean vector of α 1, in this way the trend is initialized to be matched up with average of the first season, and the slope and seasonality are initialized to be equal to 0. We update a 1 by using information of α. We let the first two coordinates (trend and slope) of a 1 to be equal to those of α 1, and we let the remaining coordinates (seasonality) of a 1 to be equal to those of α S+1. The reason why we do not let a 1 to be equal to α 1 entirely is due to the consideration on convergence and robustness. Since we initialize the seasonality part in a 1 as 0, it will remain 0 if we let a 1 equals α 1 entirely (due to the mechanism how we update α 1 as described in Section 4.1. We can avoid such trouble via using α S+1 .For p, we initialize them to be equal to 1/n. If we have additional information on the number of change points or anomaly points, we can initiate them with different values, for example, 0.1/n, or 10/n. We can update p after obtaining z, but we choose not to, also for the sake of robustness. In the early iterations when the algorithm is far from convergence, it is highly possible that z a or z c may turn out to be all 0. If we update p, say, by taking the proportion of change point or anomaly points in z. Then p a or p c might be 0, and it may get stuck in 0 in the remaining iterations. Once we infer all the latent variables α, z and tune all the parameters p, a 1, σ, we are able to forecast the future time series y future . From the graphical model described in Section 3, the future forecasting only involves α n instead of the whole α. Note that we assume that there exists no change point and anomaly point in the future. This is reasonable as in most cases we have no additional information on the future time series. Given α n and σ we can use our predictive procedures (i.e., Algorithm 1) to generate future time series y future. We can further integrate out α n to have the posterior predictive distribution as p σ (y future |y).The forecasting on future time series is not deterministic. There are two sources for the randomness in y future. One comes from the inference of α n (and also σ) from y. Under the Bayesian framework in Section 3, we have a posterior distribution over α n rather than a single point estimation. The second one comes from the forecasting function itself. The forecasting involves intrinsic noise like t, u t, v t and w t. Thus, the predictive density function p σ (y future |y, α n) will lead to different path even with fixed σ and α n. In this way we are able to obtain distribution and predictive interval for forecasting. We also suggest to take the average of multiple forecasting paths, as the posterior mean for the forecasting. The average of multiple forecasting paths (denoted asȳ future), if the number of paths is large enough, always takes the form as a combination of linear trend and seasonality. This can be observed in both our synthesis data (Section 7) and real data analysis (Section 8). This seems to be surprising at the first glance, but makes some sense intuitively. Under our assumption, we have no information on the future, and thus a reliable way to forecast the future is to use the information collected at the end of observed time series, i.e., trend µ n, slope δ n and seasonality structure. Theorem 1 gives mathematical explanation of the linearity ofȳ future, in both mean and standard deviation. Theorem 1. Let N be the number of future time series paths we generate from Algorithm 1). Let m be the number of points we are going to forecast. Denote {y to be the future paths. Defineȳ future = (ȳ n+1,ȳ n+2, . . .,ȳ n+m) to be the average such that DISPLAYFORM0 Then for all j = 1, 2,..., N, we haveȳ n+j as a normal distribution with mean and variance as DISPLAYFORM1 Consequently, for all j = 1, 2,..., m, E[ȳ n+j] is in a linear form with respect to j, and the standard deviation ofȳ n+j also takes a approximately linear form with respect to j. Proof. Recall that α n, σ are given and fixed, and we assume there is no change point or anomaly in the future time series. The Equation leads to δ n+j = δ n + j l=1 v n+l, which implies that DISPLAYFORM2 For the seasonality part, simple linear algebra together with Equation 3 leads to γ n+j = γ n−S+(j mod S) + j l=1 w n+l. Thus, DISPLAYFORM3 Due to the independence and Gaussian distribution of all the noises,ȳ n+j is also normally distributed and its means and variance can be calculated accordingly. Our proposed method can be divided into three parts: initialization, inference, and forecasting. Section 4 and Section 5 provide detailed explanation and reasoning for each of them. We present a whole picture of our proposed methodology in Algorithm 3.Algorithm 3: Proposed Algorithm Input: Observed time series y = (y 1, y 2, . . ., y n), seasonality length S, length of time series for forecasting m, number of predictive paths N, change point minimum segment l Output: Change point detection z c, anomaly points z a, forecasting y future = (y n+1, y n+1, . . ., y n+m) and its distribution or predictive intervals Part I: Initialization; 1 Initialize σ, σ o, σ u, σ r, σ v, σ w all with the empirical standard deviation of y; 2 Initialize a 1 such that its first coordinate equals to the average of (y 1, y 2, . . ., y S) and all the remaining S coordinates with 0; 3 Initialize p a and p c by 1/n. Then generate z a and z c as independent Bernoulli random variables with success probability p a and p c respectively; DISPLAYFORM0 Infer α by Kalman filter, Kalman smoothing and "fake-path" trick described in Section 4.1; Update z a and z c by sampling from DISPLAYFORM0, where the success probability {p Update σ by Equation FORMULA11 to; Update a 1 such that its first two coordinates equal to the those of α 1 and the remaining (S − 1) coordinates equals to those of α S+1; Calculate the likelihood function L a1,p,σ (y, α, z) given in Equation; end Part III: Forecasting; 10 With a n and σ, use the generate procedure in Algorithm 1 to generate future time series y future with length m. Repeat the generative procedure to obtain multiple future paths y DISPLAYFORM0 future,..., yfuture; 11 Combine all the predictive paths give the distribution for the future time series forecasting. If needed, calculate the point-wise quantile to obtain predictive intervals. Use the point-wise average as our final forecasting . It is worth mentioning that our proposed methodology is downward compatible with many simpler state space time series models. By letting p c = 0, we assume there is no change point in the time series. By letting p a = 0, we assume there is no anomaly point in the time series. If both p c and p a are set to be 0, then our model is reduced to the classic state space time series model. Also, the seasonality and slope can be removed from our model, if we know there exists no such structure in the data. In this section, we study the synthetic data generated from our model. We let S = 7 and provide values for σ and a 1. The change points and anomaly points are randomly generated. We use our generative procedure (Algorithm 1) to generate time series with total length 500 by fixed parameters. The first 350 points will be used as training set and the remaining 150 points will be used to evaluate the performance of forecasting. When generating, we let the time series have weekly seasonality with S = 7. For σ we have σ = 0.1, σ u = 0.1, σ v = 0.0004, σ w = 0.01, σ r = 1, σ o = 4. For α 1 we have value for µ as 20, value for δ as 0, and value for seasonality as (1, 2, 4, −1, −3, −2)/10. For p we have p c = 4/350 and p a = 10/350. Despite that, to make sure that at least one change point is in existence, we force z c 330 = 1 and r 330 = 2. That is, for each time series we generate, its 330th point is a change point with the mean shifted up by 3. Also to be consistence with our assumption, we force z c i = z a i = 0, ∀351 ≤ i ≤ 500 so there exists no change point or anomaly point in the testing part. The top panel of FIG9 shows one example of synthesis data. The blue line marks the separation between training and testing set. The blue dashed line indicates the locations for the change point, while the yellow dots indicate the positions of anomaly points. Also see FIG9 for illustration on the returned by implementing our proposed algorithm on the same dataset. The red line gives the fitting in the first 350 points and forecasting in the last 150 points. The change points detected are marked with vertical red dotted line, and the anomaly detected are flagged with purple squares. FIG9 shows that on this dataset, our proposed algorithm yields perfect detection on both change points and anomaly points. In FIG9, the gray part indicates the 90% predictive interval for forecasting. We run our generative model 100 times to produce 100 different time series, and implement multiply methods on each of them, and aggregate the together for comparison. We include the following methodologies. For time series forecasting, we compare our method against Bayesian Structural Time Series (BSTS) BID24 BID5 ), Seasonal Decomposition of Time Series by Loess (STL) BID7 ), Seasonal ARIMA BID4, Holt-Winters , Exponential Smoothing State Space Model (ETS) ), and the Prophet R package by BID27. We evaluate the performances by mean absolute percentage error (MAPE), mean square error (MSE) and mean absolute error (MAE) on forecasting set. The mathematical definition of these three criterion is given as follows. Let x 1, x 2,..., x n be the true value andx 1,x 2,...,x n be the estimation or predictive values. Then we have DISPLAYFORM0 The comparison of our proposed algorithm and the aforementioned algorithms are included below in TAB1. As we mentioned in Section 6, our algorithm is downward compatible with the cases ignoring the existence of change point or anomaly, by setting p c = 0 or p a = 0. We also run proposed algorithm on the synthetic data with p c = 0 (no change point), or p a = 0 (no anomaly point), or p c = p a = 0 (no change and anomaly point), for the purpose of numeric comparison. From TAB1 it turns out that our proposed algorithm achieves the best performance compared to other existing methods. Our proposed algorithm also performs better compared with the cases ignoring change point or anomaly point. This is a convincing evidence on the importance of incorporating both change point structure and anomaly point structure when modeling, for time series forecasting. We also compare our proposed method with other existing change point detection methods and anomaly detection algorithm with respect to the performance of detections. We evaluate the performance by two criterions: True Positive Rate (TPR) and False Positive (FP). TPR measures the percentage of change points or anomalies to be correctly detected. FP count the number of points wrongly detected as change points or anomaly points. The mathematical definitions of TPR and FP are as follows. Let (z 1, z 2, . . ., z n) be the true binary vector for change points or anomalies, and (ẑ 1,ẑ 2, . . .,ẑ n) are the estimated ones. Then DISPLAYFORM1 From the definition, we can see high TPR and low FP means the algorithm has better performance in detection. The comparison on change point detection is shown in TAB2. We compare our against three popular change point detection methods: Bayesian Change Point (BCP) BID3, Change-Point (CP) BID21 and Breakout (twitter, 2017). From TAB2 our proposed method outperforms the most of the others by both TPR and FP. We have smaller TPR compared to CP, but we are better in FP. In TAB3, we also compare the performance of our algorithm on anomaly detection with three existing common anomaly detection methods: the AnomalyDetection package by BID29, RAD by BID22 and Tsoutlier by BID6. The comparison is listed in TAB3. We can see our method also outperforms most of the others with respect to anomaly detection, by both TPR and FP. RAD has slightly better TPR but its FP is much worse compared with ours. In this section, we implement our proposed method on real-world datasets. We also compare its performance against other existing time series forecasting methodologies. We consider two datasets, one is a public data called Well-log dataset, and the other is an unpublished internet traffic dataset. The bottom panels of FIG10 and FIG11 give the of our proposed algorithms. The blue line separates the training set and testing set. We use red line to show our fitting and forecasting , vertical red dashed line to indicate change points and purple dots to indicate anomaly points. The gray part shows 90% predication interval. This dataset BID11 BID20 was collected when drilling a well. It measures the nuclear magnetic response, which provides geophysical information to analyze the structure of rock surrounding the well. This dataset is public and available online 2. It has 4050 points in total. We split it such that the first 3000 points are used as training set and last 1000 points are used to evaluate the forecasting performance. From FIG10, it is obvious that there exists no seasonality or slope structure in the dataset. This motivates us not to include these two components in our model. We implement our proposed algorithm without seasonality and slope, and compare the forecasting performance with other methods in TAB4. Our method outperforms BSTS, ARIMA, ETS and Prophet. However in TAB4 the performance can be slightly improved if we ignore the existence of anomaly points by letting p a = 0. This may be caused by model mis-specification as the data may not generated in a way not entirely captured by our model. Nevertheless, the performances of our method considering anomaly points or not, are comparable to each other. In this dataset there is no ground-truth of change point and anomaly point on their locations or even existence. However, from bottom panel of FIG10, there are some obvious changes in the sequence and they all successfully captured by our algorithm. Our second real data is an Internet traffic data acquired from a major Tech company (see FIG11). It is a daily traffic data, with seasonality S = 7. We use the first 800 observations as training set and evaluate the performance of forecasting on the remaining 265 points. The bottom panel of FIG11 show the from implementing our algorithm. We also do the comparison of forecasting performance of our proposed algorithm together with other existing methods, shown in TAB5. We can also see that our algorithm outperforms all the other algorithms with respect to MAPE, MSE and MAE. From FIG11 our proposed algorithm identifies one change point (the 576th point, indicated by the vertical red dashed line), which can be confirmed that this is exactly the only one change point existing in this time series caused by the change of counting methods, by some external information. Thus, we give the perfect change point detection in this Internet traffic data. For this Internet traffic dataset, since we have ground-truth for change point, we can compare the performance of change point detection of different methodologies. BCP returns posterior distribution, which peaks in the the 576th point with posterior probability value 0.5. And it also returns with many other points with posterior probability value around 0.1. CP returns 4 change points, where the 576th point (the only true one) is one of them. Breakout returns 8 change points without including the 576th point. To sum up, our proposed method achieves the best change point detection in this real dataset. Compared to the aforementioned models, our work differs in Bayesian modeling which samples posterior to estimate hidden components given the independent Bernoulli priors of changing point and anomalies.

Represent the TLDR for finding the corresponding scientific article

We propose a novel state space time series model with the capability to capture the structure of change points and anomaly points, so that it has a better forecasting performance when there exist change points and anomalies in the time series.

Represent the short summary for finding the corresponding scientific article (long)

GANs are shown to provide us a new effective robust mean estimate against agnostic contaminations with both statistical optimality and practical tractability.