and test sets When evaluating the quality of forecasts, it is invalid to look at how well a model fits the historical data; the accuracy of forecasts can only be determined by considering how well a model performs on new data that were not used when fitting the model. When choosing models, it is common to use a portion of the available data for fitting, and use the rest of the data for testing the model, as was done in the above examples. Cross-validation Cross-validation is a more sophisticated version of training a test set. For cross-sectional data, one approach to cross-validation works as follows: Select observation i for the test set, and use the remaining observations in the training set. Compute the error on the test observation. Repeat the above step for i = 1,2,..., N where N is the total number of observations. Compute the forecast accuracy measures based on the errors obtained. This makes efficient use of the available data, as only one observation is omitted at each step For time series data, the training set can only include observations prior to the test set. Therefore, no future observations can be used in constructing the forecast. Suppose k observations are needed to produce a reliable forecast; then the process works as follows: Starting with i=1, select the observation k + i for the test set, and use the observations at times 1, 2, ..., k+i–1 to estimate the forecasting model. Compute the error on the forecast for k+i. Repeat the above step for i = 2,...,T–k where T is the total number of observations. Compute the forecast accuracy over all errors. This procedure is sometimes known as a "rolling forecasting origin" because the "origin" (k+i -1) at which the forecast is based rolls forward in time. Further, two-step-ahead or in general p-step-ahead forecasts can be computed by first forecasting the value immediately after the training set, then using this value with the training set values to forecast two periods ahead, etc. See also Calculating demand forecast accuracy Consensus forecasts Forecast error Predictability Prediction intervals, similar to confidence intervals Reference class forecasting Seasonality and cyclic behaviour Seasonality Seasonality is a characteristic of a time series in which the data experiences regular and predictable changes which recur every calendar year. Any predictable change or pattern in a time series that recurs or repeats over a one-year period can be