# How is Gram-Schmidt procedure used in the following time series context?

I was reading the innovation algorithm in Brickel's Time Series Theory and Methods (page 171-172).

Let $H$ denotes a Hilbert space, $P$ denotes the projection operator and $\bar{sp}$ denotes closed span.

It mentioned that

The innovation algorithm depends on the decomposition of $H_{n}$ into $n$ orthogonal subspaces by means of the Gram-Schmidt procedure.

Then it says

As before, we define $H_n = \bar{sp}\{X_1, ..., X_n\}$ and and the one-step predictors $$\hat{X}_{n+1} = 0 \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \text{if } n = 0$$ $$\hat{X}_{n+1} = P_{H_n} X_{n+1}\;\;\;\;\;\;\; \text{if } n \geq 1$$

and we also define $\hat{X_1}:= 0$, $$H_{n} = \bar{sp}\{X_1 - \hat{X}_1, X_2 - \hat{X}_2,..., X_n - \hat{X}_n\} \;\;\;\;\;\;\;\;\;\; n \geq 1 \;\;\;\;\;\;\;\;\;\;(1)$$ so that $$\hat{X}_{n+1} = \sum^n_{j=1}\theta_{nj}(X_{n+1-j} - \hat{X}_{n+1-j})\;\;\;\;\;\;\;\;\;\;\;(2)$$

The recursive scheme for computing ${\theta_{nj}, j = 1,...,n = 1,2,...}$, is derived later on in the book.

My first question is: how does $(1)$ come about from the Gram-Schmidt procedure ? Because from my understanding, according to the Gram-Schmidt procedure, the orthogonal vectors of $H_n$ that is also a closed span should be the following: $$u_1 = X_1 = X_1 - \hat{X}_1$$ $$u_2 = X_2 - P_{H1}X_2 = X_2 - \hat{X}_2$$ $$u_3 = X_3 - P_{H1}X_3 - P_{H_2}X_3= X_3 - \hat{X}_{1+2} - \hat{X}_{2+1}$$ $$....$$ and I thought it should be $H_n = \bar{sp}\{u_1, u_2, u_3,...\}$ which is clearly different from $(1)$ above.

My second question is how does $(1)$ lead to $(2)$ , suppose that we can ignore the what exactly $\theta_{nj}$ is for now?

• Brickel may mean Brockwell and Davis. Either way, full citations of books and papers as expected in research texts and papers is always helpful. – Nick Cox Mar 27 '15 at 11:43
• Cross-posted here math.stackexchange.com/questions/1183974/… and indeed elsewhere. Even if no-one answers a question elsewhere, leaving it open carries an obligation to cross-refer to answers received, as here. – Nick Cox Mar 27 '15 at 11:45

First question

$$u_3 = X_3 - P_{H1}X_3 - P_{H_2}X_3= X_3 - \hat{X}_{1+2} - \hat{X}_{2+1}$$

is not right. Notice $H_1 \subset H_2$. $P_{H1}$ is not orthogonal to $P_{H_2}$.

Second question

It's immediate, exactly the same as, say, when the Hilbert space is $\mathbb{R}^3$. It's by induction: take $X_0$, and $X_1$, then in your notation,

$$\hat{X}_1 = \frac{\langle X_1, X_0\rangle}{\langle X_0, X_0\rangle} (X_0 - 0) = \theta (X_0 - 0) .$$

For real random variables with finite seconds moments, the inner product is expectation of the product. If there's an $X_2$, then

$$\hat{X}_2 = P_{\{ X_1, X_0 \} } X_2 = P_{\{ X_1 - \hat{X}_1, X_0 \} } X_2 = P_{ X_1 - \hat{X}_1} X_2 + P_{X_0 } X_2.$$

This is exactly the GS procedure. You see how the $\theta$'s can be computed recursively.

• can you elaborate a bit more about the answer for second question. Because I don't quite follow. Thanks. – mynameisJEFF Mar 27 '15 at 10:08
• Are you sure you're not missing some normalization in your question there? – Michael Mar 27 '15 at 10:44
• It's actually exactly GS. As you've written, though, all $\theta$'s are 1's. – Michael Mar 27 '15 at 11:08
• I understand what you are saying . But the $\theta$ s are not actually ones. Perhaps , I should attach a picture of the original piece of text later on so everybody who is interested in this problem can gain a good understanding of the question. – mynameisJEFF Mar 27 '15 at 11:16
• Never mind, I misread. The $\theta$'s are the coefficients of the projection onto $\{ X_1, \cdots, X_{n-1}\}$. Will edit answer accordingly. – Michael Mar 27 '15 at 11:25
{"ft_lang_label":"__label__en","ft_lang_prob":0.78136766,"math_prob":0.9994154,"size":1612,"snap":"2021-21-2021-25","text_gpt3_token_len":613,"char_repetition_ratio":0.16293532,"word_repetition_ratio":0.0,"special_character_ratio":0.45781636,"punctuation_ratio":0.26546392,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.99988425,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2021-05-08T00:52:54Z\",\"WARC-Record-ID\":\"<urn:uuid:5bf71393-ca87-46f2-a284-a4774bf2d5d8>\",\"Content-Length\":\"172603\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:24d8258d-36c8-4afa-b8e9-a31cf74eb573>\",\"WARC-Concurrent-To\":\"<urn:uuid:9019b2a5-58c3-4114-82f4-ab2b8484f171>\",\"WARC-IP-Address\":\"151.101.193.69\",\"WARC-Target-URI\":\"https://stats.stackexchange.com/questions/141235/how-is-gram-schmidt-procedure-used-in-the-following-time-series-context\",\"WARC-Payload-Digest\":\"sha1:DIZIAPOPCQORSTSOZMPCFHDS6EIPVJ7Y\",\"WARC-Block-Digest\":\"sha1:4XO5RUKNWIWWI4IZ7KK6LGHVIOSVTKV3\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2021/CC-MAIN-2021-21/CC-MAIN-2021-21_segments_1620243988831.77_warc_CC-MAIN-20210508001259-20210508031259-00383.warc.gz\"}"}