Is Linear Prediction ever exact?
Theoretically, yes. So you can for instance predict values taken by polynomials, with the following initial patterns:


degree $0$: $\hat{S}[n] = {S}[n-1]$;
degree $1$: $\hat{S}[n] = 2{S}[n-1]-{S}[n-2]$;
degree $2$: $\hat{S}[n] = 3{S}[n-1]-3{S}[n-2]+{S}[n-3]$;
degree $3$: $\hat{S}[n] = 4{S}[n-1]-6{S}[n-2]+4{S}[n-3]-{S}[n-4]$;


and so on. You can recognize binomial coefficients. 

There are similar  linear relations for other signals when sampling conditions are met, and for different cost functions. I am happy to dig into old stuff here. In a 1927, G.  Udny Yule (the same as in autoregressive modeling known under Yule-Walker), On a Method of Investigating Periodicities in Disturbed Series, with Special Reference to Wolfer's Sunspot Numbers,  gave the following trigonometric recurrence relation  (using the same notations), for:

$$\begin{align} 
u_0 &amp; =&amp; A \sin\left({2\pi\frac{t+\tau}{T}}\right)\\
u_1 &amp; =&amp; A \sin\left({2\pi\frac{t+\tau+h}{T}}\right)\\
u_2 &amp; =&amp; A \sin\left({2\pi\frac{t+\tau+2h}{T}}\right)\end{align}
 $$
then 
$$ u_2 = (2-\mu)u_1-u_0$$
with $\mu = 4 \sin^2 \pi\frac{h}{T}$. One can notice that, when $\mu =0$, one recognizes the linear extrapolation: $ u_2 = 2u_1-u_0$.

I would suggest: 


The Theory of Linear Prediction by P. P. Vaidyanathan, or 
chapter 13.6 Linear Prediction and Linear Predictive Coding  in Numerical Recipes.


In practice, the exactness is limited by the data precision (in time and amplitude), for instance by finite-word quantization (rounding errors, floating point operations) and jitter, in time.