%matplotlib inline
import numpy as np
import statsmodels.api as sm
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')

np.random.seed(123)
n = 100
x = np.linspace(0.01, 2, n)
y = 2 * np.log(x)
y_noise = y + np.random.normal(size=(n))

"""
Explanation: Bias-Variance Trade-Off
Introduction
Let us assume we have some sample that somewhat follows the function $f(x) = 2 \cdot \ln(x)$.
Obviously, in reality we never know the true function (otherwise what would be the point of doing any machine learning). However, what we usually have is a (noisy) sample based on which we wish to infer on the true underlying relationship between feature ($X$) and response ($y$). 
Before we start talking about bias and variance and its trade-off, let us first load some packages needed to visualize this and generate some random data (following the function states above).
End of explanation
"""


plt.figure(figsize=(12, 8))
plt.scatter(x, y_noise);

"""
Explanation: And here is how our random sample looks like. Without knowing the true relation between feature and response one could easily argue the dependent variable follow some linear function (or maybe a polynomail function with low order).
End of explanation
"""


# Create OLS object
reg = sm.OLS(y_noise, exog=sm.add_constant(x)).fit()

print('R2 = ', reg.rsquared)

reg_ln = sm.OLS(y_noise, exog=sm.add_constant(np.log(x))).fit()
print('R2 = ', reg_ln.rsquared)

# Plot data
plt.figure(figsize=(12, 8))
plt.scatter(x, y_noise, label="Sample")
plt.plot(x, y, color='g', label="True function")
plt.plot(x, reg.fittedvalues, c='k', label='Linear Fit')  # Linear fit
#plt.plot(x, reg_ln.fittedvalues, c='r', label='Ln Fit')
plt.legend();

"""
Explanation: Bias
We defined 'bias' as follows: 
* The inability for a machine learning method (like linear regression) to capture the true relationship is called bias.
* Refers to the error that is introduced by approximating a real-life problem, which may be extremely complicated, by a much simpler model.
* The lower the bias, the closer the model approaches the underlying true pattern. The higher the bias, the further the model is from the true complexity of the underlying true pattern.
What if we approximate our sample with a linear regression? Below code does exactly this.
End of explanation
"""


# Kreate kernel regression object
kreg = sm.nonparametric.KernelReg(y_noise, exog=x, var_type='o', bw=[0.20]).fit()

# Plot results
plt.figure(figsize=(12, 8))
plt.plot(x, y, color='g', label="True")
plt.scatter(x, y_noise, label="Sample")
plt.plot(x, reg.fittedvalues, c='k', label='Linear Fit') # Linear fit
plt.plot(x, kreg[0], c='cyan', label='Kernel Fit')  # Kernel fit
plt.legend();

"""
Explanation: Knowing the true relationship between feature and response we can see that we have a certain bias present in that we approximate a logarithmic function with a much more static linear function. Especially towards the edges of the spectrum we tend to misclasify the sample.
Let us instead look at a kernel regression, which (given low bandwidth, follows the observation very closely.
End of explanation
"""


plt.figure(figsize=(16, 8))
for i in range(1, 7):
    indices = np.sort(np.random.randint(0, x.shape[0], 20))
    reg = sm.OLS(y_noise[indices], exog=sm.add_constant(x[indices])).fit()
    kreg = sm.nonparametric.KernelReg(y_noise[indices], exog=x[indices], var_type='o', bw=[0.05]).fit()
    plt.subplot(2, 3, i)
    plt.plot(x[indices], reg.fittedvalues, c='k', label='Fit')
    plt.plot(x[indices], kreg[0], c='cyan', label='Kernel Fit')
    plt.plot(x, y, color='g', label="True")
    plt.ylim(-10, 4)
    plt.scatter(x[indices], y_noise[indices], label="Sample")
    plt.text(0.1, -8, 'Lin.fit: f(x) = {0: .2f} + {1: .2f} $x$'.format(reg.params[0], reg.params[1]))
    plt.legend();


"""
Explanation: Here we see that a Kernel regression is much better at approximating the true function than a simple linear model. Therefore, the bias for the more complex function is lower and should be preferred. 
Variance
This is how we defined variance: 
* Variance refers to the amount by which $\hat{f}$ would change if we estimated it using a different (training) sample.
* Ideally, our estimate of $\hat{f}$ does not vary too much between training sets.
* Tells us how sensitive the learning algorithm is to small changes in the training set
Now let us see how the two approximation functions (linear and kernel regression) fare when we just use a subset of overall sample. This we do to see the variance in the function $\hat{f}$ between different training sets.
End of explanation
"""


from IPython.display import YouTubeVideo
from datetime import timedelta

YouTubeVideo('EuBBz3bI-aA', start=int(timedelta(hours=0, minutes=0, seconds=17).total_seconds()))

"""
Explanation: Here it becomes pretty clear that for the kernel regression the variance is high as this regression is simply too dependent on the actual sample. The linear regression however, is much more stable and produces +/- the same results independent of the sample. Therefore its model variance is low and linear model should be preferred over a kernel model. 
Conclusion
The goal is obviously to have both low bias as well as low variance. We have seen from the above plots that in one case the more complex approach (kernel) produces lower bias than the less complex. At the same time, we have also observed that for a more complex model the variance is much higher from sample to sample than with a more static/less complex mode. This shows the trade-off we are: it is impossible to have both at the same time meaning that we have to find a sweetspot in between. 
For further information see e.g. this or other ressources.
End of explanation
"""


