x = np.linspace(-3, 3, 100)

y = np.sin(4 * x) + x + np.random.uniform(size=len(x))

plt.plot(x, y, 'o')

"""
Explanation: Regression
In regression we try to predict a continuous output variable. This can be most easily visualized in one dimension.
We will start with a very simple toy example. We will create a dataset out of a sinus curve with some noise:
End of explanation
"""


print(x.shape)
X = x[:, np.newaxis]
print(X.shape)

"""
Explanation: Linear Regression
One of the simplest models again is a linear one, that simply tries to predict the data as lying on a line. One way to find such a line is LinearRegression (also known as ordinary least squares).
The interface for LinearRegression is exactly the same as for the classifiers before, only that y now contains float values, instead of classes.
To apply a scikit-learn model, we need to make X be a 2d-array:
End of explanation
"""


from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y)
print(X_train.shape, X_test.shape)

"""
Explanation: We split our data in a training and a test set again:
End of explanation
"""


from sklearn.linear_model import LinearRegression
regressor = LinearRegression()
regressor.fit(X_train, y_train)

"""
Explanation: Then we can built our regression model:
End of explanation
"""


y_pred_train = regressor.predict(X_train)

plt.plot(X_train, y_train, 'o', label="data")
plt.plot(X_train, y_pred_train, 'o', label="prediction")
x = np.linspace(-3, 3)
plt.plot(x, regressor.coef_*x + regressor.intercept_, '-r')
plt.legend(loc='best')

"""
Explanation: And predict. First let us try the training set:
End of explanation
"""


y_pred_test = regressor.predict(X_test)

plt.plot(X_test, y_test, 'o', label="data")
plt.plot(X_test, y_pred_test, 'o', label="prediction")
plt.legend(loc='best')

"""
Explanation: The line is able to capture the general slope of the data, but not many details.
Let's try the test set:
End of explanation
"""


print(regressor.score(X_test, y_test))
print(regressor.score(X_train, y_train))

"""
Explanation: Again, scikit-learn provides an easy way to evaluate the prediction quantitatively using the score method. For regression tasks, this is the R2 score. Another popular way would be the mean squared error.
End of explanation
"""


from sklearn.neighbors import KNeighborsRegressor
kneighbor_regression = KNeighborsRegressor(n_neighbors=3)
kneighbor_regression.fit(X_train, y_train)

"""
Explanation: KNeighborsRegression
As for classification, we can also use a neighbor based method for regression. We can simply take the output of the nearest point, or we could average several nearest points. This method is less popular for regression than for classification, but still a good baseline.
End of explanation
"""


y_pred_train = kneighbor_regression.predict(X_train)

plt.plot(X_train, y_train, 'o', label="data")
plt.plot(X_train, y_pred_train, 'o', label="prediction")
X_ = np.linspace(-3, 3, 1000)[:, np.newaxis]
y_ = kneighbor_regression.predict(X_)
plt.plot(X_, y_, 'r-')
plt.legend(loc='best')

"""
Explanation: Again, let us look at the behavior on training and test set:
End of explanation
"""


y_pred_test = kneighbor_regression.predict(X_test)

plt.plot(X_test, y_test, 'o', label="data")
plt.plot(X_test, y_pred_test, 'o', label="prediction")
plt.legend(loc='best')

"""
Explanation: On the training set, we do a perfect job: each point is its own nearest neighbor!
End of explanation
"""


print(kneighbor_regression.score(X_test, y_test))
print(kneighbor_regression.score(X_train, y_train))

"""
Explanation: On the test set, we also do a better job of capturing the variation, but our estimates look much more messy then before.
Let us look at the R2 score:
End of explanation
"""


from sklearn.datasets import load_boston
boston = load_boston()
X_train, X_test, y_train, y_test = train_test_split(boston.data, boston.target, random_state=0)


# Linear regression
regressor = LinearRegression()
regressor.fit(X_train, y_train)
print('train score: ', regressor.score(X_train, y_train))
print('test score:  ', regressor.score(X_test, y_test))

# KNN
kn_regressor = KNeighborsRegressor(n_neighbors=5)
kn_regressor.fit(X_train, y_train)
print('train score: ', kn_regressor.score(X_train, y_train))
print('test score:  ', kn_regressor.score(X_test, y_test))

"""
Explanation: Much better then before! Here, the linear model was not a good fit for our problem.
Exercise
Compare the KNeighborsRegressor and LinearRegression on the boston housing dataset. You can load the dataset using sklearn.datasets.load_boston.
End of explanation
"""


