print("Hello, world!")

"""
Explanation: So far, we've been running all of our code linearly, i.e. we execute a file an we let Python do its job. Here we will be running code interactively. Each cell will contain some code, which we can execute by clicking on the cell and pressing Shift + Enter. Here is an example:
End of explanation
"""


hello = "Hello, world!"

"""
Explanation: As you can see, the code executed and the output is now in the notebook. One thing that's special about Jupyter is that our variables are persistent. I.e. we could create the block:
End of explanation
"""


print(hello)

"""
Explanation: And then in a new block execute:
End of explanation
"""


import pandas as pd
import sklearn as sk
from sklearn import datasets as ds
from sklearn.linear_model import LinearRegression
from matplotlib import pyplot as plt

"""
Explanation: You might have noticed the In [ ] to the side of the cells. The numbers inside indicate the order in which the code in the cells was executed. You should imagine this notebook as maintaining a hidden Python program, with all of our variables and functions, etc. executed in the order of the In [ ] blocks.
Now, lets analyze some data. To start off with, we will import the libraries which we will need.
End of explanation
"""


boston = ds.load_boston()

"""
Explanation: In our first example, we will analyze a dataset of Boston house prices, which comes packaged with the sklearn module.
End of explanation
"""


boston

"""
Explanation: When you run a cell in Jupyter, it will try to print out the value on the last line of the cell. So far, we've loaded the dataset into a variable boston. Lets investigate how this dataset is actually structured.
End of explanation
"""


boston.keys()

"""
Explanation: As we can see, this dataset is actually a dict, containing a few fields. We can get a list of those fields:
End of explanation
"""


print(boston['DESCR'])

"""
Explanation: The DESCR key is particularly interesting, as it contains a string which describes the dataset. Lets check it out:
End of explanation
"""


boston['data']

boston['target']

boston['feature_names']

"""
Explanation: So as we can see, the data field contains the first 13 attributes, the target field contains the median value of a house, and the feature_names contains the names of the features. We could view these all individually.
End of explanation
"""


df = pd.DataFrame(data=boston['data'], columns=boston['feature_names'])
df

"""
Explanation: However, this is very inconvenient. To address this issue, we will use the DataFrame class from the pandas module we imported.
End of explanation
"""


df.query("CRIM >= 20") # List all towns with average crime rate >= 20 per capita

"""
Explanation: Much nicer. There are settings in Jupyter which allow us to force rendering more rows, but we will not go into these now. The important thing is that we can inspect our data. We might wish to query our data to filter some results. Pandas supports a few ways of doing this. One of them is the query function of DataFrames.
End of explanation
"""


plt.scatter(df['CRIM'], boston['target'])
plt.xlabel('Average crime rate per capita')
plt.ylabel('Median house price in $1000')
plt.show()

"""
Explanation: The query function is quite intuitive to use and if your query looks like python code, it will probably get executed. 
Now, maybe we want to inspect our data, to see how each of the attributes relates to the median house price (the "target"). To do that we will use the pyplot submodule of the matplotlib module. Lets first plot the crime rate against the house price.
End of explanation
"""


plt.figure(figsize=(20,15))
for i, column in enumerate(df):
    plt.subplot(4, 4, i+1)
    plt.scatter(df[column], boston['target'])

"""
Explanation: As we might expect, the house price goes down as the crime rate increases. However, at the lower end, crime rate doesn't seem to be such a good predictor. We could do the same thing and plot each attribute against the house price. Or we could do it programatically.
End of explanation
"""


plt.scatter(df['RM'], boston['target'])
plt.xlabel('Average number of rooms per dwelling')
plt.ylabel('Median house price in $1000')
plt.show()

"""
Explanation: As we can see, some attributes seem more predictive than others. Let's focus on the RM attribute, which is the average number of rooms per dwelling.
End of explanation
"""


reg = LinearRegression().fit(df[['RM']], boston['target'])

"""
Explanation: As we might expect, this also seems like a pretty good predictor of house price. But what if we wanted to model the correlation between the number of rooms in a dwelling and its expected price in that town? One simple model is called Linear Regression, and it essentially involves drawing a line through the data. The module sklearn provides us with such a model, which we will use in our program.
End of explanation
"""


plt.scatter(df['RM'], boston['target'])
plt.plot(df['RM'], reg.predict(df[['RM']]), color='r')
plt.xlabel('Average number of rooms per dwelling')
plt.ylabel('Median house price in $1000')
plt.show()

"""
Explanation: Now the variable reg contains our model. It is fit on our data, meaning that it is the line which best fits our data (according to the linear regression interpretation). The red line is how our model fits the data. Essentially, it will always predict a house price that lies on the red line, given the number of rooms.
End of explanation
"""


reg.score(df[['RM']], boston['target'])

"""
Explanation: One way to evaluate how good our model fits the data is called an $R^2$ score, or a coefficient of determination. Essentially, it assigns a number between $-\infty$ and $1$ which indicates how well our model fits the data. The best possible score is $1$. We can check out the $R^2$ score of our model using the score function.
End of explanation
"""


reg_full = LinearRegression().fit(df, boston['target'])

"""
Explanation: Not great, not terrible.
We can use the same technique to try to use all of our attributes to predict the house price. This would look like:
End of explanation
"""


plt.scatter(df['RM'], boston['target'])
plt.scatter(df['RM'], reg_full.predict(df), color='r')
plt.xlabel('Average number of rooms per dwelling')
plt.ylabel('Median house price in $1000')
plt.show()

"""
Explanation: Now, to get an intuition of how well our model fits the data now, we will again plot the average number of rooms against the median price. This time, the actual data points are in blue, and the predictions of our model are in red.
End of explanation
"""


reg_full.score(df, boston['target'])

"""
Explanation: This looks much better. In fact, we can try the score again:
End of explanation
"""


my_df = pd.read_csv('my_dataset.csv')

"""
Explanation: As we can see, this model is a lot better than the first one. As an exercise, you might want to try removing some of the attributes, and seeing which ones affect the score a lot, and which ones don't.
If we wanted to import our own dataset, we can do that as well. Pandas has functions like read_csv and read_excel which allow you to import your dataset from a .csv or .xlsx file and analyze it like any other DataFrame. What you would do is:
End of explanation
"""


my_df = pd.read_excel('my_dataset.xlsx')

"""
Explanation: Or
End of explanation
"""


