%matplotlib inline

import numpy as np

from sklearn import tree

import pandas as pd

"""
Explanation: Data Munging with Pandas
Table of Contents
    1. Let's load the dataset
    2. Let's plot the dataset
        2.1  scatter_matrix
        2.2  Bar plot
        2.3  Histogram
        2.4  Box plot
        2.5  Area plot
        2.6  Scatter plot
        2.7  Hexbin plot
        2.8  Pie plot
        2.9  Missing data on plots
        2.10 Density/KDE plots
        2.11 Andrews Curves
        2.12 Parallel Coordinates
        2.13 Lag plot
        2.14 Autocorrelation plot
        2.15 Bootstrap plot
        2.16 RadVis plot
    3. Data holders
        3.1 Series
            3.1.1 Series from ndarray
            3.1.2 Series from dict
            3.1.3 Series from scalar
        3.2 DataFrame
            3.2.1 DataFrame from dict of Series o dicts
            3.2.2 DataFrame from dict of ndarrays
            3.2.3 DataFrame from structured/record arrays
            3.2.4 DataFrame from lists of dicts 
            3.2.5 DataFrame from dict of tuples
            3.2.6 DataFrame from Series
    4. Dealing with missing data
        4.1 Automatic Formating of date strings
        4.2 Handle of NaNs
            4.2.1 Find NaN values
                4.2.1.1 NaN and None
            4.2.2 Fill NaN with fixed value
            4.2.3 Fill NaN with mean or median
            4.2.4 Drop NaN rows
            4.2.5 Calculations with NaN
        4.3 Reading Bad Input files
        4.4 
Let's focus on the Pandas dataframe Munging stuff
The modules that we'll need:
End of explanation
"""


from sklearn import datasets

iris = datasets.load_iris()

"""
Explanation: 1. Let's load the dataset
We'll load one of the example datasets from sklearn!
End of explanation
"""


iris.DESCR;
iris.data;
iris.feature_names

iris.target;
iris.target_names

iris.data.shape

type(iris.data)

"""
Explanation: This is a sklearn dataset including the 
-  .DESCR : general description
- .data  : all features
- .features_names : names of features
- .target : target values expressed as values / numbered classes
- .target_names : names of the target classes 
-  .shape : can be applied both on .data and .target and give the (row,column) tuple
End of explanation
"""


dataframe = pd.DataFrame(iris.data, columns=iris.feature_names)

dataframe.head()

target_df = pd.DataFrame(iris.target, columns=['Species'])

target_df.tail(2)

"""
Explanation: Let's use pandas' to convert the ndarray into dataframe, keeping the column names!
End of explanation
"""


colors = list()

palette = { 0: "red", 1: "green", 2:"blue"}

for c in np.nditer(iris.target):
    colors.append(palette[int(c)])  # c is 0,1 or 2 and we append red, green blue 

scatterplot = pd.scatter_matrix(dataframe, 
                               alpha    = 0.3,
                               figsize  = (10,10),
                               diagonal = 'hist', #'kde'
                               color    = colors, 
                               marker   = 'o',
                               grid     = True)

"""
Explanation: 2. Let's plot the dataset!
2.1 Scatter_matrix
Using pandas' scatter_matrix function.
This is similar to R's pairs and makes a quick scatter plot map for the quantitative
features of the dataframe
First we'll make a list named colors as a book-keeper.
Then to visualise the different target categories (i.e. the Species) we'll use the target numerical categorisation (0,1,2) to differently colour each row of the dataframe based on the target variable
End of explanation
"""


dataframe.plot(kind='bar', stacked=True, figsize=(10,8)) #barh

"""
Explanation: There are various plot methods inside pandas.
Now to make the scatter plot matrix
2.2 A bar plot
End of explanation
"""


dataframe['sepal length (cm)'].plot(kind='hist', color='red', alpha=0.3, figsize=(10,8))

dataframe.plot(kind='hist', alpha=0.3, orientation='horizontal', 
               cumulative=True, bins=15, figsize=(10,8))

dataframe.diff().hist(color='k', alpha=0.5, bins=50, figsize=(10,8))
                    # diff() gets the difference of all
                    # rows with row-0 

"""
Explanation: 2.3 Histogram (for one column, for all, or split by diff )
End of explanation
"""


dataframe.plot(kind='box', vert=False, by='X', figsize=(10,8))

"""
Explanation: 2.4 Box plots
Let's remind here that box plot is a convenient way of depicting groups of numerical data and their quartiles. 
The first quartile is the 25th percentile (splits off the lowest 25% of data from the highest 75%), the second quartile (median) is the 50th percentile (cuts the dataset in half) and the third quartile is the upper quartile or the 75th percentile (splits off the highest 25% from the lowest 75%). IRQ (interquantile range) is the difference between upper and lower quartiles IRQ=Q3-Q1).
The line is the median (Q2 or 50%). 
The box represents the IRQ, the from median to highest values is the Q2 to Q3 region, while the lowest is the Q1 to Q2 region. 
```
      _   -> maximum
        |
        |
       ---    -> third quartile
      |   |
      |_|   -> median (Q2)
      |   |
      |   |
       ---    -> first quartile
        |
        |
      _____   -> minimum  
```
 Outliers  --> 3 x IRQ
 Suspected Outliers  1.5 x IRQ
End of explanation
"""


dataframe.plot(kind='area', alpha=0.6, figsize=(10,8))

"""
Explanation: 2.5 Area plots!
End of explanation
"""


dataframe.plot(kind='scatter', x='sepal length (cm)', y='sepal width (cm)', figsize=(10,8))

ax = dataframe.plot(kind='scatter', x='sepal length (cm)', y='sepal width (cm)', 
                    color='DarkBlue', label='Group 1');
dataframe.plot(kind='scatter', x='petal length (cm)', y='petal width (cm)', 
                    color='DarkGreen', label='Group 2' , figsize=(10,8),
                    ax=ax); # ax = ax to overlay
                            # if not set 2 plots made

dataframe.plot(kind='scatter', x='petal length (cm)', y='petal width (cm)', 
               c='sepal width (cm)', s=50, figsize=(10,8))

"""
Explanation: 2.6 Scatter Plots

Defining the x and y axis
plotting two sets with the ax=ax
palette height
End of explanation
"""


dataframe.plot(kind='hexbin', x='petal length (cm)', y='petal width (cm)', 
               gridsize=25, figsize=(10,8))
               

"""
Explanation: 2.7 Hexbin
Hexbin are alternative to scatter plots if the data are too dense to plot individually
End of explanation
"""


series = pd.Series(3 * np.random.rand(4), index=['a', 'b', 'c', 'd'], name='series')
series.plot(kind='pie', figsize=(10,10))

"""
Explanation: 2.8 Pie plots
Pies are good for series
End of explanation
"""


dataframe.ix[:,0:2].plot(kind='kde', figsize=(10,8))
# this is to take all rows for the first two columns of the dataframe (or df.iloc[:,0:2])

"""
Explanation: 2.9 Missing data
Depending on the plot type pandas handle by default the NAs
Plot type | Handling
---|---
Line  | Leave gaps at NA
Line stacked | Fill 0's
Bar | Fill 0's
Scatter | Drop NA
Histogram | Drop NA
Box | Drop NA
Area | Fill 0's
KDE (density) | Fill 0's
Hexbin | Drop NA
Pie | Fill 0
If this is not the wanted behaviour use: fillna() or dropna() before plotting

2.10  Density/KDE Plot
End of explanation
"""


from pandas.tools.plotting import andrews_curves

new_df = dataframe;
new_df['Species']=target_df;
new_df.head(5)

andrews_curves(new_df, "Species")

"""
Explanation: 2.11 Andrews Curves
They are used to plot multivariate data as a large number of curves that are created using the atributes of samples as coefficients for Fourier series. By coloring these curves per class it is possible to visualize data clustering.
It is a good way to visualize structure in high-dimensional data. Each data point $x={x_1 , x_2, ..., x_d}$ defines a finite Fourrier series:
$f_{x}(t) = \frac{x_1}{\sqrt{2}} + x_2 sin(t) + x_3 cos(t) + x_4 sin(2t) + x_5 cos(2t)+...$
This function is plotted for $-\pi < t < \pi$. Therefore, each data point may be viewed as a line between $-\pi$ and $\pi$. The formula can be thought of as the projection of the data point onto the vector 
$(\frac{1}{\sqrt{2}}, sin(t), cos(t), sin(2t), cos(2t),...)$
Each df row is a line per category. On the horizontal axis is the $-\pi < t < \pi$ and on the vertical the $f_x(t)$.
Curves belonging to samples of the same class will usually be closer together and form larger structures!
End of explanation
"""


from pandas.tools.plotting import parallel_coordinates
parallel_coordinates(new_df, "Species")

"""
Explanation: 2.12 Parallel Coordinates
Parallel coordinates is a way of visualising high-dimensional geometry and analysing multivariate data.
To show a set of points in a n-dimensional space, the horizontal axis is split consisting n parallel vertical (typically) lines. A point in the n-dimensional space is represented as a polyline with vertices on the parallel axes; the position of the vertex on the i-th axis corresponds to the i-th coordinate of the point.
This type of visualisation is closely related to time series visualisation, with the difference that it is applied on data that have no time dependance.
Three points must be taken into account when the plot is used for statistical inference:
1. The order of the axis is critical for finding features (many reorderings are done in practice).
2. The rotation of the axis is a translation in the parallel coordinates; if the lines intersected outside the parallel axes, it can be translated between them by rotations (i.e. 180$\circ$).
3. The scaling is necessary since the plot is based on interpolation of consecutive pairs of variables. Thus the variables must be on a common scale and orthogonal to each parallel axis. 
Points that tend to cluster will appear closer together.
End of explanation
"""


from pandas.tools.plotting import lag_plot
lag_plot(new_df)

"""
Explanation: 2.13 Lag plot
Lag plots are used to identify randomness in data. Random data should not exhibit any structure in the lag plot
Non-random structure implies that data are not random.
A lag is a fixed time displacement. For example, given a dataset $Y1,~Y2~...,~Yn$, $Y2$ and $Y7$ have lag $5$ since $7 - 2 = 5$. Lag plots can be generated for any arbitrary lag, although the most commonly used lag is $1$.
End of explanation
"""


from pandas.tools.plotting import autocorrelation_plot
data = pd.Series(0.7 * np.random.rand(1000) +
                 0.3 * np.sin(np.linspace(-9 * np.pi, 9 * np.pi, num=1000)))

autocorrelation_plot(data)

"""
Explanation: 2.14 Autocorrelation Plot
This is to check randomness in series. It computes the autocorrelations for data values at varying time lags. 
If series are random, the autocorrelation should be near zero for any and all time-lag separation.
If time series are non-random then one or more of the autocorrelations will be significantly* non-zero. *
The horizontal lines correspond to the 95% and 99% confidence bands (dashed = 99%)
definitions:


Vertical axis : autocorrelation coefficient: $R_{h}=C_{h}/ C_{0}$, 
where $C_{h}$ is the autocovariance :
$$C_{h} = \frac{1}{N} \sum_{t=1}^{N-h}(Y_{t}-\bar{Y})(Y_{t+h}-\bar{Y})$$
and $$C_{0} = \frac{\sum_{t=1}^{N}(Y_{t}-\bar{Y})^{2}}{N}$$
 $-1 \leq R_{h}\leq 1$ 
N.B. Some times the the autocovariance function is given as 
$$C_{h} = \frac{1}{N-h} \sum_{t=1}^{N-h}(Y_{t}-\bar{Y})(Y_{t+h}-\bar{Y})$$
which has less bias (1/N-h) 


Horizontal Axis: time lag $h$, $h=(1,2,...)$


The above line containes several reference lines :

the middle line is at zero
the other four lines are the 95% and 99% confidence bands. Note that there are two distinct formulas for generating confidence bands.
if the autocorrelation plot is being used to test randomness (no time dependence in data) the following formula is recomended $$CL = \pm\frac{z_{1-a/2}}{\sqrt{N}}$$, where $N$ is the sample size, $z$ the cumulative distribution function of the standard normal distribution and $\alpha$ the the significance level 
if autocorrelation plot is made using the ARIMA model fitting then the the confidence bands are generated by : $$CL= \pm  z_{1-a/2} \cdot \sqrt{ \frac{1}{N}(1+2\sum_{i}^{k}y_{i}^2) } $$ where $k$ is the lag, $N$ the sample size, $z$ the cumulative distribution function of the standard normal distribution and $\alpha$ the significance level.
End of explanation
"""


from pandas.tools.plotting import bootstrap_plot
data = pd.Series(np.random.rand(1000));
    
bootstrap_plot(data, size=50, samples=500, color='grey')    

"""
Explanation: 2.15 Bootstrap plot
Bootstrap plots are made to assess the uncertainty of a statistic metric (mean, median, midrange etc)
To make a bootstrap uncertainty estimate for one metric given a dataset, a subset of the sample of size les or equal to the size of the dataset is generated from the data and the statistic is calculated. This subset is generated with replacement, and thus each data point can be resampled multiple times or not at all. The process is repeated, usually 500 to 1000 times. The computed values for the statistic form an estimate of the sampling distribution of the statistics.
For example, in a sample of 50 values you want to bootstrap the median. You generate a subset of the sample with 50 elements and calculate the median. Repeat this 500 times, so that you have at least 500 values for the median. To calculate the 90% confidence interval for the median, the sample of medians are sorted into ascending order and the value of the 25th median is the lower confidence limit, while the value of 475th median is the upper confidence limit (first and third quartiles).
The plots generated are the series and the histograms for mean, median and mid-range.  For uniformly distributed values, mid-range has the smallest variance.
End of explanation
"""


from pandas.tools.plotting import radviz

radviz(new_df, 'Species')

"""
Explanation: 2.16 RadViz Plot
RadVis is a way to visualise multivariate data; to visualise n-dimensional points into a two dimensional space. In this case, the mapping is not linear. The technique is based on a simple spring tension minimization algorithm.
Imagine $n$ points, $S_1,S_2,...S_n$ arranged to be equally spaced around the cirumference of the unit circle. Now suppose a set of $n$ springs being fixed at one end to each of these points and the other ends to a puck. Finally, assume the stiffness constant (as in Hooke's law) of the j$th$ string in the $x_{ij}$ for one of the data points $i$. 
If the puck is released and allowed to reach equilibrium position, the coordinates of this position, $(u_i, v_j)^{T}$, are the projection in the two dimensional space of the point $(x_{i1}, x_{i2}, ... x_{in})^{T}$ in the $n$-dimensional space. If the $(u_i, v_j)^{T}$ is computed for $i=1,2...n$ and the points plotted, a visualisation of the $n$-dimensional dataset in the two dimensions is achieved.
To understand more about the projection of the $n$-dimensional space into the two dimensional one, consider the forces acting on the puck. When the puck is in equilibrium, there are no resultant forces acting on it (their sum is 0). Denoting the position vectors of $S_1$ to $S_n$ by $\mathbf{S_1}$ to $\mathbf{S_n}$ and putting $\mathbf{u_{i}}=(u_i, v_i)^{T}$ we have:
$$\sum_{j=1,n}=(\mathbf{S}{j}-\mathbf{u}{i})x_{ij} = 0 $$
which when solved for $\mathbf{u}_{i}$ :
$$\mathbf{u}{i} = \sum{j=1,n}w_{ij}\mathbf{S}_j$$
where 
$$ w_{ij} = \left( \sum_{j=1,n} x_{ij} \right)^{-1} x_{ij}$$
This means that for each $i$ (i.e. dataframe row), $\mathbf{u_i}$ is the weighted mean of the $\mathbf{S_j}$'s whose weights are the $n$ variables for case $i$ normalised to unity. 
N.B. This normalisation makes the mapping $\mathcal{R}^{n}\rightarrow \mathcal{R}^{2}$ non-linear.
End of explanation
"""


s = pd.Series(np.random.randn(5), index = ['a','b','c','d','e']) 
                                    # 5 is the # of observations
                                    # only one axis in tuple -> only 1D series

s

s.plot(kind='kde')

"""
Explanation: ----
3. Data Holders
Now let's do see some 'data wrangling' commands with pandas framework. These are to re-shape, select and extract information from a given dataset.
Let's dive a bit into the different data structures
3.1. Series
Series is in fact an 1D array with labels that holds a specific data type (double, int, string etc). The labels (axis labels) are referred to as index. 
To create a Series you need data and an index definition. 


Data can be:

dictionary
ndarray
scalar value (i.e. 5)



Index can be:

any list of strings 



If no index is included in the definition, the indices will be numerical from 0 to len(ndarray)-1.
3.1.1. Series from ndarray
End of explanation
"""


#s = pd.Series(np.random.randn(5,5)) # this returns an exception : Data must be 1-dimensional

"""
Explanation: this does not work for a 2D ndarray
End of explanation
"""


d = {'a':125., 'b':500, 'c':400, 'e':240}; d

s2 = pd.Series(d); s2

s2.plot(kind='bar', alpha=0.8, color='pink', ylim=(0, 1000))

s3 = pd.Series(d, index=['a','b','c','d','e','f']); s3

s3.plot(kind='bar', alpha=0.8, color='cyan', ylim=(0, 1000))

"""
Explanation: 3.1.2. Series from dictionary
Generating from a dictionary, the keys are taken as index.
If extra index argument is given  this the keys values are overwritten for the matching cases, while if the index argument is larger than the len(dict.keys()) then NaN's are created.
End of explanation
"""


s4 = pd.Series(5, index=['a','b','c']); s4

"""
Explanation: So indeed the series are filled with NaN's and the plots with 0's!.

3.1.3. Series from scalar
Now introducing series with scalar input, the values will all be identical.
End of explanation
"""


s2[3]

s2[:4]

s2.median(); s2.mean(); s2.std(); s2.quantile(0.50)

s2[s2>s2.median()]

s2['a']

'a' in s2

"""
Explanation: Series can behave as ndarrays or as dictionaries including all the operations!
For example
End of explanation
"""


s+s2; s2*2; np.exp(s)

s5 = pd.Series(np.random.randn(5), name='something'); s5.name

"""
Explanation: Also vectorised operations can be done. The difference with ndarray is that the result is aligned automatically based on the labels.
End of explanation
"""


d = {'one' : pd.Series([1.,2.,3.], index = ['a','b','c']),
     'two' : pd.Series([1., 2., 3., 4.], index=['a','b','c','d']) }; d

df = pd.DataFrame(d); df

pd.DataFrame(d, index = ['d','b','a']) # rearanging stuff

pd.DataFrame(d, index = ['d','b','a'], columns=['two' , 'three']) # there is no 'three' col

"""
Explanation: 3.2  DataFrames
On the other side, we have DataFrames. A dataframe is like an Excel spreadsheet or a SQL table. Dataframes can accept various many different kinds of inputs

Dictionay of 1D ndarrays, lists, dict or Series
2D numpy.ndarray
structured or record ndarray
series
another DataFrame

Apart from data, the input arguments can be columns and index.
3.2.1 DataFrame from dict of Series or dicts
The result index will be the union of the indexes of the various Series. Nested dicts will be converted to Series first.
End of explanation
"""


df.index

df.columns

"""
Explanation: The index and columns can be accessed directly:
End of explanation
"""


d = {'one' : [1.,2.,3.,4.],
     'two' : [4.,3.,2.,1.]}; d 

pd.DataFrame(d)

pd.DataFrame(d, index=['a','b','c','d'])

"""
Explanation: 3.2.2 DataFrame from dict of ndarrays/lists
End of explanation
"""


x = np.array([(1,2,'Hello'), (2,3,'World') ],
             dtype=[('foo','i4'),('bar','f4'), ('baz', 'S10')])   # structured array

x[0]

x['foo']

x.dtype.names

x[['foo','bar']]  # access multiple fields simultaneously

"""
Explanation: 3.2.3 DataFrame from structured/record arrays
These arrays are made using named indices.
For example
End of explanation
"""


y = np.rec.array([(1,2.,'Hello'),(2,3.,"World")], 
                          dtype=[('foo', 'i4'),('bar', 'f4'), ('baz', 'S10')]) ## record array

y[0]

y['foo']

y.dtype.names

y[['foo','bar']]  # access multiple fields simultaneously

y[1].baz  # access one element

"""
Explanation: 
End of explanation
"""


data = np.zeros((2,), dtype=[('A', 'i4'),('B', 'f4'),('C', 'a10')])

data[:] = [(1,2.,'Hello'), (2,3.,"World")]

pd.DataFrame(data)

pd.DataFrame(data, index=['first', 'second'])

pd.DataFrame(data, columns=['C', 'A', 'B'])

"""
Explanation: So now making a DF from such arrays:
End of explanation
"""


data2 = [{'a': 1, 'b': 2}, {'a': 5, 'b': 10, 'c': 20}]

pd.DataFrame(data2)

"""
Explanation: 3.2.4 DataFrame from list of dicts
Now make a list of dictionaries
End of explanation
"""


pd.DataFrame({('a', 'b'): {('A', 'B'): 1, ('A', 'C'): 2},
                ('a', 'a'): {('A', 'C'): 3, ('A', 'B'): 4},
                ('a', 'c'): {('A', 'B'): 5, ('A', 'C'): 6},
                ('b', 'a'): {('A', 'C'): 7, ('A', 'B'): 8},
                ('b', 'b'): {('A', 'D'): 9, ('A', 'B'): 10}})

"""
Explanation: 3.2.5 DataFrame from dict of tuples
Now make a dict containing tuples:
End of explanation
"""


# Create the csv file
out = open("file.csv", 'write');
s = '''Date,Temperature_city_1,Temperature_city_2,Temperature_city_3,Which_destination\n
20140910,80,32,40,1\n
20140911,100,50,36,2\n
20140912,102,55,46,1\n
20140912,60,20,35,3\n
20140914,60,,32,3\n
20140914,,57,42,2''';
out.write(s);
out.close();

# read it in pandas
df_csv = pd.read_csv("file.csv", sep=','); df_csv

"""
Explanation: 3.2.6 DataFrame from Series
This is including various series for the creation of a DF


4. Dealing with dates and problematic data
Now let's make one dataframe and see what we can do when we have dates and NaN values in it.
End of explanation
"""


# read it in pandas and process dates
df_csv2 = pd.read_csv("file.csv", sep=',', parse_dates=[0]); df_csv2

"""
Explanation: 4.1 Automatic Formating of "date strings"
The read_csv() function has an option that specifyies which column contains dates. Then these dates can be parsed and formated accordingly. 
Just include the argiment parse_dates=[list of columns with dates]
End of explanation
"""


df_nulls = pd.read_csv("file.csv", sep=',', parse_dates=[0]); df_nulls

pd.isnull(df_nulls['Temperature_city_1'])  # looking for nulls in one column

df_nulls['Temperature_city_1'].isnull()   # same thing different format

df_nulls.notnull()   # look for actual values (NON NaN's) in the whole df

"""
Explanation: 4.2 Handle NaNs
There are various ways of dealing with NaN values within pandas. But first one must find them!
4.2.1 Find NaN Values
There are various ways of finding NaN values. Mainly these are revolting around the isnull() and notnull() functions.
End of explanation
"""


print np.nan;
np.nan == None

"""
Explanation: 4.2.1.1 NaN and None
It is very important to remember that NaN is not None :
End of explanation
"""


np.nan == np.nan

"""
Explanation: Also NaN are not comparable, this means that
End of explanation
"""


None == None

"""
Explanation: Pandas/Numpy use this to separate None and NaN, since
End of explanation
"""


# read it in pandas and process dates
df_csv3 = pd.read_csv("file.csv", sep=',', parse_dates=[0]);
df_csv3.fillna(-1)

"""
Explanation: 4.2.2 Fill NaN with fixed value
One may choose to replace NaN with a certain fixed value (fillna(val))
End of explanation
"""


df_csv4 = df_csv3.fillna(-1); df_csv4

"""
Explanation: keep in mind that this does not replace the values in the original DataFrame, but it must be stored in a copy. 
So let's replace and see the values:
End of explanation
"""


df_csv3

"""
Explanation: the df_csv4 has indeed replaced NaN with -1, but what happened of df_csv3 ?
End of explanation
"""


print type(df_csv3.mean(axis=0));
df_csv3.mean(axis=0)

df_csv5 = df_csv3.fillna(df_csv3.mean(axis=0)); df_csv5

"""
Explanation: df_csv3 is the original DF and has not replaced its NaN values.

4.2.2 Fill NaN with mean or Median
One can replace the NaN values with the column mean or median so to minimise the guessing error. This can easily be done by
End of explanation
"""


df_csv6 = df_csv3.dropna(); df_csv6

"""
Explanation: N.B. The argument axis=0 implies the calculation of the means that span the rows so the obtained means that they extend column-wise. The axis=1 spans columns and the row-wise results are obtained.

4.2.5 Drop NaN rows
One might choose to drop the NaN values this can be done by
End of explanation
"""


df_nulls

df_nulls.count()

df_nulls.interpolate().count() # first interpolate the NaN

df_nulls.interpolate()

"""
Explanation: 4.2.6 Interpolate NaN
Series and DataFrames have the interpolate method, which performs linear interpolation (by default) for the missing datapoints
End of explanation
"""


df_nulls.plot(title='no linear interpolation')
df_nulls.interpolate().plot(title='with linear interpolation')

"""
Explanation: to see how this interpolation is done we can plot it
End of explanation
"""


## Let's make some dfs with NaNs
a1 = pd.Series(np.random.randn(3), index = ['e','f','h']);
a2 = pd.Series(np.random.randn(5), index = ['a','c','e','f','h']);
a = {'one': a1, 'two': a2};
dfa = pd.DataFrame(a); 
print dfa

b1 = pd.Series(np.random.randn(2), index = ['e','f']);
b2 = pd.Series(np.random.randn(5), index = ['a','c','e','f','h']);
b3 = pd.Series(np.random.randn(5), index = ['a','c','e','f','h']);
b = {'one':b1, 'two':b2, 'three':b3};
dfb = pd.DataFrame(b);
print dfb

dfa+dfb ## NaN are propagated properly

(dfa+dfb).mean(axis=0) #mean for rows

"""
Explanation: 4.2.5 Calculations with NaNs
Missing values do propagate naturally in operations with Pandas Objects
End of explanation
"""


dfa.cumsum()

dfa.groupby('one').mean()

"""
Explanation: Functions like cumsum() (cumulative sum over requested axis) and cumprod() (cumulative product over requested axis) ignore the NaNs. Similarly the groupby()
End of explanation
"""


f = open('bad_dataset.csv', 'write')
f.write('Val1,Val2,Val3\n')
f.write('1,1,1\n')
f.write('2,2,2,2\n') # 4 columns, 3 expected!
f.write('3,3,3\n')
f.close()

"""
Explanation: 4.3 Reading Bad Input Files
There is also the case - especially in the real world - that the dataset which is loaded is bad or erroeneous. When loading such dataset with load_csv() the program will exit with an error. A workaround is to ignore bad lines by the error_bad_lines option. 
Assume that we have the following bad dataset:
End of explanation
"""


bdf = pd.read_csv('bad_dataset.csv')

"""
Explanation: Now lets load it as always
End of explanation
"""


bdf = pd.read_csv('bad_dataset.csv', error_bad_lines=False) #i.e. "I don't want a parser check"

bdf

"""
Explanation: Notice the CParserError: Error tokenizing data. C error: Expected 3 fields in line 3, saw 4
Now trying with the option:
End of explanation
"""


