precipitation_df.head()

precipitation_df.tail()

"""
Explanation: Analyze precipitation data
This notebook shows you the basic steps when analyzing precipitation data in a notebook. You will learn how to load a data file to a notebook, access this data file in Object Storage, and then load the content of the data file into a pandas DataFrame. Furthermore, you will learn how to explore and analyze the data in the DataFrame and how to vizualize your results by using the matplotlib library. 
This notebook runs on Python 2 and 3.
Table of contents


Get data


Load data


Access data


Explore data


Plot the results


Download the notebook


Summary


<a id="data_set"></a> 
1. Get data
In this notebook, you will explore and analyze historical annual precipitation data.
The raw precipitation data set is from <a href="http://data.un.org/" target="_blank" rel="noopener no referrer">UNData</a>, an Internet search engine for statistical databases provided by the United Nations Statistics Division. You will use a curated version of this data set in this sample. The precipitation measurements are in million cubic meters.
To get the precipitation data set:
1. Click <a href="https://cdsax.cloudant.com/public-samples/test/precipitation.csv" target="_blank" rel="noopener no referrer">annual precipitation data set</a> to download the data in CSV format. 
    Note: If you use Safari, right-click the link and select Download Linked File. The CSV file is downloaded to your Download folder.            
1. Save the CSV file to your computer.
<a id="load_data"></a>
2. Load data
You can add the CSV file that you downloaded to this notebook by clicking the Data icon on the notebook action bar and dragging the file to the Data pane in the notebook. 
The data file is listed on the Data pane and is saved in the Object Storage instance associated with your project.
In the next step, you will learn how to access files in Object Storage.
<a id="access_data"></a> 
3. Access data
<a href="http://pandas.pydata.org/" target="_blank" rel="noopener no referrer">pandas</a> is a useful Python package for managing structured data. The two primary data structures of pandas are <a href="http://pandas-docs.github.io/pandas-docs-travis/dsintro.html#series" target="_blank" rel="noopener no referrer">Series</a> (1-dimensional) and <a href="http://pandas-docs.github.io/pandas-docs-travis/dsintro.html#dataframe" target="_blank" rel="noopener no referrer">DataFrame</a> (2-dimensional).
In the following steps, you will load the precipitation.csv file into a DataFrame in memory.
To run code cells in a notebook, click Run Cell (&#9658;) in the notebook toolbar.  While the code in the cell is running, a [*] appears next to the cell. After the code has run, the [*] is replaced by a number indicating that the code cell is the Nth cell to run in the notebook.
Insert code to access the data source
To add the code to access the precipitation.csv file which is stored in Object Storage, click the next code cell and select Insert to code>pandas DataFrame on the data file that you loaded in the Data pane in the notebook.
This function inserts the setup and data loading code for you. The setup code includes importing the required packages and defining a helper function. The helper function includes the Object Storage credentials to access the precipitation.csv file. Then, the data in the precipitation.csv file is loaded into a pandas DataFrame.
<div class="alert alert-block alert-info">Note: The DataFrame that is created for you and filled with data is given a generic name. Rename the DataFrame to `precipitation_df` and run the code cell to proceed.</div>

<a id="explore_data"></a> 
4. Explore data
Now that your data is in memory, you can begin to explore and manipulate it.
Show the first five and the last five rows of the data by using the head() and tail() methods.  Run each code cell:
End of explanation
"""


precipitation_df['Country or Area'].values

"""
Explanation: Each row in the table lists:

The country or region where the measurements were taken
The annual precipitation for 1990, and for the years 1995 to 2009

By using the DataFrame API, you can list all countries or areas for which precipitation measurements exist. Run the following cell to list these countries:
End of explanation
"""


precipitation_df = precipitation_df.set_index(precipitation_df["Country or Area"])
precipitation_df.drop(['Country or Area'], axis=1, inplace=True)
precipitation_df.head()

"""
Explanation: Having the names of the countries or areas as a column comes in handy, but it would be more useful to have the names as an index to the rows. To manipulate the DataFrame in this way, run the next cell:
End of explanation
"""


%matplotlib inline 

"""
Explanation: <a id="plot_data"></a> 
5. Plot the results
When you work in notebooks, you can decide how to present your anlysis results and derived information. So far, you have used normal print functions, which are informative. However, you can also show your results in a visual way by using the popular matplotlib package to create plots.
First, render your results as inline charts:
End of explanation
"""


germany = precipitation_df.loc['Germany']
ax = germany.plot(kind='bar', figsize=(8,5), title="Precipitation of %s between 1990 to 2009" % germany.name)
ax.set_ylabel("Precipitation (million cubic meters)")
ax.set_xlabel("Years");

"""
Explanation: Now use pandas wrappers around the matplotlib routines. Because you changed the index of the table rows, you can access the precipitation measurements of a country or area by name. For example, run the following cell to plot the annual precipitation of Germany:
End of explanation
"""


import matplotlib.pyplot as plt, numpy as np

france = precipitation_df.loc['France']
years = france.index
index = np.arange(len(years))  
plt.figure(figsize=(8,5))
bar_width = 0.5
plt.bar(index, france, bar_width, color='b')
plt.xlabel("Year")
plt.ylabel("Precipitation (million cubic meters)")
plt.title("Precipitation of %s between 1990 to 2009" % france.name, y=1.08)
plt.xticks(index + bar_width, years, rotation=90)
plt.show()

"""
Explanation: Another possibility is to import the matplotlib library and use it directly for plotting the precipitation values in France:
End of explanation
"""


precipitation_df["SUM"] = precipitation_df.sum(axis=1)

"""
Explanation: Plot the countries or areas with the highest total precipitation
To get a list of the countries with the highest total precipitation, you have to compute the sum of all annual precipitation. You can do this by using the pandas sum() method. This method adds an  additional column called SUM, which contains the sums of the annual precipitation values to the precipitation_df DataFrame.
End of explanation
"""


precipitation_sorted_df = precipitation_df.sort_values(by="SUM",ascending=False)
top5_sums = pd.Series(precipitation_sorted_df["SUM"].head(5))
top5_sums

"""
Explanation: Next, sort the DataFrame by total precipitation and print the top five countries or areas with the highest total precipitation:
End of explanation
"""


top5_bars = precipitation_sorted_df[years][0:5].transpose()
ax = top5_bars.plot(figsize=(10,8), marker='o', linestyle='-', title="Top 5 Countries with highest Precipitation")
ax.set_xlabel("Years")
ax.set_ylabel("Precipitation (million cubic meters)");

"""
Explanation: Now, plot the top five countries next to each another in a line graph for a better comparison of the recordings:
End of explanation
"""


precipitation_sums = top5_sums
other_sums = precipitation_sorted_df["SUM"][5:].sum()
precipitation_sums["Other"] = other_sums

plt.axis('equal')
plt.title("Annual precipitation percentage",y=1.08)
plt.pie(
    precipitation_sums,
    labels=precipitation_sums.index,
    colors=['blue', 'green', 'red', 'turquoise', 'magenta','yellow'],
    autopct="%1.2f%%",
    radius=1.25);

"""
Explanation: Finally, compare the annual precipitation for the top five countries. China has the highest annual precipitation, followed by Colombia. The line graph also shows that some values are missing, which makes comparing measurements difficult. 
To show the total precipitation for the top five countries in relation to the total precipitation for all countries over the entire time period, use a pie chart. Run the next code cell to create a pie chart by using the matplotlib library:
End of explanation
"""


chile = precipitation_sorted_df.loc['Chile'][:-1]
years = chile.index
chile.plot(kind='bar', figsize=(8,5), title="Precipitation of %s between 1990 to 2009" % chile.name)
plt.xlabel("Year")
plt.ylabel("Precipitation (million cubic meters)");

"""
Explanation: The pie chart shows you that nearly a quarter of the precipitation fell in China, and that more than  half of the precipitation fell in the five countries with the highest annual precipitation.
Plot the countries with a negative annual precipitation trend
Going through each row in the DataFrame and looking at the figures is not a viable solution to determine trends. Plotting a bar chart for each country or area is possible, but inconvenient. 
One way to determine trends in annual precipitation is to fit a line to the data points. Run the next code cell to create a bar chart showing the annual precipitation trend for Chile:
End of explanation
"""


index = chile[chile>0.0].index
fit = np.polyfit(index.astype('int'), chile[index].values,1)
print("slope: " + str(fit[0]))

plt.plot(index, chile[index],'.')
plt.plot(index, fit[0]*index.astype('int') + fit[1], '-', color='red')
plt.title("Precipitation Trend for Chile")
plt.xlabel("Year")
plt.ylabel("Precipitation (million cubic meters)")
plt.show()

"""
Explanation: A line of best fit is a straight line that is the best approximation of the given set of data.
The polyfit() function fits a line through data points. Because the DataFrame you are using is two-dimentional, it returns two values, the slope and the intercept. To determine whether the trend is positive or negative, you need only the slope. And naturally, you have to exclude any data points that correpond to the value 0.0. 
Run the next cells to determine and line plot the trend for Chile:
End of explanation
"""


def compute_slope(row):
    """ This method computes the slope of a fittet line for given data points. 
    For arguments containing less than two valid data points it returns a NaN."""
    slope = np.nan
    years = row[row>0.0].index
    if(len(years)>=2):
        fit = np.polyfit(years.astype('int'), row[years].values,1)
        slope = fit[0]
    return slope

"""
Explanation: For Chile, the line plot shows a positive trend in annual precipitation, despite the fact that during the last couple of years the annual precipitation decreased. 
The next cells show you how to determine the precipitation trends for 91 countries.
Run the next code cell to define the compute_slope() method to compute the slope of a fitted line  between given points:
End of explanation
"""


precipitation_df["TREND"] = precipitation_sorted_df[years].apply(compute_slope, axis=1)
precipitation_df.head()

"""
Explanation: After defining the compute_slope() method, use the pandas apply() method to compute the slope for each row in the DataFrame and add the computed slope value to a new column called TREND:
End of explanation
"""


negative_trends = precipitation_df[precipitation_df["TREND"]<0.0]
negative_trends.index

"""
Explanation: If the value is positive, the annual precipitation trend for the country is positive. If the value is negative, the annual precipitation trend is negative. Values near zero indicate a stable condition. 
To determine the countries with a negative precipitation trend, find the rows in the DataFrame with a negative value in the TREND column:
End of explanation
"""


