import sys

sys.version

import warnings
warnings.simplefilter('ignore', FutureWarning)

from pandas import *

show_versions()

"""
Explanation: Table of Contents
<p><div class="lev1 toc-item"><a href="#Project-3:-GDP-and-life-expectancy" data-toc-modified-id="Project-3:-GDP-and-life-expectancy-1"><span class="toc-item-num">1&nbsp;&nbsp;</span>Project 3: GDP and life expectancy</a></div><div class="lev2 toc-item"><a href="#Env" data-toc-modified-id="Env-11"><span class="toc-item-num">1.1&nbsp;&nbsp;</span>Env</a></div><div class="lev2 toc-item"><a href="#Getting-the-data" data-toc-modified-id="Getting-the-data-12"><span class="toc-item-num">1.2&nbsp;&nbsp;</span>Getting the data</a></div><div class="lev2 toc-item"><a href="#Cleaning-the-data" data-toc-modified-id="Cleaning-the-data-13"><span class="toc-item-num">1.3&nbsp;&nbsp;</span>Cleaning the data</a></div><div class="lev2 toc-item"><a href="#Transforming-the-data" data-toc-modified-id="Transforming-the-data-14"><span class="toc-item-num">1.4&nbsp;&nbsp;</span>Transforming the data</a></div><div class="lev2 toc-item"><a href="#Combining-the-data" data-toc-modified-id="Combining-the-data-15"><span class="toc-item-num">1.5&nbsp;&nbsp;</span>Combining the data</a></div><div class="lev2 toc-item"><a href="#Calculating-the-correlation" data-toc-modified-id="Calculating-the-correlation-16"><span class="toc-item-num">1.6&nbsp;&nbsp;</span>Calculating the correlation</a></div><div class="lev2 toc-item"><a href="#Showing-the-data" data-toc-modified-id="Showing-the-data-17"><span class="toc-item-num">1.7&nbsp;&nbsp;</span>Showing the data</a></div><div class="lev2 toc-item"><a href="#Conclusions" data-toc-modified-id="Conclusions-18"><span class="toc-item-num">1.8&nbsp;&nbsp;</span>Conclusions</a></div>

# Project 3: GDP and life expectancy

by Michel Wermelinger, 27 August 2015, updated 5 April 2016

This is the project notebook for Week 3 of The Open University's [_Learn to code for Data Analysis_](http://futurelearn.com/courses/learn-to-code) course.

Richer countries can afford to invest more on healthcare, on work and road safety, and other measures that reduce mortality. On the other hand, richer countries may have less healthy lifestyles. Is there any relation between the wealth of a country and the life expectancy of its inhabitants?

The following analysis checks whether there is any correlation between the total gross domestic product (GDP) of a country in 2013 and the life expectancy of people born in that country in 2013.

## Env
End of explanation
"""


YEAR = 2013
GDP_INDICATOR = 'NY.GDP.MKTP.CD'
LIFE_INDICATOR = 'SP.DYN.LE00.IN'

gdpReset = read_csv('WB GDP 2013.csv')

gdpReset.head()

lifeReset = read_csv('WB LE 2013.csv')

lifeReset.head()

"""
Explanation: Getting the data
Two datasets of the World Bank are considered. One dataset, available at http://data.worldbank.org/indicator/NY.GDP.MKTP.CD, lists the GDP of the world's countries in current US dollars, for various years. The use of a common currency allows us to compare GDP values across countries. The other dataset, available at http://data.worldbank.org/indicator/SP.DYN.LE00.IN, lists the life expectancy of the world's countries. The datasets were downloaded as CSV files in March 2016.
End of explanation
"""


len(gdpReset)

len(lifeReset)

gdpCountries = gdpReset[34:].dropna()
lifeCountries = lifeReset[34:].dropna()

len(gdpCountries)

len(lifeCountries)

gdpCountries.head()

lifeCountries.head()

"""
Explanation: Cleaning the data
Inspecting the data with head() and tail() shows that:

the first 34 rows are aggregated data, for the Arab World, the Caribbean small states, and other country groups used by the World Bank;
GDP and life expectancy values are missing for some countries.

The data is therefore cleaned by:
1. removing the first 34 rows;
- removing rows with unavailable values.
End of explanation
"""


def roundToMillions (value):
    return round(value / 1000000)

def usdToGBP (usd):
    return usd / 1.564768

GDP = 'GDP (£m)'
gdpCountries[GDP] = gdpCountries[GDP_INDICATOR].apply(usdToGBP).apply(roundToMillions)
gdpCountries.head()

"""
Explanation: Transforming the data
The World Bank reports GDP in US dollars and cents. To make the data easier to read, the GDP is converted to millions of British pounds (the author's local currency) with the following auxiliary functions, using the average 2013 dollar-to-pound conversion rate provided by http://www.ukforex.co.uk/forex-tools/historical-rate-tools/yearly-average-rates.
End of explanation
"""


COUNTRY = 'country'
headings = [COUNTRY, GDP]
gdpClean = gdpCountries[headings]
gdpClean.head()

"""
Explanation: The unnecessary columns can be dropped.
End of explanation
"""


LIFE = 'Life expectancy (years)'
lifeCountries[LIFE] = lifeCountries[LIFE_INDICATOR].apply(round)
headings = [COUNTRY, LIFE]
lifeClean = lifeCountries[headings]
lifeClean.head()

"""
Explanation: The World Bank reports the life expectancy with several decimal places. After rounding, the original column is discarded.
End of explanation
"""


gdpVsLife = merge(gdpClean, lifeClean, on=COUNTRY, how='inner')
gdpVsLife.head()

"""
Explanation: Combining the data
The tables are combined through an inner join on the common 'country' column.
End of explanation
"""


from scipy.stats import spearmanr

gdpColumn = gdpVsLife[GDP]
lifeColumn = gdpVsLife[LIFE]
(correlation, pValue) = spearmanr(gdpColumn, lifeColumn)

print('The correlation is: {}\nAnd p-value is: {}\n'.format(correlation, format(pValue, ".15f")))

if pValue < 0.05:
    print('It is statistically significant.')
else:
    print('It is not statistically significant.')

"""
Explanation: Calculating the correlation
To measure if the life expectancy and the GDP grow together, the Spearman rank correlation coefficient is used. It is a number from -1 (perfect inverse rank correlation: if one indicator increases, the other decreases) to 1 (perfect direct rank correlation: if one indicator increases, so does the other), with 0 meaning there is no rank correlation. A perfect correlation doesn't imply any cause-effect relation between the two indicators. A p-value below 0.05 means the correlation is statistically significant.
End of explanation
"""


%matplotlib inline
gdpVsLife.plot(x=GDP, y=LIFE, kind='scatter', grid=True, logx=True, figsize=(16, 10));

"""
Explanation: The value shows a direct correlation, i.e. richer countries tend to have longer life expectancy, but it is not very strong.
Showing the data
Measures of correlation can be misleading, so it is best to see the overall picture with a scatterplot. The GDP axis uses a logarithmic scale to better display the vast range of GDP values, from a few million to several billion (million of million) pounds.
End of explanation
"""


# the 10 countries with highest GDP
gdpVsLife.sort_values(GDP).tail(10)

# the 10 countries with lowest GDP
gdpVsLife.sort_values(GDP).head(10)

# the 10 countries with highest life expectancy
gdpVsLife.sort_values(LIFE).tail(10)

# the 10 countries with lowest life expectancy
gdpVsLife.sort_values(LIFE).head(10)

"""
Explanation: The plot shows there is no clear correlation: there are rich countries with low life expectancy, poor countries with high expectancy, and countries with around 10 thousand (10<sup>4</sup>) million pounds GDP have almost the full range of values, from below 50 to over 80 years. Towards the lower and higher end of GDP, the variation diminishes. Above 40 thousand million pounds of GDP (3rd tick mark to the right of 10<sup>4</sup>), most countries have an expectancy of 70 years or more, whilst below that threshold most countries' life expectancy is below 70 years. 
Comparing the 10 poorest countries and the 10 countries with the lowest life expectancy shows that total GDP is a rather crude measure. The population size should be taken into account for a more precise definiton of what 'poor' and 'rich' means. Furthermore, looking at the countries below, droughts and internal conflicts may also play a role in life expectancy.
End of explanation
"""


# Some experiments with the data
BRICS_LABEL = ['Brazil', 'Russian Federation', 'India', 'China', 'South Africa']

bricsGdpVsLife = gdpVsLife.loc[gdpVsLife['country'].isin(BRICS_LABEL)]
bricsGdpVsLife

bricsGdpVsLife.plot(x=GDP, y=LIFE, kind='scatter', grid=True, figsize=(16, 10));

"""
Explanation: Conclusions
To sum up, there is no strong correlation between a country's wealth and the life expectancy of its inhabitants: there is often a wide variation of life expectancy for countries with similar GDP, countries with the lowest life expectancy are not the poorest countries, and countries with the highest expectancy are not the richest countries. Nevertheless there is some relationship, because the vast majority of countries with a life expectancy below 70 years is on the left half of the scatterplot.
Using the NY.GDP.PCAP.PP.CD indicator, GDP per capita in current 'international dollars', would make for a better like-for-like comparison between countries, because it would take population and purchasing power into account. Using more specific data, like expediture on health, could also lead to a better analysis.
End of explanation
"""


