# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python [default]
#     language: python
#     name: python3
# ---

# # 1. Introduction
#
# Welcome to the 2nd part of this course - now that (presumably) you have a solid grasp of the principles surrounding Numerical computing in NumPy, we will move on to data management in Python. The most common way to do this is in **tabular** format (i.e in a table) with relational databases. The most commonly used powerful library which provides in-memory database-like data handling is **Pandas**. Pandas is well suited for:
#
# * **Tabular** data with heterogeneously-typed columns, such as in an SQL database or Excel spreadsheet.
# * Ordered and unordered **time-series** data.
# * Arbitrary **matrix** data with row and column labels.
#
# Some of the interesting features include:
#
# * Handling missing data fluently
# * Size mutability
# * Easy-to-use *data alignment*
# * Label-based *slicing*, *fancy indexing* and *subsetting*
# * Intuitive *merging* and *joining* of datasets by label
# * Hierarchical labelling of axes
# * Decent IO tools for importing from an array of different formats
# * Flexible reshaping and *pivoting* of tables

import pandas as pd

# **Pandas** is broken down into two primary classes:
#
# 1. **Series**: think of this as an any-type (templated) unordered array with an index. A generalized *numpy array*.
# 2. **DataFrame**: think of this as a 2-D heterogeneous table with a *Series* for each column.
#
# ## Series

counts = pd.Series([644, 1276, 3554, 154])
counts

# If we don't specify an index, a default sequence of integers (from `np.arange()`) is assigned as the index. A numpy array comprises the values of the *Series*, which the index is another *Pandas* object: 

counts.values

counts.index

# We can assign meaningful labels to the series, as:

foods = pd.Series([644, 1276, 3554, 154], index=['Oranges', 'Apples', 'Melons', 'Pumpkins'])
foods

# A useful way to think of a *Series* is to use **key-value** pairs, i.e input using a dictionary:

# +
food_d = {
    'Oranges': 644,
    'Apples': 1276,
    'Melons': 3554,
    'Pumpkins': 154
}

pd.Series(food_d)
# -

# This can also be achieved via separate lists:

labels = ['Oranges', 'Apples', 'Melons', 'Pumpkins']
counts = [644, 1276, 3554, 154]
pd.Series(dict(zip(labels,counts)))

# ## DataFrame
#
# One of the really nice aspects about Dataframes, particularly in Jupyter notebook, is the automatic HTML/Javascript generated when visualizing tables:

data = pd.DataFrame({'value': [632, 1638, 569, 115, 433, 1130, 754, 555],
                     'patient': [1, 1, 1, 1, 2, 2, 2, 2],
                     'phylum': ['Firmicutes', 'Proteobacteria', 'Actinobacteria', 
                                'Bacteroidetes', 'Firmicutes', 'Proteobacteria',
                                'Actinobacteria', 'Bacteroidetes']})
data

# For most datasets it is impractical to display all the values, there are methods to only view the first $n$ rows: head by default only views the first 5 rows.

data.head()

# We can extract the column names as:

data.columns

# ### Reading and Writing Files
#
# There are a number of powerful functions that can achieve this, for instance if our data is in excel:

titanic = pd.read_excel("titanic.xlsx")
titanic.head()

# We can also extract from csv or any other flat-file k-delimited style format. This can be specified in the 'sep' argument within a call to `read_csv` or `read_table`.
#
# Checking the size of the dataset is a priority:

titanic.shape

# As well as determining the number of missing values from each column:

titanic.isnull().sum()

# We can select a column using the square-bracket notation [] or using direct.dot notation:

titanic.Age
titanic['Age'].head()

# Like NumPy, we can index and select using similar methods:

titanic.Age[:5]

titanic[2:10:2]

# Given that this dataset is by passengers, it would be wise to set PassengerID as the index, as such:

titanic = titanic.set_index("PassengerId")
titanic.head()

# ### Querying, Selection
#
# We can select passengers by the row, using `.loc[]`. Loc can also give us a dataframe and is multidimensional.

titanic.loc[3]

# Or values by including a column term

titanic.loc[3, 'Age']

# We can quickly subset the dataset using boolean operators:
#
# These can be done directly, such as:
#
#     [pandas.Series] < $value
#     
# Or function-chained using pre-defined functions:
#
#     - pandas.Series.lt (less than)
#     - pandas.Series.gt (greater than)
#     - pandas.Series.eq (equals)
#     - pandas.Series.neq (not equal to)
#     - pandas.Series.lte (less than or equal to)
#     - pandas.Series.gte (greater than or equal to)

titanic.Age.lt(30).values

titanic[titanic.Age > 30].head()

# Or select columns between two identified as:

titanic.loc[:3, "Cabin":"Fare"]

# Alternatively, we can index using the absolute *position* using `iloc[]`.

titanic.iloc[1, 2]

titanic.iloc[1]

# We can use the `isin()` method to search if a value or values exist within a Series:

titanic['Port Embarked'].isin(['Cherbourg']).head()

# We can find all the indices where the condition is met, and returns the values that satisfy the condition but retains the shape of the original dataframe, which is crucial when alignment is required:

import numpy as np
x = pd.DataFrame(np.random.rand(5,7))
x.where(x < 0.5)

# Alternatively, instead of just getting NaNs, we could place a value or an array into the 'other' argument of `where`.

x.where(x < 0.5, other=-x)

x.where(x > 0.5, other=lambda y: y**3-1)

# Selection using `query()` feels an awful lot like SQL, which can take raw variables as part of it using @

n_parents = 2
titanic.query("(Age < 25) & ((Pclass == '1st class') | (n_parents == @n_parents))").head()

# #### Aggregation
#
# The toys of NumPy are back in a similar form: max, min, mean, sum etc.

titanic.sum()

titanic.Age.mean()

titanic.describe()

# We could check the correlation between two factors.

titanic.Fare.corr(titanic.Age)

# Or generate the correlation matrix, with variance as the diagonal (=1).

titanic.corr()

titanic.agg(['min','max'])

titanic.agg({'Fare': ['mean','std'], 'Age': ['min', 'max']})

# Or we can apply another operation not found in Pandas but in NumPy, or our own, as:

titanic[['Age','Fare','n_parents','n_siblings']].dropna().apply(np.median)


# +
def age_fare_ratio(x):
    if (x.Fare > 0.):
        return x.Age / x.Fare
    else:
        return 0.

titanic['Age_Fare_rat'] = titanic.apply(age_fare_ratio, axis=1)
titanic.head()
# -

# One of the most powerful forms of aggregation is **groupby**. This allows us to perform an aggregation function not *only on one column*, but on multiple ones, allowing us to control for different factors:

titanic.groupby(['Sex',"Pclass"]).agg(['mean', 'std'])

# #### Sorting, Ranking

titanic.sort_values(by='Age', ascending=False).head(3)

titanic.sort_index(ascending=False).head(3)

titanic.sort_values(by=['n_parents','Fare'], ascending=[False,True]).head()

# We can `rank()` each value relative to the others if desired:

titanic.Fare.rank().head()

# ### Counts
#
# We can count the number of unique values in a column with `value_counts()` - incredibly useful!

titanic.Survived.value_counts()

titanic.Sex.value_counts()

# ### Handling Complex String columns
#
# We may wish to break down the 'name' category into title, first and last names.

titanic.Name.head()

complex_names = titanic.Name.str.extract("(?P<Surname>[a-zA-Z]+),\s(?P<Title>[a-zA-Z]+).\s(?P<Forename>[a-zA-Z]+)",
                         expand=True)
complex_names.head()

# or alternatively, splitting a string by a common character, such as comma
titanic.Name.str.split(" ", expand=True).head()

# make a new titanic with names appended!
titanic = pd.concat([ complex_names, titanic ], axis=1)
titanic.head()

# # Tasks
#
# The titanic dataset is a rich dataset to begin using Pandas with, as it is not too large or cumbersome and has plenty of categorical variables to work with.
#
# ### Task 1.
#
# Select all of the passengers who are under the age of 30 and calculate the mean Fare that they paid, by gender.

# your answer here
titanic.query("Age < 30").groupby("Sex").Fare.mean()

# ### Task 2.
#
# Calculate the number of people that embarked from each port; by sex, survival and class.

# your answer here
titanic.groupby(["Sex","Survived","Pclass","Port Embarked"]).count().Forename


# ### Task 3. 
#
# The **Gini coefficient** is a measure of dispersion usually related to represent an income or wealth distribution between individuals, which always exists between 0 and 1.
#
# Gini coefficients of 0 express perfect equality, whereby coefficients of 1 expresses maximal inequality among values.
#
# The gini coefficient is calculated in a sample, that is indexed in a non-decreasing fashion ($y_i \leq y_{i+1}$), $i=1,2,\dots,n$ as:
#
# $$
# G=\frac{1}{n}\left(n+1-2\frac{\sum_{i=1}^n\left(n+1-i\right)y_i}{\sum_{i=1}^n y_i}\right)
# $$
#
# where $y_i$ refers to a continuous value representing wealth, and $i$ refers to the index.
#
# Write a function `gini` that calculates the gini index for an array, and calculate the Titanics' Fare distribution equality.

def gini(x):
    xl = len(x)
    return (1./(xl - 1))*(xl + 1 - 2*(((x*(xl+1-np.arange(xl))).sum())/(x.sum())))


gini(titanic.Fare.sort_values().dropna().reset_index(drop=True))

StarcoderdataJupyterScriptsDedupFiltered