# Get ready!

import pandas

"""
Explanation: Working with Numbers in Python
Python Data Analysis Library - Pandas
We've started to grapple with numbers, strings, and lists and how they might be used in text analysis. Sometimes we will use these methods to dig deeper into the text - to enable more productive deep readings. Other times we might want to turn text into numbers, either for "distant reading," or even to perform some statistical analyses on the text.
Today we will learn the basics of the Python library pandas, which is designed for (numerical) data analysis. It is similar to spreadsheets, such as those in Excel, or the dataframe in Stata or R, if you have used those. It is my favoriate place to do data/text munging and basic statistics. There are other libraries for more sophisticated statistics in Python.
Today we will just explore numerical analysis. In the weeks to come we will explore how we can use pandas in the text analysis pipeline.
Learning Goals

Get familiar with the pandas libarary
Learn some basic functions to summarize numerical data
Have a foundation in pandas so we can use it on text as data in the weeks to come

Outline

Introduction to pandas and the dataframe
Dataframe Subsetting
Arithmetic
Summarizing real data using pandas

<a id='pandas'></a>
Pandas
Pandas is a popular and flexible package whose primary use is its datatype: the <i>DataFrame</i>. The dataframe is essentially a spreadsheet, like you would find in Excel, but it integrates seamlessly into a Natural Language Processing workflow and it has a few tricks up its sleeve.
End of explanation
"""


# Create a list of three sub-lists, each with two entries.

square_list = [[100,'high school'],[400,'BA'],[700,'MA']]

# We can slice lists of lists like we do lists
# Question: What do you expect the output to be?

square_list[2]

# Now the interesting stuff. Create a dataframe from that list. 
# The output here should look familiar, like a spreadsheet

pandas.DataFrame(square_list)

# We can create a list for our column labels to add to our dataframe

column_names = ['income', 'education']
row_names = ['Laura', 'Heather', 'KT']


# A-ha!

pandas.DataFrame(square_list, columns = column_names, index=row_names)

# Assign this to a variable

df = pandas.DataFrame(square_list, columns = column_names, index=row_names)
df

# Call up a column of the dataframe

df['income']
df.income

# Make that column into a list

list(df['income'])

# Get the indices for the entries in the column

df['income'].index

# Call up a row from the indices
df.iloc[1]

# Call up a couple of rows, using a list of indices

df.loc[['Laura','Heather']]

# Get a specific entry by calling both row and column

df.loc['Laura']['income']

# Temporarily re-order the dataframe by values in the 'income' column

df.sort_values('income', ascending=False)

# Create a new column

df['age'] = [22,31,35]

# Inspect

df

## EX. Call up the entries (400 and BA) from the middle of the dataframe 'df' individually

## CHALLENGE: Call up both entries at the same time

df.loc['Heather'][['income', 'education']]


"""
Explanation: We have to first introduce a new datatype, lists of lists, or lists with sub-lists. We will use this to create a toy example of a dataset in pandas. Our toy dataset will include a sample of three people, and their weekly income and education level.
End of explanation
"""


# Slice out a column

df['income']

# Evaluate whether each element in the column is equal to 100

df['income']==100

# We can also use evaluation to subset the table. This time we'll use the greater than evaluator.
#df['income']>200
df[df['income']>200]

## EX. Slice 'df' to contain only rows in which 'education' equals 'BA' (remeber how to specify a string!)
df[df['education']=='BA']

"""
Explanation: <a id='subsetting'></a>
DataFrame Subsetting
End of explanation
"""


# Our dataframe

df

# Pandas will produce a few descriptive statistics for each row, but only columns that are numbers

df.describe()

# Multiply entries of the dataframe by 10

df*10

# Add 10 to each entry

df+10

"""
Explanation: <a id='arithmetic'></a>
Arithmetic!
End of explanation
"""


#We can do it if we specify a column

df['income']+10

# Of course our dataframe hasn't changed

df

# What if we just want to add the values in the column?

df['income'].sum()

# We can also perform operations among columns
# Pandas knows to match up individual entries in each column

df['income_age'] = df['income']/df['age']
df

# EX: .sum() adds the values in a column. .mean() calculates the mean value in a column.
#     Find the mean income and the mean age for the dataframe df()

"""
Explanation: We can't do it! Why not?
End of explanation
"""


#we'll assign it to the variable name df again, to keep things consistent
df = pandas.read_csv("../data/nsfg_data1.csv.bz2", compression='bz2', index_col = 0)
#view our data. Note: 'NaN' indicates the value is missing
df

#Typically the first thing we want to do is describe our data.
df.describe()

"""
Explanation: <a id='exercise'></a>
Summarzing Real Data
Now let's read in a dataset containing real world data. The data is a .csv file you should have saved in the 'data' folder. These data are from the National Survey of Family Growth. It's from Cycle 6, the years 2002-2003. The survey contains information on pregnancy and births in the U.S.
First read in the data:
End of explanation
"""


df['insuranc'].value_counts()

"""
Explanation: This describes the columns that contain numbers as values. To summarize the columns that contain strings, we can use the .value_counts() function. This counts the number of rows for each unique value in the specified column.
End of explanation
"""


# The main visualization, or plotting, library in Python is matplotlib
import matplotlib.pyplot as plt
df.hist()
plt.show()

"""
Explanation: We can visualize the distribution of the two number columns by creating a histogram.
End of explanation
"""


#group by the column insuranc, and calculate the mean for each group for the rest of the columns
grouped = df.groupby('insuranc').mean()
grouped

## EX: Create a new grouped by dataframe, called 'grouped_new', that groups the dataframe by pregordr
##     and calculates the mean for the other variables. Print this out.

## EX: Sort this new grouped by dataframe by the age at pregnancy, in descending order
##     so that the oldest average age at pregnancy is at the top.

## EX: Print the subset of the dataframe with only those rows where the pregordr is 15 or greater

## EX: In the original dataframe ('df'), create a new column that contains the age at pregnancy
##   divided by the pregnancy order. Print out the dataframe to see the new column.
##   (Note: There is not a sensible mathematical reason for this that I can think of. This is just to practice pandas.)

"""
Explanation: Quite often when we are analyzing data we want to know the relationship between two variables. Here we may want to know the relationship between whether the respondent had health insurance and how it is related to their age at pregnancy and the birthweight of their baby.
We can easily do this by using the groupby function, which groups all rows that share the same value in a particular column and performs an operation on the rest of the columns. 
Let's find out if the average age at pregnancy and the average birth weight are different for respondents who have no health insurance, private insurance, and public insurance.
End of explanation
"""


