# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: python3-datasci
#     language: python
#     name: python3-datasci
# ---

# Pandas
# ======
#
# Pandas (Python Data Analysis Library)is another a Python library whose purpose is to allow one to do statistical data analysis easily in Python.  Pandas adds a new fundamental data type, the DataFrame to the Python language.  The DataFrame is inspired by the same data type from the R statistical analysis programming envrionment.  DataFrames give us easy ways to load and massage a large data set of values.  The rest of the Pandas library adds methods to do various complex statistical analysis on the data.  Pandas is inspired by, and meant to replace functions
# one would do in R with pure Python.  Though a relatively new library, Pandas has quickly matured.  It still does not cover some of the more complex analysis you could do with R, but for many people it will have everything they need already.  Plus it is written in Python to boot, meaning one can avoid the somewhat unintuitive implementation that R provides.  Pandas is also being designed with an eye towards computational performance.  It is already as good, if not often better than R performance wise on very large data sets.
#
# References
# ----------
#
# * [Pandas web site](http://pandas.pydata.org/)
# * [Python for Data Analysis](http://www.amazon.com/Python-Data-Analysis-Wes-McKinney/dp/1449319793/ref=sr_1_1?ie=UTF8&qid=1377800910&sr=8-1&keywords=python+for+data+analysis) by <NAME>, 2012. (developer of Pandas) 

import numpy as np
import pandas as pd

# Important Features in pandas
# ============================
#
# Loading Data from delimited text files
# --------------------------------------
#
# A simple yet very common data format used by scientists are plain text files, with a delimiter of some kind between each data item.  This basically arranges the data as a table.  By convention, each row of the table represents one observation, or data point.  And each column of the table represents a feature or dimension that was measured or observed.  A common delimiter between each feature is to use a comma ',', though you will see files that use a blank space ' ', a tab '\\t' or sometimes something else to delimit fields.  Pandas allows you to easily load even very large plain text delimited files for analysls.
#
# Here is an example of a comma separated data file (from [Luis Zaman's](http://research.luiszaman.com/) digital parasite data set):

# !head ../data/parasite_data.csv

# !tail ../data/parasite_data.csv

# Notice that some of the rows at the end have a blank space in the first column.  We will pass pandas a parameter to tell it how to treat these values.

exp_df = pd.read_csv("../data/parasite_data.csv", na_values=[" "])

print(exp_df)

# Notice that pandas automatically names the column features, using the 1st line from the data file (known as the header).  Notice also that the first feature, Virulance, is missing 50 values (it had 300 non-null values, while the other features had 350 values.

# Accessing data in pandas DataFrames
# -----------------------------------
#
# We can directly access any column and row by indexing the DataFrame (kinda like a dictionary):

print(exp_df['Virulence'])

print(exp_df['ShannonDiversity'][12])

# We can also access all of the values in a column meeting a certain criteria:

print(exp_df[exp_df['ShannonDiversity'] > 2.0])

# NumPy use of pandas DataFrame data
# ----------------------------------
#
# Another powerful feature of the pandas DataFrame is that they are fully integrated with NumPy.  

np.sort(exp_df['ShannonDiversity'])

# Blank/Omitted data
# ==================
#
# One of the toughest preprocessing steps that is needed in analyzing a real data step is trying to identify and figure out how to handle missing data.  Pandas gives great support for identifying and specifying how to handle these missing values.

print(exp_df[np.isnan(exp_df['Virulence'])])

# DataFrame methods automatically ignore NA/NaN values (which is usually the right thing to do).

print("Mean virulence across all treatments:", exp_df['Virulence'].mean())

# Although we can filter out NA/NaN values if we need to, to for example pass to other libraries that don't handle these so well.  Notice that the following has dropped those 50 rows that had null Virulence values.

print(exp_df.dropna())

# Or if needed, we can instead replace all NA/NaN entries with a valid (default) value.

print(exp_df.fillna(0.0)['Virulence'])

# Some Simple Summary Statistics with Pandas
# ------------------------------------------
#
# Lets look at some very simple summary statistics with pandas.  Conveniently DataFrames have all kinds of built-in functions to perform standard operations on them en masse: add() sub() mul() div() mean() std() etc.

print("Mean Shannon Diversity w/ 0.8 Parasite Virulence = ", exp_df[exp_df['Virulence'] == 0.8]['ShannonDiversity'].mean())
print("Variance Shannon Diversity w/ 0.8 Parasite Virulence = ", exp_df[exp_df['Virulence'] == 0.8]['ShannonDiversity'].var())

# Some More Examples
# ------------------
#
# The following are more examples of how you can manipulate pandas DataFrames (taken from the pandas [10-minute tutorial](http://pandas.pydata.org/pandas-docs/stable/10min.html))

dates = pd.date_range('20130101', periods=6)
print(dates)
df = pd.DataFrame(np.random.randn(6,4), index=dates, columns=list('ABCD'))
print(df)

df2 = pd.DataFrame({'A': 1.0,
                    'B': pd.Timestamp('20130102'),
                    'C': pd.Series(1, index=range(4), dtype='float32'),
                    'D': np.array([3] * 4, dtype='int32'),
                    'E': 'foo'})
df2

df2.dtypes

df.head()

df.tail(3)

df.columns

df.values # notice, the easiest way to turn whole pandas DataFrame into a NumPy array with same rows/columns

# Describe shows a quick statistics summary of your data:

df.describe()

# Sort data by values of some column

df.sort_values(by='B')

# Select a single column, which yields a Series, equivalent to df.A

df['A']

# Copy a column values into a NumPy array, for passing to a library for example:
#

col_A = df['A'].values
print(col_A)
print(type(col_A))

# Select rows of a DataFrame using slices.

print(df[1:3])
print(df['20130103':'20130105'])

# Selecting multiple columns.  Unfortunately

df.loc[:,['A','B']]

# Getting particular columns and particular rows

df.loc['20130104':'20130106', ['C', 'A']]

# Selecting only particular rows by some condition.

df[df.A > 0]

# # Versions

# +
import sys
sys.path.append("../src") # add our class modules to the system PYTHON_PATH

from ml_python_class.custom_funcs import version_information
version_information()

StarcoderdataJupyterScriptsDedupFiltered