# load packages we will be using for this lesson
import pandas as pd

"""
Explanation: title: "Data Manipulation in Python"
subtitle: "CU Psych Scientific Computing Workshop"
weight: 1301
tags: ["core", "python"]

Goals of this lesson
Students will learn:

How to group and categorize data in Python
How to generative descriptive statistics in Python

Links to Files
The files for all tutorials can be downloaded from the Columbia Psychology Scientific Computing GitHub page. This particular file is located here: /content/tutorials/python/3-data-manipulation/index.ipynb.
End of explanation
"""


# use pd.read_csv to open data into python
df = pd.read_csv("uncapher_2016_repeated_measures_dataset.csv")

"""
Explanation: 0. Open dataset and load package
This dataset examines the relationship between multitasking and working memory. Link here to original paper by Uncapher et al. 2016.
End of explanation
"""


df.head()

df.shape

df.columns

"""
Explanation: 1. Familiarize yourself with the data
Quick review from data cleaning: take a look at the basic data structure, number of rows and columns.
End of explanation
"""


df = df[["subjNum", "groupStatus", "adhd", "hitRate", "faRate", "dprime"]]
df.head()

"""
Explanation: 2. Selecting relevant variables
Sometimes datasets have many variables that are unnecessary for a given analysis. To simplify your life, and your code, we can select only the given variables we'd like to use for now.
End of explanation
"""


df.describe()

"""
Explanation: 3. Basic Descriptives
Summarizing data
Let's learn how to make simple tables of summary statistics.
First, we will calculate summary info across all data using describe(), a useful function for creating summaries. Note that we're not creating a new object for this summary (i.e. not using the = symbol), so this will print but not save.
End of explanation
"""


df.groupby(["groupStatus"]).mean()

"""
Explanation: 3. Grouping data
Next, we will learn how to group data based on certain variables of interest.
We will use the groupby() function in pandas, which will automatically group any subsequent actions called on the data.
End of explanation
"""


df["adhdF"] = pd.qcut(df["adhd"],q=2,labels=["Low","High"])

"""
Explanation: We can group data by more than one factor. Let's say we're interested in how levels of ADHD interact with groupStatus (multitasking: high or low). 
We will first make a factor for ADHD (median-split), and add it as a grouping variable using the qcut() function in pandas:
End of explanation
"""


df.groupby(["groupStatus","adhdF"]).size()

"""
Explanation: Then we'll check how evenly split these groups are by using groupby() the size() functions:
End of explanation
"""


df.groupby(["groupStatus","adhdF"]).mean()

"""
Explanation: Then we'll calculate some summary info about these groups:
End of explanation
"""


subList = df["subjNum"].unique()
nSubs = len(subList)
nSubs

"""
Explanation: A note on piping / stringing commands together
In R, we often use the pipe %&gt;% to string a series of steps together. We can do the same in python with many functions in a row
This is how we're able to take the output of df.groupby(["groupStatus","adhdF"]) and then send that output into the mean() function

5. Extra: Working with a long dataset
This is a repeated measures ("long") dataset, with multiple rows per subject. This makes things a bit tricker, but we are going to show you some tools for how to work with "long" datasets.
How many unique subjects are in the data?
End of explanation
"""


nTrialsPerSubj = df.groupby(["subjNum"]).size().reset_index(name="nTrials")
nTrialsPerSubj.head()

"""
Explanation: How many trials were there per subject?
End of explanation
"""


avgHR = df.groupby(["subjNum"])["hitRate"].mean().reset_index(name="avgHR")
avgHR.head()

df = df.merge(avgHR,on="subjNum")
df.head()

"""
Explanation: Combine summary statistics with the full data frame
For some analyses, you might want to add a higher level variable (e.g. subject average hitRate) alongside your long data. We can do this by summarizing the data in a new data frame and then merging it with the full data.
End of explanation
"""


