# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: python3-datasci
#     language: python
#     name: python3-datasci
# ---

# # Pandas Introduction
#
#
# Pandas (Python Data Analysis Library) is a Python library whose purpose is to
# allow one to do statistical data analysis easily in Python.
# Pandas adds a new fundamental data type, the `DataFrame` to the Python
# language.  The `DataFrame` is inspired by the data type with the same name
# from the R statistical analysis programming envrionment.  `DataFrames`
# give us easy ways to load, clean, analyse and massage a large data
# set of values. The rest of the Pandas library adds methods to do
# various complex statistical analysis on the data.  
#
# Pandas is inspired by, and meant to replace functions one would do in
# R with pure Python.  Though a relatively new library, Pandas has
# quickly matured. Pandas is  designed with an eye towards computational performance.  It is already as good, if not often better than R
# performance wise on very large data sets.
#
# **References**
#
# - [Pandas web site](http://pandas.pydata.org/)
# - [Python for Data Analysis](http://www.amazon.com/Python-Data-Analysis-Wes-McKinney/dp/1449319793/ref=sr_1_1?ie=UTF8&qid=1377800910&sr=8-1&keywords=python+for+data+analysis) by <NAME>Kinney, 2012. (developer of Pandas) 
# - [Pandas 10 minutes tutorial](https://pandas.pydata.org/pandas-docs/stable/getting_started/10min.html)

# By convention we import pandas using `pd` as an alias name.  Pandas 
# `DataFrame` objects work in conjunction with NumPy `ndarray` arrays,
# and we will be using both in the following examples, so lets import
# both of the needed libraries in the usual manner.

import numpy as np
import pandas as pd

# The `DataFrame` object is the main object added by and implemented from
# the Pandas library.  A `DataFrame` is basically a table of data, with
# rows representing samples of data or subjects/individuals of a sampled
# population.  The columns of a Pandas `DataFrame` represent what we
# call in machine learning the individual **features** or **attributes**
# of the data set.  Each column in Pandas is represented by what is
# known as a `Series` object.  So a `DataFrame` is a collection of `Series`
# that represent the columns or features of the data set.  All of the
# `Series` or columns in the `DataFrame` have to be of the same length, so
# that each attribute has 1 attribute for each row of the `DataFrame`.
#
# One of the main differences between a Pandas `DataFrame` and a 2-dimensional
# table in a NumPy array is that the `Series` attribute columns in a `DataFrame`
# can be of different types, unlike in a NumPy array where all data in all
# columns must be of the same type.  
#
# The other major difference that a Pandas `DataFrame` provides is that 
# all `Series` column attributes can be accessed with an attribute
# name.  The rows and columns of a `DataFrame` can be accessed by
# index values (and sliced) in ways we have seen for Python series and for
# the NumPy array type.  But `DataFrame` `Series` columns can also be
# accessed by and sliced by their attribute names.  This has become a 
# common way to think of and organize a related data set, with rows
# representing samples, and named attribute columns the data, that can
# be accessed by these attribute names.

# # Series 
#
# Lets start first by creating some `Series` attributes by hand.

freq = pd.Series([1, 3, 5, np.nan, 6, 8], name='frequency')
freq

freq.dtype

freq.name

freq.size

freq.shape

freq.ndim

freq.hasnans

# A series represents a column of attribute data in a data set.  The reference API for a 
# [pands Series](https://pandas.pydata.org/pandas-docs/stable/reference/series.html)
#
# A series
# can have a name that we will use to identify it, like we gave it here.
# All of the values in a series will be of the same type.  As shown here,
# the type will default to floats unless you explicitly request a type
# or all values can be interpreted as a different type.
#
# A `Series` can be accessed by index, like a Python sequence.

freq[0]

freq[:3]

freq[1::2]

# # DataFrame
#
# A `DataFrame` is a collection of named `Series` objects, where all `Series`
# objects are of the same size and represent the column attributes of the
# data set.
#
# There are many ways to construct a `DataFrame` by hand.  
#
# For example, on way that emphasize the series nature of the attributes and
# the ide that each series has a name is to initialize the `DataFrame`
# with a dictionary of `Series` or other items like NumPy arrays or 
# other sequences of the same size.

# +
data_dict = {
    'A': 1., # not a series or even of the correct size, scalars are repeated
    'B': pd.Timestamp('20200704'), # also only a single value, will be repeated
    'C': pd.Series(1, index=list(range(4)), dtype='float32'), # series of 4 floats
    'D': np.array([3] * 4, dtype='int32'), # 4 integers as a numpy array
    'D': pd.Categorical(['test', 'train', 'test', 'train']), # a categorical variable
    'F': pd.Series([1, 3, 5, np.nan]), # our freq Series from first example
    'G': 'foo', # a string
}

df = pd.DataFrame(data_dict)
# -

df

df.columns

df.index

df.dtypes

df.ndim

df.shape

df.size

df.values

# The reference API for a
# [pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html)
#
#
# We showed some of the attributes you can access for a data frame, like the column names,
# the `dtypes` data types for each column, the `DataFrame` shape, etc.  
#
# The `df.values` actually creates and returns a NumPy array of the `DataFrame`.  As you can see here,
# if you have values of different types, especially things like strings, the resulting array will have to 
# be of the general `object` type.  But if your `DataFrame` (or some columns you select) are all
# numerical, the resulting NumPy array can be of a numerical type.
#
#
# Notice that when we create the DataFrame, the number of rows ended up being 4.  Some of the attributes
# in our data dictionary were scalar values (only a single value).  Scalar values will simply get repeated
# the number of times needed so that their `Series` size matches the number of rows or attribute size of
# the `DataFrame`.  All attribute series will be expanded to the largest series size when creating
# a `DataFrame`.  Scalar or single values will be repeated, but `Series` that are too small will get filled
# in with missing `NaN` values.

data_dict = {
    'A': pd.Series([1, 2, 3, 4, 5]), # only have 5 items for this attribute
    'B': pd.Series([1.0, 2.0, 3.0, 4.0, 5.0, 6.0]),
}
df2 = pd.DataFrame(data_dict)

df2

# One other quick thing to mention, each column `Series` of a `DataFrame` is also accessible as a 
# member attribute of the `DataFrame.

df.B

df.F

df2.B

# ## Viewing Data
#
# Lets create a bigger data frame and fill it in with some random values.
# Here we also create an index for our `DataFrame`.  By default an integer index starting
# at 0 is created to index the rows of a `DataFrame`.  But you can have a more complex
# value as a row index, as long as all index values are unique.

dates = pd.date_range('20200701', periods=60) # 60 days starting July 1 2020
dates

# data frame with 60 rows, indexed by date, and 26 columns all filled with random values
data = np.random.randn(60, 26)
columns = list('ABCDEFGHIJKLMNOPQRSTUVWXYZ')
df = pd.DataFrame(data, index=dates, columns=columns)

df.shape

# `DataFrame` objects have many member methods to access, view and manipulate the data.
# Again check the API reference for a full list.
#
# Get the top and bottom rows of the frame:

# shows first 5 by default
df.head()

# last 3 sample rows
df.tail(3)

# We already demontrated you can access any column attribute by name.  You can also access the index:

df.index

df.F

# The `to_numpy()` member function is what is called when you access the `.values` member attribute, so you
# can use either to convert the `DataFrame` into a NumPy array.  
#
# Notice here, since all columns are numerical, the resulting NumPy array has a float dtype.

df.to_numpy()

df.to_numpy().dtype

df.values

df.values.dtype

# Note that `to_numpy()` and `.values` do not include the column names nor the index, as these
# are features particular to `DataFrames` and are not how NumPy arrays are organized or accessed.

# ## Selecting Data
#
# You can slice data frames rows by numerical indexes, like NumPy arrays.

# first 3 rows
df[0:3]

# You can also select columns by their label.  And you can essentially use fancy indexing, giving a list
# of column names to select.  Though note the syntax here, it may surprise you if you were expecting 
# access by row,col as with a NumPy array.

df['C']

# as we already showed, accessing by indexing the attribute name is equivalent to getting the attribute
df.C

# you can give a list of column names, which you cannot do using the attributes
df[ ['D', 'Q', 'P'] ]

# However the Pandas `DataFrame` does not support full slicing by an index number for rows and columns.
# This is because of the nature of how the `DataFrame` is organized.  There is a way to do the following,
# but you need a slightly different syntax.
#
# For example, if you wanted to select columns by index, you may have thought to try:

# the following is not supported syntatically by a DataFrame
try:
    df[:, 0:3]
except TypeError:
    print('TypeError is generated, this type of slicing syntax is not supported for DataFrame')

# In some sense it is understandable, even admirable, if you thought and expected to be able to 
# apply what you learned from slicing a NumPy array to be able to do the same thing with a Pandas
# `DataFrame`.
#
# However, first of all, if you find yourself wanting to try and do a lot of selection and slicing
# of the attributes of a `DataFrame` by their column index number, then you are not quite grasping 
# what a `DataFrame` is and how it is meant to be used.  We primarily think of a `DataFrame` as a 
# collection of named attributes.  So most of the time we will be using the column attribute names to
# select a column or multiple columns from a data frame for some manipulations.  Thus the
# only default slicing supported with the top level `[]` is either the rows by an index number, or
# column attributes by name or by a list of names.
#
# You can get full integer based indexing and slicing on a dataframe using the `.iloc[]` member
# accessor.  This supports slicing rows and columns by index, just like a NumPy array.

# this is how you get the first 3 columns by index number, including all rows, from a DataFrame
df.iloc[:, 0:3]

# select last 3 columns only for the first 5 rows
df.iloc[:5, -3:]

# But like we just mentioned, if you find yourself using `.iloc[]` a lot, you are probably misusing and/or
# misunderstanding the purpose and uses of the Pandas `DataFrame`.
#
# There is also a `.loc[]` member function which is meant to be used to access a `DataFrame` primarily by
# labels.  For example, we didn't mention, but if the index differs from an integer based index starting at 0,
# you can't use the previous basic `[]` to access by index, e.g. this doesn't work

try:
    df[dates[0]]
except KeyError:
    print('KeyError is generated because accessing rows by an index name/label is not supported with []')

# But you can use `.loc[]` to access complex index types, singly or by slices, e.g.:

# all rows for the 30th date index, which was July 31
df.loc[dates[30]]

# an example of slicing a selection of rows by date indexes
df.loc[ dates[5]:dates[15] ]

# the slicing by labels can be pretty sophisticated, it can analyze a string and convert
# to a series like for this
# date
df.loc[ '20200706' : '20200716' ]

# Likewise you can use `.loc[]` to access columns by an attribute name, though this is eqivalent to using `[]`

# get all values in column L
df.loc[:, 'L']

# previous is same as
df['L']

# get a slice of columns by their attribute name
df.loc[:, 'L':'P' ]

# Though one thing to note here that I'm not sure I really agree with.  But when you slice by the
# attribute names, the last attribute is included in the slice.  Thus we ended up with attribute
# column `P` in the last example.  This differs from integer index slices that go up to but do not
# include the last item in the slice.
#
# This also shows that, though the columns are accessed by name like keys to a dictionary, there is still
# an order to the columns that is maintained by the `DataFrame`, unlike dictionary keys.  You can
# change the order of the attributes in a `DataFrame` if needed.

# +
cols = df.columns.tolist()
#cols

# put M and X in front
cols.remove('M')
cols.remove('X')
cols = ['M', 'X'] + cols

#cols
# -

df = df[cols]
df

# ## Boolean Indexing
#
# Boolean indexing is supported, and is a very common operation to do to select a subset of samples based
# on the feature values of 1 or more attribute columns.
#
# For example, lets get only the rows where column 'M' (now our first column) have values greater than 2.

df[ df['M'] > 1 ]

# The result of `df['M'] > 2` is an array (actually a Series here) of boolean results.  And if this boolean 
# sequence is used as in index, only the rows where the result was true will be selected and others are filtered
# out.
#
# We can combine boolean expressions using logical operators to create more complex conditions for 
# selection.

df[ (df['M'] > 1) & (df['X'] < -1) ]

# +
# if the boolean expression begins to get too complex, I often break this up into multiple steps, e.g.
# create a mask to find rows where M is greater than 1 and also where attribute X is less than -1
mask = (df['M'] > 1) & (df['X'] < -1)

# now select rows that match the mask
df[mask]
# -

# The `isin()` member method of a `Series` can be useful to select categorical attributes when we want more
# than one attribute.  For example

# +
# need a function from the random library so can randomly shuffle our list
import random 

# add a categorical attribute, need 60 strings, and we will shuffle them at random
category = ['one'] * 5 + ['two'] * 50 + ['three'] * 5
random.shuffle(category)
# category
# -

# add the new category attribute to our example dataframe
df2 = df.copy()
df2['cat'] = category

# select only rows in category 'one' or three
mask = df2['cat'].isin(['one', 'three'])
df2[mask].loc[:, 'Y':'cat']

# ## Adding and Setting Data
#
# If you were paying attention, we just showed an example of appending a whole new column to a 
# `DataFrame`.  If you have a list, sequence or NumPy array of the same number of elements as the
# rows of your data frame, you can simply create a new column and assign it the new list of
# attributes.
#
# We can likewise create subsets of data frames using the methods to slice and select like we have
# shown before.  Lets create a new data frame using our last mask, and add a a few more 
# attributes.

# we will actually extract the previous DataFrame to a new one
df3 = df2[mask].loc[:, 'Y':'cat']
df3

# +
# add a scalar value, the value is repeated as before
df3['scalar'] = 5

# add a numpy array of ints
values = np.arange(10)
df3['values'] = values

# add another category
cat2 = ['pos', 'neg', 'pos', 'pos', 'neg', 'neg', 'pos', 'pos', 'pos', 'neg']
df3['cat2'] = cat2

df3
# -

# We can change any particular value or values.  If you select a value or values
# in a `DataFrame` and put them on the left hand side of an assignment, then you
# change the value(s).

# select a single value, the last value in Y make it 0
df3.iloc[-1, 0] = 0
df3

# +
# change middle 3 of cat to two
df3.loc[ '20200726':'20200802', 'cat'] = 'two'

df3

# +
# change all of the values attributes to new values newvalue = value^2 + 3*value - 5
df3['values'] = df3['values']**2.0 + 3.0 * df3['values'] - 5

df3

# +
# make all of cat2 positive
df3['cat2'] = 'pos'

df3
# -

# # Loading Data
#
# So far we have used some made up `DataFrame`s with random data to illustrate some of 
# the basic operations you can do with them.
#
# Of course for real data analysis we won't usually be making up data, but our data will be collected
# from sampling or from experiments.  The data will be saved in some sort of relatively permanent
# electronic storage, like a file, database, excel spreadsheet,  HDF5 store, or other method,
# from which we want to load it for processing. You can references the
# [Pandas input/output](https://pandas.pydata.org/pandas-docs/stable/reference/io.html)
# API section for a list of all of the supported methods for getting data into an in-memory
# Pandas `DataFrame`.
#
#
# In this class we will mostly read small to medium-ish sized example data sets stored in
# simple flat-file comma separated values (CSV) format.  This format for a data file is pretty
# common for many data sets that comfortably fit into normally sized computer memory for
# analysis. 
#
# The CSV format is really just a layout of a table of information.  Each line in a CSV file is a row
# of data.  The attributes, or columns, are separted by some delimiter character.  Most commonly a
# comma ",", but you can use tabs, whitespace, or other special delimiters (in which case I guess it would
# technically be a tab separated values, etc. file).
#
# Take a look at the `parasite_data.csv` comma separated values file in the data subdirectory of
# our class repository.  We will load this file into a `DataFrame` and do a bit of data
# exploration on the data therein.  Notice that the first line (row) of the file is actually the 
# attribute name for the column.  Also there is no particular column that is a unique index for this
# data, so we will let pandas create a default index for us, starting numbering at 0 for each row
# of the data.
#
# To load a comma separated values file, we use the `pd.read_csv()` Pandas library function, which
# tries to construct and return a Pandas `DataFrame` of the file data.

# by default the first row will be used for the attribute labels, and it will assume a comma with no extra
# whitespace as the delimiter between attributes for the file
parasite = pd.read_csv('../../data/parasite_data.csv')

# If we didn't peek into the file first, and if it loaded successfully, we can start to try and understand it
# by looking at the column names, types, size of the data, etc. that we got from loading it.

parasite.columns

parasite.shape

parasite.dtypes

parasite.head()

# Hmm... the data type of the first attribute was object, which is kind of the default when Pandas can't figure
# out what to do with it, though the first values look like floats, and the attribute name would suggest
# this should be a attribute with some type of numerical or categorical interpretation.
#
# But in any case we see that this data set contains 350 samples.  The index created for the rows thus
# ranges from 0 to 349.  We have 3 attributes.  
#
# We might first want to use some summary `DataFrame` functions to help us understand the data, though 
# until we do something about the representation of the `Virulence` attribute, describing or
# summarizing that attribute won't be of much use.
#
# We can start with the `describe()` member function.

parasite.describe()

# `describe()` will summarize numerical attributes by counting them, calculating the mean and the
# standard deviation, and finding the minimum, maximum and percental values.  This gives you some
# information about the distribution of the variables.
#
# First of all notice both of the attributes that Pandas could treat as numerical have a count of 350, meaning
# there is no missing data in either of these attributes.
#
# The `Replicate` attribute doesn't look exactly like data.  It ranges from 1 to 50 and has a mean of about
# 25.
#
# Since there are only 350 samples in this dataset, we could look at all of them.  But for a large dataset this
# might not always be practical.  
#
# We could try counting the unique values, as it appears like they are actually integers.

parasite.Replicate.value_counts()

# There are 7 instances of each of the integers 1 through 50.  This indicates that `Replicate` is an experimental
# condition here.  Each of 50 different variations were tried 7 different times, giving a total of 350
# trials/samples for this data set.
#
# So the `ShannonDiversity` is probably the actual experimental measure of this data, and the `Replicate`
# is part of the experimental protocol.
#
# What about `Virulence`?

parasite.Virulence.value_counts()

# So there are 50 of each of what look like Virulence levels.  The reason this data is an `object` instead of
# a float type appears to be because one of the levels was missing and had a blank or space in the data file, thus
# Pandas had to cast all of the values as strings.
#
# We can confirm this, lets see which sample rows might have a blank as their value.

mask = (parasite.Virulence == ' ') 
parasite[ mask ]

# So it is actually the last 50 row samples that have a blank here.  We might guess that, since it looks like
# 'Virulence' is being controlled and varied at levels from 0.5, 0.6, ... to 1.0 that these last rows were
# a virulence of 1.1.  Though we should consult with the original researchers who gathered this data to
# confirm this.

# # Operations on Data
#
# We might want to treat these values as missing values.  The advantage of this is, if we load a blank
# and tell Pandas those are missing values, pandas can then treat the 'Virulence' attribute as a 
# `Series` of floating point numbers, were some are missing values.
#
# So lets reload and resummarize the data frame, specifying that blank spaces be treated as missing values.

# by default the first row will be used for the attribute labels, and it will assume a comma with no extra
# whitespace as the delimiter between attributes for the file
# This time ask pandas to treat a blank space as a value as a missing value instead.
parasite = pd.read_csv('../../data/parasite_data.csv', na_values=[' '])

parasite.columns

parasite.dtypes

parasite.describe()

parasite.tail()

# So now the `Virulence` column is being treated as a floating point type with values ranging from
# 0.5 to 1.0, but with some missing values.
#
# `DataFrame` methods automatically ignore NA/NaN values (which is usually the right thing to do).
# So the mean and standard deviation reported for `Virulence` are only for the 300 present values.

parasite.Virulence.mean()

parasite.Virulence.std()

# We will need to determine what to do with the missing data values if we want to analyze this data.
# One way to proceed would be to simply just drop the samples with the missing attribute from
# our analysis.

parasite.dropna()

# We could use this as our `DataFrame` for analysis.  Notice there are now only the 300 rows were all
# data was present for all of the 3 features.
#
# Or if needed we could instead replace the NA/NaN entries with a valid (default) value.  We could use 0.0,
# although another likely guess from the progression of the data in the table is that this
# was a Virulence of 1.1

parasite.Virulence.fillna(1.1)

# lets fill the missing values with 1.1 from this point forward for analysis
parasite.Virulence = parasite.Virulence.fillna(1.1)


# the numpy sum will treat true as 1, so if anything is still missing we will get a non-zero result
np.sum(parasite.isna().values)

# ## Grouping 
#
# So here we might expect now there are 50 replications of the experiment at each 'Virulence'
# level.  We might want to start with a summary statistic, like the mean and the standard deviation,
# grouped by the 'Virulence' manipulation.
#
# We could extract out each of the 7 'Virulence' levels individuall, but this is a bit tedious:

# +
# mean and std for Virulence of 0.5
virulence_level = 0.5
mask = (parasite.Virulence == virulence_level)

mu = parasite[mask].ShannonDiversity.mean()
sigma = parasite[mask].ShannonDiversity.std()

print('Virulence level: %f   mean=%f   std=%f' % (virulence_level, mu, sigma))

virulence_level = 0.6
mask = (parasite.Virulence == virulence_level)

mu = parasite[mask].ShannonDiversity.mean()
sigma = parasite[mask].ShannonDiversity.std()

print('Virulence level: %f   mean=%f   std=%f' % (virulence_level, mu, sigma))

# etc
# -

# However, Grouping is a powerful concept that can be applied to a `DataFrame`.  So for example if you group
# the `parasite` data by the `Virulence` manipulation, you get this:

parasite.groupby(['Virulence']).mean()

# As you can see this created a group for each of the `Virulence` levels, and the report in the table is
# the `mean()` value of all samples that were in each grouping.
#
# We don't really need the mean for the `Replicate` attribute, but we migth like to the
# the mean and standard deviation, and have a summary table or figure of this information.

virulance_groups = parasite.groupby(['Virulence'])
virulance_groups['ShannonDiversity'].mean(), virulance_groups['ShannonDiversity'].std()

# The result of these 2 summaries are both `Series` with the `Virulence` levels as the index.  It is common
# to put this back into a `DataFrame` so we can create some quick summary tables and figures.
#

data_dict = {
    'mean': virulance_groups['ShannonDiversity'].mean(),
    'std' : virulance_groups['ShannonDiversity'].std()
}
virulance_manipulation = pd.DataFrame(data_dict)


virulance_manipulation

# bar plot with error bars representing standard error
virulance_manipulation['mean'].plot.bar(yerr=virulance_manipulation['std']/np.sqrt(50));

# So assuming higher virulence is bad here, the Shannon diversity seems to mostly increase as virulence increases,
# though it decreases drastically for virulence level 1.0.  We know our 1.1 level may be problematic
# as we were guessing the virulence level to fill in missing values there.  I do wonder if the virulence level
# 1.0 is also mislabeled (0.1?), as it would make sense that the more diverse a virus is, the more virulent
# and dangerous it would be.

# ## Categorical Attributes
#
# Pandas can include and work with categorical data in a `DataFrame`.  A category is a variable that
# takes on a discrete set of values.  To handle categorical data in a NumPy array, and in machine learning
# libraries like Scikit-Learn, we usually have to map the categories to integer values (or use other
# methods like one-hot encoding).
#
# But when we are doing data analytics, we usually want our categories to have readable, meaningful names, 
# even if behind the scenes the categoreis are being mapped to whole numbers like 0, 1, 2,...
#
# Pandas has powerful mechanisms for defining and working with categorical data.

# an example set of data, the second series will be our category
data_dict = {
    'id': [1, 2, 3, 4, 5, 6], # student id
    'raw_grade': ['a', 'b', 'b', 'a', 'a', 'e'], # student grades, but e?
}
df = pd.DataFrame(data_dict)
df

df.dtypes

# Notice that the `raw_grade` attribute is currently an object.  Pandas will not identify and treat 
# attributes as categories automatically, because sometimes these can be a general string or text attribute,
# but sometimes they are true categories.
#
# We can convert the grade strings to a categorical data type.

# +
df['grade'] = df['raw_grade'].astype('category')

df.grade
# -

# Notice that the type of the new attribute is a category and it has 3 levels `[a, b, e]`.
# Internally Pandas represents the category as a mapping, but you can use the 
# category names to manipulate and perform operations on the category.
#
# It would now be illegal, for example, to try and change a value in the category to a level that is
# not defined for the category.

try:
    df.grade.iloc[-1] = 'f'
except ValueError:
    print('ValueError is generated here because f is not a valid category value')

# We can rename the category variable labels.  We could change the 'e' to an 'f', or we can use any
# labels we want.

df.grade.cat.categories = ['very good', 'good', 'very bad']
df

# We can also add new category levels, reorder them, etc.  Notice that the reordering does not change
# the existing category labels incorrectly, they still map to the level you would expect.

df.grade = df.grade.cat.set_categories(['very bad', 'bad', 'medium', 'good', 'very good'])
df

# For some categories the order makes sense and it is important.  For example, the order defined above will
# be used if we want to sort the results by this category attribute.

df.sort_values(by='grade')

# Grouping by a categorical column shows empty columns.

df.groupby('grade').size()

# And if you convert to a NumPy array, the category levels will be used, so this is useful to 
# convert a string category label to an finite integer value.

df['grade_code'] = df.grade.cat.codes
df

df[ ['id', 'grade_code'] ].values

StarcoderdataJupyterScriptsDedupFiltered