# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     name: python3
# ---

# + [markdown] id="37vrnAFPW6_C" colab_type="text"
# #Exercise 10.02: Analyse the categorical variables from the Ames Housing dataset
# In this exercise, we will continue our dataset exploration by analysing the categorical variables of this dataset. To do so, we will implement our own describe functions.
# The dataset used for this exercise is the Ames Housing dataset compiled by <NAME>: http://www.amstat.org/publications/jse/v19n3/decock.pdf

# + [markdown] id="2Ox-pHUPX78-" colab_type="text"
# 1. Open on a new Colab notebook and import the pandas package

# + id="HEiOAwQPW0qb" colab_type="code" colab={}
import pandas as pd

# + [markdown] id="C6NbwMIwX_84" colab_type="text"
# 2. Assign the link to the AMES dataset to a variable called 'file_url':

# + id="j7whidfaYjns" colab_type="code" colab={}
file_url = 'https://raw.githubusercontent.com/TrainingByPackt/The-Data-Science-Workshop/master/Chapter10/dataset/ames_iowa_housing.csv'

# + [markdown] id="VzoH_wvLuD0p" colab_type="text"
# 3. Using the read_csv method from the package pandas, load the dataset into a new variable called 'df':

# + id="nFMz2jNVt-xy" colab_type="code" colab={}
df = pd.read_csv(file_url)

# + [markdown] id="qJ-1EVGS6yZL" colab_type="text"
# 4. Create a new dataframe called 'obj_df' with only the columns that are of numerical types using the method select_dtypes from pandas packages and pass in the value 'number' to the parameter 'include: 
#

# + id="qwSvQHzZhN2q" colab_type="code" colab={}
obj_df = df.select_dtypes(include='object')

# + [markdown] id="eW-0FR24hJ5T" colab_type="text"
#
# 5. Using the attribute **columns** from pandas, extract the list of columns of this dataframe 'obj_df' and assign it to a new variable called 'num_cols' and print its content:

# + [markdown] id="mVuWicDG3X7B" colab_type="text"
# Expected Output:
#
# ![Figure 27 - List of categorical variables](https://docs.google.com/uc?export=download&id=1n5gkiNhA0bPeZFUTdrXEssgqJtqwuBW0)
#

# + id="vdBz2p9w6w4K" colab_type="code" outputId="06c091c4-c294-4ed2-8438-69a1f27c6b00" colab={"base_uri": "https://localhost:8080/", "height": 187}
obj_cols = obj_df.columns
obj_cols


# + [markdown] id="kE2u0_H3aJWs" colab_type="text"
# 6. Create a function called 'describe_object' that takes a **pandas** dataframe and a column name as input parameters. Then inside the function, print out the name of the given column, its number of unique values using the method **nunique()** and the list of values and their occurence using the method **value_counts()**:

# + id="3yRUe0E7aJgk" colab_type="code" colab={}
def describe_object(df, col_name):
  print(f"\nCOLUMN: {col_name}")
  print(f"{df[col_name].nunique()} different values")
  print(f"List of values:")
  print(df[col_name].value_counts(dropna=False, normalize=True))


# + [markdown] id="H1YRwGZgOYmp" colab_type="text"
# 7. Test this function by providing the 'df' dataframe and the column 'MSZoning':

# + [markdown] id="hTglykjp3hZG" colab_type="text"
# Expected Output:
#
# ![Figure 28 - Display of the created function for the column ‘MSZoning’](https://docs.google.com/uc?export=download&id=1PE8Ive1mpHZeeC0GN5wDrjxvkaufT5Pv)
#

# + id="QRLrQjc5bl5b" colab_type="code" outputId="0b94009d-3775-41ce-a4b5-a2e5f40b36a3" colab={"base_uri": "https://localhost:8080/", "height": 187}
describe_object(df, 'MSZoning')

# + [markdown] id="z-c3WNFIwWDG" colab_type="text"
# For the column ‘MSZoning’, the value ‘RL’ represents almost 79% of the values while ‘C (all) is only present less than 1% of the rows.

# + [markdown] id="6sE7pqfMf-NR" colab_type="text"
# 8. Create a for loop that will call the created function for every element from the list 'obj_cols':

# + [markdown] id="CkuSqu9p3tJK" colab_type="text"
# Expected Output:
#
# ![Figure 29 - Display of the created function for the first columns contained in ‘obj_cols’](https://docs.google.com/uc?export=download&id=1cou8u_aD4Z8ec1EqDKRPFXJGGyEQUzh_)
#
#

# + id="hv3qvtqZf-TL" colab_type="code" outputId="adf18174-a440-4fd6-c7f2-59461011644f" colab={"base_uri": "https://localhost:8080/", "height": 1000}
for col_name in obj_cols:
  describe_object(df, col_name)

# + [markdown] id="ZO9eRjpFwQvD" colab_type="text"
# We can confirm that the column ‘Street’ is almost constant as 99.6% of the rows contains the same value: ‘Pave’. For the column ‘Alley’, almost 94% of the rows have missing values. We can also notice that 'MiscFeature' contains missing value for 96% of the rows so this column doesn't contain much information.
#

# + [markdown] id="_-yGDjckRbwV" colab_type="text"
# Excellent! We just analysed all the categorical variables from this dataset in one go. We saw how to look at the distribution of all the values contained in one feature. We also found that some of them are dominated by a single value and others have mainly missing values.

StarcoderdataJupyterScriptsDedupFiltered