# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: 'Python 3.9.2 64-bit (''base'': conda)'
#     language: python
#     name: python392jvsc74a57bd0249cfc85c6a0073df6bca89c83e3180d730f84f7e1f446fbe710b75104ecfa4f
# ---

# STAT 451: Machine Learning (Fall 2021)  
# Instructor: <NAME> (<EMAIL>)  

# + [markdown] tags=[]
# # L05 - Data Preprocessing and Machine Learning with Scikit-Learn
# -

# # 5.1 Reading a Dataset from a Tabular Text File

# %load_ext watermark
# %watermark -v -a '<NAME>' -p pandas

# ## Overview

# In this lecture, we are closing the "Computational Foundation" section by introducing yet another Python library, pandas, which is extremely handy for data (pre)processing. The second focus of this lecture is on the [Scikit-learn](http://scikit-learn.org) machine learning library, which is widely considered as the most mature and most well-designed general machine learning library.

# ## Pandas -- A Python Library for Working with Data Frames

# - Pandas is probably the most popular and convenient data wrangling library for Python (official website: https://pandas.pydata.org) 
# - Pandas stands for PANel-DAta-S.
# - Relativ similar to data frames in R.
# - How is it different from NumPy arrays? 
#     - Allows for heterogenous data (columns can have different data types)
#     - Adds some more convenient functions on top that are handy for data processing

# ### Loading Tabular Datasets from Text Files

# - Here, we are working with structured data, data which is organized similar to a "design matrix" (see lecture 1) -- that is, examples as rows and features as columns (in contrast: unstructured data such as text or images, etc.).
# - CSV stands for "comma separated values" (also common: TSV, tab seperated values).
# - The `head` command is a Linux/Unix command that shows the first 10 rows by default; the `!` denotes that Jupyter/the IPython kernel should execute it as a shell command (`!`-commands may not work if you are on Windows, but it is not really important).

# !head data/iris.csv

# - We use the `read_csv` command to load the CSV file into a pandas data frame object f of the class `DataFrame`.
# - Data frames also have a `head` command; here it shows the first 5 rows.

# +
import pandas as pd


df = pd.read_csv('data/iris.csv')
df.head()
# -

type(df)

# - It is always good to double check the dimensions and see if they are what we expect. 
# - The `DataFrame` `shape` attribute works the same way as the NumPy array `shape` attribute (Lecture 04).

df.shape

StarcoderdataJupyterScriptsDedupFiltered