# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     name: python3
# ---

# + [markdown] id="view-in-github" colab_type="text"
# <a href="https://colab.research.google.com/github/allenalvin333/College7/blob/master/T1/T1.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# + [markdown] id="UVjpftgcVgHH"
# # **<NAME>** - 18BIS0043
# # **Lab Task - 1, Submitted to: Dr. <NAME>**

# + [markdown] id="p9RMLu7GWwZ2"
# ## Example

# + colab={"base_uri": "https://localhost:8080/"} id="TaSkzzJ48Rz7" outputId="f5e2fddf-407f-4923-d3e2-1433c5e87948"
dataset = pd.read_csv('sample_data.csv')
print(dataset)

# + colab={"base_uri": "https://localhost:8080/"} id="Ixu6Dh9r50L8" outputId="d1a44c5f-e196-4558-907d-baa27f4d4463"
dataset = dataset.drop_duplicates()
print(dataset)

# + [markdown] id="Zxyz4_1gC971"
# # **0. Initialization**

# + colab={"base_uri": "https://localhost:8080/"} id="_NxlLHiL3tGr" outputId="ac10a27c-4f86-4a57-f4e2-621747e18a6e"
pip install fuzzywuzzy

# + colab={"base_uri": "https://localhost:8080/"} id="Un1O0J8ADpMM" outputId="207466bd-82cb-45b5-9f12-7e2138bfd053"
from google.colab import drive
drive.mount("/content/gdrive")

# + id="kyE8Ju5CFMM8"
# !cp -r /content/gdrive/MyDrive/Academic/Labs/ML/T1 /content/

# + id="L-wofYF9r-m9"
import warnings
warnings.filterwarnings("ignore")

# + id="VebPMaCk71qX"
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt

# + id="Vm_4cNrEed4Y"
#1
from sklearn.impute import SimpleImputer
from sklearn.metrics import mean_absolute_error
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split

#2
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import LabelBinarizer

#3
import fuzzywuzzy
from fuzzywuzzy import process
import chardet

#4
import datetime

#5
from scipy import stats
from numpy import asarray
from mlxtend.preprocessing import minmax_scaling
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import MinMaxScaler
from sklearn.pipeline import Pipeline
from pandas.plotting import scatter_matrix
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold

# + [markdown] id="XgSQ_CZ7ZjOa"
# # **1. Dealing with Missing Values** 

# + [markdown] id="ICB2I9CdJn9i"
# ## References
# - https://www.kaggle.com/dansbecker/handling-missing-values
# - https://towardsdatascience.com/how-to-deal-with-missing-data-in-python-1f74a9112d93
# - https://medium.com/x8-the-ai-community/handling-missing-values-in-data-54e1dc77e24f

# + [markdown] id="aG2eNPabJuaG"
# ## Dataset
# - https://www.kaggle.com/dansbecker/melbourne-housing-snapshot/home?select=melb_data.csv

# + [markdown] id="W8Pt9PsMZ0T6"
# ## Explanation
# *A common occurrence in a data-set is missing values. This can happen due to multiple reasons like unrecorded observations or data corruption. Pandas library provides a variety of functions for marking these corrupt values. We will study how we can remove or impute these values. There are many ways data can end up with missing values.*
# - *A 2 bedroom house wouldn't include an answer for "How large is the third bedroom?"*
# - *Someone being surveyed may choose not to share their income*
#
# *Python libraries represent missing numbers as nan which is short for "not a number". Most libraries (including scikit-learn) will give you an error if you try to build a model using data with missing values.*
# - *A Simple Option: **Drop Columns with Missing Values** - 
# In many cases, you'll have both a training dataset and a test dataset. You will want to drop the same columns in both DataFrames*
# - *A Better Option: **Imputation** - Imputation fills in the missing value with some number. The imputed value won't be exactly right in most cases, but it usually gives more accurate models than dropping the column entirely. The default behavior fills in the mean value for imputation. Statisticians have researched more complex strategies, but those complex strategies typically give no benefit once you plug the results into sophisticated machine learning models.*

# + [markdown] id="xzyyUZ6Kr-bZ"
# ## **Sample Program**

# + [markdown] id="UwmyImYUsM4H"
# #### Dataset Inspection

# + id="tDoeKxhw0aGR"
df1 = pd.read_csv('/content/T1/2.csv')

# + colab={"base_uri": "https://localhost:8080/", "height": 347} id="7_uya6xi0bS5" outputId="1f96b3a0-2382-421e-c32b-c1809a8fd6a3"
df1.head()

# + id="7Ii-L2gr0ef4" colab={"base_uri": "https://localhost:8080/", "height": 320} outputId="88782747-1085-40a6-fc01-17b8f7573e80"
df1.describe()

# + id="3QXrbAUz0hfJ" colab={"base_uri": "https://localhost:8080/"} outputId="59eb3188-6604-45af-ea94-05a937cf3b79"
df1.info

# + colab={"base_uri": "https://localhost:8080/", "height": 348} id="TXVKPmadQLeI" outputId="23e74aa9-9301-4eca-eead-2b132ac624e9"
col1 = df1.columns[:30]
colours1 = ['#000099', '#ffff00']
sns.heatmap(df1[col1].isnull(), cmap=sns.color_palette(colours1))

# + [markdown] id="TXa7vKseQneQ"
# *Yellow represents the missing data in the first 30 entries of the dataframe*

# + [markdown] id="eHn_Z3wDsOqT"
# #### **Drop Columns**
# *If those columns had useful information (in the places that were not missing), your model loses access to this information when the column is dropped. Also, if your test data has missing values in places where your training data did not, this will result in an error.*

# + id="6DdsoJWIhJY6"
df11 = df1.dropna(axis=1)

# + colab={"base_uri": "https://localhost:8080/", "height": 347} id="MIFvmtDGi82B" outputId="de7e253f-fcb5-442c-cfa2-f0911de37194"
df11.head()

# + colab={"base_uri": "https://localhost:8080/", "height": 300} id="GeCEW2sgi_Vq" outputId="21013a64-fa79-4feb-a7e7-3b3c184fe12b"
df11.describe()

# + id="Sp2Lz2t4jdsx"
df1_cols = [col for col in df1.columns 
            if df1[col].isnull().any()]

# + colab={"base_uri": "https://localhost:8080/"} id="ESsaCwFajnJS" outputId="712acae8-4dd7-4612-e60a-dd82026f5ba5"
print(df1_cols)

# + id="jmKaqhNbjtmw"
df12 = df1.drop(df1_cols, axis=1)

# + colab={"base_uri": "https://localhost:8080/", "height": 347} id="RffI8KZZj2EI" outputId="f3010f0f-5fbf-4c6b-ce39-2576d2dc22fe"
df12.head()

# + colab={"base_uri": "https://localhost:8080/", "height": 300} id="A11UES7Oj3eB" outputId="c44c097c-5768-4e4f-a15a-8bcb44c8f711"
df12.describe()

# + [markdown] id="IJNm4oOAlkEw"
# *So, it's somewhat usually not the best solution. However, it can be useful when most values in a column are missing.*

# + [markdown] id="ZRpseaA1scgZ"
# #### **Imputation**
# *Imputation is the standard approach, and it usually works well. However, imputed values may by systematically above or below their actual values (which weren't collected in the dataset). Or rows with missing values may be unique in some other way.*

# + id="34kljwUbqf2K"
df13 = df1.select_dtypes(exclude=['object'])

# + id="62zjUMvyodL_"
im1 = SimpleImputer()
df14 = im1.fit_transform(df13)

# + colab={"base_uri": "https://localhost:8080/"} id="DFmBN64uyYAG" outputId="4f65ba3f-0ba8-4938-c9d0-165b079b0ec9"
print(df14)

# + [markdown] id="ui9CJQukJwaS"
# #### **Comparitive Study**
# *Let's see which method is better for dealing with the missing values in this dataset*

# + id="XWjDAvGGKFeh"
x1 = df1.drop(['Price'], axis=1)               # Predictors
y1 = df1.Price                                 # Target
x1_num = x1.select_dtypes(exclude=['object'])  # Numeric Predictors

# + id="XtIpJoC_KtM-"
xtrain1, xtest1, ytrain1, ytest1 = train_test_split(x1_num,y1,train_size=0.7,
                                                    test_size=0.3,random_state=0)


# + id="vYk4TZ76LOPb"
def score_dataset_1(xtrain1, xtest1, ytrain1, ytest1):
    model1 = RandomForestRegressor()
    model1.fit(xtrain1, ytrain1)
    p1 = model1.predict(xtest1)
    return mean_absolute_error(ytest1, p1)


# + colab={"base_uri": "https://localhost:8080/"} id="_PluuPW9LhPq" outputId="06bab438-e6fc-4161-8adb-310b17c89437"
# Dropping Values
df15 = [col for col in xtrain1.columns if xtrain1[col].isnull().any()]
red_xtrain1 = xtrain1.drop(df15, axis=1)
red_xtest1  = xtest1.drop(df15, axis=1)
print("Mean Absolute Error from Dropping columns with Missing Values:")
print(score_dataset_1(red_xtrain1, red_xtest1, ytrain1, ytest1))

# + colab={"base_uri": "https://localhost:8080/"} id="CgxXdmsRMB8H" outputId="6178fe33-90dd-42ef-9b4c-cc6b619a8fa4"
# Imputation
im2 = SimpleImputer()
imp_xtrain1 = im2.fit_transform(xtrain1)
imp_xtest1 = im2.transform(xtest1)
print("Mean Absolute Error from Imputation:")
print(score_dataset_1(imp_xtrain1, imp_xtest1, ytrain1, ytest1))

# + [markdown] id="IlGlGEWHNUIJ"
# *Mean Absolute Error from Imputation is lesser. Using this method will help us to improve the model compared to dropping values*

# + [markdown] id="y1wzGQNVZrrs"
# # **2. Handling categorical data**

# + [markdown] id="brcVXKcNvd_r"
# ## References
# - https://pbpython.com/categorical-encoding.html
# - https://analyticsindiamag.com/complete-guide-to-handling-categorical-data-using-scikit-learn/
# - https://towardsdatascience.com/handling-categorical-data-the-right-way-9d1279956fc6

# + [markdown] id="gqsJyWs0wtv6"
# ## Dataset
# - https://www.kaggle.com/dansbecker/melbourne-housing-snapshot/home?select=melb_data.csv

# + [markdown] id="ZXop1lJgwNPa"
# ## Explanation
# *Categorical features have a lot to say about the dataset thus it should be converted to numerical to make it into a machine-readable format. Focusing only on numerical variables in the dataset isn’t enough to get good accuracy. Often categorical variables prove to be the most important factor and thus identify them. In many practical Data Science activities, the data set will contain categorical variables.*
#
# *These variables are typically stored as text values which represent various traits. Some examples include color (“Red”, “Yellow”, “Blue”), size (“Small”, “Medium”, “Large”) or geographic designations (State or Country). Regardless of what the value is used for, the challenge is determining how to use this data in the analysis. Many machine learning algorithms can support categorical values without further manipulation but there are many more algorithms that do not. Therefore, the analyst is faced with the challenge of figuring out how to turn these text attributes into numerical values for further processing.*
#
# *Currently, many resources advertise a wide variety of solutions that might seem to work at first, but are deeply wrong once thought through. This is especially true for non-ordinal categorical data, meaning that the classes are not ordered (As it might be for Good=0, Better=1, Best=2). A bit of clarity is needed to distinguish the approaches that Data Scientists should use from those that simply make the models run.*
#
# *Two major types of categorical features are:*
# - ***Nominal*** – *These are variables which are not related to each other in any order such as colour (black, blue, green).*
# - ***Ordinal*** – *These are variables where a certain order can be found between them such as student grades (A, B, C, D, Fail).*
#
# *There are many more ways by which categorical variables can be changed to numeric I’ve discussed some of the important and commonly used ones. Handling categorical variables is an important step for feature engineering. New variables can be formed by categorical variables and get more insight into the dataset.*

# + [markdown] id="_Q2mC0bTyYrI"
# ## **Sample Program**

# + [markdown] id="6SBvtgm2Qtdd"
# #### Dataset Inspection

# + id="WmzAXfqAvYqL"
df2 = pd.read_csv('/content/T1/2.csv')

# + colab={"base_uri": "https://localhost:8080/", "height": 347} id="saqey1yPWNOR" outputId="68fcb42e-a3ba-4c54-ac68-d3334e7af172"
df2.head()

# + colab={"base_uri": "https://localhost:8080/", "height": 320} id="T3jmsHQ_WkjM" outputId="7e6d7a8f-bba6-4854-84cf-d9c501f4971a"
df2.describe()

# + colab={"base_uri": "https://localhost:8080/"} id="oyouGJDtWs04" outputId="21abb29c-e4a0-4ff6-b11a-c809d3c009b0"
df2.info

# + colab={"base_uri": "https://localhost:8080/"} id="eq3Xi9zLW0Gi" outputId="f135c821-9bea-407b-adcb-7651c93868cb"
s2 = (df2.dtypes=='object')
o2 = list(s2[s2].index)
print(o2)

# + colab={"base_uri": "https://localhost:8080/", "height": 206} id="N_vYwreyZNaD" outputId="22e9f04e-16c1-4cfa-99fa-103e4c063e38"
features2 = df2[['Type','Method','Regionname']]
features2.head()

# + colab={"base_uri": "https://localhost:8080/", "height": 175} id="PbHjTfRVZX1o" outputId="0511b993-be33-4a2f-fc8b-ceb93d2d833a"
features2.describe()

# + colab={"base_uri": "https://localhost:8080/"} id="xfjrnQSuZe5a" outputId="f13e5014-154d-4e25-9c02-9ec1c23937e2"
features2.Regionname.value_counts()

# + colab={"base_uri": "https://localhost:8080/"} id="-bHt_mgMZmGM" outputId="17a53dc8-8c76-45d2-cd12-88ab4b5ba312"
features2.Type.value_counts()

# + colab={"base_uri": "https://localhost:8080/"} id="bWFsSNGKZsLR" outputId="af6f74a4-0724-41ad-d5f4-a8e15682161e"
features2.Method.value_counts()

# + [markdown] id="E3fMIaWp6oaT"
# #### **Label Encoding**
# *Label encoding can uniquely number the different categories from 0 to n-1. Thus also termed as Integer encoding. LabelEncoder class from the scikit-learn library is used for this purpose.*

# + id="sk0F_Av47kzT" colab={"base_uri": "https://localhost:8080/"} outputId="4b1e9b7b-9736-4f2e-a9cc-f81459aee790"
le21 = LabelEncoder()
df21 = features2[['Regionname']]
df21['Region'] = le21.fit_transform(features2['Regionname'])
df21.value_counts()

# + colab={"base_uri": "https://localhost:8080/"} id="UrIevLXBrmg6" outputId="092ef1f8-fd54-4d26-d290-354f6b4c9e78"
le22 = LabelEncoder()
df22 = features2[['Type']]
df22['Type'] = le22.fit_transform(features2['Type'])
df22.value_counts()

# + colab={"base_uri": "https://localhost:8080/"} id="rhGnjQYarxDx" outputId="73965784-09dc-44c4-84d5-a9662789414d"
le23 = LabelEncoder()
df23 = features2[['Method']]
df23['Method'] = le23.fit_transform(features2['Method'])
df23.value_counts()

# + [markdown] id="739FJzmjlTN4"
# *LabelEncoder cannot handle missing values so it’s important to impute them. LabelEncoder can be used to store values using less disk space. This is simple to use and works well on tree-based algorithms. It cannot work for linear models, SVMs, or neural networks as their data needs to be standardized.*

# + [markdown] id="NH2UGcj527Rr"
# #### **One Hot Encoding**
# *One hot encoding is a binary encoding applied to categorical values. To increase performance one can also first perform label encoding then those integer variables to binary values which will become the most desired form of machine-readable.*

# + id="nqmwqV4L2_9j" colab={"base_uri": "https://localhost:8080/"} outputId="574c7b6c-d0a6-4ec3-d077-f8cd1becbbb4"
df24 = pd.get_dummies(features2['Regionname'])
df24.value_counts()

# + colab={"base_uri": "https://localhost:8080/"} id="5HzxX6V_s2md" outputId="9549e987-82ba-4c29-e32e-a709ae4a6725"
df25 = pd.get_dummies(features2['Type'])
df25.value_counts()

# + colab={"base_uri": "https://localhost:8080/"} id="LhPvy-UBs7lq" outputId="607d3ce3-9edc-4ca0-8021-e733e4aae3cc"
df26 = pd.get_dummies(features2['Method'])
df26.value_counts()

# + [markdown] id="RCLalD2XlZbS"
# *One hot encoding overcomes the limitations of label encoding and can be used in both tree-based and non-tree-based machine learning algorithms. The disadvantage is that for high cardinality, the feature space can really blow up quickly. The binary variables are often called “dummy variables” in statistics.*

# + [markdown] id="aA48nazBsgEy"
# #### **Label Binarizer**
# *Scikit-learn also supports binary encoding by using the LabelBinarizer. We use a similar process as above to transform the data for the process of creating a pandas DataFrame.*

# + colab={"base_uri": "https://localhost:8080/"} id="MjxicUP-sjOe" outputId="0c60b376-8fc8-41e2-9e7a-450e22d411a7"
lb21 = LabelBinarizer()
lb24 = lb21.fit_transform(features2["Regionname"])
pd.DataFrame(lb24, columns=lb21.classes_).value_counts()

# + colab={"base_uri": "https://localhost:8080/"} id="Ioy2nBkFv9Kp" outputId="203c673a-7df3-41ef-e2fd-fa4730640eff"
lb22 = LabelBinarizer()
lb25 = lb22.fit_transform(features2["Type"])
pd.DataFrame(lb25, columns=lb22.classes_).value_counts()

# + colab={"base_uri": "https://localhost:8080/"} id="gFQgu1hZwGbw" outputId="0c2a5c02-8f73-47f4-afd7-8ff8bdb6e96f"
lb23 = LabelBinarizer()
lb26 = lb23.fit_transform(features2["Method"])
pd.DataFrame(lb26, columns=lb23.classes_).value_counts()

# + [markdown] id="V5Q8Gsh3Zr_c"
# # **3. Dealing with inconsistent data entry**

# + [markdown] id="22Fe9ukyOv9O"
# ## References
# - https://towardsdatascience.com/how-to-deal-with-missing-data-in-python-1f74a9112d93
# - https://www.kaggle.com/rtatman/data-cleaning-challenge-inconsistent-data-entry
# - https://www.kaggle.com/alexisbcook/inconsistent-data-entry
# - https://towardsdatascience.com/data-cleaning-in-python-the-ultimate-guide-2020-c63b88bf0a0d

# + [markdown] id="p_xNUNQkO1yn"
# ## Dataset
# - https://www.kaggle.com/zusmani/pakistansuicideattacks

# + [markdown] id="Few7kLcHO5JE"
# ## Explanation
# *Data cleaning or cleansing is the process of detecting and correcting (or removing) corrupt or inaccurate records from a record set, table, or database and refers to identifying incomplete, incorrect, inaccurate or irrelevant parts of the data and then replacing, modifying, or deleting the dirty or coarse data.*
# *It is also crucial to have the dataset follow specific standards to fit a model. We need to explore the data in different ways to find out the inconsistent data. Much of the time, it depends on observations and experience. There is no set code to run and fix them all.*
#
# *Three inconsistent data types are:*
# - ***Capitalization*** - *Inconsistent  usage of upper and lower cases in categorical values is a common mistake. It could cause issues since analyses in Python is case sensitive.*
# - ***Formats*** - *Another standardization we need to perform is the data formats. One example is to convert the feature from string to DateTime format.*
# - ***Categorical Values*** - *Inconsistent categorical values are the last inconsistent type we cover. A categorical feature has a limited number of values. Sometimes there may be other values due to reasons such as typos.*
#
# *To avoid **capitalization inconsistencies**, we can put all letters to lower cases (or upper cases).*
#
# *For avoiding **Format inconsistencies** we can convert it and extract the date or time values by using the code below. After this, it’s easier to analyze the transaction volume group by either year or month.*
#
# ***Categorical Value inconsistencies*** *can easily be identified by fuzzy logic (or edit distance). It measures how many letters (distance) we need to change the spelling of one value to match with another value.*

# + [markdown] id="sLmmKra2O8KQ"
# ## **Sample Program**

# + [markdown] id="AL2cxDz24Zo7"
# #### Dataset Inspection

# + id="trdYjQMLRdR6"
np.random.seed(0)

# + [markdown] id="pu1SV0bT5B3g"
# `df3 = pd.read_csv('/content/T1/1.csv')` 
# > This statement gives a character encoding error.

# + colab={"base_uri": "https://localhost:8080/"} id="XTOn92EU42lS" outputId="42ab6f9f-74cf-4b57-b5f8-da2fe828aaef"
with open("/content/T1/1.csv", 'rb') as rd:
    res = chardet.detect(rd.read(100000))
print(res)

# + id="ETL3teSc4c9q"
df3 = pd.read_csv('/content/T1/1.csv',encoding='Windows-1252')

# + colab={"base_uri": "https://localhost:8080/", "height": 573} id="fHR8OiPL5cDa" outputId="00232df4-f4ce-48ba-e824-a7d6487219b0"
df3.head()

# + colab={"base_uri": "https://localhost:8080/", "height": 300} id="LZHiGxs-5gnU" outputId="012c4d24-423d-4264-88c6-172c43f33a1d"
df3.describe()

# + colab={"base_uri": "https://localhost:8080/"} id="zXe0XGGQ5iAi" outputId="21dc1cfe-14c5-4ff5-bd06-a55b03bf8816"
df3.info

# + [markdown] id="tZijc_O45kJK"
# #### **Manual Correction**
# *The first thing to do is to convert everything to lower case and remove any white spaces at the beginning and end of cells. Inconsistencies in capitalizations and trailing white spaces are very common in text data.*

# + id="auo8h82w5pNU"
x3 = df3['City'].unique()

# + colab={"base_uri": "https://localhost:8080/"} id="tOggjfLc_3fw" outputId="e5dfc136-5b58-48e6-d847-d20888233c60"
len(x3)

# + colab={"base_uri": "https://localhost:8080/"} id="nKxEi8co-tV4" outputId="247fd542-63fa-4f49-98a7-db47f399c459"
x3.sort()
x3

# + id="FiaTl3nY_ACq"
df3['City'] = df3['City'].str.lower()

# + id="LPQn3aQQ_aOJ"
df3['City'] = df3['City'].str.strip()

# + id="BHbnlit5_a94"
y3 = df3['City'].unique()

# + colab={"base_uri": "https://localhost:8080/"} id="3dQ2AnzP_78P" outputId="8f49675b-6509-4f94-efcd-7428f9190b94"
len(y3)

# + colab={"base_uri": "https://localhost:8080/"} id="YybMnqrI_ftx" outputId="e9b3313d-68b4-43e4-89f4-75fb195c4824"
y3.sort()
y3

# + colab={"base_uri": "https://localhost:8080/"} id="f78vEQF1_-AR" outputId="5161737c-b340-4d29-a7b5-e61210216020"
print("Redundant Values: {}".format(len(x3)-len(y3)))

# + [markdown] id="SAmzHLY5AII4"
# *We can fix a good 80% of your text data entry inconsistencies by doing this. There's a more efficient way to do this though*

# + [markdown] id="0NbXaKjL6OpQ"
# #### **Fuzzy Matching**
# *The process of automatically finding text strings that are very similar to the target string. In general, a string is considered "closer" to another one the fewer characters you'd need to change if you were transforming one string into another.*

# + colab={"base_uri": "https://localhost:8080/"} id="FGY8vq0uAj1A" outputId="8c966876-f38c-4dcb-e893-3b397fd2e8bc"
a3 = fuzzywuzzy.process.extract("d.i khan", y3, limit=5, scorer=fuzzywuzzy.fuzz.token_sort_ratio)
a3


# + id="E9thUFPVBiDR"
def replace_matches_in_column_3(df, column, string_to_match, min_ratio = 85):
    strings = df[column].unique()
    matches = fuzzywuzzy.process.extract(string_to_match, strings, 
                                         limit=10, scorer=fuzzywuzzy.fuzz.token_sort_ratio)
    close_matches = [matches[0] for matches in matches if matches[1] >= min_ratio]
    rows_with_matches = df[column].isin(close_matches)
    df.loc[rows_with_matches, column] = string_to_match  
    print("Completed")


# + colab={"base_uri": "https://localhost:8080/"} id="FLqYnlGjB4AI" outputId="aa2af36b-3c8c-4d99-d015-78e78cd8b12c"
replace_matches_in_column_3(df=df3, column='City', string_to_match="d.i khan")

# + id="pdi2lSehCBpf"
z3 = df3['City'].unique()

# + colab={"base_uri": "https://localhost:8080/"} id="w7XQwkWECIml" outputId="f82d6e0d-077d-459d-d5e5-baf109414315"
len(z3)

# + colab={"base_uri": "https://localhost:8080/"} id="GV1sWPNgCf8h" outputId="b1be4a53-ceb0-47f5-dbb8-18914081cbb1"
z3.sort()
z3

# + colab={"base_uri": "https://localhost:8080/"} id="wejm4kgFCkBQ" outputId="54c4c0e1-fa8f-4e49-e02b-a3394ea40343"
print("Redundant Values: {}".format(len(x3)-len(z3)))

# + [markdown] id="hSxhad0xA0O8"
# *So "apple" and "snapple" are two changes away from each other (add "s" and "n") while "in" and "on" are one change away (replace "i" with "o"). We won't always be able to rely on fuzzy matching 100%, but it is comparitively efficient.*

# + [markdown] id="hAaPC92WZsV6"
# # **4.	Handling Dates and Times**

# + [markdown] id="cBzx7jGeSx_9"
# ## References
# - https://www.kaggle.com/rtatman/data-cleaning-challenge-parsing-dates
# - https://towardsdatascience.com/machine-learning-with-datetime-feature-engineering-predicting-healthcare-appointment-no-shows-5e4ca3a85f96
# - https://predictablynoisy.com/posts/2017/2017-03-16-dates_in_python/
# - https://medium.com/jbennetcodes/dealing-with-datetimes-like-a-pro-in-pandas-b80d3d808a7f

# + [markdown] id="bb-gklbrTZUX"
# ## Dataset
# - https://www.kaggle.com/nasa/landslide-events

# + [markdown] id="fJ3b1MxOTxY9"
# ## Explanation
# *Dates and times are rich sources of information that can be used with machine learning models. However, these datetime variables do require some feature engineering to turn them into numerical data. In this post, I will demonstrate how to create datetime features with built in pandas functions for your machine learning models.*
#
# *Pandas uses the **object** dtype for storing various types of data types, but most often when you see a column with the dtype **object** it will have strings in it. There's also a specific **datetime64** dtypes in the pandas documentation. Since the dtype of our column is **object** rather than **datetime64**, Python doesn't know that this column contains dates.*
#
# *The basic idea is that you need to point out which parts of the date are where and what punctuation is between them. There are lots of possible parts of a date, but the most common are **%d** for day, **%m** for month, **%y** for a two-digit year and **%Y** for a four digit year.*

# + [markdown] id="gCphB5TfTzJD"
# ## **Sample Program**

# + [markdown] id="4O9Ogduk5HHz"
# #### Dataset Inspection

# + id="mLfUMMUzC1VZ"
np.random.seed(0)

# + id="u7xxvK7k5k-a"
df4 = pd.read_csv('/content/T1/3.csv')

# + colab={"base_uri": "https://localhost:8080/", "height": 400} id="J2NwTQnL5nhY" outputId="0ebcc352-aa15-466b-f9a0-e29bc798c845"
df4.head()

# + colab={"base_uri": "https://localhost:8080/", "height": 300} id="dPXvHNVA5ozo" outputId="a7477823-e8ab-46f8-f5f5-8b724f152505"
df4.describe()

# + colab={"base_uri": "https://localhost:8080/"} id="d1OzgzTx5qf2" outputId="079d8a8a-5541-43d8-c617-15eb0d8499af"
df4.info

# + [markdown] id="Y9Ok__cZ5gNO"
# #### **Date Parsing**
# *Now that we know that our date column isn't being recognized as a date, it's time to convert it so that it is recognized as a date. This is called "parsing dates" because we're taking in a string and identifying its component parts.*

# + colab={"base_uri": "https://localhost:8080/"} id="rIbG3yxsBKsW" outputId="40aaf517-a3b2-4335-b59f-802a11ae92c4"
print(df4['date'].head())

# + colab={"base_uri": "https://localhost:8080/"} id="-a5vfBF7DARa" outputId="ae204c28-609f-4dd5-93b8-dce6bebd9c5e"
df4['date'].dtype

# + [markdown] id="ac4nAYdtD3PL"
# `df4['new_date'] = pd.to_datetime(df4['Date'], format = "%m/%d/%y")` 
# > This statement gave an error because there are multiple date formats in a single column.

# + id="glnzZtG3DEcn"
df4['new_date'] = pd.to_datetime(df4['date'], infer_datetime_format=True)

# + colab={"base_uri": "https://localhost:8080/"} id="GeVUfQA4FyC3" outputId="1c95d3bd-aea1-4840-e295-3f6f6c07a689"
df4['new_date'].dtype

# + colab={"base_uri": "https://localhost:8080/"} id="IvQIkX45FVr7" outputId="3a103ff8-8d3b-4ae5-f843-09afbd3b18da"
day4 = df4['new_date'].dt.day
day4

# + colab={"base_uri": "https://localhost:8080/"} id="p5pHPZZXHx2G" outputId="76d9b0f3-835c-464a-beb9-57952386e904"
month4 = df4['new_date'].dt.month
month4

# + colab={"base_uri": "https://localhost:8080/"} id="EROKiNrUH7yB" outputId="56f67074-c429-424c-c9f6-15813d58c33e"
year4 = df4['new_date'].dt.year
year4

# + id="JC3SbVEPH_Nn"
day4 = day4.dropna()
month4 = month4.dropna()
year4 = year4.dropna()

# + colab={"base_uri": "https://localhost:8080/", "height": 299} id="hIcYaVnyIM3u" outputId="de87374f-8fc0-4fcb-8109-3d776df47226"
sns.distplot(day4, kde=False, bins=31)

# + colab={"base_uri": "https://localhost:8080/", "height": 299} id="M0P0AnLmIPVi" outputId="d6b298a2-26e0-4015-b74f-5e00fa57e538"
sns.distplot(month4, kde=False, bins=12)

# + colab={"base_uri": "https://localhost:8080/", "height": 299} id="mXibPg0sIRjx" outputId="53050793-6383-4a87-bc8c-24ef5a6a7152"
sns.distplot(year4, kde=False)

# + [markdown] id="CJU1EsNLE4Fh"
# *There are two big reasons not to always have pandas guess the time format. The first is that pandas won't always been able to figure out the correct date format, especially if someone has gotten creative with data entry. The second is that it's much slower than specifying the exact format of the dates.*

# + [markdown] id="LNYm8fOVZskK"
# # **5. Scaling and Normalization**

# + [markdown] id="Umdea-MyUFgF"
# ## References
# - https://www.kaggle.com/rtatman/data-cleaning-challenge-scale-and-normalize-data
# - https://www.analyticsvidhya.com/blog/2020/04/feature-scaling-machine-learning-normalization-standardization/
# - https://machinelearningmastery.com/standardscaler-and-minmaxscaler-transforms-in-python/
# - https://machinelearningmastery.com/how-to-improve-neural-network-stability-and-modeling-performance-with-data-scaling/

# + [markdown] id="E0D6P9hYUKTw"
# ## Dataset
# - https://raw.githubusercontent.com/jbrownlee/Datasets/master/sonar.csv

# + [markdown] id="yYGpMWB0UMEX"
# ## Explanation
# *Deep learning neural network models learn a mapping from input variables to an output variable. As such, the scale and distribution of the data drawn from the domain may be different for each variable. Input variables may have different units (e.g. feet, kilometers, and hours) that, in turn, may mean the variables have different scales.*
#
# *Differences in the scales across input variables may increase the difficulty of the problem being modeled. An example of this is that large input values (e.g. a spread of hundreds or thousands of units) can result in a model that learns large weight values. A model with large weight values is often unstable, meaning that it may suffer from poor performance during learning and sensitivity to input values resulting in higher generalization error.*
#
# #### ***Scaling***
# *This means that you're transforming your data so that it fits within a specific scale, like 0-100 or 0-1. You want to scale data when you're using methods based on measures of how far apart data points, like support vector machines, or SVM or k-nearest neighbors, or KNN. With these algorithms, a change of "1" in any numeric feature is given the same importance. By scaling your variables, you can help compare different variables on equal footing. To help solidify what scaling looks like, let's look at a made-up example.*
#
# #### ***Normalization***
# *Scaling just changes the range of your data. Normalization is a more radical transformation. The point of normalization is to change your observations so that they can be described as a normal distribution. In general, you'll only want to normalize your data if you're going to be using a machine learning or statistics technique that assumes your data is normally distributed. Some examples of these include t-tests, ANOVAs, linear regression, linear discriminant analysis (LDA) and Gaussian naive Bayes.*
#

# + [markdown] id="x_KNtJwlUNwq"
# ## **Sample Program**

# + [markdown] id="wrK0EyWfJpeN"
# #### Dataset Inspection

# + id="x2H9ELpRTbCn"
np.random.seed(0)

# + id="DDUHWh5zTbtc"
df5 = pd.read_csv('/content/T1/4.csv',header=None)

# + colab={"base_uri": "https://localhost:8080/", "height": 226} id="D6RpFuC3TtC9" outputId="1063c893-822c-44c5-9ce7-1683018dc475"
df5.head()

# + colab={"base_uri": "https://localhost:8080/", "height": 320} id="imwN02bUTuhM" outputId="14b56ca8-453d-4784-b640-5f4e9e869319"
df5.describe()

# + colab={"base_uri": "https://localhost:8080/"} id="SOiXtUB-XKAc" outputId="f1f106c4-ff56-4373-9742-65032bead106"
df5.shape

# + colab={"base_uri": "https://localhost:8080/"} id="5mY2njzkTwm2" outputId="5716290d-8de1-4320-d213-2bac81038ba8"
df5.info

# + [markdown] id="bNr4cbbcSKW7"
# #### **Scaling vs Normalization**
# >*Standardization:* 
# ***y = (x – mean) / standard_deviation***
#
# >*Normalization:* 
# ***y = (x – min) / (max – min)***

# + id="WxXkG1hoT3JF"
# Scaling
df5_original1 = np.random.exponential(size = 1000)
df5_scaled1 = minmax_scaling(df5_original1, columns = [0])

# + colab={"base_uri": "https://localhost:8080/", "height": 300} id="Atc3Dw00WQLE" outputId="93bf80c3-3e45-49ca-f519-6c573eb38041"
fig, ax=plt.subplots(1,2)
sns.distplot(df5_original1, ax=ax[0])
ax[0].set_title("Original Data")
sns.distplot(df5_scaled1, ax=ax[1])
ax[1].set_title("Scaled data")

# + id="ZXlEx5CTUVVU"
# Normalization
df5_original2 = np.random.exponential(size = 1000)
df5_normalized2 = stats.boxcox(df5_original2)

# + colab={"base_uri": "https://localhost:8080/", "height": 300} id="ZBxk5PH4WS67" outputId="d038ae3b-ba57-4fc9-fe02-5785684be30c"
fig, ax=plt.subplots(1,2)
sns.distplot(df5_original2, ax=ax[0])
ax[0].set_title("Original Data")
sns.distplot(df5_normalized2[0], ax=ax[1])
ax[1].set_title("Normalized data")

# + [markdown] id="IB6xDCMCSDc2"
# #### **Comparitive Study**
# *Let's see which method yields better accuracy for this dataset*

# + id="VuN1Fvr_ZENq"
data51 = df5.values
x5, y5 = data51[:, :-1], data51[:, -1]
x5 = x5.astype('float32')
y5 = LabelEncoder().fit_transform(y5.astype('str'))

# + id="CsBs934bfDTe"
# No Scaler
model51 = KNeighborsClassifier()

# + colab={"base_uri": "https://localhost:8080/"} id="rxndiZu3Z1b_" outputId="e3bf8ec1-90f2-494e-b16b-65da8d6fb024"
cv51 = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
n_scores_51 = cross_val_score(model51, x5, y5, scoring='accuracy', cv=cv51, n_jobs=-1, error_score='raise')
print('Accuracy: %.3f (%.3f)' % (np.mean(n_scores_51), np.std(n_scores_51)))

# + id="bi79kTccalMV"
data52 = df5.values
x5, y5 = data52[:, :-1], data52[:, -1]
x5 = x5.astype('float32')
y5 = LabelEncoder().fit_transform(y5.astype('str'))

# + id="jz5E0Dtget8a"
# MinMaxScaler
ms5 = MinMaxScaler()
model52 = KNeighborsClassifier()
pipeline52 = Pipeline(steps=[('t', ms5), ('m', model52)])

# + colab={"base_uri": "https://localhost:8080/"} id="tuegfxsme4Ou" outputId="2f48fa21-f775-4234-9f88-155ded612674"
cv52 = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
n_scores_52 = cross_val_score(pipeline52, x5, y5, scoring='accuracy', cv=cv52, n_jobs=-1, error_score='raise')
print('Accuracy: %.3f (%.3f)' % (np.mean(n_scores_52), np.std(n_scores_52)))

# + id="KzgY-rpqfRKG"
data53 = df5.values
x5, y5 = data53[:, :-1], data53[:, -1]
x5 = x5.astype('float32')
y5 = LabelEncoder().fit_transform(y5.astype('str'))

# + id="c6PQJ27wfv3A"
# StandardScaler
ss5 = StandardScaler()
model53 = KNeighborsClassifier()
pipeline53 = Pipeline(steps=[('t', ss5), ('m', model53)])

# + colab={"base_uri": "https://localhost:8080/"} id="HGCidTt0f7Kx" outputId="c996df44-f1b4-48a2-f78b-6c00dfe5fc23"
cv53 = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
n_scores_53 = cross_val_score(pipeline53, x5, y5, scoring='accuracy', cv=cv53, n_jobs=-1, error_score='raise')
print('Accuracy: %.3f (%.3f)' % (np.mean(n_scores_53), np.std(n_scores_53)))

# + [markdown] id="XM6KTjcjgdV8"
# *It can be inferred that the accuracy is maximum using the MinMaxScaler for this dataset*

StarcoderdataJupyterScriptsDedupFiltered