# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # Capstone 2 Exploratory Data Analysis
#
# In this part we will perform Exploratory Data Analysis of out dataset. During this analysis we will try to answer several questions about our data.
# What is the minimum, average and maximum age in the dataset?
# Which gender has more positive cases? What is the percentage of positive cases in each gender?
# What features are intercorellated and what are the coefficients of correlation?
# What feature is the most correlated with the target?
# What feature is the least correlated with the target?
# What feature explains the most of variance?

import pandas as pd
import numpy as np
import os
import matplotlib.pyplot as plt
import seaborn as sns
import pickle

df = pd.read_csv('D:\Tutorials\SDST\My Projects\Capstone2\Data Wrangling\Diabetes_Data_wrangling.csv')

df.head()

df.info()

# Let's take closer look at our data. First we'll find out if we have the same amount of male and female patients.

df['Gender'].value_counts()

plt.hist(df['Gender'], bins=10)
plt.xlabel('Gender')
plt.ylabel('Counts')
plt.show();

# As we can see, there are 328 males and 192 females. Now let's take a look at the patients' age.

df_age = df['Age']

df_age.describe()

plt.hist(df_age, bins=10)
plt.xlabel('Age')
plt.ylabel('Count')
plt.grid()
plt.show;

# We can see that the data is normally distributed. The mean age of our patients is 48 years old with the standard deviation of approximately 12 years. The youngest patiens is 16 years old and the oldest one is 90 years old.
#
# Now let's take a look which gender has more positive cases and what it the percentage of positive cases for each gender.

df_gender = df[['Gender', 'class']]
df_gender['class'].value_counts()

class_pos = df[df['class'] == 'Positive']
prop_pos = (len(class_pos) / len(df))*100
print(prop_pos)

plt.hist(df_gender['class'], bins=10)
plt.xlabel('Cases')
plt.ylabel('Count')
plt.grid()
plt.show;

# #### Overall, we have 520 cases, 320 of which are positive and 200 are negative. Percent of positive cases is 61.5%, negative - 38.5%.

df_male = df_gender[df_gender['Gender'] == 'Male']
df_female = df_gender[df_gender['Gender'] == 'Female']

df_male['Gender'].value_counts()

df_male['class'].value_counts()

df_male_pos = df_male[df_male['class'] == 'Positive']
df_male_neg = df_male[df_male['class'] == 'Negative']
df_male_pos_prop = len(df_male_pos)/len(df_male)
df_male_pos_perc = df_male_pos_prop * 100
print(df_male_pos_perc)

plt.figure(figsize=[15, 5])
plt.hist(df_male['class'], bins=10)
plt.xlabel('Class (males)')
plt.ylabel('Count')
plt.grid()
plt.show();

# We can see, that 147 out of 328 males (approximately 44.8%) in the dataset were classified as 'Positive'. Since we only have two classes, we can assume that aproximately 55.2% of males were classified as 'Negative'.
#
# Now let's find out the percentage for females. 

df_female['class'].value_counts()

df_female['Gender'].value_counts()

df_female_pos = df_female[df_female['class'] == 'Positive']
df_female_pos_prop = len(df_female_pos)/len(df_female)
df_female_pos_perc = df_female_pos_prop * 100
print(df_female_pos_perc)

plt.figure(figsize=[15, 5])
plt.hist(df_female['class'], bins=10)
plt.xlabel('Class (females)')
plt.ylabel('Count')
plt.grid()
plt.show();

# As we can see, 173 out of 192 females (approximately 90.1%) in the dataset were classified as 'Positive'. Since we only have two classes, we can assume that approximately 9.9% of females were classified as 'Negative'. The data is heavily skewed towards 'Positive' class and is highly unbalanced.
#
# Let's take a look at our original dataset once again.

df.head()

# We only have one numeric column in the dataset - 'Age'. All other columns are categorical values which we need to encode in order to perform future exploratory analysis and modeling. Let's check if those categorical variables are binary.

df_enc = df

#numerical column
num_col = df_enc['Age']

#categorical columns
cat_cols = df_enc.columns.drop('Age')

#see how many unique values for each feature
df_enc[cat_cols].apply(lambda x: x.nunique(), axis=0)

# It seems like all our categorical columns have two unique values. For all columns except 'Gender' and 'class' those values are strings 'Yes' and 'No', for 'Gender' column - strings 'Male' and 'Female' and for 'class' column - strings 'Positive' and 'Negative'. Since all of the values are binary, we can encode them using pandas .map() function. In order to keep our data from increasing dimensionality we will not use pandas.get_dummies()

df_enc['Gender'] = df_enc['Gender'].map({'Female':1, 'Male':0})

for column in df_enc.columns:
    if column == 'Age':
        continue
    elif column == 'Gender':
        continue
    else:
        df_enc[column] = df_enc[column].map({'No':0, 'Yes':1, 'Negative':0, 'Positive':1})

df_enc.head()

# The list of encoding labels and values: Male - 0, Female - 1, No - 0, Yes - 1, Negative - 0, Positive - 1.
#
# Now let's see how different features are correlated to each other using seaborn heatmap.

corr = df_enc.corr()
plt.figure(figsize=[15, 7])
sns.heatmap(corr);

# Let's take a look at the heatmap. We are interested in how our target feature ('class') is correlated with other features (which are a patient's charachteristics).
# From the map we can derive that the highest correlation is with 'Polyuria' (production of abnormally large volumes of dilute urine) and 'Polydipsia' (abnormally great thirst as a symptom of disease). It is also correlated with 'Gender' (we saw that percentage of women with positive class in the dataset is 90.1% against 44.8% of men), 'sudden weight loss', and 'partial paresis'(a condition of muscular weakness caused by nerve damage or disease).

# Now let's look at the numbers. Here is a table with pairwise correlation of each feature.

corr

# We will only consider those features which have correlation coefficients above 'moderate' (0.40 - 0.59) and 'strong' (0.60-0.79) thresholds.
# We are paricularly interested in out target variable, so let's see how it looks.

corr['class'].sort_values(ascending=False)

# As it is clear from the table we only have two features whose correlation with the target can be considered 'strong'. Those are 'Polyuria' (0.67) and 'Polydipsia (0.65). The features with 'moderate' correlation are 'Gender' (0.45), 'Sudden Weight Loss' (0.44) and 'Partial Paresis' (0.43). Surprisingly enough, 'Age' does not seem to be a significant factor with the coefficient of only 0.11. Our target is also very weakly correlated with 'Obesity', which a lot of people think would be one of the primary causes of diabetes. It is also negatively correlated with 'Alopecia' - a condition that causes hair to fall out in small patches.
# Let's create a dataframe for above mentioned important features and see how they are correlated with each other.

# +
# features = pd.DataFrame(corr['class'].sort_values(ascending=False))
# features_list = features.index.to_list()
# for feature in features_list:
#    display(corr[feature].sort_values(ascending=False))
# -

imp_feat = df_enc[['Polyuria', 'Polydipsia', 'Gender', 'sudden weight loss', 'partial paresis', 'class']]
imp_feat.head()

# Now let's take a look only at those features which correlation coeffitient with a target is above 0.4 and see how they are correlated with each other. If we will find independent variables which are highly correlated it might imply that feature collinearity takes place.

corr_imp = imp_feat.corr()

top_corr_features = corr_imp.index
plt.figure(figsize=(10,10))
g=sns.heatmap(df_enc[top_corr_features].corr(),annot=True,cmap="RdYlGn")

# As we can see, among these features we don't have those which are higly correlated with each other. We can assume that there is no feature collinearity in our dataset and it will not affect our future model's performance. Let's convert the correlation matrix to the dataframe and save it for later use.

corr_imp_df = pd.DataFrame(corr_imp)

corr_imp_df

# Now let's find top-5 feature-to-target and feature-to-feature correlations. We already know that 'Polyuria' and 'Polydipsia' are correlated the most with the target, but let's check if it's true one more time.

f_to_t = (corr_imp_df.where(np.triu(np.ones(corr_imp_df.shape), k=1).astype(np.bool)).stack().sort_values(ascending=False))

print(f_to_t[:5])

# As we expected, those two features are indeed the most correlated with target. Now, let's find out what features are the most correlated with each other. In order to do it we will drop our target feature from the correlation matrix.

corr_imp_df_nt = corr_imp_df.drop('class', axis=1)
f_to_f = (corr_imp_df_nt.where(np.triu(np.ones(corr_imp_df_nt.shape), k=1).astype(np.bool)).stack().sort_values(ascending=False))

print(f_to_f[:5])

# Two features that have the highest intercorrelation are 'Polyuria' to 'Polydipsia' (correlation 0.60) and 'Polyuria' to 'partial paresis' (0.45). Now let's convert the series to dataframes and save them as pickle files.

f_to_t_df = pd.DataFrame(f_to_t[:5])
f_to_f_df = pd.DataFrame(f_to_f[:5])

corrpath = '../Correlations'
if not os.path.exists(corrpath):
    os.mkdir(corrpath)
diabetes_ftot_df_path = os.path.join(corrpath, 'Feature-to-Target.pkl')
if not os.path.exists(diabetes_ftot_df_path):
    with open(diabetes_ftot_df_path, 'wb') as ft:
        pickle.dump(f_to_t_df, ft)
diabetes_ftof_df_path = os.path.join(corrpath, 'Feature-to-Feature.pkl')
if not os.path.exists(diabetes_ftof_df_path):
    with open(diabetes_ftof_df_path, 'wb') as ff:
        pickle.dump(f_to_f_df, ff)

# And now let's built a pairplot in order to see the visual representation of our features' interrelationships.

sns.pairplot(corr_imp_df);

#saving the notebook and dataset
datapath = 'D://Tutorials/SDST/My Projects/Capstone2/EDA'
df_enc.to_csv('df_enc', index=False)

# ## Exploratory Data Analysis summary
#
# Summarizing results of the EDA we can conclude that our dataset is imbalanced - there are 320 positive cases (61.5%) and 200 negative cases (38.5%). The distribution of patients age is approximately normal with mean age of 48 and standard deviation of 12.2. The youngest patient is 16 and the oldest patient is 90 years old. There are 328 male patients and 192 female patients. Percent of male patients classified as 'Positive' is approximately 44.8% (147 out of 328) and percent of female patients classified as 'Positive' is approximately 90.1% (173 out of 192). Features that are 'strongly' and 'moderately' correlated to the target are (from stongest to weakest correlation) 'Polyuria', 'Polydipsia', 'Gender', 'sudden weight loss' and 'partial paresis'.

StarcoderdataJupyterScriptsDedupFiltered