# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# + [markdown] application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "88399a27-9868-485f-9cb1-7b225a16e33a"}
# # Short Introduction
# The analysis explores the candy power ranking dataset from FiveThirtyEight (for more detailed informations see https://fivethirtyeight.com/features/the-ultimate-halloween-candy-power-ranking/).
#
# This notebook provides initial exploratory analysis of the data to better visualize and understnad variables.

# + [markdown] application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "86fd4150-0af0-41d0-b80d-b9b0c5fe711a"}
# # Overview

# + [markdown] application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "03610d76-4eec-43b7-b495-5324d7fd84fa"}
# First of all we load all needed packages.

# + _kg_hide-input=false application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "108046d0-324d-48ad-9dd7-c360b2ccdbb3"} _uuid="d629ff2d2480ee46fbb7e2d37f6b5fab8052498a" _cell_guid="79c7e3d0-c299-4dcb-8224-4455121ee9b0"
# %run "./Libraries"

# + _kg_hide-input=true application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "b4b9ea3d-ce30-472c-accc-ff029ef8257a"}
df = pd.read_csv('https://raw.githubusercontent.com/fivethirtyeight/data/master/candy-power-ranking/candy-data.csv')
df['competitorname'] = df.competitorname.apply(lambda l: l.replace('Õ', '\''))
display(df)

# + application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "ab6c705a-df5e-4ca1-96c9-f72dee5b4cdb"}
df.info()

# + application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "513e8456-3b94-4a09-a8cd-dca8ca29023a"}
df.isnull().sum()

# + application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "b0d9bec4-33ef-4816-aa41-76781809af2e"}
df['competitorname'] = df.competitorname.apply(lambda l: l.replace('Õ', '\''))
display(df)

# + application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "ce688cc9-4ec4-4baa-a55f-6ecdad568b3f"}
plt.figure(figsize=(8,19))
sns.barplot(x=df["winpercent"],y=df["competitorname"],order=df[["competitorname","winpercent"]].sort_values(by="winpercent",ascending=False).iloc[:,0],palette="PuRd_r")
plt.xlabel("Win percentage")
plt.ylabel("")
plt.title("Candy Ranking")

# + [markdown] application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "4294f945-a557-4162-9a55-a7646c62ca23"}
# The characteristics of the binary features (all features except for sugarpercent and pricepercent):

# + application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "a3d641d7-f406-4595-a1bf-a653943e7ea6"}
df[df.columns[1:-3]].agg(['sum','count'])

# + [markdown] application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "c0cf0fa1-d019-4c14-94f4-b6c0607890ff"}
# The characteristics of the non-binary features sugarpercent and pricepercent:

# + application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "b5996310-6177-4ed1-8b7e-a54e30f6231f"}
df[['sugarpercent', 'pricepercent']].describe().T

# + [markdown] application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "4f435000-376e-413c-afba-d1b72c2bc65b"}
# ## Top Ten & Worst Ten

# + [markdown] application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "c68fea3e-b071-4bf7-8ab1-96fe084e8e25"}
# For a first impression we take a short look at the top ten:

# + _kg_hide-input=false application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "d75bb093-a87a-4c4b-a533-ffbef6240fe3"}
df.sort_values('winpercent', ascending=False)[:10]

# + [markdown] application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "745c2049-b1bc-4512-b94c-71e724ba697e"}
# and the worst ten:

# + application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "734e5a4a-7944-466f-b269-890500127711"}
df.sort_values('winpercent')[:10]

# + [markdown] application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "fec02c33-f5c5-42fd-9fcf-b36fbdee4071"}
# We can see that chocolate and perhaps peanutyalmondy have a positive effect to winpercent. Additionaly fruity seems to have a negative impact.
#
# Now let us using statistical and machine learning methods for a more reliable analysis. Because nougat and crispedricewafer are rare we will omit them.

# + [markdown] application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "f6f913b0-23a6-40f6-903b-c4f9886d52cf"}
# # Box Plot

# + application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "1cb6425a-81ee-4b76-86a0-ef1afd2508c4"}
display(df)

# + [markdown] application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "f2c08f84-8dc5-45d7-bec5-431d437bb551"}
# # Histogram

# + application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "c3112dfa-4470-4119-957e-1fbcafef74b0"}
display(df)

# + [markdown] application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "3d18c39b-b7b0-4818-b3ca-8bd24142d69c"}
# # Correlation Matrics
#
# The correlation matrics contains every correlation coefficiant for every pair of features and target variable.

# + application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "e26c06c7-8d52-4baf-a90f-15c77645b80a"}
plt.figure(figsize=(10,10))
mask=np.zeros_like(df.corr(),dtype=np.bool)
mask[np.triu_indices_from(mask)]=True
sns.heatmap(data=df.corr(),annot=True,square=True,mask=mask,cmap="GnBu",linewidths=1,linecolor="white")
plt.title("Candy characteristics correlation")

# + [markdown] application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "9097677f-43c8-40e6-ba93-446a5c005a5b"}
# Since the matrics is large and therefore a little bit confusing, here the top ten of correlations (positive and negative):

# + application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "2f0e4fae-e697-4b3b-ac91-775c82bdeb44"}
corr_df = pd.DataFrame([(f1, f2, df.corr()[f1].loc[f2]) for (f1, f2) in itertools.combinations(df.columns[1:].sort_values(), 2)],
             columns=['feature1', 'feature2', 'corr']).sort_values('corr')

corr_df = corr_df.iloc[(-corr_df['corr'].abs()).argsort()].reset_index(drop=True)
corr_df[:10]

# + [markdown] application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "80e464b0-eb8b-4d3e-b37b-4d37d49f34de"}
# And here the correlation coefficients for winpercent only:

# + application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "b1cc5f6a-799c-47f7-a230-37a2efeb3fa7"}
corr_df[(corr_df.feature1 == 'winpercent') | (corr_df.feature2 == 'winpercent')]

# + [markdown] application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "850cd23f-cb0e-45e1-8c31-0e6a2592486c"}
# We see that chocolate and winpercent have the largest correlation. Also bar and winpercent just as peanutyalmondy and winpercent have noticeable correlations. But we have to be careful because also chocolate and bar have a large correlation.

# + [markdown] application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "aefd8179-7e49-4051-b4f4-9211f18add16"}
# # Summary
#
# We cleansed the data for some anomolous characters and checked for outliers.  We then visualized the features to understand correlated variables and will further analyze these variables in the next notebook.

# + application/vnd.databricks.v1+cell={"title": "", "showTitle": false, "inputWidgets": {}, "nuid": "96f85f01-2788-4184-a3d9-c71e89f7675c"}


StarcoderdataJupyterScriptsDedupFiltered