"""
## Problem Definition / Goals
Our goals in this analysis are:

> Need to perform clustering to summarize customer segments

> Given the customer routine spending, can we predict at which segment a customer can be associated?
"""
"""
## 3. Features

The features of this dataset are divided in three macro categories:

**People**

* ID: Customer's unique identifier
* Year_Birth: Customer's birth year
* Education: Customer's education level
* Marital_Status: Customer's marital status
* Income: Customer's yearly household income
* Kidhome: Number of children in customer's household
* Teenhome: Number of teenagers in customer's household
* Dt_Customer: Date of customer's enrollment with the company
* Recency: Number of days since customer's last purchase
* Complain: 1 if customer complained in the last 2 years, 0 otherwise

**Products**

* MntWines: Amount spent on wine in last 2 years
* MntFruits: Amount spent on fruits in last 2 years
* MntMeatProducts: Amount spent on meat in last 2 years
* MntFishProducts: Amount spent on fish in last 2 years
* MntSweetProducts: Amount spent on sweets in last 2 years
* MntGoldProds: Amount spent on gold in last 2 years


**Promotion**

* NumDealsPurchases: Number of purchases made with a discount
* AcceptedCmp1: 1 if customer accepted the offer in the 1st campaign, 0 otherwise
* AcceptedCmp2: 1 if customer accepted the offer in the 2nd campaign, 0 otherwise
* AcceptedCmp3: 1 if customer accepted the offer in the 3rd campaign, 0 otherwise
* AcceptedCmp4: 1 if customer accepted the offer in the 4th campaign, 0 otherwise
* AcceptedCmp5: 1 if customer accepted the offer in the 5th campaign, 0 otherwise
* Response: 1 if customer accepted the offer in the last campaign, 0 otherwise

**Place**

* NumWebPurchases: Number of purchases made through the company’s web site
* NumCatalogPurchases: Number of purchases made using a catalogue
* NumStorePurchases: Number of purchases made directly in stores
* NumWebVisitsMonth: Number of visits to company’s web site in the last month
"""
"""
## Preparing the tools

Let's import the main libraries
"""
# This Python 3 environment comes with many helpful analytics libraries installed
# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from datetime import datetime
import seaborn as sns
import matplotlib.pylab as pylab
params = {'legend.fontsize': 'x-large',
          'figure.figsize': (15, 5),
         'axes.labelsize': '18',
         'axes.titlesize':'24',
         'xtick.labelsize':'12',
         'ytick.labelsize':'12'}
pylab.rcParams.update(params)

# Models from Scikit-Learn
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import MinMaxScaler

# Model evaluation
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.model_selection import RandomizedSearchCV, GridSearchCV
from sklearn.metrics import confusion_matrix, classification_report
from sklearn.metrics import precision_score, recall_score, f1_score
from sklearn.metrics import plot_roc_curve

# Input data files are available in the read-only "../input/" directory
# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory

import os
for dirname, _, filenames in os.walk('/kaggle/input/customer-personality-analysis'):
    for filename in filenames:
        print(os.path.join(dirname, filename))

# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using "Save & Run All" 
# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session
data = pd.read_csv(os.path.join(dirname, filename), sep='\t', parse_dates=['Dt_Customer'])
"""
# Data Exploration

In this section we are going to analyze our data in order to visualize and compare features and their values, try to look at the creation of new features if necessary and look for other kind of pre-processing needs as normalization, transformation and so on.

The following questions can guide our EDA:

* Which kind of customer tends to accept promotions at first offer? A the second / third and so on...?
* Which kind of customer make more web visit of the web?
* Which kind of customer make more purchases with discounts?
* What is the best place (web/store) in which customer spends more?
* What is the average age of the enrolled customer?
* Who complain more? Customer with High Education or Low Education? Aged people or young?

In update...

Add description of the features added, preprocessing and so on...
"""
"""
## Let's start

Let's discover the dataset structure and compute some new feature
"""
data.info()
"""
Some columns could be converted in categorical type and we can add more information to this data. Let's work with features
"""
df_copy = data.copy()
"""
### Feature Engineering
What we've done:
* Aggregated some Education categories
* Aggregated some Marital Status categories
* Computed a new feature: the fidelity years (customer's loyalty years)
* Computed a new feature: the age of the customers
* Computed a new feature: Sum up the kid and teen of customers as unique feature Children
* Computed a new feature: If a customer has or not a child
* Computed a new feature: Sum of all expenditures in different categories of products

"""
def campaign_acceptance(row):
    toReturn = row
    campaigns = ['AcceptedCmp1', 'AcceptedCmp2', 'AcceptedCmp3', 'AcceptedCmp4', 'AcceptedCmp5', 'Response']
    toReturn['accepted'] = 0
    if row[campaigns].any() == 1:
        toReturn['accepted'] = 1
    
    return pd.Series(toReturn, index=list(toReturn.keys()))
# There are too much redundant value for Education
df_copy['Education'].replace({'Basic': 'Undergraduate', '2n Cycle': 'Undergraduate'}, inplace=True)

# The same for Marital Status
df_copy['Marital_Status'].replace({'Absurd': 'Single', 'YOLO': 'Single',
                                    'Alone': 'Single', 'Widow': 'Single', 'Together': 'Couple'}, inplace=True)

# Let's see the customers fidelity years and their age
df_copy['fidelity_years'] = df_copy['Dt_Customer'].dt.year - df_copy['Year_Birth']
df_copy['age'] = datetime.now().year - df_copy['Year_Birth']

# sum up the kid and teen as children
df_copy['children'] = df_copy['Kidhome'] + df_copy['Teenhome']
df_copy['has_child'] = df_copy['children'] > 0

# Calculate total amount of expenses in different kind of products
df_copy['total_spend'] = df_copy['MntWines'] + df_copy['MntFruits'] + df_copy['MntMeatProducts'] +\
                        df_copy['MntFishProducts'] + df_copy['MntSweetProducts'] + df_copy['MntGoldProds']

# Let's aggregate the acceptance of compaigns in one feature
df_copy = df_copy.apply(campaign_acceptance, axis=1)

# Let's convert the strings in categories
for label, content in df_copy.items():
    if pd.api.types.is_string_dtype(content):
        df_copy[label] = content.astype('category').cat.as_ordered()
        
# Remove the NaN values from Income
df_copy['Income'].dropna(inplace=True)
"""
Moreover it seems there are columns without a description and with only one value, we are going to remove those columns
"""
df_copy = df_copy.drop(['Z_Revenue', 'Z_CostContact'], axis=True)
df_copy.columns
df_copy['accepted'].value_counts()
"""
### Discovering Insights
"""
"""
The company has a lot of Graduate customers follower by PhD people. From the point of view of civil status they are mostly in couple (married and together)
"""
fig, (ax1, ax2) = plt.subplots(figsize=(20, 10), ncols=2)
grouped = df_copy.groupby('Education').count().reset_index()

sns.countplot(ax=ax1, x="Education", data=df_copy, palette= sns.color_palette("hls", 4))

sns.countplot(ax=ax2, x='Marital_Status', data=df_copy, palette= sns.color_palette("hls", 4))

ax2.set_xlabel('Marital Status')
ax2.set_title('Civil Status of Customers')
ax1.set_title('Education status of customers')
ax1.set_ylabel('Count');
ax2.set_ylabel('Count');
has_child_perc = df_copy[df_copy['has_child'] == True].shape[0] / df_copy.shape[0]
has_nochild_perc = df_copy[df_copy['has_child'] == False].shape[0] / df_copy.shape[0]
fig, ax = plt.subplots(figsize=(10, 10))
# Anche qui vedere la percentuale di quanti hanno i figli

sns.barplot(x=['Has Child', 'No Child'], y= [has_child_perc, has_nochild_perc], 
            palette = sns.color_palette('hls', 2));
ax.set_ylabel('Percentage of customers')
ax.set_title(f'The {round(has_child_perc, 2) * 100}% of customers has children');
"""
Less than 1% of customers complain about company services, but we can see that in this little pool of customer graduate and in-couple people tend to complain more.
"""
# Complain in the last 2 years
complain = df_copy[df_copy['Complain'] == 1]

fig, (ax1, ax2) = plt.subplots(figsize=(20, 10), ncols=2)

ed_compl_perc = complain.groupby('Education').count()['ID'] / df_copy.shape[0] * 100
marital_compl_perc = complain.groupby('Marital_Status').count()['ID'] / df_copy.shape[0] * 100

sns.barplot(ax=ax1, x = ed_compl_perc.index, y = ed_compl_perc.values, palette = sns.color_palette('hls', 4))
sns.barplot(ax=ax2, x = marital_compl_perc.index, y = marital_compl_perc.values, 
            palette = sns.color_palette('hls', 4));

ax2.set_xlabel('Marital Status')
ax2.set_title('Civil Status of Customers')
ax1.set_title('Education status of customers')
ax1.set_ylabel('Percentage of complain customers');
print(f'Percentage of complaining cutomers: {round((complain.shape[0] / df_copy.shape[0]) * 100, 2)}%')
"""
Also the acceptance rate seems to be very low, but even though the acceptance rate is very low for all campaigns, it seems that people "gave up" at the last campaign. In few words, it seems that the last campaign was more effective.
"""
campaigns = {
    'Campaign': ['First', 'Second', 'Third', 'Fourth', 'Fifth', 'Last'],
    'Acceptance Rate': [(df_copy[df_copy['AcceptedCmp1'] == 1].shape[0] / df_copy.shape[0]) * 100,
                       (df_copy[df_copy['AcceptedCmp2'] == 1].shape[0] / df_copy.shape[0]) * 100,
                       (df_copy[df_copy['AcceptedCmp3'] == 1].shape[0] / df_copy.shape[0]) * 100,
                       (df_copy[df_copy['AcceptedCmp4'] == 1].shape[0] / df_copy.shape[0]) * 100,
                       (df_copy[df_copy['AcceptedCmp5'] == 1].shape[0] / df_copy.shape[0]) * 100,
                       (df_copy[df_copy['Response'] == 1].shape[0] / df_copy.shape[0]) * 100]
}
fig, (ax1, ax2) = plt.subplots(figsize=(20, 10), ncols=2, sharex=True)

offers = np.array([1, 2, 3, 4, 5])
sns.barplot(ax=ax1, x = campaigns['Campaign'], y = campaigns['Acceptance Rate'], palette = sns.color_palette('hls', 6))
#for offer in offers:
#    sns.barplot(x = offer, y = (df_copy[df_copy['AcceptedCmp' + str(offer)] == 1].shape[0] / df_copy.shape[0]) * 100)
sns.lineplot(ax=ax2, x = campaigns['Campaign'], y= campaigns['Acceptance Rate'], markers=True, dashes=False)    

fig.suptitle('Campaigns Acceptance Rate', fontsize=24)
ax1.set_ylabel('Acceptance rate')
ax1.set_xlabel('Campaigns')
ax2.set_xlabel('Campaigns');
"""
In general only the 27% of customers joined to the promoted campaigns
"""
fig, ax = plt.subplots(figsize=(7, 7))
#define data
slices = [df_copy[df_copy['accepted'] == 1].shape[0], 
          df_copy[df_copy['accepted'] == 0].shape[0]]

labels = ['Accepted', 'Rejected']

#define Seaborn color palette to use
colors = sns.color_palette('pastel')[0:2]

#create pie chart
ax.pie(slices, labels = labels, colors = colors, autopct='%.0f%%');
ax.set_title('Campaign Joined People');
"""
customers purchases frequency seems to be high, on average 49 days pass from one purchase to the next one.
"""
df_copy['Recency'].describe()
"""
We are going to work more on this feature in the next chunks. 
"""
"""
Customers focus their purchases mainly on meat products
"""
fig, ax = plt.subplots(figsize=(10, 10))

labels = ['Meat', 'Fish', 'Sweet', 'Gold']
values = [df_copy['MntMeatProducts'].mean(), df_copy['MntFishProducts'].mean(),
         df_copy['MntSweetProducts'].mean(), df_copy['MntGoldProds'].mean()]

sns.barplot(x = labels, y=values, palette = sns.color_palette('hls', 4))

ax.set_xlabel('Products')
ax.set_ylabel('Average Expenditure ($)')
ax.set_title('Average expenditure by products');
"""
Now we can see some outliers from the point of view of Income, so we are going to remove them
"""
df_copy = df_copy[df_copy['Income'] < 120000]
df_copy['Income'].describe()
"""
Customers are centered around 50K dollars of annual Income. I guess that customers are mostly middle class people.
"""
fig, ax = plt.subplots(figsize=(15, 10))
sns.histplot(df_copy['Income'], stat='density');
df_copy['total_spend'].describe()
fig, ax = plt.subplots(figsize=(15, 10))
sns.histplot(df_copy['total_spend'], stat='density')

ax.set_xlabel('Total Expenditures($)');
"""
Total expenditure is concentrated around an average value of 600+ dollars with a high level of standard deviation. Thus, we can ipotize the presence of very loyal customers and just occasional customers.
"""
"""
Let's try to compute the purchases made in a store or on the web in percentage.
"""
ed_group = df_copy.groupby('Education').sum().reset_index()
def compute_percentage_purchases(row, group):
    toReturn = row
    toReturn['WebPerc'] = 0
    toReturn['StorePerc'] = 0
    
    # Get Total web and store purchases
    total_web_purchases = group['NumWebPurchases'].sum()
    total_store_purchases = group['NumStorePurchases'].sum()
    
    if row['Education'] == 'Graduation':
        # Divide the customer store and web purchases by the total
        toReturn['WebPerc'] = row['NumWebPurchases'] / total_web_purchases
        toReturn['StorePerc'] = row['NumStorePurchases'] / total_store_purchases
    
    if row['Education'] == 'Undergraduate':
        toReturn['WebPerc'] = row['NumWebPurchases'] / total_web_purchases
        toReturn['StorePerc'] = row['NumStorePurchases'] /total_store_purchases
    
    if row['Education'] == 'PhD':
        toReturn['WebPerc'] = row['NumWebPurchases'] / total_web_purchases
        toReturn['StorePerc'] = row['NumStorePurchases'] / total_store_purchases
        
    if row['Education'] == 'Master':
        toReturn['WebPerc'] = row['NumWebPurchases'] /total_web_purchases
        toReturn['StorePerc'] = row['NumStorePurchases'] / total_store_purchases
    
    return pd.Series(toReturn, index = list(toReturn.keys()))
df_copy = df_copy.apply(compute_percentage_purchases, group = ed_group, axis=1)
#df_copy['NumStorePurchasesPerc'] = df_copy['NumStorePurchases'] / ed_group['NumWebPurchases']
# Let'se the mean of the computed value
ed_group = df_copy.groupby('Education').mean().reset_index()
"""
There aren't very difference between the percentage of web and store purchases by education level, for example Graduation category has the similar percentage both for store and web. Maybe this problem is due to the unbalanced level in Education categories.

**Question for readers:** Can be sampling a good solution in this case?
"""
fig, (ax1, ax2) = plt.subplots(figsize=(20, 10), ncols = 2)

sns.barplot(ax=ax1, x = ed_group['Education'], y = ed_group['StorePerc'], palette = sns.color_palette('hls', 4));
sns.barplot(ax=ax2, x = ed_group['Education'], y = ed_group['WebPerc'], palette = sns.color_palette('hls', 4));

ax1.set_ylabel('Purchases Percentage in Store')
ax1.set_title('Store')
ax2.set_ylabel('Purchases Percentage on Web')
ax2.set_title('Web')
plt.suptitle('Percentage Purchases by Education level', fontsize=24);
"""
## Cluster

We are going to clusterize our customers as goal of the notebook and to discover other insights
"""
df_copy.dropna(inplace=True)
from sklearn.cluster import KMeans
from sklearn.preprocessing import MinMaxScaler
# define the attribute to cluster ['age', 'children', 'total_spend', 'Income']
cluster_features = ['age', 'children', 'total_spend', 'Income'] #['children', 'total_spend', 'Income', 'WebPerc', 'StorePerc']
toCluster = df_copy[cluster_features]
# Normalization
scaler = MinMaxScaler()
toCluster[cluster_features] = scaler.fit_transform(toCluster)
# Elbow method to select the right number of cluster
K = range(1,10)
distortions = []
for k in K:
    kmeanModel = KMeans(n_clusters=k)
    kmeanModel.fit(toCluster)
    distortions.append(kmeanModel.inertia_)
plt.figure(figsize=(16,8))
plt.plot(K, distortions, 'bx-')
plt.xlabel('k')
plt.ylabel('Distortion')
plt.title('The Elbow Method showing the optimal k')
plt.show()
clusters = KMeans(n_clusters = 4, random_state=42)
labels = clusters.fit_predict(toCluster)
df_copy['cluster'] = labels
df_copy['cluster'].value_counts()
df_copy.groupby('cluster').mean()
fig, ax = plt.subplots(figsize=(15, 10))

df_copy = df_copy[df_copy['Income'] < 200000]


x_axis = 'Income'
y_axis = 'total_spend'
sns.scatterplot(x = df_copy[df_copy['cluster'] == 0][x_axis],
                y = df_copy[df_copy['cluster'] == 0][y_axis], 
            palette='salmon', legend = 'auto')

sns.scatterplot(x= df_copy[df_copy['cluster'] == 1][x_axis], 
                y = df_copy[df_copy['cluster'] == 1][y_axis], 
            palette='blue', legend = 'auto')

sns.scatterplot(x= df_copy[df_copy['cluster'] == 2][x_axis], 
                y = df_copy[df_copy['cluster'] == 2][y_axis], 
            palette='green', legend = 'auto')

sns.scatterplot(x= df_copy[df_copy['cluster'] == 3][x_axis], 
                y = df_copy[df_copy['cluster'] == 3][y_axis], 
            palette='red', legend = 'auto')

linedf = pd.DataFrame({'x': df_copy['Income'].values,'y': np.full(df_copy.shape[0], 1400)})
plt.plot(linedf['x'], linedf['y'], color='blue', linestyle='dashed')
#sns.lineplot(x = 'x', y = 'y', data = linedf, palette = 'salmon')

linedf2 = pd.DataFrame({'x': df_copy['Income'].values,'y': np.full(df_copy.shape[0], 800)})
plt.plot(linedf2['x'], linedf2['y'], color='salmon', linestyle='dashed')
#sns.lineplot(x = 'x', y = 'y', data = linedf2, palette = 'red')

ax.set_title("Income vs Total amount spent")
ax.set_xlabel('Income')
ax.set_ylabel('Total Spent')
ax.legend(['Rich Class', 'Middle Class', '0', '1', '2', '3']);
"""
It is interesting to note how rich people (people with high annual income) tends to overlap with middle class people and in few cases with poor class. Indeed, it is possible to note different samples with high annual income and low total expenditure in products and, on the other side, samples with low annual income with high total spending. In few words, here we have a depiction of different behaviors and habits between rich and poor people, where true rich people tend to handle and hold their money for investments (we can ipotize) and poor people that are not able to manage money and tend to spend more to show off their no-poor condition. A classic example is Warren Buffet, he is a multi-bilionare but he lives in the same house bought in 1950s years and he don't spend its money if it not necessary, instead in a lot of cases poorer class tend to live beyond their means. We can have more evidence by the following charts, where are compared the cluster 3 containing people with a highest mean of annual income and cluster 1 containing people with the lowest annual income in mean.
"""
cluster1 = df_copy[df_copy['cluster'] == 3]
cluster1 = cluster1[cluster1['age'] < 120]
cluster2 = df_copy[df_copy['cluster'] == 1]
cluster2 = cluster2[cluster2['age'] < 120]

fig, (ax1, ax2) = plt.subplots(figsize=(15, 20), nrows=2)

sns.regplot(ax=ax1, x = cluster1['age'], y = cluster1['total_spend'])
sns.regplot(ax=ax2, x = cluster2['age'], y = cluster2['total_spend'])

from scipy import stats
        
slope1, intercept2, r_value2, pv2, se2= stats.linregress(cluster1['age'], cluster1['total_spend'])
slope2, intercept2, r_value2, pv2, se2 = stats.linregress(cluster2['age'], cluster2['total_spend']) 
print(slope1, slope2)
"""
The total spend of rich people has a decreasing trend as well as the age increase with a negative slope of -0.09, instead the poor people have a slightly opposite behaviour. This means rich people at age increasing tend to save their money, meanwhile poor people increase their expenditures.

**Note:** Of course, it is a my opinion, I'm really far to offend someone. It is just what I'm reading from analyzed data and it is just an attempt to show a my intuition in light of the above chart. Is it my reasoning linear and according with the shown charts?

"""
"""
### Let's add more features to our customers

I want to segment the customer by different features as age, fidelity years, income and recency days. I borrowed from the linked notebook: *[Customer Personality Analysis with Python](https://thecleverprogrammer.com/2021/02/08/customer-personality-analysis-with-python/)*.
"""
#Create Age segment
cut_labels_Age = ['Young', 'Adult', 'Mature', 'Senior']
cut_bins = [0, 30, 45, 65, 120]
df_copy['age_group'] = pd.cut(df_copy['age'], bins=cut_bins, labels=cut_labels_Age)

cut_labels_Age = ['0-30', '30-45', '45-65', '65+']
cut_bins = [0, 30, 45, 65, 120]
df_copy['age_range'] = pd.cut(df_copy['age'], bins=cut_bins, labels=cut_labels_Age)

#Create Income segment
cut_labels_Income = ['Low income', 'Low to medium income', 'Medium to high income', 'High income']
df_copy['Income_group'] = pd.qcut(df_copy['Income'], q=4, labels=cut_labels_Income)

#Create Fidelity segment. The time the customer is client
cut_labels_Seniority = ['New customers', 'Discovering customers', 'Experienced customers', 'Old customers']
df_copy['Fidelity_group'] = pd.qcut(df_copy['fidelity_years'], q=4, labels=cut_labels_Seniority)

# Create Recency segment. Recency is some sort of purchases frequency
cut_labels_Recency = ['High Frequency', 'Normal Frequency', 'Low Frequency', 'Occasional']
df_copy['Recency_group'] = pd.qcut(df_copy['Recency'], q=4, labels=cut_labels_Recency)
#data=data.drop(columns=['Age','Income','Seniority'])
fig, ax = plt.subplots(figsize=(20, 20), nrows=2, ncols=2)
ax1 = ax[0][0]
ax2 = ax[0][1]
ax3 = ax[1][0]
ax4 = ax[1][1]

sns.countplot(ax = ax1, x = df_copy['age_group'], palette = sns.color_palette('hls', 4))
ax1.set_xlabel('Age Group')

sns.countplot(ax = ax2, x = df_copy['Income_group'], palette = sns.color_palette('hls', 4))
ax2.set_xlabel('Income Group')

sns.countplot(ax = ax3, x = df_copy['Fidelity_group'], palette = sns.color_palette('hls', 4))
ax3.set_xlabel('Fidelity Group')

sns.countplot(ax = ax4, x = df_copy['Recency_group'], palette = sns.color_palette('hls', 4))
ax4.set_xlabel('Recency Group');
"""
Let's suppose the company is promoting discounts campaign. The below chart show us possible target that could partecipate to campaign, in particular is possible to focus more effort on young and adult people with low income.
"""
income_age = pd.crosstab(df_copy['age_group'], df_copy['Income_group']).reset_index()
income_age['total'] = income_age['Low income'] + income_age['Low to medium income'] + income_age['Medium to high income'] + income_age['High income']
income_age['low-perc'] = round(income_age['Low income'] / income_age['total'], 2)
income_age['lowmed-perc'] = round(income_age['Low to medium income'] / income_age['total'], 2)
income_age['medhigh-perc'] = round(income_age['Medium to high income'] / income_age['total'], 2)
income_age['high-perc'] = round(income_age['High income'] / income_age['total'], 2)
income_young_dict = {
    'age_group': 'Young',
    'income': ['low-income', 'low-med-income', 'med-high-income', 'high-income'],
    'percentage': [income_age[income_age['age_group'] == 'Young']['low-perc'].values[0],
                  income_age[income_age['age_group'] == 'Young']['lowmed-perc'].values[0],
                  income_age[income_age['age_group'] == 'Young']['medhigh-perc'].values[0],
                  income_age[income_age['age_group'] == 'Young']['high-perc'].values[0],]
}

income_adult_dict = {
    'age_group': 'Adult',
    'income': ['low-income', 'low-med-income', 'med-high-income', 'high-income'],
    'percentage': [income_age[income_age['age_group'] == 'Adult']['low-perc'].values[0],
                  income_age[income_age['age_group'] == 'Adult']['lowmed-perc'].values[0],
                  income_age[income_age['age_group'] == 'Adult']['medhigh-perc'].values[0],
                  income_age[income_age['age_group'] == 'Adult']['high-perc'].values[0],]
}

income_mature_dict = {
    'age_group': 'Mature',
    'income': ['low-income', 'low-med-income', 'med-high-income', 'high-income'],
    'percentage': [income_age[income_age['age_group'] == 'Mature']['low-perc'].values[0],
                  income_age[income_age['age_group'] == 'Mature']['lowmed-perc'].values[0],
                  income_age[income_age['age_group'] == 'Mature']['medhigh-perc'].values[0],
                  income_age[income_age['age_group'] == 'Mature']['high-perc'].values[0],]
}

income_senior_dict = {
    'age_group': 'Senior',
    'income': ['low-income', 'low-med-income', 'med-high-income', 'high-income'],
    'percentage': [income_age[income_age['age_group'] == 'Senior']['low-perc'].values[0],
                  income_age[income_age['age_group'] == 'Senior']['lowmed-perc'].values[0],
                  income_age[income_age['age_group'] == 'Senior']['medhigh-perc'].values[0],
                  income_age[income_age['age_group'] == 'Senior']['high-perc'].values[0],]
}

factor_df = pd.DataFrame(income_young_dict).append(pd.DataFrame(income_adult_dict), ignore_index=True)\
            .append(pd.DataFrame(income_mature_dict), ignore_index=True).append(pd.DataFrame(income_senior_dict), ignore_index=True)
sns.catplot(x='age_group', y='percentage', hue='income', data=factor_df, kind='bar', palette=sns.color_palette('hls', 4),
           height=8, aspect=1.5);
cluster = df_copy[df_copy['cluster'] == 0]

cluster['age_group'].mode().values[0], cluster['age_range'].mode().values[0], cluster['Income_group'].mode().values[0], cluster['Fidelity_group'].mode().values[0], cluster['Recency_group'].mode().values[0]
cluster = df_copy[df_copy['cluster'] == 1]

cluster['age_group'].mode().values[0], cluster['age_range'].mode().values[0], cluster['Income_group'].mode().values[0], cluster['Fidelity_group'].mode().values[0], cluster['Recency_group'].mode().values[0]
cluster = df_copy[df_copy['cluster'] == 2]

cluster['age_group'].mode().values[0], cluster['age_range'].mode().values[0], cluster['Income_group'].mode().values[0], cluster['Fidelity_group'].mode().values[0], cluster['Recency_group'].mode().values[0]
cluster = df_copy[df_copy['cluster'] == 3]

cluster['age_group'].mode().values[0], cluster['age_range'].mode().values[0], cluster['Income_group'].mode().values[0], cluster['Fidelity_group'].mode().values[0], cluster['Recency_group'].mode().values[0]
"""
Thus, the company have four kind of customers:
* Mature custmomers (45 - 65 years old) middle class, old customer with a normal frequency of purchases (Cash Cow)
* Adult customers (30 - 45 years old) low income, new customer with a high level of frequency purchases (Opportunity)
* Mature customers similar to the first one except for the annual income, that in this case is low to medium
* Mature customers with high income but low frequency of purchases. They are very experienced customers.
"""
"""
## Prepare data for modelling
"""
df_copy.head().T
"""
We can drop some columns because they were used to compute other columns and are redundant
"""
df_copy = df_copy.drop(['ID', 'Year_Birth', 'Kidhome',
                        'Teenhome', 'Dt_Customer'], axis=1)
# This will turn all the strings value into category values
df_copy['has_child'] = df_copy['has_child'].astype('category').cat.as_ordered()
for label, content in df_copy.items():
    if pd.api.types.is_string_dtype(content):
        df_copy[label] = content.astype('category').cat.as_ordered()

# Turn categorical variables into numbers
for label, content in df_copy.items():
    if not pd.api.types.is_numeric_dtype(content):
        # Turn categories into numbers and add + 1
        df_copy[label] = pd.Categorical(content).codes + 1
toScale = ['Income', 'Recency', 'MntWines', 'MntFruits', 'MntMeatProducts', 'MntFishProducts', 'MntSweetProducts', 'MntGoldProds',
          'NumDealsPurchases', 'NumWebPurchases', 'NumCatalogPurchases', 'NumStorePurchases', 'NumWebVisitsMonth',
          'fidelity_years', 'age', 'children', 'total_spend', 'WebPerc', 'StorePerc']

scaler = MinMaxScaler()
df_copy[toScale] = scaler.fit_transform(df_copy[toScale])
np.random.seed(42)

X = df_copy.drop('cluster', axis=1)
y = df_copy['cluster']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=0)

lr_model = LogisticRegression(solver='liblinear')
knn_model = KNeighborsClassifier()
rf_model = RandomForestClassifier()


models = {
    'LogisticRegression': lr_model,
    'KNeighborsClassifier': knn_model,
    'RandomForestClassifier': rf_model
}

for name, model in models.items():
    model.fit(X_train, y_train)
    print(name, ', score:', model.score(X_test, y_test))
"""
Since our Random Forest model provides the best scores so far with default parameters, we can proceed towars the cross-validation. From results the model doesn't seem to soffer the overfitting.
"""
# Cross validation on the best model
from sklearn.model_selection import cross_val_score

best_clf = models['RandomForestClassifier']

cv_acc = cross_val_score(best_clf, X, y, cv=5, scoring='accuracy')

cv_acc
# Helper function for plotting feature importance
def plot_features(columns, importances, n=20):
    df = (pd.DataFrame({'features': columns, 
                        'feature_importances': importances})
          .sort_values('feature_importances', ascending=False)
          .reset_index(drop=True))
    
    fig, ax = plt.subplots(figsize=(10, 10))
    
    ax.barh(df['features'][:n], df['feature_importances'][:n])
    ax.set_ylabel('Features')
    ax.set_xlabel('Feature importance')
    ax.invert_yaxis()
    
plot_features(X.columns, best_clf.feature_importances_)
"""
So, we can predict the customer segment with high accuracy given its purchases behaviors. Moreover, from the above chart we can see the best attributes impacting the prediction
"""
"""
## Conclusion
I propose again the main goals of this analysis
> Need to perform clustering to summarize customer segments

The customer segments defined are the following:
* Mature custmomers (45 - 65 years old) middle class, experienced customer with a normal frequency of purchases, we can consider this kind of customer as a **Cash Cow**, due to their loyalty and constant expenditures
* Adult customers (30 - 45 years old) low income, new customer with a high level of frequency purchases, we can consider them as an opportunity to grow the company income making the right campaign for them.
* Mature customers similar to the first one except for the annual income, that in this case is low to medium
* Mature customers with high income but low frequency of purchases. They are very experienced customers, they tends to save their money it could be difficult to get more from them.

> Given the customer routine spending, can we predict at which segment a customer can be associated?

It is possible to predict the customer segment given its expenditure routine. In this way, it is possible to run more specific campaigns for each kind of customers.
"""