# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # Employee Retention
#
# ## Goal
#
# Employee turnover is a very costly problem for companies. The cost of replacing an employee if often larger than 100K USD, taking into account the time spent to interview and find a replacement, placement fees, sign-on bonuses and the loss of productivity for several months.
#
# t is only natural then that data science has started being applied to this area. Understanding why and when employees are most likely to leave can lead to actions to improve employee retention as well as planning new hiring in advance. This application of DS is sometimes called people analytics or people data science (if you see a job title: people data scientist, this is your job).
#
# In this challenge, you have a data set with info about the employees and have to predict when employees are going to quit by understanding the main drivers of employee churn.
#
# ## Challenge Description
#
# We got employee data from a few companies. We have data about all employees who joined from 2011/01/24 to 2015/12/13. For each employee, we also know if they are still at the company as of 2015/12/13 or they have quit. Beside that, we have general info about the employee, such as avg salary during her tenure, dept, and yrs of experience.
#
# As said above, the goal is to predict employee retention and understand its main drivers
#
# ## Hints
#
# What are the main factors that drive employee churn? Do they make sense? Explain your findings.
#
# What might you be able to do for the company to address employee Churn, what would be follow-up actions?
#
# If you could add to this data set just one variable that could help explain employee churn, what would that be?
#
# Your output should be in the form a a jupyter notebook and pdf output of a jupyter notebook in which you specify your results and how you got them.
#
# ## Data
#
# The table is: "employee_retention" - comprehensive information about employees
#
# Columns:
# - employee_id : id of the employee. Unique by employee per company 
# - company_id : company id. 
# - dept : employee dept 
# - seniority : number of yrs of work experience when hired 
# - salary: avg yearly salary of the employee during her tenure within the company 
# - join_date: when the employee joined the company, it can only be between 2011/01/24 and 2015/12/13 
# - quit_date: when the employee left her job (if she is still employed as of 2015/12/13, this field is NA)
#
#
# ## Additional variable 
#
# - Internal mobility: did the patient get a promotion?
# - satisfaction: reported at annual / bi-annual evaluation

# # Loading and preparing the data

# +
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

sns.set(style="ticks", color_codes=True)
# -

# Load the data and first look at it
df = pd.read_csv('employee_retention_data.csv')
display(df.head(10))
display(df.describe())

# Generating a new binary outcome variable, 
df['quit'] = 0
df.loc[~df['quit_date'].isnull(), 'quit'] = 1
# Filling the NaNs in quit_date with the end time of the study
df['quit_date'].fillna('2015-12-13', inplace=True)
# Adding a new retention variable
df['retention'] = (pd.to_datetime(df['quit_date']) - pd.to_datetime(df['join_date'])).dt.days
df['start_month'] = pd.to_datetime(df['join_date']).dt.month
# Dropping seniority of 99 years
df = df[df.seniority < 98]
df.head(10)

# # First look at the data
#
# There are no very strong notable association in the data. 
# Seniority seems to be correlated with salary. 
# Interestingly, there could a trend between the month of hire and the duration of employment.
# Looking at the retention more carefully, I can see that there are prefered time when people quit: around year 1, then around year 2, ... This seems to be true across companies

# Visualization of the relationship between variables (scatter)
g = sns.pairplot(df)

# Visualization of the relationship between variables (correlation)
plt.matshow(df.corr());
plt.colorbar();

# Looking at retention overall and by company
df['retention'].hist(bins=40);
df.hist(column='retention', by='company_id', figsize=(10,10), sharex=True);

# # Looking at the company level
#
# There is a wide range in terms of nb of employees between those company (16 to 8486 employees).
# There doesn't seems to be any strong association between variables at the company level, except possibly between seniority and retention.

# Building some company level averages
df_company = pd.DataFrame({
    'retention': df.groupby('company_id')['retention'].mean(),
    'employee_nb': df.groupby('company_id')['employee_id'].count(),
    'seniority': df.groupby('company_id')['seniority'].mean(),
    'salary': df.groupby('company_id')['salary'].mean(),
})
df_company

# Visualization of the relationship between variables
g = sns.pairplot(df_company)
plt.matshow(df_company.corr());
plt.colorbar();

# # Looking at the dept level
#
# Some departments have higher turnaround (e.g., sales).

# Building some dpt level averages
df_dept = pd.DataFrame({
    'count': df.groupby('dept')['retention'].count(),
    'retention': df.groupby('dept')['retention'].mean(),
    'seniority': df.groupby('dept')['seniority'].mean(),
    'salary': df.groupby('dept')['salary'].mean(),
})
df_dept['retention'].sort_values().plot.barh()
g = sns.pairplot(df_dept)
plt.matshow(df_dept.corr());
plt.colorbar();
df_dept

# # Building a regression model to predict employees leaving the company
#
# Both the linear regression and the random forest achieve low accuracy on the training data, which limits the validity of the interpretation. However, a few interesting points:
# - for both methods, it seems that salary, seniority and start month play an important role
# - the lin reg that employees in company 11 and 12 quit earlier and that sales leave earlier

# +
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import r2_score
from sklearn.preprocessing import MinMaxScaler, PolynomialFeatures

df_reg = df[df.quit == 1]
y = df_reg['retention'].values
X = pd.get_dummies(df_reg[['company_id', 'dept', 'seniority', 'salary', 'start_month']], 
                   columns=['company_id', 'dept'], prefix='d')
cols = X.columns
X = X.values
X = MinMaxScaler().fit_transform(X)

reg = LinearRegression().fit(X, y)
print(reg.score(X, y))

from sklearn.model_selection import cross_val_score,cross_validate
scores = cross_validate(reg, X, y, cv=5)
print(scores)

pd.DataFrame({
    'name': cols,
    'importance': reg.coef_
}).sort_values(by='importance')


# +
reg = RandomForestRegressor().fit(X, y)
print(reg.score(X, y))

from sklearn.model_selection import cross_val_score,cross_validate
scores = cross_validate(reg, X, y, cv=5)
print(scores)

pd.DataFrame({
    'name': cols,
    'importance': reg.feature_importances_
}).sort_values(by='importance')
# -

# # Training a classifier to find employees that quit
#
# The RF gets a better accuracy (0.81), and uses mostly retention, salary, start month and seniority to make a decision.
# The logistic regression has consistent results except for seniority that has a low coefficient.
# There might be an imbalance in the distribution of dept, which could affect how those are influencing the classifier, but the current results indicate that employee get unhappy faster in marketing, customer service and sales compared to the other dept.

# Selecting the features and creating the dummy variables for the categorical ones
df2 = pd.get_dummies(df, columns=['dept', 'company_id'], prefix='d')
df2 = df2.drop(['employee_id', 'join_date', 'quit_date'], axis=1)
plt.matshow(df2.corr())
df2.head(10)

# +
# Creating the train/test sets and balancing the classes
from sklearn.model_selection import train_test_split

Xb = df2.drop('quit', axis=1).values
Xb = MinMaxScaler().fit_transform(Xb)
col_names = df2.drop('quit', axis=1).columns
yb = df2['quit'].values
X_train, X_test, y_train, y_test = train_test_split(Xb, yb, test_size=0.3, random_state=42)#, stratify=yb)

sel = y_train == 1
X0 = X_train[sel]
y0 = y_train[sel]
sub = np.random.choice(range(len(y0)), size=sum(~sel), replace=False)
X_train = np.concatenate([X_train[~sel,:], X0[sub,:]])
y_train = np.concatenate([y_train[~sel], y0[sub]])

plt.hist(y_train);
# -

# running a random classifier
from sklearn.dummy import DummyClassifier
dummy = DummyClassifier(strategy='most_frequent')
dummy.fit(X_train, y_train)
print(dummy.score(X_test, y_test))

# +
# Running a logistic regression classifier
from sklearn.linear_model import LogisticRegressionCV

clf = LogisticRegressionCV(cv=5, max_iter=300, solver='liblinear')
clf.fit(X_train, y_train)
print(clf.score(X_test, y_test))

df3 = pd.DataFrame({'importance': clf.coef_[0]}, index=col_names)
df3.sort_values(by='importance').plot.barh(legend=False, figsize=(6,6), title='Logistic regression coefficients');

# +
# Random forest
from sklearn.ensemble import RandomForestClassifier

clf = RandomForestClassifier(n_estimators=200)
clf.fit(X_train, y_train)
print(clf.score(X_test, y_test))

df3 = pd.DataFrame({'importance': clf.feature_importances_}, index=col_names)
df3.sort_values(by='importance').plot.barh(legend=False, figsize=(6,6), title='RF Feautre importance');
# -

# # Overall conclusion
#
# Based on my investigation of the regression and the classification, here are some possible conclusions:
# - a higher salary seems to keep employees happy
# - senior employees might leave the company earlier
# - the month of hire seems to have an influence 
# - retention is predictive of who is quitting, meaning that the longer employees stay the more likely to leave, so it is wise to plan for it
# - client facing dept seem to have more difficulties to keep employees happy.

StarcoderdataJupyterScriptsDedupFiltered