# Parameters for Logistic Regression
logReg_parameters = {"C":[0.5,1,5,5.5,6],
                     "penalty":["l1",  "l2"],
                     "tol":[1e-2, 1e-3, 1e-4, 1e-5]
                     } 
# Parameters for AdaBoostClassifier
AdaBoost_parameters = {'n_estimators':[25,50,75], 
                       "learning_rate":[0.25, 0.5, 0.75, 1.0],
                       } 
# Parameters for DecisionTreeClassifier
DecisionTree_parameters = {"criterion":["gini","entropy"], 
                           "min_samples_split":[2,3,4],
                           "min_samples_leaf":[1,2,3]
                           } 
# Parameters for RandomForestClassifier
RandomForest_parameters = {"max_depth":[3,4,5,6,7], 
                           "criterion":["gini","entropy"],
                           "n_estimators":[20,25,30],
                           "random_state":[35,40,45]
                           } 

"""
Explanation: Identify Fraud from Enron Email
1. Introduction
In 2000, Enron was one of the largest companies in the United States. By 2002, it had collapsed into bankruptcy due to widespread corporate fraud. In the resulting Federal investigation, there was a significant amount of typically confidential information entered into public record, including tens of thousands of emails and detailed financial data for top executives. 
In this project, I will play detective, and put my machine learning skills to use by building a person of interest identifier based on financial and email data made public as a result of the Enron scandal. 
The identifier will be build based on the provided dataset that contains several financial as well as email features for 
146 employees. 18 out of these employees are labeled as persons of interest (POI).
The features and labels will help me build a classifier that will classify a random employee as a person of interest.
2. Data Wrangling
2.1 Data Overview
The dataset contains financial features (all units are in US dollars):
- salary
- deferral_payments
- total_payments
- loan_advances
- bonus
- restricted_stock_deferred
- deferred_income
- total_stock_value
- expenses
- exercised_stock_options
- other
- long_term_incentive
- restricted_stock
- director_fees
Email features (units are generally number of emails messages; notable exception is ‘email_address’, which is a text string): 
- to_messages
- email_address
- from_poi_to_this_person
- from_messages
- from_this_person_to_poi
- shared_receipt_with_poi
Label (boolean, represented as integer): 
- poi 
The dataset is not complete. The features often do not have an entry. Here is an overview of the misssing values there are for each feature.
| Feature  | Number of NaN's|
|---|---|
| total_stock_value|  18| 
| total_payments|  20| 
| restricted_stock|  34| 
| exercised_stock_options|  42| 
| expenses|  49| 
| salary|  49| 
| other|  52| 
| shared_receipt_with_poi|  57| 
| to_messages|  57| 
| from_poi_to_this_person|  57| 
| from_this_person_to_poi|  57| 
| from_messages|  57| 
| bonus|  62| 
| long_term_incentive|  78| 
| deferred_income|  95| 
| deferral_payments|  105| 
| restricted_stock_deferred|  126| 
| director_fees|  127| 
| loan_advances|  140| 
2.2 Data removal
After Exploratory Data Analysis, I found these three records that must be removed from the dataset:

LOCKHART EUGENE E (Contains only NaN values)
TOTAL (contains the sum for each features of each employee)
THE TRAVEL AGENCY IN THE PARK (does not represent an employee)

2.3 Creating new features
In this part I do some additional feature engineering by constructing three new features from the ones in the raw data. I hope that these engineered features will enhance the training of the algorithm by providing information that better differentiates the patterns in the data. I expect them also to provide additional information that is not easily apparent or not clearly captures in the raw dataset. The new features are:

email_from_poi_ratio: Ratio of emails received from poi's in relation to all received emails
email_to_poi_ratio: Ratio of emails sent to poi's in relation to all sent emails
total_wealth: Sum of the salary, bonus, exercised_stock_options, total_stock_value

2.4 Features Scaling
Feature scaling is a method used to standardize the range of features in the data. Since the range of values in the raw dataset my vary widely, some machine learning algorithms will work badly without feature scalling. The majority of classifiers calculate the distance of two points by the Euclidean distance. In the case that one of the features has a broad range of values the distance will be dominated by this feature. Because of this fact the range of all features should be scalled so that every feature in the dataset contribute proportionately to the final distance. In this project I wil use Scikit-learn's StandardScaler() to normalize the features. However not every classifier will use feature scaling.
3. Algorithm Tuning
In this project I try out several algorithms to determine which classifier provide me in the end with the best result. The used algorithms are:

Gaussian Naive Bayes
Logistic Regression
Decision Tree
Random Forest
Ada Boost 

Before the algorithms are used in practice I perform an algorithm tuning. The tuning involves the determination of the best input parameters to increase the algorithm´s performance.
Algorithms have numerous input parameters that assist the decision making process during the training of an algorithm. The input parameters determine for instance which heuristics are used or the probabilities of certain events occurring.
The performance of an algorithm is greatly affected by the parameter setting. However these setting are not easy to determine. To find out the best combination of parameter settings that provide the best performance I use Scikit-learn's GridSearchCV(). This function takes a classifier and a dictionary of input parameters and returns the parameters which results in the best performance of the algorithm.
Here are the dictionarys of input parameters that were used by GridSearchCV() to determine the best parameter settings for each classifier:
End of explanation
"""


Pipeline(steps=[("Scaler", None),
                 ("K_best", None),
                 ("PCA", PCA(n_components=12)),
                 ("clf", LogisticRegression(C=6,
                                            penalty=l1,
                                            tol=0.001)
                                            )
               ]
        )


"""
Explanation: The execution of GridSearchCV() resulted in the following best parameter settings:
LogisticRegression

C=6
penalty=l1
tol=0.001

AdaBoostClassifier

n_estimators=50,
learning_rate=0.25

RandomForestClassifier

n_estimators=20
criterion='entropy'
max_depth=3,
random_state=40

DecisionTreeClassifier

min_samples_split=3
criterion='entropy'
min_samples_leaf=2

4. Feature Selection and Further Enhancement
To increase the performance even more I use the Pipeline function to perform a number of operation such as StandardScaler(), SelectKBest(k) and PCA(n_components). StandardScaler() was already mentioned. Scikit-learn's SelectKBest(k) is used for feature selection. The function selects features according to the k highest scores. Here is an overview over the features with the corresponding scores.
| Feature  | Score  |
|---|---|
|  total_wealth | 28.26  |
|  exercised_stock_options | 24.81  |
|  total_stock_value | 24.18  |
|  bonus |  20.79  |
|  salary | 18.28  |
|  deferred_income |  11.45 |
| long_term_incentive  |  9.92 |
| restricted_stock  |  9.21 |
| total_payments  | 8.77  |
|  shared_receipt_with_poi | 8.58  |
|  loan_advances | 7.18  |
|  expenses | 7.18  |
| from_poi_to_this_person  | 5.24  |
| email_from_poi_ratio | 5.12|
|  other | 4.18  |
|  email_to_poi_ratio | 4.09  |
|  from_this_person_to_poi | 2.38  |
|  director_fees | 2.12  |
|  to_messages |  1.64 |
|  deferral_payments | 0.22  |
|  from_messages | 0.16  |
|  restricted_stock_deferred | 0.06  |
The new feature tota_wealth has the highest score, in opposite email_from_poi_ratio and email_to_poi_ratio occupy 14th and 16th place out of 22.
Scikit-learn's PCA() is used to transform the features into Principal Components, which are used as new features. This step may increase the performance of an algorithm. The final dimension of Principal Componenets is determined by using n_components as input parameter for PCA().
While the set of best input parameters that were determined by GridSearchCV() for each algorithm stay the same I use different combinations of StandardScaler(), SelectKBest(), PCA() as well as combinations of k for SelectKBest() and n_components for PCA(). In the end I kept the combination of the operations and their input parameters which resulted in the best performance.
Since I did the combinations manually I could not try out all the possible combinations there are. I am aware of the fact that some combinations I did not try out may result in even better performances.
Here I provide an overview over the final manually chosen operations used in the Pipepline as well as the optimal input parameters for each algorithm determined by GridSearchCV() during the tuning. The expressions "Scaler", "K_best", "PCA" and "clf" correspond to StandardScaler(), SelectKBest(), PCA() and the used algorithm.
'None' means that the corresponding function was not used in the pipeline. In the case of "K_best" it means that all features were selected.
LogisticRegression
End of explanation
"""


Pipeline(steps=[("Scaler", StandardScaler()),
                ("K_best", None),
                ("PCA", None),
                ("clf", AdaBoostClassifier(n_estimators=50,
                                           learning_rate=0.25))
               ]
        )


"""
Explanation: AdaBoostClassifier
End of explanation
"""


Pipeline(steps=[("Scaler", None),
                 ("K_best", SelectKBest(k=10)),
                 ("PCA", PCA(n_components=5)),
                 ("clf", RandomForestClassifier(n_estimators=20,
                                                criterion='entropy',
                                                max_depth=3,
                                                random_state=40))
               ]
        )

"""
Explanation: RandomForestClassifier
End of explanation
"""


Pipeline(steps=[("Scaler", None),
                 ("K_best", SelectKBest(k=13)),
                 ("PCA", PCA(n_components=10)),
                 ("clf", GaussianNB())
               ]
        )


"""
Explanation: Gaussian Naive Bayes
End of explanation
"""


Pipeline(steps=[("Scaler", StandardScaler()),
                 ("K_best", SelectKBest(k=4)),
                 ("PCA", None),
                 ("clf", DecisionTreeClassifier(min_samples_split=3,
                                                criterion='entropy',
                                                min_samples_leaf=2))
               ]
        )


"""
Explanation: DecisionTreeClassifier
End of explanation
"""


