import os
import csv
import codecs
import string


TRAINSET_PATH = '../data/train/'
TESTSET_PATH = '../data/test/'
LINGSPAM_TRAIN_CSV_PATH = TRAINSET_PATH + 'lingspam_train.csv'
LINGSPAM_TEST_CSV_PATH = TESTSET_PATH + 'lingspam_test.csv'


def generate_trainset(input_dir, output_path):
    l = []

    for root, dirs, files in os.walk(input_dir):
        path = root.split(os.sep)
        part_name = os.path.basename(root)
        for file in files:
            if not file.endswith('.txt'):
                continue

            d = {}
            file_name = file.replace('.txt', '')
            file_path = os.path.join(root, file)
            
            with codecs.open(file_path, mode='r', encoding='utf8', errors='ignore') as f:
                line_counter = 0
                for line in f.readlines():
                    line = line.strip()
                    if line_counter == 0:   # subject
                        subject = line.replace('Subject:', '').strip()
                    if line_counter == 2:
                        email = line
                    line_counter += 1
            d['email_subject'] = subject
            d['email_body'] = email
            d['part_name'] = part_name
            d['file_name'] = file_name
            d['is_spam'] = 1 if file_name.startswith('spmsg') else 0
            l.append(d)
    
    with codecs.open(output_path, mode='w', encoding='utf8', errors='ignore') as out_file:
        writer = csv.DictWriter(out_file, l[0].keys())
        writer.writeheader()
        for row in l:
            writer.writerow(row)


# generate_trainset(TRAINSET_PATH, LINGSPAM_TRAIN_CSV_PATH)
# generate_trainset(TESTSET_PATH, LINGSPAM_TEST_CSV_PATH)

"""
Explanation: Introduction
In this project, I will design and implement several e-mail spam filters using Naive Bayes and SVM based classification on the ling-spam dataset. 
Here is a list of email spam filters:
* Naive Bayes Classifier
  * Bernoulli NB classiﬁer with    binary features
  * Multinomial NB with binary features
  * Multinomial NB with term frequency (TF) features 
* SVM based Classifier
* Adversarial Classification
Dataset
The ling-spam corpus contains e-mails from the Linguist mailing list categorized as either legitimate or spam emails. The corpus is divided into four sub-folders that contain the same emails that are pre-processed with/without lemmatization and with/without stop-word removal. The e-mails in each sub-folder partitioned into 10 folds.
In this project, we will use the ﬁrst 9 folds from the ling-spam corpus as training data, and the 10th fold as test data. Also, we will use the corpus with both lemmatization and stop-word enabled (under the lemm_stop folder).
Download the ling-spam dataset from: http://www.aueb.gr/users/ion/data/lingspam_public.tar.gz
Before we start the experiments, I need to do some preprocessing on the corpus.
Data Preprocessing
The purpose of preprocessing is to organize corpus in a clean and unified format and make the following experiments easier. In this step, I will generate 2 csv files lingspam_train.csv and lingspam_test.csv based on lingspam corpus (lemm_stop folder). The first 9 parts make up the training set and the 10th part is the testing set.
In the csv file, it includes all the information we need, such as email subject, email body, is email spam or legitimate.
The python script I used to do data preprocessing:
End of explanation
"""


import csv
import pandas as pd

lingspam_train_path = './lingspam_train.csv'
lingspam_test_path = './lingspam_test.csv'
dtype={
    'email_subject': str,
    'email_body': str,
    'part_name': str,
    'file_name':str,
    'is_spam': int
    }

lingspam_train_df = pd.read_csv(lingspam_train_path, dtype=dtype)
lingspam_test_df = pd.read_csv(lingspam_test_path, dtype=dtype)

print("Dataset column names:")
for col in lingspam_train_df.columns:
  print(col)

print('\nlingspam trainset:')
print(lingspam_train_df)
print('\nlingspam testset:')
print(lingspam_test_df)

trainset_size = lingspam_train_df.shape[0]
testset_size = lingspam_test_df.shape[0]
print("\nTrainset size: " + str(trainset_size))
print("Testset size: " + str(testset_size))

y_train = lingspam_train_df['is_spam'].to_numpy()
y_test = lingspam_test_df['is_spam'].to_numpy()

"""
Explanation: We can't really run the above code in this notebook since it's hard to upload the dataset folders to colab. So I run this script offline and generated 2 expected csv files. I put the link here
* lingspam_train.csv
* lingspan_test.csv
Please download these 2 files and upload them to this colab notebook, click the upload button on the left Files tab.
Now we have cleaned corpus, let's take a look at them using Pandas.
End of explanation
"""


import string
from collections import Counter

words = []
dictionary = {}

for index, row in lingspam_train_df.iterrows():
  email = row['email_body'].split(' ')
  email = [word for word in email if word not in string.punctuation]
  email = [word for word in email if len(word) > 1]
  email = [word for word in email if word.isalpha() == True]
  words += email

dictionary = Counter(words)

unique_num = len(dictionary)
total_num = sum(dictionary.values())

print("The number of unique words in lingspam trainset: " + str(unique_num))
print("The total times they appeared: " + str(total_num))

print("The 20 most common words in trainset:")
print(*dictionary.most_common(20), sep='\n')

print('\nThe length of current dictionary: ' + str(len(dictionary)))

"""
Explanation: From the shape of csv dataframes, the trainset has 2602 samples, the testset has 291 test cases.
Next, I will make an dictionary from the trainset lingspam_train.csv to count word frequency. With this dictonary, we can easily extract binary features or term frequency features from the trainset and testset.
For the eamil body, I remove all the punctuations since I believe they do not affect whether the mail is spam. Also, words which length less than 2 are removed since I don't think single letters are real words. Besides, I only keep those words which all the characters are alphabet letters (a-z).
For the feature selection step, we don't want the dimension of our features is too large since it will bring too much computational cost. There are about 55k unique words in lingspam training dataset, we will do some feature selection based on information gain.
End of explanation
"""


import math

total_legit_emails = 0.0
total_spam_emails = 0.0

for index, row in lingspam_train_df.iterrows():
  is_spam = row['is_spam']
  if is_spam == 1:
    total_spam_emails += 1
  else:
    total_legit_emails += 1

print("Total legit email number = {}".format(total_legit_emails))
print("Total spam email number = {}".format(total_spam_emails))

p = total_legit_emails / (total_spam_emails + total_legit_emails)
print("p = {}".format(p))

h_c = -1 * p * math.log(p, 2) - (1 - p) * math.log(1 - p, 2)
print("H(C) = {}".format(h_c))


def count_legit_emails_with_word(word):
  num_legit_emails_with_word = 0
  for index, row in lingspam_train_df.iterrows():
    if row['is_spam'] == 0 and word in row['email_body'].split(' '):
      num_legit_emails_with_word += 1
  return num_legit_emails_with_word
      
def count_spam_emails_with_word(word):
  num_spam_emails_with_word = 0
  for index, row in lingspam_train_df.iterrows():
    if row['is_spam'] == 1 and word in row['email_body'].split(' '):
      num_spam_emails_with_word += 1
  return num_spam_emails_with_word

def h_legit_word_not_present(word):
  num_legit_emails_with_word = count_legit_emails_with_word(word)
  num_spam_emails_with_word = count_spam_emails_with_word(word)
  return (total_legit_emails - num_legit_emails_with_word) / (total_spam_emails + total_legit_emails) * math.log((total_legit_emails - num_legit_emails_with_word) / (total_spam_emails - num_spam_emails_with_word + total_legit_emails - num_legit_emails_with_word), 2)

def h_spam_word_not_present(word):
  num_legit_emails_with_word = count_legit_emails_with_word(word)
  num_spam_emails_with_word = count_spam_emails_with_word(word)
  return (total_spam_emails - num_spam_emails_with_word) / (total_spam_emails + total_legit_emails) * math.log((total_spam_emails - num_spam_emails_with_word) / (total_spam_emails - num_spam_emails_with_word + total_legit_emails - num_legit_emails_with_word), 2)

def h_legit_word_is_present(word):
  num_legit_emails_with_word = count_legit_emails_with_word(word)
  num_spam_emails_with_word = count_spam_emails_with_word(word)
  return num_legit_emails_with_word / (total_spam_emails + total_legit_emails) * math.log(num_legit_emails_with_word / (num_spam_emails_with_word + num_legit_emails_with_word), 2)

def h_spam_word_is_present(word):
  num_legit_emails_with_word = count_legit_emails_with_word(word)
  num_spam_emails_with_word = count_spam_emails_with_word(word)
  return num_spam_emails_with_word / (total_spam_emails + total_legit_emails) * math.log(num_spam_emails_with_word / (num_spam_emails_with_word + num_legit_emails_with_word), 2)

def info_gain(word):
  h_c_x = -1 * (h_legit_word_not_present(word) + h_spam_word_not_present(word) + h_legit_word_is_present(word) + h_spam_word_is_present(word))
  ig = h_c - h_c_x
  return ig

word = "language"
print("word: {}, info_gain: {}".format(word, info_gain(word)))

"""
Explanation: Experiments
Feature Selection
In the next 3 Naive Bayes classifiers, we will use 2 ways to encode features:
* Binary/Boolean Features
  * $x_i=1$ if term $i$ appears in a doc; $0$ otherwise
  * Each doc has a $M$ dimension boolean features, $x = (x_1, x_2, ..., x_M)$ 
* Term Frequencies (TF) Features
  * $x_i$: number of times term $i$ appears in a doc
  * Each doc is represented by $x = (x_1, x_2, ... x_M)$, a vector of term frequencies.
Information gain (IG) will be used as metric to perform feature selection (rank features). Here, IG metric only accounts for the occurrence of (and not frequency with which terms appear) in the dataset. Also, I use Laplacian Smoothing to ensure that there is no event with zero probability.
Next let's calculate IG for words in the training dataset.
End of explanation
"""


ig_filepath = './ig.csv'

dtype={
    'word': str,
    'freq': int,
    'ig': float,
    }

ig_df = pd.read_csv(ig_filepath, dtype=dtype)


print('\nInformation Gain dictionary:')
print(ig_df)

"""
Explanation: Since the computation time for all words in dataset (45k words!) is quite long, I did the computation offline and save the dictionary with information gain values to a csv file. You can download this file from this link. After you download it, upload it to this colab notebook and we will need this in the following experiments.
End of explanation
"""


sorted_ig_df = ig_df.sort_values(by=['ig'], ascending=False)

print('\nDictionary sorted by IG on desending order:')
print(sorted_ig_df)

top_10_features = sorted_ig_df.head(10)
top_100_features = sorted_ig_df.head(100)
top_1000_features = sorted_ig_df.head(1000)

print("\nTop-10 features:")
print(top_10_features)
print("\nTop-100 features:")
print(top_100_features)
print("\nTop-1000 features:")
print(top_1000_features)

top_10_features_list = [row['word'] for index, row in top_10_features.iterrows()]
top_100_features_list = [row['word'] for index, row in top_100_features.iterrows()]
top_1000_features_list = [row['word'] for index, row in top_1000_features.iterrows()]

print("\nTop-10 words:")
print(top_10_features_list)
print("\nTop-100 words:")
print(top_100_features_list)
print("\nTop-1000 words:")
print(top_1000_features_list)

"""
Explanation: Now we get Information Gain values for each word in the dataset, we are able to perform feature selection by ranking word on descending order based on IG. From the training data, select the top-N features ($N = {10, 100, 1000}$) on terms of the highest Information Gain (IG) scores.
End of explanation
"""


import numpy as np
from pandas import DataFrame

def extract_binary_features(df: DataFrame, N: int):

  if N == 10:
    top_n_features_list = top_10_features_list
  elif N == 100:
    top_n_features_list = top_100_features_list
  elif N == 1000:
    top_n_features_list = top_1000_features_list
  else:
    print('Please choose a right value for N (10, 100 or 1000)!')
    return

  assert N == len(top_n_features_list), "The length of top_n_features_list should be equal with N!"

  features_matrix = np.zeros((df.shape[0], N))
  for email_idx, row in df.iterrows():
    email_body = row['email_body'].split(' ')
    for word_idx in range(len(top_n_features_list)):
      word = top_n_features_list[word_idx]
      if word in email_body:
        features_matrix[email_idx, word_idx] = 1
  return features_matrix


"""
Explanation: Now we have 3 feature dictonaries, top-10, top-100 and top-1000. Next we can extract binary feature matrixes or term frequency feature matrixes from datasets based on them. Let's start!
Bernoulli NB classiﬁer with Binary Features
Bernoulli Naive Bayes classifier for multivariate Bernoulli models. This classifier is suitable for discrete data. BernoulliNB is designed for binary/boolean features. Here, bianry feature $x_i = 1$ if word $i$ appears in a document/email, 0 otherwise.
Next I will extract binary feature matrix from trainset/testset based on the top-N feature dictonaries calculated before.
End of explanation
"""


from sklearn.naive_bayes import BernoulliNB
from sklearn.metrics import precision_score, recall_score

print("Bernoulli NB classiﬁer with Binary Features")


n = 10
x_train_10 = extract_binary_features(lingspam_train_df, n)
x_test_10 = extract_binary_features(lingspam_test_df, n)
bernoulii_nb_binary_10 = BernoulliNB()
bernoulii_nb_binary_10.fit(x_train_10, y_train)

y_pred = bernoulii_nb_binary_10.predict(x_test_10)
bernoulii_nb_binary_10_precision = precision_score(y_test, y_pred)
bernoulii_nb_binary_10_recall = recall_score(y_test, y_pred)

print("N = {}".format(n))
print("precision: {}".format(bernoulii_nb_binary_10_precision))
print("recall: {}\n".format(bernoulii_nb_binary_10_recall))


n = 100
x_train_100 = extract_binary_features(lingspam_train_df, n)
x_test_100 = extract_binary_features(lingspam_test_df, n)
bernoulii_nb_binary_100 = BernoulliNB()
bernoulii_nb_binary_100.fit(x_train_100, y_train)

y_pred = bernoulii_nb_binary_100.predict(x_test_100)
bernoulii_nb_binary_100_precision = precision_score(y_test, y_pred)
bernoulii_nb_binary_100_recall = recall_score(y_test, y_pred)

print("N = {}".format(n))
print("precision: {}".format(bernoulii_nb_binary_100_precision))
print("recall: {}\n".format(bernoulii_nb_binary_100_recall))


n = 1000
x_train_1000 = extract_binary_features(lingspam_train_df, n)
x_test_1000 = extract_binary_features(lingspam_test_df, n)
bernoulii_nb_binary_1000 = BernoulliNB()
bernoulii_nb_binary_1000.fit(x_train_1000, y_train)

y_pred = bernoulii_nb_binary_1000.predict(x_test_1000)
bernoulii_nb_binary_1000_precision = precision_score(y_test, y_pred)
bernoulii_nb_binary_1000_recall = recall_score(y_test, y_pred)

print("N = {}".format(n))
print("precision: {}".format(bernoulii_nb_binary_1000_precision))
print("recall: {}\n".format(bernoulii_nb_binary_1000_recall))

"""
Explanation: Train a Bernoulli Naive Bayes classifier with binary features and get the precision score, recall score on the testing dataset.
End of explanation
"""


from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import precision_score, recall_score

print("Multinomial NB classiﬁer with Binary Features")


n = 10
multinomial_nb_binary_10 = MultinomialNB()
multinomial_nb_binary_10.fit(x_train_10, y_train)

y_pred = multinomial_nb_binary_10.predict(x_test_10)
multinomial_nb_binary_10_precision = precision_score(y_test, y_pred)
multinomial_nb_binary_10_recall = recall_score(y_test, y_pred)

print("N = {}".format(n))
print("precision: {}".format(multinomial_nb_binary_10_precision))
print("recall: {}\n".format(multinomial_nb_binary_10_recall))


n = 100
multinomial_nb_binary_100 = MultinomialNB()
multinomial_nb_binary_100.fit(x_train_100, y_train)

y_pred = multinomial_nb_binary_100.predict(x_test_100)
multinomial_nb_binary_100_precision = precision_score(y_test, y_pred)
multinomial_nb_binary_100_recall = recall_score(y_test, y_pred)

print("N = {}".format(n))
print("precision: {}".format(multinomial_nb_binary_100_precision))
print("recall: {}\n".format(multinomial_nb_binary_100_recall))


n = 1000
multinomial_nb_binary_1000 = MultinomialNB()
multinomial_nb_binary_1000.fit(x_train_1000, y_train)

y_pred = multinomial_nb_binary_1000.predict(x_test_1000)
multinomial_nb_binary_1000_precision = precision_score(y_test, y_pred)
multinomial_nb_binary_1000_recall = recall_score(y_test, y_pred)

print("N = {}".format(n))
print("precision: {}".format(multinomial_nb_binary_1000_precision))
print("recall: {}\n".format(multinomial_nb_binary_1000_recall))

"""
Explanation: Multinomial NB with Binary Features
Multinomial NB classifier is for multinomial models. It is suitable for classification with discrete features (e.g., word counts for text classification). The multinomial distribution normally requires integer feature counts. We will use binary features in this multinomial NB classifier. The process of extrating binary features is the same with prior classifer.
Next train a multinomial NB classifier with binary features and evaluate the model.
End of explanation
"""


import numpy as np
from pandas import DataFrame

def extract_tf_features(df: DataFrame, N: int):

  if N == 10:
    top_n_features_list = top_10_features_list
  elif N == 100:
    top_n_features_list = top_100_features_list
  elif N == 1000:
    top_n_features_list = top_1000_features_list
  else:
    print('Please choose a right value for N (10, 100 or 1000)!')
    return

  assert N == len(top_n_features_list), "The length of top_n_features_list should be equal with N!"

  features_matrix = np.zeros((df.shape[0], N))
  for email_idx, row in df.iterrows():
    email_body = row['email_body'].split(' ')
    for word_idx in range(len(top_n_features_list)):
      word = top_n_features_list[word_idx]
      if word in email_body:
        features_matrix[email_idx, word_idx] = email_body.count(word)
  return features_matrix

"""
Explanation: Multinomial NB with Term Frequency (TF) Features
In this multinomial NB classifer, we will use term frequency (TF) features. In other words, we will use word occurence count. TF feature $x_i$ represents the times term $i$ appears in a document/email.
Next, I will extract TF features from datasets.
End of explanation
"""


from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import precision_score, recall_score

print("Multinomial NB classiﬁer with TF Features")


n = 10
x_train_10 = extract_tf_features(lingspam_train_df, n)
x_test_10 = extract_tf_features(lingspam_test_df, n)
multinomial_nb_tf_10 = MultinomialNB()
multinomial_nb_tf_10.fit(x_train_10, y_train)

y_pred = multinomial_nb_tf_10.predict(x_test_10)
multinomial_nb_tf_10_precision = precision_score(y_test, y_pred)
multinomial_nb_tf_10_recall = recall_score(y_test, y_pred)

print("N = {}".format(n))
print("precision: {}".format(multinomial_nb_tf_10_precision))
print("recall: {}\n".format(multinomial_nb_tf_10_recall))


n = 100
x_train_100 = extract_tf_features(lingspam_train_df, n)
x_test_100 = extract_tf_features(lingspam_test_df, n)
multinomial_nb_tf_100 = MultinomialNB()
multinomial_nb_tf_100.fit(x_train_100, y_train)

y_pred = multinomial_nb_tf_100.predict(x_test_100)
multinomial_nb_tf_100_precision = precision_score(y_test, y_pred)
multinomial_nb_tf_100_recall = recall_score(y_test, y_pred)

print("N = {}".format(n))
print("precision: {}".format(multinomial_nb_tf_100_precision))
print("recall: {}\n".format(multinomial_nb_tf_100_recall))


n = 1000
x_train_1000 = extract_tf_features(lingspam_train_df, n)
x_test_1000 = extract_tf_features(lingspam_test_df, n)
multinomial_nb_tf_1000 = MultinomialNB()
multinomial_nb_tf_1000.fit(x_train_1000, y_train)

y_pred = multinomial_nb_tf_1000.predict(x_test_1000)
multinomial_nb_tf_1000_precision = precision_score(y_test, y_pred)
multinomial_nb_tf_1000_recall = recall_score(y_test, y_pred)

print("N = {}".format(n))
print("precision: {}".format(multinomial_nb_tf_1000_precision))
print("recall: {}\n".format(multinomial_nb_tf_1000_recall))

"""
Explanation: Train a Multinomial Naive Bayes classifier with term frequency features and get the precision score, recall score on the testing dataset.
End of explanation
"""


from sklearn import svm
from sklearn.metrics import precision_score, recall_score
from sklearn.model_selection import cross_validate


print("SVM classiﬁer with TF Features")

n = 1000
kernel = ['linear', 'poly', 'rbf', 'sigmoid']
c = [1, 2]

x_train = extract_tf_features(lingspam_train_df, n)
x_test = extract_tf_features(lingspam_test_df, n)

# train svm models
for k in kernel:
  for reg in c:
    svm_model = svm.SVC(kernel=k, C=reg).fit(x_train, y_train)

    y_pred = svm_model.predict(x_test)
    svm_precision = precision_score(y_test, y_pred)
    svm_recall = recall_score(y_test, y_pred)

    print("Kernel = {}, C = {}".format(k, reg))
    print("precision: {}".format(svm_precision))
    print("recall: {}\n".format(svm_recall))

    print("Perform 5-fold Cross Validation on training set:")
    scores = cross_validate(
        svm_model, 
        x_train, 
        y_train, 
        cv=5,
        scoring=('precision', 'recall')
        )
    print(scores)

    cv_best_precision = max(scores['test_precision'])
    cv_best_reall = max(scores['test_recall'])
    print("best cross validation precision: {}".format(cv_best_precision))
    print("best cross validation recall: {}\n".format(cv_best_reall))

"""
Explanation: SVM based Spam Filter
In this section, I will design and implement a SVM based spam filter. The top-1000 features will be selected on terms of the highest Information Gain (IG) scores. I will use TF feature selection method as the function extract_tf_features() did. Also, cross validation will be employed to select the best model.
Let's train SVM models with different hyperparameters and compare their performance.
End of explanation
"""


from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score

n = 10
x_train = extract_tf_features(lingspam_train_df, n)
x_test = extract_binary_features(lingspam_test_df, n)
multinomial_nb_binary_baseline = MultinomialNB()
multinomial_nb_binary_baseline.fit(x_train, y_train)

y_pred = multinomial_nb_binary_baseline.predict(x_test)

acc_score = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
conf_mat = confusion_matrix(y_test, y_pred, labels = [0, 1])

tn = conf_mat[0][0]
fn = conf_mat[1][0]
tp = conf_mat[1][1]
fp = conf_mat[0][1]
# False positive rate
fpr = fp / (fp + tn)
# False negative rate
before_fnr = fn / (tp + fn)

print("baseline nb classifier accuracy rate: {}, precision: {}, recall: {}".format(acc_score, precision, recall))
print("confusion matrix: \n{}".format(conf_mat))
print("tn: {}, fp: {}, fn: {}, tp: {}".format(tn, fp, fn, tp))
print("fpr: {}, fnr: {}".format(fpr, before_fnr))

"""
Explanation: Adversarial Classification based Spam Filter
Adversarial Classification is an approach to update NB based e-mail spam ﬁlters in response to attacks that try to evade a basic NB ﬁlter. I will implement the techniques presented in this paper. More specifically, the following assumptions are made:
* The baseline NB classiﬁer (Multinomial NB with binary features) uses the top-10 terms identiﬁed using the IG metric and using Boolean features.
* The adversary uses the ADD-WORDS strategy. Cost of adding a word is 1. The attacker seeks to ﬁnd the minimum cost solution such that each spam email in the test set gets classiﬁed as legitimate by the baseline NB classiﬁer.
* Adversary cannot modify negative instances, which means adversary can only modify spam emails in the testset.
* Update the baseline NB classiﬁer in response to the attacker’s strategy above. The defender pays a unit price for both false positives and false negatives.
Before attackers make modifications to test emails, let's calculate the False Negative rate of the baseline NB classifier through confusion matrix. Here, the baseline model is multinomial NB with binary features (N=10). Assume positve class is spam email class, negative class is legit email class.
End of explanation
"""


spam_email_list = []

for email_idx, row in lingspam_test_df.iterrows():
  if row['is_spam'] == 1:
    spam_email_list.append(email_idx)

spam_email_size = len(spam_email_list)
print("spam email size: {}".format(spam_email_size))
print("spam email list: {}".format(spam_email_list))


"""
Explanation: First, let's get all spam emails in the testset.
End of explanation
"""


print('performing ADD-WORDS strategy on testset...')
all_cost = 0
modified_x_test = x_test

for email_idx, row in lingspam_test_df.iterrows():
  if email_idx in spam_email_list:
    cost = 0
    print('email index: {}'.format(email_idx))
    features_matrix = np.zeros((1, 10))
    email_body = row['email_body'].split(' ')
    for word_idx in range(len(top_10_features_list)):
      word = top_10_features_list[word_idx]
      if word in email_body:
        features_matrix[0, word_idx] = 1
    
    print('original feature matrix: {}'.format(features_matrix))

    while multinomial_nb_binary_baseline.predict(features_matrix) != 0:
      idx = next((i for i, x in enumerate(features_matrix[0]) if x == 0), None)
      if idx == None:
        break
      features_matrix[0, idx] = 1
      cost += 1
    
    all_cost += cost
    modified_x_test[email_idx] = features_matrix
    print('modified featurex matrix: {}'.format(features_matrix))
    print('cost: {}'.format(cost))
  
avg_cost = all_cost / spam_email_size
print('all cost on testset: {}'.format(all_cost))
print('average cost: {}'.format(avg_cost))

"""
Explanation: Next perform ADD-WORDS strategy for the adversary on these spam emails. The attacker seeks to ﬁnd the minimum cost solution such that each spam email in the test set gets classiﬁed as legitimate by the baseline NB classiﬁer. I will add words to spam emails in testset so that they can fool the classifier. The words are chosen from top-10 terms ranked by the IG metric. The ADD-WORDS strategy is like a greedy algorithm here, once the prediction result of a spam email flips, stop adding words.
The cost of adding a word is 1. I will calculate the average cost of the attacker’s modiﬁcations, averaged over all spam emails in the test set.
End of explanation
"""


after_y_pred = multinomial_nb_binary_baseline.predict(modified_x_test)

acc_score = accuracy_score(y_test, after_y_pred)
precision = precision_score(y_test, after_y_pred)
recall = recall_score(y_test, after_y_pred)
conf_mat = confusion_matrix(y_test, after_y_pred, labels = [0, 1])

tn = conf_mat[0][0]
fn = conf_mat[1][0]
tp = conf_mat[1][1]
fp = conf_mat[0][1]
# False positive rate
fpr = fp / (fp + tn)
# False negative rate
after_fnr = fn / (tp + fn)

print("After the attacker's modifications to test emails")
print("baseline nb classifier accuracy rate: {}, precision: {}, recall: {}".format(acc_score, precision, recall))
print("confusion matrix: \n{}".format(conf_mat))
print("tn: {}, fp: {}, fn: {}, tp: {}".format(tn, fp, fn, tp))
print("fpr: {}, fnr: {}".format(fpr, after_fnr))

"""
Explanation: Evaluate baseline classifier on modified testset.
End of explanation
"""


train_spam_email_list = []

for email_idx, row in lingspam_train_df.iterrows():
  if row['is_spam'] == 1:
    train_spam_email_list.append(email_idx)

train_spam_email_size = len(train_spam_email_list)
print("spam email size: {}".format(train_spam_email_size))
print("spam email list: {}".format(train_spam_email_list))


print('performing ADD-WORDS strategy on trainset...')
all_cost_train = 0
modified_x_train = x_train

for email_idx, row in lingspam_train_df.iterrows():
  if email_idx in train_spam_email_list:
    cost = 0
    # print('email index: {}'.format(email_idx))
    features_matrix = np.zeros((1, 10))
    email_body = row['email_body'].split(' ')
    for word_idx in range(len(top_10_features_list)):
      word = top_10_features_list[word_idx]
      if word in email_body:
        features_matrix[0, word_idx] = 1
    
    # print('original feature matrix: {}'.format(features_matrix))

    while multinomial_nb_binary_baseline.predict(features_matrix) != 0:
      idx = next((i for i, x in enumerate(features_matrix[0]) if x == 0), None)
      if idx == None:
        break
      features_matrix[0, idx] = 1
      cost += 1
    
    all_cost_train += cost
    modified_x_train[email_idx] = features_matrix
    # print('modified featurex matrix: {}'.format(features_matrix))
    # print('cost: {}'.format(cost))
  
avg_cost_train = all_cost_train / train_spam_email_size
print('all cost on trainset: {}'.format(all_cost_train))
print('average cost: {}'.format(avg_cost_train))

print('Updateing baseline classifier...')
multinomial_nb_binary_updated = MultinomialNB()
multinomial_nb_binary_updated.fit(modified_x_train, y_train)

updated_y_pred = multinomial_nb_binary_updated.predict(modified_x_test)

acc_score = accuracy_score(y_test, updated_y_pred)
precision = precision_score(y_test, updated_y_pred)
recall = recall_score(y_test, updated_y_pred)
conf_mat = confusion_matrix(y_test, updated_y_pred, labels = [0, 1])

tn = conf_mat[0][0]
fn = conf_mat[1][0]
tp = conf_mat[1][1]
fp = conf_mat[0][1]
# False positive rate
updated_fpr = fp / (fp + tn)
# False negative rate
updated_fnr = fn / (tp + fn)

print("baseline nb classifier accuracy rate: {}, precision: {}, recall: {}".format(acc_score, precision, recall))
print("confusion matrix: \n{}".format(conf_mat))
print("tn: {}, fp: {}, fn: {}, tp: {}".format(tn, fp, fn, tp))
print("fpr: {}, fnr: {}".format(updated_fpr, updated_fnr))

"""
Explanation: Apply the same ADD-WORDS strategy on training set, then retrain the baseline classifier and evaluate the updated baseline classifier.
End of explanation
"""


print('False Negative Rate of the baseline NB classifier before attacker\'s modification: {}'.format(before_fnr))
print('False Negative Rate of the baseline NB classifier after attacker\'s modification: {}'.format(after_fnr))
print('Average cost of attacker\'s modifications: {}'.format(avg_cost))
print('False Negative Rate of updated NB classifier: {}'.format(updated_fnr))
print('False Positvive Rate of updated NB classifier: {}'.format(updated_fpr))

"""
Explanation: Result and Conclusion
The evaluation results of Naive Bayes classifers are in the table below. It has 9 rows, one for each classifier and N combination ($N = {10, 100, 1000}$).
Naive Bayes Classifier | N | Precision | Recall
--- | --- | --- | ---
Bernoulli NB classiﬁer with binary features | 10 | 0.8888888888888888 | 0.8163265306122449
Bernoulli NB classiﬁer with binary features | 100 | 1.0 | 0.673469387755102
Bernoulli NB classiﬁer with binary features | 1000 | 1.0 | 0.6122448979591837
Multinomial NB with binary features | 10 | 0.8888888888888888 | 0.8163265306122449
Multinomial NB with binary features | 100 | 0.9782608695652174 | 0.9183673469387755
Multinomial NB with binary features | 1000 | 1.0 | 0.9387755102040817
Multinomial NB with term frequency (TF) features | 10 | 0.8518518518518519 | 0.9387755102040817
Multinomial NB with term frequency (TF) features | 100 | 0.9787234042553191 | 0.9387755102040817
Multinomial NB with term frequency (TF) features | 1000 | 1.0 | 0.9387755102040817
The evaluation results of SVM classifers with different hyperparameters are in the table below. 4 different kernels are tested, they are ['linear', 'poly', 'rbf', 'sigmoid']. $C$ represents regularization parameter. Here, $C \in [1, 2]$. The strength of the regularization is inversely proportional to $C$. SVM classifers use top-1000 features selected based on information gain and use term frequency features. Also, 5-fold cross validation is performed on training set.
SVM Classifier | Kernel | C | Cross Validation | Precision | Recall
--- | --- | --- | --- | --- | ---
SVM classifier with TF features | linear | 1 | False | 0.7924528301886793 | 0.8571428571428571
SVM classifier with TF features | linear | 1 | True | 0.9879518072289156 | 0.9651162790697675
SVM classifier with TF features | linear | 2 | False | 0.7777777777777778 | 0.8571428571428571
SVM classifier with TF features | linear | 2 | True | 0.9764705882352941 | 0.9651162790697675
SVM classifier with TF features | poly | 1 | False | 1.0 | 0.14285714285714285
SVM classifier with TF features | poly | 1 | True | 1.0 | 0.39080459770114945
SVM classifier with TF features | poly | 2 | False | 1.0 | 0.20408163265306123
SVM classifier with TF features | poly | 2 | True | 1.0 | 0.40229885057471265
SVM classifier with TF features | rbf | 1 | False | 1.0 | 0.673469387755102
SVM classifier with TF features | rbf | 1 | True | 1.0 | 0.7441860465116279
SVM classifier with TF features | rbf | 2 | False | 1.0 | 0.7551020408163265
SVM classifier with TF features | rbf | 2 | True | 1.0 | 0.813953488372093
SVM classifier with TF features | sigmoid | 1 | False | 0.85 | 0.6938775510204082
SVM classifier with TF features | sigmoid | 1 | True | 0.7974683544303798 | 0.7325581395348837
SVM classifier with TF features | sigmoid | 2 | False | 0.8571428571428571 | 0.7346938775510204
SVM classifier with TF features | sigmoid | 2 | True | 0.7386363636363636 | 0.7558139534883721
The evaluation results of Adversarial Attack classifers are as following. Here, baseline NB classifier is multinomial NB with binary features (N=10, top-10 features are selected).
End of explanation
"""


