import numpy as np
import pandas as pd
import xgboost
from sklearn import model_selection, metrics

"""
Explanation: Detecting Encrypted TOR Traffic with Boosting and Topological Data Analysis
HJ van Veen - MLWave
We establish strong baselines for both supervised and unsupervised detection of encrypted TOR traffic.
**Note: This article uses the 5-second lag dataset. For better comparison we will use the 15-second lag dataset in the near future.
Introduction
Gradient Boosted Decision Trees (GBDT) is a very powerful learning algorithm for supervised learning on tabular data <a href="#References">[1]</a>. Modern implementations include XGBoost <a href="#References">[2]</a>, Catboost <a href="#References">[3]</a>, LightGBM <a href="#References">[4]</a> and scikit-learn's GradientBoostingClassifier <a href="#References">[5]</a>. Of these, especially XGBoost has seen tremendous successes in machine learning competitions <a href="#References">[6]</a>, starting with its introduction during the Higgs Boson Detection challenge in 2014 <a href="#References">[7]</a>. The success of XGBoost can be explained on multiple dimensions: It is a robust implementation of the original algorithms, it is very fast -- allowing data scientists to quickly find better parameters <a href="#References">[8]</a>, it does not suffer much from overfit, is scale-invariant, and it has an active community providing constant improvements, such as early stopping <a href="#References">[9]</a> and GPU support <a href="#References">[10]</a>.
Anomaly detection algorithms automatically find samples that are different from regular samples. Many methods exist. We use the Isolation Forest in combination with nearest neighbor distances. The Isolation Forest works by randomly splitting up the data <a href="#References">[11]</a>. Outliers, on average, are easier to isolate through splitting. Nearest neighbor distance looks at the summed distances for a sample and its five nearest neighbors. Outliers, on average, have a larger distance between their nearest neighbors than regular samples <a href="#References">[12]</a>.
Topological Data Analysis (TDA) is concerned with the meaning, shape, and connectedness of data <a href="#References">[13]</a>. Benefits of TDA include: Unsupervised data exploration / automatic hypothesis generation, ability to deal with noise and missing values, invariance, and the generation of meaningful compressed summaries. TDA has shown efficient applications in a number of diverse fields: healthcare <a href="#References">[14]</a>, computational biology <a href="#References">[15]</a>, control theory <a href="#References">[16]</a>, community detection <a href="#References">[17]</a>, machine learning <a href="#References">[18]</a>, sports analysis <a href="#References">[19]</a>, and information security <a href="#References">[20]</a>. One tool from TDA is the $MAPPER$ algorithm. $MAPPER$ turns data and data projections into a graph by covering it with overlapping intervals and clustering <a href="#References">[21]</a>. To guide exploration, the nodes of the graph may be colored with a function of interest <a href="#References">[22]</a>. There are an increasing number of implementations of $MAPPER$. We use the open source implementation KeplerMapper from scikit-TDA <a href="#References">[23]</a>.
The TOR network allows users to communicate and host content while preserving privacy and anonimity <a href="#References">[24]</a>. As such, it can be used by dissidents and other people who prefer not to be tracked by commercial companies or governments. But these strong privacy and anonimity features are also attractive to criminals. A 2016 study in 'Survival - Global Politics and Strategy' found at least 57% of TOR websites are involved in illicit behavior, ranging from the trade in illegal arms, counterfeit ID documents, pornography, and drugs, money laundering & credit card fraud, and the sharing of violent material, such as bomb making tutorials and terrorist propaganda <a href="#References">[25]</a>.
Network Intrusion Detection Systems are a first line of defense for governments and companies <a href="#References">[26]</a>. An undetected hacker will try to elevate their priviledges, moving from the weakest link to more hardened system-critical network nodes <a href="#References">[27]</a>. If the hacker's goal is to get access to sensitive data (for instance: for resale -, industrial espionage -, or extortion purposes) then any stolen data needs to be exfiltrated. Similarly, cryptolockers often need to communicate with a command & control server outside the network. Depending on the level of sophistication of the malware or hackers, exfiltration may be open and visible, run encrypted through the TOR network in an effort to hide the destination, or use advanced DNS tunneling techniques.
Motivation

Current Network Intrusion Detection Systems, much like the old spam detectors, rely mostly on rules, signatures, and anomaly detection. Labeled data is scarce. Writing rules is a very costly task requiring domain expertise. Signatures may fail to catch new types of attacks until they are updated. Anomalous/unusual behavior is not necessarily suspicous/adversarial behavior.
Machine Learning for Information Security suffers a lot from poor false positive rates. False positives lead to alarm fatigue and can swamp an intelligence analyst with irrelevant work.
Despite the possibility of false positives, it is often better to be safe than sorry. Suspicious network behavior, such as outgoing connections to the TOR network, require immediate attention. A network node can be shut down remotely, after which a security engineer can investigate the machine. The best practice of a multi-layered security makes this possible <a href="#References">[28]</a>: Instead of a single firewall to rule them all, hackers can be detected in various stages of their network intrusion, up to the final step of data exfilitration.

Data
We use a dataset written for the paper "Characterization of Tor Traffic Using Time Based Features" (Lashkari et al.) <a href="#References">[29]</a>, graciously provided by the Canadian Institute for Cybersecurity <a href="#References">[30]</a>. This dataset combines older research on nonTOR network traffic with more recently captured TOR traffic (both were created on the same network) <a href="#References">[31]</a>. The data includes features that are more specific to the network used, such as the source and destination IP/Port, and a range of time-based features with a 5 second lag.
|Feature|Type|Description|Time-based|
|---|---|---|
|'Source IP'|Object|Source IP4 Address. String with dots.|No|
|' Source Port'|Float|Source Port sending packets.|No|
|' Destination IP'|Object|Destination IP4 Address.|No|
|' Destination Port'|Float|Destination Port receiving packets.|No|
|' Protocol'|Float|Integer [5-17] denoting protocol used.|No|
|' Flow Duration'|Float|Length of connection in seconds|Yes|
|' Flow Bytes/s'|Float|Bytes per seconds send|Yes|
|' Flow Packets/s'|Object|Packets per second send. Contains "infinity" strings.|Yes|
|' Flow IAT Mean'|Float|Flow Inter Arrival Time.|Yes|
|' Flow IAT Std'|Float||Yes|
|' Flow IAT Max'|Float||Yes|
|' Flow IAT Min'|Float||Yes|
|'Fwd IAT Mean'|Float|Forward Inter Arrival Time.|Yes|
|' Fwd IAT Std'|Float||Yes|
|' Fwd IAT Max'|Float||Yes|
|' Fwd IAT Min'|Float||Yes|
|'Bwd IAT Mean'|Float|Backwards Inter Arrival Time.|Yes|
|' Bwd IAT Std'|Float||Yes|
|' Bwd IAT Max'|Float||Yes|
|' Bwd IAT Min'|Float||Yes|
|'Active Mean'|Float|Average amount of time in seconds before connection went idle.|Yes|
|' Active Std'|Float||Yes|
|' Active Max'|Float||Yes|
|' Active Min'|Float||Yes|
|'Idle Mean'|Float|Average amount of time in seconds before connection became active.|Yes|
|' Idle Std'|Float|Zero variance feature.|Yes|
|' Idle Max'|Float||Yes|
|' Idle Min'|Float||Yes|
|'label'|Object|Either "nonTOR" or "TOR". ~17% TOR signal.|-|
Experimental setup

Supervised ML. We establish a strong baseline with XGBoost on the full data and on a subset (only time-based features, which generalize better to new domains). We follow the dataset standard of creating a 20% holdout validation set, and use 5-fold stratified cross-validation for parameter tuning <a href="#References">[32]</a>. For tuning we use random search on sane parameter ranges, as random search is easy to implement and given enough time, will equal or beat more sophisticated methods <a href="#References">[33]</a>. We do not use feature selection, but opt to let our learning algorithm deal with those. Missing values are also handled by XGBoost and not manually imputed or hardcoded.
Unsupervised ML. We use $MAPPER$ in combination with the Isolation Forest and the summed distances to the five nearest neighbors. We use an overlap percentage of 150% and 40 intervals per dimension for a total of 1600 hypercubes. Clustering is done with agglomerative clustering using the euclidean distance metric and 3 clusters per interval. For these experiments we use only the time-based features. We don't scale the data, despite only Isolation Forest being scale-invariant.
End of explanation
"""


df = pd.read_csv("CSV/Scenario-A/merged_5s.csv")

df.replace('Infinity', -1, inplace=True)
df["label"] = df["label"].map({"nonTOR": 0, "TOR": 1})
df["Source IP"] = df["Source IP"].apply(lambda x: float(x.replace(".", "")))
df[" Destination IP"] = df[" Destination IP"].apply(lambda x: float(x.replace(".", "")))

features_all = [c for c in df.columns if c not in 
            ['label']]

features = [c for c in df.columns if c not in 
            ['Source IP',
             ' Source Port',
             ' Destination IP',
             ' Destination Port',
             ' Protocol',
             'label']]
features

X = np.array(df[features])
X_all = np.array(df[features_all])
y = np.array(df.label)
print(X.shape, np.mean(y))

"""
Explanation: Data Prep
There are string values "Infinity" inside the data, causing mixed types. We need to label-encode the target column. We turn the IP addresses into floats by removing the dots.
We also create a subset of features by removing Source Port, Source IP, Destination Port, Destination IP, and Protocol. This to avoid overfitting/improve future generalization and focus only on the time-based features, like most other researchers have done.
End of explanation
"""


splitter = model_selection.StratifiedShuffleSplit(
    n_splits=1,
    test_size=0.2,
    random_state=0)

for train_index, test_index in splitter.split(X, y):
    X_train, X_holdout = X[train_index], X[test_index]
    X_train_all, X_holdout_all = X_all[train_index], X_all[test_index]
    y_train, y_holdout = y[train_index], y[test_index]
    
print(X_train.shape, X_holdout.shape)

"""
Explanation: Local evaluation setup
We create a stratified holdout set of 20%. Any modeling choices (such as parameter tuning) are guided by 5-fold stratified cross-validation on the remaining dataset.
End of explanation
"""


model = xgboost.XGBClassifier(seed=0)
print(model)

skf = model_selection.StratifiedKFold(
    n_splits=5,
    shuffle=True,
    random_state=0)

for i, (train_index, test_index) in enumerate(skf.split(X_train, y_train)):
    X_train_fold, X_test_fold = X_train[train_index], X_train[test_index]
    y_train_fold, y_test_fold = y_train[train_index], y_train[test_index]
    model.fit(X_train_fold, y_train_fold)
    probas = model.predict_proba(X_test_fold)[:,1]
    preds = (probas > 0.5).astype(int)
    
    print("-"*60)
    print("Fold: %d (%s/%s)" %(i, X_train_fold.shape, X_test_fold.shape))
    print(metrics.classification_report(y_test_fold, preds, target_names=["nonTOR", "TOR"]))
    print("Confusion Matrix: \n%s\n"%metrics.confusion_matrix(y_test_fold, preds))
    print("Log loss : %f" % (metrics.log_loss(y_test_fold, probas)))
    print("AUC      : %f" % (metrics.roc_auc_score(y_test_fold, probas)))
    print("Accuracy : %f" % (metrics.accuracy_score(y_test_fold, preds)))
    print("Precision: %f" % (metrics.precision_score(y_test_fold, preds)))
    print("Recall   : %f" % (metrics.recall_score(y_test_fold, preds)))
    print("F1-score : %f" % (metrics.f1_score(y_test_fold, preds)))

"""
Explanation: 5-fold non-tuned XGBoost
End of explanation
"""


model = xgboost.XGBClassifier(base_score=0.5, colsample_bylevel=0.68, colsample_bytree=0.84,
    gamma=0.1, learning_rate=0.1, max_delta_step=0, max_depth=11,
    min_child_weight=1, missing=None, n_estimators=1122, nthread=-1,
    objective='binary:logistic', reg_alpha=0.0, reg_lambda=4,
    scale_pos_weight=1, seed=189548, silent=True, subsample=0.98)

"""
Explanation: Hyper parameter tuning
We found the below parameters by running a random gridsearch on the first fold in ~50 iterations (minimizing log loss). We use an AWS distributed closed-source auto-tuning library called "Cher" with the following parameter ranges:
"XGBClassifier": {
    "max_depth": (2,12),
    "n_estimators": (20, 2500),
    "objective": ["binary:logistic"],
    "missing": np.nan,
    "gamma": [0, 0, 0, 0, 0, 0.01, 0.1, 0.2, 0.3, 0.5, 1., 10., 100.],
    "learning_rate": [0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1, 0.15, 0.2, 0.1 ,0.1],
    "min_child_weight": [1, 1, 1, 1, 2, 3, 4, 5, 1, 6, 7, 8, 9, 10, 11, 15, 30, 60, 100, 1, 1, 1],
    "max_delta_step": [0, 0, 0, 0, 0, 1, 2, 5, 8],
    "nthread": -1,
    "subsample": [i/100. for i in range(20,100)],
    "colsample_bytree": [i/100. for i in range(20,100)],
    "colsample_bylevel": [i/100. for i in range(20,100)],
    "reg_alpha": [0, 0, 0, 0, 0, 0.00000001, 0.00000005, 0.0000005, 0.000005],
    "reg_lambda": [1, 1, 1, 1, 2, 3, 4, 5, 1],
    "scale_pos_weight": 1,
    "base_score": 0.5,
    "seed": (0,999999)
}
End of explanation
"""


print(model)
for i, (train_index, test_index) in enumerate(skf.split(X_train, y_train)):
    X_train_fold, X_test_fold = X_train[train_index], X_train[test_index]
    y_train_fold, y_test_fold = y_train[train_index], y_train[test_index]
    model.fit(X_train_fold, y_train_fold)
    probas = model.predict_proba(X_test_fold)[:,1]
    preds = (probas > 0.5).astype(int)

    print("-"*60)
    print("Fold: %d (%s/%s)" %(i, X_train_fold.shape, X_test_fold.shape))
    print(metrics.classification_report(y_test_fold, preds, target_names=["nonTOR", "TOR"]))
    print("Confusion Matrix: \n%s\n"%metrics.confusion_matrix(y_test_fold, preds))
    print("Log loss : %f" % (metrics.log_loss(y_test_fold, probas)))
    print("AUC      : %f" % (metrics.roc_auc_score(y_test_fold, probas)))
    print("Accuracy : %f" % (metrics.accuracy_score(y_test_fold, preds)))
    print("Precision: %f" % (metrics.precision_score(y_test_fold, preds)))
    print("Recall   : %f" % (metrics.recall_score(y_test_fold, preds)))
    print("F1-score : %f" % (metrics.f1_score(y_test_fold, preds)))

"""
Explanation: 5-fold tuned XGBoost
End of explanation
"""


model.fit(X_train, y_train) 
probas = model.predict_proba(X_holdout)[:,1]
preds = (probas > 0.5).astype(int)

print(metrics.classification_report(y_holdout, preds, target_names=["nonTOR", "TOR"]))
print("Confusion Matrix: \n%s\n"%metrics.confusion_matrix(y_holdout, preds))
print("Log loss : %f" % (metrics.log_loss(y_holdout, probas)))
print("AUC      : %f" % (metrics.roc_auc_score(y_holdout, probas)))
print("Accuracy : %f" % (metrics.accuracy_score(y_holdout, preds)))
print("Precision: %f" % (metrics.precision_score(y_holdout, preds)))
print("Recall   : %f" % (metrics.recall_score(y_holdout, preds)))
print("F1-score : %f" % (metrics.f1_score(y_holdout, preds)))

"""
Explanation: Holdout set evaluation
End of explanation
"""


model.fit(X_train_all, y_train) 
probas = model.predict_proba(X_holdout_all)[:,1]
preds = (probas > 0.5).astype(int)

print(metrics.classification_report(y_holdout, preds, target_names=["nonTOR", "TOR"]))
print("Confusion Matrix: \n%s\n"%metrics.confusion_matrix(y_holdout, preds))
print("Log loss : %f" % (metrics.log_loss(y_holdout, probas)))
print("AUC      : %f" % (metrics.roc_auc_score(y_holdout, probas)))
print("Accuracy : %f" % (metrics.accuracy_score(y_holdout, preds)))
print("Precision: %f" % (metrics.precision_score(y_holdout, preds)))
print("Recall   : %f" % (metrics.recall_score(y_holdout, preds)))

"""
Explanation: Results
|Model|Precision|Recall|F1-Score
|---|---|---|---|
|Logistic Regression (Singh et al., 2018) <a href="#References">[34]</a>|0.87|0.87|0.87|
|SVM (Singh et al., 2018)|0.9|0.9|0.9|
|Naïve Bayes (Singh et al., 2018)|0.91|0.6|0.7|
|C4.5 Decision Tree + Feature Selection (Lashkari et al., 2017) <a href="#References">[29]</a>|0.948|0.934|-|
|Deep Learning (Singh et al., 2018)|0.95|0.95|0.95|
|Random Forest (Singh et al., 2018)|0.96|0.96|0.96|
|XGBoost + Tuning|0.974|0.977|0.976|
Holdout evaluation with all the available features
Using all the features results in near perfect performance, suggesting "leaky" features (These features are not to be used for predictive modeling, but are there for completeness). Nevertheless we show how using all features also results in a strong baseline over previous research.
End of explanation
"""


import kmapper as km
import pandas as pd
import numpy as np
from sklearn import ensemble, cluster

df = pd.read_csv("CSV/Scenario-A/merged_5s.csv")
df.replace('Infinity', -1, inplace=True)
df[" Flow Bytes/s"] = df[" Flow Bytes/s"].apply(lambda x: float(x))
df[" Flow Packets/s"] = df[" Flow Packets/s"].apply(lambda x: float(x))
df["label"] = df["label"].map({"nonTOR": 0, "TOR": 1})
df["Source IP"] = df["Source IP"].apply(lambda x: float(x.replace(".", "")))
df[" Destination IP"] = df[" Destination IP"].apply(lambda x: float(x.replace(".", "")))
df.fillna(-2, inplace=True)

features = [c for c in df.columns if c not in 
            ['Source IP',
             ' Source Port',
             ' Destination IP',
             ' Destination Port',
             ' Protocol',
             'label']]

X = np.array(df[features])
y = np.array(df.label)

projector = ensemble.IsolationForest(random_state=0, n_jobs=-1)
projector.fit(X)
lens1 = projector.decision_function(X)

mapper = km.KeplerMapper(verbose=3)
lens2 = mapper.fit_transform(X, projection="knn_distance_5")

lens = np.c_[lens1, lens2]

G = mapper.map(
    lens,
    X,
    nr_cubes=40,
    overlap_perc=1.5,
    clusterer=cluster.AgglomerativeClustering(3))

_ = mapper.visualize(
    G,
    custom_tooltips=y,
    color_function=y,
    path_html="tor-tda.html",
    inverse_X=X,
    inverse_X_names=list(df[features].columns),
    projected_X=lens,
    projected_X_names=["IsolationForest", "KNN-distance 5"],
    title="Detecting encrypted Tor Traffic with Isolation Forest and Nearest Neighbor Distance"
)

"""
Explanation: Results
|Model|Precision|Recall|Accuracy
|---|---|---|---|
|ANN (Hodo et al., 2017) <a href="References">[35]</a>|0.983|0.937|0.991|
|SVM (Hodo et al., 2017)|0.79|0.67|0.94|
|ANN + Feature Selection (Hodo et al., 2017)|0.998|0.988|0.998|
|SVM + Feature Selection (Hodo et al., 2017)|0.8|0.984|0.881|
|XGBoost + Tuning|0.999|1.|0.999|
Topological Data Analysis
End of explanation
"""


