Predicting outliers in network data I'm implementing KNN algorithm to predict outliers in network data which contains the following columns: source IP address, source port number, protocol and total bytes transferred. To achieve this, I'm using the Python library sklearn to predict response variable (total bytes) with regards to explanatory variables (source IP address, source port number, protocol).

However, the accuracy of prediction on test data is extremely low (0.21). Any recommendations to improve the accuracy score will be much appreciated.

import pandas as pd
from sklearn.neighbors import KNeighborsRegressor
import random
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from numpy.random import permutation
from sklearn.metrics import accuracy_score
import sys
import math
import netaddr as nt 
from netaddr import IPAddress

#Import csv into dataframe
df = pd.read_csv('data.csv')
df = df[['sourceaddress','sourceport','protocol','total_bytes']]

df_arr = df.as_matrix()

for i in range(df_arr.shape[0]):
    df_arr[i,0] = int(nt.IPAddress(df_arr[i,0]))

df = pd.DataFrame(df_arr)
df.columns = ['sourceaddress','sourceport','protocol','total_bytes']
df = df.astype('str')

# Design matrix X and target vector y
X = np.array(df.ix[:, 0:3])     
Y = np.array(df['total_bytes']) 

# split into train and test
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.33, random_state=42)


knn = KNeighborsClassifier(n_neighbors=24)
# fitting the model
knn.fit(X_train, Y_train)

# prediction of the response variable
pred = knn.predict(X_test)
print pred

# evaluate accuracy
print accuracy_score(Y_test, pred)
You mention accuracy, but talk about the number of bytes, so I'm going to assume that you apply a cut-off point to number of bytes and convert it to binary variable behind the screen. Now the answer:

The reason why the performance is so low is most probably lack of feature engineering. You should definitely inject as much domain knowledge as possible. Let's go over it step by step:


You are, if I understand correctly one-hot encoding IP addressees - however, it could be done in more concise and pythonic way without casting dataframe to matrix:

dummies = pd.get_dummies(df['sourceaddress'], prefix='ip')  # also called one-hot encoding
df = df.merge(dummies, left_index=True)


Protocol should be one-hot encoded as well, as it is a categorical variable. 

Now you might want to group IPs by geographic region - this will allow the algorithm to pick up any correlations between not only particular IP but a country or continent. That's how you can add such a feature:

df['region'] = determine_region([df['sourceaddress'])


If you write determine_region(ip) in a way that allows array-wide operations (in other words using numpy), it will be applied to whole dataset at once. This vectorization tends to be faster than .apply, especially for complex functions. And you should also one-hot encode the regions.

Same probably applies to sourceport (not an expert on web technology, so making assumptions here) - there are certain ports that are used in general for some thing and not for others, so another feature that reflects that could be crafted and then one-hot encoded (all your features are categorical so far, tough luck). 

Now, speculating here, but maybe combination of region and sourceport is meaningful as well? Really think about anything and everything that might explain the value of your target and create a feature out of that. Then you could choose the most relevant features, using one of the feature selection techniques, but that's not nearly as important as creating features in the first place.
Next, I expect that your dataset is highly imbalanced, which means that your algorithm will be biased towards majority class. What is typically done is rebalancing. Depending on the size of your dataset, you either upsample the minority class (with replacement obviosuly), or downsample the majority class (without replacement, that's important). An example of downsampling when majority class is ~total_bytes - so normal amount of bytes:

df_majority = df[~df['total_bytes']]
df_minority = df[df['total_bytes']]
df_majority_downsampled = resample(df_majority,
                               replace=False,
                               n_samples=len(df_minority),
                               random_state=0)  
df_downsampled = pd.concat([df_majority_downsampled, df_minority])


Now the amount of positive and negative examples is equal, so no bias.

Then you should try to tune the hyper parameters. Good way to do that is gridsearch with crossvalidation. Docs are excellent, so no need to explain further here.


Doing all this should improve the performance of your model.