Selecting features for malware analysis I am trying to build a classifier that detects if I have a malaware by predicting the provenance compiler. To do so I have a dataset composed of assembly code in json format :



In particular, I want to select as features the instructions, so the push, mov, jmp,..etc and create a feature vector that contains the number of times a feature appears. So, I want to apply the bag of words. To do so my code is the following:

#libraries
import numpy as np
import pandas as pd
import json as j
import re
import nltk
from nltk.tokenize import word_tokenize

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import *
from sklearn.metrics import confusion_matrix, classification_report
from sklearn import svm

#for visualizing data
import matplotlib.pyplot as plt
import seaborn as sns; sns.set(font_scale=1.2)

%matplotlib inline

json_data = None;
with open('training_dataset.jsonl') as data_file:
    lines = data_file.readlines()
    joined_lines = "[" + ",".join(lines)+"]"

    json_data = j.loads(joined_lines)  

data = pd.DataFrame(json_data)
data.head()



vect = CountVectorizer()

data['instructions'] = data['instructions'].apply(lambda x: ' '.join(x))

vect.fit_transform(data['instructions'])

a =vect.vocabulary_
a


and from this I obtain a dictionary:



with more and more key, value pairs. At this point what I tried to do is to eliminate the registers so my idea to do this is to iterate the dictionary and eliminate the keys that have numbers, and so I have written the following:

def hasNumbers(inputString):
    return any(char.isdigit() for char in inputString)

for k in list(a.keys()):
    if hasNumbers(k):
        del a[k]


but actually it wasn't a really good idea because I have also registers without numbers, for example rdi.

At this point, I don't know how to move, and I am not sure I am going in the correct way. Can somebody please help me? Thank's in advance.

[EDIT 2] 
Now I am trying to take the json file, not opening it with pandas, but open it as a dictionary, so:



and my intenction is try to split, as suggested, each key in such a way to eliminate the registers. So, my code for this is :

for key in json_data:
    splitted = key.split[0]


but I recieve the following error message:

AttributeError: 'dict' object has no attribute 'split'


I am getting really confused about how to operate now.

[EDIT 3] or I tried to do the following:

for v in json_data.values():
    splitted = v.split[0]


but it gives me the following error message:

AttributeError: 'list' object has no attribute 'values'


a problem I think is the fact that the various push r12,.. etc are in a list, so I don't know how to do.

[EDIT 4] I actually tried to solve the above problem this way:

for v in json_data[0].values():
    splitted = v.split[0]


but again I get the error message:

AttributeError: 'list' object has no attribute 'split'


[EDIT 5] Now, I am trying to do this using the file opened with pandas, so:

data['features'] = [i.split[0] for i in data['instructions']]


but still I get the following error:

AttributeError: 'list' object has no attribute 'split'


or I tried also:

feat = [i.split[0] for i in json_data[0].values()]


but get the same error as above.
In particular, I want to select as features the instructions, so the push, mov, jmp,..etc and create a feature vector that contains the number of times a feature appears.


Instead of reading each instruction entirely for a sample program, you can read the first word of each instruction to get the opcode and then create a feature vector out of these opcodes.

Suppose,

sample_program[i] =  ['push r12', 'push rbp', 'xor edx edx', ...]


i.e. a list of instructions, then define opcodes[i] as

opcodes[i] = [instruction.split[0] for instruction in sample_program[i]] 
#['push', 'push', 'xor']


Now you can create feature vectors out of collections of such opcodes.