arXiv:1801.00318v1 [] 31 Dec 2017

Towards Building an Intelligent Anti-Malware System: A Deep
Learning Approach using Support Vector Machine (SVM) for
Malware Classification
Abien Fred M. Agarap∗

Francis John Hill Pepito†

Department of Computer Science
Adamson University
Manila, Philippines
abien.fred.agarap@adamson.edu.ph

Department of Computer Science
Adamson University
Manila, Philippines
francis.john.hill.pepito@gmail.com

ABSTRACT
Effective and efficient mitigation of malware is a long-time endeavor in the information security community. The development of
an anti-malware system that can counteract an unknown malware
is a prolific activity that may benefit several sectors. We envision
an intelligent anti-malware system that utilizes the power of deep
learning (DL) models. Using such models would enable the detection
of newly-released malware through mathematical generalization.
That is, finding the relationship between a given malware x and
its corresponding malware family y, f : x 7→ y. To accomplish this
feat, we used the Malimg dataset[12] which consists of malware
images that were processed from malware binaries, and then we
trained the following DL models1 to classify each malware family:
CNN-SVM[16], GRU-SVM[3], and MLP-SVM. Empirical evidence
has shown that the GRU-SVM stands out among the DL models
with a predictive accuracy of ≈84.92%. This stands to reason for
the mentioned model had the relatively most sophisticated architecture design among the presented models. The exploration of an
even more optimal DL-SVM model is the next stage towards the
engineering of an intelligent anti-malware system.

CCS CONCEPTS
• Security and privacy → Malware and its mitigation; • Computing methodologies → Supervised learning by classification; Support vector machines; Neural networks;

mentioned database contains the information on all known malware by the particular system[15], which is then used for malware
detection. Consequently, newly-released malware which are not
yet included in the database will go undetected.
We envision an intelligent anti-malware system that employs
a deep learning (DL) approach which would enable the detection
of newly-released malware through its capability to generalize on
data. Furthermore, we amend the conventional DL models to use
the support vector machine (SVM) as their classification function.
We take advantage of the Malimg dataset[12] which consists of
visualized malware binaries, and use it to train the DL-SVM models
to classify each malware family.

2 METHODOLOGY
2.1 Machine Intelligence Library
Google TensorFlow[2] was used to implement the deep learning
algorithms in this study, with the aid of other scientific computing
libraries: matplotlib[8], numpy[17], and scikit-learn[13].

2.2

The Dataset

The deep learning (DL) models in this study were evaluated on
the Malimg dataset[12], which consists of 9,339 malware samples
from 25 different malware families. Table 1 shows the frequency
distribution of malware families and their variants in the Malimg
dataset[12].

KEYWORDS
artificial intelligence; artificial neural networks; classification; convolutional neural networks; deep learning; machine learning; malware classification; multilayer perceptron; recurrent neural network; supervised learning; support vector machine

1

INTRODUCTION

Effective and efficient mitigation of malware is a long-time endeavor in the information security community. The development of
an anti-malware system that can counteract an unknown malware
is a prolific activity that may benefit several sectors.
To intercept an unknown malware or even just an unknown variant is a laborious task to undertake, and may only be accomplished
by constantly updating the anti-malware signature database. The
∗ Lead

Author, implemented the DL-SVM models and performed the experiments
† Co-author, converted the Malimg dataset[12] to NumPy[17] arrays
1 Code available at https://github.com/AFAgarap/malware-classification

Figure 1: Image from [12]. Visualizing malware as a
grayscale image.
Nataraj et al. (2011)[12] created the Malimg dataset by reading
malware binaries into an 8-bit unsigned integer composing a matrix
M ∈ Rm×n . The said matrix may be visualized as a grayscale image
having values in the range of [0, 255], with 0 representing black
and 1 representing white.

2.3

Dataset Preprocessing

Similar to what [6] did, the malware images were resized to a 2dimensional matrix of 32 × 32, and were flattened into a n × n-size

,,

Abien Fred M. Agarap and Francis John Hill Pepito

Table 1: Malware families found in the Malimg Dataset[12].
No.

Family

Family Name

No. of Variants

01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

Dialer
Backdoor
Worm
Worm
Trojan
Worm:AutoIT
Trojan
Trojan
Dialer
Trojan Downloader
Rogue
Dialer
PWS
PWS
PWS
PWS
Trojan
Trojan Downloader
Backdoor
Trojan
Trojan Downloader
Trojan Downloader
Worm
Trojan Downloader
Worm

Adialer.C
Agent.FYI
Allaple.A
Allaple.L
Alueron.gen!J
Autorun.K
C2Lop.P
C2Lop.gen!G
Dialplatform.B
Dontovo.A
Fakerean
Instantaccess
Lolyda.AA 1
Lolyda.AA 2
Lolyda.AA 3
Lolyda.AT
Malex.gen!J
Obfuscator.AD
Rbot!gen
Skintrim.N
Swizzor.gen!E
Swizzor.gen!I
VB.AT
Wintrim.BX
Yuner.A

122
116
2949
1591
198
106
146
200
177
162
381
431
213
184
123
159
136
142
158
80
128
132
408
97
800

(4)

where wT w is the Manhattan norm (also known as L1 norm), C
is the penalty parameter (may be an arbitrary value or a selected
value using hyper-parameter tuning), and ξ is a cost function.
The corresponding unconstrained optimization problem of Eq. 2
is given by Eq. 5.
(5)

where y ′ is the actual label, and wT x+b is the predictor function.
This equation is known as L1-SVM, with the standard hinge loss.
Its differentiable counterpart, L2-SVM (given by Eq. 6), provides
more stable results[16].
p
Õ
2
1
min ∥w∥22 + C
max 0, 1 − yi′ (wT xi + b)
p
i=1

X −µ
(1)
σ
where X is the feature to be standardized, µ is its mean value, and
σ is its standard deviation. The standardization was implemented using StandardScaler().fit_transform() of scikit-learn[13].
Granted that the dataset consists of images, and standardization
may not be suitable for such data, but take note that the images
originate from malware binary files. Hence, the features are not
technically images to begin with.
z=

Computational Models

This section presents the deep learning (DL) models, and the support
vector machine (SVM) classifier used in the study.
2.4.1 Support Vector Machine (SVM). The support vector machine (SVM) was developed by Vapnik[5] for binary classification.
Its objective is to find the optimal hyperplane f (w, x) = w · x + b
to separate two classes in a given dataset, with features x ∈ Rm .
SVM learns the parameters w and b by solving the following
constrained optimization problem:
p
Õ
1
min wT w + C
ξi
p
i=1

(3)

ξ i ≥ 0, i = 1, . . . , p

p
Õ

1
min wT w + C
max 0, 1 − yi′ (wT xi + b)
p
i=1

array, resulting to a 1 × 1024-size array. Each feature array was then
labelled with its corresponding indexed malware family name (i.e.
0 − 24). Then, the features were standardized using Eq. 1.

2.4

s.t yi′ (w · x + b) ≥ 1 − ξ i

(2)

(6)

where ∥w∥2 is the Euclidean norm (also known as L2 norm),
with the squared hinge loss.
Despite intended for binary classification, SVM may be used
for multinomial classification as well. One approach to achieve
this is the use of kernel tricks, which converts a linear model to a
non-linear model by applying kernel functions such as radial basis
function (RBF). However, for this study, we utilized the linear L2SVM for the multinomial classification problem. We then employed
the one-versus-all (OvA) approach, which treats a given class c i as
the positive class, and others as negative class.
Take for example the following classes: airplane, boat, car. If a
given image belongs to the airplane class, it is taken as the positive
class, which leaves the other two classes the negative class.
With the OvA approach, the L2-SVM serves as the classifier of
each deep learning model in this study (CNN, GRU, and MLP). That
is, the learning parameters weight and bias of each model is learned
by the SVM.
2.4.2 Convolutional Neural Network. Convolutional Neural Networks (CNNs) are similar to feedforward neural networks for they
also consist of hidden layers of neurons with “learnable” parameters.
These neurons receive inputs, performs a dot product, and then
follows it with a non-linearity such as siдmoid or tanh. The whole
network expresses the mapping between raw image pixels x ∈ Rm
and class scores y, f : x 7→ y. For this study, the CNN architecture
used resembles the one laid down in [1]:
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)

INPUT: 32 × 32 × 1
CONV5: 5 × 5 size, 36 filters, 1 stride
LeakyReLU: max(0.01hθ (x)), hθ (x))
POOL: 2 × 2 size, 1 stride
CONV5: 5 × 5 size, 72 filters, 1 stride
LeakyReLU: max(0.01hθ (x)), hθ (x))
POOL: 2 × 2 size, 1 stride
FC: 1024 Hidden Neurons
LeakyReLU: max(0.01hθ (x)), hθ (x))
DROPOUT: p = 0.85
FC: 25 Output Classes

Towards Building an Intelligent Anti-Malware System: A Deep Learning Approach using Support Vector Machine (SVM) for Malware
Classification
,,
The modification introduced in the architecture design was the
size of layer inputs and outputs (e.g. input of 32 × 32 × 1 instead
of 28 × 28 × 1, and output of 25 classes), the use of LeakyReLU
instead of ReLU, and of course, the introduction of L2-SVM as the
network classifier instead of the conventional Softmax function.
This paradigm of combining CNN and SVM was actually proposed
by Tang (2013)[16].
2.4.3 Gated Recurrent Unit. Agarap (2017)[3] proposed a neural
network architecture combining the gated recurrent unit (GRU)[4]
variant of a recurrent neural network (RNN) and the support vector
machine (SVM)[5] for the purpose of binary classification.
z = σ (Wz · [ht −1 , x t ])

(7)

r = σ (Wr · [ht −1 , x t ])

(8)

h̃t = tanh(W · [r t ∗ ht −1 , x t ])

(9)

ht = (1 − zt ) ∗ ht −1 + zt ∗ h̃t

(10)

where z and r are the update gate and reset gate of a GRU-RNN
respectively, h̃t is the candidate value, and ht is the new RNN cell
state value[4]. In turn, the ht is used as the predictor variable x in
the L2-SVM predictor function (given by wx + b) of the network
instead of the conventional Softmax classifier.
2.4.4 Multilayer Perceptron. The perceptron model was developed by Rosenblatt (1958)[14] based on the neuron model by McCulloch & Pitts (1943)[11]. A perceptron may be represented by a
linear function (given by Eq. 11), which is then passed to an activation function such as sigmoid σ , sign, or tanh. These activation
functions introduce non-linearity (except for the sign function) to
represent complex functions.
As the term itself implies, a multilayer perceptron (MLP) is a
neural network that consists of hidden layers of perceptrons. In this
study, the activation function used was the LeakyReLU[10] function
(given by Eq. 12).
hθ (x) =

n
Õ

θ i xi + b

(11)

i=0



f hθ (x) = max 0.01hθ (x), hθ (x)

(12)

The learning parameters weight and bias for each DL model
were learned by the L2-SVM using the loss function given by Eq. 6.
The computed loss is then minimized through Adam[9] optimization. Then, the decision function f (x) = siдn(wx + b) produces a
vector of scores for each malware family. In order to get the predicted labels y for a given data x, the arдmax function is used (see
Eq. 13).

y ′ = arдmax siдn(wx + b)
(13)
The arдmax function shall return the indices of the highest
scores across the vector of predicted classes wx + b.

2.5

Data Analysis

There were two phases of experiment for this study: (1) training
phase, and (2) test phase. All the deep learning algorithms described
in Section 2.4 were trained and tested on the Malimg dataset[12].

The dataset was partitioned in the following fashion: 70% for training phase, and 30% for testing phase.
The variables considered in the experiments were the following:
(1) Test Accuracy (the predictive accuracy on unseen data)
(2) Epochs (number of passes through the entire dataset)
(3) F1 score (harmonic mean of precision and recall, see Eq. 14)
(4) Number of data points
(5) Precision (Positive Predictive Value, see Eq. 15)
(6) Recall (True Positive Rate, see Eq. 16)
PPV · T PR
PPV + T PR

(14)

True Positive
True Positive + False Positive

(15)

F =2·
PPV =

True Positive
(16)
True Positive + False N eдative
The classification measures F1 score, precision, and recall were
all computed using the classification_report() function of
sklearn.metrics[13].
T PR =

3

RESULTS

All experiments in this study were conducted on a laptop computer
with Intel Core(TM) i5-6300HQ CPU @ 2.30GHz x 4, 16GB of DDR3
RAM, and NVIDIA GeForce GTX 960M 4GB DDR5 GPU. Table 2
shows the hyper-parameters used by the DL-SVM models in the
conducted experiments. Table 3 summarizes the experiment results
for the presented DL-SVM models.
Table 2: Hyper-parameters used in the DL-SVM models.
Hyper-parameters

CNN-SVM

GRU-SVM

MLP-SVM

Batch Size
Cell Size
No. of Hidden Layers
Dropout Rate
Epochs
Learning Rate
SVM C

256
N/A
2
0.85
100
1e-3
10

256
[256 × 5]
5
0.85
100
1e-3
10

256
[512, 256, 128]
3
None
100
1e-3
0.5

As opposed to what [6] did on dataset partitioning, the relative
populations of each malware family were not considered in the splitting process. All the DL-SVM models were trained on ≈70% of the
Table 3: Summary of experiment results on the DL-SVM
models.
Variables

CNN-SVM

GRU-SVM

MLP-SVM

Accuracy
Data points
Epochs
F1
Precision
Recall

77.2265625%
256000
100
0.79
0.84
0.77

84.921875%
256000
100
0.85
0.85
0.85

80.46875%
256000
100
0.81
0.83
0.80

,,

Abien Fred M. Agarap and Francis John Hill Pepito

preprocessed Malimg dataset[12], i.e. 6400 malware family variants
(6400 mod 256 = 0), for 100 epochs. On the other hand, the models
were tested on ≈30% of the preprocessed Malimg dataset[12], i.e.
2560 malware family variants (2560 mod 256 = 0), for 100 epochs.

Figure 2: Plotted using matplotlib[8]. Training accuracy
of the DL-SVM models on malware classification using
the Malimg dataset[12].
Figure 2 summarizes the training accuracy of the DL-SVM models for 100 epochs (equivalent to 2500 steps, since 6400 × 100 ÷ 256 =
2500). First, the CNN-SVM model accomplished its training in
3 minutes and 41 seconds with an average training accuracy of
80.96875%. Meanwhile, the GRU-SVM model accomplished its training in 11 minutes and 32 seconds with an average training accuracy
of 90.9375%. Lastly, the MLP-SVM model accomplished its training
in 12 seconds with an average training accuracy of 99.5768229%.

Figure 4: Plotted using matplotlib[8]. Confusion Matrix
for GRU-SVM testing results, showing its predictive accuracy for each malware family described in Table 1.

Figure 5: Plotted using matplotlib[8]. Confusion Matrix
for MLP-SVM testing results, showing its predictive accuracy for each malware family described in Table 1.

Figure 3: Plotted using matplotlib[8]. Confusion Matrix
for CNN-SVM testing results, showing its predictive accuracy for each malware family described in Table 1.
Figure 3 shows the testing performance of CNN-SVM model
in multinomial classification on malware families. The mentioned
model had a precision of 0.84, a recall of 0.77, and a F1 score of 0.79.
Figure 4 shows the testing performance of GRU-SVM model
in multinomial classification on malware families. The mentioned
model had a precision of 0.85, a recall of 0.85, and a F1 score of 0.85.

Figure 5 shows the testing performance of MLP-SVM model
in multinomial classification on malware families. The mentioned
model had a precision of 0.83, a recall of 0.80, and a F1 score of 0.81.
As shown in the confusion matrices, the DL-SVM models had
better scores for the malware families with the high number of
variants, most notably, Allaple.A and Allaple.L. This may be
pointed to the omission of relative populations of each malware
family during the partitioning of the dataset into training data and
testing data. However, unlike the results of [6], only Allaple.A
and Allaple.L had some misclassifications between them.

Towards Building an Intelligent Anti-Malware System: A Deep Learning Approach using Support Vector Machine (SVM) for Malware
Classification
,,

4

DISCUSSION

It is palpable that the GRU-SVM model stands out among the DLSVM models presented in this study. This finding comes as no
surprise as the GRU-SVM model did have the relatively most sophisticated architecture design among the presented models, most
notably, its 5-layer design. As explained in [7], the number of layers
of a neural network is directly proportional to the complexity of
a function it can represent. In other words, the performance or
accuracy of a neural network is directly proportional to the number
of its hidden layers. By this logic, it stands to reason that the less
number of hidden layers that a neural network has, the less its
performance or accuracy is. Hence, the findings in this study corroborates the literature explanation as the MLP-SVM came second
(having ≈80.47% test accuracy) to GRU-SVM with a 3-layer design,
and the CNN-SVM came last (having ≈77.23% test accuracy) with a
2-layer design.
The reported test accuracy of ≈84.92% clearly states that the
GRU-SVM model has the strongest predictive performance among
the DL-SVM models in this study. This is attributed to the fact that
the GRU-SVM model has the relatively most complex design among
the presented models. First, its 5-layer design allows it to represent
increasingly complex mappings between features and labels, i.e.
function mappings f : x 7→ y. Second, its capability to learn from
data of sequential nature, in which an image data belongs. This
nature of the GRU-RNN comes from its gating mechanisms, given
by equations in Section 2.4.3. Through the mentioned mechanisms,
the GRU-RNN solves the problem of vanishing gradients and exploding gradients[4]. Thus, being able to connect information with
a considerable gap. However, as indicated by the training summary
given by Figure 2, the GRU-SVM has the caveat of relatively longer
computing time. Having finished its training in 11 minutes and 32
seconds, it was the slowest among the DL-SVM models. From a
high-level inspection of the presented equations of each DL-SVM
model (CNN-SVM in Section 2.4.2, GRU-SVM in Section 2.4.3, and
MLP-SVM in Section 2.4.4), it was a theoretical implication that the
GRU-SVM would have the longest computing time as it had more
non-linearities introduced in its computation. On the other hand,
with the least non-linearities (having only used LeakyReLU), it was
also theoretically implied that the MLP-SVM model would have the
shortest computing time.
From the literature explanation[7] and empirical evidence, it
can be inferred that increasing the complexity of the architectural
design (e.g. more hidden layers, better non-linearities) of the CNNSVM and MLP-SVM models may catapult their predictive performance, and would be more on par with the GRU-SVM model. In
turn, this implication warrants a further study and exploration that
may be prolific to the information security community.

5

CONCLUSION AND RECOMMENDATION

We used the Malimg dataset prepared by [12], which consists of
malware images for the purpose of malware family classification.
We employed deep learning models with the L2-SVM as their final
output layer in a multinomial classification task. The empirical data
shows that the GRU-SVM model by [3] had the highest predictive
accuracy among the presented DL-SVM models, having a test accuracy of ≈84.92%.

Improving the architecture design of the CNN-SVM model and
MLP-SVM model by adding more hidden layers, adding better nonlinearities, and/or using an optimized dropout, may provide better
insights on their application on malware classification. Such insights may reveal an information as to which architecture may
serve best in the engineering of an intelligent anti-malware system.

6

ACKNOWLEDGMENT

We extend our statement of gratitude to the open-source community, especially to TensorFlow. An appreciation as well to Lakshmanan Nataraj, S. Karthikeyan, Gregoire Jacob, and B.S. Manjunath
for the Malimg dataset[12].
We would also like to express our appreciation to friends who
took time reading parts of this paper during its development, and
who gave their support to this endeavor. F.J.H. Pepito gives his
appreciation to Hyacinth Gasmin, and A.F. Agarap gives his appreciation to Ma. Pauline de Ocampo, Abqary Alon, Rhea Jude Ferrer,
and Julius Luis H. Diaz.

REFERENCES
[1] 2017. Deep MNIST for Experts. (Nov 2017). https://www.tensorflow.org/get_
started/mnist/pros
[2] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen,
Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard,
Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike
Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul
Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals,
Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.
2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems.
(2015). http://tensorflow.org/ Software available from tensorflow.org.
[3] Abien Fred Agarap. 2017. A Neural Network Architecture Combining Gated
Recurrent Unit (GRU) and Support Vector Machine (SVM) for Intrusion Detection
in Network Traffic Data. arXiv preprint arXiv:1709.03082 (2017).
[4] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau,
Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase
representations using RNN encoder-decoder for statistical machine translation.
arXiv preprint arXiv:1406.1078 (2014).
[5] C. Cortes and V. Vapnik. 1995. Support-vector Networks. Machine Learning 20.3
(1995), 273–297. https://doi.org/10.1007/BF00994018
[6] Felan Carlo C. Garcia and Felix P. Muga II. 2016. Random Forest for Malware
Classification. arXiv preprint arXiv:1609.07770 (2016).
[7] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT
Press. http://www.deeplearningbook.org.
[8] J. D. Hunter. 2007. Matplotlib: A 2D graphics environment. Computing In Science
& Engineering 9, 3 (2007), 90–95. https://doi.org/10.1109/MCSE.2007.55
[9] Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[10] Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. 2013. Rectifier nonlinearities
improve neural network acoustic models. In Proc. ICML, Vol. 30.
[11] Warren S McCulloch and Walter Pitts. 1943. A logical calculus of the ideas
immanent in nervous activity. The bulletin of mathematical biophysics 5, 4 (1943),
115–133.
[12] Lakshmanan Nataraj, S Karthikeyan, Gregoire Jacob, and BS Manjunath. 2011.
Malware images: visualization and automatic classification. In Proceedings of the
8th international symposium on visualization for cyber security. ACM, 4.
[13] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M.
Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine
Learning in Python. Journal of Machine Learning Research 12 (2011), 2825–2830.
[14] Frank Rosenblatt. 1958. The perceptron: A probabilistic model for information
storage and organization in the brain. Psychological review 65, 6 (1958), 386.
[15] Gary B Shelly and Misty E Vermaat. 2011. Discovering Computers, Complete: Your
Interactive Guide to the Digital World. Cengage Learning.
[16] Yichuan Tang. 2013. Deep learning using linear support vector machines. arXiv
preprint arXiv:1306.0239 (2013).
[17] Stéfan van der Walt, S Chris Colbert, and Gael Varoquaux. 2011. The NumPy
array: a structure for efficient numerical computation. Computing in Science &
Engineering 13, 2 (2011), 22–30.

