An Architecture Combining Convolutional Neural Network
(CNN) and Support Vector Machine (SVM) for Image
Classification
Abien Fred M. Agarap

arXiv:1712.03541v1 [cs.CV] 10 Dec 2017

Department of Computer Science
Adamson University
Manila, Philippines
abien.fred.agarap@adamson.edu.ph

ABSTRACT
Convolutional neural networks (CNNs) are similar to “ordinary”
neural networks in the sense that they are made up of hidden layers
consisting of neurons with “learnable” parameters. These neurons
receive inputs, performs a dot product, and then follows it with a
non-linearity. The whole network expresses the mapping between
raw image pixels and their class scores. Conventionally, the Softmax
function is the classifier used at the last layer of this network.
However, there have been studies [2, 3, 11] conducted to challenge
this norm. The cited studies introduce the usage of linear support
vector machine (SVM) in an artificial neural network architecture.
This project is yet another take on the subject, and is inspired by
[11]. Empirical data has shown that the CNN-SVM model was able
to achieve a test accuracy of ≈99.04% using the MNIST dataset[10].
On the other hand, the CNN-Softmax was able to achieve a test
accuracy of ≈99.23% using the same dataset. Both models were
also tested on the recently-published Fashion-MNIST dataset[13],
which is suppose to be a more difficult image classification dataset
than MNIST[15]. This proved to be the case as CNN-SVM reached
a test accuracy of ≈90.72%, while the CNN-Softmax reached a test
accuracy of ≈91.86%. The said results may be improved if data
preprocessing techniques were employed on the datasets, and if
the base CNN model was a relatively more sophisticated than the
one used in this study.

CCS CONCEPTS
• Computing methodologies → Supervised learning by classification; Support vector machines; Neural networks;

KEYWORDS
artificial intelligence; artificial neural networks; classification; image classification; machine learning; mnist dataset; softmax; supervised learning; support vector machine

1

INTRODUCTION

A number of studies involving deep learning approaches have
claimed state-of-the-art performances in a considerable number of
tasks. These include, but are not limited to, image classification[9],
natural language processing[12], speech recognition[4], and text
classification[14]. The models used in the said tasks employ the
softmax function at the classification layer.
However, there have been studies[2, 3, 11] conducted that takes
a look at an alternative to softmax function for classification – the
support vector machine (SVM). The aforementioned studies have

claimed that the use of SVM in an artificial neural network (ANN)
architecture produces a relatively better results than the use of the
conventional softmax function. Of course, there is a drawback to
this approach, and that is the restriction to binary classification.
As SVM aims to determine the optimal hyperplane separating two
classes in a dataset, a multinomial case is seemingly ignored. With
the use of SVM in a multinomial classification, the case becomes a
one-versus-all, in which the positive class represents the class with
the highest score, while the rest represent the negative class.
In this paper, we emulate the architecture proposed by [11],
which combines a convolutional neural network (CNN) and a linear SVM for image classification. However, the CNN employed in
this study is a simple 2-Convolutional Layer with Max Pooling
model, in contrast with the relatively more sophisticated model and
preprocessing in [11].

2 METHODOLOGY
2.1 Machine Intelligence Library
Google TensorFlow[1] was used to implement the deep learning
algorithms in this study.

2.2

The Dataset

MNIST[10] is an established standard handwritten digit classification dataset that is widely used for benchmarking deep learning
models. It is a 10-class classification problem having 60,000 training
examples, and 10,000 test cases – all in grayscale. However, it is
argued that the MNIST dataset is “too easy” and “overused”, and “it
can not represent modern CV [Computer Vision] tasks”[15]. Hence,
[13] proposed the Fashion-MNIST dataset. The said dataset consists
of Zalando’s article images having the same distribution, the same
number of classes, and the same color profile as MNIST.
Table 1: Dataset distribution for both MNIST and FashionMNIST.
Dataset

MNIST

Fashion-MNIST

Training
Testing

60,000
60,000

10,000
10,000

Both datasets were used as they were, with no preprocessing
such as normalization or dimensionality reduction.

,,

2.3

Abien Fred M. Agarap

Support Vector Machine (SVM)

The support vector machine (SVM) was developed by Vapnik[5] for
binary classification. Its objective is to find the optimal hyperplane
f (w, x) = w · x + b to separate two classes in a given dataset, with
features x ∈ Rm .
SVM learns the parameters w by solving an optimization problem
(Eq. 1).
p
Õ

1
min wT w + C
max 0, 1 − yi′ (wT xi + b)
p
i=1

(1)

where wT w is the Manhattan norm (also known as L1 norm), C
is the penalty parameter (may be an arbitrary value or a selected
value using hyper-parameter tuning), y ′ is the actual label, and
wT x + b is the predictor function. Eq. 1 is known as L1-SVM, with
the standard hinge loss. Its differentiable counterpart, L2-SVM (Eq.
2), provides more stable results[11].
p
Õ
2
1
max 0, 1 − yi′ (wT xi + b)
min ∥w∥22 + C
p
i=1

Figure 1: The Rectified Linear Unit (ReLU) activation
function produces 0 as an output when x < 0, and then
produces a linear with slope of 1 when x > 0.

(2)

(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)

where ∥w∥2 is the Euclidean norm (also known as L2 norm),
with the squared hinge loss.

2.4

Convolutional Neural Network (CNN)

Convolutional Neural Network (CNN) is a class of deep feed-forward
artificial neural networks which is commonly used in computer
vision problems such as image classification. The distinction of
CNN from a “plain” multilayer perceptron (MLP) network is its
usage of convolutional layers, pooling, and non-linearities such as
tanh, siдmoid, and ReLU.
The convolutional layer (denoted by CONV) consists of a filter,
for instance, 5 × 5 × 1 (5 pixels for width and height, and 1 because
the images are in grayscale). Intuitively speaking, the CONV layer
is used to “slide” through the width and height of an input image,
and compute the dot product of the input’s region and the weight
learning parameters. This in turn will produce a 2-dimensional
activation map that consists of responses of the filter at given
regions.
Consequently, the pooling layer (denoted by POOL) reduces the
size of input images as per the results of a CONV filter. As a result,
the number of parameters within the model is also reduced – called
down-sampling.
Lastly, an activation function is used for introducing non-linearities
in the computation. Without such, the model will only learn linear
mappings. The commonly-used activation function these days is
the ReLU function[6] (see Figure 1). ReLU is commonly-used over
tanh and siдmoid for it was found out that it greatly accelerates
the convergence of stochastic gradient descent compared the other
two functions[9]. Furthermore, compared to the extensive computation required by tanh and siдmoid, ReLU is implemented by simply
thresholding matrix values at zero (see Eq. 3).


f hθ (x) = hθ (x)+ = max 0, hθ (x)
(3)
In this paper, we implement a base CNN model with the following
architecture:
(1) INPUT: 32 × 32 × 1
(2) CONV5: 5 × 5 size, 32 filters, 1 stride

ReLU: max(0, hθ (x))
POOL: 2 × 2 size, 1 stride
CONV5: 5 × 5 size, 64 filters, 1 stride
ReLU: max(0, hθ (x))
POOL: 2 × 2 size, 1 stride
FC: 1024 Hidden Neurons
DROPOUT: p = 0.5
FC: 10 Output Classes

At the 10th layer of the CNN, instead of the conventional softmax
function with the cross entropy function (for computing loss), the
L2-SVM is implemented. That is, the output shall be translated to
the following case y ∈ {−1, +1}, and the loss is computed by Eq. 2.
The weight parameters are then learned using Adam[8].

2.5

Data Analysis

There were two parts in the experiments for this study: (1) training
phase, and (2) test case. The CNN-SVM and CNN-Softmax models
were used on both MNIST and Fashion-MNIST.
Only the training accuracy, training loss, and test accuracy were
considered in this study.

3

EXPERIMENTS

The code implementation may be found at https://github.com/AFAgarap/cnnsvm. All experiments in this study were conducted on a laptop computer with Intel Core(TM) i5-6300HQ CPU @ 2.30GHz x 4, 16GB
of DDR3 RAM, and NVIDIA GeForce GTX 960M 4GB DDR5 GPU.
Table 2: Hyper-parameters used for CNN-Softmax and CNNSVM models.
Hyper-parameters

CNN-Softmax

CNN-SVM

Batch Size
Dropout Rate
Learning Rate
Steps
SVM C

128
0.5
1e-3
10000
N/A

128
0.5
1e-3
10000
1

An Architecture Combining Convolutional Neural Network (CNN) and Support Vector Machine (SVM) for Image Classification

,,

The hyper-parameters listed in Table 2 were manually assigned,
and were used for the experiments in both MNIST and FashionMNIST.

Figure 4: Plotted using matplotlib[7]. Training accuracy of
CNN-Softmax and CNN-SVM on image classification using
Fashion-MNIST[13].
Figure 2: Plotted using matplotlib[7]. Training accuracy of
CNN-Softmax and CNN-SVM on image classification using
MNIST[10].

Figure 5: Plotted using matplotlib[7]. Training loss of CNNSoftmax and CNN-SVM on image classification using FashionMNIST[13].
Figure 3: Plotted using matplotlib[7]. Training loss of
CNN-Softmax and CNN-SVM on image classification using
MNIST[10].
Figure 2 shows the training accuracy of CNN-Softmax and CNNSVM on image classification using MNIST, while Figure 3 shows
their training loss. At 10,000 steps, both models were able to finish
training in 4 minutes and 16 seconds. The CNN-Softmax model
had an average training accuracy of 98.4765625% and an average
training loss of 0.136794931, while the CNN-SVM model had an
average training accuracy of 97.671875% and an average training
loss of 0.268976859.
Figure 4 shows the training accuracy of CNN-Softmax and CNNSVM on image classification using MNIST, while Figure 5 shows
their training loss. At 10,000 steps, the CNN-Softmax model was
able to finish its training in 4 minutes and 47 seconds, while the
CNN-SVM model was able to finish its training in 4 minutes and 29
seconds. The CNN-Softmax model had an average training accuracy
of 94% and an average training loss of 0.259750089, while the CNNSVM model had an average training accuracy of 90.15% and an
average training loss of 0.793701683.
After 10,000 training steps, both models were tested on the test
cases of each dataset. As shown in Table 1, both datasets have

Table 3: Test accuracy of CNN-Softmax and CNN-SVM on image classification using MNIST[10] and Fashion-MNIST[13].
Dataset

CNN-Softmax

CNN-SVM

MNIST
Fashion-MNIST

99.23%
91.86%

99.04%
90.72%

10,000 test cases each. Table 3 shows the test accuracies of CNNSoftmax and CNN-SVM on image classification using MNIST[10]
and Fashion-MNIST[13].
The test accuracy on the MNIST dataset does not corroborate
the findings in [11], as it was CNN-Softmax which had a better
classification accuracy than CNN-SVM. This result may be attributed to the fact the there were no data pre-processing than on the
MNIST dataset. Furthermore, [11] had a relatively more sophisticated model and methodology than the simple procedure done in
this study. On the other hand, the test accuracy of the CNN-Softmax
model matches the findings in [15], as both methodology did not
involve data preprocessing of the Fashion-MNIST.

,,

4

Abien Fred M. Agarap

CONCLUSION AND RECOMMENDATION

The results of this study warrants an improvement on its methodology to further validate its review on the proposed CNN-SVM of
[11]. Despite its contradiction to the findings in [11], quantitatively
speaking, the test accuracies of CNN-Softmax and CNN-SVM are
almost the same with the related study. It is hypothesized that with
data preprocessing and a relatively more sophisticated base CNN
model, the results in [11] shall be reproduced.

5

ACKNOWLEDGMENT

An expression of gratitude to Yann LeCun, Corinna Cortes, and
Christopher J.C. Burges for the MNIST dataset[10], and to Han
Xiao, Kashif Rasul, and Roland Vollgraf for the Fashion-MNIST
dataset[13].

REFERENCES
[1] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen,
Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard,
Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike
Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul
Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals,
Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.
2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems.
(2015). http://tensorflow.org/ Software available from tensorflow.org.
[2] Abien Fred Agarap. 2017. A Neural Network Architecture Combining Gated
Recurrent Unit (GRU) and Support Vector Machine (SVM) for Intrusion Detection
in Network Traffic Data. arXiv preprint arXiv:1709.03082 (2017).
[3] Abdulrahman Alalshekmubarak and Leslie S Smith. 2013. A novel approach
combining recurrent neural network and support vector machines for time series
classification. In Innovations in Information Technology (IIT), 2013 9th International
Conference on. IEEE, 42–47.
[4] Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and
Yoshua Bengio. 2015. Attention-based models for speech recognition. In Advances
in Neural Information Processing Systems. 577–585.
[5] C. Cortes and V. Vapnik. 1995. Support-vector Networks. Machine Learning 20.3
(1995), 273–297. https://doi.org/10.1007/BF00994018
[6] Richard HR Hahnloser, Rahul Sarpeshkar, Misha A Mahowald, Rodney J Douglas,
and H Sebastian Seung. 2000. Digital selection and analogue amplification coexist
in a cortex-inspired silicon circuit. Nature 405, 6789 (2000), 947–951.
[7] J. D. Hunter. 2007. Matplotlib: A 2D graphics environment. Computing In Science
& Engineering 9, 3 (2007), 90–95. https://doi.org/10.1109/MCSE.2007.55
[8] Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[9] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information
processing systems. 1097–1105.
[10] Yann LeCun, Corinna Cortes, and Christopher JC Burges. 2010. MNIST handwritten digit database. AT&T Labs [Online]. Available: http://yann. lecun. com/exdb/mnist 2 (2010).
[11] Yichuan Tang. 2013. Deep learning using linear support vector machines. arXiv
preprint arXiv:1306.0239 (2013).
[12] Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-Hao Su, David Vandyke,
and Steve Young. 2015. Semantically conditioned lstm-based natural language
generation for spoken dialogue systems. arXiv preprint arXiv:1508.01745 (2015).
[13] Han Xiao, Kashif Rasul, and Roland Vollgraf. 2017. Fashion-mnist: a novel
image dataset for benchmarking machine learning algorithms. arXiv preprint
arXiv:1708.07747 (2017).
[14] Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander J Smola, and Eduard H Hovy. 2016. Hierarchical Attention Networks for Document Classification..
In HLT-NAACL. 1480–1489.
[15] Zalandoresearch. 2017. zalandoresearch/fashion-mnist. (Dec 2017). https://
github.com/zalandoresearch/fashion-mnist

