Semi-supervised Fisher vector network

arXiv:1801.04438v1 [cs.CV] 13 Jan 2018

Petar Palasek, Ioannis Patras
School of Electrical Engineering and Computer Science
Queen Mary University of London
London E1 4NS, United Kingdom
p.palasek@qmul.ac.uk, i.patras@qmul.ac.uk

Abstract

Defining such a procedure would allow us to optimize our
network using mini-batch gradient descent based methods,
the usual methods used for training deep learning architectures. As the PCA dimensionality reduction step is optional
in the Fisher vector pipeline, we will ignore it and first focus
on defining a method that learns a Gaussian mixture model
incrementally.
The final and the main goal of this work is to propose a
method which allows incorporating unlabeled data into the
process of the Fisher vector pipeline training, i.e. a method
for learning a semi-supervised Fisher vector encoding of
the input data. To test how the proposed method works
on real data, we choose to apply it at an image classification problem; classifying tiny RGB images from the Cifar10 [5] dataset into 10 classes, while artificially varying the
amount of available labeled and unlabeled data. We also
show how the same method can be used in an action recognition problem by running similar experiments on the UCF101 dataset.
The main contributions of this work can be summarized
as follows:

In this work we explore how the architecture proposed
in [8], which expresses the processing steps of the classical Fisher vector pipeline approaches, i.e. dimensionality
reduction by principal component analysis (PCA) projection, Gaussian mixture model (GMM) and Fisher vector descriptor extraction as network layers, can be modified into a
hybrid network that combines the benefits of both unsupervised and supervised training methods, resulting in a model
that learns a semi-supervised Fisher vector descriptor of the
input data. We evaluate the proposed model at image classification and action recognition problems and show how the
model’s classification performance improves as the amount
of unlabeled data increases during training.

1. Introduction
The work of [8] has shown how the standard Fisher vector pipeline for classification consisting of a local feature
extraction step, a PCA dimensionality reduction step, a step
where a Gaussian mixture model is learned and the final
step of training a classifier can all be viewed as a single
neural network, allowing finetuning of the model in a supervised, end-to-end fashion. However, even when all the
mentioned steps were defined as network layers, the finetuning could start only after the layers had been initialized with
parameters learned using offline batch learning procedures
that require access to the whole training set at once. That
is, the initialization consisted of running PCA on the whole
training set, followed by using expectation-maximization to
fit to data of lower dimensionality with a GMM and then
training an SVM using Fisher vectors extracted using the
GMM. After that the network was ready for finetuning.
What we are interested in exploring in this work are
methods that would give us a way of training all the parameters in the Fisher vector pipeline in an online fashion, this
way becoming suitable for applications where the training
data is not available in advance, but is arriving in batches.

• We describe a method for fitting Gaussian mixture
models that can be used when the training data is not
available all at once, but it is arriving in mini-batches,
i.e. the data is arriving in small subsets of the training set. The same method can be used for training
of the GMM defined as a network layer, described in
[8]. Also, the method can be run on a GPU, leading to
smaller training times compared to methods that can
only run on CPUs.
• We define a method that leverages the availability
of unlabeled data for training of an improved, semisupervised version of the Fisher vector encoding. Similarly as in [8], we define a network that extracts the
Fisher vector descriptor of the input data, which we
name the semi-supervised Fisher vector network. We
perform a number of experiments which show how
increasing the amount of unlabeled data helps im1

prove the classification performance of our model at
the problem of image classification (on CIFAR-10) and
action recognition (on UCF-101).
We start by describing the problem of Gaussian mixture
model fitting.

2. Gaussian mixture model training
We are faced with a problem of estimating a probability
distribution of a random variable X for which we assume
that it depends on another random variable Z, whose values
we cannot observe, i.e. Z is a hidden random variable. If
we parameterize the joint distribution of the two variables
with a set of parameters θ, the distribution of X can then be
written by marginalizing over Z as
X
p(x|θ) =
p(x, z|θ).
(1)
z

Finding the maximum likelihood estimate parameters of
the model above, that is finding the θ which maximizes
L(θ) = log p(x|θ), cannot be done as a direct maximum
δ
L(θ) to
likelihood estimation by setting the derivatives δθ
zero, as this leads to entangled equations without a closed
form solution. However, the parameters can be found
in the case when the posterior distribution p(z|x, θ) is
known. This observation lead to the idea of the expectationmaximization (EM) algorithm [3] which offers a way of
finding maximum likelihood parameters using an iterative
approach.
Starting from some initial estimate of the parameters θ0 ,
the EM algorithm works by iteratively performing two steps
at each time step t; the E-step which determines the posterior distribution p˜t (z) = p(z|x, θt−1 ) using the parameter estimates from the previous time step θt−1 , and the
M-step which finds new estimates of the parameters θt as
the ones that maximize the expected value of the log likelihood L(θ) under the estimated posterior distribution from
the E-step, p˜t . That is, θt is set to the θ that maximizes
Ep˜t [log p(x, z|θ)]. This procedure is guaranteed to never
worsen the log likelihood of the data [3].
Following the work of [6], we can define a function
F (p̃, θ) whose value is maximized or at least increased by
both the E-step and the M-step of the EM algorithm as
F (p̃, θ) = Ep̃ [log p(x, z|θ)] + H(p̃),

(2)

H(p̃) = −Ep̃ [log p̃(x)] .

(3)

with
This function can also be written in terms of a KullbackLiebler divergence between p̃(z) and pθ (z) = p(z|x, θ) as:
F (p̃, θ) = −D(p̃||pθ ) + L(θ),

(4)

where
D(p̃||pθ ) =

X

p̃(z) log

z

p̃(z)
pθ (z)

(5)

is the KL divergence and L(θ) is the log-likelihood, L(θ) =
log p(x|θ). It was shown in [6] that the local and global
maxima of F (p̃, θ) are at the same time the local and global
maxima of L(θ), so maximizing F (p̃, θ) also leads to the
maximum likelihood estimate parameters θ which we want
to find. When the objective function is defined as in Equation 2 the E-step of the EM algorithm maximizes F (p̃, θ)
with respect to p̃ and the M-step maximizes it with respect
to θ.

3. Online Gaussian mixture model training
In the case when the expression Ep˜t [log p(x, z|θ)]
which we wish to optimize in the maximization step of
the EM algorithm is not completely maximized, but only
moved towards its maximum, the procedure will still result
in improvements of the data likelihood [3]. Algorithms that
perform the maximization step only partially are called generalized expectation-maximization (GEM) algorithms and
are the type of algorithms that we are interested in for our
problem.
As our goal is to have a way of training a GMM incorporated in a neural network as a layer, we want to be
able to use the usual, gradient based optimization methods
for learning the GMM’s parameters. The approach which
we will take for implementing the M-step will be to simply calculate the gradients of the function defined in Equation 4 with respect to all of the GMM parameters and perform a number of gradient ascent steps to move the value of
F (p̃, θ) toward its maximum.
Here we define the Gaussian mixture model as a sum of
K weighted Gaussians:
uθ (x) =

K
X

wk uk (x),

(6)

k=1

with

1
0 −1
− (x − µk ) Σk (x − µk )
uk (x) =
D
1 exp
2
(2π) 2 |Σk | 2
(7)
being the probability density function of a single Gaussian
distribution. In order to meet the constraints that the weights
wk need to be positive and sum up to one, we rewrite wk in
k)
. The
terms of internal weights αk as wk = Pexp(α
K
l exp(αl )
posteriors or responsibilities describing how likely it is that
a sample x was generated by the k-th mixture component
can be calculated as:
1



wk uk (x)
γ(k) ≡ p(Z = zk |x) = PK
.
l wl ul (x)

(8)

We represent the set of the GMM parameters as θ =
{αk , µk , Σk , k = 1, ..., K}, where αk is the k-th internal
component weight, µk is its mean vector and Σk its covariance matrix. We will denote the set of GMM parameters at
iteration t as θt .
Finally, we will describe the steps of the generalized
mini-batch EM algorithm which we use in this chapter.
Given a mini-batch of training samples, i.e. only a small
subset of the available training data, with t denoting the iteration of the algorithm, we can summarize the algorithm
as follows:
• E-step: Maximize F (p̃t , θt−1 ) with respect to p̃t by
setting p˜t (zk ) ← γt−1 (k), where γt−1 (k) is the posterior distribution over all the samples in the current
mini-batch, calculated using the parameters θt−1 from
the previous time step, t − 1.
• M-step: Repeat nM times:
– Calculate the gradients of F (p̃t , θt−1 ) with respect to each of the parameters θt−1 in θt−1 ;
∇θt−1 F (p̃t , θt−1 ).
– Update each θt in θt using a gradient based
method, e.g. θt ← θt−1 + λ∇θt−1 F (p̃t , θt−1 ).
Note that the gradient based optimization method used in
the M step is not limited to gradient ascent, but other methods such as Adagrad [4] or RMSProp [11] could be used
too.

3.1. Experiments on a 2D toy dataset
In order to evaluate the performance of the method proposed above for online training of Gaussian mixture models, we first use an implementation of the standard batch
expectation-maximization algorithm [3] from the scikitlearn [9] library and apply it to a toy problem in which
we want to model a distribution of a set of points in twodimensional space.
To generate the toy dataset we first randomly pick
K means, µk , and K standard deviations, σk , and use
them to parameterize a set of K normal distributions
{N (µk , σk2 ), k ∈ [0, K − 1]}. We then define a multinomial distribution parameterized by {π0 , . . . , πK−1 }, with
PK−1
k=0 πk = 1, where each πk represents the probability of
picking the k-th normal distribution for sampling. We finally generate a set of n two-dimensional vectors by choosing one of the K normal distributions according to the
multinomial distribution and sampling from it. This results in data grouped in K random clusters whose means
and variances are known. By using the described procedure
we can generate arbitrary many random point distributions,
allowing us to compare the performance of our method to
scikit-learn’s implementation of the batch EM algorithm.

Examples of the generated toy datasets are shown in Figures 1a and 2a.
3.1.1

Parameter initialization

To be able to directly compare the two methods we run both
of them from the same parameter initialization. We use the
k-means++ algorithm [2], a method used for initializing the
locations of centroids for the k-means algorithm, to initialize the means µk for each of the Gaussians in our mixture
model. We initialize each of the K component weights wk
to 1/K and set the standard deviations σk to fixed values.
3.1.2

Results

We generated a number of random 2D toy datasets consisting of 10 000 training and 10 000 testing samples from
32 different Gaussian distributions. We show the generated
datasets in the top left corners of Figures 1, 2 and 3. We then
use the implementation of the batch EM algorithm from the
scikit-learn library to fit a GMM with 32 components, starting from the intialization described above. The training was
stopped after the improvement of the training data log likelihood between two iterations fell below a certain threshold
value.
The results acquired using the batch EM algorithm are
shown in top right corners of Figures 1, 2 and 3. After that
we run the version of the EM algorithm we described in
Section 3 using different gradient based optimization methods, different learning rates and different number of M-step
iterations. We run the training for a fixed number of iterations. In the first two experiments the number of samples
given in a mini-batch was the same as the total number of
available training samples. The results of running the proposed method are shown in the bottom row of Figures 1,
2. In the last experiment we use a mini-batch size of 500
and show the results in the bottom row of Figure 3. We
also report the log likelihood of the testing data in each of
the experiments below the figures. It can be seen that the
learned Gaussians identified most of the groundtruth clusters. The proposed algorithm also learned some Gaussians
with a small variance, which also increased the average likelihood. The values of the variances during training were
limited to not fall bellow a predefined value.

4. Semi-supervised Fisher vector encoding
In this section we will define a method for leveraging
unlabeled data in the Fisher vector classification pipeline.
An illustration of the architecture that we describe in this
chapter is shown in Figure 4.
Having a method for training Gaussian mixture models
with a gradient based optimization method which we described in the previous section (Section 3) gives us a natural

(a) Groundtruth data generated (b) GMM learned using scikitby sampling from 32 Gaussians. learn batch EM implementation.
Mean log p(xtest ) =
−13.48343.

(a) Groundtruth data generated (b) GMM learned using scikitby sampling from 32 Gaussians. learn batch EM implementation.
Mean log p(xtest ) =
−14.46651.

(c) GMM learned using the pro- (d) GMM learned using the proposed algorithm with Adagrad, posed algorithm with Nesterov
1 000 M-step iterations, learning momentum, 1 000 M-step iterarate 1. Mean log p(xtest ) = tions, learning rate 10. Mean
−10.9322.
log p(xtest ) = −10.9354.

(c) GMM learned using the pro- (d) GMM learned using the proposed algorithm with Nesterov posed algorithm with RMSProp,
momentum, 1 000 M-step itera- 1 000 M-step iterations, learning
tions, learning rate 10. Mean rate 0.01. Mean log p(xtest ) =
log p(xtest ) = −11.76504.
−11.80651.

Figure 1: Experiments on the 2D toy dataset. The first
subfigure shows the generated dataset, the second subfigure shows the GMM learned using scikit-learn’s EM implementation, the third subfigure shows the GMM learned using the proposed EM algorithm with Adagrad and the fourth
subfigure shows the GMM learned using the proposed EM
algorithm with Nesterov mementum. The training set contained 10 000 samples and the mini-batch size in the last
two experiments was set to 10 000.

Figure 2: Experiments on the 2D toy dataset. The first
subfigure shows the generated dataset, the second subfigure shows the GMM learned using scikit-learn’s EM implementation, the third subfigure shows the GMM learned
using the proposed EM algorithm with Nesterov momentum and the fourth subfigure shows the GMM learned using
the proposed EM algorithm with RMSProp. The training
set contained 10 000 samples and the mini-batch size in the
last two experiments was set to 10 000.
loss also used in [8]:
Cunsup ≡ F (x, p̃, θ) = −

X

p̃(z) log

z

way to build a hybrid network that is trained with the goal
of optimizing two different objective functions; an unsupervised objective function and a supervised objective function. The unsupervised objective function is the one whose
goal is to fit a Gaussian mixture model to the given data
and the supervised objective function would be the one that
forces the network to output the correct label given a training sample. We already have both of the needed functions
defined - the unsupervised objective function, Cunsup is
F (p̃, θ) from Equation 2 and the supervised objective function, which we will denote as Csup , is the squared hinge

p̃(z)
+log p(x|θ),
pθ (z)
(9)

and
m

Csup ≡ C(x, y, w, b) =

X
λ
2
||w||2 +
max (0, 1 − yj · sj ) ,
2
j

(10)
where s = xwT + b is the SVM score, sj denotes the j-th
element of s and y is the label l of sample x encoded as
a vector where, in case of dealing with m classes, m − 1
elements are set to -1 and a single element at position l is
set to 1. After this, we can define the combined objective

(a) Groundtruth data generated (b) GMM learned using scikitby sampling from 32 Gaussians. learn batch EM implementation.
Mean log p(xtest ) =
−13.48343.

beled and unlabeled data, at each step taking a mini-batch
from the set. Each chosen mini-batch can either consist of
a mix of labeled and unlabeled samples, all the samples can
be unlabeled or all the samples can be labeled. For each
mini-batch we first perform the E-step of the EM algorithm
described in Section 3, setting p̃ to the one that minimizes
−λ · Cunsup while keeping the other parameters fixed. This
is followed by nM M-steps in which all the GMM parameters θ from θ are updated using a gradient-based optimization method. Finally, we update all the parameters of both
the GMM and the SVM layers by taking gradients of the
combined cost, Chybrid , with respect to each of the parameters and moving towards its minimum again using a
gradient-based optimization method. In the case when a
sample is unlabeled, its supervised part of the cost is set to
0. We evaluate the proposed method in the following section.

5. Experiments and results

(c) GMM learned using the pro- (d) GMM learned using the
posed algorithm with Adagrad, proposed algorithm with Ada100 M-step iterations, learn- grad, 10 M-step iterations, learning rate 1.
log p(xtest ) = ing rate 1.
log p(xtest ) =
−11.03016.
−11.14458.

Figure 3: Experiments on the 2D toy dataset. The first
subfigure shows the generated dataset, the second subfigure shows the GMM learned using scikit-learn’s EM implementation, the third and the fourth subfigures show the
GMM learned using the proposed EM algorithm with Adagrad, but with different numbers of M-step iterations. The
training set contained 10 000 samples and the mini-batch
size in the last two experiments was set to 500.

In this section we report how we evaluated the proposed
semi-supervised Fisher vector network on the problems of
image classification and action recognition.

5.1. Image classification on CIFAR-10
The CIFAR-10 dataset [5] is a standard image classification dataset containing small 32 × 32 px images from 10
different categories. In our experiments the images are preprocessed by subtracting the mean calculated on the training
set from each image.
The architecture that we will use for the experiments we
run on the CIFAR-10 dataset is shown in Figure 4, where the
local feature extraction layers were replaced by the layers
taken from the VGG-16 network [10], pretrained on the ImageNet dataset. In each experiment we will specify which
layer is used as the input layer to the GMM layer.

cost as
Chybrid = Csup − λ · Cunsup ,

(11)

where λ is a weighting coefficient. We subtract the unsupervised cost in order to turn its optimization from a maximization into a minimization problem.
Trying to optimize the local feature extraction layers’
weights jointly with the GMM and SVM layers is not
straightforward as the feature extraction layers’ weights
might collapse to zeros, leading to a trivial but not meaningful solution that minimizes the unsupervised part of the
cost. Here we focus only at the GMM and SVM layers and
keep the parameters of the local feature extraction layers
fixed. The training of the proposed model can in this case
be done as follows.
We loop through the training set consisting of both la-

5.1.1

Architecture without a GMM layer

The first experiments that we run are with an architecture
that does not contain a GMM layer, that is, the outputs of the
VGG-16 network layer are fed straight into the SVM layer.
First we keep the VGG layer weights fixed and only train
the SVM layer for a fixed number of epochs. We then try
training the whole network using the SVM cost (Equation
10). We check how adding an additional convolutional layer
with 512 1 × 1 px filters affects the accuracy. We also try
adding an additional fully connected layer with 16416 units.
We use different layers of VGG-16 as the input to the SVM
layer. The results are shown in Table 1.

Input data

Local feature
extraction layers

Fisher vector
descriptor layer

Classification layer

Gaussian mixture
model layer

Predicted label

Network input

Extracted feature
maps

Posterior
tensor

Fisher vector
descriptor

Network output

Figure 4: An illustration of the proposed architecture. It is a simplified version of the network proposed in [8], without the
spatio-temporal pooling layer and the dimensionality reduction layer. For example, we can think of the network input as an
RGB image that is fed through local feature extraction layers that give some feature maps as output. The ”pixels” of the
feature maps are modeled with a GMM layer that outputs a tensor of posteriors for each ”pixel” in the given feature map.
Feeding the extracted feature maps and the posterior tensor into the Fisher vector descriptor layer gives the FV descriptor as
an output. The FV descriptor is then fed into the classification layer that outputs a class prediction for the image given at the
input of the network.
Table 1: Classification results on the CIFAR-10 test set using different outputs of the VGG-16 network directly fed into the
SVM layer. We train the network using RMSProp for 100 epochs.
Output layer
conv 3 3
conv 3 3
conv 3 3 + conv layer
conv 3 3 + FC layer
conv 4 3
conv 5 3

5.1.2

Output shape
(128, 256, 8, 8)
(128, 256, 8, 8)
(128, 32768)
(128, 16416)
(128, 512, 4, 4)
(128, 512, 2, 2)

Architecture with a GMM layer trained offline

In these experiments we pass images from the training set
through the VGG network and model the outputs of its different layers with a GMM trained using an implementation
of the EM algorithm from the scikit-learn library. Once the
GMM parameters have been learned, we extract Fisher vector representations of the input images and use them to train
a classifier by optimizing the SVM squared hinge loss. We
use mini-batch gradient descent with momentum to do the
training for 100 epochs, with the learning rate set to 0.001,
momentum 0.9 and mini-batch size of 128. The number of
GMM components was 64.

Parameters trained
SVM
all
all
all
all
all

5.1.3

Accuracy
69.96%
81.97%
81.99%
82.33%
85.44%
86.39%

Architecture with a GMM layer trained using the
proposed hybrid objective function

In these experiments we will evaluate the proposed method
for training a semi-supervised Fisher vector network, as described in Section 4. We are given a training set consisting
of a fixed number of labeled samples and we vary the number of unlabeled data throughout the experiments. We start
by initializing the GMM layer using the k-means++ algorithm [2], as it was done in Section 3.1.1. Only the labeled
data is used to do the initialization.
The optimization method we use for the M-step is Adagrad with the initial learning rate set to 1. The updates with
respect to the supervised objective function are done using
RMSProp using an initial learning rate that we mention for

Table 2: Classification results on the CIFAR-10 test set using different outputs of the VGG-16 network. The GMM layer
contained 64 components and was trained using scikit-learn’s implementation of the batch EM algorithm. The SVM was
trained using gradient descent with momentum for 100 epochs.
Output layer
pool 5
conv 5 3
conv 4 3
conv 3 3
conv 3 3

Output shape
(128, 512, 1, 1)
(128, 512, 2, 2)
(128, 512, 4, 4)
(128, 256, 8, 8)
(128, 256, 8, 8)

FV size
(128, 65600)
(128, 65600)
(128, 65600)
(128, 32832)
(128, 32832)

each of the experiments.
In the first experiment on the CIFAR-10 dataset we fix
the number of labeled samples to 10 000 and change the
amount of unlabeled samples from 0, 1 000, 5 000, 10 000,
20 000 to 40 000. The learning rate is set to 0.0001 and
we perform 5 M-step iterations for each mini-batch. The
training charts for this experiment can be seen in Figure 5.
In the second experiment on the CIFAR-10 dataset we
fix the number of labeled samples to 1 000 and change the
amount of unlabeled samples from 0, 1 000, 5 000, 10 000,
20 000 to 49 000. The learning rate is set to 0.0001 and we
perform only a single M-step iteration for each mini-batch.
The training charts for this experiment can be seen in Figure
6. In all of the mentioned experiments the coefficient λ was
set to 0.1.

5.2. Action recognition on UCF-101
In order to evaluate how the semi-supervised Fisher vector network performs on an action recognition problem
when the amount of unlabeled data is being increased we do
the following experiments. We use the representation used
in [7], where instead of performing dense sampling as in
[8], only random subvolumes are extracted from a video for
calculating the FV. We use the layer conv5 3 from the VGG16 network as the feature extraction layer. We extract 1 000
subvolumes from each video and pool them spatially and
temporally as illustrated in the mentioned figure. As we are
not interested in learning the PCA dimensionality reduction
mapping, we use the PCA learned in the experiments from
[8] to lower the dimensionality of each extracted subvolume
to 100. We then end up with 1 000 feature vectors of size
100 for each video, which is what we feed into our GMM
layer. The rest of the training process is the same as with
the experiments we performed on the CIFAR-10 dataset.
We also start by initializing the GMM using the k-means++
algorithm on the labeled part of the training data. The optimization method used in the M-step is Adagrad with the
initial learning rate set to 1, and the optimization method
used to update the parameters with respect to the supervised
objective function is RMSProp, as it was in the CIFAR-10
experiments.
We start by using 1 000 labeled samples and a varying

Parameters trained
SVM
SVM
SVM
SVM
all

Accuracy
53.79%
57.10%
72.48%
77.73%
86.03%

amount of unlabeled samples (0, 1 000, 5 000 and 8 500)
to train the semi-supervised Fisher vector network with 64,
128 and 256 GMM components using the proposed method
for 100 epochs. We show the effect of using different
amount of unlabeled data for the network with 64 GMM
components in 7. Very similar behavior was observed when
128 or 256 GMM components were used, so we do not include the corresponding figures. In all of the mentioned
experiments the coefficient λ was set to 0.1. We show the
achieved classification performance from these experiments
in Table 3.
Table 3: Classification results on the UCF-101 test set using
semi-supervised Fisher vector networks with different numbers of GMM components. The number of labeled samples
was fixed to 1 000, λ = 0.1 and the training ran for 100
epochs.
Number unlabeled
samples

0
1 000
2 000
5 000
8 500

64 GMM
components

128 GMM
components

256 GMM
components

42.21%
42.65%
45.65%
45.76%
46.41%

41.76%
43.73%
43.78%
45.76%
46.16%

42.62%
43.51%
44.59%
47.03%
46.32%

In the next experiment we chose to train the GMM component of the network offline, using the standard batch EM
algorithm using both unlabeled and labeled data, and then
we finetune the network using the labeled data only. We do
this in order to be able to compare the achieved performance
to the performance of the proposed semi-supervised training that updates all the network parameters at the same time,
in an online fashion. The results shown in Table 4 demonstrate that the performance of the network where the GMM
part was trained offline almost always surpasses the performance of the network trained in an online fashion. This is
an expected result as the batch EM algorithm has access to
the information from the whole training set at each step of
the training. On the other hand, our online version can only
make updates to the network parameters based only on the
information contained in the current mini-batch.

Table 4: Classification results on the UCF-101 test set using semi-supervised Fisher vector networks with different
numbers of GMM components. The GMMs are trained offline using labeled and unlabeled data and the networks are
finetuned using the 1 000 labeled samples. The training ran
for 100 epochs.
Number unlabeled
samples

0
1 000
2 000
5 000
8 500

64 GMM
components

128 GMM
components

256 GMM
components

46.70%
46.78%
46.08%
47.41%
45.94%

46.08%
46.05%
45.67%
46.83%
46.19%

43.76%
44.16%
44.11%
44.64%
44.70%

In order to see how much information is gained by increasing the available number of labeled samples compared
to when the number of unlabeled samples is increased, we
run a new set of experiments in which we do not use any
unlabeled data, we only modify the amount of available labeled data. The results of this experiment are shown in Table 5. When comparing to the results of the previous experiment shown in Table 3 we can see that we only needed
to add on average 600 labeled samples to surpass the performance of the semi-supervised Fisher vector network that
was trained with 1 000 labeled and 8 500 unlabeled samples.
Table 5: Classification results on the UCF-101 test set using
the semi-supervised Fisher vector network trained without
any unlabeled data and different amounts of labeled data.
Number labeled
samples

Test
accuracy

1 000
1 200
1 400
1 600

42.21%
42.47%
44.67%
47.12%

In order to see how the performance of the proposed network changes as a function of the mini-batch size used during training, we run a set of experiments where we trained
the network in a semi-supervised way while changing the
size of the mini-batch. We trained the network using minibatch size 1, 5, 50, 100, 250 and 500. We show the results in
Table 6. It can be seen that, out of the different mini-batch
sizes we have used for the experiment, the best performance
is achieved using a mini-batch size of 50.

6. Discussion and conclusion
In the first experiments (Table 1) we have shown how
training an SVM using different output layers of the VGG16 network affects the classification accuracy evaluated on
the CIFAR-10 dataset. We can notice that using higher lay-

Table 6: Classification results on the UCF-101 test set using the semi-supervised Fisher vector network trained using
different mini-batch sizes. The numbers of both labeled and
unlabeled samples were fixed to 1 000, we used 64 GMM
components and the learning rate was set to 0.001.
Mini-batch
size

Test
accuracy

1
5
50
100
250
500

12.24%
37.39%
46.31%
42.65%
39.44%
32.29%

ers improved the performance as these layers capture more
high-level features making it easier for the SVM to discriminate between the classes. We have also shown that finetuning the whole network leads to significant improvements
in the performance. This is expected as the VGG-16 network that we use was pretrained on a different dataset (ImageNet), so the weights are not tailored for the CIFAR-10
dataset.
In the experiments where a Fisher vector encoding was
used (Table 2) we showed that having feature maps of larger
spatial size results in better performance. This is because
the Fisher vector combines local features into a global descriptor and having feature maps with an image structure
provides more local information than when the whole image is represent as only a single vector as it is the case when
e.g. the pool 5 layer is used.
As can be observed in Figure 5, where the proposed hybrid objective function was used to train the network for the
image classification problem on CIFAR-10, increasing the
amount of unlabeled samples resulted in increased classification performance. The highest test accuracy was achieved
when 40 000 unlabeled samples were used (yellow line).
The same can be seen in Figure 7 and Table 3, where different amounts of unlabeled data were used to train the network for the action recognition problem on UCF-101. The
highest test accuracy in this case was also achieved when
the largest number of unlabeled samples was used during
training. The effect of using a small number of M-step iterations in our algorithm are shown in Figure 6 where only a
single iteration is used. It can be seen that the improvement
in the training is not as stable as when more iterations are
used as shown in e.g. Figure 5 and Figure 7 (both using 5-M
step iterations).
In Table 4 we show that the performance of the network
when the GMM part of it is trained offline surpasses the performance of the network trained using our semi-supervised
online approach. As already mentioned, this is because the
batch EM-algorithm used to train the GMM offline has ac-

cess to all the data during each EM iteration, while our online algorithm only sees a single mini-batch at a time. We
note that the batch EM cannot be used in all situations - all
data might not be available immediately, or it might be impossible to apply the batch algorithm because of memory
constraints. Our proposed training method does not have
this problem.
We tried matching the performance of the network
trained in a semi-supervised way with a network trained
only with labeled data in an experiment shown in Table
5. We show that only 600 additional labeled samples
were needed to match the performance gained from adding
8 500 unlabeled samples. We checked how the performance changes as a function of training time mini-batch
size, shown in Table 6. It can be seen that the smallest
mini-batch size resulted in the worst performance, as a single sample does not bring much information about the distribution being modeled. If the mini-batch is too large the
performance again degrades, as it is often observed when
training neural networks.
To conclude, we presented a novel semi-supervised
Fisher vector network and showed can be applied on an
image classification problem (on CIFAR-10) and an action
recognition problem (on UCF-101). We believe that we
have shown some promising results and further experiments
should be performed to see how well the semi-supervised
Fisher vector network would perform on the whole UCF101 dataset when unlabeled data is added from a larger
dataset, e.g. the YouTube-8M dataset [1].

References
[1] S. Abu-El-Haija, N. Kothari, J. Lee, P. Natsev, G. Toderici,
B. Varadarajan, and S. Vijayanarasimhan. Youtube-8m: A
large-scale video classification benchmark. arXiv preprint
arXiv:1609.08675, 2016. 9
[2] D. Arthur and S. Vassilvitskii. k-means++: The advantages
of careful seeding. In Proceedings of the eighteenth annual
ACM-SIAM symposium on Discrete algorithms, pages 1027–
1035. Society for Industrial and Applied Mathematics, 2007.
3, 6
[3] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum
likelihood from incomplete data via the EM algorithm. Journal of the royal statistical society. Series B (methodological),
pages 1–38, 1977. 2, 3
[4] J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization.
Journal of Machine Learning Research, 12(Jul):2121–2159,
2011. 3
[5] A. Krizhevsky. Learning multiple layers of features from
tiny images. 2009. 1, 5
[6] R. M. Neal and G. E. Hinton. A view of the EM algorithm that justifies incremental, sparse, and other variants.
In Learning in graphical models, pages 355–368. Springer,
1998. 2

[7] P. Palasek and I. Patras. Action recognition using convolutional restricted Boltzmann machines. In Proceedings of the
1st International Workshop on Multimedia Analysis and Retrieval for Multimodal Interaction, MARMI ’16, pages 3–8,
New York, NY, USA, 2016. ACM. 7
[8] P. Palasek and I. Patras. Discriminative convolutional
fisher vector network for action recognition. arXiv preprint
arXiv:1707.06119, 2017. 1, 4, 6, 7
[9] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel,
B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss,
V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau,
M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011. 3
[10] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. In International
Conference on Learning Representations, 2015. 5
[11] T. Tieleman and G. Hinton. Lecture 6.5-rmsprop: Divide
the gradient by a running average of its recent magnitude.
COURSERA: Neural networks for machine learning, 4(2),
2012. 3

Figure 5: Training of the semi-supervised Fisher vector network on CIFAR-10 using 10 000 labeled samples and changing
the number of unlabeled samples. The training was run for 30 epochs, the learning rate was 0.0001 and we used 5 M-step
iterations.

Figure 6: Training of the semi-supervised Fisher vector network on CIFAR-10 using 1 000 labeled samples and changing the
number of unlabeled samples. The training was run for 30 epochs, the learning rate 0.0001 and we used only a single M-step
iteration.

Figure 7: Training of the semi-supervised Fisher vector network on UCF-101, 64 GMM components, using 1000 labeled
samples and changing the number of unlabeled data. Learning rate 0.001, 5 M-steps, λ = 0.1.

