arXiv:1711.04598v1 [] 13 Nov 2017

Convolutional neural networks pretrained on large face
recognition datasets for emotion classification from video
Boris Knyazev∗

Roman Shvetsov

NTechLab
Moscow, Russia
b.knyazev@ntechlab.com

NTechLab
Moscow, Russia
r.shvetsov@ntechlab.com

Natalia Efremova

Artem Kuharenko

University of Oxford
Oxford, UK
natalia.efremova@sbs.ox.ac.uk

NTechLab
Moscow, Russia
a.kuharenko@ntechlab.com

ABSTRACT
In this paper we describe a solution to our entry for the emotion
recognition challenge EmotiW 2017. We propose an ensemble of several models, which capture spatial and audio features from videos.
Spatial features are captured by convolutional neural networks,
pretrained on large face recognition datasets. We show that usage
of strong industry-level face recognition networks increases the
accuracy of emotion recognition. Using our ensemble we improve
on the previous best result on the test set by about 1 %, achieving a
60.03 % classification accuracy without any use of visual temporal
information.

1

INTRODUCTION

Emotion recognition potentially has many applications in academia
and industry, and emotional intelligence is an important part of
artificial intelligence. However, in contrast to such tasks as face
recognition (FR), emotion recognition has not yet become so widespread. We believe that the reason for this is the fact that emotion
recognition is much harder and requires more research and efforts
to gain success. Face recognition is also hard, but training data with
clean ground truth labels can be collected easier and benchmarks
are usually objective (i.e. we know the identity). In emotion recognition, there is a lack of understanding and the agreement of what
the labels should be. This can be proved by recent appearance of
datasets with compound emotions [4] or with dominant and complementary emotions [11]. There is also a lack of training data due
to difficulty of collecting rare emotions (how often do you clearly
show fear?).
Emotion recognition from video is also more difficult than general video recognition. State of the art methods such as (Improved)
Dense Trajectories [14] or 3D convolutional neural networks, which
typically show a 70-90 % accuracy on video datasets, fail to provide
results above 40 % on emotion datasets [8, 15].
The emotion recognition challenges, in particular EmotiW 2017
and its predecessors [5, 6], allow to boost the progress in this area
by providing data and benchmarks for training and evaluating
novel methods. Once emotions can be recognized reliably and well
understood, it can provide the same or even more benefits than
face recognition. Due to the presence of concealed emotions it can
lead to even more benefits because humans need expert and rare
∗ Currently

at the University of Guelph.

knowledge to recognize concealed emotions, while machines could
potentially perform this task easily opening up new research areas
as well as new privacy challenges analogously to face recognition.
In this work, we attempt to further contribute to the field of
emotion recognition by presenting our solution to the fifth Emotion
Recognition in the Wild Challenge (EmotiW) 2017, in particular to
its audio-video emotion recognition sub-challenge.
Recent NIST reports show that our FR networks perform as
state-of-the-art or better in most benchmarks [13] and, in this work,
we employ them for emotion recognition by fine-tuning them on
the emotion datasets. Similarly to other video recognition challenges [1], we make features extracted from facial images of all
frames publicly available.

2 EXPERIMENTS
2.1 Networks
We experiment with four convolutional neural networks: VGGFace [12] and three proprietary state of the art face recognition
networks which we notate as FR-Net-A, FR-Net-B and FR-Net-C.
Compared to VGG-Face, which is trained on 2600 individuals with
around 3 million images, our networks are trained on a much larger
data volume (Table 1), which make them more powerful both for
face and emotion recognition tasks.

2.2

Datasets

In this work we use two emotion datasets to train our models
(Table 1): FER2013 [9] and data of this challenge EmotiW 2017.
The FER2013 dataset was also used in the previous year’s winning
method [8]. It consists of 28709 training, 3589 validation and 3589
test images. We fine-tune our models using only a training set.
In the audio-video emotion recognition sub-challenge of Emotiw
2017 there are the same training (773) and validation (383) videos
as in the EmotiW 2016 version, but this year 60 new test videos
were added with 653 in total.

2.3

Pipeline

2.3.1 Face detection. To extract and align faces both from images of FER2013 and EmotiW video frames we use the dlib face
detector [10]. If a face was not found on the frame, the entire frame
is passed to the network. To limit such cases, we apply a low face
detection threshold.

Table 1: Training datasets used in this work. For our face
recognition data we only provide the number of images.
Dataset

#classes

FER 2013
EmotiW 2017

7 emotions
7 emotions

VGG-Face data
Our face recognition data

2600 persons
-

Table 2: Model comparison using frames feature averaging
on the validation set before and after fine-tuning (FT) on
FER2013. Best results in each column are in bold.

#images
Model

35k
50k frames
3m
50m

2.3.2 Frames feature extraction. In all experiments we follow the
same pipeline to obtain results on EmotiW validation videos. First,
features for all frames are computed using all four networks. For
VGG-Face, we empirically choose fc6 features (4096 dimensional),
while for other networks we use 1024 dimensional features of the
layer before the last one.

After FT

# features

VGG-Face
FR-Net-A
FR-Net-B
FR-Net-C

37.9
33.7
33.4
37.6

48.3
44.6
48.8
45.2

4096
1024
1024
1024

Audio features

35.0

-

1582

Table 3: Feature comparison on the validation set for models
fine-tuned on FER2013. STAT* is STAT without max. ** - our
best submission. A - augmentation. Ensemble: VGG-Face +
FR-Net-A+B+C + Audio. Best results in each row are in bold.
Model

2.3.3 Frame-level feature aggregation. Motivated by results of
the previous year [3], we compute features of videos with one
or several aggregation functions (e.g., mean, standard deviation)
(Table 3) followed by rescaling to range [−1, 1] and concatenation
(e.g., in case of VGG-Face if we compute mean and std features, then
featurespare 8192 dimensional). In this work, rootsift normalization
(sign(x) |x|/∥x∥1 ) [2] and global standardization is also applied to
concatenated features.

Mean STAT STAT* STAT*+A STAT*+fft STAT*+fft+A
52.2
47.8
52.5
45.2

52.5
47.5
52.2
45.7

50.4
49.3
52.5
53.0

53.0
48.3
53.3
45.2

50.7
49.6
53.5
52.5

Ensemble 52.7 55.1

54.8

56.4**

56.4

56.7

VGG-Face
FR-Net-A
FR-Net-B
FR-Net-C

2.3.4 Classification. A linear SVM was trained on training data
(one SVM per network) in case of reporting validation accuracy and,
as in [8], on training plus validation data in case of test accuracy. The
regularization constant of SVMs is found by 5-fold cross-validation.

2.5

48.3
44.6
48.8
45.2

Feature comparison

To improve STAT features used in [3], which include mean, standard
deviation, minimum and maximum features we first performed
an ablation study and removed the max features (Table 3). This
appeared to be important for improving generalization on the test
set.
Afterwards, we found that frame-based augmentation in the
form of horizontal flipping, rotation and scaling usually improves
features except for VGG-Face. We compute 18 transformations per
frame and average features of these transformations.
We then tried to add spectral features by computing the 1 dimensional Fourier transform (fft) for each neuron and then taking the
average of that. For instance, for VGG-Face for one video we have
4096 dimensional fft features. These features significantly improved
performance on the validation set (Table 3), however, due to the
limit of test submissions, we were not able to evaluate this and
several other features.

2.3.5 Ensembling with audio features. We compute 1582 dimensional audio features using the Opensmile library [7]. We train a
linear SVM in this case as well, so that our ensemble averages scores
of 5 SVMs in total.

2.4

Before FT

FER2013 fine-tuning

All four networks are fine-tuned on FER2013 by replacing the last
1-2 fully connected layers with new ones (for VGG we only replace
the classification layer) and then training all layers for 30k iterations
with Nesterov SGD with the following parameters: learning rate
0.0001-0.0005, weight decay 0.0005, momentum 0.9, batch size 32
and polynomial learning rate decay. The networks achieved 7072 % accuracy on FER2013 validation data, but we did not pay much
attention to these results, because the relationship between the
FER2013 and validation accuracies was not always straightforward.
As expected, fine-tuning (FT) on FER2013 boosts performance
on EmotiW in all cases (Table 2). Before FT better FR models usually
have worse performance in the emotion recognition task, because
face recognition should be emotion invariant, but after FT the
results invert. For instance, FR-Net-A and FR-Net-B have the same
architecture, but the latter was pretrained on much larger face
recognition data. As a result, FR-Net-B is the worst before finetuning, but the best afterwards. This experiment confirms that
pretraining on larger FR data positively affects emotion recognition
accuracy.

2.6

Class distribution

In this challenge we noticed that fitting the validation set was not
useful or was even harmful, because it differs quite a lot from the
test set. Specifically, the validation set is relatively balanced, i.e. each
emotion has about the same number of samples, while according to
the confusion matrix (provided for submissions) of the test set it is
imbalanced with many more samples of happy, neutral and angry
emotions (Table 5). Therefore, if during fitting of the validation set
the model that predicts all emotions equally well will be chosen,
that would decrease performance on the test set, because on the
test set not all emotions are equally important.
2

Table 4: Comparison of our ensembles with other results. Ensemble: VGG-Face + FR-Net-A+B+C + Audio.
Model
Baseline [6]
Previous year best results

Table 6: For this task the order of frames is not very important, so the videos can be seen as unordered sets of frames.

Val acc

Test acc

Model

38.81
59.42 [3]

40.47
59.02 [8]

LSTM + FR-NET-B
LSTM + FR-NET-B + frame shuffling

54.83
56.40
48.30

54.98
60.03

Ensemble
Ensemble + A
Ensemble + A + class weights

Val acc
46.48
50.39

order is fixed). Our results imply that each video in this task is
not a sequence of video frames, but rather a set of frames. Manual
examination of videos was consistent with our finding, because
after shuffling frames of some video we still could (or still couldn’t)
recognize the correct emotion. LSTMs are trained on top of our
best FR network analogously to [8] on full sequences. However, we
were not able to improve our performance on the test set by adding
LSTMs.

4

CONCLUSIONS

In this work, we present an ensemble of models that achieves better
emotion classification accuracy than the previous year’s winner.
We rely on strong face recognition convolutional networks which
can be easily fine-tuned to perform the emotion recognition task.
Audio features are also used to complement our models with an
additional modality. We make frame level features computed with
our networks publicly available to help the research community by
reducing the task of emotion recognition from video to learning
from high level features.
Figure 1: The confusion matrix of our final predictions on
the test set.

ACKNOWLEDGMENTS
The authors would like to thank other members of the NTechLab
team for useful advice and technical support.

Table 5: Training, validation and test sets class distribution
in EmotiW 2017.

REFERENCES
Train
Validation
Test

An

Di

Fe

Ha

Ne

Sa

Su

Total

133
64
98

74
40
40

81
46
70

150
63
144

144
63
193

117
61
80

74
46
28

773
383
653

Class weights 0.15 0.10 0.13 0.19 0.21 0.14 0.08

[1] Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici,
Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. 2016. YouTube8M: A large-scale video classification benchmark. arXiv preprint arXiv:1609.08675
(2016).
[2] Relja Arandjelović and Andrew Zisserman. 2012. Three things everyone should
know to improve object retrieval. In Computer Vision and Pattern Recognition
(CVPR), 2012 IEEE Conference on. IEEE, 2911–2918.
[3] Sarah Adel Bargal, Emad Barsoum, Cristian Canton Ferrer, and Cha Zhang. 2016.
Emotion recognition in the wild from videos using images. In Proceedings of the
18th ACM International Conference on Multimodal Interaction. ACM, 433–436.
[4] C Fabian Benitez-Quiroz, Ramprakash Srinivasan, Qianli Feng, Yan Wang, and
Aleix M Martinez. 2017. EmotioNet Challenge: Recognition of facial expressions
of emotion in the wild. arXiv preprint arXiv:1703.01210 (2017).
[5] Abhinav Dhall and others. 2012. Collecting large, richly annotated facialexpression databases from movies. (2012).
[6] Abhinav Dhall, Roland Goecke, Jyoti Joshi, Jesse Hoey, and Tom Gedeon. 2016.
Emotiw 2016: Video and group-level emotion recognition challenges. In Proceedings of the 18th ACM International Conference on Multimodal Interaction. ACM,
427–432.
[7] Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. Opensmile: the munich
versatile and fast open-source audio feature extractor. In Proceedings of the 18th
ACM international conference on Multimedia. ACM, 1459–1462.
[8] Yin Fan, Xiangju Lu, Dian Li, and Yuanliu Liu. 2016. Video-based emotion
recognition using cnn-rnn and c3d hybrid networks. In Proceedings of the 18th
ACM International Conference on Multimodal Interaction. ACM, 445–450.
[9] Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi
Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun
Lee, and others. 2013. Challenges in representation learning: A report on three
machine learning contests. In International Conference on Neural Information
Processing. Springer, 117–124.

1

This observation enables us to significantly improve results by
weighting scores of emotions according to the square root of the
observed test set frequencies. As a result, our model minimizes
the presence of disgust and surprise emotions in order to improve
recognition of happy and neutral facial expressions and to achieve
better classification accuracy of 60.03% overall (Table 4).

3

FRAME SHUFFLE AUGMENTATION FOR
LSTM

In this experiment, we trained a Long Short-Term Memory (LSTM)
by randomly shuffling the order of frames during training (Table 6),
which can be seen as a form of augmentation. Although it sounds
counterintuitive, we achieve considerable accuracy gain compared
to training on frames in the original order (during inference the
3

[10] Davis E. King. 2009. Dlib-ml: A Machine Learning Toolkit. Journal of Machine
Learning Research 10 (2009), 1755–1758.
[11] Iiris Lüsi, Julio CS Jacques Junior, Jelena Gorbova, Xavier Baró, Sergio Escalera,
Hasan Demirel, Juri Allik, Cagri Ozcinar, and Gholamreza Anbarjafari. 2017.
Joint challenge on dominant and complementary emotion recognition using
micro emotion features and head-pose estimation: Databases. In Automatic Face
& Gesture Recognition (FG 2017), 2017 12th IEEE International Conference on. IEEE,
809–813.
[12] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and others. 2015. Deep
Face Recognition.. In BMVC, Vol. 1. 6.
[13] Kayee Hanaoka Patrick Grother, Mei Ngan. 2017. Ongoing Face Recognition Vendor Test (FRVT). (2017). https://w3auth.nist.gov/sites/default/files/documents/
2017/08/07/frvt_report_2017_08_07.pdf
[14] Heng Wang and Cordelia Schmid. 2013. Action recognition with improved
trajectories. In Proceedings of the IEEE international conference on computer vision.
3551–3558.
[15] Anbang Yao, Dongqi Cai, Ping Hu, Shandong Wang, Liang Sha, and Yurong Chen.
2016. HoloNet: towards robust emotion recognition in the wild. In Proceedings of
the 18th ACM International Conference on Multimodal Interaction. ACM, 472–478.

4

