arXiv:1709.05675v1 [cs.CV] 17 Sep 2017

Organizing Multimedia Data in Video
Surveillance Systems Based on Face Verification
with Convolutional Neural Networks
Anastasiia D. Sokolova, Angelina S. Kharchevnikova, Andrey V. Savchenko
National Research University Higher School of Economics, Nizhny Novgorod, Russian
Federation
adsokolova96@mail.ru

Abstract. In this paper we propose the two-stage approach of organizing information in video surveillance systems. At first, the faces are
detected in each frame and a video stream is split into sequences of
frames with face region of one person. Secondly, these sequences (tracks)
that contain identical faces are grouped using face verification algorithms
and hierarchical agglomerative clustering. Gender and age are estimated
for each cluster (person) in order to facilitate the usage of the organized
video collection. The particular attention is focused on the aggregation
of features extracted from each frame with the deep convolutional neural
networks. The experimental results of the proposed approach using YTF
and IJB-A datasets demonstrated that the most accurate and fast solution is achieved for matching of normalized average of feature vectors of
all frames in a track.
Keywords: Organizing video data, video surveillance system, deep convolutional neural networks, clustering, face verification.

1

Introduction

Nowadays, due to the growth of the multimedia data volume, the task of forming an automatic approach to the ordering of digital information is attracting
increasing attention [1]. The various photo organizing systems allows the user to
speed up the search for the required frame, and also to increase the efficiency of
work with the media library. Such modern solutions include services like Apple
iPhoto, Google Photos, etc., which are designed to store, organize and display
media data. However, multimedia data organization systems are required not
only for a particular user who has an archive of photographs, but also for the
field of public safety, where video surveillance technologies are used for monitoring purposes [2]. Consequently, there is a challenge of ordering the visitors,
whose faces are observed in a surveillance system. To solve the problem the
clustering of video tracks that contains the same person can be performed using
the known face verification methods [3,4] based on deep convolutional neural
networks (CNNs) [5,6,7,8]. Unlike the traditional technologies of ordering digital

2

A.D.Sokolova, A.S.Kharchevnikova, A.V. Savchenko

information, video surveillance systems are characterized by a large amount of
data, because hundred frames can be obtained in dynamics in a few seconds
[4,9]. Therefore the goal of our research is to improve the verification efficiency
by the combination for features extracted from individual frames. The rest of the
paper is organized as follows: in Section 2, we formulate the pro-posed approach
of organizing multimedia data in video surveillance system. In Section 3, we
present the experimental results in unconstrained face verification. Concluding
comments are given in Section 4.

2

Automatic Organization of Video Data

The task of this paper is to split the given video sequence of T frames into
subsequences with observations of one person, and then unite different subsequences containing the same person. At first, the facial regions are detected [9]
using, e.g., the Viola-Jones method. For simplicity, we assume that each frame
in the given video consists of images (frames) of exactly one face. Next, an
appropriate tracker algorithm [9,10] divides the input sequence into M < T
disjoint subsequences (tracks) {X(m)}, m = 1, 2, ..., M , where the m-th frame
is characterized by its borders (t1 (m),t2 (m)), where the m-th track contains
∆t(m) = t2 (m) − t1 (m) + 1 frames. Finally, we search for similar tracks using,
e.g., hierarchical agglomerative clustering methods [11]: similar objects are sequentially grouped together. In order to implement any clustering method, a
dissimilarity measure between video tracks should be defined. Let us extract
appropriate facial features from every frame.
Nowadays feature extraction is implemented using the deep CNNs trained
with an external large dataset, e.g., Casia WebFaces or MS-Celeb-1M [6,12].
The outputs of the CNNs last (bottleneck) layer for the t-th frame are stored
in the D-dimensional feature vector x(t). These bottleneck features are usually
matched with the Euclidean distance (x(t1 ), x(t2 )) [12]. It is possible to define
the dissimilarity of tracks X(m1 ) and X(m2 ) as a summary statistic of distances
between individual frames. In our experiments the highest accuracy was achieved
with the average distance:

ρ(X(m1 ), X(m2 )) =

1
∆t(m1 )∆t(m2 )

X

t2 (m1 )

X

t2 (m2 )

ρ(x(t), x(t′ )).

(1)

t=t1 (m1 ) t′ =t1 (m2 )

However, the run-time complexity of such distance is high due to the pair-wise
matching of all frames in these tracks causing the computation of ∆t(m1 )∆t(m2 )
distances between high-dimensional features. Hence, in this paper we examine
the computation of the distance between tracks X(m1 ) and X(m2 ) as the distance between their fixed-size representations. Yang et al [5] proposed the 2-layer
neural network with attention blocks to aggregate the CNN features of all frames.
However, in our experiments the robustness of this approach was insufficient,
hence, we use straight-forward aggregation (or pooling [5]) techniques:

Organizing Multimedia Data in Video Surveillance Systems

3

1. The distance between tracks is defined as the distance between their medoids:
ρ(X(m1 ), X(m2 )) = ρ(x*(m1 ), x*(m2 )),
X

t2 (mi )

x*(mi ) =

argmin

ρ(x(t), x(t′ )), i ∈ {1, 2}. (2)

x(t),t∈[t1 (mi ),t2 (mi )] t′ =t (m )
1
i

2. Average features of each track are matched:
1
ρ(X(m1 ), X(m2 )) = ρ(x̄(m1 ), x̄(m2 )), x̄(mi ) =
∆t(mi )

X

t2 (mi )

x(t). (3)

t′ =t1 (mi )

It is worth noting that in static image recognition tasks the CNN bottleneck
feature vectors are typically divided into their L2 norm [12]. Such normalization
is known to make these features more robust to variations of observation conditions, e.g., camera resolution, illumination and occlusion. However, in our task
the sequences of frames are matched, so it is possible to slightly defer the normalization. Thus, in this paper we consider either conventional approach with
aggregation of the normalized features (hereinafter “L2 -norm -> Medoid” (2)
and “L2 -norm -> AvePool” (3)), or its slightly modified version with normalization of aggregated vectors (2), (3) (hereinafter “Medoid -> L2 -norm” and
“AvePool -> L2 -norm”, respectively).
We implemented the described approach in a special in MS Visual Studio
2015 project (github link will be provided after double-blind peer review) using
C++ language and the OpenCV library, especially, its DNN and Tracking extra
modules. The complete data flow in this system is presented in Fig. 1.

Fig. 1. The data flow in the organizing video data system.

Here we detect faces using the Viola-Jones cascades with the Haar features.
The obtained facial regions are verified by additional eye detection [9] and
tracked using the KCF algorithm [10]. Face detection is repeated periodically
in order to: 1) verify the tracking results; 2) look for new faces, and 3) mark

4

A.D.Sokolova, A.S.Kharchevnikova, A.V. Savchenko

disappeared persons. In the latter case we extract D CNN bottleneck features
for each frame of the track. These features are extended with the probabilities
at the output of the gender and age prediction CNNs [13]. In the data flow (Fig.
1) the simple online clustering of the normalized features is presented [14]: the
feature vector of the last track is matched with the features of previously detected clusters. If the distance to the nearest cluster does not exceed a certain
threshold, this track is added to the cluster, and the information about the latter
is updated. Namely, we compute the aggregated features (2) or (3) of a whole
cluster and re-estimate the gender and age of persons discovered in the input
video in order to facilitate the navigation through the organized collection of
tracks.

3

Experimental Results

In this section we provide experimental study of the key part of the proposed
system (Fig. 1), namely, the matching of video tracks in unconstrained face
verification task. In addition to described aggregation techniques (“L2 -norm ->
Medoid” and “Medoid -> L2 -norm” (2), “L2 -norm -> AvePool” (3)) we examined the pairwise comparison of all frames in both tracks (1). Moreover, we
implemented the described techniques (1)-(3) unnormalized features. To extract
features, we used the Caffe framework and two publicly available CNNs suitable
for face recognition, namely, the VGGNet [6] and Lightened CNN (version C)
[12]. The VGGNet extracts D = 4096 non-negative features in the output of
“fc7” layer from 224x224 RGB images. D = 256 features (“eltwise fc2” layer)
are computed when 128x128 grayscale image of the facial region is fed into the
Lightened CNN. All experiments were performed on the computer Lenovo ideapad 310, 64-bit operating system with NVIDIA GeForce 920MX.
Our first experiments were conducted on the YouTube Faces (YTF) database
[15], which contains 3,425 videos of 1,595 different people. An average of 2.15
videos are available for each subject. The shortest track duration is 48 frames,
the longest track contains 6,070 frames, and the average length of a video clip is
181.3. The estimates of AUC (Area under curve) and FRR (False Reject Rate)
for fixed FAR (False Accept Rate) using the YTF face verification protocol are
presented in Table 1 (in the format mean standard deviation). Here we do not
display a row for “L2 -norm -> Medoid”, because its results are identical to the
“Medoid -> L2 -norm” due to the independence of the computed medoid (2) on
the order of normalization.
These results emphasize the need for proper normalization of feature vectors.
The most efficient algorithm is to normalize the features of all frames and then
find the average distance (1). The obtained state-of-the-art result is 0.982, the
difference be-tween it and 0.988 [5] is not statistically significant. However, the
normalization of AvePool (3) features is characterized by practically the same
quality, though it is much faster. The AUC for matching of medoids (2) is 1012% less when compared to the AUC of the AvePool (3). It is worth noting that

Organizing Multimedia Data in Video Surveillance Systems

5

Table 1. Results of video-based face verification, YTF dataset
Lightened CNN

Distance (1)
L2 -norm
->
Distance (1)
Medoid (2)
Medoid (2) ->
L2 -norm
AvePool (3)
L2 -norm
->
AvePool (3)
AvePool (3) > L2 -norm

VGGNet

AUC(%)

ERR(%)

FRR@FAR
= 1%

AUC(%)

ERR(%)

FRR@FAR
= 1%

90.7±0.6
98.2±0.4

15.7±0.2
6.0±0.1

77.0±8.4
14.1±3.6

83.3±0.8
97.9±0.6

24.0±0.3
6.0±0.1

85.8±9.0
23.2±6.3

84.7±0.7
88.8±0.6

25.0±0.3
19.0±0.2

72.9±7.8
54.1±5.9

80.8±1.2
85.2±0.7

27.0±0.4
23.0±0.2

83.9±7.7
69.9±7.9

91.8±1.4
96.8±0.5

13.0±0.1
12.0±0.1

72.3±11.5
37.2±7.6

87.4±1.2
96.3±0.7

39.0±0.3
39.0±0.3

81.2±5.8
76.9±6.8

97.6±0.5

7.5±0.1

12.5±3.1

97.7±0.6

13.0±0.1

25.3±7.8

the latter method is also 1-2% more accurate than the conventional approach
[4,5] of averaging the preliminarily normalized features.
In the next experiment we obtained a cluster threshold by fixing FAR =
1% and using training set, then applied the clustering of all tracks from the
YTF. By using the Lightened CNN features, 1800 clusters were identified, 20 of
them contain videos of different persons. The application of the VGGNet feature
extraction increases the number of clusters to 2000 with 30 incorrect clusters.
The first experiment was repeated for rough grouping of tracks with the
persons of approximately identical age and same gender using the probabilities
at the outputs of the pre-trained CNNs [13]. Table 2 contains AUC achieved for
matching of D = 8 posterior probabilities of age categories, D = 2 ((male/female)
posterior probabilities) and the union of these two feature sets. We used L1 norm to treat the features as posterior probabilities and compared them with
either Euclidean (L2 ) distance of the Kullback-Leibler (KL) divergence, which is
assumed to be more suitable for comparison of discrete probability distributions.
Here the prior feature normalization is not needed, as the outputs of the CNNs
softmax layers are L1 normed. These results are much worse when compared to
facial features from the previous experiment. Nevertheless, the age and gender
features can be potentially used to refine the results obtained by conventional
face verification techniques (Table 1) AUC is 8-19% higher than the random
guess.
The last experiment was conducted on the IARPA Janus Benchmark A (IJBA) (IJB-A) dataset [16] with 2043 videos of 500 identities. Table 3 contains
the results of several best aggregation techniques in the face verification with
bottleneck features extracted by VGGNet and Lightened CNN.
In contrast to the first experiment, here the VGGNet [6] is much more accurate than the Lightened CNN [12]. Our conclusions about relative efficiency

6

A.D.Sokolova, A.S.Kharchevnikova, A.V. Savchenko

Table 2. AUC (%) of video-based face verification, YTF dataset, age and gender
features

Distance (1)
Medoid (2)
AvePool (3) -> L2 -norm

Distance

Age

L2
KL
L2
KL
L2
KL

60.8±1.3
61.6±1.1
58.4±1.4
58.9±1.4
60.4±1.3
63.2±1.2

Gender Age and Gender
65.8±0.8
65.8±1.0
63.3±1.0
63.4±1.0
65.7±0.9
65.3±0.9

68.7±0.8
65.8±0.9
64.9±1.0
64.8±0.9
67.9±0.9
68.8±0.8

Table 3. Results of video-based face verification, IJB-A dataset

Lightened CNN

L2 -norm
->
Distance (1)
Medoid (2) ->
L2 -norm
L2 -norm
->
AvePool (3)
AvePool (3) > L2 -norm

VGGNet

AUC(%)

ERR(%)

FRR@FAR
= 1%

AUC(%)

ERR(%)

FRR@FAR
= 1%

87.9±0.5

20.5±0.9

67.9±3.3

97.5±0.4

8.0±0.4

30.3±4.6

76.6±0.4

30.0±1.2

77.1±4.0

92.4±0.7

15.5±0.6

50.0±6.9

79.6±0.7

27.8±0.8

67.5±4.4

96.1±0.4

13.6±0.8

40.0±4.4

88.2±0.4

20.0±0.4

59.3±2.9

97.7±0.3

8.0±0.3

26.8±4.2

of discussed aggregation techniques remain similar to the previous experiments.
However, this dataset highlights the superiority of the normalized average features (AvePool -> L2 -norm): it drastically improves AUC and FRR, when compared to traditional imple-mentation of average pooling in aggregation of video
features [3,4].

4

Conclusion

In this paper we considered the automatic organizing the data in video surveillance systems (Fig. 1). We particularly focused on the ways to efficiently compute
the dissimilarity of video tracks by using rather simple aggregation techniques.
We experimentally supported the claim that the most accurate and computationally cheap technique involves the L2 -normed average vector of unnormalized
frame features. It was noticed that the sequence of this two operations is very
important. In fact, much more widely used aggregation of normalized features
[2] is usually less accurate (Table 3).

Organizing Multimedia Data in Video Surveillance Systems

7

The main direction for further research is applying our approach in organizing data from real video surveillance systems. It is also important to examine
more sophisticated distances between video tracks, e.g., metric learning [17] or
statistical homogeneity testing [11]. If the number of observed persons is high, it
is necessary to deal with insufficient performance of our simple online clustering
by using, e.g., approximate nearest neighbor search [14,7,8,18]. Moreover, we
are planning to introduce the weighing for different features including age and
gender probabilities to make our algorithm more accurate. In fact, the accuracy
of the age and gender prediction CNNs [13] is rather low, hence, it is necessary
to implement contemporary CNN architectures including Inception or ResNets.
Acknowledgements. The article was prepared within the framework of the
Academic Fund Program at the National Research University Higher School of
Economics (HSE) in 2017 (grant 17-05-0007) and by the Russian Academic Excellence Project “5-100”. Andrey V. Savchenko is partially supported by Russian
Federation President grant no. MD-306.2017.9.

References
1. Manju, A., Valarmathie, P.: Organizing multimedia big data using semantic based
video content extraction technique. In: IEEE International Conference on SoftComputing and Networks Security (ICSNS), pp. 1-4 (2015)
2. Zhang, Y.J., Lu, H.B.: A hierarchical organization scheme for video data. Pattern
Recogni-tion 35(11), 2381-2387 (2002)
3. Chen, J.C., Ranjan, R., Kumar, A., Chen, C.H., Patel, V.M., Chellappa, R.: An
end-to-end system for unconstrained face verification with deep convolutional neural networks. In: IEEE International Conference on Computer Vision Workshops,
pp. 118-126 (2015)
4. Li H., Hua G., Shen X., Lin Z., Brandt J.: Eigen-PEP for video face recognition.
In: Cremers D., Reid I., Saito H., Yang M.H. (eds.) Asian Conference on Computer
Vision. ACCV 2014. LNCS, vol. 9005, pp. 17-33. Springer, Cham (2014)
5. Yang J., Ren P., Chen D., Wen F., Li H., Hua G.: Neural aggregation network for
video face recognition, arXiv: 1603.05474 (2016)
6. Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition. In: Proceedings
of the British Machine Vision, pp. 6-17 (2015)
7. Savchenko, A. V.: Deep Convolutional Neural Networks and Maximum-Likelihood
Principle in Approximate Nearest Neighbor Search. In: L.A. Alexandre et al.
(Eds.): IbPRIA 2017, LNCS, vol. 10255. pp. 42-49. Springer International Publishing (2017)
8. Savchenko, A. V.: Deep neural networks and maximum likelihood search for approximate nearest neighbor in video-based image recognition. In: Optical Memory
and Neural Networks (Information Optics), vol. 26, no. 2, pp. 129-136 (2017)
9. Szeliski, R.: Computer vision: algorithms and applications. Springer Science and
Business Media (2010)
10. Henriques J. F., Caseiro R., Martins P., Batista J.: Exploiting the circulant structure of tracking-by-detection with kernels. In: Fitzgibbon A., Lazebnik S., Perona
P., Sato Y., Schmid C. (eds) European Conference on Computer Vision (ECCV
2012). LNCS, vol 7575, pp. 702-715. Springer, Berlin, Heidelberg (2012)

8

A.D.Sokolova, A.S.Kharchevnikova, A.V. Savchenko

11. Friedman, J., Hastie, T., Tibshirani, R.: The Elements of Statistical Learning: Data
Mining, Inference, and Prediction. Springer Series in Statistics (2009)
12. Wu, X., He, R., Sun, Z.: A Lightened CNN for deep face representation.
arXiv:1511.02683 (2015)
13. Levi, G., Hassner, T.: Age and gender classification using convolutional neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 34-42 (2015)
14. Savchenko, A.V.: Clustering and maximum likelihood search for efficient statistical
classification with medium-sized databases. Optimization Letters 11(2), 329-341
(2017)
15. Wolf, L., Hassner, T., Maoz, I.: Face recognition in unconstrained videos with
matched background similarity. In: IEEE International Conference on Computer
Vision and Pattern Recognition (CVPR), pp. 529-534 (2011)
16. B. F. Klare, B. Klein, E. Taborsky, A. Blanton, J. Cheney, K. Allen, P. Grother,
A. Mah, A. K. Jain. Pushing the frontiers of unconstrained face detection and
recognition: IARPA Janus Benchmark A. In: IEEE International Conference on
Computer Vision and Pattern Recognition (CVPR), pp. 1931-1939 (2015).
17. Kulis, B.: Metric learning: A survey. Foundations and Trends in Machine Learning
5(4), pp. 287-364 (2013)
18. Babenko A., Lempitsky V.: The inverted multi-index. In: IEEE International Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp.3069-3076
(2012).

