Maximum A Posteriori Estimation of Distances Between Deep Features in Still-to-Video Face Recognition
Andrey V. Savchenko Natalya S. Belova







Maximum A Posteriori Estimation of Distances Between Deep Features in Still-to-Video Face Recognition
    Andrey V. Savchenko

National Research University Higher School of Economics

Laboratory of Algorithms and Technologies for Network Analysis, 
36 Rodionova St., Nizhny Novgorod, Russia

avsavchenko@hse.ru
Natalya S. Belova

National Research University Higher School of Economics

20 Myasnitskaya St., Moscow, Russia

nbelova@hse.ru

    December 30, 2023
=================================================================================================================================================================================================================================================================================================================================================



The paper deals with the still-to-video face recognition for the small sample size problem based on computation of distances between high-dimensional deep bottleneck features. We present the novel statistical recognition method, in which the still-to-video recognition task is casted into Maximum A Posteriori estimation. In this method we maximize the joint probabilistic density of the distances to all reference still images. It is shown that this likelihood can be estimated with the known asymptotically normal distribution of the Kullback-Leibler discriminations between nonnegative features. The experimental study with the LFW (Labeled Faces in the Wild), YTF (YouTube Faces) and IJB-A (IARPA Janus Benchmark A) datasets has been provided. We demonstrated, that the proposed approach can be applied with the state-of-the-art deep features and dissimilarity measures. Our algorithm achieves 3-5% higher accuracy when compared with conventional aggregation of decisions obtained for all frames.






§ ABBREVIATIONS



  * CNN - Convolution Neural Network

  * i.i.d. - independent identically distributed

  * IJB-A - IARPA Janus Benchmark A

  * KL - Kullback-Leibler divergence

  * LFW - Labeled Faces in the Wild

  * MAP - Maximum A Posteriori

  * ML - Maximum Likelihood

  * NN - Nearest Neighbor 

  * YTF - YouTube Faces




§ INTRODUCTION


The usage of deep learning technologies <cit.> instead of traditional computer vision methods <cit.> has recently made it possible to achieve a near human-level performance in various face recognition tasks, such as verification <cit.> or identification <cit.>. Moreover, contemporary techniques can deal even with well-known crucial issues appeared in practical applications of face recognition, e.g., unconstrained environment (various illumination and pose, partial occlusion) <cit.>, or the small sample size problem <cit.>, when usually only single facial image per person is available <cit.>. The latter problem is solved using transfer learning methods <cit.>, in which large external datasets of celebrities are used to train deep convolutional neural network (CNN) <cit.>. This CNN is further applied to extract features of the training images from the limited sample of subjects of interest using the outputs of a bottleneck layer <cit.>. Such transfer learning technique makes it possible to create a classifier that ideally performs nearly as well as if rich dataset of photos of these individuals were present <cit.>. 

Though the algorithms that do recognition from a single test photo have its own practical applications, e.g., search for a person in a social network, one of the most significant use-cases is provided by surveillance and authentication systems. Hence, all the more attention is nowadays paid to video face recognition <cit.>, in which a set or a sequence of observed images of the same individual are available for decision-making <cit.>. Despite the large number of recent papers devoted to video face recognition <cit.>, this task still remains very challenging. For example, most of the known algorithms suffered from heavy off-line training load <cit.>, and they were experimentally studied only with databases, in which the still images are gathered under controlled environment. In this paper we proposed the novel still-to-video face recognition method suitable for application with the small sample of still photos per person, which is based on the probabilistic interpretation <cit.> without learning temporal coherence <cit.>. At first, the nearest neighbor (NN) reference images are examined for each video frame. Next, computed distances to all still images are used to weight the recognition results for each frame based on the estimation of their reliability. The more is the likelihood of the computed vector of distances for particular individual, the more is the weight corresponding to this frame and this subject. The likelihoods (joint probabilistic densities) are computed using the idea of the maximum-likelihood approximate NN algorithm <cit.> by assuming that the Kullback-Leibler (KL) divergence <cit.> is used to compare distances. However, we demonstrate that our approach can be successfully applied with much more widely used dissimilarity measures.

The rest of the paper is organized as follows. Section <ref> contains the literature survey of related papers. In Section <ref> we present a simple statistical formulation of the still-to-video recognition task <cit.> using the KL minimum discrimination principle. In Section <ref> we propose the novel approach, in which the maximum a-posteriori (MAP) rule is regularized using the computation of the joint probability densities of distances based on the asymptotic properties of the KL divergence. In Section <ref> we present the complete algorithm, in which the initial assumptions are facilitated and the computational complexity is taken into account. Section <ref> presents the experimental results in recognition of videos from either IJB-A (IARPA Janus Benchmark A) <cit.> or YouTube Faces (YTF) datasets <cit.> with the still images from the Labeled Faces in the Wild (LFW) dataset <cit.> using such deep CNNs as VGGNet <cit.> and Lightened CNN <cit.>. Finally, concluding comments are given in Section <ref>.



§ RELATED WORKS


Compared to still-to-still recognition, the requirement of real-time processing in video face recognition prevents an implementation of too sophisticated recognition algorithms <cit.>, especially, when the number of individuals is rather large <cit.>. However, more information of the individuals can be exploited from several video frames <cit.>, and the recognition results can be generally improved when using multiple images or video sequences <cit.>. Most of the papers devoted to person identification or retrieval in surveillance videos <cit.> address the video-to-video face recognition problem, in which query videos are matched against a set of reference video sets <cit.>. However, real-world applications usually involve the still-to-video face recognition scenario, when only very few still images per person are enrolled into the training set while the input sequence of video frames is captured <cit.>. This problem is very difficult, because the testing videos have generally low quality and are captured under poor illumination and pose <cit.>. 

There are two types of still-to-video recognition methods <cit.>. First group of methods exploits the temporal information in a video sequence. Liu and Chen used the hidden Markov models in recognition of video frames <cit.>. The paper <cit.> describes successful application of a state space model <cit.> parametrized by a tracking state vector and a recognizing identity variable, simultaneously characterizing the kinematics and identity of humans. All such methods require large datasets of video clips to train the extraction of dynamic information. Moreover, their performance relied on strong temporal coherence in training and testing environment <cit.>. Hence, another group of methods with accumulation of multiple observations of the same individual has recently been studied. These methods are usually based on fusion of evidence from multiple measurements <cit.>. For example, the authors of the paper <cit.> extended the probabilistic appearance-based face recognition approach to work with multiple images and video sequences. Shakhnarovich et al <cit.> assumed that the multiple frames for each human subject followed the normal distribution, and they used the KL divergence <cit.> to measure the distance between distributions. Several important works on still-to-video face recognition adopted a metric learning framework. For example, Zhu et al <cit.> extended the well-known Mahalonbis metric learning to point-to-set and set-to-set distance learning. Huang et al <cit.> applied Euclidean-to-Riemannian metric learning framework to image-to-set object classification and still-to-video face recognition tasks. 

Nowadays, it is more typical to map each face image into a feature vector using a deep CNN. Next, a naive approach would be representing a video face as a set of frame-level face features from the CNN <cit.>. Such a representation obviously maintains all the information across all frames. To speed-up the recognition process, the features of all video frames are aggregated into a single vector (video face representation) using such pooling strategies as average and max pooling <cit.>. The Eigen-PEP representations <cit.> integrates the visual information from all relevant video frames using part-based average pooling through the probabilistic elastic part model. Then the intermediate representation is compressed through principal component analysis, and only a number of principal eigen dimensions are kept. Yang et al <cit.> proposed the learning of the frames weights in the aggregation module, which consists of two attention blocks driven by a memory storing all the extracted features. Canziani and Culurciello proposed the CortexNet neural architecture <cit.> in order to obtain a robust and stable representation of temporal visual inputs. In the paper <cit.> two video sequences representing head motion and facial expressions are compared using a new positive definite kernel based on the concept of principal angles between two linear subspaces. Most of described methods learned the relationship between the still images and video frames but did not directly handle bad quality frames, which very likely make the recognition perform badly. To deal with this problem, another kind of methods such as <cit.> was proposed, which first select the best quality frames, perform facial alignments and then integrate the recognition results of selected frames <cit.>. Thus, one can notice from this brief survey, that the methods without learning temporal coherence are more widely used nowadays. Hence, in our paper we decided to focus on application of statistical recognition of individual frames <cit.> described by the modern deep CNN features <cit.>.



§ STATISTICAL STILL-TO-VIDEO FACE RECOGNITION



The task of closed-set still-to-video face identification is to assign an observed sequence of T video frames to one of C>1 classes (identities). The classes are specified by the training set of R≥ C still images. We consider the supervised learning case, when the class label (subject id) c(r) ∈{1,...,C} of the rth photo is known. For simplicity, we assume that only one image per person is available (C=R,c(r)=r), and all frames contain facial region of only one identity, i.e., the faces are detected, and the whole video clip is clustered to extract sequential frames containing the same subject. At first, each image should be described with a feature vector using the preliminarily trained deep CNN, as it was described in introduction. The outputs at the last layer of this CNN for all t frames and each r-th reference image are used as the D-dimensional feature vectors 𝐱(t)=[x_1(t),...,x_D(t)] and 𝐱_r=[x_r;1,...,x_r;D], respectively. 

Let us assume that the normalized feature vector 𝐱(t) is the estimate of multinomial distribution of (hypothetical) random variable X(t),t ∈{1,...,T}. We also assume that each rth instance represents the probabilistic distribution of the random variable X_r,r ∈{1,...,R}. The video face recognition task is reduced to a problem of statistical testing of simple hypothesis W_r, r ∈{1,...,R} for all frames. Statistically optimal decision is defined as follows:


    r ∈{1,...,R}max P(W_r|𝐱(1),...,𝐱(T)),

where the posterior probability is estimated using the Bayes rule:


    P(W_r|𝐱(1),...,𝐱(T))=f(𝐱(1),...,𝐱(T)|W_r)/∑_i=1^Rf(𝐱(1),...,𝐱(T)|W_i),


In this paper we focus on the case of full prior uncertainty, hence, the prior probabilities of observing each subject here are equal, so the rule (1) represents the maximum likelihood (ML) criterion. Finally, it is necessary to perform aggregation of the frame recognition results <cit.>. Kittler et al <cit.> presented a statistical interpretation of a number of common methods for cross-modal fusion, such as the product, sum, maximum, and majority vote rules, which are also appropriate for late integration over a set of observations from a single modality. For example, it can be shown that under the assumption that the frames in the video sequence correspond to independent identically distributed (i.i.d) random variables, the likelihood is estimated using the product rule <cit.>


    f(𝐱(1),...,𝐱(T)|W_r)=∏_t=1^T∏_d=1^D(x_r;d)^n · x_d(t),

where n is the sample size used to estimate the distribution 𝐱(t). In practice, this parameter can be chosen proportional to the total number of pixels in the input image. By substituting (3) in (2) and dividing its numerator and denominator to ∏_t=1^T∏_d=1^D(x_d(t))^n · x_d(t), one can obtain the following estimate of the posterior probability:


    P(W_r|𝐱(1),...,𝐱(T))= exp(-n ∑_t=1^T∑_d=1^Dx_d(t) ln(x_d(t)/x_r;d))/∑_i=1^Rexp(-n ∑_t=1^T∑_d=1^Dx_d(t) ln(x_d(t)/x_i;d)),

or

    P(W_r|𝐱(1),...,𝐱(T))= exp(-n ∑_t=1^TI(𝐱(t):𝐱_r))/∑_i=1^Rexp(-n ∑_t=1^TI(𝐱(t):𝐱_i)),

where I(𝐱(t):𝐱_r)) is the KL divergence between feature vectors 𝐱(t) and 𝐱_r.

Hence, the ML criterion (1) can be written in the simplified form:


    r ∈{1,...,R}min∑_t=1^TI(𝐱(t):𝐱_r).


Thus, the ML decision (1) of the still-to-video face recognition task <cit.> and assumptions about i.i.d. frames is equivalent to the KL minimum information discrimination principle <cit.>.



§ MAXIMUM A POSTERIORI ESTIMATION OF DISTANCES BETWEEN DEEP BOTTLENECK FEATURES



In this paper we use the idea of the maximal-likelihood approximate NN method from the paper <cit.>. Namely, we exploit the known property of the KL divergence between two densities, which can be considered as the information for discrimination in favor of the first density against the second one <cit.>. Hence, another recognition criterion is used, in which the maximum posterior probability based on the joint distribution of all R-dimensional random vectors of distances 𝐈(t)=[I(𝐱(t):𝐱_1),..., I(𝐱(t):𝐱_R)] is obtained:


    r ∈{1,...,R}max f(W_r|𝐈(1),...,𝐈(T)).


To deal with this conditional density of distances let us firstly consider much more simple marginal conditional distribution f(𝐈(t)|W_r), t ∈{1,...,T}. By using a natural assumption about independence of the still photos of all identities from the training set, we estimate the joint density of distances between reference images and the tth frame f(𝐈(t)|W_r) as follows:


    f(𝐈(t)|W_r)=f(I(𝐱(t):𝐱_r)|W_r) ·∏_i=1,i ≠ r^Rf(I(𝐱(t):𝐱_i)|W_r).


At first, let us consider each term in the second multiplier. We propose to estimate the conditional density function f(I(𝐱(t):𝐱_i)|W_r) of the distance between the observed object from the rth class and the ith instance using the known asymptotic properties of the KL divergence. It is known <cit.> that if i ≠ r, then 2n-times KL divergence I(𝐱(t):𝐱_r) is asymptotically distributed as the non-central chi-squared with D-1 degrees of freedom and the non-centrality parameter 2n · I(𝐱_r:𝐱_i). This asymptotic distribution is also known to be achieved for such probabilistic dissimilarities as the chi-squared distance, the Jenson-Shannon divergence, etc. <cit.>. As the number of features D is usually large, it is possible to approximate the non-central chi-squared distribution with the Gaussian distribution <cit.>. Hence, we use the asymptotically normal distribution of the distance between the facial image from the rth class and the ith reference photo:

    𝒩(I(𝐱_r:𝐱_i)+D-1/n,4n · I(𝐱_r:𝐱_i)+(D-1)/2n^2)


Thus, the conditional distributions in the second multiplier in (8) are estimated as follows:

    f(I(𝐱(t):𝐱_i)|W_r) = exp( -n ·ϕ_r;i (𝐱(t))),

where we denote ϕ_r;i (𝐱(t)) for 

    ϕ_r;i (𝐱(t)) = 1/2nln (4I(𝐱_r:𝐱_i)+π+D-1/n)+
    +(I(𝐱(t):𝐱_i)-I(𝐱_r:𝐱_i)-(D-1)/n)^2/4I(𝐱_r:𝐱_i)+(D-1)/n.


On the one hand, the number of outputs (features) in the last layer in modern DNNs is very high (D ≫ 1) <cit.>. On the other hand, the feature vector dimensionality D should be much less than the number of samples n to estimate the conditional densities (3) reliably (D ≪ n). Hence, it is possible to approximately rewrite eq. (11) <cit.>:


    ϕ_r;i (𝐱(t))≈(I(𝐱(t):𝐱_i)-I(𝐱_r:𝐱_i))^2/4I(𝐱_r:𝐱_i).


The first multiplier in (8) cannot be computed similarly, because, if i=r, then I(𝐱_r:𝐱_i)=0, and asymptotic distribution (9) does not hold in practice. However, in such a case this conditional density can be estimated using eq. (3). Similarly to (4), (5) one can show that 


    f(I(𝐱(t):𝐱_r)|W_r)) ∝exp(-n I(𝐱(t):𝐱_r)).


By combining expressions (10), (12), (13), the marginal distribution (8) is written using softmax operation as follows
 

    f(𝐈(t)|W_r)=exp(-n · (I(𝐱(t):𝐱_r)+∑_i=1^Rϕ_r;i (𝐱(t))))/∑_j=1^Rexp(-n · (I(𝐱(t):𝐱_j)+∑_i=1^Rϕ_j;i (𝐱(t)))).


After that it is possible to apply some known rules <cit.> of combining statistical classifiers (14). For example, in case of the i.i.d. frames assumptions the criterion (7) is equivalent to rather simple video face recognition rule

    r ∈{1,...,R}min∑_t=1^T(I(𝐱(t):𝐱_r)+∑_i=1^Rϕ_r;i (𝐱(t))),

which can be viewed as an extension of conventional statistical classification (6) using the new regularization term ∑_i=1^Rϕ_r;i (𝐱(t)). However, Kittler et al <cit.> argued, that the sum rule is much more robust to the errors in unknown density estimation. In our task this rule can be formulated in the MAP way:

    r ∈{1,...,R}max∑_t=1^Tf(W_r|𝐈(t)),

where posterior probability f(W_r|𝐈(t)) is defined using the Bayes rule (2) for the proposed ML of distances estimate (14).



§ PROPOSED ALGORITHM



Though the previous sections contain many unrealistic assumptions about independence of sequential frames and representation of feature vectors as the estimates of the distributions of hypothetical random variables, the resulted criteria (15), (16) looks rather straightforward. In this section we demonstrate the possibility to implement this approach in more realistic scenarios. At first, it is necessary to emphasize, that the criteria discussed in the previous section contain only computations of the KL divergence between features of all frames and every instances from the database (12). Thus, it is possible to replace the KL divergence I(𝐱(t):𝐱_r) in (12), (13) with an arbitrary dissimilarity measure ρ(𝐱(t),𝐱_r) between deep bottleneck features 𝐱(t) and 𝐱_r. For instance, the KL minimum information discrimination criterion (6) is equivalent to the NN rule with the simple accumulation <cit.>:


    r ∈{1,...,R}min∑_t=1^Tρ(𝐱(t),𝐱_r).


The sum rule <cit.> implements the MAP estimate of the the class, which corresponds to the maximal average posterior probability <cit.>:


    c ∈{1,...,C}max∑_t=1^Texp(-n·ρ_c(𝐱(t))/∑_i=1^Cexp(-n·ρ_i(𝐱(t)),


It is important to emphasize that our conclusion about asymptotical distribution (10) of the KL divergence is in agreement with the well-known assumption about Gaussian distribution of dissimilarity measures between high-dimensional feature vectors, which is supported by many experiments <cit.>.

Secondly, though the small sample size problem is usually appeared in practical applications of video face recognition, it is not typical to have only one instance per class in the available training set. Hence, we propose to extend criterion (15) to the case of C≤ R:

    c ∈{1,...,C}max∑_t=1^Texp(-n · (ρ_c(𝐱(t))+λ/C∑_i=1^C(ρ_i(𝐱(t))-ρ_c;i)^2/ρ_c;i))/∑_j=1^Cexp(-n · (ρ_j(𝐱(t))+λ/C∑_i=1^C(ρ_i(𝐱(t))-ρ_j;i)^2/ρ_j;i)).


Here we added the smoothing parameter λ>0 to tune the influence of our regularization. The distance between feature vector 𝐱(t) and the cth identity in this criterion can be defined using the idea of the single-linkage clustering:

    ρ_c(𝐱(t))=r ∈{1,...,R},c(r)=cminρ(𝐱(t),𝐱_r).


The distance between images from cth and ith identities can be computed as the average distance between instances from these classes:

    ρ_c;i=1/R_cR_i∑_r=1^R∑_r_i=1^Rδ(c-c(r))δ(i-c(r_i))ρ(𝐱_r,𝐱_r_i),

where δ(c) is the discrete delta function (indicator) and R_c=∑_r=1^Rδ(c-c(r)) is the total number of still photos of the cth subject. 

Finally, it is necessary to notice that the run-time complexity of the proposed criterion (19) O(T(R+C^2
)) is more than C-times higher than the complexity of the baseline ML rule (17). Hence, our approach cannot be applied in practical tasks when the number of classes is rather large <cit.>. To speed-up the decision process, we propose to modify criterion (19) by examining only M ≪ C candidate classes. Namely, the sum of distances between all frames and each class are computed identically to the conventional approach (17):

    ρ_c(𝐱(1),...,𝐱(T))=1/T∑_t=1^Tρ_c(𝐱(t)),

and M candidate classes {c_1,...,c_M} with the lowest distances (18) are chosen. This Mth element search (selection algorithm) is known to have linear average complexity: O(C). In the final decision only these M candidates are checked, so the criterion (19) is modified as follows:


    c ∈{c_1,...,c_M}max∑_t=1^Texp(-n · (ρ_c(𝐱(t))+λ/C∑_i=1^C(ρ_i(𝐱(t))-ρ_c;i)^2/ρ_c;i))/∑_m=1^Mexp(-n · (ρ_c_m(𝐱(t))+λ/C∑_i=1^C(ρ_i(𝐱(t))-ρ_c_m;i)^2/ρ_c_m;i)).





Our complete procedure is presented in Algorithm <ref>. Its runtime complexity is equal to O(T(R+MC)). Here the decision is made only after observation of all T frames. However, it is not difficult to implement online recognition in our approach by looking for the class label c^* right after the next video frame with the same subject is available.




§ EXPERIMENTAL RESULTS



In most experiments of still-to-video face recognition methods it is usually assumed that the still photos are of high quality and resolution, in frontal view, with normal lighting and neutral expression <cit.>. However, in this paper we decided to consider the most challenging experimental setup, in which the training set contains photos gathered in uncontrolled environment <cit.>. Namely, we choose all photos and videos of C=1589 classes as the intersection of people from the well-known LFW and YTF datasets. It is important to mention that though the YTF dataset contains the videos of celebrities from the LFW, the quality of the images in these datasets are completely different. The training set is filled by all R=4732 photos of these C=1589 identities taken from the LFW dataset <cit.>. Our testing set contains 3353 videoclips from the YTF <cit.> (in average, 183 frames per person). In our experiments we discovered that it is not necessary to process all frames in each video, hence, only each fifth frame in every video is presented in the testing set, given T=36 frames per video in average.

The Caffe framework is applied to extract deep bottleneck features using two publicly available CNN models for unconstrained face recognition, namely, the well-known VGG-13 Network (VGGNet) <cit.>, and the Lightened CNN <cit.>. The fc8 layer of the VGGNet extracts D=4096 non-negative features from 224x224 RGB image. The Lightened CNN (version C) extracts D=256 (possibly negative) features from 128x128 grayscale image. The outputs of these CNNs were L_2 normalized to form the final feature vectors, which are matched using Euclidean distance. The features in the VGGNet are positive, hence we also perform L_1 normalization to treat them as the probability distributions and match them using the KL divergence.

The proposed Algorithm <ref> is implemented in a stand-alone C++ application with Qt 5 framework. In the first experiments we examined the dependence of the top-1 accuracy on the most important parameters of our algorithm, namely, the importance λ of regularization term ∑_t=1^T∑_i=1^C(ρ_i(𝐱(t))-ρ_c;i)^2/ρ_c;i (23), and the number of candidate classes M. The results are presented in Fig. <ref> and Fig. <ref>, respectively.





Here, firstly, error rates drastically depend on the deep CNN used for feature extraction. The accuracy of the Lightened CNN <cit.> is approximately 20% higher than the accuracy of much widely used VGGNet <cit.>. It is worth noting that treating the features from the latter network as probability distributions and matching them with the KL divergence (4) allows decreasing error rate at 1.4% in the best case. Secondly, the U-curves in Fig. <ref> prove that the proper choice of the parameter λ can significantly influence the recognition accuracy. It is especially true for the VGGNet features, in which the difference in error rates achieves 2.5%. However, the optimal value of this parameter is practically identical (λ=6..8) for all three combination of the CNN and dissimilarity measure. At the same time, the curves in Fig. <ref> reaches stability very fast with an increase of the number of candidate classes M. For example, the accuracy for the VGGNet features does not increase further if M ≥ 16. Though the examination of all M=C classes causes the lowest error rates, this value is not recommended for practical usage due to the very low recognition speed. Hence, in the further part of this paper we use near-optimal values of these parameters: λ=7 and M=64.

In the next experiments we examine the influence of additive noise on the face recognition quality. The uniform random number from the range [-X_max;X_max] was added to every pixel of each frame of all testing videoclips, where X_max≥ 0 determines the noise level. Original videos from the YTF dataset are recognized, when X_max=0. The proposed approach (23) is compared with conventional video face recognition techniques, namely, 1) the ML rule (17) applied to all frames <cit.>; 2) selection of one representative frame with the k-medoids clustering method (k=1) and the NN matching (17) of this frame <cit.> (hereinafter "ML with clustering"); and 3) estimation of the MAP class for each frame (18) (hereinafter "MAP").
The parameter n of the proposed criterion and MAP rule is tuned in order to provide the highest validation accuracy. The main results of these experiments are shown in Fig. <ref>, Fig. <ref> and Fig. <ref> for VGGNet features matched with the Euclidean distance and the KL divergence, together with the Lightened CNN features, respectively.







Based on these results one can draw the following conclusions. Firstly, though the additive noise leads to degraded recognition accuracy in most cases, the increase of the error rate is rather low even if up to X_max=10 number is added to the value of each pixel. Secondly, though the KL divergence outperforms the conventional Euclidean distance for original YTF dataset (X_max=0), the latter dissimilarity is more robust to the presence of noise. In fact, it is better to apply more robust probabilistic dissimilarities, which are based on testing for statistical homogeneity of the feature vectors <cit.>. Nevertheless, the Lightened CNN again significantly outperforms the VGGNet even for very high noise levels. Thirdly, though the rather popular accumulation of all frames in one centroid (video face representation <cit.>) in the "ML with clustering" method makes it possible to increase performance in T times, its error rate is 2-5% and 2-6% higher than the error rate of the baseline ML (17) and the average posterior probability pooling (18) <cit.>. Moreover, if this approach is applied in online video recognition right after new tth frame is observed, the gain in performance over other methods is not so obvious. Really, it is easy to implement criteria (17), (23) in adaptive mode <cit.> so that the processing of the tth frame is aggregated with the results obtained for recognition of previous (t-1) frames. It is not surprising that the frame selection methods <cit.> have more potential in video-to-video recognition or video verification tasks <cit.>, in which the matching of all pairs of frames from two video clips is very expensive. Finally, the most important conclusion here is the highest recognition accuracy of the proposed algorithm <ref> in all cases. Our approach is 3-4% more accurate when compared to other methods. Moreover, though our idea is based on looking for correspondence of the distances ρ_i(𝐱(t)) (17) and ρ_c;i (18), (19), the gain in error rate does not significantly degrade even with the presence of noise in the testing videos. It is important to highlight that though the original version of the proposed approach (14) was based on the properties of the KL divergence between positive features, our algorithm can be successfully used with the state-of-the-art distances and arbitrary feature vectors.

In the last part of this section we provide preliminary results for IJB-A dataset <cit.>. The IJB-A 1:N protocol is primarily used for testing of the video-to-video recognition. Moreover, each split is used to recognize only 33% subjects, the rest part can be used to train an algorithm under the strict condition that no such imagery contain the same subjects that are in the test split. Hence, we decided to implement more difficult scenario of the still-to-video recognition with all C=500 subjects. We put all R=5712 still photos of these subjects to the training set. The testing set contains all 2043 videos from this dataset with approximately 8 frames per video. The average accuracies for all methods described above are shown in Table <ref>.



In contrast to the previous experiments, here the features from the VGGNet <cit.> allows classifying videos 25% more accurately when compared to the Lightened CNN <cit.>, which can be probably caused by the presence of many non-frontal photos in this dataset. However, all the previous conclusions about the still-to-video recognition methods remain the same. The selection of the most representative frame in the ML with clustering is 0.6-0.9% less accurate that the matching of all frames in criterion (17). The pooling of posterior probabilities (18) here is much preferable than the ML rule (17). At the same time, the error rate of the proposed algorithm <ref> is 4.5-5.5% and 1.1-2.5% lower, than the error rates of the widely-used criteria (17) and (18), respectively.




§ CONCLUSION



In this paper we proposed the novel statistical approach to still-to-video face recognition (Algorithm <ref>), in which the joint density of distances to all reference images is maximized (7), (8). We have shown that this approach is implemented by introduction of a special regularization term (23) in the nearest neighbor matching (17) of high-dimensional features from the outputs of the deep CNNs. In this regularization it is assumed that the resulted class c^*(t) for recognition of individual tth video frame is reliable only if the distances between the features of this frame and the rth still image are approximately equal to the distances between the reference images from the c^*(t) and c(r) classes for all r ∈{1,...,R}. This assumption is known to be asymptotically correct for the KL divergence and rather simple probabilistic model from Section <ref>  <cit.>. However, it was experimentally demonstrated that the proposed approach can be combined even with the state-of-the-art Euclidean distance (Fig. <ref>). Moreover, our algorithm allows to increase accuracy even for general feature vectors with possible negative values (Fig. <ref>). We demonstrated how to tune the parameters of the proposed algorithm (Fig. <ref>) in order to drastically reduce its computation complexity by proper choice of M candidate classes (compare the criterion (23) with the original one (19)).

The main direction for further research of the proposed algorithm is its applications with more accurate approximation of the distance probability distributions, e.g., the usage of the more appropriate Weibull distribution <cit.>. Secondly, it is important to make our regularization smoother by taking into account the temporal coherence of sequential frames <cit.>. Thirdly, it is necessary to combine our approach with the modern techniques with frames weighting in the aggregation module (22) <cit.> to make distinction between frames with different quality  <cit.>. 
Finally, though the deep bottleneck features are known to be highly embedded in the manifold space, non-Euclidean metrics, e.g, such techniques as the point-to-set metric learning <cit.> or geometry aware feature matching <cit.> can also be discussed in the future experiments.



§ ACKNOWLEDGEMENT

The paper is supported by Russian Federation President grant no. MD-306.2017.9. The work in Section 3 was conducted by A.V. Savchenko at Laboratory of Algorithms and Technologies for Network Analysis, National Research University Higher School of Economics and supported by RSF grant 14-41-00039.


§ REFERENCES


10
url<#>1urlprefixURL href#1#2#2 #1#1

lecun2015deep
Y. LeCun, Y. Bengio, G. Hinton, Deep learning, Nature 521 (7553) (2015)
  436–444.

prince2012computer
S. J. Prince, Computer vision: models, learning, and inference, Cambridge
  University Press, 2012.

taigman2014deepface
Y. Taigman, M. Yang, M. Ranzato, L. Wolf, Deepface: Closing the gap to
  human-level performance in face verification, in: Proceedings of the IEEE
  Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp.
  1701–1708.

liu2015targeting
J. Liu, Y. Deng, T. Bai, Z. Wei, C. Huang, Targeting ultimate accuracy: Face
  recognition via deep embedding, arXiv preprint arXiv:1506.07310.

learned2016labeled
E. Learned-Miller, G. B. Huang, A. RoyChowdhury, H. Li, G. Hua, Labeled faces
  in the wild: A survey, in: Advances in Face Detection and Facial Image
  Analysis, Springer, 2016, pp. 189–248.

savchenko2016search
A. V. Savchenko, Search techniques in intelligent classification systems,
  Springer, 2016.

savchenko2015statistical
A. V. Savchenko, N. S. Belova, Statistical testing of segment homogeneity in
  classification of piecewise–regular objects, International Journal of
  Applied Mathematics and Computer Science 25 (4) (2015) 915–925.

tan2006face
X. Tan, S. Chen, Z.-H. Zhou, F. Zhang, Face recognition from a single image per
  person: A survey, Pattern recognition 39 (9) (2006) 1725–1745.

cao2013practical
X. Cao, D. Wipf, F. Wen, G. Duan, J. Sun, A practical transfer learning
  algorithm for face verification, in: Proceedings of the IEEE International
  Conference on Computer Vision, 2013, pp. 3208–3215.

parkhi2015deep
O. M. Parkhi, A. Vedaldi, A. Zisserman, Deep face recognition, in: BMVC,
  Vol. 1, 2015, p. 6.

wu2016light
X. Wu, R. He, Z. Sun, T. Tan, A light cnn for deep face representation with
  noisy labels, arXiv preprint arXiv:1511.02683.

savchenko2017deep
A. V. Savchenko, Deep convolutional neural networks and maximum-likelihood
  principle in approximate nearest neighbor search, in: 8th Iberian Conference
  on Pattern Recognition and Image Analysis (IbPRIA 2017), Springer, 2017, pp.
  42–49.

cui2013fusing
Z. Cui, W. Li, D. Xu, S. Shan, X. Chen, Fusing robust face region descriptors
  via multiple metric learning for face recognition in the wild, in:
  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
  (CVPR), 2013, pp. 3554–3561.

cevikalp2010face
H. Cevikalp, B. Triggs, Face recognition based on image sets, in: Proceedings
  of the IEEE International Conference on Computer Vision and Pattern
  Recognition (CVPR), 2010, pp. 2567–2573.

mian2013image
A. Mian, Y. Hu, R. Hartley, R. Owens, Image set based face recognition using
  self-regularized non-negative coding and adaptive distance metric learning,
  IEEE Transactions on Image Processing 22 (12) (2013) 5252–5262.

wolf2003kernel
L. Wolf, A. Shashua, Kernel principal angles for classification machines with
  applications to image sequence interpretation, in: Proceedings of the IEEE
  International Conference on Computer Vision and Pattern Recognition (CVPR),
  Vol. 1, 2003.

huang2013coupling
Z. Huang, X. Zhao, S. Shan, R. Wang, X. Chen, Coupling alignments with
  recognition for still-to-video face recognition, in: Proceedings of the IEEE
  International Conference on Computer Vision (ICCV), 2013, pp. 3296–3303.

liu2014toward
L. Liu, L. Zhang, H. Liu, S. Yan, Toward large-population face identification
  in unconstrained videos, IEEE Transactions on Circuits and Systems for Video
  Technology 24 (11) (2014) 1874–1884.

zhu2015still
Y. Zhu, Z. Zheng, Y. Li, G. Mu, S. Shan, G. Guo, Still to video face
  recognition using a heterogeneous matching approach, in: Proceedings of the
  IEEE International Conference on Biometrics Theory, Applications and Systems
  (BTAS), 2015, pp. 1–6.

yang2016neural
J. Yang, P. Ren, D. Chen, F. Wen, H. Li, G. Hua, Neural aggregation network for
  video face recognition, arXiv preprint arXiv:1603.05474 (accepted for
  CVPR17).

savchenko2017maximum
A. V. Savchenko, Maximum-likelihood approximate nearest neighbor method in
  real-time image recognition, Pattern Recognition 61 (2017) 459–469.

kullback1997information
S. Kullback, Information theory and statistics, Courier Corporation, 1997.

shakhnarovich2002face
G. Shakhnarovich, J. W. Fisher, T. Darrell, Face recognition from long-term
  observations, in: European Conference on Computer Vision, Springer, 2002, pp.
  851–865.

klare2015pushing
B. F. Klare, B. Klein, E. Taborsky, A. Blanton, J. Cheney, K. Allen,
  P. Grother, A. Mah, A. K. Jain, Pushing the frontiers of unconstrained face
  detection and recognition: Iarpa janus benchmark a, in: Proceedings of the
  IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp.
  1931–1939.

wolf2011face
L. Wolf, T. Hassner, I. Maoz, Face recognition in unconstrained videos with
  matched background similarity, in: Proceedings of the IEEE International
  Conference on Computer Vision and Pattern Recognition (CVPR), 2011, pp.
  529–534.

savchenko2012adaptive
A. V. Savchenko, Adaptive video image recognition system using a committee
  machine, Optical Memory and Neural Networks 21 (4) (2012) 219–226.

zhang2006weighted
Y. Zhang, A. M. Martínez, A weighted probabilistic approach to face
  recognition from multiple images and video sequences, Image and Vision
  Computing 24 (6) (2006) 626–638.

liu2003video
X. Liu, T. Chen, Video-based face recognition using adaptive hidden markov
  models, in: Proceedings of the IEEE International Conference on Computer
  Vision and Pattern Recognition (CVPR), Vol. 1, 2003.

huang2012benchmarking
Z. Huang, S. Shan, H. Zhang, S. Lao, A. Kuerban, X. Chen, Benchmarking
  still-to-video face recognition via partial and local linear discriminant
  analysis on cox-s2v dataset, in: Asian Conference on Computer Vision,
  Springer, 2012, pp. 589–600.

zhou2002face
S. Zhou, V. Krueger, R. Chellappa, Face recognition from video: A condensation
  approach, in: Proceedings of the IEEE International Conference on Automatic
  Face and Gesture Recognition, 2002, pp. 221–226.

kittler1998combining
J. Kittler, M. Hatef, R. P. Duin, J. Matas, On combining classifiers, IEEE
  transactions on pattern analysis and machine intelligence 20 (3) (1998)
  226–239.

zhu2013point
P. Zhu, L. Zhang, W. Zuo, D. Zhang, From point to set: Extend the learning of
  distance metrics, in: Proceedings of the IEEE International Conference on
  Computer Vision, 2013, pp. 2664–2671.

huang2014learning
Z. Huang, R. Wang, S. Shan, X. Chen, Learning euclidean-to-riemannian metric
  for point-to-set classification, in: Proceedings of the IEEE Conference on
  Computer Vision and Pattern Recognition, 2014, pp. 1677–1684.

schroff2015facenet
F. Schroff, D. Kalenichenko, J. Philbin, Facenet: A unified embedding for face
  recognition and clustering, in: Proceedings of the IEEE Conference on
  Computer Vision and Pattern Recognition, 2015, pp. 815–823.

chen2015end
J.-C. Chen, R. Ranjan, A. Kumar, C.-H. Chen, V. M. Patel, R. Chellappa, An
  end-to-end system for unconstrained face verification with deep convolutional
  neural networks, in: Proceedings of the IEEE International Conference on
  Computer Vision (ICCV) Workshops, 2015, pp. 118–126.

li2014eigen
H. Li, G. Hua, X. Shen, Z. Lin, J. Brandt, Eigen-pep for video face
  recognition, in: Proceedings of the Asian Conference on Computer Vision,
  Springer, 2014, pp. 17–33.

yang2017cortexnet
A. Canziani, E. Culurciello, Cortexnet: a generic network family for robust
  visual temporal representations, arXiv preprint arXiv:1706.02735.

wong2011patch
Y. Wong, S. Chen, S. Mau, C. Sanderson, B. C. Lovell, Patch-based probabilistic
  image quality assessment for face selection and improved video-based face
  recognition, in: Proceedings of the IEEE Computer Vision and Pattern
  Recognition Workshops (CVPRW), 2011, pp. 74–81.

burghouts2008distribution
G. Burghouts, A. Smeulders, J.-M. Geusebroek, The distribution family of
  similarity distances, in: Proceedings of the International Conference on
  Advances in neural information processing systems (NIPS), 2008, pp. 201–208.

harandi2014manifold
M. T. Harandi, M. Salzmann, R. Hartley, From manifold to manifold:
  Geometry-aware dimensionality reduction for spd matrices, in: European
  conference on computer vision, Springer, 2014, pp. 17–32.



