1

On the Capacity of Face Representation

arXiv:1709.10433v2 [cs.CV] 15 Feb 2018

Sixue Gong, Student Member, IEEE, Vishnu Naresh Boddeti, Member, IEEE
and Anil K. Jain, Life Fellow, IEEE
Abstract—Face recognition is a widely used technology with numerous large-scale applications, such as surveillance, social media
and law enforcement. There has been tremendous progress in face recognition accuracy over the past few decades, much of which
can be attributed to deep learning based approaches during the last five years. Indeed, automated face recognition systems are now
believed to surpass human performance in some scenarios. Despite this progress, a crucial question still remains unanswered: given a
face representation, how many identities can it resolve? In other words, what is the capacity of the face representation? A scientific
basis for estimating the capacity of a given face representation will not only benefit the evaluation and comparison of different face
representation methods, but will also establish an upper bound on the scalability of an automatic face recognition system. We cast the
face capacity estimation problem under the information theoretic framework of capacity of a Gaussian noise channel. By explicitly
accounting for two sources of representational noise: epistemic (model) uncertainty and aleatoric (data) variability, our approach is
able to estimate the capacity of any given face representation. To demonstrate the efficacy of our approach, we estimate the capacity
of a 128-dimensional deep neural network based face representation, FaceNet [1], and that of the classical Eigenfaces [2]
representation of the same dimensionality. Our numerical experiments on unconstrained faces indicate that, (a) our capacity estimation
model yields a capacity upper bound of 5.8×108 for FaceNet and 1×100 for Eigenface representation at a false acceptance rate (FAR)
of 1%, (b) the capacity of the face representation reduces drastically as you lower the desired FAR (for FaceNet representation; the
capacity at FAR of 0.1% and 0.001% is 2.4×106 and 7.0×102 , respectively), and (c) the empirical performance of the FaceNet
representation is significantly below the theoretical limit.
Index Terms—Face Recognition, Face Representation, Channel Capacity, Gaussian Noise Channel, Bayesian Inference

F

1

I NTRODUCTION

Face recognition has witnessed rapid progress and wide
applicability in a variety of practical applications: social
media, surveillance systems and law enforcement to name
a few. Fueled by copious amounts of data, ever growing computational resources and algorithmic developments,
current state-of-the-art face recognition systems are believed
to surpass human capability in certain scenarios [3]. Despite
this tremendous progress, a crucial question still remains
unaddressed, what is the capacity of a given face representation?
Tackling this question is the central aim of this paper.
Consider the following scenario where we would like to
deploy a face recognition system and can afford to tolerate
a given maximum false acceptance rate (FAR). With a single
user in our database, the face representation has a verification rate of 100%. As we add users to this system, it might
continue to exhibit 100% verification rate as long as their
representations do not overlap with each other. However,
at some point, the face representation’s verification rate is
likely to drop below 100%, or in other words, reach a limit
at which it can no longer completely resolve all of the users
within the tolerable FAR. We define the maximal number of
users at which the face representation reaches this limit as
the capacity1 of the representation.
The ability to determine this capacity affords the following benefits. As the deployment scale of face recognition
The authors are with the Department of Computer Science and Engineering, Michigan State University, East Lansing, MI, 48864 USA e-mail:
(gongsixu@msu.edu).
1. This is different from the notion of capacity of a space of functions
as measured by its VC dimension.

z

x

y

Fig. 1: An illustration of a face representation embedding,
where all the faces typically lie inside the population hyperellipsoid. The embedding of images belonging to each
identity or a class are ideally clustered into their own
class-specific hyper-ellipsoids. The capacity of a face representation is the number of identities (class-specific hyperellipsoids) that can be packed into the population hyperellipsoid of the embedding within an error tolerance.

systems grows larger (e.g., FBI face database [4] and Aadhar [5]), it is critical to obtain reliable statistical estimates
of the upper bound on the number of identities the face
representation can resolve. This would allow for informed
deployment of face recognition systems based on the ex-

2

Detection

Alignment

Normalization

Embedding Function Representation

..
.

y ∈ Rd
Fig. 2: A typical face representation pipeline comprises of face detection, alignment, normalization and representation
or feature extraction. While each of these components affect the capacity of the representation, in this paper, we focus on
the capacity of the embedding function that maps a high-dimensional normalized face image to a d-dimensional vector
representation.
pected scale of operation. Face representations are typically
compared by their recognition performance on benchmark
datasets. However, this metric of comparison conflates the
twin effects of the complexity2 and scale of the dataset into
a single number and does not showcase the full potential
and limitations of the representation. Capacity, on the other
hand, allows us to estimate the maximal operating scale
of the face representation as we vary the complexity of
the data without having to exhaustively evaluate the face
representation at that scale. Consequently, capacity offers an
alternative dataset3 agnostic metric for comparing different
face representations.
In this paper, we propose an approach to estimate the
capacity of a given face representation. Our solution relies
on the notion of capacity that has been well studied by the
information theory community in the context of wireless
communication [6]. The setting, commonly referred to as
the Gaussian channel, consists of a source signal x ∈ Rd
that is additively corrupted by Gaussian noise z ∈ Rd to
generate observations y ∈ Rd . The capacity of this Gaussian
channel is defined as the number of distinct source signals
in the signal representation. Figure 1 provides a pictorial
illustration of the geometrical structure of a face representation under the setting of a Gaussian channel. The capacity
of this Gaussian channel can be analyzed under various
assumptions for the distributions of the source signal x and
the noise z .
Despite the rich theoretical understanding of the capacity of a Gaussian channel, there has been limited practical
application of this theory in the context of estimating the
capacity of learned embeddings like face representations.
Many challenges contribute to this limited practical utility.
1)
2)

Estimating the distributions of the source Px and
the noise Pz for a high-dimensional embedding,
such as a face representation, is an open problem.
Accounting for the different sources of noise and
reliably inferring the probability distributions, especially in high dimensions (typically, several hundred), is a very challenging task. In the context of
face representations, all the components of a typical
face representation pipeline (see Fig. 2), including
face detection, landmarking, normalization and the
mapping from a high-dimensional image to the

2. Variations in facial appearance due to pose, illumination, expressions, occlusions etc.
3. Class of datasets as opposed to a specific dataset.

3)

4)

low-dimensional embedding are potential sources
of noise.
The Gaussian channel [6] setting makes independence4 i.e., x ⊥ z and additivity i.e., x + z assumptions. These assumptions do not hold in many
practical scenarios.
Existing capacity estimates for the Gaussian channel
are predicated upon assumptions on the Gaussianity of the distributions of x and z , assumptions
that are again difficult to justify in many practical
scenarios.

In this paper, we propose a framework that addresses the
aforementioned challenges to obtain reliable estimates of the
capacity of any given face representation. We leverage recent
advances in deep neural networks (DNNs) to estimate highdimensional distributions of Px and Pz from an observed
face representation y . Given an embedding function that
maps normalized high-dimensional facial images to a lowdimensional vector, we model two sources of uncertainty
that contribute to the noise in the embeddings: (i) uncertainty in the data, and (ii) uncertainty in the embedding
function. We note that, in this paper, we only focus on
the noise inherent to the embedding function and leave
the noise introduced by face detection, landmarking, and
normalization components for a future study. Finally, for
tractability purposes, the data and noise models for our
embedding are designed to conform to the Gaussian channel
setting, in terms of independence, additivity and Gaussianity5 , instead relying on the ability of deep neural networks to approximate complex non-linear functions. These,
rather simplifying, assumptions make the capacity estimates
tractable and enable the direct applicability of existing capacity estimates of Gaussian channels to the problem at
hand. As these assumptions are relaxed in future studies,
our capacity estimates will serve as a baseline. The key
technical contributions of this paper are:
1)

2)

A general approach for estimating highdimensional distributions of Gaussian channel
based embeddings i.e., Py , Px and Pz , given
samples Y from Py .
A noise model Pz for facial embeddings that explicitly accounts for uncertainty due to data and the

4. Px,z = Px Pz ∀x ∈ Rd and ∀z ∈ Rd
5. We provide empirical analysis on the Gaussianity of the embedding in the supplementary material.

3

TABLE 1: A summary of face recognition systems reported in the literature.
Authors

Probe
# Images
# classes

Taignman et al. [9]
Liao et al. [10]
Kemelmacher-Shlizerman et al. [11]
Yan et al. [12]
Best-Rowden et al. [13]
Wang et al. [14]
Schroff et al. [1]
Wen et al. [15]
Parikhi et al. [16]

4,000,000
8,707
100,000
16,028
9,074
4,350
N/A
N/A
N/A

4,000
4,249
500
466
596
1,000
N/A
N/A
N/A

220
120

N/A
12

Gallery
# Images
# classes

Training

Datasets
Verification

Performance

Verification (TAR @ 0.1% FAR)
13,113
1,000
690,572
116,028
4,249
1,000
N/A
N/A
N/A

5,749
1,000
1,027,060
466
4,249
1,000
N/A
N/A
N/A

WebD1
N/A
[1]
FRGC v2.0
COTS3
CASIA
Google
WebD1
SelfD2

LFW
LFW
MegaFace+FaceScrub
FRGC v2.0(Exp. 1)+SelfD2
LFW
LFW
LFW
LFW
LFW

97.35%
41.66%
98.00%
97.03%
89.00%
97.52%
99.63%
99.28%
98.95%

N/A
N/A

WebD +LFW
LFW

44.00%
18.60%

FERET(fb)+SelfD2
PCSO

99.58%
85.00%

Retrieval (Mean Average Precision)
Wu et al. [17]
Chen et al. [18]

1M+
13,113

N/A
5,749

Identification (Detection and Identification Rate @ RANK 1)
Yi et al. [19]
Klare et al. [20]
1
2
3

1,196
50,000

1,196
50,000

201,196
50,000

FERET
PCSO

WebD are datasets downloaded from the Internet and used to augment the gallery; different retrieval systems use their own web datasets
SelfD are datasets collected by authors of the papers to augment the gallery
COTS is a commercial off the shelf face recognition system

3)
4)

5)

uncertainty in the parameters of the representation
function.
The first practical attempt at estimating the capacity
of face representations.
Establishing a relationship between the capacity of
a Gaussian channel and the discriminant function
of a nearest neighbor classifier. Consequently, we
can estimate capacity as a function of the desired
operating point, in terms of the maximum desired
probability of false acceptance error.
An estimate of the capacity of a DNN based face
representation, namely FaceNet, consisting of 128dimensional feature vector.

Numerical experiments suggest that our proposed
model can provide reasonable estimates of face representation capacity, serving as a proxy for the discriminative
capability of face representations. Applying our approach
to a DNN based face representation, FaceNet, yields an
upper bound on the capacity of 2.0×108 and 2.4×106 for
constrained and unconstrained face recognition scenarios,
respectively, at a false accept rate (FAR) of 0.1%. This implies
that the representation should have a true accept rate (TAR)
of 100% at FAR of 0.1% over 2.0×108 and 2.4×106 subjects
under constrained and unconstrained scenarios. However,
empirically the FaceNet representation only achieves a TAR
of 50% at a FAR of 0.1% on the IJB-B [7] (unconstrained)
dataset over 1,845 subjects and a TAR of 95% at a FAR
of 0.1% on the LFW [8] (constrained) dataset over 5,749
subjects.

2

N/A
50,000

R ELATED W ORK

The subject of face recognition is as old as the field of
computer vision [21]. Not surprisingly, face recognition
has received tremendous attention in the computer vision
and biometrics communities over the past several decades.
While an exhaustive survey of the vast literature on facial
feature extraction and matching is beyond the scope of this
paper, we present a few notable approaches. Eigenfaces [2]
and Fisherfaces [22] are among the earliest learning based

approaches for face recognition that relied on second-order
statistics of the data, through principal component analysis
(PCA) and linear discriminant analysis (LDA), respectively.
Later on, feature representations to encode the local texture
in a face image became very popular for face recognition,
including local binary patterns (LBP) [23] and histogram of
oriented gradients (HoG) [24]. Developments in deep neural
network based representation learning have contributed to
massive strides in face recognition capabilities. The defining
characteristic of such methods is the use of convolutional
neural network (CNN) based feature extractor, a learnable
embedding function comprised of several sequential linear
and non-linear operators [25]. Taigman et al. [9] presented
the DeepFace system, a deep CNN trained to classify faces
using a dataset of 4 million example faces spanning 4,000
unique identities, that demonstrated remarkable performance on the Labeled Faces in the Wild (LFW) dataset [8].
Researchers from Google [1] used a massive dataset of about
200 million face images of 8 million identities to train a CNN
directly for face verification. They optimize a loss function
based on triplets of images comprising a pair of similar
and a pair of dissimilar faces. This model is currently the
state-of-the-art for face verification and achieves the best
performance on LFW and YouTube Faces dataset [26]. For
the benefit of the readers, we present an overview of recent
facial representation approaches in Table 1. While the focus
of the majority of the work in the literature has been on the
accuracy of facial recognition on benchmark datasets, our
goal in this paper is to characterize the maximal discriminative capacity of a given face representation at a specified
error tolerance.
Information theory has been widely used in the domains of data compression and channel coding [6]. Beyond
the studies on the capacity estimates of communication
channels [6], information theory has also been used in
signal processing to derive information-theoretic limits on
subspace classification. Erdogmus et al. [27] theoretically
studied how information transfer through a classifier affects its performance. They illustrated a theoretical use of
Renyi’s definition of information, extending Fano’s result,

4

to derive an upper bound on the probability of classification error. Motivated by applications in high-dimensional
signal processing, Nokleby et al. [28] derived fundamental
limits on the performance of compressive linear classifiers.
They identified a duality between classification (through
Gaussian mixture models) and communications over noncoherent multiple-antenna channels. And, in their followup work, Nokleby et al. [29] derived tighter upper bounds
on the classification of linear and affine subspaces from
noisy linear features, where the subspaces are modeled
by high-dimensional Gaussian distributions with approximately low-rank covariances. In contrast to the linear subspace based classifiers that are the target of these prior
studies, in this paper, we seek to estimate the capacity of
any given face representation embedding, including but not
limited to deep neural network based face representations.
Information theoretic concepts have also been used in
the context of analyzing biometric recognition systems.
Schmid et al. [30], [31] theoretically analyzed the performance of biometric systems, deriving bounds on the probability of error. Bhatnagar et al. [32] proposed an information theoretic approach to formulate performance indices
for biometric authentication. They obtained the capacity of
a biometric system following Shannon’s channel capacity
formulation along with a rate-distortion theory framework
to estimate the FAR. Wang et al. [33] proposed an approach
to model and predict the performance of a face recognition
system based on an analysis of the similarity scores. The
common theme across this entire body of work is that
the performance bounds of biometric systems are analyzed
purely based on the similarity scores obtained as part of
the matching process. In contrast, our work in this paper
directly analyzes the geometry of the representation space
of face recognition systems.
One of the key technical contributions of our paper is
the reliable estimation of the underlying probability distributions of noisy signal model Py , ideal signal model Px and
the noise model Pz from samples y . Gaussian Processes [34]
are a popular and powerful tool in statistics that allows us
to model distributions over functions, offering nice properties such as uncertainty estimates over function values,
robustness to over-fitting, and principled ways for hyperparameter tuning. A number of approaches have been proposed for modeling uncertainties in deep neural networks
[35], [36], [37]. Along similar lines, Kendall et al. [38] study
the benefits of explicitly modeling epistemic6 (model) and
aleatoric 7 (data) uncertainties [39] in Bayesian deep neural
networks for semantic segmentation and depth estimation
tasks. Drawing inspiration from this work, we account for
these two sources of uncertainties in the process of mapping a normalized facial image into a low-dimensional face
representation.
Apart from the face, there are a number of other physical
human traits that serve as biometric signatures for human
identification, most notably fingerprints and iris. Unlike the
face, capacity estimates to determine the uniqueness of these
two biometric modalities have already been established.
Pankanti et al. [40] derived an expression for estimating
6. Uncertainty due to lack of information about a process.
7. Uncertainty stemming from the inherent randomness of a process.

the probability of a false correspondence between minutiaebased representations from two arbitrary fingerprints belonging to two different fingers. Zhu et al. [41] later developed a more realistic model of fingerprint individuality
through a finite mixture model to represent the distribution
of minutiae in fingerprint images, including minutiae clustering tendencies and dependencies in different regions of
the fingerprint image domain. Daugman [42] proposed an
information theoretic approach to compute the capacity of
IrisCode. He first developed a generative model of IrisCode
based on Hidden Markov Models and then estimated the
capacity of IrisCode by calculating the entropy of this generative model. Adler et al. [43] proposed an information
theoretic measure of the information contained in a biometric representation, estimating the average information
contained within a face representation like Eigenfaces [2].
To the best of our knowledge, no such capacity estimation models have been proposed in the literature for stateof-the-art representations for face. Moreover, the distinct
nature of representations for fingerprint8 , iris9 and face10
based human recognition does not allow capacity estimation
approaches to carry over from one biometric modality to
another. Therefore, we believe that a customized model is
necessary to establish the capacity of face representations.

3

C APACITY OF FACE R EPRESENTATIONS

We first describe the setting of our problem and provide a
high-level outline of our approach. Our goal is to estimate
the capacity of a given face representation model, dubbed
teacher11 , that is assumed to be available in the form of a
black-box embedding function that maps an image s to a
low-dimensional embedding y . We first cast the teacher’s
face representation embedding process in the framework of
a Gaussian channel (Section 3.1). We then learn a student
model that mimics the teacher while also providing uncertainty estimates of the embedding, enabling us to estimate
the parameterized probability distributions, Py , Px and Pz ,
inherent to the Gaussian channel (Section 3.2). Then we
empirically estimate the capacity of the student model by
leveraging the theoretical framework for the capacity of a
Gaussian noise channel (Section 3.3). Finally, we establish a
relationship between the Gaussian channel and the discriminant function of a nearest neighbor classifier, allowing us
to obtain capacity estimates as a function of the probability
of false acceptance (Section 3.4). A pictorial outline of the
approach described above is shown in Fig. 3.
3.1

Face Representation Model

A face representation model M is a parametric embedding
function that maps a face image s of identity c to a, typically,
lower dimensional vector space y ∈ Rd , i.e., y = fM (s; θ),
where θ is the set of parameters of the embedding function.
For example, in the case of a linear embedding function
like Principal Component Analysis (PCA), the parameter set
8. An unordered collection of minutiae points.
9. A binary representation, called the iris code.
10. A fixed-length vector of real values.
11. We adopt the terminology of teacher-student models from the
model compression community [44].

5

Gaussian Channel

Teacher-Student Model

z ∼ N (0, Σz )


s

x1





z1



 . L . 
 .  .
 .  .
xd
zd
x = f (s)
x ∼ N (µx , Σx )

Teacher
(blackbox)
y

Capacity Estimate

yi
Σi

si

Σ2
c

Σ3
c

µi
Σxc

yi

Σ4
c

µi

Σ1
c

Student

y ∼ N (µy , Σy )

Σ5
c

Σi

Fig. 3: Overview of Face Representation Capacity Estimation: We cast the face representation process in the information
theoretic framework of a Gaussian noise channel. Given a black-box teacher face representation model, we learn a student
model that mimics this teacher while being able to provide explicit estimates of the uncertainty (noise) in the embedding due
to the data and the model. The uncertainty estimates at the image level are leveraged to generate empirical approximations
of the probability distributions of the various components of the Gaussian channel at the identity level, which in turn
facilitates an empirical estimate of the capacity of the teacher face representation.

θ would represent the eigenvectors. And, in the case of a
deep neural network based non-linear embedding function,
θ represents the parameters of the deep neural network.
The face embedding process can be approximately cast
within the framework of a Gaussian noise channel as
follows. Face representations y of an image s from the
teacher are modeled as observations of a true underlying
embedding x that is corrupted by noise z . The nature of
the relationship between these entities is determined by the
assumptions of a Gaussian channel, namely, (i) additivity
of the noise i.e., y = x + z , (ii) independence of the true
embedding and the additive noise, i.e., x ⊥ z , and (iii)
all entities, y , x and z follow a Gaussian distribution, i.e.,
Px ∼ N (µg , Σg ), Pz ∼ N (0, Σz ) and Py ∼ N (µy , Σy ).
Statistical estimates of these parameterized distributions
will enable us to compute the capacity of the teacher face
representation model as described in Section 3.3.
But, before we proceed further, we would like to point
out a limitation of our face representation model, namely
the Gaussianity assumption of the distribution Px . For a
given black-box face representation, in practice, the embeddings could potentially lie on an arbitrary and unknown
low-dimensional manifold. Approximating this manifold
through a normal distribution potentially over-estimates
the support of the embedding in Rd , resulting in an overestimation of the capacity of the representation. Similarly,
the Gaussianity assumption of the noise distribution Pz is
another limitation of our model. The potentially non-linear
nature of the embedding function would, in general, result
in an arbitrary noise distribution.
While this particular representation model suffers from
the aforestated drawbacks, it affords the practical advantage
of trading-off the realism and complexity of the theoretical
model and the complexity of the inferential algorithm12 .
Specifically, our representation model allows us to compen12. Such simplifying assumptions were also made by Pankanti et al.
[40] for fingerprints and Daugman [42] for iris to keep the problem
tractable. Zhu et al. [41] later relaxed the “uniformity” assumption on
minutiae distribution made by Pankanti et al.

sate for the lack of theoretical machinery for modeling the
manifolds induced by non-linear high-dimensional embedding functions by leveraging the functional approximation
capabilities, flexibility and computational efficiency of deep
neural networks as feed-forward inference machines. In
summary, these assumptions make the face capacity estimation problem tractable.
3.2 Estimating Uncertainties in Representations
We first build a probabilistic model for the space of noisy
embeddings y generated by a black-box facial representation model (teacher) Mt with parameters θ .
p(y|S ∗ , Y ∗ )

=
=

Z

p(y|x, S ∗ , Y ∗ )p(x|S ∗ , Y ∗ )dx
(1)
Z Z
p(y|x, θ)p(θ|S ∗ , Y ∗ )p(x|S ∗ , Y ∗ )dθdx

where Y ∗ = {y1 , . . . , yN } and S ∗ = {s1 , . . . , sN } are
the training samples to estimate the model parameters
θ , p(y|x, θ) is the aleatoric (data) uncertainty given a set
of parameters, p(θ|S ∗ , Y ∗ ) is the epistemic (model) uncertainty in the parameters given the training samples and
p(x|S ∗ , Y ∗ ) ∼ N (µg , Σg ) is the Gaussian approximation
of the underlying manifold of noiseless embeddings. Furthermore, we assume that the true mapping between the
image s and the noiseless embedding x is a deterministic
but unknown function i.e., x = f (s).
The black-box nature of the teacher model however only
provides D = {si , yi }N
i=1 , pairs of facial images si and
their corresponding noisy embeddings yi , a single sample
from the distribution p(y|S ∗ , Y ∗ ). Therefore, we learn a
student model Ms with parameters w to mimic the teacher
model. Specifically, the student model approximates the data
dependent aleatoric uncertainty p(yi |si , w) ∼ N (µi , Σi ),
where µi = xi = f (si ) represents the data dependent mean
noiseless embedding and Σi represents the data dependent
uncertainty around the mean. This student is an approximation of the unknown underlying probabilistic teacher, by
which an input image s generates noisy embeddings y of
ideal noiseless embeddings x, for a given set of parameters

6

w, i.e., p(yi |si , w) ≈ p(yi |xi , θ). Finally, we employ a variational distribution to approximate the epistemic uncertainty
of the teacher i.e., p(w|S ∗ , Y ∗ ) ≈ p(θ|S ∗ , Y ∗ ).

Learning: Given pairs of facial images and their corresponding embeddings from the teacher model, we learn a student
model to mimic the outputs of the teacher for the same
inputs in accordance to the probabilistic model described
above. We use parameterized functions, µi = f (si ; wµ )
and Σi = f (si ; wΣ ) to characterize the aleatoric uncertainty p(yi |si , w), where w = {wµ , wΣ }. We choose deep
neural networks, specifically convolutional neural networks
as our functions f (·; wµ ) and f (·; wΣ ). For the epistemic
uncertainty, while many deep learning based variational
inference [35], [45], [46] approaches have been proposed, we
use the simple interpretation of dropout as our variational
approximation [35]. Practically, this interpretation simply
characterizes the uncertainty in the deep neural network
weights w through a Bernoulli sampling of the weights.
We learn the parameters of our probabilistic model
φ = {wµ , wΣ , µg , Σg } through maximum-likelihood estimation i.e., minimizing the negative log-likelihood of the
observations Y = {y1 , . . . , yN }. This translates to optimizing a combination of loss functions:

min
φ

Ls + λLg + γLrs + δLrg

(2)

Similarly, we reparameterize the likelihood of the noiseless
embedding as a function of lg , the log variance along
each dimension. The regularization terms are
 also repaj
1 PN Pd
and Lrg =
exp
l
rameterized as, Lrs = 2N
i
j=1
i=1

1 Pd
j
j=1 exp lg .
2
Since the noise z is a normal distribution with zero-mean
in our Gaussian channel model, E(y) = E(x). Therefore,
1 PN
we empirically estimate µg as µg = N
i=1 yi . We estimate
the other parameters φ = {wµ , wΣ , Σg } through stochastic
gradient descent [47]. The gradients of the parameters are
computed by backpropagating [48] the gradients of the
outputs through the network.
Inference: The student model that has been learned can
now be used to infer the uncertainty in the embeddings
of the original teacher model. For a given facial image s,
the aleatoric uncertainty can be predicted by a feed-forward
pass of the image s through the network i.e., µ = f (s, wµ )
and Σ = f (s, wΣ ). The epistemic uncertainty can be approximately estimated through Monte-Carlo integration over
different samples of model parameters w. In practice the parameter sampling is performed through the use of dropout
at inference. In summary, the total uncertainty in the embedding of each facial image s is estimated by performing
Monte-Carlo integration over a total of T evaluations,

T
where λ, γ and δ are the weights for the different loss
1X t
µ̂
=
µ
(4)
1 PN
1
2
2
i
functions, Lrs = 2N i=1 kΣi kF and Lrg = 2 kΣg kF are
T t=1 i
the regularization terms and Ls is the loss function of the
T
T

T
1X t
1X t
student that captures the log-likelihood of a given noisy
µi − µ̂i µti − µ̂i +
Σ (5)
Σ̂i =
representation yi under the distribution N (µi , Σi ).
T t=1
T t=1 i
!
N
N
h
i
X
1
1X
−1
T
where µti and Σti are the predicted aleatoric uncertainty for
ln|Σi |+ Trace
Σi (yi − µi )(yi − µi )
Ls =
2 i=1
2
each feed-forward evaluation of the network.
i=1

Lg is the log-likelihood of the manifold of noiseless embeddings under the approximation by a multi-variate normal
distribution N (µg , Σg ).
!
N h
i
X
N
1
−1
T
Lg =
ln|Σg |+ Trace Σg
(yi − µg )(yi − µg )
2
2
i=1
For computational tractability we make a simplifying assumption on the covariance matrix Σ by parameterizing it
as a diagonal matrix i.e., the off-diagonal elements are set
to zero. This parametrization corresponds to independence
assumptions on the uncertainty along each dimension of the
embedding. The sparse parametrization of the covariance
matrix yields two computational benefits in the learning
process. Firstly, it is sufficient for the student to predict
only the diagonal elements of the covariance matrix. Secondly, positive semi-definitiveness constraints on a diagonal
matrix can be enforced simply by forcing all the diagonal
elements of the matrix to be non-negative. To enforce nonnegativity on each of the diagonal variance values, we
predict the log variance, lj = log σj2 . This allows us to reparameterize the student likelihood in terms of li :

2
j
j
N
d
N
d
1 X X j 1 X X yi − µi
 
Ls =
l +
(3)
2 i=1 j=1 i
2 i=1 j=1 exp lj
i

3.3

Capacity From Uncertainty Estimates

The goal of this paper is to estimate the capacity of a given
teacher face representation, where capacity is defined as the
maximal number of unique identities that can be resolved
by the representation. Given xc and yc , the respective noiseless population embedding and the noisy embeddings of a
representative class, the capacity of the face representation
channel is defined as,

C = max I(xc ; yc ),
p(xc )

(6)

RR
p(x,y)
is the mutual inforwhere I(x; y) =
p(x, y) log p(x)p(y)
mation between the random variables x and y . For general
distributions, it may not be possible to analytically calculate
the capacity of a channel. However, for the Gaussian channel that we consider in this paper, analytical expressions for
the capacity do exist. In fact, the capacity of this channel
has been extensively studied (we refer the reader to [6]
for a more in-depth treatment of this topic) across various
relaxations of the Gaussian distributions, namely, isotropic
Gaussian, axis aligned Gaussian and finally the more general case of anisotropic Gaussian distribution with colored
noise, the scenario that is most relevant to our model. Under

7

this general case, the capacity of the face representation
Gaussian channel is given by:

C

=

max I(xc ; yc )

p(xc )

(7)

|Σxc + Σzc |
1
log
Trace(Σxc )≤P 2
|Σzc |
|UzTc Σxc Uzc + Λzc |
1
=
max
log
Trace(Σxc )≤P 2
|Λzc |
|Σx¯c + Λzc |
1
=
max
log
|Λzc |
Trace(Σx̄c )≤P 2
=

max

where Σxc is the covariance matrix of the noiseless population embedding of classes, Σzc is the covariance matrix
of the embedding noise of a representative class, Uzc and
Λzc are the matrices with the eigenvectors and the eigenvalues of Σzc respectively. The capacity of this channel is
maximized when Σx̄c is diagonalized. Under the assumption of axis aligned Gaussian distribution (i.e., off-diagonal
elements of covariance matrix are zero) the expression for
the capacity of this channel simplifies as,
!
d
σxi c
1X
C = P max
log 1 + i
(8)
d
i
2 i=1
σzc
i=1 σxc ≤P

i
where σx
is the variance of the i-th dimension of the noisec
less population embedding and σzi c is the variance of the
i-th dimension of the embedding noise of a representative
class.
While we were able to extract uncertainty estimates of
each individual image in Section 3.2, we now need to estimate the distributions Pxc and Pzc , the distributions for the
noiseless population embedding of classes and the embedding noise of a representative class, respectively. We approximate each of these distributions by multi-variate normal
distributions, xc ∼ N (µxc , Σxc ) and zc ∼ N (0, Σzc ).
We empirically estimate the parameters of these distributions as follows. The mean of the population embedding is
PNc c
PC
µ̂i .
computed as µxc = C1 c=1 µ̂c , where µ̂c = N1c i=1
The covariance of the population embedding Σxc is estimated as,

Σ̃c = arg max Σ̂c +
Σ̂c

C
1 X c
(µ̂ − µxc )(µ̂c − µxc )T
C c=1

Σxc + Σzc = Σ̃c +

C
1 X c
(µ̂ − µxc )(µ̂c − µxc )T
C c=1

(9)

PNc c
where Σ̂c = N1c i=1
Σ̂i . Along the same lines, the covariance of the embedding noise of a class c is estimated as,

Σzc

=

Nc X
T h
i

T
1 X
µti − µ̂i µti − µ̂i + Σti
Nc T i=1 t=1

We use these estimates of Σxc and Σzc in Eq. 7 to compute
the capacity of the face representation.
3.4

Decision Theory and Model Capacity

The capacity of the face representation, in terms of the
Gaussian channel, described thus far can be alternatively
motivated from a geometrical perspective. Recall that our

representation space is composed of two components: the
noiseless population embedding of the classes approximated by a multi-variate Gaussian distribution and the embedding noise of each class approximated by a homoscedastic multi-variate Gaussian distribution. Under these settings, the decision boundaries between the classes13 that
minimizes the classification error rate are determined by
discriminant functions [49]. As illustrated in Fig. 4, for a
two-class problem, the discriminant function is a hyperplane in Rd with this optimal hyper-plane being equidistant
from both the classes. Moreover, the separation between the
classes determines the operating point and hence the FAR.
In the multi-class setting the optimal discriminant function
is the surface encompassed by all the pairwise hyper-planes,
which asymptotically reduces to a high-dimensional hyperellipsoid. The extent of this enclosing hyper-ellipsoid can be
determined by the desired operating point in terms of the
maximal error probability of false acceptance.
Under the multi-class setting, the capacity estimation
problem is equivalent to the geometrical problem of ellipse
packing, which seeks to estimate the maximum number
of small hyper-ellipsoids that can be packed into a larger
hyper-ellipsoid. In the context of face representations the
small hyper-ellipsoids correspond to the class-specific enclosing hyper-ellipsoids as described above while the large
hyper-ellipsoid corresponds to the entire space spanned by
the population of all classes. The volume V of a hyperellipsoid corresponding to a Mahalanobis distance r2 =
(x − µ)T Σ−1 (x − µ) with covariance matrix Σ is given
1
by the following expression, V = Vd |Σ| 2 rd , where Vd is the
volume of the d-dimensional hypersphere. An upper bound
on the capacity of the face representation can be computed
simply as the ratio of the volumes of the population and the
class-specific hyper-ellipsoids,


Vxc ,zc
(10)
C ≤
Vzc
!
1
Vd |Σxc + Σzc | 2 rxd c
=
1
Vd |Σzc | 2 rzdc
!
1
|Σxc + Σzc | 2 rxd c
=
1
|Σzc | 2 rzdc
!
1
|Σ̄xc ,zc | 2
=
1
|Σ̄zc | 2

where Vxc ,zc is the volume of population hyper-ellipsoid
and Vzc is the volume of the class-specific hyper-ellipsoid.
The size of the population hyper-ellipsoid rxc is chosen such
that a desired fraction of all the classes lie within the hyperellipsoid and rzc determines the size of the class-specific
hyper-ellipsoid. Σ̄xc ,zc and Σ̄zc are the effective sizes of
the enclosing population and class-specific hyper-ellipsoids
respectively. For each of the hyper-ellipsoidspthe effective
√
radius √
along the i-th principal direction is λ̄i = r λi ,
where λi is the radius of the original hyper-ellipsoid along
the same principal direction.
This geometrical interpretation of the capacity reduces
to the Shannon capacity in Eq. 7 when rxc = rzc . Con13. In the case of face recognition, each class is an identity (subject)
and the number of classes corresponds to the number of identities.

8

Receiver Operating Characteristics
R
F AR = ∞
y∗

1
exp
2πσ 2

q

(y−µ1 )2
−
2σ 2

!

Representation Discriminant Function
Two-Class Decision Boundary

dy

Shannon Capacity: y ∗ = µ1 + σ

p(ykci )

c1

Ω = {xkr2 ≥ (x − µ)T Σ−1 (x − µ)}
R

q = 1 − x∈Ω √ 1d
exp − 21 xT Σ−1 x dx

c2

Embedding Decision Boundary (Ω)

σ

µ1

y

∗

(2π) kΣk

Σ̄ = U Λ̄U T
p
√
λ̄i = r λi

where r2 = χ2 (1 − q, d)−1

µ2

y

Fig. 4: Decision Theory and Capacity: We illustrate the relation between capacity and the discriminant function
corresponding to a nearest neighbor classifier. Left: Depiction of the notion of decision boundary and probability of
false accept between two identical one dimensional Gaussian distributions. Shannon’s definition of capacity corresponds
to the decision boundary being one standard deviation away from the mean. Right: Depiction of the decision boundary
induced by the discriminant function of nearest neighbor classifier. Unlike in the definition of Shannon’s capacity, the size
of the ellipsoidal decision boundary is determined by the maximum acceptable false accept rate. The probability of false
acceptance can be computed through the cumulative distribution function of a χ2 (r2 , d) distribution.
sequently, in this instance, the choice of rxc for the population hyper-ellipsoid implicitly determines the boundary
of separation between the classes and hence the operating
false accept rate (FAR) of the embedding. For instance, when
computing the Shannon capacity of the face representation
choosing rxc such that 95% of the classes are enclosed
within the population hyper-ellipsoid would implicitly correspond to operating at a FAR of 5%.
However, very often, practical face recognition systems
need to operate at different false accept rates, dictated by
the desired level of security. The geometrical interpretation
of the capacity described in Eq. 10 directly enables us to
compute the representation capacity as a function of the
desired operating point as determined by its corresponding
false accept rate. The size of the population hyper-ellipsoid
rxc will be determined by the desired fraction of classes
to enclose or alternatively other geometric shapes like the
minimum volume enclosing hyper-ellipsoid or the maximum volume inscribed hyper-ellipsoid of a finite set of
classes, both of which correspond to a particular fraction
of the population distribution. Similarly, the desired false
accept rate q determines the size of the class-specific hyperellipsoid rzc .
Let Ω = {x | r2 ≥ (x−µ)T Σ−1 (x−µ)} be the enclosing
hyper-ellipsoid. Without loss of generality, assuming that
the class-specific hyper-ellipsoid is centered at the origin,
the false accept rate q can be computed as,


Z
xT Σ−1 x
1
p
exp −
dx (11)
q = 1−
2
(2π)d |Σ|
x∈Ω
1

Reparameterizing the integral as y = Σ− 2 x, we have Ω =
{y | r2 ≥ y T y} and,


Z
1
yT y
p
q =1−
exp −
dy
(12)
2
(2π)d
y∈Ω

where {y1 , . . . , yn } are independent standard normal random variables. The Mahalanobis distance r2 is distributed
according to the χ2 (r2 , d) distribution with d degrees of

freedom and 1 − q is the cumulative distribution function
of χ2 (r2 , d). Therefore, given the desired FAR q , the corresponding Mahalanobis distance rzc can be obtained from
the inverse CDF of the χ2 (rz2 , d) distribution. Along the
same lines, the size of the population hyper-ellipsoid rxc
2
, d) discan be estimated from the inverse CDF of the χ2 (rx
tribution given the desired fraction of classes to encompass.

4

N UMERICAL E XPERIMENTS

In this section we will evaluate the efficacy of the proposed
capacity model. We estimate the capacity of a deep neural
network based face representation model. Furthermore, we
estimate the capacity of the face representation using multiple datasets, corresponding to different practical scenarios.
4.1

Datasets

We first provide a brief description of the face datasets that
we use, both for learning the teacher and student models as
well as for estimating the capacity of the teacher. Figure 5
shows a few examples of the kind of faces in each dataset.
PCSO: A large collection of mugshot images (not in the
public domain) acquired from the Pinellas County Sheriffs
Office (PCSO), comprising of 1,447,607 images of 403,619
subjects. We use a subset of 84,164 images from 10,000
subjects for face verification and about 100,000 classes for
capacity estimation.
LFW [8]: 13,233 face images of 5,749 subjects, downloaded
from the web. These images exhibit limited variations in
pose, illumination, and expression, since only faces that
could be detected by the Viola-Jones face detector [50] were
included in the dataset. One limitation of this dataset is that
only 1,680 subjects among the total of 5,749 subjects have
more than one face image.
CASIA [51]: A large collection of labeled images downloaded from the web (based on names of famous personalities) typically used for training deep neural networks. It
consists of 494,414 images across 10,575 subjects, with an

9

(a) PCSO

(b) LFW

(c) CASIA

(d) IJB-A

(e) IJB-B

Fig. 5: Example images from each of the five datasets considered in our paper in increasing order of complexity.
average of about 500 face images per subject. This dataset is
used for training both the teacher and student models.
IJB-A [52]: IARPA Janus Benchmark-A (IJB-A) contains 500
subjects with a total of 25,813 images (5,399 still images
and 20,414 video frames), an average of 51 images per
subject. Compared to the LFW and CASIA datasets, the IJBA dataset is more challenging due to: i) full pose variation
making it difficult to detect all the faces using a commodity
face detector, ii) a mix of images and videos, and iii) wider
geographical variation of subjects. The face locations are
provided with the IJB-A dataset (and used in our experiments when needed).
IJB-B [7]: IARPA Janus Benchmark-B (IJB-B) dataset is a
superset of the IJB-A dataset consisting of 1,845 subjects
with a total of 76,824 images (21,798 still images and 55,026
video frames from 7,011 videos), an average of 41 images
per subject. Images in this dataset are labeled with ground
truth bounding boxes and other covariate meta-data such
as occlusions, facial hair and skin tone. A key motivation
for the IJB-B dataset is to make the face database less
constrained compared to the IJB-A dataset and have a more
uniform geographic distribution of subjects across the globe.
4.2

Face Representation Model

We consider the capacity of two different face representation models: (i) a deep neural network model, FaceNet
introduced by Schroff et al. [1], and (ii) the classical PCA
based EigenFaces [2] representation of image pixels. These
are illustrative of the two extremes of various face representations proposed in the literature with FaceNet providing
close to state-of-the-art recognition performance (see Table
1). The FaceNet representation is based on a non-linear
multi-layered deep convolutional network architecture introduced by Szegedy et al. [53]. The EigenFaces representation, in contrast, is a linear model for representing faces.
However, unlike the original FaceNet model, both of our
models are trained using the entire CASIA dataset.
Since the student model is purposed to mimic the teacher
model, we base the student network architecture (see supplementary material for exact network architecture) on the
teacher’s14 architecture with a few notable exceptions. First,
14. In the scenario where the teacher is a black-box model, the design
of the student network architecture needs more careful consideration
but it also affords more flexibility. See Fig. 3 for an illustration of this
process.

we introduce dropout before every convolutional layer of
the network, including all the convolutional layers of the
inception [54] and residual [55] modules. Second, the last
layer of the network is modified to generate two outputs µ
and Σ instead of the noisy embedding y , the output of the
teacher.
4.3

Implementation Details

Before we train the teacher and student networks, the face
images are pre-processed and normalized to a canonical
face image. The faces are detected and normalized using
the joint face detection and alignment system introduced by
Zhang et al. [56]. Given the facial landmarks, the faces are
normalized to a canonical image of size 182×182 from which
RGB patches of size 160×160 are extracted as the input to the
teacher and student networks. During training, these patches
are cropped randomly from the normalized image and also
flipped horizontally at random for data augmentation.
The teacher is trained through a combination of the
softmax loss for classification, and the center loss [15] that
minimizes the intra-class distance between the embeddings.
Training is performed through stochastic gradient descent
with Nesterov Momentum 0.9 and weight decay 0.0005. We
use a batch size of 16, a learning rate of 0.01 dropping by
a factor of 2 every 20 epochs and is trained for about 100
epochs. The teacher model also uses dropout (with dropout
probability set to 0.8) at training, but only in the final fullyconnected classification layer. As is standard practice, we do
not use dropout on the teacher model during inference.
The student is trained to minimize the loss function
defined in Eq. 2, where the hyper-parameters are chosen
through cross-validation. Training is performed through
stochastic gradient descent with Nesterov Momentum 0.9
and weight decay 0.0005. We used a batch size of 16, a
learning rate of 0.01 that is dropped by a factor of 2 every 20
epochs. We observed that it is sufficient to train the student
model for about 100 epochs for convergence. The student
model includes dropout with a probability of 0.05 after each
convolutional layer and with a probability of 0.8 after the
final fully-connected classification layer. At inference each
image is passed through the student network 1,000 times
as a way of performing Monte-Carlo integration through
the space of network parameters {wµ , wΣ }. These sampled
outputs are used to empirically estimate the mean and
covariance of the image embedding.

10

TABLE 2: Face Recognition Results for DNN, PCA and state-of-the-art.
Dataset
LFW
LFW (BLUFR)
PCSO
IJB-A
IJB-B

4.4

Teacher: DNN
0.1% FAR 1% FAR
95.50
98.20
93.90
98.51
96.01
99.16
45.92
70.26
48.31
74.47

Student: DNN
0.1% FAR 1% FAR
91.97
98.47
90.68
97.22
95.94
99.16
44.35
70.36
46.04
73.96

Teacher: PCA
0.1% FAR 1% FAR
2.07
9.80
0.81
4.3
4.89
15.03
0.31
1.87
1.16
4.87

Face Recognition Experiments

We evaluate and compare the performance of the teacher
and student models on the four test datasets that we consider, namely, LFW, PCSO, IJB-A and IJB-B. To evaluate the
student model we estimate the face representation through
Monte-Carlo integration. We pass each image through the
student model 1,000
times to extract {µi , Σi }1000
i=1 and com1 P1000
pute µ = 1000 i=1 µi as the representation. Following standard practice, we match a pair of representations
through the nearest neighbor classifier on the normalized
feature vectors i.e., by computing the euclidean distance
2
dij = kui − uj k2 between the normalized representations
xi
ui = kxi k2 , where xi is the un-normalized representation
from either the teacher or student model. We note that the
capacity model considered in this paper does not take into
account this normalization process i.e., the projection of the
un-normalized representations in Rd onto the surface of a
unit hyper-sphere in Rd , leading to overestimates of the face
representation capacity. We leave the topic of relaxing this
simplifying assumption, to obtain more accuracte capacity
estimates, to a future study.
We evaluate the face representation models on the LFW
unconstrained face dataset, using two protocols: the standard LFW protocol and the BLUFR protocol [10]. Since
there is no standard predefined protocol for evaluation on
the PCSO dataset, we use the BLUFR protocol. Lastly, the
protocol for the IJB-A and IJB-B datasets defines matching
between templates, where each template is composed of
possibly multiple images of the class. Following the protocol
in [14], we define the match score between templates as the
average of the match scores between all pairs of images in
the two templates.
Figure 6 and Table 2 report the performance of the teacher
and student models, both FaceNet and Eigenfaces, on each
of these datasets at different operating points. We make the
following observations: (1) The performance of DNN based
representation on PCSO and LFW, consisting largely of
frontal face images with minimal pose variations and facial
occlusions, is comparable to the state-of-the-art. However,
its performance on IJB-A and IJB-B, datasets with large pose
variations, is lower than state-of-the-art approaches. This
is due to the fact that unlike these methods we do not
fine-tune the DNN model on the IJB-A and IJB-B training
sets since our goal in the paper is estimate the capacity of
a generic face representation as opposed to achieving the
best verification performance on each individual datasets.
(2) Our results indicate that the student models are able
to mimic the teacher models very well as demonstrated
by the similarity of their recognition accuracies as well as
the receiving operating curves, and (3) As expected, the
DNN significantly outperforms the PCA based Eigenfaces
representation in terms of recognition accuracy.

4.5

Student: PCA
0.1% FAR 1% FAR
1.43
7.27
0.64
3.55
2.07
13.49
0.17
1.39
0.84
4.39

State-of-the-Art
0.1% FAR
1% FAR
N/A
99.77(accuracy, [57])
99.41 [57]
N/A
99.6
99.7 [14]
90.6
95.2 [58]
70.0
83.0 [7]

Capacity Estimates

Shannon Capacity: We estimate the capacity of the face
representations by evaluating Eq. 7. For each of the datasets
we empirically determine the shape and size of the population hyper-ellipsoid Σxc and the class-specific hyperellipsoids Σzc . These quantities are computed through the
predictions obtained by sampling the weights (wµ , wΣ ) of
the model, via dropout. We obtain 1,000 such predictions for
a given image, by feeding the image through the network a
1,000 different times with dropout. For robustness against
outliers15 we only consider classes with at least two images
per class for LFW and five images per class for all the other
datasets for the capacity estimates.
Although the expressions in Eq. 7 and Eq. 10 assume
that the shape of the class-specific embedding and the population embedding are hyper-ellipsoids, we obtain capacity
estimates for different modeling assumptions on the shape
of these entities. For instance the shapes could also be modeled as hyper-spheres corresponding to a diagonal covariance matrix with the same variance in each dimension. We
generalize the hyper-sphere model to an axis aligned hyperellipsoid corresponding to a diagonal covariance matrix
with possibly different variances along each dimension. For
the sake of modeling efficiency we make the same modeling
assumption for both the global shape of the embedding and
the embedding shape of each class.
The capacity estimates of the face representations, under
the Gaussian noise channel setting, are dependent on the
size of a canonical class-specific hyper-ellipsoid (the capacity expression Eq. 7 depends on Σzc ), that is representative
of a typical subject among all possible subjects the face
representation system is going to be used for. However,
the hyper-ellipsoid corresponding to each class could potentially be of a different size. For instance, in Fig. 1 each classspecific hyper-ellipsoid is of a different size, orientation
and shape. Defining or identifying a canonical subject, from
among all possible identities, is in itself a challenging task
and beyond the scope of this paper. We overcome this practical problem by reporting the capacity for different choices
of classes, i.e., ranging from classes with very low intra-class
variability and classes with very high intra-class variability.
We report estimates of the capacity for different datasets and
for each dataset select classes with the minimum, mean, median and maximum hyper-ellipsoid volume as our canonical
class-specific hyper-ellipsoid. Datasets whose class distribution is similar to the distribution of the data that was used
to train the face representation, are expected to exhibit low
intra-class uncertainty, while datasets with classes that are
out of the training distribution can potentially have high
15. We also removed classes with wrongly labeled images from the
CASIA and PCSO datasets.

100

100

80

80

80

80

60
40

Teacher(DNN)
Student(DNN)
Teacher(PCA)
Student(PCA)

20
0

10

1

100

101

False Accept Rate (%)

102

60
40

Teacher(DNN)
Student(DNN)
Teacher(PCA)
Student(PCA)

20
0
10

(a) LFW Protocol

2

10

1

100

101

False Accept Rate (%)

60
40

Teacher(DNN)
Student(DNN)
Teacher(PCA)
Student(PCA)

20
0
10

102

Verification Rate (%)

100

Verification Rate (%)

100

Verification Rate (%)

Verification Rate (%)

11

6

10

5

10

(b) LFW BLUFR

10

4

3

10

2

10

1

100

False Accept Rate (%)

101

60
40
20
0

102

Teacher(DNN)
Student(DNN)
Teacher(PCA)
Student(PCA)

(c) PCSO

10

4

10

3

10

2

10

1

100

False Accept Rate (%)

101

102

(d) IJB-B

Fig. 6: Face recognition performance of the teacher and student models on the various datasets. We report the performance
of both DNN and PCA based face representations, (a) face verification on the LFW dataset evaluated through the standard
protocol, (b) face verification on the LFW dataset evaluated through the BLUFR protocol, (c) face verification on a subset
of the PCSO dataset (84,164 images of 10,000 classes), (d) face verification on the IJB-B dataset.
TABLE 3: Capacity of Face Representation Model
Dataset

Examples

Noise

Hypersphere

Hyper-ellipsoid
(Axis Aligned)

Hyper-ellipsoid

Model

DNN

PCA

DNN

PCA

DNN

PCA

PCSO

Min
Mean
Median
Max

2.3×100
1.4×100
1.4×100
1.0×100

6.8×1038
3.4×1014
7.0×1022
2.0×1012

2.3×100
1.4×100
1.4×100
1.0×100

2.7×1028
6.3×1021
9.0×1021
1.1×1015

2.3×100
1.4×100
1.4×100
1.0×100

LFW

Min
Mean
Median
Max

5.9×1040
2.3×1014
3.9×1023
2.0×1012

CASIA

Min
Mean
Median
Max

4.2×1031
2.5×108
4.1×1014
5.6×106

IJB-A

Min
Mean
Median
Max

1.9×100
1.4×100
1.4×100
1.0×100

IJB-B

Min
Mean
Median
Max

Dataset Pool

Min
Mean
Median
Max

1.9×1042
3.6×1011
8.4×1024
8.9×109

2.8×100
1.7×100
1.7×100
1.0×100

6.1×1024
1.8×108
3.4×1012
1.3×107

1.2×101
1.5×100
1.5×100
1.0×100

3.9×1032
1.8×108
1.5×1015
2.9×106

7.7×1030
1.3×108
1.7×1013
5.8×106
8.3×1045
2.1×1011
5.4×1025
9.0×108

intra-class uncertainty, and consequently lower capacity.
Figure 7 show examples of the images corresponding to the
lowest and highest intra-class variability in each dataset.
Table 3 reports the Shannon capacity (corresponding to
1% FAR) of the DNN and PCA based face representation
estimated on various datasets and the pooled collection of
data across different modeling assumptions. We make the
following observations from our numerical results,
DNN and PCA: The capacity of the FaceNet representation
is significantly higher than the capacity of Eigenfaces, with
the former and later being of the order of ≈ 1010 and 100
respectively. The Eigenfaces representation based on linear
projections of the raw pixel values is unable to scale beyond
a handful of identities, while FaceNet representation based
on learned non-linear functions is able to resolve significantly more number of identities. The relative difference in
the capacity is also reflected in the vast difference in the
verification performance between the two representations.

2.7×100
1.7×100
1.7×100
1.0×100

1.3×101
1.8×100
1.8×100
1.0×100

3.8×1041
3.2×1011
3.5×1026
7.9×109

2.8×100
1.7×100
1.7×100
1.0×100

7.8×1028
1.6×1023
2.6×1023
2.1×1013

1.9×1025
6.7×107
4.5×1012
4.1×106

1.2×101
1.5×100
1.5×100
1.0×100

2.1×1020
1.4×1014
1.5×1014
2.9×108

1.2×101
1.5×100
1.5×100
1.0×100

3.4×1044
1.5×1011
5.8×1025
9.9×108

1.3×101
1.8×100
1.8×100
1.0×100

2.5×1031
3.2×1023
1.0×1024
4.8×1010

1.3×101
1.8×100
1.8×100
1.0×100

4.2×1030
1.4×108
3.2×1013
4.8×106

1.9×100
1.4×100
1.4×100
1.0×100

2.7×100
1.7×100
1.7×100
1.0×100

2.9×1024
1.6×1016
2.3×1016
5.1×108

7.5×1022
1.1×1016
1.5×1016
5.8×108

2.8×100
1.7×100
1.7×100
1.0×100

1.9×100
1.4×100
1.4×100
1.0×100

2.7×100
1.7×100
1.7×100
1.0×100

Capacity Estimates: The upper bound on the capacity estimate of the FaceNet model in constrained scenarios (LFW
and PCSO) is ≈ 1014 , in unconstrained environments (CASIA and IJB) is ≈ 108 and ≈ 1010 over the pooled collection of all these datasets under the general model of a
hyper-ellipsoid. Therefore, theoretically, the representation
should be to resolve 1014 , 108 and 1010 subjects with a
true acceptance rate (TAR) of 100% at a FAR of 1% under
the constrained, unconstrained and aggregate operational
settings, respectively. While this capacity estimate is on
the order of the population of the earth, in practice, the
performance of the representation is lower than the theoretical performance, about 95% across only 10,000 subjects
in the constrained and only 50% across 1,845 subjects in the
unconstrained scenarios. These results suggest a significant
room for improvement in the empirical performance of face
recognition systems, especially under unconstrained scenarios. The relative order of the capacity estimates mimics the

12

(a) PCSO

(b) LFW

(c) CASIA

(d) IJB-A

(e) IJB-B

Fig. 7: Example images of classes that correspond to different sizes of the class-specific hyper-ellipsoids, based on the
FaceNet representation, for each of the five datasets considered in the paper. Top Row: Images of the class with the
largest class-specific hyper-ellipsoid for each database. Notice that in the case of database with predominantly frontal faces
(PCSO and LFW), large variations in facial appearance lead to the greatest uncertainty in the class representation. On
more challenging datasets (CASIA, IJB-A, IJB-B), the face representation exhibits most uncertainty due to pose variations.
Bottom Row: Images of the class with the smallest class-specific hyper-ellipsoid for each database. As expected, across all
the datasets, frontal face images with the minimal change in appearance result in the least amount of uncertainty in the
class representation. More examples like these can be found in the supplementary material.
relative order of the verification accuracy on these datasets.
Data Bias: Even within each dataset, the capacity estimates
exhibit significant variations depending on the choice of
the size of the class-specific hyper-ellipsoid. Whilst these
variations are a consequence of the simplicity of our capacity estimation model and the inhomogeneity of distribution
of samples from each class, they help establish an upper
bound on the capacity of the representation. Empirically,
we have observed that classes with the highest capacity
(smallest hyper-ellipsoid) are typically classes with very
few images and very little variation in facial appearance.
Similarly, classes with high intra-class uncertainty, with low
capacity estimates, are typically classes with a very large
number of images spanning a wide range of variations
in pose, expression, illumination conditions etc., variations
that one can expect under any real-world deployments of
face recognition systems. Coupled with the fact that the
capacity of the face representation is estimated from a very
small sample of the population (less than 11,000 subjects),
we argue that capacity estimates with large intra-class
uncertainty (within the datasets considered in this paper)
are an accurate representation of a canonical subject’s face
representation in unconstrained real-world deployments of
face recognition systems.
Gaussian Distribution Parameterization: The capacity estimates of the anisotropic Gaussian (hyper-ellipsoid) are two
orders of magnitude higher than the capacity estimates of
the reduced approximations, hyper-sphere (isotropic Gaussian) and axis-aligned hyper-ellipsoid. At the same time, the
isotropic and the axis-aligned hyper-ellipsoid approximations result in very similar capacity estimates.
Capacity at Specified FAR: We extend the point estimates of
Shannon’s capacity described so far to establish capacity as
a function of different operating points, as defined by differ-

ent false accept rates. We define rxc and rzc corresponding
to the desired operating points and evaluate Eq. 10. In
all our experiments we choose rxc to encompass 99% of
the classes within the population hyper-ellipsoid. Different
FARs define different decision boundary contours that, in
turn, define the size of the class-specific hyper-ellipsoid.
Figure 8 shows how the capacity of the representation
changes as a function of the FARs for different datasets.
We note that at the operating point of F AR = 0.1%, the
capacity of the maximum face representation is ≈ 1011 in the
constrained and ≈ 106 in the unconstrained case. However,
at stricter operating points (FAR of 0.001% or 0.0001%), that
is more meaningful at larger scales of operation [11], the
capacity of the FaceNet representation is significantly lower
(700 and 27 respectively for IJB) than the typical desired
scale of operation of face recognition systems.

5

C ONCLUSION

In this paper, we make the first attempt to estimate the
capacity of any given face representation. We approximate
the face representation process as a Gaussian noise channel and explicitly account for the different sources of uncertainty in the representation. We proposed an efficient
approach to infer the states of this representation channel
and estimate its capacity under the information theoretic
notion of capacity of a Gaussian channel. We also extended
our model to establish a relation between the capacity of
the Gaussian noise channel and the false accept rate of a
nearest neighbor classifier. Finally, we empirically estimated
the capacity, under the Gaussian noise channel, of two face
representations: a deep neural network, FaceNet and classical Eigenfaces. Numerical results yield an upper bound of
4.8×1010 for FaceNet and 1×100 for Eigenfaces at a FAR of
1% and a large gap between the theoretical and empirical

13

5

10 4 10 3 10
False Accept Rate

2

10

1

(a) CASIA

1021
1017
1013
109
105
101
10 3
10

10

Hypersphere
Axis Aligned Hyper-ellipsoid
Hyper-ellipsoid

6

10

5

10 4 10 3 10
False Accept Rate
(d) IJB-A

2

10

1

Hypersphere
Axis Aligned Hyper-ellipsoid
Hyper-ellipsoid

Capacity

Capacity
6

1024
1020
1016
1012
108
104
100
10

6

10

5

10 4 10 3 10
False Accept Rate

2

10

10

1

1023
1019
1015
1011
107
103
10 1
10

Hypersphere
Axis Aligned Hyper-ellipsoid
Hyper-ellipsoid

6

10

5

10 4 10 3 10
False Accept Rate
(e) IJB-B

2

10

Hypersphere
Axis Aligned Hyper-ellipsoid
Hyper-ellipsoid

1023
1019
1015
1011
107
103

1

(b) LFW

1021
1017
1013
109
105
101
10 3
10

1027

Capacity

10

Capacity

Hypersphere
Axis Aligned Hyper-ellipsoid
Hyper-ellipsoid

1017
1013
109
105
101
10 3

Capacity

Capacity

1021

6

10

5

10 4 10 3 10
False Accept Rate

2

10

1

2

10

1

(c) PCSO
Hypersphere
Axis Aligned Hyper-ellipsoid
Hyper-ellipsoid

6

10

5

10 4 10 3 10
False Accept Rate

(f) Dataset Pool

Fig. 8: For each of the datasets (as well as the dataset pool) under consideration, we estimate the capacity of the deep neural
network at different false accept rates corresponding to different operating points. Under the limit, the capacity tends to
zero as the FAR tends to zero. Similarly, the capacity tends to ∞ as the FAR tends to 1.0. Exact capacity values for different
FAR are available in the supplementary material.
verification performance of the representations. At lower
FAR of 0.001%, the FaceNet capacity drops-off significantly
to only 700 under unconstrained scenarios, impairing the
scalability of the face representation.
As face recognition technology makes rapid strides in
performance and witnesses wider adoption, quantifying
the capacity of a given face representation is an important
problem, both from an analytical as well as from a practical
perspective. However, due to the challenging nature of the
technical problem we make simplifying assumptions, for
tractability purposes, in this first attempt. Our experimental
results demonstrate that even this simplified model is able to
provide reasonable capacity estimates of a high performance
DNN based face representation. Relaxing the assumptions
of the approach presented here is an exciting direction of
future work, leading to more realistic capacity estimates.

R EFERENCES
[1]
[2]
[3]
[4]
[5]
[6]
[7]

F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified
embedding for face recognition and clustering,” in CVPR, 2015.
M. A. Turk and A. P. Pentland, “Face recognition using eigenfaces,” in CVPR, 1991.
C. Lu and X. Tang, “Surpassing human-level face verification
performance on lfw with gaussianface,” in AAAI, 2015.
U. S. G. A. Office, “Face recognition technology: FBI should
better ensure privacy and accuracy,” 2016. [Online]. Available:
http://www.gao.gov/assets/680/677098.pdf
UIDAI, “Aadhar,” 2017. [Online]. Available: https://portal.uidai.
gov.in/uidwebportal/dashboard.do
T. M. Cover and J. A. Thomas, Elements of Information Theory. John
Wiley & Sons, 2012.
C. Whitelam, E. Taborsky, A. Blanton, B. Maze, J. Adams, T. Miller,
N. Kalka, A. K. Jain, J. A. Duncan, K. Allen et al., “IARPA janus
benchmark-b face dataset,” in CVPRW, 2017.

[8]

[9]
[10]
[11]
[12]
[13]

[14]
[15]
[16]
[17]

[18]
[19]
[20]
[21]

G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeled
faces in the wild: A database for studying face recognition in
unconstrained environments,” Technical Report 07-49, University
of Massachusetts, Amherst, Tech. Rep., 2007.
Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “Deepface: Closing
the gap to human-level performance in face verification,” in CVPR,
2014.
S. Liao, Z. Lei, D. Yi, and S. Z. Li, “A benchmark study of largescale unconstrained face recognition,” in IJCB, 2014.
I. Kemelmacher-Shlizerman, S. M. Seitz, D. Miller, and E. Brossard,
“The megaface benchmark: 1 million faces for recognition at
scale,” in CVPR, 2016.
J. Yan, Z. Lei, D. Yi, and S. Z. Li, “Towards incremental and large
scale face recognition,” in IJCB, 2011.
L. Best-Rowden, H. Han, C. Otto, B. F. Klare, and A. K. Jain,
“Unconstrained face recognition: Identifying a person of interest
from a media collection,” IEEE Transactions on Information Forensics
and Security, vol. 9, no. 12, pp. 2144–2157, 2014.
D. Wang, C. Otto, and A. K. Jain, “Face search at scale: 80
million gallery,” IEEE Transactions on Pattern Analysis and Machine
Intelligence, vol. 39, no. 6, pp. 1122 – 1136, 2017.
Y. Wen, K. Zhang, Z. Li, and Y. Qiao, “A discriminative feature
learning approach for deep face recognition,” in ECCV. Springer,
2016.
O. M. Parkhi, A. Vedaldi, and A. Zisserman, “Deep face recognition.” in BMVC, 2015.
Z. Wu, Q. Ke, J. Sun, and H.-Y. Shum, “Scalable face image retrieval with identity-based quantization and multireference
reranking,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 10, pp. 1991–2001, 2011.
B.-C. Chen, Y.-Y. Chen, Y.-H. Kuo, and W. H. Hsu, “Scalable face
image retrieval using attribute-enhanced sparse codewords,” IEEE
Transactions on Multimedia, vol. 15, no. 5, pp. 1163–1173, 2013.
D. Yi, Z. Lei, Y. Hu, and S. Z. Li, “Fast matching by 2 lines of code
for large scale face recognition systems,” arXiv:1302.7180, 2013.
B. F. Klare, A. Blanton, and B. Klein, “Efficient face retrieval using
synecdoches,” in IJCB, 2014.
A. K. Jain, K. Nandakumar, and A. Ross, “50 years of biometric re-

14

[22]

[23]
[24]
[25]
[26]
[27]
[28]

[29]

[30]

[31]

[32]
[33]

[34]
[35]
[36]
[37]
[38]
[39]
[40]
[41]
[42]
[43]
[44]
[45]
[46]
[47]

search: Accomplishments, challenges, and opportunities,” Pattern
Recognition Letters, vol. 79, pp. 80–105, 2016.
P. N. Belhumeur, J. P. Hespanha, and D. J. Kriegman, “Eigenfaces
vs. fisherfaces: Recognition using class specific linear projection,”
IEEE Transactions on Pattern Analysis and Machine Intelligence,
vol. 19, no. 7, pp. 711–720, 1997.
T. Ahonen, A. Hadid, and M. Pietikäinen, “Face recognition with
local binary patterns,” in ECCV, 2004.
A. Albiol, D. Monzo, A. Martin, J. Sastre, and A. Albiol, “Face
recognition using hog–ebgm,” Pattern Recognition Letters, vol. 29,
no. 10, pp. 1537–1543, 2008.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in NIPS, 2012.
L. Wolf, T. Hassner, and I. Maoz, “Face recognition in unconstrained videos with matched background similarity,” in CVPR,
2011.
D. Erdogmus and J. C. Principe, “Lower and upper bounds for
misclassification probability based on renyi’s information,” The
Journal of VLSI Signal Processing, vol. 37, no. 2, pp. 305–317, 2004.
M. Nokleby, R. Calderbank, and M. R. Rodrigues, “Informationtheoretic limits on the classification of gaussian mixtures: Classification on the grassmann manifold,” in Information Theory Workshop, 2013, pp. 1–5.
M. Nokleby, M. Rodrigues, and R. Calderbank, “Discrimination
on the grassmann manifold: Fundamental limits of subspace classifiers,” IEEE Transactions on Information Theory, vol. 61, no. 4, pp.
2133–2147, 2015.
N. A. Schmid and J. A. O’Sullivan, “Performance prediction
methodology for biometric systems using a large deviations approach,” IEEE Transactions on Signal Processing, vol. 52, no. 10, pp.
3036–3045, 2004.
N. A. Schmid, M. V. Ketkar, H. Singh, and B. Cukic, “Performance
analysis of iris-based identification system at the matching score
level,” IEEE Transactions on Information Forensics and Security, vol. 1,
no. 2, pp. 154–168, 2006.
J. Bhatnagar and A. Kumar, “On estimating performance indices
for biometric identification,” Pattern Recognition, vol. 42, no. 9, pp.
1803–1815, 2009.
P. Wang, Q. Ji, and J. L. Wayman, “Modeling and predicting face
recognition system performance based on analysis of similarity
scores,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, no. 4, pp. 665–670, 2007.
C. E. Rasmussen and C. K. Williams, Gaussian Processes for Machine
Learning. MIT Press Cambridge, 2006, vol. 1.
Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,”
arXiv:1506.02142, vol. 2, 2015.
——, “A theoretically grounded application of dropout in recurrent neural networks,” in NIPS, 2016.
——, “Bayesian convolutional neural networks with bernoulli
approximate variational inference,” arXiv:1506.02158, 2015.
A. Kendall and Y. Gal, “What uncertainties do we need in bayesian
deep learning for computer vision?” in NIPS, 2017.
A. Der Kiureghian and O. Ditlevsen, “Aleatory or epistemic? does
it matter?” Structural Safety, vol. 31, no. 2, pp. 105–112, 2009.
S. Pankanti, S. Prabhakar, and A. K. Jain, “On the individuality
of fingerprints,” IEEE Transactions on Pattern Analysis and Machine
Intelligence, vol. 24, no. 8, pp. 1010–1025, 2002.
Y. Zhu, S. C. Dass, and A. K. Jain, “Statistical models for assessing
the individuality of fingerprints,” IEEE Transactions on Information
Forensics and Security, vol. 2, no. 3, pp. 391–401, 2007.
J. Daugman, “Information theory and the iriscode,” IEEE Transactions on Information Forensics and Security, vol. 11, no. 2, pp. 400–
409, 2016.
A. Adler, R. Youmaran, and S. Loyka, “Towards a measure of
biometric feature information,” Pattern Analysis and Applications,
vol. 12, no. 3, pp. 261–270, 2009.
J. Ba and R. Caruana, “Do deep nets really need to be deep?” in
NIPS, 2014.
D. P. Kingma, T. Salimans, and M. Welling, “Variational dropout
and the local reparameterization trick,” in NIPS, 2015.
C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra,
“Weight uncertainty in neural networks,” arXiv:1505.05424, 2015.
L. Bottou, “Large-scale machine learning with stochastic gradient
descent,” in Proceedings of COMPSTAT’2010. Springer, 2010, pp.
177–186.

[48] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning
representations by back-propagating errors,” Cognitive Modeling,
vol. 5, no. 3, p. 1, 1988.
[49] R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification. John
Wiley & Sons, 2012.
[50] P. Viola and M. J. Jones, “Robust real-time face detection,” International Journal of Computer Vision, vol. 57, no. 2, pp. 137–154, 2004.
[51] D. Yi, Z. Lei, S. Liao, and S. Z. Li, “Learning face representation
from scratch,” arXiv:1411.7923, 2014.
[52] B. F. Klare, B. Klein, E. Taborsky, A. Blanton, J. Cheney, K. Allen,
P. Grother, A. Mah, and A. K. Jain, “Pushing the frontiers of unconstrained face detection and recognition: IARPA janus benchmark
A,” in CVPR, 2015.
[53] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. Alemi, “Inceptionv4, inception-resnet and the impact of residual connections on
learning,” arXiv:1602.07261, 2016.
[54] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov,
D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with
convolutions,” in CVPR, 2015.
[55] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for
image recognition,” in CVPR, 2016.
[56] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, “Joint face detection
and alignment using multitask cascaded convolutional networks,”
IEEE Signal Processing Letters, vol. 23, no. 10, pp. 1499–1503, 2016.
[57] J. Liu, Y. Deng, T. Bai, Z. Wei, and C. Huang, “Targeting ultimate accuracy: Face recognition via deep embedding,”
arXiv:1506.07310, 2015.
[58] R. Ranjan, C. D. Castillo, and R. Chellappa, “L2-constrained softmax loss for discriminative face verification,” arXiv:1703.09507,
2017.
Sixue Gong received her bachelor degree from
Communication University of China in 2015.
Since 2016, she has been working towards a
Ph.D. degree in the Department of Computer
Science and Engineering at Michigan State University. Her research interests include pattern
recognition, image processing, and computer vision, with applications to face recognition.

Vishnu Naresh Boddeti is currently an Assistant Professor in the computer science department at Michigan State University. He received
a Ph.D. degree in Electrical and Computer Engineering program at Carnegie Mellon University
in 2007. His research interests are in Computer
Vision, Pattern Recognition and Machine Learning. He received the best paper award at the
BTAS conference in 2013.

Anil Jain is a University Distinguished Professor in the Department of Computer Science at
Michigan State University. His research interests
include pattern recognition, computer vision and
biometric authentication. He served as a member of the United States Defense Science Board
and the Forensic Science Standards Board. He
has received Fulbright, Guggenheim, Alexander
von Humboldt, and IAPR King-Sun Fu awards
and is a Fellow of ACM and IEEE. He is a
member of the United States National Academy
of Engineering and Foreign Fellow of the Indian National Academy of
Engineering.

1

On the Capacity of Face Representation
(Supplementary Material)
Sixue Gong, Student Member, IEEE, Vishnu Naresh Boddeti, Member, IEEE
and Anil K. Jain, Life Fellow, IEEE

F

In this supplementary material we include additional
analysis and results that could not be included in the main
paper due to space constraints.

1

via maximimum likelihood estimation over N samples.

µ =

C APACITY E STIMATES : A TOY E XAMPLE

In this section we provide an example to demonstrate the
capacity estimation process given a constellation of classes
in a two-dimensional representation space. The distribution
of the population space of classes (class centers to be specific) is modelled as a multi-variate normal distribution. The
feature representations of each class itself are modelled as a
two-dimensional normal distribution.
Figure 1 provides an illustration of the representation
space showing the population space and four different
classes corresponding to classes with the minimum, mean,
median and maximum volume from among the 100 population classes that were sampled. We also illustrate the
different approximations of the population and class distributions, isotropic normal (sphere), axis-aligned normal
(axis-aligned ellipse) and general normal (ellipse).
Table 1 shows the capacity estimates for each of these
classes and for the different approximations (hypersphere,
axis-aligned hyperellipsoid and hyperellipsoid) of the representation space. These capacity estimates were computed
as follows: we sample 100 different classes from the ground
truth population distribution and for each of these classes
we sample the representations from the ground truth distribution of that class. From these samples, we estimate the
covariance matrix of the population space distribution and
that of the individual classes under the different approximations of a multivariate normal distribution.
Isotropic Normal Distribution We estimate the parameters
of the isotropic normal distribution (µ1 , µ2 , σ ) via maximimum likelihood estimation over N samples.

µ =
min
σ

N
1 X i
x
(1)
N i=1

N 
X
(xi1 − µ1 )2 + (xi2 − µ2 )2
+
log
σ
(2)
2σ 2
i=1

Axis-Aligned Normal Distribution We estimate the parameters of the axis-aligned normal distribution (µ1 , µ2 , σ1 , σ2 )

min

σ1 ,σ2

N
1 X i
x
(3)
N i=1


1 (xi1 − µ1 )2
(xi2 − µ2 )2
+
+
log
σ
σ
(4)
1
2
N
2σ12
2σ22

Normal Distribution We estimate the parameters of a multivariate normal distribution (µ, Σ) via maximimum likelihood estimation over N samples.

µ =
min
Σ

N
1 X i
x
N i=1

(5)

N
i
1 Xh
(x − µ)T Σ−1 (x − µ) + log|Σ| (6)
2N i=1

Table 1 shows the actual covariance matrix of the population and class distributions. We also show the estimated
covariance matrices under the different modeling assumptions as described above. We also report the volume of the
population distribution and that of the class distributions
estimated from the covariance matrices, and thus the capacity of this example computed as the ratio of the population
volume and the class volume. Our results indicate that a
capacity of two for the the class with the largest volume,
i.e., we will not be able to fit more than two classes with
that volume into the population sphere without any overlap
between the classes. Visually, this is also apparent from the
Figure 1.

2

G AUSSIAN C HANNEL

The validity of our capacity estimates are dependent on
how well the face representations conform to the modeling
assumptions of the Gaussian noise channel. The Gaussian
noise channel is predicated upon the Gaussianity of the
distributions Py , Px and Pz . While there are many tests
to numerically quantify the Gaussianity of the embedding,
most of these tests are either very unstable or are too
conservative. Therefore, we resort to visual tests through
Q-Q plots, univariate as well as multi-variate.
Univariate Test: Figure 2 displays the heat map of the
pair-wise covariance between the dimensions of the representation averaged over the entire CASIA dataset. The

2

Illustrative Example of 2D Representation

Fig. 1: 2D Representation Space: An illutration of a two dimensional represenatation space. The white color represents the
population space, the green, gray, orange and red correspond to classes with the minimum, mean, median, and maximum
noise, respectively. Furthermore, as described in the text, we consider three different models, hyper-sphere (dashes), axisaligned hyper-ellipsoid (dots), and general hyper-ellipsoid (solid), to approximate the population and class distribution.
covariance is predominantly a diagonal matrix with very
low correlation between the feature dimensions. Figure 3
shows the marginal histograms and the normal distribution
estimated from the data. Across all dimensions the data
conforms very well to a normal distribution. This is also
apparent in the marginal Q-Q plots for all the dimensions
as shown in Fig. 4.

Multivariate Test: Figure 5 displays the Q-Q plot to compare
the face representations of the entire CASIA dataset and
the mulit-variate Gaussian distribution estimated from the
same representations. The plots show that apart from a few
outlying samples the FaceNet representation does indeed
follow a Gaussian distribution for both Py and Px .

3
0
20

0.8

40

0.6

60

0.4

80

0.2

100

0.0

120

0.2
0

20

40

60

80

100

120

Fig. 2: Heat map of the empirical covariance of the representation estimated over the entire CASIA dataset.

Teacher -Student R EPRESENTATIONS

The efficacy of our capacity estimation model is predicated
on the ability of the student to mimic the teacher. It is critical
that the student model preserves the overall structure of
the embedding space. Figure 6 provides a visualization of
the embeddings after projecting onto the top two principal
eigenvectors of the embedding. We observe that the student
model not only mimics the teacher at the level of each individual image but also preserves the overall global structure
of the embedding space.
Our main motivation for learning a student model is
to explicitly account for and extract uncertainty estimates
that the teacher itself is unable to provide. Furthermore, by
explicitly accounting for the different sources of uncertainty,
epistemic and aleatoric, the student model can identify outlier samples, those that are outside the distribution of the
training data. For an inlier sample, the epistemic uncertainty
can be explained away by the uncertainty in the model
weights w while the same does not hold for an outlier image.
In Fig. 6 we provide a pictorial demonstrative example of
the embedding of an outlier and an inlier and quantitative

3

TABLE 1: Capacity of 2D Toy Example
Population
Covariance
Estimation
Ground-Truth

Assumptions

Circle

Ellipse
(Axis Aligned)

Ellipse


12.07
0


11.22
0


11.22
0.76


0
12.07


0
12.93


0.76
12.93








1.11
12.47

10.38
1.11


1.11
12.47

10.38
1.11


1.11
12.47

10.38
1.11

Noise
Area

Epistemic

(T)

Aleatoric

Covariance
Estimation

Area

Ground-Truth

Estimated

Ground-Truth

Capacity

Capacity

Min


1.14
0



0
1.14


1.37
0.48


0.48
1.09

1.14

10.64

–

Mean


3.73
0


0
3.73


5.99
0.31


0.31
3.04

3.73

3.24

–

Median


3.52
0


0
3.52


2.22
0.39


0.39
5.37

3.52

3.43

–

Max


6.36
0


0
6.36


5.88
0.72


0.72
5.47

6.36

1.90

–

Min


1.39
0


0
0.88


1.37
0.48


0.48
1.09

1.11

10.89

–

Mean


5.02
0


0
2.53


5.86
0.89


0.89
2.83

3.56

3.38

–

Median


3.75
0


0
2.98


4.53
0.03


0.03
3.32

3.34

3.60

–

Max


7.03
0


0
5.68


5.88
0.72


0.72
5.47

6.32

1.91

–

Min


1.39
0.46


0.46
0.88


1.37
0.48


0.48
1.09

1.01

11.95

10.09

Mean


3.02
0.79


0.79
4.23


2.65
0.49


0.49
4.93

3.49

3.45

3.16

Median


2.69
0.69


0.69
4.17


3.48
0.80


0.80
3.58

3.28

3.67

3.29

Max


7.03
1.11


1.11
5.68


5.88
0.72


0.72
5.47

6.22

1.93

2.01

12.07

12.04

12.02

TABLE 2: Epistemic and Aleatoric Uncertainty
Sample Class

Model

of uncertainty compared to the inlier example.
(T)

4

FACE R EPRESENTATION A RCHITECTURES

20

200

1000

20

200

1000

3.66

3.47

3.37

7.88

7.79

7.79

3.18

3.06

3.03

7.25

7.21

7.25

Figure 7a provides a complete description of the inceptionresnet [1] network architecture of the teacher model, the
basis of the FaceNet representation while Fig. 7b provides a
pictorial representation of the student network architecture.

7.34

7.08

7.13

13.01

13.02

13.14

5

9.32

9.53

9.63

14.88

15.36

15.30

examples in Table 2. Notice that, both qualitatively and
quantitatively, the outlier example exhibits a greater amount

A DDITIONAL E XPERIMENTAL R ESULTS

Capacity at FAR: Table 3 shows the capacity as a function
of the False Accept Rate for the FaceNet representation.
Distribution of Class Uncertainty: The capacity estimates
of a given representation is dependent on the choice of a
canonical class-specific hyper-ellipsoid, that is representative of a typical subject among all possible subjects the face
representation system is going to be used for. However,

4

Fig. 3: Histogram and marginal distributions of the 128 dimensional FaceNet representation.

Fig. 4: Q-Q plot of the marginal distributions of the 128 dimensional FaceNet representation.
determining this canonical subject is a difficult task in itself.
In Fig. 8 we show histograms of the distribution of classspecific hyper-ellipsoids for each of the datasets. Notice that
classes from constrained datasets, LFW and PCSO, have
smaller vokumes while the classes from the unconstrained
datasets, CASIA, IJB-A and IJB-B, have much larger volumes. Furthermore, the classes in each dataset span a wide
range of hyper-ellipsoid sizes resulting in the wide range
of capacity estimates. Some datasets even have a number of
outlying classes with hyper-ellipsoids that are much bigger
or much smaller than the rest of the dataset. These spurious
classes (often the result of mis-labeled data) can have a
significant effect on the capacity estimates. To overcome
this problem, we manually identified and accounted for
classes with mis-labeled images. We also note that the for the
Eigenfaces representation, the volumes of the classes across

different datasets are very similar.
We also show examples of the classes and the images that
correspond to the smallest and largest class hyper-ellipsoids
in each dataset. First for FaceNet: PCSO in Fig. 9, LFW in
Fig. 10, CASIA in Fig. 11, IJB-A in Fig. 12 and IJB-B in Fig.
13, followed by Eigenfaces: PCSO in Fig. 14, LFW in Fig. 15,
CASIA in Fig. 16, IJB-A in Fig. 17 and IJB-B in Fig. 18.
Figure 19 shows the capacity of Eigenfaces on different
datasets as a function of the operating point defined by the
acceptable false accept rate.

R EFERENCES
[1] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. Alemi, “Inceptionv4, inception-resnet and the impact of residual connections on
learning,” arXiv:1602.07261, 2016.

5

(a) FaceNet Py

(b) FaceNet Px

(c) Eigenfaces Py

(d) Eigenfaces Px

Fig. 5: Q-Q plot for each of the 128 dimensions of the FaceNet representation for both Py and Px

(a) Teacher Embeddings

(b) Student Embeddings

(c) Inlier and Outlier Embeddings

Fig. 6: Visualization of the embeddings of the (a) teacher, (b) student, (c) an inlier and outlier class. We project the face
representation onto the first two principal eigenvectors for the purpose of this visualization.

TABLE 3: apacity vs FAR (FaceNet)
FAR
0.0001%
0.001%
0.01%
0.1%
1.0%
10.0%

PCSO

LFW

CASIA

IJB-A

IJB-B

Dataset Pool

5.3×107
1.4×109
5.9×1010
4.8×1012
1.1×1015
2.8×1018

9.9×105
2.6×107
1.1×109
8.9×1010
2.1×1013
5.1×1016

2.4×101
6.3×102
2.6×104
2.1×106
5.1×108
1.2×1012

1.3×101
3.5×102
1.5×104
1.2×106
2.9×108
6.9×1011

2.7×101
7.0×102
2.9×104
2.4×106
5.8×108
1.4×1012

2.2×103
5.8×104
2.4×106
2.0×108
4.8×1010
1.1×1014

6

yc1

yc2
yc5
yc7

Red-A 10 ×
Stem Stem 5 ×
IncepR-B
Input
IncepR-A

Red-B

5×
IncepR-C

AvgPool
+Dropout

yc3

yc4

yc

yc6

Softmax

(a) Teacher Network Architecture: The input to the teacher is a face image of size 160 × 160 and the output is a 128-dimensional
vector for each image. The entire face representation space can be approximated by a 128-dimensional hyper-ellipsoid.

Σ1
c

Σ2
c
Σ4
c
Σ5
c
Σ7
c

Red-A 10 ×
Stem Stem 5 ×
IncepR-B
Input
IncepR-A

Red-B

5×
IncepR-C

AvgPool
+Dropout

Σ3
c

Σ6
c

Softmax

(b) Student Network Architecture: The input to the student is a face image of size 160 × 160 and the output is a 128-dimensional
hyper-ellipsoid of the uncertainty of the representation for the corresponding image. This entire face representation can be
approximated by a 128-dimensional population hyper-ellipsoid.

Fig. 7: Face Representation Architectures: The network architectures for estimating the representation capacity. (a) The
Inception-ResNet [1] architecture of the teacher face representation model. This architecture is the current state-of-the-art
network for face representations. It was designed to represent a 160×160 face image by a 128-dimensional feature vector.
(b) We adopt the same Inception-ResNet architecture for the student network. The student network, however, differs from
the teacher network in the following ways; (1) it has two outputs: the mean and the covariance of the uncertainty of the
embedding, and (2) incorporates dropout after every convolutional layer in the network.

7

(a) FaceNet

(b) Eigenfaces

Fig. 8: Histograms of the class-specific hyper-ellipsoids for all the dataset under the three different approximations of the
normal distribution, isotropic Gaussian, axis-aligned Gaussian and the anisotropic Gaussian distributions.

(a) DNN: PCSO Large

(b) DNN: PCSO Small

Fig. 9: PCSO

(a) DNN: LFW Large

(b) DNN: LFW Small

Fig. 10: LFW

8

(a) DNN: CASIA Large

(b) DNN: CASIA Small

Fig. 11: CASIA

(a) DNN: IJB-A Large

(b) DNN: IJB-A Small

Fig. 12: IJB-A

(a) DNN: Large IJB-B

(b) DNN: Small IJB-B

Fig. 13: IJB-B

(a) PCA: PCSO Large

(b) PCA: PCSO Small

Fig. 14: PCSO

(a) PCA: LFW Large

(b) PCA: LFW Small

Fig. 15: LFW

9

(a) PCA: CASIA Large

(b) PCA: CASIA Small

Fig. 16: CASIA

(a) PCA: IJB-A Large

(b) PCA: IJB-A Small

Fig. 17: IJB-A

(a) PCA: Large IJB-B

(b) PCA: Small IJB-B

6

10

5

10 4 10 3 10
False Accept Rate

(a) CASIA

2

10

1

109
106
103
100
10 3
10 6
10 9
10 12
10 1510

Hypersphere
Axis Aligned Hyper-ellipsoid
Hyper-ellipsoid

6

10

5

10 4 10 3 10
False Accept Rate

(b) LFW

2

10

1

108
105
102
10 1
10 4
10 7
10 10
10 13
10

Hypersphere
Axis Aligned Hyper-ellipsoid
Hyper-ellipsoid

Capacity

Hypersphere
Axis Aligned Hyper-ellipsoid
Hyper-ellipsoid

Capacity

108
105
102
10 1
10 4
10 7
10 10
10 13
10

Capacity

Capacity

Fig. 18: IJB-B

6

10

5

10 4 10 3 10
False Accept Rate

(c) PCSO

2

10

1

109
106
103
100
10 3
10 6
10 9
10 12
10

Hypersphere
Axis Aligned Hyper-ellipsoid
Hyper-ellipsoid

6

10

5

10 4 10 3 10
False Accept Rate

2

10

1

(d) IJB-B

Fig. 19: For each of the datasets under consideration we estimate the capacity of Eigenfaces at different false accept rates
corresponding to different operating points. Under the limit, the capacity tends to zero as the FAR tends to zero. Similarly
the capacity tends to ∞ as the FAR tends to 1.0.

