arXiv:1711.06246v1 [cs.CV] 16 Nov 2017

LDMNet: Low Dimensional Manifold Regularized Neural
Networks∗
Wei Zhu1 , Qiang Qiu2 , Jiaji Huang3 , Robert Calderbank2 , Guillermo Sapiro2 , and Ingrid
Daubechies1
1

Department of Mathematics, Duke University
2
ECE, Duke University
3
Baidu Silicon Valley AI Lab

Abstract

efits of LDMNet in the experiments. First, we show that
LDMNet significantly outperforms widely-used network
regularizers such as weight decay and DropOut. Second,
we show that LDMNet can be designed to extract common features of an object imaged via different modalities, which implies, in some imaging problems, LDMNet
is more likely to find the model that is subject to different illumination patterns. This proves to be very useful in
real-world applications such as cross-spectral face recognition.

Deep neural networks have proved very successful on
archetypal tasks for which large training sets are available, but when the training data are scarce, their performance suffers from overfitting. Many existing methods of
reducing overfitting are data-independent, and their efficacy is often limited when the training set is very small.
Data-dependent regularizations are mostly motivated by
the observation that data of interest lie close to a manifold, which is typically hard to parametrize explicitly
and often requires human input of tangent vectors. These
methods typically only focus on the geometry of the input data, and do not necessarily encourage the networks
to produce geometrically meaningful features. To resolve
this, we propose a new framework, the Low-DimensionalManifold-regularized neural Network (LDMNet), which
incorporates a feature regularization method that focuses
on the geometry of both the input data and the output
features. In LDMNet, we regularize the network by encouraging the combination of the input data and the output features to sample a collection of low dimensional
manifolds, which are searched efficiently without explicit
parametrization. To achieve this, we directly use the manifold dimension as a regularization term in a variational
functional. The resulting Euler-Lagrange equation is a
Laplace-Beltrami equation over a point cloud, which is
solved by the point integral method without increasing
the computational complexity. We demonstrate two ben∗ Work

1

Introduction

In this era of big data, deep neural networks (DNNs) have
achieved great success in machine learning research and
commercial applications. When large amounts of training
data are available, the capacity of DNNs can easily be increased by adding more units or layers to extract more
effective high-level features [9, 10, 26]. However, big
networks with millions of parameters can easily overfit
even the largest of datasets. It is thus crucial to regularize
DNNs so that they can extract “meaningful” features not
only from the training data, but also from the test data.
Many widely-used network regularizations are dataindependent. Such techniques include weight decay, parameter sharing, DropOut [24], DropConnect [27], etc.
Intuitively, weight decay alleviates overfitting by reducing the magnitude of the weights and the features, and
DropOut and DropConnect can be viewed as computa-

partially supported by NSF, DoD, NIH, and Google.

1

the variational problem.
In the experiments, we demonstrate two benefits of
LDMNet: First, by extracting geometrically meaningful features, LDMNet significantly outperforms widelyused regularization techniques such as weight decay
and DropOut. For example, Figure 1 shows the twoMost of the data-dependent regularizations are motidimensional projections of the test data from MNIST and
vated by the empirical observation that data of interest
their features learned from 1,000 training data by the same
typically lie close to a manifold, an assumption that has
network with different regularizers. It can be observed
previously assisted machine learning tasks such as nonthat the features learned by weight decay and DropOut
linear embedding [2], semi-supervised labeling [3], and
typically sample two-dimensional regions, whereas the
multi-task classification [6]. In the context of DNN, datafeatures learned by LDMNet tend to lie close to onedependent regularization techniques include the tangent
dimensional and zero-dimensional manifolds (curves and
distance algorithm [20, 22], tangent prop algorithm [21],
points). Second, in some imaging problems, LDMNet is
and manifold tangent classifier [18]. Typically, these almore likely to find the model that is subject to different ilgorithms only focus on the geometry of the input data, and
lumination patterns. By regularizing the network outputs,
do not encourage the network to produce geometrically
LDMNet can extract common features of the same subject
meaningful features. Moreover, it is typically hard to eximaged via different modalities so that these features samplicity parametrize the underlying manifold, and some of
ple the same low dimensional manifold. This can be obthe algorithms require human input of the tangent planes
served in Figure 2d, where the features of the same subject
or tangent vectors [9].
extracted by LDMNet from visible (VIS) spectrum images
Motivated by the same manifold observation, we pro- and near-infrared (NIR) spectrum images merge to form
pose a new network regularization technique that focuses a single low dimensional manifold. This significantly inon the geometry of both the input data and the output creases the accuracy of cross-modality face recognition.
features, and build low-dimensional-manifold-regularized The details of the experiments will be explained in Secneural networks (LDMNet). This is inspired by a recent tion 4.
algorithm in image processing, the low dimensional manifold model [16, 19, 30]. The idea of LDMNet is that the
concatenation (xi , ξi ) of the input data xi and the output 2 Model Formulation
features ξi should sample a collection of low dimensional
manifolds. This idea is loosely related to the Gaussian For simplicity of explanation, we consider a K-way clasN
mixture model: instead of assuming that the generating sification using a DNN. Assume that {(xi , yi )}i=1 ⊂
d1
distribution of the input data is a mixture of Gaussians, R × {1, . . . , K} is the labeled training set, and θ is the
we assume that the input-feature tuples (xi , ξi ) are gen- collection of network weights. For every datum xi with
erated by a mixture of low dimensional manifolds. To em- class label yi ∈ {1, . . . , K}, the network first learns a d2 d2
phasize this, we explicitly penalize the loss function with dimensional feature ξi = fθ (xi ) ∈ R , and then applies
a term using an elegant formula from differential geome- a softmax classifier to obtain the probability distribution
try to compute the dimension of the underlying manifold. of xi over the K classes. The softmax loss `(fθ (xi ), yi )
The resulting variational problem is then solved via al- is then calculated for xi as a negative log-probability of
ternating minimization with respect to the manifold and class yi . The empirical loss function J(θ) is defined as
the network weights. The corresponding Euler-Lagrange the average loss on the training set:
equation is a Laplace-Beltrami equation on a point cloud,
N
1 X
which is efficiently solved by the point integral method
J(θ) =
`(fθ (xi ), yi ).
(1)
(PIM) [15] with O(N ) computational complexity, where
N i=1
N is the size of the input data. In LDMNet, we never have
to explicitly parametrize the manifolds or derive the tan- When the training samples are scarce, statistical learning
gent planes, and the solution is obtained by solving only theories predict that overfitting to the training data will
tionally inexpensive ways to train an exponentially large
ensemble of DNNs. Their effectiveness as network regularizers can be quantified by analyzing the Rademacher
complexity, which provides an upper bound for the generalization error [1, 27].

2

(a) 10,000 test data from (b) Features learned from (c) Features learned from (d) Features learned from
MNIST
a network regularized by a network regularized by a network regularized by
weight decay
DropOut
LDMNet

Figure 1: Test data of MNIST and their features learned by the same network with different regularizers. All networks
are trained from the same set of 1,000 images. Data are visualized in two dimensions using PCA, and ten classes are
distinguished by different colors. In (b) and (c), the features learned by weight decay and DropOut typically sample
two dimensioanl regions. Whereas in (d), features learned by LDMNet tend to concentrate on one-dimensional and
zero-dimensional manifolds (curves and points).

60

10

40

10

1

5

0.5

5
20
0

0

-5

-0.5

0
0
-20
-40
40

-5
10
20

40
20

0

0
-20

-20

-10
10
10

0
-10

-60

(a) VGG-face

5

-10

0.5
0
-0.5

0

0

-10
-20

1

10

-5

-40
-40

-1
2
15

0

5
0

-20

-15

(b) Weight decay

-5

(c) DropOut

-1
-1

-1.5

(d) LDMNet

Figure 2: Features generated from VGG-face model [17] with or without embedding learned from the network specified in Table 8 with different regularizations. A total of five subjects are displayed, with one color per subject. VIS
images are denoted as filled circles and NIR images are denoted as unfilled diamonds. All features are visualized in
two dimensions using PCA. In (a), (b), and (c), features of the same subject typically form two clusters, one for NIR
and the other for VIS. In (d), with LDMNet, features of the same subject from different modalities merge to form a
single low dimensional manifold.

3

Pd
where j=1 k∇M αj k2L2 (M) corresponds to the L1 norm
of the local dimension. To solve (3), we alternate the
direction of minimization with respect to M and θ.
More specifically, given (θ (k) , M(k) ) at step k satisfying
(k)
{(xi , fθ(k) (xi ))}N
, step k + 1 consists of the
i=1 ⊂ M
following

occur [25]. What this means is that the average loss on
the testing set can still be large even if the empirical loss
J(θ) is trained to be small.
LDMNet provides an explanation and a solution for
network overfitting. Empirical observation suggests that
many data of interest typically sample a collection of low
L
dimensional manifolds, i.e. {xi }N
i=1 ⊂ N = ∪l=1 Nl ⊂
d1
R . One would also expect that the feature extractor, fθ ,
of a good learning algorithm be a smooth function over N
so that small variation in x ∈ N would not lead to dramatic change in the learned feature ξ = fθ (x) ∈ Rd2 .
Therefore the concatenation of the input data and output features, {(xi , ξi )}N
i=1 , should sample a collection of
d
low dimensional manifolds M = ∪L
l=1 Ml ⊂ R , where
d = d1 + d2 , and Ml = {(x, fθ (x))}x∈Nl is the graph
of fθ over Nl . We suggest that network overfitting occurs when dim(Ml ) is too large after training. Therefore,
to reduce overfitting, we explicitly use the dimensions of
Ml as a regularizer in the following variational form:
Z
λ
dim(M(p))dp
(2)
min J(θ) +
θ,M
|M| M

• Update θ (k+1) and the perturbed coordinate func(k+1)
(k+1)
tions α(k+1) = (α1
, · · · , αd
) as the minimizers of (4) with the fixed manifold M(k) :
d

min
θ,α

J(θ) +

λ X
k∇M(k) αj k2L2 (M(k) )
|M(k) | j=1
(4)

s.t.

α(xi , fθ(k) (xi )) = (xi , fθ (xi )),

∀i = 1, . . . , N

• Update M(k+1) :
M(k+1) = α(k+1) (M(k) )

(5)

Remark 1. As mentioned in Theorem 1, α =
(α1 , · · · , αd ) is supposed to be the coordinate functions.
s.t.
⊂ M,
In (4), we solve a perturbed version α(k+1) so that it maps
of point cloud {(xi , fθ(k) (xi ))}N
i=1
where for any p ∈ M = ∪L
the the previous iterates
l=1 Ml , M(p) denotes
(k)
PL
and manifold M to their corresponding updated vermanifold Ml to which p belongs, and |M| = l=1 |Ml |
sions. If the iteration converges to a fixed point, the conis the volume of M. The following theorem from differsecutive iterates of the manifolds M(k+1) and M(k) will
ential geometry provides an elegant way of calculating the
be very close to each other for sufficiently large k, and
manifold dimension in (2).
α(k+1) will be very close to the coordinate functions.
Theorem 1. [16] Let M be a smooth submanifold isoNote that (5) is straightforward to implement, and (4)
metrically embedded in Rd . For any p = (pi )di=1 ∈ M,
is an optimization problem with linear constraint, which
d
can be solved via the alternating direction method of mulX
2
dim(M) =
|∇M αi (p)| ,
tipliers (ADMM). More specifically,
{(xi , fθ (xi ))}N
i=1

i=1

d
X
where αi (p) = pi is the coordinate function, and ∇M is α(k+1) = arg min
k∇M(k) αj kL2 (M(k) )
ξ
αξ
the gradient operator on the manifold M. More specifij=d
+1
1
Pk
cally, ∇M αi = s,t=1 g st ∂t αi ∂s , where k is the intrinN
µ|M(k) | X
(k)
sic dimension of M, and g st is the inverse of the metric +
kαξ (xi , fθ(k) (xi )) − (fθ(k) (xi ) − Zi )k22 .
2λN
tensor.
i=1
(6)
As a result of Theorem 1, (2) can be reformulated as:
d

min
θ,M

s.t.

J(θ) +

λ X
k∇M αj k2L2 (M)
|M| j=1

{(xi , fθ (xi ))}N
i=1

(3)

θ (k+1) = arg min J(θ) +
θ

⊂M

N
µ X (k+1)
kαξ
(xi , fθ(k) (xi ))
2N i=1

(k)

− (fθ (xi ) − Zi )k22 .
4

(7)

(k+1)

Zi

(k)

= Zi

(k+1)

+ αξ

(xi , fθ(k) (xi )) − fθ(k+1) (xi ),
(8)

second term with respect to the output layer fθ (xi ) is:


∂Ei
(k)
(k+1)
= µ fθ (xi ) − Zi − αξ
(xi , fθ(k) (xi ))
∂fθ (xi )
(11)

where αξ is defined as α = (αx , αξ ) =
((α1 , . . . , αd1 ), (αd1 +1 , . . . , αd )), and Zi is the dual variable. Note that we need to perturb only the coordinate
functions αξ corresponding to the features in (6) because
the inputs xi are given and fixed. Also note that because
the gradient and the L2 norm in (6) are defined on M
instead of the projected manifold, we are not simply minimizing the dimension of the manifold M projected onto
the feature space. For computational efficiency, we update
α, θ and Zi only once every manifold update (5).
Among (6),(7) and (8), (8) is the easiest to implement,
(7) can be solved by stochastic gradient descent (SGD)
with modified back propagation, and (6) can be solved
by the point integral method (PIM) [15]. The detailed
implementation of (6) and (7) will be explained in the next
section.

3

This means that we need to only add the extra term (11)
to the original gradient, and then use the already known
procedure to back-propagate the gradient. This essentially
leads to no extra computational cost in the SGD updates.

3.2

Point Integral Method for the α Update

Note that the objective funtion in (6) is decoupled with
respect to j, and each αj update can be cast into:
min
1

u∈H (M)

k∇M uk2L2 (M) + γ

X

|u(q) − v(q)|2 , (12)

q∈P

where u = αj , M = M(k) , γ = µ|M(k) |/2λN ,
N
and P = {pi = (xi , fθ(k) (xi ))}i=1 ⊂ M. The EulerLagrange equation of (12) is:
X
−∆M u(p) + γ
δ(p − q)(u(q) − v(q)) = 0, p ∈ M

Implementation Details and Complexity Analysis

q∈P

In this section, we present the details of the algorithmic
implementation, which includes back propagation for the
θ update (7), point integral method for the α update (6),
and the complexity analysis.

3.1

∂u
= 0, p ∈ ∂M
∂n
(13)
It is hard to discretize the Laplace-Beltrami operator ∆M
and the delta function δ(x, y) on an unstructured point
cloud P . We instead use the point integral method to solve
(13). The key observation in PIM is the following theorem:

Back Propagation for the θ Update

We derive the gradient of the objective function in (7). Let

µ (k+1)
3
(k)
kα
(xi , fθ(k) (xi )) − (fθ (xi ) − Zi )k22 . Theorem 2. [15] If u ∈ C (M) is a function on M, then
2 ξ
Z
Z
(9)
∂u(q)
Rt (p, q)dτq
∆M u(q)Rt (p, q)dq − 2
M
∂M ∂n
Then the objective function in (7) is
Z
1
+
(u(p) − u(q))Rt (p, q)dq
= O(t1/4 ).
N
N
t M
2
X
X
L (M)
1
1
˜
J(θ)
=
`(fθ (xi ), yi ) +
Ei (θ). (10)
(14)
N i=1
N i=1

Ei (θ) =

where Rt is the normalized heat kernel:


|p − q|2
Rt (p, q) = Ct exp −
.
4t

Usually the back-propagation of the first term in (7) is
known for a given network. As for the second term, let
xi be a given datum in a mini-batch. The gradient of the
5

(15)

After convolving equation (13) with the heat kernel Rt , Algorithm 1 LDMNet Training
we know the solution u of (13) should satisfy
d1
Require: Training data {(xi , yi )}N
× R, hyi=1 ⊂ R
Z
perparameters λ̃ and µ, and a neural network with the
−
∆M u(q)Rt (p, q)dq
weights θ and the output layer ξi = fθ (xi ) ∈ Rd2 .
Ensure: Trained network weights θ ∗ .
XM
+γ
Rt (p, q) (u(q) − v(q)) = 0.
(16)
Randomly initialize the network weights θ (0) . The dual
(0)
q∈P
variables Zi ∈ Rd2 are initialized to zero.
while not converge do
Combined with Theorem 2 and the Neumann boundary
1. Compute the matrices W and L as in (20) and
condition, this implies that u should approximately satisfy
(21) with pi = (xi , fθ(k) (xi )).
Z
2. Update α(k+1) in (6): solve the linear systems
(u(p) − u(q)) Rt (p, q)dq
(19), where
M
(k)
X
ui = αj (pi ), vi = fθ(k) (xi )j − Zi,j . (22)
+γt
R (p, q) (u(q) − v(q)) = 0
(17)
t

3. Update θ (k+1) in (7): run SGD for M epochs with
an extra gradient term (11).
4. Update Z (k+1) in (8).
5. k ← k + 1.
end while
θ ∗ ← θ (k) .

q∈P

Note that (17) no longer involves the gradient ∇M or
the Laplace-Beltrami operator ∆M . Assume that P =
{p1 , . . . , pN } samples the manifold M uniformly at random, then (17) can be discretized as
N
N
X
|M| X
Rt,ij (ui − uj ) + γt
Rt,ij (uj − vj ) = 0,
N j=1
j=1

3.3

(18)

Based on the analysis above, we present a summary of the
traning for LDMNet in Algorithm 1.
The additional computation in Algorithm 1 (in steps 1
and 2) comes from the update of weight matrices in (20)
and solving the linear system (19) from PIM once every
M epochs of SGD. We now explain the computational
complexity of these two steps.
When N is large, it is not computationally feasible to
compute the pairwise distances in the entire training set.
Therefore the weight matrix W is truncated to only 20
nearest neighbors. To identify those nearest neighbors,
we first organize the data points {p1 , . . . , pN } ⊂ Rd into
a k-d tree [7], which is a binary tree that recursively partitions a k-dimensional space (in our case k = d). Nearest neighbors can then be efficiently identified because
branches can be eliminated from the search space quickly.
Modern algorithms to build a balanced k-d tree generally
at worst converge in O(N log N ) time [4, 5], and finding
nearest neighbours for one query point in a balanced k-d
tree takes O(log N ) time on average [8]. Therefore the
complexity of the weight update is O(N log N ).
Since W and L are sparse symmetric matrices with

where ui = u(pi ), and Rt,ij = Rt (pi , pj ). Combining
the definition of γ in (12), we can write (18) in the matrix
form


µ
µ
(19)
L + W u = W v, λ̃ = 2λ/t,
λ̃
λ̃
where λ̃ can be chosen instead of λ as the hyperparameter
to be tuned, u = (u1 , . . . , uN )T , W is an N × N matrix


|pi − pj |2
,
Wij = Rt,ij = exp −
4t

(20)

and L is the graph Laplacian of W :
Lii =

X

Wij ,

and

Lij = −Wij

if

Complexity Analysis

i 6= j.

j6=i

(21)
Therefore, the update of αξ , which is cast into the canonical form (12), is achieved by solving a linear system (19).
6

a fixed maximum number of non-zero entries in each
row, the linear system (19) can be solved efficiently with
the preconditioned conjugate gradients method. After restricting the number of matrix multiplications to a maximum of 50, the complexity of the α update is O(N ).

4

Layer

Type

1

conv

2

max pool

3

conv

4

max pool

5

conv

6
7
8

ReLu (DropOut)
fully connected
softmaxloss

Experiments

In this section, we compare the performance of LDMNet
to widely-used network regularization techniques, weight
decay and DropOut, using the same underlying network
structure. We point out that our focus is to compare the
effectiveness of the regularizers, and not to investigate the
state-of-the-art performances on the benchmark datasets.
Therefore we typically use simple network structures and
relatively small training sets, and no data augmentation or
early stopping is implemented.
Unless otherwise stated, all experiments use mini-batch
SGD with momentum on batches of 100 images. The
momentum parameter is fixed at 0.9. The networks are
trained using a fixed learning rate r0 on the first 200
epochs, and then r0 /10 for another 100 epochs.
As mentioned in Section 3.3, the weight matrices W
are truncated to 20 nearest neighbors. For classification
tasks, nearest neighbors can be searched within each class
in the labeled training set. We also normalize the weight
matrices with local scaling factors σ(p) [29]:


kp − qk2
w(p, q) = exp −
,
(23)
σ(p)σ(q)

Parameters
size: 5 × 5 × 1 × 20
stride: 1, pad: 0
size: 2 × 2, stride: 2, pad: 0
size: 5 × 5 × 20 × 50
stride: 1, pad: 0
size: 2 × 2, stride: 2, pad: 0
size: 4 × 4 × 50 × 500
stride: 1, pad: 0
N/A
500 × 10
N/A

Table 1: Network structure in the MNIST experiments.
The outputs of layer 6 are the extracted features, which
will be fed into the softmax classifier (layer 7 and 8).
training per
class
50
100
400
700
1000
3000
6000

λ̃

µ

w

0.05
0.05
0.01
0.01
0.005
0.001
0.001

0.01
0.01
0.01
0.01
0.01
0.01
0.01

0.1
0.05
0.01
0.005
0.005
0.001
0.001

Table 2: Hyperparamters used in the MNIST experiments

where σ(p) is chosen as the distance between p and its
10th nearest neighbor. This is based on the empirical analysis on choosing the parameter t in [15], and has been
used in [30]. The weight matrices and α are updated once
every M = 2 epochs of SGD.
All hyperparameters are optimized so that those reported are the best performance of each method. For
LDMNet, λ̃ defined in (19) typically decreases as the
training set becomes larger, whereas the paramter µ for
augmented Lagrangian can be fixed to be a constant. For
weight decay,

4.1

MNIST

The MNIST handwritten digit dataset contains approximately 60,000 training images (28 × 28) and 10,000 test
images. Tabel 1 describes the network structure. The
learned feature fθ (xi ) for the training data xi is the output of layer six, and is regularized for LDMNet in (2).
While state-of-the-art methods often use the entire
training set, we are interested in examining the performance of the regularization techniques with varying train2
min J(θ) + wkθk2 ,
(24) ing sizes from 500 to 60,000. In this experiment, the iniθ
tial learning rate is set to 0.001, and the hyperparameters
the parameter w also usually decreases as the training size are reported in Table 2. Table 3 displays the testing acincreases. For DropOut, the corresponding DropOut layer curacy of the competing algorithms. The dependence of
is always chosen to have a drop rate of 0.5.
the classification error and generalization error (which is
7

training per
class
50
100
400
700
1000
3000
6000

weight
decay
91.32%
93.38%
97.23%
97.67%
98.06%
98.87%
99.15%

DropOut

LDMNet

92.31%
94.05%
97.95%
98.07%
98.71%
99.21%
99.41%

95.57%
96.73%
98.41%
98.61%
98.89%
99.24%
99.39%

Table 3: MNIST: testing accuracy for different regularizers
0.5
weight decay
DropOut
LDMNet

8

generalization error

test error(%)

10

6
4
2
0
0.5

1

4

7 10

30

weight decay
DropOut
LDMNet

0.4
0.3
0.2
0.1

60

size of training set x 103

0
0.5

1

4

7 10

30

60

size of training set x 103

Figure 3: Comparison of the regularizers on the MNIST
dataset. The first (second) figure shows the dependence of
the classification (generalization) error on the size of the
training set.
the difference between the softmax loss on the testing and
training data) on the size of the training set is shown in
Figure 3. Figure 1 provides a visual illustration of the
features of the testing data learned from 1,000 training
samples. It is clear to see that LDMNet significantly outperforms weight decay and DropOut when the training set
is small, and the performance becomes broadly similar as
the size of the training set reaches 60,000.

4.2

Layer

Type

1

conv

2
3

ReLu
max pool

4

conv

5
6

ReLu
max pool

7

conv

8
9
10
11
12
13
14
15

ReLu
max pool
fully connected
ReLu (DropOut)
fully connected
ReLu (DropOut)
fully connected
softmaxloss

Parameters
size: 5 × 5 × 3 × 96
stride: 1, pad: 2
N/A
size: 3 × 3, stride: 2, pad: 0
size: 5 × 5 × 96 × 128
stride: 1, pad: 2
N/A
size: 3 × 3, stride: 2, pad: 0
size: 4 × 4 × 128 × 256
stride: 1, pad: 0
N/A
size: 3 × 3, stride: 2, pad: 0
output: 2048
N/A
output: 2048
N/A
2048 × 10
N/A

Table 4: Network structure in the SVHN and CIFAR-10
experiments. The outputs of layer 13 are the extracted features, which will be fed into the softmax classifier (layer
14 and 15).

SVHN and CIFAR-10

SVHN and CIFAR-10 are benchmark RGB image
datasets, both of which contain 10 different classes. These
two datasets are more challenging than the MNIST dataset
because of their weaker intraclass correlation. All algorithms use the network structure (similar to [11]) specified in Table 4. The outputs of layer 13 are the learned
features, and are regularized in LDMNet. All algorithms
start with a learning rate of 0.005 for SVHN and 0.001
8

training
per class
50
100
400
700

λ̃
0.1
0.05
0.05
0.01

SVHN
w
10−6
10−6
10−7
10−8

training per
class
50
100
400
700
full data

CIFAR-10
w
λ̃
0.01 5 × 10−4
0.01 5 × 10−5
0.01 5 × 10−5
0.01 5 × 10−7

Table 5: Hyperparameters used in the SVHN and CIFAR10 experiments.
training per
class
50
100
400
700

weight
decay
71.46%
79.05%
87.38%
89.69%

DropOut

LDMNet

71.94%
79.94%
87.16%
89.83%

74.64%
81.36%
88.03%
90.07%

weight
DropOut
decay
34.70%
35.94%
42.45%
43.18%
56.19%
56.79%
61.84%
62.59%
87.72%

LDMNet
41.55%
48.73%
60.08%
65.59%
88.21%

Table 7: CIFAR-10: testing accuracy for different regularizers. The first four experiments use training sets varied
in size from 500 to 7,000 and a simple network specified
in Table 4. In the last experiment, we first train a DNN
with VGG-16 architecture [23] from the full training data
using both weight decay and DropOut, then the DNN is
fine-tuned by regularizing the output layer with LDMNet.
No data augmentation is implemented in any of the experiment.

Table 6: SVHN: testing accuracy for different regularizers
ment is to match a probe image of a subject captured in
the near-infrared spectrum (NIR) to the same subject from
a gallery of visible spectrum (VIS) images. The CASIA
NIR-VIS 2.0 benchmark dataset [14] is used to evaluate
the performance. This dataset contains 17,580 NIR and
VIS face images of 725 subjects. Figure 4 shows eight
sample images of two subjects after facial landmark alignment and cropping [12]. Despite recent breakthroughs for
VIS face recognition by training DNNs from millions of
VIS images, such approach cannot be simply transferred
to NIR-VIS face recognition. The reason is that, unlike
VIS face images, we have only limited number of availabe NIR images. Moreover, the NIR-VIS face matching
is a cross-modality comparison.
The authors in [13] introduced a way to transfer the
breakthrough in VIS face recognition to the NIR spectrum. Their idea is to use a DNN pre-trained on VIS
images as a feature extactor, while making two independent modifications in the input and output of the DNN.
They first modify the input by “hallucinating” a VIS image from the NIR sample, and then apply a low-rank embedding of the DNN features at the output. The combination of these two modifications achieves the state-of-theart performance on cross-spectral face recognition.
We follow the second idea in [13], and learn a nonlinear low dimensional manifold embedding of the output features. The intuition is that faces of the same

for CIFAR-10. µ has been fixed as 0.5 for SVHN and
1 for CIFAR-10, and the remaining hyperparameters are
reported in Table 5.
We report the testing accuracies of the competing regularizers in Table 6 and Table 7 when the number of training samples is varied from 50 to 700 per class. Again, it is
clear to see that LDMNet outperforms weight decay and
DropOut by a significant margin.
To demonstrate the generality of LDMNet, we conduct
another experiment on CIFAR-10 using the entire training data with a different network structure. We first train
a DNN with VGG-16 architecture [23] on CIFAR-10 using both weight decay and DropOut without data augmentation. Then, we fine-tune the DNN by regularizing the
output layer with LDMNet. The testing accuracies are reported in the last row of Table 7. Again, LDMNet outperforms weight decay and DropOut, demonstrating that
LDMNet is a general framework that can be used to improve the performance of any network structure.

4.3

NIR-VIS Heterogeneous Face Recognition

Finally, we demonstrate the effectiveness of LDMNet for
NIR-VIS face recognition. The objective of the experi9

VGG-face
VGG-face + triplet [13]
VGG-face + low-rank [13]
VGG-face weight Decay
VGG-face DropOut
VGG-face LDMNet

Figure 4: Sample images of two subjects from the CASIA
NIR-VIS 2.0 dataset after the pre-procssing of alignment
and cropping [12]. Top: NIR. Bottom: VIS.
Layer
1
2
3
4

Type
fully connected
ReLu (DropOut)
fully connected
ReLu (DropOut)

Parameters
output:2000
N/A
output:2000
N/A

Table 8: Fully connected network for the NIR-VIS nonlinear feature embedding. The outputs of layer 4 are the
extracted features.

subject in two different modalities should sample the
same low dimensional feature manifold in a transformed
space. In our experiment, we use the VGG-face model
(downloaded at http://www.robots.ox.ac.uk/
˜vgg/software/vgg_face/) [17] as a feature extractor. The learned 4,096 dimensional features can be reduced to a 2,000 dimensional space using PCA and used
directly for face matching. Meanwhile, we also put the
4,096 dimensional features into a two-layer fully connected network described in Table 8 to learn a nonlinear
embedding using different regularizations. The features
extracted from layer 4 are regularized in LDMNet.
All nonlinear embeddings using the structure specified
in Table 8 are trained with SGD on mini-batches of 100
images for 200 epochs. We use an exponentially decreasing learning rate that starts at 0.1 with a decaying factor of
0.99. The hyperparameters are chosen to achieve the optimal performance on the validation set. More specifically,
λ̃, µ, w are set to 5 × 10−5 , 5, and 5 × 10−4 respectively.
We report the rank-1 performance score for the stan-

Accuracy (%)
74.51 ± 1.28
75.96 ± 2.90
80.69 ± 1.02
63.87 ± 1.33
66.97 ± 1.31
85.02 ± 0.86

Table 9: NIR-VIS cross-spectral rank-1 identification rate
on the 10-fold CASIA NIR-VIS 2.0 benchmark. The first
result is obtained by reducing the features learned from
VGG-face to 2,000 dimensional space using PCA. The
next two results use triplet [28] and low-rank embedding
of the learned features, and are reported in [13]. The last
three results are achieved by training the nonlinear embedding network in Table 8 with the corresponding regularizations.

dard CASIA NIR-VIS 2.0 evaluation protocol in Table 9. Because of the limited amount of training data
(around 6,300 NIR and 2,500 VIS images), the fullyconnected networks in Table 8 trained with weight decay and DropOut clearly overfit the training data: they
actually yield testing accuracies that are worse than using a simple PCA embedding of the features learned from
VGG-face. However, the same network regularized with
LDMNet has achieved a significant 10.5% accuracy boost
(from 74.51% to 85.02%) to using VGG-face directly. It
is also better than the results reported in [13] using the
popular triplet embedding [28] and low-rank embedding.
Figure 2 provides a visual illustration of the learned features from different regularizations. The generated features of five subjects are visualized in two dimensions using PCA, with filled circle for VIS, and unfilled diamond
for NIR, and one color for each subject. Note that in
Figure 2a,2b,2c, features of one subject learned directly
from VGG-face or from a nonlinear embedding regularized with weight decay or DropOut typically form two
clusters, one for NIR and the other for VIS. This contrasts
with the behavior in Figure 2d, where features of the same
subject from two different modalities merge to form a single low dimensional manifold.
10

5

Conclusion

[7] J. H. Friedman, J. L. Bentley, and R. A. Finkel. An algorithm for finding best matches in logarithmic expected
time. ACM Trans. Math. Softw., 3(3):209–226, Sept. 1977.
6

We proposed a general deep neural network regularization
technique LDMNet. The intuition of LDMNet is that the
concatenation of the input data and output features should [8] J. H. Friedman, J. L. Bentley, and R. A. Finkel. An algorithm for finding best matches in logarithmic expected
sample a collection of low dimensional manifolds, an idea
time. ACM Trans. Math. Softw., 3(3):209–226, Sept. 1977.
that is loosely related to the Gaussian mixture model.
6
Unlike most data-dependent regularizations, LDMNet fo[9]
I. Goodfellow, Y. Bengio, and A. Courville. Deep
cuses on the geometry of both the input data and the outLearning.
MIT Press, 2016.
http://www.
put features, and does not require explicit parametrization
deeplearningbook.org. 1, 2
of the underlying manifold. The dimensions of the manifolds are directly regularized in a variational form, which [10] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In The IEEE Conference on
is solved by alternating direction of minimization with a
Computer Vision and Pattern Recognition (CVPR), June
slight increase in the computational complexity (O(N ) in
2016. 1
solving the linear system, and O(N log N ) in the weight
[11]
G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever,
update). Extensive experiments show that LDMNet has
and R. R. Salakhutdinov. Improving neural networks
two benefits: First, it significantly outperforms widely
by preventing co-adaptation of feature detectors. arXiv
used regularization techniques, such as weight decay and
preprint arXiv:1207.0580, 2012. 8
DropOut. Second, LDMNet can extract common features
[12] V. Kazemi and J. Sullivan. One millisecond face alignof the same subject imaged via different modalities so that
ment with an ensemble of regression trees. In The IEEE
these features sample the same low dimensional maniConference on Computer Vision and Pattern Recognition
fold, which significantly increases the accuracy of cross(CVPR), June 2014. 9, 10
spectral face recognition.
[13] J. Lezama, Q. Qiu, and G. Sapiro. Not afraid of the
dark: Nir-vis face recognition via cross-spectral hallucination and low-rank embedding. In The IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), July
2017. 9, 10

References
[1] P. L. Bartlett. The sample complexity of pattern classification with neural networks: the size of the weights is more
important than the size of the network. IEEE transactions
on Information Theory, 44(2):525–536, 1998. 2

[14] S. Z. Li, D. Yi, Z. Lei, and S. Liao. The casia nir-vis
2.0 face database. In The IEEE Conference on Computer
Vision and Pattern Recognition (CVPR) Workshops, June
2013. 9

[2] M. Belkin and P. Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation, 15(6):1373–1396, 2003. 2

[15] Z. Li, Z. Shi, and J. Sun. Point integral method for solving poisson-type equations on manifolds from point clouds
with convergence guarantees. Communications in Computational Physics, 22(1):228258, 2017. 2, 5, 7

[3] M. Belkin, P. Niyogi, and V. Sindhwani. Manifold regularization: A geometric framework for learning from labeled
and unlabeled examples. Journal of machine learning research, 7(Nov):2399–2434, 2006. 2
[4] M. Blum, R. W. Floyd, V. Pratt, R. L. Rivest, and R. E.
Tarjan. Time bounds for selection. Journal of Computer
and System Sciences, 7(4):448 – 461, 1973. 6

[16] S. Osher, Z. Shi, and W. Zhu. Low dimensional manifold
model for image processing. SIAM Journal on Imaging
Sciences, 10(4):1669–1690, 2017. 2, 4
[17] O. M. Parkhi, A. Vedaldi, A. Zisserman, et al. Deep face
recognition. In BMVC, volume 1, page 6, 2015. 3, 10

[5] T. H. Cormen. Introduction to algorithms. MIT press,
2009. 6

[18] S. Rifai, Y. N. Dauphin, P. Vincent, Y. Bengio, and
X. Muller. The manifold tangent classifier. In Advances
in Neural Information Processing Systems, pages 2294–
2302, 2011. 2

[6] T. Evgeniou, C. A. Micchelli, and M. Pontil. Learning
multiple tasks with kernel methods. Journal of Machine
Learning Research, 6(Apr):615–637, 2005. 2

[19] Z. Shi, S. Osher, and W. Zhu. Generalization of the
weighted nonlocal laplacian in low dimensional manifold
model. Journal of Scientific Computing, Sep 2017. 2

11

[20] P. Simard, Y. LeCun, and J. S. Denker. Efficient pattern
recognition using a new transformation distance. In Advances in neural information processing systems, pages
50–58, 1993. 2
[21] P. Simard, B. Victorri, Y. LeCun, and J. Denker. Tangent
prop-a formalism for specifying selected invariances in an
adaptive network. In Advances in neural information processing systems, pages 895–903, 1992. 2
[22] P. Y. Simard, Y. A. LeCun, J. S. Denker, and B. Victorri.
Transformation Invariance in Pattern Recognition – Tangent Distance and Tangent Propagation, pages 235–269.
Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. 2
[23] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. arXiv preprint
arXiv:1409.1556, 2014. 9
[24] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever,
and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of machine
learning research, 15(1):1929–1958, 2014. 1
[25] V. N. Vapnik. An overview of statistical learning theory. IEEE transactions on neural networks, 10(5):988–
999, 1999. 4
[26] L. Wan, M. Zeiler, S. Zhang, Y. L. Cun, and R. Fergus.
Regularization of neural networks using dropconnect. In
S. Dasgupta and D. Mcallester, editors, Proceedings of
the 30th International Conference on Machine Learning
(ICML-13), volume 28, pages 1058–1066. JMLR Workshop and Conference Proceedings, May 2013. 1
[27] L. Wan, M. Zeiler, S. Zhang, Y. L. Cun, and R. Fergus.
Regularization of neural networks using dropconnect. In
S. Dasgupta and D. Mcallester, editors, Proceedings of
the 30th International Conference on Machine Learning
(ICML-13), volume 28, pages 1058–1066. JMLR Workshop and Conference Proceedings, May 2013. 1, 2
[28] K. Q. Weinberger, J. Blitzer, and L. K. Saul. Distance metric learning for large margin nearest neighbor classification. In Advances in neural information processing systems, pages 1473–1480, 2006. 10
[29] L. Zelnik-Manor and P. Perona. Self-tuning spectral clustering. In Advances in neural information processing systems, pages 1601–1608, 2005. 7
[30] W. Zhu, B. Wang, R. Barnard, C. D. Hauck, F. Jenko, and
S. Osher. Scientific data interpolation with low dimensional manifold model. Journal of Computational Physics,
352:213–245, 2018. 2, 7

12

