Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
Timur Garipov Pavel Izmailov Dmitrii Podoprikhin Dmitry Vetrov Andrew Gordon Wilson



Spin-polaron ladder spectrum of the spin-orbit-induced Mott insulator Sr_2IrO_4 probed by scanning tunneling spectroscopy
    Christian Hess
    December 30, 2023
=========================================================================================================================





The loss functions of deep neural networks are complex and their geometric properties
are not well understood.  We show that the optima of these complex loss functions
are in fact connected by simple curves 
over which training and test accuracy are nearly constant.  We introduce a training procedure to discover these 
high-accuracy pathways between modes.  Inspired by this new geometric insight, 
we also propose a new ensembling method entitled Fast Geometric Ensembling (FGE).
Using FGE we can train 
high-performing ensembles in the time required to train a single model.  We achieve
improved performance compared to 
the recent state-of-the-art Snapshot Ensembles, on  
CIFAR-10, CIFAR-100, and ImageNet. 




§ INTRODUCTION



The loss surfaces of deep neural networks (DNNs) are highly non-convex and can depend on 
millions of parameters.  The geometric properties of these loss surfaces are not well understood.  Even for simple networks, the number of local optima and saddle points is large and can grow exponentially in the number of 
parameters <cit.>.  Moreover, the loss is
high along a line segment connecting two optima  <cit.>.
These two observations suggest that the local optima are isolated.  

In this paper, we provide a new training procedure which can in fact find paths of near-constant accuracy 
between the modes of large deep neural networks. Furthermore, we 
show that for a wide range of architectures we can find these paths in the form of a simple polygonal chain of two 
line segments.  Consider, for example, Figure <ref>, which illustrates the ResNet-164 ℓ_2-regularized
cross-entropy train loss on CIFAR-100, through three different planes. 
We form each two dimensional plane by all affine combinations of three weight
vectors.[Suppose we have three weight vectors w_1, w_2, w_3. 
We set u = (w_2 - w_1), v = (w_3 - w_1) -⟨ w_3 - w_1, w_2 -  w_1 ⟩ /  w_2 - w_1^2 · (w_2 - w_1).
Then the normalized vectors û = u / u, v̂ = v / v form
an orthonormal basis in the plane containing w_1, w_2, w_3. To visualize
the loss in this plane, we define a Cartesian grid in the basis û, v̂
and evaluate the networks corresponding to each of the points in the grid. A point
P with coordinates (x, y) in the plane would then be given by 
P = w_1 + x ·û + y ·v̂.]

The left panel shows
a plane defined by three independently trained networks.  In this plane, all 
optima are isolated, which corresponds to the standard intuition.  However, the middle and 
right panels show two different paths of near-constant loss between the modes in weight space, 
discovered by our proposed training procedure.  
The endpoints of these paths are the two independently trained DNNs corresponding
to the two lower modes on the left panel.

We believe that this geometric discovery has major implications for research into 
multilayer networks, including (1) improving the efficiency, reliability, and accuracy of training, 
(2) creating better ensembles, and (3) deriving more effective posterior approximation 
families in Bayesian deep learning. Indeed, in this paper we are inspired by this geometric
insight to propose a new ensembling procedure that can efficiently discover multiple 
high-performing but diverse deep neural networks.


In particular, our contributions include:

	
  * The discovery that the local optima for modern deep neural networks are connected by very simple curves, such as a polygonal chain with only one bend.
	
  * A new method that finds such paths between two local optima, 
	such that the train loss and
	test error remain low along these paths.
	
  * Using the proposed method we demonstrate that such mode connectivity
	holds for a wide range of modern deep neural networks, on key benchmarks such 
	as CIFAR-100.  We show that these paths correspond to meaningfully different representations that can be efficiently 
	ensembled for increased accuracy. 
		
  * Inspired by these observations, we propose Fast Geometric Ensembling (FGE),
	which outperforms the recent state-of-the-art Snapshot Ensembles <cit.>, on CIFAR-10 and
	CIFAR-100, using powerful deep neural networks 
	such as VGG-16, Wide ResNet-28-10, and ResNet-164. On ImageNet we achieve
	0.56% top-1 error-rate improvement for a pretrained 
  ResNet-50 model by running FGE for only 5 epochs. 
  
  * We release the code for reproducing the results in this paper at 

<https://github.com/timgaripov/dnn-mode-connectivity>


The rest of the paper is organized as follows. Section <ref> 
discusses existing literature on DNN loss geometry and ensembling techniques.
Section <ref> introduces the proposed method to find 
the curves with low train loss and test error between local optima, which 
we investigate empirically in Section <ref>.  Section
<ref> then introduces our proposed ensembling technique,
FGE, which we empirically compare to the alternatives in Section <ref>.
Finally, in Section <ref> we discuss connections
to other fields and directions for future work.  


Note that we interleave two sections where we make methodological 
proposals (Sections <ref>, <ref>), 
with two sections where we perform experiments (Sections <ref>, <ref>).
Our key methodological proposal for ensembling, FGE, is in Section <ref>.



§ RELATED WORK



Despite the success of deep learning across many application domains, the loss 
surfaces of deep neural networks are not well understood.  These loss surfaces 
are an active area of research, which falls into two distinct categories.

The first category explores the local structure of minima found by SGD and its modifications. 
Researchers typically distinguish sharp and 
wide local minima, which are respectively found by using large and small
mini-batch sizes during training. <cit.> and <cit.>, 
for example, claim that flat minima
lead to strong generalization, while sharp minima deliver poor results 
on the test dataset. However, recently <cit.> argue that most existing 
notions of flatness cannot directly explain generalization. To better understand the local structure of DNN loss minima, 
<cit.> proposed a new visualization method for 
the loss surface near the minima found by SGD. 
Applying the method for a variety of different architectures, they
showed that the loss surfaces of modern residual networks are 
seemingly smoother than those of VGG-like models.

The other major category of research
considers global loss structure. One of the main questions in this area is 
how neural networks are able to
overcome poor local optima. <cit.> investigated the link 
between the loss function of a simple fully-connected network  and  the  Hamiltonian
of the spherical spin-glass model. Under strong simplifying assumptions they
showed that the values of the investigated loss function at local optima are within
a well-defined bound. In other research, 
<cit.> showed that under mild conditions gradient descent 
almost surely
converges to a local minimizer and not a saddle point, starting from a random 
initialization.  

In recent work <cit.>
theoretically show that local minima of a neural network with one hidden layer
and ReLU activations can be connected with a curve along which the loss is 
upper-bounded by a constant that depends on the number
of parameters of the network and the “smoothness of the data”.  Their 
theoretical results do not readily generalize to multilayer networks.  
Using a dynamic programming approach they empirically construct a polygonal chain for a CNN on MNIST and an RNN on PTB next word prediction. However, in more difficult settings such as AlexNet on CIFAR-10 their approach struggles to achieve even the modest test accuracy of 80%. Moreover, they do not consider ensembling.

By contrast, we propose a much simpler training procedure that can 
find near-constant accuracy polygonal chains with only one bend between 
optima, even on a range of modern state-of-the-art architectures. 
Inspired by properties of the loss function discovered by our procedure, we also 
propose a new state-of-the-art ensembling method that can be trained 
in the time required to train a single DNN, with compelling performance on many
key benchmarks (e.g., 96.4% accuracy on CIFAR-10).

<cit.> proposed a related ensembling approach that
gathers outputs of neural networks from different epochs at the end of training 
to stabilize final predictions.  More recently, <cit.> proposed 
snapshot ensembles, which use a cosine cyclical learning rate
<cit.> to save “snapshots” of the model during training at times
when the learning rate achieves its minimum.  In our experiments, we compare
our geometrically inspired approach to <cit.>, showing improved 
performance.

<cit.> simultaneously and independently discovered the existence of 
curves connecting local optima in DNN loss landscapes.
To find these curves they used a different approach inspired by the 
Nudged Elastic Band method <cit.> from quantum chemistry.



§ FINDING PATHS BETWEEN MODES



We describe a new method to minimize the training error along a path that connects 
two points in the space of DNN weights.  Section <ref> 
introduces this general procedure for arbitrary parametric curves, and Section 
<ref> describes polygonal chains and Bezier curves
as two example parametrizations of such curves. 
In the supplementary material, we discuss the 
computational complexity of the proposed approach and how to apply 
batch normalization at test time to points on these curves.  We note that after 
curve finding experiments in Section <ref>, we make our key 
methodological proposal for ensembling in Section <ref>.



 §.§ Connection Procedure



Let ŵ_1 and ŵ_2 in ℝ^|net| be two sets of weights
corresponding to two neural networks independently trained by minimizing any
user-specified loss ℒ(w), such as the cross-entropy loss.  Here, |net| is 
the number of weights of the DNN. Moreover, let
ϕ_θ: [0, 1] →ℝ^|net| be
a continuous piecewise smooth parametric curve, with parameters
θ, such that 
ϕ_θ(0) = ŵ_1,  ϕ_θ(1) = ŵ_2.  

To find a path of high accuracy between ŵ_1 and ŵ_2, we propose to find the parameters
θ that minimize the expectation over a uniform distribution on 
the curve, ℓ̂(θ):

    ℓ̂(θ) = 
    ∫ L(ϕ_θ) dϕ_θ/∫ dϕ_θ =  
    ∫_0^1 L(ϕ_θ(t))ϕ'_θ(t) dt/∫_0^1ϕ'_θ(t) dt = ∫_0^1 ℒ(ϕ_θ(t)) q_θ(t) dt = 𝔼_t ∼ q_θ(t)[ ℒ(ϕ_θ(t)) ],

where the distribution q_θ(t) on t ∈ [0, 1] is defined as:
q_θ(t) = ϕ_θ'(t) ·(∫_0^1ϕ_θ'(t) dt)^-1.
The numerator of (<ref>) is the line integral of the loss ℒ on the curve, and the denominator ∫_0^1 ϕ'_θ(t) dt is the normalizing constant of the uniform distribution on the curve defined by ϕ_θ(·).
Stochastic gradients of ℓ̂(θ) in 
Eq. (<ref>) are
generally intractable since q_θ(t) depends on θ. 
Therefore we also propose a more computationally tractable loss

    ℓ(θ) = ∫_0^1ℒ(ϕ_θ(t)) dt =
    𝔼_t ∼ U(0, 1)ℒ(ϕ_θ(t)),

where U(0, 1) is the uniform distribution on [0, 1]. The difference
between (<ref>) and (<ref>) is that the
latter is an expectation of the loss ℒ(ϕ_θ(t)) with
respect to a uniform distribution on t ∈ [0, 1], while (<ref>)
is an expectation with respect to a uniform distribution on the curve.
The two losses coincide, for example, when ϕ_θ(·) defines a polygonal
chain with two line segments of equal length and the parametrization of 
each of the two segments is linear in t. 

To minimize (<ref>), at each iteration we 
sample  t̃ from the uniform distribution U(0,1) and make a gradient step for 
θ  with respect to the loss ℒ(ϕ_θ(t̃)). This way we obtain unbiased estimates of the gradients of ℓ(θ), as

    ∇_θℒ(ϕ_θ(t̃)) ≃𝔼_t ∼ U(0, 1)∇_θℒ(ϕ_θ(t)) = 
    ∇_θ𝔼_t ∼ U(0, 1)ℒ(ϕ_θ(t)) = 
    ∇_θℓ(θ).

We repeat these updates until convergence.



 §.§ Example Parametrizations




  
Polygonal chain

The simplest parametric curve we consider is the polygonal chain 
(see Figure <ref>, right). The trained networks 
ŵ_1 and ŵ_2 serve as the endpoints of the chain and the bends of the chain are
the parameters θ of the curve parametrization. Consider the simplest case of a chain with 
one bend θ. Then

    ϕ_θ(t) = 
    {[        2 ( t θ + (0.5 - t) ŵ_1),                      0 ≤ t ≤ 0.5; 2 ((t - 0.5)  ŵ_2 + (1 - t) θ) ,                     0.5 ≤ t ≤ 1. ] .





  
Bezier curve

A Bezier curve (see Figure <ref>, middle)
provides a convenient parametrization of smooth paths with given endpoints. 
A quadratic Bezier curve ϕ_θ(t) with
endpoints ŵ_1 and ŵ_2 is given by

    ϕ_θ(t) = 
    (1 -t)^2 ŵ_1 + 2t(1-t) θ + t^2 ŵ_2,   0 ≤ t ≤ 1.


These formulas naturally generalize for n bends θ = {w_1, w_2, …, w_n} 
(see supplement).






§ CURVE FINDING EXPERIMENTS



We show that the proposed training procedure in Section <ref>
does indeed find high accuracy paths connecting different modes, across 
a range of architectures and datasets. Moreover,
we further investigate the properties of these curves, showing that they 
correspond to meaningfully different representations that can be ensembled
for improved accuracy.  We use these insights to propose an improved ensembling
procedure in Section <ref>, which we empirically validate in 
Section <ref>.

In particular, we test VGG-16 <cit.>, a
28-layer Wide ResNet with widening factor 10 <cit.>
and a 158-layer ResNet <cit.> on CIFAR-10, and VGG-16, 164-layer ResNet-bottleneck <cit.> on CIFAR-100. For CIFAR-10
and CIFAR-100 we use the same standard data augmentation as <cit.>. 
We provide additional
results, including detailed experiments for fully connected and recurrent networks, in the 
supplement.  

For each model and dataset we train two networks with different random 
initializations to find two modes. Then we use the proposed algorithm of Section <ref> 
to find a path connecting these two modes in the weight space with a quadratic 
Bezier curve and a polygonal chain with one bend. We also connect the two modes
with a line segment for comparison. In all experiments we optimize the loss 
(<ref>), as for Bezier curves the gradient of loss 
(<ref>) is intractable, and for polygonal chains we found 
loss (<ref>) to be more stable. 

Figures <ref> and <ref> show the results of the proposed mode connecting 
procedure for ResNet-164 on CIFAR-100.  Here loss refers to ℓ_2-regularized
cross-entropy loss. 
For both the Bezier curve and polygonal chain, train loss (Figure <ref>, left) and
test error (Figure <ref>, middle) are indeed nearly constant. In addition, we provide plots
of train error and test loss in the supplementary material.
In the supplement, we also include a comprehensive table summarizing all path finding experiments on 
CIFAR-10 and CIFAR-100 for VGGs, ResNets and Wide ResNets, as well as fully connected networks and 
recurrent neural networks, which follow the same general trends. In the supplementary
material we also show that the connecting 
curves can be found consistently as we vary the number of parameters 
in the network, although the ratio of the arclength for the curves to the length
of the line segment connecting the same endpoints decreases with increasing parametrization.
In the supplement, we also measure the losses
(<ref>)  and (<ref>) for all the curves we 
constructed, and find that the values of the two losses are very close,
suggesting that the loss (<ref>) is a good practical approximation 
to the loss (<ref>).

The constant-error curves connecting two given networks 
discovered by the proposed method are not unique. We trained two different 
polygonal chains with the same endpoints and
different random seeds using VGG-16 on CIFAR-10. We then measured the
Euclidean distance between the turning points of these curves. 
For VGG-16 on CIFAR-10 this distance is equal to 29.6 and the distance between 
the endpoints is 50, showing that the curves are not unique. In this instance, we expect
the distance between turning points to be less than the distance between endpoints,
since the locations of the turning points were initialized to the same value (the center
of the line segment connecting the endpoints).

Although high accuracy connecting curves can often be very simple, such as a polygonal chain
with only one bend, we note that line segments directly connecting two modes generally incur high 
error. For VGG-16 on CIFAR-10 the test error goes up to 90% in the center of the
segment. For ResNet-158 and Wide ResNet-28-10 the worst errors along
direct line segments are still high, but relatively less, at 80% and 66%, respectively.
This finding suggests that the loss surfaces of state-of-the-art residual networks are 
indeed more regular than those of classical models like VGG, in accordance with
the observations in <cit.>.

In this paper we focus on connecting pairs of networks trained using the same
hyper-parameters, but from different random initializations.
Building upon our work, <cit.> have 
recently shown that our mode connectivity approach applies to pairs of networks
trained with different batch sizes, optimizers, data augmentation strategies,
weight decays and learning rate schemes.

To motivate the ensembling procedure proposed in the next
section, we now examine how far we 
need to move along a connecting curve to find a point that produces 
substantially different, but still useful, predictions.
Let ŵ_1 and ŵ_2 be two 
distinct sets of weights corresponding to optima obtained
by independently training a DNN two times.
We have shown that there
exists a path connecting ŵ_1 and ŵ_2 with 
high test accuracy. Let ϕ_θ(t), t ∈ [0, 1] parametrize
this path with ϕ_θ(0) = ŵ_1, ϕ_θ(1) = ŵ_2.
We investigate the performance of an ensemble of two networks: the
endpoint ϕ_θ(0) of the curve and a point ϕ_θ(t) on the curve corresponding 
to t ∈ [0, 1]. Figure <ref> (right) shows the test error
of this ensemble as a function of t, for a ResNet-164 on CIFAR-100.  
The test error starts decreasing at t ≈ 0.1
and for t ≥ 0.4 the error of an ensemble is already as low as 
the error of an ensemble of the two independently trained networks used as
the endpoints of the curve. Thus even by moving away from the 
endpoint by a relatively small distance along the curve we can find a network that
produces meaningfully different predictions from the network at the endpoint.
This result also demonstrates that these curves do not exist only due to degenerate
parametrizations of the network (such as rescaling on either side of a ReLU); instead, points 
along the curve correspond to meaningfully different representations of the data that can be 
ensembled for improved performance. In the supplementary
material we show how to create trivially connecting curves that do not have this property.



§ FAST GEOMETRIC ENSEMBLING



In this section, we introduce a practical ensembling procedure, Fast Geometric Ensembling (FGE), 
motivated by our observations about mode connectivity.



In the previous section, we considered ensembling along mode connecting
curves. Suppose now we instead only have one set of weights ŵ 
corresponding to a mode of the loss. We cannot explicitly construct a path ϕ_θ(·)
as before, but we know that multiple paths passing through ŵ exist,
and it is thus possible to move away from ŵ in the weight
space without increasing the loss. Further, we know that we can find
diverse networks providing meaningfully different predictions by making
relatively small steps in the weight space (see Figure <ref>, right).  

Inspired by these observations,
we propose the Fast Geometric Ensembling (FGE) method that aims to find diverse 
networks with 
relatively small steps in the weight space, without leaving a region
that corresponds to low test error. 

While inspired by mode connectivity, 
FGE does not rely on explicitly finding a connecting curve, and thus 
does not require pre-trained endpoints, and so can be trained in the time 
required to train a single network. 

Let us describe Fast Geometric Ensembling. First, we initialize a copy of the network with weights w set equal to the 
weights of the trained network ŵ. Now, to force w to move away from
ŵ without substantially decreasing the prediction accuracy we adopt a 
cyclical learning rate schedule α(·) (see Figure <ref>, left), with the learning rate at iteration i = 1, 2, … defined as

    α(i) = {[       (1 - 2 t(i)) α_1 + 2 t(i) α_2                       0 < t(i) ≤1/2; (2 - 2 t(i)) α_2 + (2 t(i) - 1) α_1                      1/2 < t(i) ≤ 1;                                     ].,
 
where t(i) = 1/c((i - 1, c) + 1), the learning rates are
α_1 > α_2, and the number of iterations
in one cycle is given by even number c. Here by iteration we
mean processing one mini-batch of data.  
We can train the
network w using the standard ℓ_2-regularized cross-entropy loss function
(or any other loss that can be used for DNN training) with 
the proposed learning rate schedule for n iterations.
In the middle of each learning rate cycle when the learning rate reaches
its minimum value α(i) = α_2 (which corresponds to (i - 1,c) + 1=c/2, t(i)=1/2)
we collect the checkpoints of weights w. When the training is finished we 
ensemble the collected models. An outline of the algorithm is provided in 
the supplement.


Figure <ref> (left) illustrates the adopted learning rate schedule.
During the periods when the learning rate is large (close to α_1), w is 
exploring the weight space doing larger steps but sacrificing the test
error. When the learning rate is small (close to α_2), w is in
the exploitation phase in which the steps become smaller and the test error
goes down. The cycle length is usually about 2 to 4 epochs,
so that the method efficiently balances exploration and exploitation
with relatively-small steps in the weight space that are still sufficient
to gather diverse and meaningful networks for the ensemble.

To find a good initialization ŵ for the proposed procedure, we first
train the network with the standard learning rate schedule (the schedule used 
to train single DNN models) for about 80% of the time required to train
a single model. After this pre-training is finished we initialize FGE 
with ŵ and run the proposed 
fast ensembling algorithm for the remaining computational
budget.
In order to get more diverse samples, one can run the algorithm described above 
several times for a smaller number of iterations
initializing from different checkpoints saved during training of 
ŵ, and then ensemble all of the 
models gathered across these runs.

Cyclical learning rates have also recently been considered in <cit.> and <cit.>.
Our proposed method is perhaps most closely related to Snapshot Ensembles <cit.>, but has 
several distinctive features, inspired by our geometric insights.
In particular, Snapshot Ensembles adopt cyclical learning rates with
cycle length on the scale of 20 to 40 epochs
from the beginning of the training as they are trying to do large steps in the 
weight space. However, according to our analysis of the curves it is sufficient
to do relatively small steps in the weight space to get diverse networks, so
we only employ cyclical learning rates with a small cycle length on the scale of
2 to 4 epochs in the last stage of the training.  
As illustrated in Figure <ref> (left), the step sizes made by FGE between saving
two models (that is the euclidean distance between sets of weights of corresponding models in the weight space) are on the scale of 7 for Preactivation-ResNet-164 on
CIFAR-100. For 
Snapshot Ensembles for the same model the distance between two snapshots 
is on the scale of 40.
We also use a piecewise
linear cyclical learning rate schedule following <cit.>
as opposed to the cosine schedule in Snapshot Ensembles. 



§ FAST GEOMETRIC ENSEMBLING EXPERIMENTS





In this section we compare the proposed Fast Geometric Ensembling (FGE) technique against
ensembles of independently trained networks (Ind), and SnapShot Ensembles (SSE) <cit.>,
a recent state-of-the-art fast ensembling approach.   

For the ensembling experiments we use a 164-layer Preactivation-ResNet
in addition to the VGG-16 and Wide ResNet-28-10 models.  Links for implementations to 
these models can be found in the supplement. 

We compare the accuracy of each method as a function of computational budget.
For each network architecture and dataset we denote the number of epochs 
required to train
a single model as B. 
For a kB budget, we run each of Ind, FGE and SSE   
k times from random initializations
and ensemble the models gathered from the k runs.
In our experiments we set B = 200 for VGG-16 and Wide ResNet-28-10 (WRN-28-10) models, and
B=150 for ResNet-164, since 150 epochs is typically sufficient to train this model.  We note
the runtime per epoch for FGE, SSE, and Ind is the same, and so the total computation associated 
with kB budgets is the same for all ensembling approaches.

For Ind, we use an initial learning rate of 0.1 for ResNet and Wide ResNet, and 0.05 for VGG.  
For FGE, with VGG we use cycle length c = 2 epochs, and a total of 22
models in the final ensemble. With ResNet and Wide ResNet we use c = 4 epochs, and the total 
number of models in the final ensemble is 12 for Wide ResNets and 6 for ResNets. 
For VGG we set the learning rates to α_1 = 10^-2, α_2 = 5 · 10^-4; for ResNet
and Wide ResNet models we set α_1 = 5 · 10^-2, 
α_2 = 5 · 10^-4. .
For SSE, we followed <cit.> and varied the initial learning rate
α_0 and number of snapshots per run M.
We report the best results we achieved, which corresponded to 
α_0 = 0.1, M = 4 for ResNet, 
α_0 = 0.1, M = 5 for Wide ResNet, and α_0 = 0.05, M = 5 for VGG.  The total number of models in the FGE ensemble is constrained by network
choice and computational budget. 
Further experimental details are in the supplement.


Table <ref> summarizes the results of the experiments. 
In all conducted experiments FGE outperforms SSE, particularly as we increase
the computational budget. The 
performance improvement against Ind is most noticeable for CIFAR-100.  With 
a large number of classes, any two models are less likely to make the same predictions.
Moreover, there will be greater uncertainty over which representation one should use on 
CIFAR-100, since the number of classes is increased tenfold from CIFAR-10, but the number
of training examples is held constant.  Thus
smart ensembling strategies will be especially important on this dataset.
Indeed in all experiments on CIFAR-100, FGE outperformed all other 
methods. On CIFAR-10, FGE consistently improved upon SSE for all budgets and architectures.  
FGE also improved against Ind for all training budgets with VGG, but is more similar in performance
to Ind on CIFAR-10 when using ResNets.

Figure <ref> (right) illustrates the results for 
Preactivation-ResNet-164 on CIFAR-100 for one and two training
budgets. The training budget B is 150 epochs. Snapshot Ensembles use a cyclical learning rate from the 
beginning of the training and they gather the models for the
ensemble throughout training.
To find a good initialization
we run standard independent 
training for the first 125 epochs 
before applying FGE.
In this case, the whole ensemble is gathered over the following 22 epochs (126-147) to fit in the budget of each of the two runs.
During these 22 epochs FGE is able to gather diverse enough networks to
outperform Snapshot Ensembles both for 1 B and 2 B budgets.

Diversity of predictions of the individual networks is crucial for the 
ensembling performance <cit.>.
We note that the diversity of the networks averaged by FGE is lower than that of
completely independently trained networks. Specifically, two independently trained 
ResNet-164 on CIFAR-100 make different predictions on 19.97% of test objects, while 
two networks from the same FGE run make different predictions on 14.57% of test objects. 
Further, performance of
individual networks averaged by FGE is slightly lower than that of fully trained networks (e.g. 78.0% against 78.5%
on CIFAR100 for ResNet-164). 
However, for a given computational budget FGE can propose many 
more high-performing networks than independent training, leading to better 
ensembling performance (see Table <ref>). 



 §.§ ImageNet


ImageNet ILSVRC-2012 <cit.>
is a large-scale dataset containing  1.2  million  training  
images  and  50000 validation images divided into 1000 classes.

CIFAR-100 is the primary focus of our ensemble experiments. However,
we also include ImageNet results for the proposed FGE procedure, using a 
ResNet-50 architecture. We used a pretrained model with top-1 test error
of 23.87 to initialize the FGE procedure. We then ran FGE for 5 epochs
with a cycle length of 2 epochs and with learning rates α_1 = 10^-3,
α_2 = 10^-5. The top-1 test error-rate of the final ensemble was 
23.31. Thus, in just 5 epochs we could improve the accuracy of the 
model by 0.56 using FGE. The final ensemble contains 4 models (including the pretrained one).
Despite the harder setting of only 5 epochs to
construct an ensemble, FGE performs comparably to the best result reported by 
<cit.> on ImageNet, 23.33 error, which was also achieved using a ResNet-50.



§ DISCUSSION AND FUTURE WORK



We have shown that the optima of deep neural networks are connected 
by simple pathways, such as a polygonal chain with a single bend, 
with near constant accuracy.  We introduced a training
procedure to find these pathways, with a user-specific curve of choice.
We were inspired by these insights to propose a practical new ensembling
approach, Fast Geometric Ensembling, which achieves state-of-the-art 
results on CIFAR-10, CIFAR-100, and ImageNet.  

There are so many exciting future directions for this research. At a high 
level we have shown that even though the loss surfaces of deep neural 
networks are very complex, there is relatively simple structure connecting
different optima.  Indeed, we can now move towards thinking about 
valleys of low loss, rather than isolated modes.  

These valleys could 
inspire new directions for approximate Bayesian inference, such as 
stochastic MCMC approaches which could now jump along these bridges
between modes, rather than getting stuck exploring a single mode.  One 
could similarly derive new proposal distributions for variational inference, 
exploiting the flatness of these pathways.  These geometric insights could also
be used to accelerate the convergence, stability and 
accuracy of optimization procedures like SGD, by helping us understand the 
trajectories along which the optimizer moves, and making it possible to develop procedures 
which can now search in more structured spaces of high accuracy.   One could
also use these paths to construct methods which are more robust to adversarial
attacks, by using an arbitrary collection of diverse models described by a high 
accuracy curve, returning the predictions of a different model for each query from
an adversary. We can also use this new property to create better visualizations 
of DNN loss surfaces.  Indeed, using the proposed training procedure, we were
able to produce new types of visualizations showing the connectivity of modes, which 
are normally depicted as isolated.  We also could continue to build on the new 
training procedure we proposed here, to find curves with particularly desirable properties,
such as diversity of networks.  Indeed, we could start to use entirely new loss 
functions, such as line and surface integrals of cross-entropy across structured
regions of weight space.



  
Acknowledgements. Timur Garipov was supported by Ministry of Education and Science of the Russian Federation (grant 14.756.31.0001). Timur Garipov and Dmitrii Podoprikhin were supported by Samsung Research, Samsung Electronics. Andrew Gordon Wilson and Pavel Izmailov were supported by Facebook Research and NSF IIS-1563887.

plainnat




§ SUPPLEMENTARY MATERIAL

We organize the supplementary material  as follows. Section <ref> 
discusses the computational complexity of the 
proposed curve finding method. Section <ref> describes how to apply batch normalization at test time to points on curves connecting pairs of local optima.
Section <ref> provides formulas for a
polygonal chain and Bezier curve with n bends.
Section <ref> provides details and results of 
experiments on curve finding and contains a table summarizing all path finding experiments.
Section <ref> provides additional visualizations of the train loss
and test accuracy surfaces.
Section <ref> contains details on curve ensembling experiments. Section <ref> describes experiments on relation between mode connectivity and the number of parameters in the networks.
Section <ref>
discusses a trivial construction of curves connecting two modes, where 
points on the curve represent reparameterization of the endpoints, unlike
the curves in the main text. Section <ref> provides details of experiments on FGE. Finally, Section <ref> describes pathways traversed by FGE.





 §.§ Computational complexity of curve finding




The forward pass of the proposed method consists of two steps: computing the point
ϕ_θ(t) and then passing a mini-batch of data through the DNN corresponding
to this point. Similarly, the backward pass consists of first computing the 
gradient of the loss with respect to ϕ_θ(t), and then multiplying the result
by the Jacobian ∂ϕ_θ/∂θ. The second step of
the forward pass and the first step of the backward pass are exactly the same as
the forward and backward pass in the training of a single DNN model. The additional
computational complexity of the procedure compared to single model training comes
from the first step of the forward pass and the second step of the backward pass and
in general depends on the parametrization ϕ_θ(·) of the curve.

In our experiments we use curve parametrizations of a specific form. The general
formula for a curve with one bend is given by

    ϕ_θ(t) = ŵ_1 · c_1(t) + θ· c(t) + ŵ_2 · c_2(t).

Here the parameters of the curve are given by θ∈ℝ^|net| and coefficients c_1, c_2, c: [0,1] →ℝ.



For this family of curves the computational complexity of the first step of the method
is 𝒪(|net|), as we only need to compute a weighted sum of ŵ_1, 
ŵ_2 and
θ∈ℝ^|net|. The Jacobian matrix

    ∂ϕ_θ(t)/∂θ = c(t) · I,

thus the additional computational complexity of the backward pass is also 
𝒪(|net|), as we only need to multiply the gradient with respect to 
ϕ_θ (t) by a scalar. Thus, the total additional computational complexity 
is 𝒪(|net|). In practice we observe that the gap in time-complexity 
between one epoch of training a single model and one epoch of the proposed method
with the same network architecture is usually below 50 %.



 §.§ Batch Normalization


Batch normalization (<cit.>) is essential to modern deep learning 
architectures. Batch normalization re-parametrizes the output of each layer as

    x̂ = γx - μ(x)/σ(x) + ϵ + β,

where μ(x) and σ(x) are the mean and standard deviation of the 
output x, ϵ > 0 is a constant for numerical stability
and γ and 
β are free parameters. 
During training, μ(x) and σ(x) are computed 
separately for each mini-batch and at test time statistics aggregated during training are used.

When connecting two DNNs that use batch normalization, along a curve 
ϕ(t), we compute μ(x) and σ(x) for any given t over 
mini-batches during training,
as usual.  In order to apply batch-normalization to a network on the curve 
at the test stage we compute these statistics with one additional
pass over the data, as running averages for these
networks are not collected during training.



 §.§ Formulas for curves with n bends




For n bends θ = {w_1, w_2, …, w_n}, the parametrization of a
polygonal chain connecting points w_0, w_n+1 is given by

    ϕ_θ(t) = 
    (n+1) ·((t - i/n + 1) · w_i+1 +  (i+1/n + 1 - t) · w_i),

for i/n + 1≤ t ≤i+1/n+1 and 0 ≤ i ≤ n.


For n bends θ = {w_1, w_2, …, w_n}, the parametrization of a
Bezier curve connecting points w_0 and w_n+1 is given by

    ϕ_θ(t) = ∑_i = 0^n+1w_i C_n + 1^i t^i (1 - t)^n +1 - i




 §.§ Curve Finding Experiments











All experiments on curve finding were conducted with TensorFlow (<cit.>) 
and as baseline models we used the following implementations:

	
  * ResNet-bottleneck-164 and Wide ResNet-28-10 (<https://github.com/tensorflow/models/tree/master/research/resnet>);
	
  * ResNet-158 (<https://github.com/tensorflow/models/tree/master/official/resnet>);
	
	
  * A reimplementation of VGG-16  without batch-normalization from (<https://github.com/pytorch/vision/blob/master/torchvision/models/vgg.py>);


Table <ref> summarizes the results of the 
curve finding experiments with 
all datasets and architectures.
For each of the models we report the properties of loss and the 
error on the train and test datasets. For 
each of these metrics we report 3 values: “Max” is the maximum values 
of the metric along the curve, “Int” is a numerical approximation of the integral
∫(ϕ_θ) dϕ_θ / ∫ dϕ_θ,
where <metric> represents the train loss or the error on the train or test
dataset and “Min” is the minimum value of the error on the
curve. “Int”
represents a mean over a uniform distribution on the curve, and for the train
loss it  coincides with the loss (1) in the paper. 
We use an equally-spaced grid with 121 points 
on [0, 1] to estimate the values of “Min”, “Max”, “Int”. 
For “Int” we use the trapezoidal rule to estimate the integral. 
For each dataset and architecture we report the performance of single models
used as the endpoints of the curve as “Single”, the performance of a line
segment connecting the two single networks as “Segment”, the
performance of a quadratic Bezier curve as “Bezier” and the performance of a polygonal chain with one bend as
“Polychain”. Finally, for each curve we report the ratio of its length to the
length of a line segment connecting the two modes.

We also examined the quantity “Mean” defined as 
∫<>(ϕ_θ (t)) dt,  which coincides with the loss
(2) from the paper, but in all our experiments it is nearly  
equal to “Int”.

Besides convolutional and fully-connected architectures we also apply our 
approach to RNN architecture on next word prediction task, PTB dataset 
(<cit.>). As a base model we used the implementation  
available at <https://www.tensorflow.org/tutorials/recurrent>.
As the main loss we consider  perplexity. The results are presented in Table
<ref>.



 §.§ Train loss and test accuracy surfaces




In this section we provide additional visualizations.
Fig. <ref> and Fig. <ref> 
show visualizations of the train loss and test accuracy for
ResNet-164 on CIFAR-100 and VGG-16 on CIFAR-10.




 §.§ Curve Ensembling




Here we explore ensembles constructed from points sampled from these high 
accuracy curves.
In particular, we train a polygonal chain with one bend connecting two independently trained 
ResNet-164
networks on CIFAR-100 and construct an ensemble of networks corresponding
to 50 points placed on an equally-spaced grid on the curve. The resulting
ensemble had 21.03% error-rate on the test dataset. The error-rate of the ensemble
constructed from the endpoints of the curve was 22.0%.  An ensemble of three
independently trained networks has an error rate of 21.01%.  
Thus, the ensemble of the networks on the curve outperformed an ensemble
of its endpoints implying that the curves found by the proposed method are
actually passing through diverse networks that produce predictions different
from those produced by the endpoints of the curve. Moreover, the ensemble based on 
the polygonal chain has the same number of parameters as three independent networks,
and comparable performance.

Furthermore, we can improve the ensemble on the chain without adding additional parameters 
or computational expense, by accounting for the pattern of increased training and
test loss towards the centres of the linear paths shown in Figure <ref>.
While the training and test accuracy are relatively constant, the pattern of loss, 
shared across train and test sets,
indicates overconfidence away from the three points defining the curve: in this 
region, networks tend to output probabilities closer to 1, sometimes with the 
wrong answers. This overconfidence decreases the performance of ensembles constructed from 
the networks sampled on the curves. In order to correct for this overconfidence and 
improve the ensembling performance we use temperature scaling <cit.>, which is
inversely proportional to the loss. 
Figure <ref>, bottom right, illustrates the test loss of ResNet-164 on CIFAR-100
before and after temperature scaling.
After rescaling the predictions of
the networks, the test loss along the curve decreases and flattens. Further, the
test error-rate of the ensemble constructed from the points on the curve went down
from 21.03% to 20.7% after applying the temperature scaling, outperforming
3 independently trained networks.

However, directly ensembling on the curves requires manual intervention for 
temperature scaling, and an additional pass over the training data for each 
of the networks (50 in this case) at test time to perform batch normalization 
as described in section <ref>.  Moreover, we also need to train
at least two networks for the endpoints of the curve. 



 §.§ The Effects of Increasing Parametrization





One possible factor that influences the connectedness of a local minima set is the overparameterization of neural networks. In this section, we investigate the relation between the observed connectedness
of the local optima and the number of parameters (weights) in the
neural network.  We 
start with a network that has three convolutional layers followed by
three fully-connected layers, where each layer has 1000K neurons.
We vary K ∈{0.3, 0.5, 0.8, 1}, and for each value of K we 
train two networks that we connect with a Bezier curve using 
the proposed procedure. 

For each value of K, Figure <ref> 
shows the worst training loss along the curve, maximum of losses
of the endpoints, and the ratio of the length of the curve
and the line segment connecting the two modes.  
Increasing the number of parameters we are able to reduce the 
difference between the worst value of the loss along the curve and
the loss of single models used as the endpoints. The ratio of the length of the found curve and the length of the line
segment connecting the two modes also decreases monotonically with K. 
This result is intuitive, since a greater parametrization allows for more flexibility in how we can 
navigate the loss surfaces.  



 §.§ Trivial connecting curves



For convolutional 
networks with ReLU activations and without batch normalization we can construct 
a path connecting two points in weight space such that the accuracy of 
each point on the curve (excluding the origin
of the weight space)
is at least as good as 
the minimum of the accuracies of the endpoints. 
Unlike the paths found by our procedure, these paths are trivial 
and merely exploit redundancies in the parametrization. Also, the training loss
goes up substantially along these curves. Below we give a construction of such
paths.

Let ŵ_1 and ŵ_2 be two sets of weights.
This path of interest consists of two parts. The first part connects the point
ŵ_1 with 0 and the second one connects the point ŵ_2 with  0. 
We describe  only the first part ϕ(t) of the path, such that 
ϕ(0) = 0, ϕ(1) = ŵ_1, as the second part is completely analogous. 
Let the weights of the network ŵ_1 be 
{W_i, b_i}_1 ≤ i ≤ n where W_i, b_i are the weights and biases of
the i-th layer, and n is the total number of layers.
Throughout the derivation we consider the inputs of the network fixed.
The output of the i-th layer o_i = W_i (o_i-1) + b_i, 1 ≤ i ≤ n, 
where i = 0 corresponds to the first layer and i = n corresponds to logits
(the outputs of the last layer).
We construct ϕ(t) = {W_i(t), b_i(t)}_1 ≤ i ≤ n in the following 
way. We set W_i(t) = W_it 
and b_i(t) = b_i t^i. It is easy to see that logits of the network with 
weights ϕ(t) are equal to o_n(t) = t^n o_n for all t > 0. 
Note that the predicted labels corresponding to the logits o_n(t) and 
o_n are the same, so the accuracy of all networks corresponding to t > 0
is the same.





 §.§ Fast geometric ensembling experiments


Alg. <ref> provides an outline of the algorithm.
As baseline models we used the following implementations:

	
  * VGG-16 (<https://github.com/pytorch/vision/blob/master/torchvision/models/vgg.py>);
	
  * Preactivation-ResNet-164 (<https://github.com/bearpaw/pytorch-classification/blob/master/models/cifar/preresnet.py>);
	
  * ResNet-50 ImageNet (<https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py>);
	
  * Wide ResNet-28-10 (<https://github.com/meliketoy/wide-resnet.pytorch/blob/master/networks/wide_resnet.py>);


For the FGE (Fast Geometric Ensembling) strategy on 
ResNet we run the FGE routine summarized in Alg. 1 after epoch 125 of the 
usual (same as Ind) training for 22 epochs. The total training time
is thus 125 + 22 = 147 epochs.
For VGG and Wide ResNet models we run the pre-training procedure for 156 epochs
to initialize FGE.
Then we run FGE for 22 epochs starting from checkpoints corresponding to 
epochs 120 and 156 and ensemble all the gathered models. 
The total training time is thus 156 + 22 + 22 = 200 epochs. 
For VGG we use cycle length c = 2 epochs, which
means that the total number of models in the final ensemble is 22.
For ResNet and Wide ResNet we use c = 4 epochs, and the total number of models in the
final ensemble is 12 for Wide ResNets and 6 for ResNets.




 §.§ Polygonal chain connecting FGE proposals



In order to better understand the trajectories followed by FGE we construct
a polygonal chain connecting the points that FGE ensembles. Suppose we run 
FGE for n learning rate cycles obtaining  n points w_1, w_2, …, w_n 
in the weight space that correspond to the lowest values of the learning rate. 
We then consider the polygonal chain consisting of the line segments connecting 
w_i to w_i+1 for i=1, …, n-1. We plot test accuracy and train error
along this polygonal chain in Figure <ref>. We observe that
along this curve both train loss and test error remain low, agreeing with
our intuition that FGE follows the paths of low loss and error. Surprisingly,
we find that the points on the line segments connecting the weights w_i, w_i+1
have lower train loss and test error than w_i and w_i+1. See 
<cit.> for a detailed discussion of this phenomenon.



