Mean Field Residual Networks: On the Edge of Chaos

Greg Yang∗

Samuel S. Schoenholz

arXiv:1712.08969v1 [cs.NE] 24 Dec 2017

Microsoft Research AI
gregyang@microsoft.com

Google Brain
schsam@google.com

Abstract
We study randomly initialized residual networks using mean field theory and the
theory of difference equations. Classical feedforward neural networks, such as
those with tanh activations, exhibit exponential behavior on the average when
propagating inputs forward or gradients backward. The exponential forward dynamics causes rapid collapsing of the input space geometry, while the exponential
backward dynamics causes drastic vanishing or exploding gradients. We show,
in contrast, that by adding skip connections, the network will, depending on the
nonlinearity, adopt subexponential forward and backward dynamics, and in many
cases in fact polynomial. The exponents of these polynomials are obtained through
analytic methods and proved and verified empirically to be correct. In terms of
the “edge of chaos” hypothesis, these subexponential and polynomial laws allow
residual networks to “hover over the boundary between stability and chaos,” thus
preserving the geometry of the input space and the gradient information flow. In
our experiments, for each activation function we study here, we initialize residual
networks with different hyperparameters and train them on MNIST. Remarkably,
our initialization time theory can accurately predict test time performance of these
networks, by tracking either the expected amount of gradient explosion or the
expected squared distance between the images of two input vectors. Importantly,
we show, theoretically as well as empirically, that common initializations such
as the Xavier or the He schemes are not optimal for residual networks, because
the optimal initialization variances depend on the depth. Finally, we have made
mathematical contributions by deriving several new identities for the kernels of
powers of ReLU functions by relating them to the zeroth Bessel function of the
second kind.

1

Introduction

Previous works [9, 3, 11] have shown that randomly initialized neural networks exhibit a spectrum of
behavior with depth, from stable to chaotic, which depends on the variance of the initializations: the
cosine distance of two input vectors converges exponentially fast with depth to a fixed point in [0, 1];
if this fixed point is 1, then the behavior is stable; if this fixed point is 0, then the behavior is chaotic.
It has been argued in many prior works [1, 9] that effective computation can only be supported by
a dynamical behavior that is on the edge of chaos. Too much stability prevents the neural network
from telling apart two different inputs. While some chaotic behavior can increase the expressivity
of a network, too much chaos makes the neural network think two similar inputs are very different.
At the same time, the same initialization variances also control how far gradient information can be
propagated through the network; the networks with chaotic forward dynamics will tend to suffer from
exploding gradients, while networks with stable forward dynamics will tend to suffer from vanishing
gradients.
∗

Work done while at Harvard University

31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.

These works have focused on vanilla (fully connected) feedforward networks. Here we consider
residual networks [6, 7] (with fully-connected layers and without batchnorm), which are a family
of recently proposed neural network architectures that has achieved state-of-the-art performance on
image recognition tasks, beating all other approaches by a large margin. The main innovation of this
family of architectures is the addition of a passthrough (identity) connection from the previous layer
to the next, such that the usual nonlinearity computes the “residual” between the next-layer activation
and the previous-layer activation.
In this work, we seek to characterize randomly initialized residual networks. One of our main results
is that random residual networks for many nonlinearities such as tanh live on the edge of chaos,
in that the cosine distance of two input vectors will converge to a fixed point at a polynomial rate,
rather than an exponential rate, as with vanilla tanh networks. Thus a typical residual network
will slowly cross the stable-chaotic boundary with depth, hovering around this boundary for many
layers. In addition, for most of the nonlinearities considered here, the mean field estimate of the
gradient grows subexponentially with depth. In fact, for α-ReLU, the αth-power of ReLU, for
α < 1, the gradient grows only polynomially. These theoretical results provide some theoretical
justification for why residual networks work so well in practice. In our experiments, we are also able
to predict surprisingly well the relative performances of trained residual networks based only on
their initialization hyperparameters, in a variety of settings. In particular, we find that the quality of
initialization for tanh resnets is determined by trainability (how much gradient explosion on average)
while that for (α-)ReLU resnets is determined by expressivity (how far can two different input vectors
be pulled apart) (see Section 6). To the best of our knowledge, this is the first time that a quantity other
than gradient explosion/vanishing has been found to control the quality of initialization. We establish
theoretically and empirically that the best initialization variances for residual networks depend on
the depth of the network (contrary to the feedforward case [11]), so that common initialization
schemes like Xavier [4] or He [5] cannot be optimal. In fact, even the rationale of He initialization
is incorrect for ReLU residual networks because it tries to control gradient dynamics rather than
expressivity. However we want to emphasize that we study a simplified model of residual networks
in this work, with no batchnorm or convolutional layers, so that these results are not necessarily
indicative of the MSRA residual network used in practice [6].
In the body of this paper, we give account of general intuition and/or proof strategy when appropriate
for our theoretical results, but we relegate all formal statements and proofs to the appendix.

2

Background

Consider a vanilla feedforward neural network of L layers, with each layer l having N (l) neurons;
here layer 0 is the input layer. For the ease of presentation we assume all hidden layer widths are the
(0)
(0)
same N (l) = N for all l > 0. Let x(0) = (x1 , . . . , xN (0) ) be the input vector to the network, and
let x(l) for l > 0 be the activation of layer l. Then a neural network is given by the equations
(l)

(l)

(l)

xi = φ(hi ),

hi =

N
X

(l) (l−1)

wij xj

(l)

+ bi

j=1

where (i) h(l) is the pre-activation at layer l, (ii) w(l) is the weight matrix, (iii) b(l) is the bias vector,
and (iv) φ is a nonlinearity, for example tanh or ReLU, which is applied coordinatewise to its input.
To lighten up notation, we suppress the explicit layer numbers l and write
X
xi = φ(hi ),
hi =
wij xj + bi
j

where • implicitly denotes •(l) , and • denotes •(l−1) (and analogously, • denotes •(l+1) ).
A series of papers [9, 10, 11] investigated the “average behavior” of random neural networks
(l)
(l)
2
sampled via wij ∼ N (0, σw
/N ), bi ∼ N (0, σb2 ), for fixed parameters σw and σb , independent of l.
P
N
Consider the expectation of N1 i=1 x2i , the normalized squared length of x, over the sampling of
w and b. Poole et al. [9] showed that this quantity converges to a fixed point exponentially fast for
sigmoid nonlinearities. Now suppose we propagate two different vectors x(0) and (x(0) )0 through the
2

PN
network. Poole et al. [9] also showed that the expectation of the normalized dot product N1 i=1 xi x0i
converges exponentially fast to a fixed point. The ratio between the normalized squared length and
the normalized dot product is the cosine distance between x and x0 . Thus these two exponential
convergence results show that the cosine distance converges exponentially fast to a fixed point as
well. Intuitively, this means that a vanilla feedforward network “forgets” the geometry of the input
space “very quickly,” after only a few layers.
In addition, Schoenholz et al. [11], under certain independence assumptions, showed that the expected
normalized squared norm of the gradient also vanishes or explodes in an exponential fashion with
depth, with the ”half-life” controlled by σw and σb . They verified that this theoretical ”half-life”
correlates in practice with the maximal number of layers that are admissible to good performance.
At the same time, Daniely et al. [3] published work of similar nature, but phrased in the language
of reproducing kernel Hilbert spaces, and provided high probability estimates that are meaningful
for the case when the width N is finite and the depth is logarithmic in N . However, they essentially
fixed the variance parameters σ• , and furthermore, their framework (for example the notion of a
“skeleton”) does not immediately generalize to the residual network case.
In this work, we show that residual networks have very different dynamics from vanilla feedforward
networks. In most cases, the cosine distance convergence rate and the gradient growth rate are
subexponential in a residual network, and in most cases, these rates may be polynomial.

3

Preliminaries

Residual networks were first introduced by [6] and later refined by [7], and they are now commonplace
among deployed neural systems. The key innovation there is the addition of a shortcut connection
from the previous layer to the next. We define the following idealized architectures for ease of
analysis. Note that we only consider fully-connected affine layers instead of convolutional layers. A
reduced residual network (RRN) has the recurrence
X
xi = φ(hi ) + x,
hi =
wij xj + bi .
j

A (full) residual network (FRN) in addition has an affine connection given by weights v and biases
a from the nonlinearity φ(h) to the next layer:
X
X
xi =
vij φ(hj ) + xi + ai ,
hi =
wij xj + bi
j

j

We are interested in the “average behavior” of these network when the weights and biases,
(l) (l) (l)
(l)
wij , bi , vij , and ai are sampled i.i.d. from Gaussian distributions resp. with standard devi(l)

2
ations σw , σb , σv , and σa , independent from l. Here we take the variance of wij to be σw
/N so that
(l)

2
the variance of each hi is σw
, assuming each xj is fixed (similarity for vij ). Such an initialization
scheme is standard in practice.

We make several key “physical assumptions” to make theoretical computations tractable:
(l)

(l)

Axiom 3.1 (Symmetry of activations and gradients). (a) We assume h(hi )2 i = h(hj )2 i and
(0)

(0)

(l)

h(xi )2 i = h(xj )2 i for any i, j, l. (b) We also assume that the gradient ∂E/∂xi with respect to
(l)

(l)

the loss function E satisfies h(∂E/∂xi )2 i = h(∂E/∂xj )2 i for any i, j, l.
One can see that Axiom 3.1(a) is satisfied if the input x(0) ∈ {±1}N and Axiom 3.1(b) is satisfied
if Axiom 3.2 below is true and the gradient at the last layer ∂E/∂xL ∈ {±1}N . But in general it
(l)
(l)
is justified both empirically and theoretically as an approximation, because (hi )2 − (hj )2 stays
(l)

(l)

about constant with l, but (hi )2 and (hj )2 grow rather quickly at the same pace with l (as will be
(l)

seen later in calculations), so that their additive difference becomes negligible; similarly for (xi )2
(l)
and (∂E/∂hi )2 .
3

Axiom 3.2 (Gradient independence). (a) We assume the we use a different set of weights for backpropagation than those used to compute the network outputs, but sampled i.i.d. from the same
(l)
distributions. (b) For any loss function E, we assume that the gradient at layer l, ∂E/∂xi , is
(l)
(l−1)
independent from all activations hj and xj
from the previous layer.
Axiom 3.2(a) was first made in [11] for computing the mean field theory of gradients for feedforward
tanh networks. This is similar to the practice of feedback alignment [8]. Even though we are the
first to explicitly formulate Axiom 3.2(b), in fact it was already applied implicitly in the gradient
(l)
(l+1)
calculations of [11]. Note that a priori Axiom 3.2(b) is not true, as ∂E/∂xi depends on φ̇(hk
)
(l)
(l−1)
for every k, which depend on hj for each j, and which depends on xk
for every k. Nevertheless,
in practice both subassumptions hold very well.
Now we define the central quantities studied in this paper. Inevitably, our paper involves a large
amount of notation that may be confusing for the first-time reader. We have included a glossary of
symbols (Table A.1) to ameliorate notation confusion.
(l)

Definition 3.3. Fix an input x(0) . Define the length quantities q(l) := h(h1 )2 i and p(l) :=
(l)
h(x1 )2 i for l > 0 and p(0) = kx(0) k2 /N . Here the expectations h•i are taken over all random
initialization of weights and biases for all layers l, as N → ∞ (large width limit).
Note that in our definition, the index 1 does not matter by Axiom 3.1.
Definition 3.4. Fix two inputs x(0) and x(0)0 . We write •0 to denote a quantity • with respect to the
0
(l) (l)
(l) (l)
input x(0) . Then define the correlation quantities γ (l) := hh1 h1 0 i and λ(l) := hx1 x1 0 i for
l > 0 and γ (0) = x(0) · x(0)0 /N , where the expectations h•i are taken over all random initialization
of weights and biases for all layers l, as N → ∞ (large width limit). Again, here the index 1
1
does not matter by Axiom 3.1. By metric expressivity, we mean s(l) := 2N
hkx(l) − x(l)0 k2 i =
1
1
(l) 2
(l)0 2
(l)
(l)0
(l)
(l)0
(l)
k i − 2hx · x i) = 2 (p + p ) − γ . Additionally, define the cosine
2N (hkx k i + hkx
p
p
distance quantities e(l) := γ (l) / p(l) p(l)0 and c(l) := λ(l) / q(l) q(l)0 , and we will also call e(l)
angular expressivity.
In this paper, for the ease of presentation, we assume p(0) = p(0)0 . Then, as we will see, p(l) =
p(l)0 , q(l) = q(l)0 for all l, and as a result, e(l) = γ (l) /p(l) and s(l) = p(l) − γ (l) = (1 − e(l) )p(l) .
(L)

Definition 3.5. Fix an input x(0) and a gradient vector (∂E/∂xi )i of some loss function E with
(l)
(l)
respect to the last layer x(L) . Then define the gradient quantities χ(l) := h(∂E/∂x1 )2 i, χ• :=
(l)
(l)
(l)
h(∂E/∂•1 )2 i for • = a, b, and χ• := h(∂E/∂•11 )2 i for • = w, v. Here the expectations are
taken with Axiom 3.2 in mind, over both random initialization of forward and backward weights and
biases, as N → ∞ (large width limit). Again, the index 1 or 11 does not matter by Axiom 3.1.
Asymptotic notations. The expressions f = O(g) ⇐⇒ g = Ω(f ) have their typical meanings,
and f = Θ(g) iff f = O(g), g = O(f ). We take f (x) = Õ(g(x)) ⇐⇒ g(x) = Ω̃(f (x)) to
mean f (x) = O(g logk x) for some k ∈ Z (this is slightly different from the standard usage of
Õ), and f = Θ̃(g) ⇐⇒ f = Õ(g) & g = Õ(f ). We introduce a new notation: f = Θ̌(g) if
f (x) = O(g(x) · x ) and f (x) = Ω(g(x) · x− ), as x → ∞, for any  > 0. All asymptotic notations
are sign-less, i.e. can indicate either positive or negative quantities, unless stated otherwise.

4

Overview

The primary reason we may say anything about the average behavior of any of the above quantities is
the central limit theorem: every time the activations of the previous layer pass through an affine layer
whose weights are sampled i.i.d., the output is a sum of a large number of random variables, and thus
follows approximately Gaussian distributions. The mean and variance of these distributions can be
computed by keeping track of the mean and variances of the activations in the previous layer.
In what follows, we use this technique to derive recurrence equations governing p, q, γ, λ, χ for
different architectures and different activation functions. We use these equations to investigate the
4

dynamics of e and s, the key quantities in the forward pass, and the dynamics of χ, the key quantity
in the backward pass.
The cosine distance e in some sense measures the angular geometry of two vectors. If e = 1, then the
vectors are parallel; if e = 0, then they are orthogonal. Just as in [9] and [11], we will show that in
all of the architectures and activations we consider in this paper, e(l) converges to a fixed point e∗ as
l → ∞ 1 . Thus, on the average, as vectors propagate through network, the geometry of the original
input space, for example, linear separability, is “forgotten” by residual networks as well as by vanilla
networks. But we will prove and verify experimentally that, while Poole et al. [9] and [11] showed
that the convergence rate to e∗ is exponential in a vanilla network, the convergence rate is rather
only polynomial in residual networks, for tanh and α-ReLU (Defn 5.2) nonlinearities; see Thm B.5,
Thm B.11, Thm B.17, and Thm B.18. This slow convergence preserves geometric information in the
input space, and allows a typical residual network to “hover over the edge of chaos”: Even when the
cosine distance e(l) converges to 0, corresponding to “chaos”, (resp. 1, corresponding to “stability”),
for the number of layers usually seen in practice, e(l) will reside well away from 0 (resp. 1).
Similarly, the quantity s measures the metric geometry of two vectors. The evolution of s(l) with l
tells us the ability of the average network to separate two input points in terms of Euclidean distance.
Again, for tanh and α-ReLU (α < 1) nonlinearities, s varies only polynomially with l.
On the other hand, χ(l) measures the size of gradient at layer l, and through it we track the dynamics
of gradient backpropagation, be it explosion or vanishing. In contrast to vanilla tanh networks, which
can experience both of these two phenomenon depending on the initialization variances, typical
residual networks cannot have vanishing gradient, in the sense of vanishing χ(l) as l → 1; see
Thm B.5 and Thm B.12. Furthermore, while vanilla tanh networks exhibit exponentially vanishing
or exploding gradients, all of the activation/architecture pairings considered here, except the full
residual network with ReLU, have subexponential
gradient dynamics. While tanh residual networks
√
(0)
(l)
(reduced or full) has χ ≈ exp(Θ( l))χ (Thm B.13), α-ReLU residual networks for α < 1
have χ(0) ≈ poly(l)χ(l) (Thm B.20). Instead of ∂E/∂xi , we may also consider the size of gradients
of actual trainable parameters. For tanh and α-ReLU with α < 1, they are still subexponential and
polynomial (Thm B.21). On the other hand, while χ(0) = exp(Θ(l))χ(l) for a ReLU resnet, its
weight gradients have size independent of layer, within O(1) (Thm B.21)! This is the only instance
in this paper of gradient norm being completely preserved across layers.
The above overviews the theoretical portion of this paper. Through experiments, we discover that we
can very accurately predict whether one random initialization leads to better performance than another
on the test set, after training, by leveraging this theory we build. Residual networks of different
nonlinearities have different controlling quantities: for resnets with tanh, the optimal initialization
is obtained by controlling the gradient explosion χ(0) /χ(L) ; whereas for ReLU and α-ReLU, the
optimal initialization is obtained by maximizing s without running into numerical issues (with floating
point computation). See Section 6 for details.
Over the course of our investigation of α-ReLU, we derived several new identities involving the
associated kernel functions, first defined in [2], which relate them to the zeroth Bessel functions
(Lemmas C.31 to C.34).

5

Theoretical Results

In what follows in the main text, we assume σ• > 0 for all • = w, v, b, a; in the appendix, the formal
statement of each main theorem will contain results for other cases. We are interested in the two
major categories of nonlinearities used today: tanh-like and rectified units. We make the following
formal definitions as a foundation for further consideration.
Definition 5.1. We say a function φ is tanh-like if φ is antisymmetric (φ(−x) = −φ(x)), |φ(x)| ≤ 1
for all x, φ(x) ≥ 0, ∀x ≥ 0, and φ(x) monotonically increases to 1 as x → ∞.
Definition 5.2. Define the α-ReLU ψα (x) = xα if x > 0 and 0 otherwise. 2
By applying the central limit theorem as described in the last section, we derive a set of recurrences
for different activation/architecture pairs, shown in Table 1 (see appendix for proofs). They leverage
certain integral transforms 3 as in the following
5

Table 1: Main Recurrences

Antisymmetric/RRN

Any/FRN

2
q = σw
p + σb2

p = Vφ(q) + p

2
q = σw
p + σb2

p = σv2 Vφ(q) + σa2 + p

2
λ = σw
γ + σb2

γ = Wφ(q, λ) + γ

2
λ = σw
γ + σb2

γ = σv2 Wφ(q, λ) + σa2 + γ

2
χ = (σw
Vφ̇(q) + 1)χ

Theorems B.2, B.3, B.5

2
χ = (σv2 σw
Vφ̇(q) + 1)χ

Theorems B.8, B.10, B.12

Table 2: Summary of Main Dynamics Results. Note that while χ(l) is exponential for ReLU/FRN, the gradients
with respect to weight parameters have norms (χw and χv ) constant in l (Thm B.21). Also, the χ(l) entry for
α-ReLU is for α ∈ (3/4, 1) only

p(l)
s(l)
e(l) − e∗
χ(l)

Tanh/RRN

Tanh/FRN

ReLU/FRN

α-ReLU/FRN, α < 1

Θ(l),
B.2
Θ(l),
B.4
2
Θ̌(l π −1 ),
B.4
√
exp(Θ( l)), B.6

Θ(l),
B.9
Θ(l),
B.11
poly(l),
B.11
√
exp(Θ( l)), B.12

exp(Θ(l)), B.16
exp(Θ(l)), B.17
Θ(l−2 ), B.17

Θ(l1/(1−α) ),
Θ(l1/(1−α) ),
poly(l),

exp(Θ(l)), B.20

Θ(l (1−α)(2α−1) ), B.20

B.16
B.18
B.18

α2

Definition 5.3. Define the transforms V 
and Wby Vφ(q) := E[φ(z)2 : z ∼ N (0, q)] and
ρ ν
Wφ(ρ, ν) := E[φ(z)φ(z 0 ) : (z, z 0 ) ∼ N (0,
)].
ν ρ
These recurrences are able to track the corresponding quantities in practice very well. For example,
Fig. 1 compares theory vs experiments for the tanh/FRN pair. The agreement is very good for
tanh/RRN (not shown, but similar to the case of tanh/FRN with σv = 1 and σa = 0) and αReLU/FRN as well (see Fig. A.1).
As mentioned in previous sections, we seek to characterize the long term/high depth behavior of all
of the quantities defined in Section 2. To do so, we solve for the asymptotics of the recurrences in
Table 1, where φ is instantiated with tanh or α-ReLU. Our main dynamics results are summarized in
Table 2.
5.1

Tanh

Forward dynamics. When φ = tanh, p(l) and q(l) increase as Θ(l) in either RRN or FRN
(Thm B.2), as one might expect by observing that V tanh(q) → 1 as q → ∞ so that, for example in
the RRN case, the recurrence p = V tanh(q) + p becomes p = 1 + p. This is confirmed graphically
by the black lines of the leftmost chart of Fig. 1. We carefully verify that this intuition is correct in its
proof in the appendix, and find that in fact p(l) ∼ l in the RRN case and p(l) ∼ (σv2 + σa2 )l in the
FRN case.
What about γ (l) ? The middle chart of Fig. 1 shows that over time, e(l) = γ (l) /p(l) contracts toward
the center of the interval [0, 1], but from the looks of it, it is not clear whether there is a stable fixed
point e∗ of e or not. We prove that, in fact, all trajectories of e not starting at 1 do converge to a
single fixed point, but only at a polynomial rate, in both the RRN and FRN cases (Thm B.2 and
Thm B.10); we can even explicitly compute the fixed point and the rate of convergence: For FRN,
there is a unique stable fixed point e∗ < 1 determined by the equation
e∗ =

1
2
[σ 2 arcsin (e∗ ) + σa2 ],
σv2 + σa2 v π

∗

and |e∗ − e(l) | decreases like l−δ , where
δ ∗ := 1 −

1
2
σv2
p
.
π 1 − (e∗ )2 σv2 + σa2
6

Figure 1: Our equations predict the relevant quantities very well in practice. These plots make the comparison
between prediction and measurements for the full resnet with tanh activation, with σv2 = 1.5, σa2 = .5,
2
σw
= 1.69, σb2 = .49. Left-to-right: (a) p(l) and γ (l) against layer l for 200 layers. (b) e(l) = γ (l) /p(l)
against l for 200 layers. Both (a) and (b) trace out curves for different initial conditions. (c) Different gradient
quantities against l for 50 layers. From left to right the layer number l decreases, following the direction of
backpropagation. Notice that the gradient increases in norm as l → 1. All three figures exhibit smooth curves,
which are theoretical estimates, and irregular curves with shades around them, which indicate empirical means
and standard deviations (both of which taken in regular scale, not log scale). (a) and (b) are made with 20 runs
of resnets of width 1000. (c) is made with 25 runs of resnets of width 250.
1.0

0.8

0.6

0.4

1-

2
π

0.2

*
δ*

0.0
0

1

2

3

4

σa /σv

∗

Figure 2: Left-to-right: (a) Plots of e∗ and δ ∗ against σa /σv . (b) In log-log scale: the dashed line is l−δ −1 ,
and the colored lines are e(l) − e(l−1) for different initial conditions e(0) . That they
√become parallel at about
∗
l = 400 on verifies that e(l) = Θ(l−δ ). 4 (c) In log-log scale: The dashed line is A l (A given in Thm B.13),
and the colored lines are log(•(1) /•(l) ) for • = χ, χb , χw . That they all converge together starting around
l = 1000 indicates that the approximation in Thm B.13 is very good for large l.

Since e∗ < 1, s = (1 − e)p = Θ(p) = Θ(l). The case of RRN can be viewed as a special case
of the above, setting σv2 = 1 and σa2 = 0, which yields e∗ = 0 and δ ∗ = 1 − π2 . We observe that
both e∗ and δ ∗ only depend on the ratio ρ := σa /σv , so in Fig. 2 we graph these two quantities as
a function of ρ. e∗ and δ ∗ both increase with ρ and asymptotically approach 1 and 1/2 respectively
from below. When ρ = σa = 0, e∗ = 0 and δ ∗ = 1 − π2 . Thus the rate of convergence at its slowest
for tanh/FRN is δ ∗ = 1 − π2 ≈ 0.36338, where asymptotically the network tends toward a chaotic
regime e∗ = 0, corresponding to a large weight variance and a small bias variance; it at its fastest is
δ ∗ = 1/2, where asymptotically the network tends toward a stable regime e∗ = 1, corresponding to
∗
a large bias variance and small weight variance. We verify δ ∗ by comparing e(l) − e(l−1) to l−δ −1
∗
∗
in log-log scale. If e(l) = Θ(l−δ ), then e(l) − e(l−1) = Θ(l−δ −1 ) and should obtain the same
∗
slope as l−δ −1 as l → ∞. The middle figure of Fig. 2 ascertains that this is indeed the case, starting
around layer number 400.
Backward dynamics. Finally, we show that the gradient is approximated by
√
√
χ(m) = exp(A( l − m) + O(log l − log m))χ(l)
(?)
q
q
σ2 σ
where A = 43 π2 σw in the RRN case and A = 43 π2 √ v2 w 2 in the FRN case (Thm B.6 and
σv +σa

Thm B.13). The rightmost plot of Fig. 2 verifies that indeed, for large l ≥ 1000, this is a very good
approximation. This demonstrates that the mean field assumption of independent backpropagation
weights is very practical and convenient even for residual networks.
7

Note that in the FRN case, the constant A can be decomposed into A =
σa2 /σv2 )−1/2 .

4
3

q

2
π

· σv · σw · (1 +

∗

Consider the ratio ρ := σa /σv . If ρ  1, then e ≈ 1 (Fig. C.17), meaning that the
typical network essentially computes a constant function, and thus unexpressive; at the same time,
large ρ makes A small, and thus ameliorating the gradient explosion problem, making the network
more trainable. On the other hand, if ρ  1, then e∗ ≈ 0 (Fig. C.17), the typical network can tease
out the finest differences between any two input vectors, and a final linear layer on top of such a
network should be able to express a wide variety of functions [9]; at the same time, small ρ increases
A, worsening the gradient explosion problem, making the network less trainable. This is the same
expressivity-trainability tradeoff discussed in [11].

5.2

α-ReLU

Forward dynamics. As with the tanh case, to deduce the asymptotic behavior of random α-ReLU
resnets, we need to understand the transforms Vψα and Wψα . Fortunately, Vψα has a closed form,
and Wψα has been studied before [2]. In particular, if α > − 12 , then Vψα (q) = cα qα , where
cα is a constant with a closed form given by Lemma B.15. In addition, by [2], we know that
Wψα (q, cq) = Vψα (q)Jα (c) for Jα given in Appendix C.7.1. Fig. C.17 shows a comparison of Jα
for different αs along with the identity function.
2
Substituting in cα qα for Vψα , we get a difference equation p − p = σv2 cα (σw
p + σb2 )α + σa2
governing the evolution of p. This should be reminiscent of the differential equation Ṗ (l) = CP (l)α ,
which has solution ∝ l1/(1−α) for α < 1, and ∝ exp(Cl) when α = 1. And indeed, the solutions p(l)
to these difference equations behave asymptotically exactly like so (Thm B.16). Thus ReLU behaves
2
very explosively compared to α-ReLU with α < 1. In fact, in simulations, for σw
= 1.69 and
2
σv = 1.5, the ReLU resnets overflows into infs after around 100 layers, while there’s no problem
from any other kind of networks we consider.

Regardless, α-ReLU for all α massages e(l) toward a fixed point e∗ that depends on α. When
φ = ψ1 , the standard ReLU, e(l) converges to 1 asymptotically as Cl−2 for an explicit constant C
depending on σv and σw only (Thm B.17), so that s = (1 − e)p = Θ(l−2 exp(Θ(l))) = exp(Θ(l)).
When φ = ψα for α < 1, then e(l) converges to the nonunit fixed point e∗ of Jα at a rate of Θ̌(l−µ ),
where µ = (1 − J̇α (e∗ ))/(1 − α) is independent of the variances (Thm B.18), so that s = Θ(p).
These rates are verified in Fig. A.2.

Backward dynamics. Finally, we have also characterized the rate of gradient growth for any
α ∈ ( 34 , 1].5 In the case of α = 1, the dynamics of χ is exponential, the same as that of p,
2
+ 1. For α ∈ ( 34 , 1), the dynamics is polynomial, but
χ(l−m) = χ(l) B m where B = 12 σv2 σw
with different exponent in general from that of the forward pass: χ(l−m) = Θ(1)χ(l) (l/(l − m))R
α2
for R = (1−α)(2α−1)
, where the constants in Θ(1) do not depend on l or m. This exponent R is
3
minimized on α ∈ [ 4 , 1) at α = 3/4, where R = 9/2 (but on α ∈ ( 12 , 1) it is minimized at α = 2/3,
where R = 4); see Fig. B.8. These exponents are verified empirically in Fig. A.2.
Looking only at χ and the gradients against the biases, it seems that ReLU suffers from a dramatic
case of exploding gradients. But in fact, because χ gains a factor of B moving backwards while p
(l−m)
(l−m)
loses a factor of B, the gradient norm χw
(and similarly for χv
) is independent of how far,
m, the gradient has been propagated (Thm B.21) — this is certainly the best gradient preservation
among all of the models considered in this paper. Thus strangely, random ReLU FRN exhibits both
the best (constant for v and w) and the worse (exponential for a and b) gradient dynamics. This begs
the question, then, is this a better deal than other α-ReLU for which for any learnable parameter we
have at most a polynomial blowup with depth in its gradient? Our experiments (discussed below)
show that α-ReLU is useful to the extent that smaller α avoids numerical issues with exponentiating
forward and backward dynamics, but the best performance is given by the largest α that avoids them
(Fig. 3(c, d)); in fact, the metric expressivity s, determines performance, not gradient explosion (see
α-ReLU experiments).
8

250

250

200

200

150

150

100

100

50

50
0.5

1.0
σw2

1.5

2.0

200
150

L

300

L

L

300

100
50

0.5

1.0
σw2

1.5

2.0

0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6
σa2 /σv2

300
1.0
2.0
3.0
4.0
5.0

200

L

100

50
40
30
20

10

0.05

0.10 0.15

0.25

0.50

1.00 1.50

2
σw

2
Figure 3: From left to right, top to bottom: (a) and (b): σw
, L, and test set accuracy of a grid of tanh reduced
(left) and full (right) resnets trained on MNIST. Color indicates performance, with ligher colors indicating
higher accuracy on test set. Other than the values on the axes, we have fixed σb2 = σa2 = 12 and σv2 = 1. The
2
white dotted lines are given by σw
L = C, where C = 170 on the left and C = 145 on the right. We see
that both dotted lines
accurately
predict
the largest optimal σw for each depth L. (c) Varying the ratio σa2 /σv2
p
2
2
while fixing σv / 1 + σa /σv , and thus fixing A, the leading constant of log χ(0) /χ(L) . (d) in log-log scale:
2
Heatmap gives the test accuracies of ReLU FRN for varying σw
and L. Curves give level sets for the log
(L)
(0)
(L)
(0)
(0)
(L)
2
ratios log s /s ≈ log p /p ≈ log χ /χ
= L log(1 + σv2 σw
/2). (e) Red heatmap shows the test
accuracies of a grid of α-ReLU FRN with varying α and L as shown, but with all σ• s fixed. The white dashed
α2
curve gives a typical contour line of LR = const, where R = (1−α)(2α−1)
. The yellow-to-blue curves form a

set of level curves for s(l) = p(l) − γ (l) = const, with yellow curves corresponding to higher levels.

6

Experimental Results

Our experiments show a dichotomy of what matters in initialization: for tanh resnets, quality of
an initialization is determined by how much gradient explosion there is (measured by χ(0) /χ(L) );
for (α-)ReLU resnets, it is determined by how expressive the random network is (measured by the
metric expressivity s(L) ). We hypothesize this is because in tanh
√ resnets, the gradient dynamics is
much more explosive than the expressivity dynamics (exp(Θ( l)) vs Θ(l)), whereas for ReLU it’s
somewhat the opposite (χw , χv = Θ(1) vs s = exp(Θ(l))).
Tanh, vary σw . We train a grid of reduced and full tanh resnets on MNIST, varying the variance
2
σw
and the number of layers (for FRN we fix σv = 1). The results are indicated in Fig. 3(a, b).
We see that in either model, deeper resnets favor much smaller σw than shallower ones. The white
dotted lines in Fig. 3(a, b) confirm our√theory: according to Eq. (?), for the same gradient ratio
R = χ(0) /χ(L) , we want log R ≈ σw L. Indeed, the white dotted lines in Fig. 3(a, b) trace out
such a level curve and it remarkably pinpoints the largest σw that gives the optimal test set accuracy
for each depth L. Why isn’t the best initialization given by R = 1 ⇐⇒ σw = 0? We believe that
when L and/or σw is small, gradient dynamics no longer dominates the initialization quality because
it has “less room to explode,” and expressivity issues start to dampen the test time performance.
Tanh, vary σa2 /σv2 . As suggested in the analysis of Eq. (?), the ratio ρ2 = σa2 /σv2 determines the
fixed point e∗ and its convergence rate by itself while also contributes to the rate of gradient explosion
in tanh
p FRN. We seek to isolate its effect on forward dynamics by varying σv with ρ such that
σv / 1 + ρ2 is kept constant, so that the leading term of the log gradient ratio is kept approximately
equal for each L and ρ. Fig. 3(c) shows the test accuracies of a grid of tanh FRN initialized with
such an ensemble of σ• s. What stands out the most is that performance is maximized essentially
9

around a fixed value of L regardless of ρ, which shows that indeed gradient dynamics determines the
initialization quality in tanh resnets. There is also a minor increase in performance with increasing ρ
regardless of L; this is counterintuitive as increasing ρ means “decreasing expressivity.” It is currently
not clear what accounts for this effect.
2
∈ [0, 1.5] while fixing
ReLU, vary σw We train a grid of ReLU FRN on MNIST, varying σw
1
2
2
2
σv = 1, σa = σb = 2 . The resulting test set accuracies are shown in Fig. 3(d). The dark upper
region signifies failure of training caused by numerical issues with exploding activation and gradient
norms: This corresponds to the region where p(L) , which is a measure of the mean magnitude of
an neuronal activation in layer L, becomes too big. We see that the best test accuracies are given
by depths just below where these numerical issues occur. However, if we were to predict that the
optimal init is the one minimizing χ(0) /χ(L) ≥ 1, then we would be wrong — in fact it is exactly
the opposite. In this case, the dynamics of s(l) , p(l) , and χ(0) /χ(l) are approximately the same (all
exp(Θ(l)) with the same hidden constants), and optimal performance corresponds to the highest s(L) ,
p(L) , and χ(0) /χ(L) without running into infs.

α-ReLU, vary α. We similarly trained a grid of α-ReLU FRN on MNIST, varying only α and the
depth, fixing all σ• . Fig. 3(e) shows their test accuracies. We see similar behavior to ReLU, where
when the net is too deep, numerical issues doom the training (black upper right corner), but the best
performance is given by L just below where this problem occurs. In this case, if we were to predict
optimality based on minimizing gradient explosion, we would be again wrong, and furthermore, the
contour plot of χ(0) /χ(L) (white dashed line) now gives no information at all on the test set accuracy.
In contrast, the contours for s(l) succeeds remarkably well at this prediction (yellow/green lines).6
By interpolation, this suggests that indeed in the ReLU case, it is expressivity, not trainability, which
determines performance at test time.
In all of our experiments, we did not find e dynamics to be predictive of neural network performance.

7

Conclusion

In this paper, we have extended the mean field formalism developed by [9, 10, 11] to residual
networks, a class of models closer to practice than classical feedforward neural networks as were
investigated earlier. We proved and verified that in both the forward and backward passes, most
of the residual networks discussed here do not collapse their input space geometry or the gradient
information exponentially. We found our theory incredibly predictive of test time performance despite
saying nothing about the dynamics of training. In addition, we overwhelmingly find, through theory
and experiments, that an optimal initialization scheme must take into account the depth of the residual
network. The reason that Xavier [4] or He [5] scheme are not the best for residual networks is in
fact not that their statistical assumptions are fragile — theirs are similar to our mean field theoretic
assumptions, and they hold up in experiments for large width — but rather that their structural
assumptions on the network break very badly on residual nets.
Open Problems. Our work thus have shown that optimality of initialization schemes can be
very unstable with respect to architecture. We hope this work will form a foundation toward a
mathematically grounded initialization scheme for state-of-the-art architectures like the original He et
al. residual network. To do so, there are still two major components left to study out of the following
three: 1. Residual/skip connection 2. Batchnorm 3. Convolutional layers. Recurrent architectures and
attention mechanisms are also still mostly unexplored in terms of mean field theory. Furthermore,
many theoretical questions still yet to be resolved; the most important with regard to mean field theory
is: why can we make Axioms 3.1 and 3.2 and still be able to make accurate predictions? We hope to
make progress on these problems in the future and encourage readers to take part in this effort.

10

Acknowledgments
Thanks to Jeffrey Ling for early exploration experiments and help with the initial draft. Thanks to
Felix Wong for offering his wisdom and experience working in statistical physics.

References
[1] Nils Bertschinger and Thomas Natschlger. Real-time computation at the edge of chaos in
recurrent neural networks. Neural Computation, 16(7):1413–1436, July 2004. ISSN 0899-7667.
doi: 10.1162/089976604323057443.
[2] Youngmin Cho and Lawrence K. Saul. Kernel methods for deep learning. In Advances in
neural information processing systems, pages 342–350, 2009. URL http://papers.nips.
cc/paper/3628-kernel-methods-for-deep-learning.

[3] Amit Daniely, Roy Frostig, and Yoram Singer.
Toward Deeper Understanding
of Neural Networks: The Power of Initialization and a Dual View on Expressivity.
In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett,
editors, Advances in Neural Information Processing Systems 29, pages 2253–
2261. Curran Associates, Inc., 2016.
URL http://papers.nips.cc/paper/
6427-toward-deeper-understanding-of-neural-networks-the-power-of-initialization-and-a-dualpdf.
[4] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward
neural networks. In PMLR, pages 249–256, March 2010. URL http://proceedings.mlr.
press/v9/glorot10a.html.
Delving deep into
[5] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
rectifiers: Surpassing human-level performance on imagenet classification.
In Proceedings of the IEEE international conference on computer vision, pages 1026–
1034, 2015. URL http://www.cv-foundation.org/openaccess/content_iccv_2015/
html/He_Delving_Deep_into_ICCV_2015_paper.html.
[6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for
Image Recognition. pages 770–778, 2016. URL https://www.cv-foundation.org/
openaccess/content_cvpr_2016/html/He_Deep_Residual_Learning_CVPR_2016_
paper.html.
[7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual
networks. In European Conference on Computer Vision, pages 630–645. Springer, 2016.
[8] Timothy P. Lillicrap, Daniel Cownden, Douglas B. Tweed, and Colin J. Akerman. Random
synaptic feedback weights support error backpropagation for deep learning. Nature Communications, 7:ncomms13276, November 2016. ISSN 2041-1723. doi: 10.1038/ncomms13276.
URL https://www.nature.com/articles/ncomms13276.
[9] Ben Poole, Subhaneil Lahiri, Maithreyi Raghu, Jascha Sohl-Dickstein, and Surya Ganguli.
Exponential expressivity in deep neural networks through transient chaos. In Advances In
Neural Information Processing Systems, pages 3360–3368, 2016.
[10] Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On
the expressive power of deep neural networks. arXiv:1606.05336 [cs, stat], June 2016. URL
http://arxiv.org/abs/1606.05336. arXiv: 1606.05336.
[11] Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep Information Propagation. 2017. URL https://openreview.net/pdf?id=H1W1UN9gg.

11

Notes
1
Under simplified conditions, Daniely et al. [3] showed that there exists a fixed point for any “well-behaved”
activation function in a feedforward net. However, this result does not apply to architectures with residual
connections.
2
Note that in practice, to avoid the diverging gradient ψ̇α (x) → ∞ as x → 0, we can use a tempered version
Ψα (x) of α-ReLU, defined by Ψα (x) = (x + )α − α on x > 0 and 0 otherwise, for some small  > 0. The
conclusions of this paper on ψα should hold similarly for Ψα as well.
3

Daniely et al. [3] called the version of Wφ with fixed ρ = 1 the “dual function” of φ.
∗

A more natural visualization is to graph e(l) − e∗ versus l−δ , but because of floating point precision,
(l)
e − e∗ doesn’t converge to 0, but a small number close to 0, so that the log-log plot wouldn’t look like what is
expected.
4

5

Our derivations actually apply to all α ∈ ( 21 , 1], where at α = 12 , the expected norm of the gradient diverges
within our mean field formalism. However, at α ≤ 34 , the variance of the gradient already diverges (Thm B.19),
so we cannot expect the empirical values to agree with our theoretical predictions. But in fact, empirically our
theoretical predictions seem to form an upper bound on the gradient norms (see Fig. A.1).
6

the contour for p(l) is similar, but its slopes are slightly off from the heatmap contours.

12

Appendices
A

Additional Figures

In figures appearing in the appendix, k means χ (due to legacy reasons).
1022

1017

° (0) = 0: 03

1.2

° (0) = 0: 70

1018

1015

° (0) = 1: 37

1016

°

(0)

1.0

= 2: 03

1013

° (0) = 2: 70

1014

° (0) = 3: 37

1012

α=1

1019

1.4
p (0) = 3: 40

1020

1011

0.8

ℸ (l)
109

1010

0.6

107

108

e (0) = 0: 01
0.4

106

e (0) = 0: 40

104

∂E=∂bi

∂E=∂wij

105

e (0) = 0: 21

103

e (0) = 0: 60

0.2

e (0) = 0: 79

102

101

e (0) = 0: 99
100

10-1

0.0
10

20

30

40

50

10

20

layer l

3.0

30

40

50

40

30

layer l

1e13

20

10

0

layer l

108

1.4
p (0) = 3: 40

107

° (0) = 0: 03
2.5

1.2

° (0) = 0: 70

106

° (0) = 1: 37
1.0

° (0) = 2: 03

2.0

° (0) = 3: 37

α = .9

105

° (0) = 2: 70

104

0.8

103

1.5
0.6

102
e (0) = 0: 01

1.0
0.4

e (0) = 0: 21
e (0) = 0: 40

0.5

e (0) = 0: 60

0.2

e

(0)

= 0: 79

101
100

ℸ (l)
∂E=∂bi

10-1

∂E=∂wij

e (0) = 0: 99
0.0

10-2

0.0
0

50

100

150

0

50

layer l

1.2

100

150

40

30

layer l

1e8

20

10

0

20

10

0

20

10

0

20

10

0

20

10

0

20

10

0

layer l

105

1.2
p (0) = 3: 40
° (0) = 0: 03

1.0

104

1.0

° (0) = 0: 70
° (0) = 1: 37

103

° (0) = 2: 03

0.8

0.8

° (0) = 2: 70

102

° (0) = 3: 37

α = .8

0.6

0.6

0.4

0.4

101
e (0) = 0: 01
e (0) = 0: 21

100

e (0) = 0: 40
0.2

0.2

e (0) = 0: 60
e (0) = 0: 79

ℸ (l)

10-1

∂E=∂bi

∂E=∂wij

e (0) = 0: 99
0.0

10-2

0.0
0

50

100

150

0

50

layer l

100

150

40

700000

layer l

105

1.2

ℸ (l)

p (0) = 3: 40
° (0) = 0: 03

600000

30

layer l

104

1.0

° (0) = 0: 70

∂E=∂bi

∂E=∂wij

° (0) = 1: 37
500000

° (0) = 2: 03

400000

° (0) = 3: 37

103
0.8

° (0) = 2: 70

α = .7

102
0.6
101

300000
0.4

e (0) = 0: 01

200000

e (0) = 0: 21

100

e (0) = 0: 40
0.2

100000

e (0) = 0: 60

10-1

e (0) = 0: 79
e (0) = 0: 99

0

10-2

0.0
0

50

100

150

0

50

layer l

100

150

40

layer l

105

1.2

ℸ (l)

p (0) = 3: 40
° (0) = 0: 03

35000

104
1.0

° (0) = 0: 70
30000

° (0) = 1: 37

25000

° (0) = 2: 70

30

layer l

40000

∂E=∂bi

∂E=∂wij

103

° (0) = 2: 03

0.8

102

° (0) = 3: 37

α = .6

20000

101

0.6

100

15000
0.4

e (0) = 0: 01
e (0) = 0: 21

10000

e (0) = 0: 40
0.2

e (0) = 0: 60

5000

e (0) = 0: 79
e

0

(0)

50

100

150

10-2

= 0: 99
10-3

0.0
0

10-1

0

50

layer l

100

150

40

30

layer l

16000

layer l

106

1.2
p (0) = 3: 40

105

° (0) = 0: 03

14000

1.0

° (0) = 0: 70
12000

° (0) = 1: 37

10000

° (0) = 2: 70

104

° (0) = 2: 03

103

0.8

° (0) = 3: 37

α = .55

102

8000

0.6
101

6000
0.4

e (0) = 0: 01

100

e (0) = 0: 21

4000

e
0.2

(0)

= 0: 40

10-1
ℸ (l)

e (0) = 0: 60

2000

e (0) = 0: 79

10-2

∂E=∂bi

∂E=∂wij

e (0) = 0: 99
0

10-3

0.0
0

50

100

150

0

50

layer l

100

150

40

9000

layer l

1017

1.2
p (0) = 3: 40

8000

1015

° (0) = 0: 03
1.0

° (0) = 0: 70
7000

°

(0)

ℸ (l)
∂E=∂bi

∂E=∂wij

1011

0.8

° (0) = 2: 70

5000

α = .51

1013

° (0) = 1: 37
° (0) = 2: 03

6000

30

layer l

109

= 3: 37

107

0.6
4000

105
3000

0.4

e (0) = 0: 01
e (0) = 0: 21

2000

e (0) = 0: 40
0.2

103
101

e (0) = 0: 60

1000

e

(0)

= 0: 79

10-1

e (0) = 0: 99
0

10-3

0.0
0

50

100

layer l

150

0

50

100

layer l

150

40

30

layer l

Figure A.1: Empirical vs theoretical dynamics for p(l) , e(l) , and different gradient quantities for α-ReLU, with
format similar to Fig. 1. We refer to each figure on each row from left to right as (a), (b), and (c). Note that in
the α = 1 case, figure (a) (p(l) and γ (l) for different initial values) has log scale y-axis and (a) and (b) have
x-axis ranging from 1 to 50, while for other α, (a) has normal y-axis and (a) and (b) have x-axis ranging from
1 to 200. We do so because the norm of the activation vector in a typical ReLU resnet blows up into NaN at
around layer 90, while this is not a problem for α < 1. Our theoretical predictions track the average of empirical
values closely for forward quantities p(l) , γ (l) , and e(l) for all α, but variance is extremely large for e(l) at
α = 1; it also predicts the average gradient norm accurately for α = 1 to α = .7 (despite the fact that we
should not expect so for α ≤ .75 due to exploding variance (Thm B.19)), although variance is large for α = 1 at
earlier layers (i.e. later layers w.r.t backpropagation). However it consistently and significantly overestimates the
average gradient norm for α = .6 to α = .5, where the variance is so large that one standard deviation below the
2
mean results in negative values. All plots are made with parameters σv2 = 1.5, σa2 = .5, σw
= 1.69, σb2 = .49;
only α is varied. All figures exhibit smooth curves, which are theoretical estimates, and irregular curves with
shades around them, which indicate empirical means and standard deviations (both of which taken in regular
scale, not log scale). For each α, figures (a) and (b) are made with 20 runs of resnets of width 1000. (c) is made
with 25 runs of resnets of width 250.

13

100

1033

e (0) = 0: 01

1029

e (0) = 0: 21

10-1

e (0) = 0: 40
e (0) = 0: 60

10-2

e (0) = 0: 79
e (0) = 0: 99

10-3

α = .9

ℸ (0) =ℸ (1)
(0)

(l)

Âb =Âb

1025

Âw(0) =Âw(l)

1021

lR
lR+1

1017

l ¡¹

l R ¡ ®=(1 ¡ ®)

1013

10-4

109
10-5

105
101

10-6

10-3
10-7

10-7

10-8
100

101

102

104

103

10-11
100

101

layer l

103

104

103

104

103

104

103

104

103

104

103

104

1020
e

(0)

= 0: 01

1018

e (0) = 0: 21

10-1
10-2

e (0) = 0: 40

1016

e (0) = 0: 60

1014

e (0) = 0: 79

1012

e (0) = 0: 99

10-3

α = .8

102

layer l

100

1010

l ¡¹

ℸ =ℸ
(0)

(1)

(0)

(l)

Âb =Âb

Âw(0) =Âw(l)
lR
lR+1
l R ¡ ®=(1 ¡ ®)

108

10-4

106
104

10-5

102
10-6

100
10-2

10-7

10-4
10-8
100

101

102

104

103

10-6
100

101

layer l

e (0) = 0: 01
10-1

e (0) = 0: 21

10-2

e (0) = 0: 60

10-3

e (0) = 0: 99

1015

e (0) = 0: 40
e (0) = 0: 79

ℸ (0) =ℸ (1)
(0)

(l)

Âb =Âb

1013

Âw(0) =Âw(l)

1011

lR
lR+1

109

l ¡¹
10-4

α = .7

102

layer l

1017

100

l R ¡ ®=(1 ¡ ®)

107
105

10-5

103

10-6

101
10-7

10-1

10-8
10-9
100

10-3

101

102

104

103

10-5
100

101

layer l

102

layer l

100
e (0) = 0: 01
e (0) = 0: 21

10-1

e (0) = 0: 40
e (0) = 0: 79

α = .6

l

ℸ (0) =ℸ (1)

1015

Âb =Âb

1011

e (0) = 0: 99

10-3

1017

1013

e (0) = 0: 60

10-2

109

¡¹

(0)

(l)

Âw(0) =Âw(l)
lR
lR+1
l R ¡ ®=(1 ¡ ®)

107

10-4

105
10-5

103
101

10-6

10-1
10-7
10-8
100

10-3

101

102

104

103

10-5
100

101

layer l

102

layer l

100
e (0) = 0: 01

1023

e (0) = 0: 21

10-1

e (0) = 0: 60
e (0) = 0: 79
e (0) = 0: 99

(l)

Âb =Âb

Âw(0) =Âw(l)

1017

lR
lR+1

1014

l R ¡ ®=(1 ¡ ®)

l ¡¹

10-3

α = .55

ℸ (0) =ℸ (1)
(0)

1020

e (0) = 0: 40

10-2

1011
108

10-4

105
10-5

102
10-1

10-6
10-4
10-7
100

101

102

104

103

10-7
100

101

layer l

102

layer l

100
e (0) = 0: 01
e (0) = 0: 21

10-1

e (0) = 0: 40

10-2

α = .51

e (0) = 0: 60
e (0) = 0: 79
e (0) = 0: 99
l ¡¹

10-3

1070

ℸ (0) =ℸ (1)

1062

Âb =Âb

1054

Âw(0) =Âw(l)

1046

lR
lR+1

1038

l R ¡ ®=(1 ¡ ®)

(0)

(l)

1030
1022
1014

10-4

106
10-2

10-5

10-10
10-18

10-6

10-26
10-7
100

101

102

103

layer l

104

10-34
100

101

102

layer l

Figure A.2: We verify the exponents of the forward and backward dynamics for α-ReLU FRN. For each row,
the figures are labeled (a) and (b) from left to right. The format is the same as in Fig. C.17. All figures are
in log-log scale. (a) We exhibit our theoretical dynamics of the cosine distance e(l) based on the recurrences
Thm B.8 and Thm B.10 for different initial conditions e(0) . We draw |e(l) − e(l−1) | for each of these dynamics
in colored solid lines. We predict that each dynamic is Θ̌(l−µ ), where µ = (1 − J̇α (e∗ ))/(1 − α), and the
dashed line gives l−µ−1 (Thm B.18), shifted vertically to better compare the slope in log scale (i.e. the exponent
of the polynomial dynamics). (See footnote 4 for why we plot the dynamics this way). We see that the our
asymptotic prediction is very accurate for the sequence of e(l) that starts with e(0) = 0.99, the closest to e∗ for
each α, while other lines only slowly converge to the same exponent (which is the slope in the log-log plot).
This is to be expected based on the proof of Thm B.18. For α = .9, the e(0) = .99 line upticks at around 103
and then turn into NaNs due to numerical instability. (b) Colored lines are •(0) /•(l) for • = χ, χb , χw (we are
not taking logs in addition to plotting in log-log scale like in Fig. C.15). The dashed lines are our asymptotic
predictions for the dynamics with corresponding colors, based on Thm B.21, again shifted appropriately to
easily compare slope visually. We see that for every alpha our asymptotic predictions are highly accurate. For
both (a) and (b), we did not show α = 1 case as ReLU FRN runs into numerical issues quickly (i.e. with even
for 100 layers) because of exponential explosions in p(l) and χ(l) as predicted by Thms B.16 and B.20, so
we cannot expect to empirically verify the precise predicted asymptotics. All plots are made with parameters
2
σv2 = 1.5, σa2 = .5, σw
= 1.69, σb2 = .49; only α is varied.

14

Table A.1: Glossary of Symbols. “Mean normalized” is abbreviated “m.n.”

B

Symbol

Meaning

Ref

σ•
x(l)
h(l)
N
p(l)
q(l)
γ (l)
λ(l)
s(l)
e(l)
e∗
c(l)
χ(l)
(l)
χ•
φ
ψα
V
W
δ∗
A
R
Jα

standard deviation of trainable parameter •
activation vector/input vector
hidden vector
width (same across all layers)
m.n. squared length of activation vector x(l)
m.n. squared length of hidden vector h(l)
m.n. dot product x(l) · x(l)0
m.n. dot product h(l) · h(l)0
m.n. squared distancepkx(l) − x(l)0 k2
cosine distance γ (l) / p(l) p(l)0
limit value of e(l) as lp→ ∞
cosine distance λ(l) / q(l) q(l)0
m.n. gradient squared norm w.r.t. x(l)
m.n. gradient squared norm w.r.t. trainable parameter •
variable nonlinearity R → R
α-ReLU
variance integral transform
covariance integral transform
∗
e(l) converges like Θ(l−δ ) in tanh FRN
leading coeff of log χ(0) /χ(L) in tanh FRN
log χ(0) /χ(L) ∼ R log L for (α < 1)-ReLU
kernel function of α-ReLU

3.3
3.3
3.4
3.4
3.4
3.4
3.4
3.5
3.5
5.2
5.3
5.3
B.11
B.13
B.20
C.30

A Listing of Main Theorems

B.1
B.1.1

Tanh
Reduced Residual Network

Lemma B.1. Suppose φ is antisymmetric. Then in an RRN, p and q satisfy the recurrence
2
q = σw
p + σb2

p = Vφ(q) + p.
Theorem B.2. Suppose φ is tanh-like. Assume RRN architecture.
• If σw = 0, then p(l) = lVφ(σb2 ) + p(0) and q(l) = σb2 .
2
• If σw > 0, liml→∞ p(l) /l = 1 and liml→∞ q(l) /(σw
l) = 1. If φ = tanh, then we can
obtain more terms of the asymptotic expansions:
−1 1/2
−2
p(l) = l − 2Cσw
l − C 2 σw
log l + O(1)
2
q(l) = σw
l − 2Cσw l1/2 − C 2 log l + O(1)
p
as l → ∞, where C = 2/π.

Theorem B.3. Suppose φ is antisymmetric. Then in an RRN, λ and γ satisfy the recurrence
2
λ = σw
γ + σb2

γ = Wφ(q, λ) + γ.
Theorem B.4. Suppose φ is a tanh-like nonlinearity in an RRN. Assume e(0) < 1.
• If σw = 0, then γ (l) = lWφ(σb2 , σb2 ) + γ (0) = lVφ(σb2 ) + γ (0) and λ(l) = σb2 , so that
e(l) → 1 and 1 − e(l) = Θ(l−1 ). As a result, s(l) = p(l) (1 − e(l) ) = Θ(1).
15

2

2

• If σw > 0, then γ (l) = Θ̌(l π ), and e(l) → 0 like Θ̌(l π −1 ). Thus s(l) = Θ(p(l) ) = Θ(l).
Theorem B.5. For any nonlinearity φ in an RRN, under assumptions Axiom 3.1 and Axiom 3.2,
whenever φ̇2 (ζ) has finite variance for Gaussian variable ζ,
2
χ = (σw
Vφ̇(q) + 1)χ,

χb = χVφ̇(q),

χw = χVφ̇(q)p.

Theorem B.6. For φ = tanh in an RRN,
• If σw = 0, χ(m) = χ(l) for all l, m.
• If σw > 0,

√
√
log(χ(m) /χ(l) ) = A( l − m) + B(log l − log m) + O(1)
q
4
2 4
where A = 43 π2 σw and B = 3π
− σw
9π .

Theorem B.7. Suppose φ = tanh. Then in an RRN
(l)

(l)

• If σw = 0, χb = χ(L) Vφ̇(σb2 ) and χw = χ(L) Vφ̇(σb2 )((l − 1)Vφ(σb2 ) + p(0) ), where L
is the last layer.
• If σw > 0,

√
√
(l)
/χb ) = A( l − m) + Bb (log l − log m) + O(1)
√
√
(l)
log(χ(m)
m) + Bw (log l − log m) + O(1)
w /χw ) = A( l −
q
where A = 43 π2 σw (same as A in Thm B.6) and Bb = B + 12 , Bw = B − 12 , with
(m)

log(χb

B=
B.1.2

4
3π

2 4
− σw
9π (same as B in Thm B.6).

Full Residual Network

Theorem B.8. For any nonlinearity φ in an FRN,
2
q = σw
p + σb2

p = σv2 Vφ(q) + σa2 + p
Theorem B.9. Suppose φ is tanh-like. Assume the FRN architecture.
• If σw = 0, then p(l) = (σv2 Vφ(σb2 ) + σa2 )l + p(0) , and q(l) = σb2 .
• If σw > 0, then p(l) = b0 l + b1 l1/2 + b2 log l + O(1), where
b0 = σv2 + σa2
−2Cσ 2 σ −1
b1 = p v w
σv2 + σa2
b2 =
and C =

q

2
π.

−2
−C 2 σv4 σw
(σv2 + σa2 )2

2
2
2
Additionally, q(l) = σw
b0 l + σw
b1 l1/2 + σw
b2 log l + O(1).

Theorem B.10. For any nonlinearity φ, in an FRN
2
λ = σw
γ + σb2

γ = σv2 Wφ(q, λ) + σa2 + γ
Theorem B.11. Assume φ = tanh in an FRN. Suppose e(0) < 1.
• If σw = 0, then λ(l) = σb2 and γ (l) = l(σv2 Wφ(σb2 , σb2 ) + σa2 ) + γ (0) = l(σv2 Vφ(σb2 ) +
σa2 )+γ (0) . Thus e(l) → 1 and 1−e(l) = Θ(l−1 ). As a result, s(l) = p(l) (1−e(l) ) = Θ(1).
16

400

10

5.5

p

p ¡ b0 l

b0 l

350

b1

p

p ¡ b0 l ¡ b1

l + 5: 4

p

l

b2 logl + 5: 4

5

5.0

0

4.5

−5

4.0

−10

3.5

300

250

200

150

100

50

0

−15
0

50

100

150

200

3.0
0

50

100

150

200

0

50

100

150

200

Figure B.3: Empirical verification of Thm B.9.

• If σw > 0, then e(l) converges to the unique fixed point e∗ 6= 1 determined by the equation
e∗ =

1
2
[σ 2 arcsin (e∗ ) + σa2 ].
σv2 + σa2 v π
∗

Furthermore, e(l) converges to e∗ polynomially: |e(l) − e∗ | is Θ̌(l−δ ), where
δ ∗ := 1 −

1
2
1
σv2
2
p
∈ [ − 1, )
π 1 − (e∗ )2 σv2 + σa2
π
2

Since e∗ < 1, s(l) = Θ(p(l) ) = Θ(l).
Theorem B.12. For any nonlinearity φ in an FRN, under assumptions Axiom 3.1 and Axiom 3.2,
whenever φ̇(ζ)2 has finite variance for Gaussian variable ζ,
2
χ = (σv2 σw
Vφ̇(q) + 1)χ,

χb = σv2 χVφ̇(q),

χw = σv2 χVφ̇(q)p,

χv = χVφ(q),

χa = χ

Theorem B.13. Assume φ = tanh in an FRN.
• If σw = 0, χ(m) = χ(l) for all l, m.
• If σw > 0, then for l ≥ m ≥ 0,
√
√
log(χ(m) /χ(l) ) = A( l − m) + B(log l − log m) + O(1)
where
r

2 σv2 σw
p
π σv2 + σa2


σv4
4
3
2
B=
− σw
9π σv2 + σa2 σv2 + σa2

A=

4
3

Fig. B.4 shows empirical verification of the asymptotic expansion of χ for various values of σ• s.
Theorem B.14. Suppose φ = tanh in an FRN.
• If σw = 0, then
(l)

χb = σv2 χ(L) Vφ̇(σb2 )
2 (L)
χ(l)
Vφ̇(σb2 )((σv2 Vφ(σb2 ) + σa2 )(l − 1) + p(0) )
w = σv χ
(L)
χ(l)
Vφ(σb2 )
v =χ
(L)
χ(l)
.
a =χ

• If σw > 0, then for l ≥ m ≥ 0,
17

σv = 0.6, σw = 0.8, σa = 0.7, σb = 0.3

σv = 0.7, σw = 0.9, σa = 0.3, σb = 0.2

σv = 0.8, σw = 0.4, σa = 0.5, σb = 0.6

20

10

8
8

15
ℸ

6

ℸ − √ l

4

6

ℸ

√ l + const

√ l + const

10

ℸ − √ l

logl + const

logl + const
5

ℸ
√ l + const

4

ℸ − √ l

logl + const

2

2
0

0

0
0

200

400

600

800

1000

0

σv = 0.9, σw = 0.8, σa = 0.8, σb = 0.1

200

400

600

800

1000

15.0
ℸ

10.0

√ l + const

2

logl + const

1

ℸ − √ l

7.5
5.0

ℸ − √ l

logl + const

400

600

800

1000

1000

ℸ
√ l + const

8

ℸ − √ l

6

logl + const

4
2

−1
200

800

10

√ l + const

2.5

0

600

12
ℸ

0

0.0

400

14

3

12.5

200

σv = 0.8, σw = 0.7, σa = 0.6, σb = 0.9

4

17.5

0

σv = 0.7, σw = 0.2, σa = 0.4, σb = 0.4

0
0

200

400

600

800

1000

0

200

400

600

800

1000

Figure B.4: Empirical verification of the asymptotic expansion of χ for various values of σ• s. Note that we
have chosen all small values for σ• s. For larger values, the constant term in Thm B.13 begins to dominate
(primarily because of the expansion log(1 + x) = x + Θ(x2 ) has large Θ term when x is large), and χ behaves
more like exp(Θ(l)) up to depth 1000.

√
√
(l)
/χb ) = A( l − m) + Bb (log l − log m) + O(1)
√
√
(l)
log(χ(m)
m) + Bw (log l − log m) + O(1)
w /χw ) = A( l −
√
√
(m)
(l)
log(χa /χa ) = A( l − m) + B(log l − log m) + O(1)
√
√
(l)
m) + B(log l − log m) + O(1)
log(χ(m)
v /χv ) = A( l −
q


σ2 σ
σv4
4
3
2
where A = 43 π2 √ v2 w 2 and B = 9π
−
σ
are as in Thm B.13 and
2
2
2
2
w
σ +σ
σ +σ
(m)

log(χb

σv +σa

Bb = B +
B.2

1
2

v

a

v

a

and Bw = B − 12 .

α-ReLU

Lemma B.15. If α > − 12 , then Vψα (q) = cα q α , where cα =

√1 2α−1 Γ
π

α+

1
2



.

Note that if α ≤ − 12 , then Vψα (q) is not defined (its defining integral does not converge).
B.2.1

Full Residual Network

By Thm B.8 and Lemma B.15, we have the length recurrences
2
q = σw
p + σb2

p = σv2 cα qα + σa2 + p
Theorem B.16. Suppose we have the nonlinearity φ = ψα . The in an FRN: If α = 1, then
2
p(l) = Θ((1 + σv2 σw
/2)l ), with the hidden constant depending on the initial condition. If 0 < α < 1,
1
1
1
2α
cα (1 − α)] 1−α .
then p(l) = Θ(l 1−α ). More precisely, liml→∞ p/l 1−α = [σv2 σw
Fig. B.5 empirically verifies the asymptotics for α = 1 for various σv and σw .
Similarly, by Thm B.10, if q = q0 , then
2
λ = σw
γ + σb2

γ = σv2 qα Wψα (1, c) + σa2 + γ

18

1048
2, 2
2, 1
0.5, 2

1044
1040
1036
1032
1028
1024
1020
1016
1012
108
104
100
0

10

20

30

40

50

layer l

Figure B.5: Verification of the exponential asymptotics of p(l) when α = 1. The lines of each color correspond
to different (σw , σv ) pairs, which are given in the legend. The solid lines are given by the recurrences Thm B.8,
2
and the dashed lines are given by our asymptotics (1 + σv2 σw
/2)l (Thm B.16). Note that the y-axis is in
log-scale.

0.99

0.98

0.97

0.96

0.95

e (0) = 0: 01
e (0) = 0: 21

0.94

e (0) = 0: 40
e (0) = 0: 60
e (0) = 0: 79

0.93

e (0) = 0: 99
e¤

0.92
0

2000

4000

6000

8000

10000

layer l

Figure B.6: Verification of fixed point e∗ in Thm B.18 for α = .6. Different colors correspond to different
initial conditions e(0) , and the dashed line gives the fixed point.

2 −1
B U l]−2 for
Theorem B.17. Suppose φ = ψ1 . Then in an FRN, e(l) → 1 and 1 − e(l) ∼ [ 14 σv2 σw
√
2
2
2
B = 1+σv2 σw
/2 and U = 3π . As a result, s(l) = (1−e(l) )p(l) = Θ(l−2 exp(Θ(l))) = exp(Θ(l)).

Theorem B.18. Suppose φ = ψα for 0 < α < 1 in an FRN. Then e converges to the unique nonunit
fixed point e∗ of Jα , and |e∗ − e(l) | is Θ̌(l−µ ), where µ = (1 − J̇α (e∗ ))/(1 − α). Additionally,
s(l) = Θ(p(l) ) = Θ(l1/(1−α) ).
Fig. B.6 verifies empirically that e∗ is indeed the fixed point of e(l) . Fig. A.2 verifies empirically the
convergence rate l−µ . Fig. B.7 plots J̇α (e∗ ) and µ versus α. It certainly looks like µ = 21 (1 − α),
but we have no proof for it. Based on this conjecture, we see there is a “discontinuity” of µ at α = 1:
µ → 0 as α → 1, but for α = 1, the actual convergence dynamics has exponent −2 by Thm B.17.
Because of the following theorem, we cannot expect the equations of Thm B.12 to hold for α ≤ 34 .
Theorem B.19. Suppose we have the nonlinearity ψα in an FRN. Var(ψ̇α (ζ)2 ) diverges for any
Gaussian variable ζ with mean 0 if α ≤ 34 but is finite if α > 43 .
Theorem B.20. Suppose we have the nonlinearity ψα in an FRN. If α = 1, then χ(l−m) =
m
α2
2
χ(l) 12 σv2 σw
+ 1 . If α ∈ ( 43 , 1), then χ(l−m) = Θ(1)χ(l) (l/(l − m))R for R = (1−α)(2α−1)
,
where the constants in Θ(1) do not depend on l or m.
2

α
This exponent (1−α)(2α−1)
is minimized at α = 34 on α ∈ (3/4, 1), where the value is
α = 23 on α ∈ (1/2, 1), where the value achieved is 4) (Fig. B.8(a)).

As a corollary,
19

9
2

(and at

dα
dc

μ

(* )

0.25

1.000

0.20

0.995
0.990

0.15

0.985

0.10
0.980

0.05

0.975

0.5

0.6

0.7

0.8

0.9

1.0

α

0.6

0.7

0.8

0.9

α

Figure B.7: (a) A plot of J̇α (e∗ ) versus α. (b) A plot of the exponent µ of the dynamics of |e∗ − e(l) | (see
Thm B.18)
α
1-α
30

-1

25

-2

-R

-3

20

-4
15
-5
10
-6

0.6

0.7

0.8

0.9

1.0

α

0.6

0.7

0.8

0.9

1.0

α

Figure B.8: (a) The exponent of the polynomial gradient dynamics with respect to α-ReLU versus α. (b) The
exponent of the dynamics of χv and χw .

Theorem B.21. If φ = ψ1 in an FRN, then for l ≥ m ≥ 0,
(l−m)

χb

= Θ(1)χ(l) B m ,

χ(l−m)
= Θ(1)χ(l) B l ,
w

χ(l−m)
= Θ(1)χ(l) B l ,
v

χ(l−m)
= Θ(1)χ(l) B m .
a

2
/2.
where B = 1 + σv2 σw

If φ = ψα in an FRN, for α < 1, then for l ≥ m ≥ 0,
(l−m)

χb

α

χ(l−m)
= Θ(1)χ(l) lR (l − m) 1−α −R ,
w

= Θ(1)χ(l) lR (l − m)−R−1 ,
α

χ(l−m)
= Θ(1)χ(l) lR (l − m) 1−α −R ,
v

χ(l−m)
= Θ(1)χ(l) (l/(l − m))R .
a

Fig. A.2 verifies the backward asymptotic dynamics empirically for different α < 1. Fig. B.8(b)
α
graphs the exponent 1−α
− R in terms of α. We see that on [0.5, 1], the maximum of this exponent
is at α = 1.

C

Proofs

A brief note about notation: We use ∼ to denote both how a random variable is sampled (ex:
x ∼ N (0, 1) for a Gaussian x) and how a function behaves asymptotically, i.e. f (x) ∼ g(x) as
x → a iff limx→a f (x)/g(x) = 1. Context should be enough to differentiate between these two
cases. We in addition use ' to denote asymptotic expansion. For example, if {αi }i≥0 is a sequence
of strictly decreasing reals and {βi }i≥0 is a sequence of nonzero reals, then
X
f (x) '
βi (x − ξ)αi
i≥0

means that as x → ξ, f (x) −

PN

i=0

βi (x − ξ)αi = Θ((x − ξ)αN +1 ).
20

C.1

Preliminary Lemmas

Lemma C.1. We have

2
σw
γ + σb2
= e(1 + O(γ −1 )).
2 p + σ2
σw
b

regardless of whether e(l) = γ (l) /p(l) converges.
But suppose e(l) = γ (l) /p(l) → e∗ . If e∗ < 1, then
2
σw
γ + σb2
= e(1 + Θ(γ −1 )).
2 p + σ2
σw
b

If e∗ = 1, then
2
σw
γ + σb2
= e(1 + Θ(p−1 )),
2
σw p + σb2

where  = 1 − e.
2
Proof. Write M = σb2 /σw
.
2
γ + σb2
σw
1 + M γ −1
=
e(1
+
)
2 p + σ2
σw
1 + M p−1
b

= e(1 + M (γ −1 − p−1 ) + O(p−1 (γ −1 − p−1 ))).
In any situation, γ −1 − p−1 = O(γ −1 ) because γ ≤ p, so this gives the first statement. If e∗ exists
and e∗ < 1, then γ −1 − p−1 = Θ(γ −1 ), which yields the second statement. If e∗ exists and e∗ = 1,
then γ −1 − p−1 = p−1 ((1 − )−1 − 1) = p−1 ( + O(2 )) = Θ(p−1 ).
For any function f that is (k + 1)-times differentiable in a neighborhood of 0, we have the asymptotic
expansion
k
X
dn f
zn
f (z) =
(0)
+ O(z k+1 ), as z → 0.
n
dz
n!
n=0
Since
dn
q 1/2 Vφ(q)
d(1/q)n

=
q→∞

(−1)n
√
2n 2π

Z

∞

φ2 (z)z 2n dz

−∞

whenever the RHS is integrable, we have
Lemma C.2. Suppose φ2 (z)z 2n is integrable over z ∈ R for all 0 ≤ n ≤ N + 1. Then Vφ(q) =
PN
q −1/2 ( n=0 Cn q −n + O(q −N −1 )) as q → ∞, where
Z ∞
(−1)n
√
Cn :=
φ2 (z)z 2n dz.
2n n! 2π −∞
Note that sechd (z) = Θ(e−d|z| ) for z → ∞ as long as d > 0, so that Cn from the above result
converges when φ = sechd . Therefore
P
Lemma C.3. Let d > 0. We have V sechd (q) ' q −1/2 n≥0 Cn q −n , where
Z ∞
(−1)n
√
Cn :=
sech2d (z)z 2n dz.
2n n! 2π −∞
As corollaries, we obtain the following asymptotics.
q
˙
Lemma C.4. Vtanh(q)
= 32 π2 q −1/2 + Θ(q −3/2 ) as q → ∞.
˙
= sech2 (z) and
Proof. Use Lemma C.3 along with the fact that tanh(z)
2
1
2 sech z tanh z.
21

R

sech4 z dz =

2
3

tanh z +

Lemma C.5. 1 − V tanh(q) =

q

2 −1/2
πq

+ Θ(q −3/2 ) as q → ∞.

Proof. Use Lemma C.3 along with the fact that 1 − tanh2 (z) = sech2 (z) and
tanh z.

R

sech2 z dz =

Lemma C.6. sech2 (t) ≥ exp(−t2 ) for all t, with equality iff t = 0.
Proof. The lower bound is equivalent to
2

2 ≥ et−t
2

The RHS has derivative (1 − t)et−t

/2

/2

+ e−t−t

− (1 + t)e−t−t

2

/2

2

/2

. This is 0 iff

1−t
= e−2t
1+t
which has a solution 0 and in general can only have solution t ∈ (−1, 1) (by considering the sign of
the LHS). Since each side is analytic in t ∈ (−1, 1), we expand
1−t
= log e−2t
1+t
log(1 − t) − log(1 + t) = −2t
log

(−t − t2 − · · · ) − (t − t2 + · · · ) = −2t
−2t − 2t3 − · · · = −2t
which shows that the only solution is t = 0. A simple plot shows that t = 0 is a maximum, where the
bound in question achieves equality.

Lemma C.7. Suppose φ = tanh. Then Vφ̇(q) ≥

√ 1
.
4q+1

As a sanity check, Lemma C.4 shows that Vφ̇(q) ∼ C0 q 1/2 where C0 ≈ .5319, which is above the
.5 in this lemma.
Proof. By Lemma C.6,
√
dµ(z)φ̇2 ( qz)
Z
1
dz exp(−z 2 /2 − 2qz 2 )
≥√
2π
Z
1
=√
dz exp(−(4q + 1)z 2 /2)
2π
1
=√
.
4q + 1
Z

Vφ̇(q) =

Fig. C.9 demonstrates Lemma C.7.
Lemma C.8. Let d ∈ R and 1 < M < N with N − M ∈ Z≥0 . Set Σ(M, N, d) :=
we fix M and let N → ∞,

Θ(1)
if d < −1





log
N
+
O(1)
if d = −1
 d+1
N
Σ(M, N, d) =
if −1 < d < 0
d+1 + O(1)



N
−
M
+
1
if d = 0


 1
1 d
d+1
max(0,d−1)
+ 2 N + O(N
) if d > 0
d+1 N
22

PN

a=M

ad . If

Vsech 2 (0; q) ¸ (4q + 1) ¡1=2

1.0

Vsech 2 (0; q)
(4q + 1) ¡1=2
0.8

f(q)

0.6

0.4

0.2

0.0
0

10

20

30

40

50

q

Figure C.9: Illustration of Lemma C.7: Vφ̇(q) vs
purposes,

√ 1
4q+1

√ 1
4q+1

for φ = tanh. This bound is very tight, and for most

can be taken as a good approximation of Vφ̇(q).

R N +1
RN
Proof. Consider the integrals A = M ad da and B = M −1 ad da. They evaluate to A =
1
1
d+1
− M d+1 ) and B = d+1
(N d+1 − (M − 1)d+1 ) when d 6= −1 and to A =
d+1 ((N + 1)
log(N + 1) − log M and B = log N − log(M − 1) when d = −1. When d ≤ 0, we have A ≤ B
and Σ(M, N, d) ∈ [A, B]; when d > 0, B ≤ A and Σ(M, N, d) ∈ [B, A]. Thus, as N → ∞ with
M fixed, when d < −1, Σ(M, N, d) = Θ(1); when d = −1, Σ(M, N, −1) = log N + O(1); and
d+1
when d > −1, we have Σ(M, N, d) = Nd+1 + O(N d ).
Now for a > 0 and d > −1 and d 6= 0, 1,
Z a+1
z d − ad dz =

1
((a + 1)d+1 − ad )
d
+
1
a
d
= (ad + ad−1 + · · · ) − ad
2
d d−1
+ Θ(ad−2 ).
= a
2
where the hidden constants in Θ depend only on d (and in fact this term vanishes if d = 1). Thus
Z N +1
N
X
d
d
Σ(M, N, d) =
z dz −
[ ad−1 + Θ(ad−2 )]
2
M
a=M

1
d
=
((N + 1)d+1 − M d+1 ) − Σ(M, N, d − 1) + Θ(Σ(M, N, d − 2))
d+1
2
If −1 < d < 0, then Σ(M, N, d − 1) = Θ(1), so that Σ(M, N, d) =
N d+1
d+1

+ O(1). If d > 0 and d 6= 1, then Σ(M, N, d − 1) =

Nd
d ,

(N +1)d+1
d+1

+ O(1) =

so that

1
1
N d+1 + N d + Θ(N max(0,d−1) ) − N d + Θ(Σ(M, N, d − 2))
d+1
2
1
1
=
N d+1 + N d + O(N max(0,d−1) ).
d+1
2

Σ(M, N, d) =

We can obtain more terms in the expansion for higher d via the Euler-Maclaurin formula, but this
suffices for our purposes.
C.2

Dynamics Zoo

This section deduces the asymptotic behaviors of some sequences governed by recurrence equations.
For the most part, the leading term of their asymptotic expansions is as one would expect from the
corresponding differential equation. However, in some cases we need subleading terms for later
results. They require slightly more nuanced reasoning. First we present a technical lemma.
Lemma C.9. Let F : R × N → R be a function such that for a subset U ⊆ R, and for all
z, z 0 ∈ U, z ≥ z 0 =⇒ F (z, n) ≥ F (z 0 , n) for every n. Suppose sequences a(l) , b(l) , c(l) satisfy
23

• a(l+1) = F (a(l) , l) for all l;
• b(l+1) ≤ F (b(l) , l) for all l above a constant Kb .
• c(l+1) ≥ F (c(l) , l) for all l above a constant Kc .
and furthermore, a(l) , b(l) , c(l) all fall into U for l above a constant KU .
If for some m ≥ max(Kb , KU ), b(m) ≤ a(m) , then b(l) ≤ a(l) , ∀l ≥ m. Similarly, if for some
n ≥ max(Kc , KU ), c(n) ≥ a(n) , then c(l) ≥ a(l) , ∀l ≥ n.
Proof. For the first claim: b(m) ≤ a(m) =⇒ b(m+1) ≤ F (b(m) , m) ≤ F (a(m) , m) = a(m+1) .
Here the last inequality used the monotonicity of F . Induction gives the desired result.
It’s similar for the second claim, where the inductive step is c(m) ≥ a(m)
F (c(m) , m) ≥ F (a(m) , m) = a(m+1) .

=⇒

c(m+1) ≥

Lemma C.10. Suppose (l) satisfies the recurrence
(l) = (l−1) (1 +

δ
).
lβ

for some nonzero constant δ ∈ R independent of l.
• If β > 1, then (l) = Θ(1).
• If β = 1, then (l) = Θ(lδ ).
δ 1−β
• If 0 < β < 1, then (l) = exp( 1−β
l
+ Θ̃(lψ1 (1−2β) )), where ψ1 (x) = max(0, x) is the
ReLU function.

Proof. We have
log (l) = log (l−1) + log(1 + δ/lβ )
= log (l−1) + δ/lβ + Θ(δ 2 /l2β )
for large l. If β > 1, then

P

l

l−β converges, and
log (l) = log (0) − Θ(1)
(l) = Θ(1).

If β = 1, then
log (l) = log (0) + δ log l + Θ(1)
(l) = Θ(lδ ).
If β < 1, then
log (l) = log (0) +
(l) = exp(

δ 1−β
l
+ Θ̃(l1−2β )
1−β

δ 1−β
l
+ Θ̃(lψ1 (1−2β) )).
1−β

Lemma C.11. Suppose (l) = Cl−α + (l−1) (1 + δ/lβ ) for α ∈ R, C 6= 0, and δ 6= 0. Then
• If β > 1, then
– (l) = Θ(l1−α ) if α ∈ (0, 1);
– (l) = Θ(log l) if α = 1;
– (l) = Θ(1) if α > 1.
24

• If β = 1, then
– (l) = Θ(lmax(δ,1−α) ) if 1 − δ 6= α.
– (l) = Θ(lδ log l) if 1 − δ = α.
Furthermore, for β = −δ = 1, (l) ∼ l−1 if α > 2, (l) ∼ l1−α if α < 2, and (l) ∼ lδ log l if α = 2.
Proof. We can unwind the recurrence to get


(l)

=

l
X

−α

m

m=1

l
Y
δ
δ
(0)
(1 + β )
(1 + β ) + 
n
n
n=1
n=m+1
l
Y

Suppose β > 1. By Lemma C.10, we get
l
X

(l) = Θ(1)

m−α + (0) Θ(1)

m=1

 1−α
) if α ∈ (0, 1)
Θ(l
= Θ(log l) if α = 1

Θ(1)
if α > 1.
Now suppose β = 1. By Lemma C.10, we get
(l) =

l
X

m−α Θ(m−δ lδ ) + (0) Θ(lδ )

m=1

where the constants hidden inside the Θ are the same in every term of the sum. If α > 1 − δ, then
Pl
m−δ−α = o(m−1 ), so that m=1 m−δ−α = Θ(1), and
(l) = Θ(lδ ) + (0) Θ(lδ )
= Θ(lδ ).
On the other hand, if α < 1 − δ, then

Pl

m=1

m−δ−α = Θ(l1−δ−α ). So

(l) = Θ(l1−α ) + (0) Θ(lδ )
= Θ(l1−α ).
If α = 1 − δ, then

Pl

m=1

m−δ−α = Θ(log l). So
(l) = Θ(lδ log l) + (0) Θ(lδ )
= Θ(lδ log l).

Finally, if β ∈ (0, 1), then
δ

(l) = e 1−β l

1−β

+Θ(l1−2β )

l
X

−δ

1−β

m−α e 1−β m

+Θ(m1−2β )

δ

+ e 1−β l

1−β

+Θ(l1−2β )

m=1

The case of δ = −1 telescopes, so that the upper and lower constants hidden in Θ can both be taken
to be 1.
Lemma C.12. Suppose for some β > 0, a sequence (l) satisfies
(l) = (l−1) (1 − µ((l−1) )β /l),
Then (l) ∼ (βµ log l)−1/β .
25

(0) ∈ (0,

1
).
µ

Proof. Consider the differential equation
ẋµ = −µxβ+1
/t
µ
for constant µ has solution xµ = [β(µ log t + C)]−1/β for some constant C determined by initial
condition. Note that
−µxµ (t)β+1 /t ≤ xµ (t + 1) − xµ (t) ≤ −µxµ (t + 1)β+1 /(t + 1) = −(1 − o(t−1 ))µxµ (t)β+1 /t.
For any small enough α > 0, we apply Lemma C.9 with F (, l) =  − µβ+1 /l (which is monotonic
in  for small enough ), c(l) = xµ (l), and b(l) = xµ−α (l) to obtain
xµ−α (l) ≤ (l) ≤ xµ (l)
for large enough l and appropriately chosen initial conditions. This shows that (l) = Θ(log l−1/β )
Taking α → 0, we also obtain the leading coefficient (l) ∼ [βµ log l]−1/β .
Lemma C.13. Suppose a sequence u(l) is governed by the equation
u(l) − u(l−1) = A(u(l−1) + B)α ,
1

α

α

where α ∈ [0, 1) and A > 0. Then u(l) = K1 l 1−α − K2 l 1−α log l + o(l 1−α log l), where K1 =
1
1
α
[A(1 − α)] 1−α and K2 = 12 A 1−α (1 − α) 1−α −1 α.
Proof. Leading term. The differential equation
ẋA,B = A(xA,B + B)α
1

has solution xA,B (l) = [A(1 − α)(l + S)] 1−α − B for some constant S. Since ẋA,B is monotonic,
we have (writing x = xA,B for brevity)
A(xA,B (l) + B)α = ẋA,B (l) ≤ xA,B (l + 1) − xA,B (l) ≤ ẋA,B (l + 1) ≤ (A + o(1))(xA,B (l) + B)α
for large enough l. We apply Lemma C.9 with F (x, l) = x + A(x + B)α (which is monotonic in x
for large x), c(l) = xA,B (l), and b(l) = xA−,B (l) to obtain
xA−,B (l) ≤ u(l) ≤ xA,B (l)
1

for large enough l and appropriate initial conditions. Therefore lim u(l) /l 1−α ∈ [[(A − )(1 −
1
1
α)] 1−α , [A(1 − α)] 1−α ]. Taking  → 0 gives the leading term.
1

1

Subleading term. Now let v (l) := u(l) − ℵl 1−a , where ℵ = [A(1 − α)] 1−α . Then we have the
recurrence
1

1

1

v (l+1) + ℵ(l + 1) 1−α − v (l) − ℵl 1−α = A(v (l) + ℵl 1−α + B)α
α
α
α
1 1−α
1
1
α
v (l+1) − v (l) + ℵ(
l
+ (
)(
)l 1−α −1 + Θ(l 1−α −2 ))
1−α
2 1−α 1−α
α

v (l+1) − v (l)

1

= A[ℵα l 1−α + α(v (l) + B)ℵα−1 l−1 + Θ((v (l) + B)l−1− 1−α )]
α
α
1
1
α
=
v (l) l−1 − ℵ(
)(
)l 1−α −1 + g(l)
1−α
2 1−α 1−α

α

for some g(l) = O(l 1−α −2 + l−1 ) and where, to get the last equation, we have used Aαα =
α
α
to cancel the l 1−α term and simplified αAℵα−1 = 1−α
.

1
1−α ℵ

α

α
For any J > 0, the differential equation v̇J (l) = 1−α
vJ (l)l−1 − Jl 1−α −1 has solution vJ (l) =
α
α
α
α
C[l(1 − α)] 1−α − Jl 1−α log l. Note that the functions FJ (z, n) = z + 1−α
zn−1 − Jn 1−α −1 and
GJ (z, n) = FJ (z, n) + g(n) is monotonic in z (for positive n). For large l, we also have v̇J (l) and
FJ (vJ (l), l) = vJ (l) + v̇J (l) decreasing in l. Thus for any  > 0 and l large enough

GJ+ (vJ (l), l) ≤ FJ+/2 (vJ (l), l) ≤ vJ (l)+v̇J (l+1) ≤ vJ (l+1) ≤ FJ (vJ (l), l) ≤ GJ− (vJ (l), l).
26

Now apply Lemma C.9 with F = GK , a(l) = v (l) , c(l) = vK− , b(l) = vK+ where K :=
1
α
1
α
1 1−α
1
(1 − α) 1−α −1 α, with appropriately chosen initial conditions. This yields
2 ℵ( 1−α )( 1−α ) = 2 A
α
liml→∞ v (l) /(l 1−α log l) ∈ [−K − , −K + ] for every  > 0, and there it must be equal to K. We
have thus obtained the asymptotic expansion
1
α
α
α
1 1
u(l) = [A(1 − α)l] 1−α − A 1−α (1 − α) 1−α −1 αl 1−α log l + o(l 1−α log l).
2

Lemma C.14. Suppose a sequence u(l) is governed by the equation
u(l) − u(l−1) = −A(u(l−1) + B)α ,
1

where α > 1 and A > 0. Then u(l) ∼ [A(α − 1)l] 1−α .
Proof. Similar to Lemma C.13.
Lemma C.15. Suppose a sequence u(l) is governed by the equation
u(l) − u(l−1) = A(u(l−1) + B)α + C,
1

where α ∈ (0, 1). Then u(l) = K1 l 1−α + R(l), where the remainder R(l) is

α
1

−K2 l 1−α log l if α > 2
1
R(l) ∼ (C − K2 )l log l if α = 2 and K2 6= C

 C(1−α) l
if α < 21
1−2α
1

1

α

where K1 = [A(1 − α)] 1−α , K2 = 12 A 1−α (1 − α) 1−α −1 α as in Lemma C.13.
Proof. u is bounded below by the dynamics v (l) − v (l−1) = A(v (l−1) + B)α and bounded above
by the dynamics w(l) − w(l−1) = (A + o(1))(w(l−1) + B)α . By Lemma C.13, both v and w are
1
asymptotic to u(l) ∼ [A(1 − α)l] 1−α , which gives the result.
1

Now define v (l) = u(l) − [A(1 − α)l] 1−α , and similar to the proof of Lemma C.13, we find
v (l+1) − v (l) =
1

α
α
v (l) l−1 − Kl 1−α −1 + C + g(l)
1−α

α

α

α
> 1 ⇐⇒ α > 12 , then
where K = 21 A 1−α (1 − α) 1−α −1 α and g(l) = O(l 1−α −2 + l−1 ). If 1−α
α
α
−1
C + g(l) = o(l 1−α ) and we can proceed as in the proof of Lemma C.13 to find v (l) ∼ Kl 1−α log l.
α
−1
α
α
If 1−α = 1 ⇐⇒ α = 1 and K 6= C, then v (l+1) − v (l) = 1−α v (l) l−1 − (K − C)l 1−α + g(l), so
α
that the technique used in Lemma C.13 would obtain v (l) ∼ (K − C)l 1−α log l = (K − C)l log l. If
α
1
α
(l+1)
− v (l) = 1−α
v (l) l−1 + C + o(1), then by using the differential
1−α < 1 ⇐⇒ α < 2 , then v
α
equation v̇J (l) = 1−α vJ (l)l−1 + J to approximate the difference equation solution and applying

Lemma C.9 as in the proof of Lemma C.13, we obtain v (l) (l) ∼
C.3

C(1−α)
1−2α l.

Forward Dynamical Equations

Here we derive the recurrences governing the forward length and correlation quantities p, q, λ, γ.
We start with reduced residual networks.
Lemma B.1. Suppose φ is antisymmetric. Then in an RRN, p and q satisfy the recurrence
2
q = σw
p + σb2

p = Vφ(q) + p.
27

Proof. We have
X
q = hh2j i = h (wji xi + bj )2 i
i

=

hb2j i

X
X
X
2 2
hwji xi bj i + 2
hwji wli x2i i
+
hwji
xi i + 2
i

i

j6=l

But wji , wli , x, and bj form an independency, so the last two sums are 0, and the terms in the first
sum split multiplicatively. Therefore
X
2
q = σb2 +
hwji
ihx2i i
i
2
σw
p
N
2
2
= σb + σw p.

=

σb2

+N ·

For the recurrence of p, we have
p = hx2i i = h(φ(hi ) + xi )2 i
= hφ(hi )2 i + hx2i i + 2hφ(hi )xi i
As N → ∞, the coefficient wii of xi in hi has vanishing covariance, so hi and xi become independent.
Therefore hφ(hi )xi i = hφ(hi )ihxi i. Because hP
i is the sum of a large number of independent random
variables, by CLT, hi is a Gaussian with mean i hwji ihxi i + hbj i = 0 since hwji i = hbj i = 0. Our
antisymmetry assumption on φ then implies hφ(hi )i = 0. Therefore,
p = hφ(hi )2 i + hx2i i
= Vφ(q) + p
as desired.
Theorem B.3. Suppose φ is antisymmetric. Then in an RRN, λ and γ satisfy the recurrence
2
λ = σw
γ + σb2

γ = Wφ(q, λ) + γ.
Proof. Similar to Lemma B.1.
Now, for the full residual networks, the proofs are similar, but we no longer need to assume that φ is
antisymmetric because of the randomization via the extra sets of weights.
Theorem B.8. For any nonlinearity φ in an FRN,
2
q = σw
p + σb2

p = σv2 Vφ(q) + σa2 + p
Proof.
q = hh2j i = h(wji xi + bj )2 i = h(wji xi )2 i + hb2j i
2
= σw
hx2i i + σb2
2
= σw
p + σb2

p = hx2i i = h(vij φ(hj ) + xi + ai )2 i
= σv2 hφ(hi )2 i + hx2i i + σa2
= σv2 Vφ(q) + σa2 + p
where in the third equality for p, we are now using the independence of vij from all other variables to
cancel out the terms, whereas before we had to rely on φ being antisymmetric.
28

Theorem B.10. For any nonlinearity φ, in an FRN
2
λ = σw
γ + σb2

γ = σv2 Wφ(q, λ) + σa2 + γ
Proof. Similar to Thm B.8.
C.4

Backward Dynamical Equations

Here we derive the recurrences governing the gradient quantities χ and χ• for different •, all under
(l)
the gradient independence assumption. Write βi = ∂E(l) for a cost function E.
∂xi

Theorem B.5. For any nonlinearity φ in an RRN, under assumptions Axiom 3.1 and Axiom 3.2,
whenever φ̇2 (ζ) has finite variance for Gaussian variable ζ,
2
χ = (σw
Vφ̇(q) + 1)χ,

χb = χVφ̇(q),

χw = χVφ̇(q)p.

Proof. For a reduced residual network, we have the following derivative computation:
∂xi
∂hi
= δji + φ̇(hi )
,
∂xj
∂xj

∂xi
= δji φ̇(hj ),
∂hj

∂hi
= wij ,
∂xj

∂hi
= xj ,
∂wij

∂hi
= δij .
∂bj

Then
β j = βj +

X

= βj +

X

βi φ̇(hi )

i

∂hi
∂xj

βi φ̇(hi )wij

i

hβ 2j i = h[βj +

X

βi φ̇(hi )wij ]2 i

i

=

hβj2 i

+

X

hβi2 φ̇2 (hi )(wij )2 i

i

+2

X

hβi βk φ̇(hi )wij φ̇(hk )wkj i + 2

X
hβj βi φ̇(hi )wij i
i

i<k

The last two terms of the above vanish as wij is independent from wkj , hi , hk and βi , βj , βk by
Axiom 3.2, and hwij i = 0.
Therefore, applying Axiom 3.1,
2
hβj2 ihφ̇2 (hi )i + hβj2 i
hβ 2j i = σw
2
= (σw
Vφ̇(q) + 1)hβj2 i

We similarly have
X ∂E ∂xi
∂E
=
= βj φ̇(hj ),
∂bj
∂xi ∂hj
i

2
∂E
h
i = hβj2 φ̇(hj )2 i = hβj2 iVφ̇(q),
∂bj
X ∂E ∂xi ∂hj
∂E
=
= βj φ̇(hj )xi ,
∂wji
∂xi ∂hj ∂wji
i

2
∂E
h
i = hβj2 φ̇2 (hj )x2i i = hβj2 iVφ̇(q)p,
∂wji

since

∂xi
= δji φ̇(hj )
∂hj

by Axiom 3.2(b);
since

∂xi
= δji φ̇(hj )
∂hj
by Axiom 3.2(b)

In the last equation we have also used the fact that as N → ∞, hj and xi become independent (they
2
are jointly Gaussian and their correlation hwji
i goes to 0 with N ).
29

Theorem B.12. For any nonlinearity φ in an FRN, under assumptions Axiom 3.1 and Axiom 3.2,
whenever φ̇(ζ)2 has finite variance for Gaussian variable ζ,
2
Vφ̇(q) + 1)χ,
χ = (σv2 σw

χb = σv2 χVφ̇(q),

χw = σv2 χVφ̇(q)p,

χv = χVφ(q),

χa = χ

Proof. For the full residual network, we have the following derivative computations:
X
∂xi
∂hk
∂xi
∂hi
∂hi
= δji +
vik φ̇(hk )
,
= vij φ̇(hj ),
= wij ,
= xj ,
∂xj
∂xj
∂hj
∂xj
∂wij
k

∂xi
= φ(hk ),
∂vik
Again let βj =

∂E
∂xj .

∂hi
= 1,
∂bi

∂xi
= 1.
∂ai

Then
βj =

X

βi (δji +

X

i

k

X

βi (δji +

X

=

i

vik φ̇(hk )

∂hk
)
∂xj

vik φ̇(hk )wkj )

k

Thus,
hβ 2j i = h[

X

βi (δji +

X

i

vik φ̇(hk )wkj )]2 i

k

X
2
2
+
hvik
ihwkj
iVφ̇(q)hβi2 i

=

hβj2 i

=

hβj2 i(1

i,k
2
+ σv2 σw
Vφ̇(q))

where in the second equality we applied the independence argument as in the proof of Thm B.5,
leveraging Axiom 3.2, and in the third equality we used Axiom 3.1 to get hβi2 i = hβj2 i.
The other computations are similar to the proof of Thm B.12.

C.5
C.5.1

Tanh: Reduced Residual Network
Forward Dynamics

Theorem B.2. Suppose φ is tanh-like. Assume RRN architecture.
• If σw = 0, then p(l) = lVφ(σb2 ) + p(0) and q(l) = σb2 .
2
l) = 1. If φ = tanh, then we can
• If σw > 0, liml→∞ p(l) /l = 1 and liml→∞ q(l) /(σw
obtain more terms of the asymptotic expansions:
−1 1/2
−2
p(l) = l − 2Cσw
l − C 2 σw
log l + O(1)
2
q(l) = σw
l − 2Cσw l1/2 − C 2 log l + O(1)
p
as l → ∞, where C = 2/π.

Proof. The case with σw = 0 is trivial. We assume σw > 0 from here on.
p and q are asymptotically linear with l. We first show that, for any ω < 1,
l + p(0) ≥ p(l) ≥ ωl
and

2
2
σw
(l + p(0) ) + σb2 ≥ q(l) ≥ σw
ω(l − 1) + σb2 ,

30

2
so that p(l) ∼ l and q(l) ∼ σw
l.

The upper bounds are trivial, given Vφ(q) ≤ 1 for any q. We show the lower bounds for any ω < 1.
For any  > 0, define ℵ by φ2 (ℵ ) = exp(−). Then
Vφ(q) ≥ exp(−) Pr[z 6∈ [−ℵ , ℵ ] : z ∼ N (0, q)]


2ℵ
≥ exp(−) 1 − √
2πq
where the second inequality follows from an overestimate of the Pr[z ∈ [−ℵ , ℵ ]] via the mode of
N (0, q).
For any q ≥ q(0) , Vφ(q) is then lower bounded by
!
p
r !


p
p
(0)
2
q
2
q(0)
1− p
q(0)
1−
φ2
= φ2
> 0.
(0)
π
2πq
Thus p(l) and q(l) are unbounded with l.

Furthermore, as q → ∞, the lower bound exp(−) 1 −
(l)

fore, for any ω < 1, p

≥ ωl and q

(l)

≥

2
σw
ω(l

− 1) +

√2ℵ
2πq
σb2 .



goes to exp(−), for any . There-

Asymptotic expansion. Now we repeat the following to get each successive asymptotic term of p(l)
(l)
and q(l) : We plug in the current
V tanh(q) = 1 − Cq−1/2 + Θ(q−3/2 )
p asymptotic form of q into(l)
Pl
(Lemma C.5), where C = 2/π. Next we take the sum q = r=1 V tanh(q(r) ), which yields
one more term in the asymptotic expansion of p than the last round. We then repeat until we get only
constant terms.
The following exhibits a trace of this procedure, where in the summation step for q(l) , we implicitly
apply
2
2
q = σw
l + o(l) = σw
l(1 + o(1))
−1 −1/2
−1 −1/2
q−1/2 = σw
l
(1 + o(1)) = σw
l
+ o(l−1/2 )

p=

l
X

1 − C(q(r) )−1/2 + Θ((q(r) )−3/2 )

r=1

=

l
X

−1 −1/2
1 − C(σw
r
+ o(r−1/2 )) + Θ(r−3/2 )

r=1
−1 1/2
= l − 2Cσw
l + o(l1/2 )
2
2
q = σw
l − 2Cσw l1/2 + o(l1/2 ) = σw
l(1 − 2Cσ2−1 l−1/2 + o(l−1/2 ))
−1 −1/2
−1 −1/2
−1 −1/2
−2 −1
q−1/2 = σw
l
(1 + Cσw
l
+ o(l−1/2 )) = σw
l
+ Cσw
l + o(l−1 )

p=

l
X

−1 −1/2
−2 −1
1 − C(σw
l
+ Cσw
l + o(l−1 )) + Θ(l−3/2 )

r=1
−1 1/2
−2
= l − 2Cσw
l − C 2 σw
log l + o(log l)
log l
2
−1 −1/2
−2 log l
q = σw
l(1 − 2Cσw
l
− C 2 σw
+ o(
))
l
l
1
log l
−1 −1/2
−1 −1/2
−2 log l
q−1/2 = σw
l
(1 + Cσw
l
+ C 2 σw
+ o(
))
2
l
l
l
X
1
log r
−3 log r
−1 −1/2
−2 −1
p=
1 − C(σw
r
+ Cσw
r + C 2 σw
+ o( 3/2 ))) + Θ(r−3/2 )
3/2
2
r
r
r=1
−1 1/2
−2
= l − 2Cσw
l − C 2 σw
log l + O(1)

which is what we want.
31

Lemma C.16. Let φ is antisymmetric. Then for τ ∈ [0, π/2],
Z
√
√
q
q
1
Wφ(q, q cos τ ) = lim
dw dw0 Υ(w, w0 ; τ )φ( √ (w + w0 ))φ( √ (w0 − w))
t→τ π sin t w0 ≥|w|
2
2
Z π
Z ∞
2
1
√
dθΣ( qr, θ; τ )
r dre−r /2
=
π 0
0
Z π
Z
1 ∞
−1 −s2 q −1 /2
dθΣ(s, θ; τ )
s dsq e
=
π 0
0
Z π Z ∞
2 −1
1
∂
=
dθ
dse−s q /2 Σ(s, θ; τ )
π 0
∂s
0
w2

1

where Υ(w, w0 ; τ ) := e− 2 ( 1−c +
φ(s sin θ)φ(s sin(θ − τ )).

(w0 )2
1+c

)

(w0 )2
1−c

1

− e− 2 (

2

w
+ 1+c
)

with c = cos τ , and Σ(s, θ; τ ) :=

Of course, in the above lemma, the limit in the first equation is only necessary when τ = 0 or
τ = π/2.
Proof. Let c := cos τ and
Γ := Wφ(q, cq) =

where Σ =

1
√
2πq 1 − c2

Z

dz exp(−zT Σ−1 z/2)φ(z)φ(z 0 ),


cq
.
q

q
cq

Our proof will have two portions: Symmetrization of the Γ integral and trigonometric change of
variables for evaluation.


−1
1
1
Symmetrization. Σ is diagonalized by Ω = √2q
,
1 1
Σ = ΩT Diag(1 − c, 1 + c)Ω.
By a change of variable w = Ωz, so that dw = q −1 dz, we have
Z
√
√
q 0
q
1
T
−1
√
dw exp(−w Diag(1 − c, 1 + c) w/2)φ( √ (w − w))φ( √ (w + w0 ))
Γ=
2
2
2
2π 1 − c
Z
√
√
(w0 )2
q 0
q
w2
1
+ 1+c )
0 − 12 ( 1−c
0
√
=
dw dw e
φ( √ (w − w))φ( √ (w + w ))
2
2
2π 1 − c2
By a change of variable swapping w with w0 , we get

Γ=−

1
2π 1 − c2
√

Z

1

dw dw0 e− 2 (

Thus
2Γ =

1
2π 1 − c2
√

Z

(w0 )2
1−c

2

w
+ 1+c
)

√
√
q
q
φ( √ (w + w0 ))φ( √ (w0 − w))
2
2

√
√
q
q
dw dw0 Υ(w, w0 ; τ )φ( √ (w + w0 ))φ( √ (w0 − w))
2
2

where
1

w2

Υ(w, w0 ; τ ) = e− 2 ( 1−c +

(w0 )2
1+c

)

1

− e− 2 (

(w0 )2
1−c

2

w
+ 1+c
)

.

Note that, by the antisymmetry of φ, the integrand K := Υ(w, w0 ; τ )φ(. . .)φ(. . .) above has the
symmetries K(w, w0 ) = K(w0 , w) = K(w, −w0 ), and is everywhere nonnegative. Fig. C.10
displays a contour plot of K for typical values of q and c. So
Z
1
Γ= √
dw dw0 K(w, w0 ).
π 1 − c2 w0 ≥|w|
32

Figure C.10: The integrand of Γ after symmetrization. Here c = .2 and q = 100 and φ = tanh.

This gives the first equation in the lemma.
Polar Coordinates. Let

√w
1−c

0

= r cos θ, √w
= r sin θ, so that
1+c

√
√
τ
w = r cos θ 1 − c = 2r cos θ sin
2
√
√
τ
0
w = r sin θ 1 + c = 2r sin θ cos
2
p
dw dw0 = 1 − c2 r dr dθ = (sin2 τ )r dr dθ.
Then
Z

w2

e−( 1−c +

A :=

(w0 )2
1+c

)/2

p
p
φ( q/2(w + w0 ))φ( q/2(w0 − w)) dw dw0

w0 ≥|w|

= sin2 τ

∞

Z

r dre−r

2

/2

0

Similarly, let

√w
1+c

π−τ /2

Z

√
√
dθφ( qr sin(θ + τ /2))φ( qr sin(θ − τ /2)).

τ /2
0

= r cos θ, √w
= r sin θ, so that
1−c
√
√
τ
w = r cos θ 1 + c = 2r cos θ cos
2
√
√
τ
0
w = r sin θ 1 − c = 2r sin θ sin
2
p
dw dw0 = 1 − c2 r dr dθ = (sin2 τ )r dr dθ,

and
Z

w2

e−( 1+c +

B=

(w0 )2
1−c

)/2

p
p
φ( q/2(w + w0 ))φ( q/2(w0 − w)) dw dw0

w0 ≥|w|

= − sin2 τ

Z

= − sin2 τ

Z

∞

r dre−r

2

/2

0

0

Z

π/2+τ /2

√
√
dθφ( qr cos(θ + τ /2))φ( qr cos(θ − τ /2))

π/2−τ /2
∞

r dre−r

2

/2

Z

τ /2

√
√
dθφ( qr sin(θ + τ /2))φ( qr sin(θ − τ /2)).

−τ /2

Thus
1
(A − B)
π 1 − c2
Z
Z π−τ /2
2
1 ∞
√
√
=
r dre−r /2
dθφ( qr sin(θ + τ /2))φ( qr sin(θ − τ /2))
π 0
−τ /2
Z ∞
Z π
2
1
√
√
=
r dre−r /2
dθφ( qr sin(θ))φ( qr sin(θ − τ )).
π 0
0

Γ=

√

33

This gives the second equation in the lemma, and a change of variables s =

√

qr gives the third.

For the fourth equality, we start from the third equality, and apply integration by parts:
Z π
Z
2 −1
1 ∞
dθΣ(s, θ; τ )
s dsq −1 e−s q /2
π 0
0
Z ∞
Z
2 −1
1 π
=
dssq −1 e−s q /2 Σ(s, θ; τ )
dθ
π 0
0

Z ∞
Z π 
∞
2 −1
2 −1
∂
1
dse−s q /2 Σ(s, θ; τ )
=
+
dθ −e−s q /2 Σ(s, θ; τ )
π 0
∂s
0
s=0
Z π Z ∞
2
−1
∂
1
dθ
dse−s q /2 Σ(s, θ; τ ).
=
π 0
∂s
0
2 −1

where the last equality follows because Σ(0, θ; τ ) = 0 and e−s

q

/2

→ 0 as s → ∞.

In the following lemmas, the “2” is not important, and can be any arbitrary finite or infinite value.
Lemma C.17. Suppose a function f : (0, 2) → R is C k on (0, 2). If limx↓0 f (i) (x) exists and is
finite for every i ∈ [0, k], then f can be extended to [0, 2) such that one sided ith derivatives exist at
0 for all i ∈ [0, k].

R1
Proof. Consider f (i) (0) := f (i) (1) − 0 f (i+1) (x) dx for i ∈ [0, k − 1], which naturally is also
R
equal to f (i) () − 0 f (i+1) (x) dx for any  > 0. Certainly f (i) (x) → f (i) (0) as x → 0 if this limit
exists — and by assumption it does, for 0 ≤ i ≤ k − 1. Therefore, we can define the extension of
f (i) to x = 0 to be f (i) (0) := f (i) (0). But we need to check that for i ∈ [0, k − 1].
1
lim (f (i) () − f (i) (0)) = f (i+1) (0)


→0

so that all one sided ith derivatives exist. But
Z
1 (i)
1  (i+1)
(f () − f (i) (0)) =
f
(x) dx

 0
Z 1
= f (i+1) (0) +
(f (i+1) (x) − f (i) (0))I(x ∈ [0, ]) dx
0

Since limx↓0 f (i+1) (x) = f (i+1) (0), f (i+1) (x)−f (i+1) (0) is bounded for small x, and by dominated
R1
R1
convergence, 0 (f (i+1) (x) − f (i) (0))I(x ∈ [0, ]) dx → 0 0 dx = 0 as  → 0. Thus
1
lim (f (i) () − f (i) (0)) = f (i+1) (0)
→0 
as desired.
Lemma C.18. If f : [0, 2) → R is C k on (0, 2) and has one sided derivatives at 0 up to order k,
then
f () = f (0) + f (1) (0) + · · · +
for any i ≤ k.
34

i−1 (i−1)
f
(0) + O(i )
(i − 1)!

Proof. We have


Z

f (1) (x) dx
Z 
(1)
= f (0) + f (0) +
f (1) (x) − f (1) (0) dx
0
Z  Z x0
f (2) (x2 ) dx2 dx1
= f (0) + f (1) (0) +
0
0
Z  Z x1
2 (2)
(1)
f (2) (x2 ) − f (2) (0) dx2 dx1
= f (0) + f (0) + f (0) +
2
0
0
..
.
Z 
Z x1
Z xi−1
i−1 (i−1)
f () = f (0) + f (1) (0) + · · · +
f
(0) +
dx1
dx2 · · ·
dxi f (i) (xi )
(i − 1)!
0
0
0

f () = f (0) +

0

for any i ≤ k. It suffices then to bound the size of the integral. Since f (i) (x) → f (i) (0) as x ↓ 0 by
assumption, |f (i) (xi )| is bounded by some constant C on the integration region A := {(x1 , . . . , xi ) :
 ≥ x1 ≥ · · · ≥ xi } for small enough . Therefore,
Z 
Z x1
Z xi−1
dx1
dx2 · · ·
dxi f (i) (xi )
0
0
0
Z
= f (i) (xi )I(~x ∈ A) d~x
≤ C|A|
= Θ(i ).

As a corollary,
Lemma C.19. If f : (0, 2) → R is smooth on (0, 2) and limx→0 f (i) (x) exists and is finite for all i,
then f can be extended to [0, 2) and be one-sided smooth at 0, and
f () = f (0) + f (1) (0) + · · · +

i−1 (i−1)
f
(0) + O(i )
(i − 1)!

for any i.
Lemma C.20. Let φ = tanh. For any fixed c, Wφ(q, cq) is smooth (infinitely differentiable) on
q ∈ (0, ∞). As a function of Q := q −1 , it can be extended smoothly to the point Q = 0, so that
Wφ(q, cq) = 0lim Wφ(q 0 , cq 0 ) + q −1 0lim ∂Wφ(q 0 , cq 0 )/∂(q 0 )−1 + · · ·
q →∞

q →∞

−i+1

+

q
lim ∂ i−1 Wφ(q 0 , cq 0 )/∂(q 0 )−i+1 + O(q −i )
(i − 1)! q0 →∞

for any i ≥ 0. Furthermore, for c bounded away from 1, the constants hidden O can be taken
independent of c.
Proof. Smoothness on (0, ∞). By the third equation of Lemma C.16, for Q ∈ (0, ∞) ⇐⇒ q ∈
(0, ∞),
Z
Z
1 ∞
∂ n  −s2 Q/2  π
s ds
Qe
dθ|φ(s sin θ)φ(s sin(θ − τ ))|
π 0
∂Qn
0
Z ∞
∂ n  −s2 Q/2 
Qe
< ∞,
≤
s ds
∂Qn
0
so by Leibniz’s integral rule and a simple induction, all derivatives of Wφ(q, cq) against Q exists for
any Q ∈ (0, ∞).
35

k

Extension to Q = 0. By Lemma C.19, it suffices to show that the limit of ∂ Wφ(q,cq)
exists and
∂Qk
is finite as Q → 0, for all k. Let τ = arccos c. By the fourth equation of Lemma C.16, we have
explicitly
Z
Z ∞
2
1 π
∂
∂ k Wφ(q, cq)
2
=
dθ
ds(−s /2)k e−s Q/2 Σ(s, θ; τ )
∂Qk
π 0
∂s
0
Z ∞
Z
2
∂
(−2)−k π
ds s2k e−s Q/2 Σ(s, θ; τ )
=
dθ
π
∂s
0
0
for any Q ∈ (0, ∞). Note that for φ = tanh, φ̇ = sech2 ,
∂
Σ(s, θ; τ ) = sin θφ̇(s sin θ)φ(s sin(θ − τ )) + sin(θ − τ )φ(s sin θ)φ̇(s sin(θ − τ )).
∂s
∂ k Wφ
∂Qk

as follows:
Z ∞
2
∂ k Wφ(q, cq)
(−2)−k π
=
dθ
ds s2k e−s Q/2 sin θφ̇(s sin θ)φ(s sin(θ − τ ))
k
∂Q
π
0
0
Z ∞
−k Z π
2
(−2)
+
dθ
ds s2k e−s Q/2 sin(θ − τ )φ(s sin θ)φ̇(s sin(θ − τ ))
π
0
0
We show that for each piece, the limit as Q → 0 exists and is finite, for any k. This will prove the
smooth extendability of Wφ to Q = 0. We will do this for the first piece; the second is similar.
We split the integral of

Z

For Q > 0, the integrand is absolutely integrable, so we may switch the integrals.
Z π Z ∞
2
dθ
ds s2k e−s Q/2 sin θφ̇(s sin θ)φ(s sin(θ − τ ))
0
Z 0∞
Z π
2k −s2 Q/2
=
dss e
dθ sin θφ̇(s sin θ)φ(s sin(θ − τ ))
0

0

We now try to bound the inner integral by an exponentially decreasing term e−sµ for some µ; clearly,
by monotone convergence on the outer integral as Q → 0, this would show the limit of the integral
exists and is finite.
Because φ is odd and φ̇ is even, the inner integrand is negative on θ ∈ [0, τ ) and positive on θ ∈ (τ, π].
We will break up the inner integral as follows, for some fixed  > 0 satisfying τ −  > 0 independent
of s (recall τ ∈ (0, π/2]).
Z π
dθ sin θφ̇(s sin θ)φ(s sin(θ − τ ))
0
Z  Z π 
Z π−
=
+
dθ sin θφ̇(s sin θ)φ(s sin(θ − τ )) +
dθ sin θφ̇(s sin θ)φ(s sin(θ − τ ))
0

π−


2

−z

Now because φ̇(z) = sech (z) ≤ 2e , and sin θ ≥ sin  on θ ∈ [, π − ],
Z π−
dθ sin θφ̇(s sin θ)φ(s sin(θ − τ ))


Z
≤2

π−

dθ exp(−s sin )


= 2(π − 2) exp(−s sin ).
For the other part:
Z

π

dθ sin θφ̇(s sin θ)φ(s sin(θ − τ ))
π−
Z 0

sin(π − θ)φ̇(s sin π − θ)φ(s sin(π − θ − τ )) d(π − θ)

=


Z
=



dθ sin θφ̇(s sin θ)φ(s sin θ + τ )
0

36

so that


Z

Z

π


dθ sin θφ̇(s sin θ)φ(s sin(θ − τ ))

+
0

π−



Z

dθ sin θφ̇(s sin θ)[φ(s sin(τ + θ)) − φ(s sin(τ − θ))]

=
0

But by intermediate value theorem, φ(s sin(τ +θ))−φ(s sin(τ −θ)) = 2θ∂φ(s sin(τ +θ))/∂θ|θ=ψ =
2θφ̇(s sin(τ + ψ))s cos(τ + ψ) for some ψ ∈ [−θ, θ]. By the assumption on , φ(s sin(τ + θ)) −
φ(s sin(τ − θ)) ≤ 2φ̇(s sin(τ − ))s cos(τ − ). Then
Z 
dθ sin θφ̇(s sin θ)[φ(s sin(τ + θ)) − φ(s sin(τ − θ))]
Z 0
≤
dθ sin θφ̇(s sin θ)2φ̇(s sin(τ − ))s cos(τ − )
0

≤ 2φ̇(s sin(τ − ))s cos(τ − )O(1)
Because τ −  > 0 by assumption on , and because φ̇(z) = exp(−Θ+ (z)), this quantity is
exp(−Θ+ (z)), as desired (here Θ+ denotes a positive quantity).
Thus
Z

π

dθ sin θφ̇(s sin θ)φ(s sin(θ − τ ))
Z  Z π 
Z
=
+
dθ sin θφ̇(s sin θ)φ(s sin(θ − τ )) +
0

0

π−

π−

dθ sin θφ̇(s sin θ)φ(s sin(θ − τ ))



= exp(−Θ+ (s))
k

Wφ
and similarly for the other piece of ∂∂Q
k , so that
Z ∞
Z π
2k −s2 Q/2
dss e
dθ sin θφ̇(s sin θ)φ(s sin(θ − τ ))
0
Z 0∞
2Q
=
dss2k e−s 2 −Θ+ (z)
Z0 ∞
→
dss2k e−Θ+ (z)
0

is finite as Q → 0, by monotone convergence.
Independence of constant hidden in O((q 0 )−i ). The constant hidden is a function of the  chosen
above, which depend on τ , but only to the extent that it must satisfy τ −  > 0. As long as we are
interested in a set C of c that is bounded away from 1, the corresponding set of τ is bounded away
from 0, so  can be taken to be some number smaller than all of the corresponding τ .
Lemma C.21. Suppose φ is tanh-like. Then for c ∈ [0, 1],
2
arcsin(c),
π
and weakly increases to this upper bound as q → ∞. Furthermore,
Wφ(q, cq) ≤

• If c = 0 or 1, then equality holds regardless of q.
• If c ∈ (0, 1) is held constant, π2 arcsin(c) − Wφ(q, cq) = Θ(q −1 ), where the hidden
constants in Θ depend on c. But the constants can be made independent of c if c ∈ [, 1 − ]
for some  > 0.
Proof. The cases of c = 0 or 1 are obvious by the definition of W. So from here on we assume
c ∈ (0, 1).
37

0.35

0.30

Wtanh(0, q, q, q cos 1)
2
arcsin(cos
π

1)

0.25

Q=
0.2

0.4

0.6

0.8

1.0

1
q

Figure C.11: We verify empirically that the subleading term in W tanh(q, cq) is linear in q −1 , for constant c.
Indeed, observe that the curve of of W tanh intersects the y-axis at an angle.

Let τ := arccos c. By the first equation of Lemma C.16 and the assumption that φ is tanh-like, it
is immediate that Wφ(q, cq) is nondecreasing in q. By dominated convergence, using the second
equation of Lemma C.16, we get
Z
2
1 ∞
r dre−r /2 (π − 2τ )
lim Wφ(q, cq) =
q→∞
π 0
π − 2τ
=
π
2
= arcsin c.
π
Then the convergence rate is O(q −1 ) by Lemma C.20 and Taylor’s theorem. Thus to show the
convergence rate is Θ(q −1 ), it suffices to show that D := ∂Wφ(q,cq)
< 0. But this is apparent from
∂Q
the first equation of Lemma C.16: For τ ∈ (0, π/2),
Z
1
1
D=
dw dw0 Υ(w, w0 ; τ )(− √ Q−3/2 )
π sin τ w0 ≥|w|
2 2
p
p
0
× [φ̇( q/2(w + w ))φ( q/2(w0 − w))
p
p
+ φ( q/2(w + w0 ))φ̇( q/2(w0 − w))]
<0
since Υ is positive on the integration domain, and φ̇ and φ are both positive for positive arguments,
by the assumption of φ being tanh-like.
Independence of the constants in Θ(q −1 ) from c when c ∈ [, 1 − ]. By Lemma C.20, the upper
constant can be made independent from c. Since D is monotonically decreasing in c (or monotonically
increasing in τ ) and |D| is monotonically increasing in c (or monotonically decreasing in τ ), we have
|D| > |D|
, which can be taken to be the lower constant in Θ(q −1 ).
c=

Fig. C.11 verifies empirically that the subleading term in W tanh(q, cq) is linear in q −1 , for constant
c.
Theorem B.4. Suppose φ is a tanh-like nonlinearity in an RRN. Assume e(0) < 1.
• If σw = 0, then γ (l) = lWφ(σb2 , σb2 ) + γ (0) = lVφ(σb2 ) + γ (0) and λ(l) = σb2 , so that
e(l) → 1 and 1 − e(l) = Θ(l−1 ). As a result, s(l) = p(l) (1 − e(l) ) = Θ(1).
2

2

• If σw > 0, then γ (l) = Θ̌(l π ), and e(l) → 0 like Θ̌(l π −1 ). Thus s(l) = Θ(p(l) ) = Θ(l).
Proof. We have by Lemma C.21,
γ=

2
arcsin(λ/q) − Θ(q−1 ) + γ.
π
38

2
2
Since q = σw
p + σb2 by Thm B.2, and λ = σw
γ + σb2 by Thm B.3,
!
2
σw
γ + σb2
2
γ = arcsin
− Θ(q−1 ) + γ.
2 p + σ2
π
σw
b
(l)
We claim that γ (l) → ∞ as l → ∞. Otherwise, there
 is some C such
 that γ ≤ C for all l. For
C
(l)
large enough l, p ≥ ωl for any ω < 1 and arcsin σ2 p(l−1) +σ2 = Θ(1/l) by linearization of
w

b

arcsin. Thus γ (l) = Θ(log l), but this contradicts our assumption that γ is bounded. This proves our
claim.
Therefore, for large enough l,
2
σw
γ + σb2
= γ/p + Θ(l−1 ).
2 p + σ2
σw
b

Fig. C.12 shows π2 arcsin x vs x. One sees that 1 is an unstable fixed point; if e < 1 − , then
2
π arcsin e < 1 −  − δ for some δ. Thus c drops monotonically until some threshold under which
the linearization of arcsin, arcsin x = x + Θ(x3 ), is applicable. So for large enough l,
2
arcsin(γ/p + Θ(l−1 )) − Θ(l−1 )
π
2
= γ/p + O(l−1 )
π

γ−γ =

2

2

As p(l) ∼ l by Thm B.2, this difference equation has solution γ = Ω(l π − ), O(l π + ) for any  by
using the dynamics of Lemma C.11 to upper and lower bound this difference equation.
C.5.2

Backward Dynamics

Theorem B.6. For φ = tanh in an RRN,
• If σw = 0, χ(m) = χ(l) for all l, m.
• If σw > 0,
√
√
log(χ(m) /χ(l) ) = A( l − m) + B(log l − log m) + O(1)
q
4
2 4
where A = 43 π2 σw and B = 3π
− σw
9π .
Proof. The σw = 0 case is obvious. We will assume σw > 0 from here on.
q
Let p(l) = b0 l + b1 l1/2 + b2 log l + O(1). Then for D = 23 π2 , we have (implicitly applying
Lemma C.4 and Lemma C.8),
−1/2 −1/2

−1
q−1/2 = σw
b0

l

−1 −1/2
−1 −1
(1 − b1 b−1
l
− b2 b−1
l log l + O(l−1 ))
0 2
0 2

Vφ̇(q) = Dq−1/2 + Θ(q−3/2 )
−1/2 −1/2

−1
= Dσw
b0

l

−1 −1/2
−1 −1
(1 − b1 b−1
l
− b2 b−1
l log l + O(l−1 ))
0 2
0 2

−1/2 −1/2

−1
log(BVφ̇(q) + 1) = BDσw
b0

l

−3/2

−1
− (BDσw
b0
l
X

−2 −1 −1 −1
b1 2−1 + B 2 D2 σw
b0 2 )l + Θ(l−3/2 log l)

−1/2 1/2

−1
log(BVφ̇(q(r) ) + 1) = 2BDσw
b0

l

r=1
−3/2

−1
− (BDσw
b0

−2 −1 −1
b1 2−1 + B 2 D2 σw
b0 2 ) log l + O(1)

39

q
−1
−2
2
In our case, we have b0 = 1, b1 = −2Cσw
, b2 = C 2 σw
, B = σw
, C = π2 , which gives
r
l
X
4
4 2
2 4
(r)
σw l1/2 + (
− σw
) log l + O(1).
log(BVφ̇(q ) + 1) =
3
π
3π
9π
r=1
so that
(m)

χ

(l)

/χ

" r
#
√
√
4
4 2
2 4
σw ( l − m) + (
− σw )(log l − log m) + O(1)
= exp
3 π
3π
9π

Theorem B.7. Suppose φ = tanh. Then in an RRN
(l)

(l)

• If σw = 0, χb = χ(L) Vφ̇(σb2 ) and χw = χ(L) Vφ̇(σb2 )((l − 1)Vφ(σb2 ) + p(0) ), where L
is the last layer.
• If σw > 0,

√
√
(l)
/χb ) = A( l − m) + Bb (log l − log m) + O(1)
√
√
(l)
log(χ(m)
m) + Bw (log l − log m) + O(1)
w /χw ) = A( l −
q
where A = 43 π2 σw (same as A in Thm B.6) and Bb = B + 12 , Bw = B − 12 , with
(m)

log(χb

B=

4
3π

2 4
− σw
9π (same as B in Thm B.6).

Proof. The σw = 0 case is obvious. We will assume σw > 0 from here on.
As in the proof of Thm B.6,
−1/2 −1/2

−1
Vφ̇(q) = Dσw
b0

where D =

2
3

q

l

+ Θ(l−1 )

2
π.

Thus by Thm B.5,
r
√
√
4 2
4
(m)
(l)
2 4
log(χ /χ ) =
σw ( l − m) + (
− σw
)(log l − log m) + O(1)
3 π
3π
9π
r
√
√
4 2
4
1
(m)
(l)
2 4
log(χb /χb ) =
σw ( l − m) + (
− − σw
)(log l − log m) + O(1)
3 √π
3π 2
9π
Similarly, since p = l + Θ( l) by Thm B.2, we have
r
√
√
1
4 2
4
2 4
(m)
(l)
σw ( l − m) + (
+ − σw
)(log l − log m) + O(1)
log(χw /χw ) =
3 π
3π 2
9π

C.6
C.6.1

Tanh: Full Residual Network
Forward Dynamics

Theorem B.9. Suppose φ is tanh-like. Assume the FRN architecture.
• If σw = 0, then p(l) = (σv2 Vφ(σb2 ) + σa2 )l + p(0) , and q(l) = σb2 .
• If σw > 0, then p(l) = b0 l + b1 l1/2 + b2 log l + O(1), where
b0 = σv2 + σa2
−2Cσ 2 σ −1
b1 = p v w
σv2 + σa2
b2 =
and C =

q

2
π.

−2
−C 2 σv4 σw
(σv2 + σa2 )2

2
2
2
Additionally, q(l) = σw
b0 l + σw
b1 l1/2 + σw
b2 log l + O(1).

40

Proof. The σw = 0 case is obvious. We will assume σw > 0 from here on.
As in Thm B.2, p will have expansion p = b0 l + b1 l1/2 + b2 log l + O(1). Then, for C =
−1/2 −1/2

−1
q−1/2 = σw
b0
l
X

Vφ(q

r=1

(r)

)=

l
X

l

q

2
π,

−1 −1/2
−1 −1
(1 − b1 b−1
l
− b2 b−1
l log l + O(l−1 ))
0 2
0 2

1 − C(q(r) )−1/2 + Θ((q(r) )−3/2 )

r=1
−1/2 1/2

−1
= l − 2Cσw
b0

p(l) = σv2

l
X

l

−3/2 −1

−1
+ Cσw
b1 b0

2

log l + O(1)

+σa2 l

r=1
−1/2 1/2

−1
= (σv2 + σa2 )l − 2Cσv2 σw
b0

l

−3/2 −1

−1
+ Cσv2 σw
b1 b0

2

log l + O(1)

which yields
b0 = σv2 + σa2
−1/2

−1
b1 = −2Cσv2 σw
b0

b2 =

−2Cσ 2 σ −1
= p v w
σv2 + σa2

−2
−C 2 σv4 σw
(σv2 + σa2 )2

Lemma C.22. Suppose φ is tanh-like. Then
γ ≤ σv2
and
σv2

2
arcsin (λ/q) + σa2 + γ,
π

2
arcsin (λ/q) + σa2 + γ − γ = Θ(q−1 ).
π

Proof. Similar to the proof of Lemma C.21.
Lemma C.23. Let u∗ ∈ [0, 1). Let ft : [0, 1) → [0, 1] be a continuous function for each t ∈ N, to
each of which we associate two numbers 0 ≤ at ≤ u∗ ≤ bt ≤ 1. Suppose for each t, ft (u) > u for
all u ∈ [0, at ) and ft (u) < u for all u ∈ (bt , 1). Assume that for each u, ft (u) − u → 0 as t → ∞
uniformly over u. If at % u∗ and bt & u∗ , then for any u0 ∈ [0, 1), the dynamics ut = ft (ut−1 ) has
a limiting point. Furthermore, either ut → u∗ or ut eventually converges monotonically (decreasing
or increasing) to a limit point.
Proof. Fix a u0 ∈ [0, 1). If ut → u∗ then we are done. Otherwise, suppose there is a neighborhood
[u∗ − , u∗ + ] such that for an infinite sequence t1 , t2 , . . ., uti 6∈ [u∗ − , u∗ + ]. WLOG assume
uti < u∗ −  for all i and (ti )i is the sequence of all ts that satisfy this inequality.
If (ti )i contains {s : s ≥ N } for some N , then for some M > N , for every t > M , at > u∗ − > ut .
By assumption, ut is monotonic for all t > M but is bounded above. Thus ut has a fixed point
û ≤ u∗ −  as desired.
Now assume there are infinite is such that ti − 1 6= ti−1 (i.e. ti − 1 is not part of the sequence
(ti )i ). We will show that this case is contradictory. Take T large enough such that at > u∗ − /2
and |ft (u) − u| < /4 for all u and for all t ≥ T (T exists by premise). Let j be the smallest index
such that tj > T and tj − 1 6= tj−1 . By the definition of j, utj −1 ≥ u∗ − . If utj −1 ≥ u∗ − /2,
then by definition of T , u∗ −  > utj = ftj (utj −1 ) > utj −1 − /4 > u∗ − 3/4 > u∗ − , a
contradiction. If u∗ −  ≤ utj −1 ≤ u∗ − /2, then by the definition of T , utj −1 ≤ atj −1 so that
utj = ftj (utj −1 ) > utj −1 ≥ u∗ − , a contradiction.
The “furthermore” claim is clear from our proof above.
Theorem B.11. Assume φ = tanh in an FRN. Suppose e(0) < 1.
41

• If σw = 0, then λ(l) = σb2 and γ (l) = l(σv2 Wφ(σb2 , σb2 ) + σa2 ) + γ (0) = l(σv2 Vφ(σb2 ) +
σa2 )+γ (0) . Thus e(l) → 1 and 1−e(l) = Θ(l−1 ). As a result, s(l) = p(l) (1−e(l) ) = Θ(1).
• If σw > 0, then e(l) converges to the unique fixed point e∗ 6= 1 determined by the equation
1
2
e∗ = 2
[σv2 arcsin (e∗ ) + σa2 ].
2
σv + σa
π
∗

Furthermore, e(l) converges to e∗ polynomially: |e(l) − e∗ | is Θ̌(l−δ ), where
δ ∗ := 1 −

2
1
σv2
1
2
p
∈ [ − 1, )
2
2
∗
2
π 1 − (e ) σv + σa
π
2

Since e∗ < 1, s(l) = Θ(p(l) ) = Θ(l).
Proof. The σw = 0 case is obvious. We will assume σw > 0 from here on.
If σa = 0, then e∗ as defined above is 0, and e =
as before.

γ
p

2

decreases as Θ(l π −1 ) to 0, by the same reason

So from now on suppose σa > 0. We apply Lemma C.23 first to show that e converges. We have
σv2 Wφ(q, cq) + σa2 = ep − ep
= ep − ep + ep − ep
= (e − e)p + e(p − p)
p
= (p − p)[(e − e)
+ e]
p−p
p
σv2 Wφ(q, cq) + σa2
+e
= (e − e)
σv2 Vφ(q) + σa2
p−p


p − p σv2 Wφ + σa2
−e =e−e
p
σv2 Vφ + σa2
i
h
2
(l)
(l−1)
σv2 Wφ(q(l) ,c(l) q(l) )+σa
If we define fl (u) := p −p
−
u
+ u (the LHS of the above), then
p(l)
σ 2 Vφ(q(l) )+σ 2
v

a

fl (u) − u = O(l−1 ) uniformly for all u because p(l) = Θ(l), p(l) − p(l−1) = Θ(1), and the part
in the bracket is O(1), with constants all (able to be taken) independent of u. We divide [0, 1) into
the following intervals I1 = [1, 1/2), I2 = [1/2, 3/4), I3 = [3/4, 7/8), . . .. For each Ik , it is clear
that the trajectories of e(l) = fl (e(l−1) ) with e(0) ∈ Ik will fall into some interval Jk bounded away
from 1 for all l ≥ L, for large enough
h 2L2 (dependent2 on k). Theniwe can apply Lemmas C.1, C.5
(l)
(l−1)
σv π arcsin(u)+σa
and C.21 to get fl (u) = p −p
− u + o(1) + u where the constants in o(1)
σ 2 +σ 2
p(l)
v

a

is uniform for all e(0) ∈ Ik . For u < e∗ (as defined in the theorem statement),
2
σ 2 2 arcsin(u)+σa
e∗ , v π σ2 +σ2
v
a

2
2
σv2 π
arcsin(u)+σa
2
σv2 +σa

>u

< u (see Fig. C.12). Thus as l → ∞, the o(1) term gets smaller
h 22
i
2
σ
arcsin(u)+σa
and smaller, and this monotonicity holds for fl (u) − u = v π σ2 +σ2
− u + o(1) > 0 (resp.
and for u >

v

a

< 0) on larger and larger intervals [0, al ] ∩ Jk (resp. [bl , 1) ∩ Jk ). This proves all the preconditions
for Lemma C.23, which yields that Ik converges to a limit point. As this argument is independent of
k, we have that for all e(0) ∈ [0, 1), e(l) converges.
Now we solve for the limit point.
Suppose e has limit point e† (possibly different from e∗ described in the theorem); if we express
γ (l) = (e† + (l) )p(l) , then
σv2 Wφ(q, cq) + σa2 = γ − γ
= (e† + )p − (e† + )p
= e† (p − p) + p − p
p
σv2 Wφ(q, cq) + σa2
= e† +  + ( − )
2
2
σv Vφ(q) + σa
p−p
42

1.0

2 sin-1 (x)
π

0.5

1.5 × 2 sin-1 (x)
π

+0.5

1.5+0.5
-1.0

0.5

-0.5

1.0

0.5 × 2 sin-1 (x)
π

+1.5

0.5+1.5
-0.5

x

-1.0

Figure C.12: Graph of y(e) =

22
1
2 +σ 2 [σv π
σv
a

arcsin(e) + σa2 ] for various σv and σa .

As l → ∞, c ∼ e → e† , and Wφ(q, e† q) → π2 arcsin(e† ), and Vφ(q) → 1. Additionally,
p/(p − p) = Θ(l) and  = o(1) so that  −  = o(l−1 ). Then we have, taking limits l → ∞,
σv2 π2 arcsin(e† ) + σa2
= e† .
σv2 + σa2
Since fl (as defined above) repels points away from 1, the only solution for e† when e(0) < 1 is
e† = e∗ as specified in the theorem statement.
We defer the proof of the convergence rate to e∗ to Thm C.25.
Lemma C.24. Let e∗ be the stable fixed point determined by σa and σv . Then as long as σv > 0,
1
1 2
2
σv2
p
∈( , ]
2
2
∗
2
π 1 − (e ) σv + σa
2 π
Proof. Write ρ :=

2
σa
σv2 .

By definition of e∗ , we get

2
arcsin e∗ + ρ
π
e∗ − π2 arcsin e∗
ρ==
1 − e∗
Substituting ρ into the expression in question, it follows that we want to show

−1
1 − π2 arcsin e∗
2
2
1 2
∗ 2 −1/2
−1
∗ 2 −1/2
(1 − e )
(1 + ρ) = (1 − e )
∈( , ]
∗
π
π
1−e
2 π
e∗ = (1 − ρ)

for e∗ ∈ [0, 1) (the endpoint at 1 is not included since σv > 0. But this is
2
2
(1 − e∗ )1/2 (1 + e∗ )−1/2 (1 − arcsin e∗ )−1 .
π
π
Set g(e∗ ) to be this expression. We could proceed by finding critical points, but a simple plot
Fig. C.13 shows that g is decreasing on [0, 1), with extremal values at the end points:
g(e∗ ) ∈ [ lim
g(e∗ ), g(0)),
∗
e →1

for e∗ ∈ [0, 1).

√
For the limit, we note that arcsin e∗ has an asymptotic expansion π2 − 2(1 −
π
1
e)1/2 + Θ((1 − e)3/2 ) at 1, so that (1 − e∗ )1/2 (1 − π2 arcsin e∗ )−1 → √ , and g(e∗ ) → as
2
2 2
e∗ → 1.
Obviously g(0) =

2
π.

Theorem C.25. If e(0) < 1, then |e(l) − e∗ | is Ω(l−δ
δ ∗ := 1 −

∗

−ε

σv2

) and O(l−δ

∗

+ε

) for any ε > 0, where

2
1
2 1
p
∈ [1 − , ),
π 1 − (e∗ )2 σv2 + σa2
π 2

where the bounds on the right follow from Lemma C.24.
43

0.64
0.62
0.60

g[* ]
0.58

1
2

0.56

2
π

0.54
0.52

0.2

0.4

0.6

0.8

1.0

*

Figure C.13: Plot of g(e∗ ) in the proof of Lemma C.24

Proof. Define ω(q, c) = π2 arcsin(c) − W tanh(q, cq). By Lemma C.21, for large enough l, c is
close to e∗ bounded away from 0 or 1, so that ω(q, c) = Θ(q−1 ) with the constant hidden in Θ
independent of c. Additionally, by Lemma C.5, 1 − V tanh(q) = Θ(q−1/2 ). Therefore,
2
arcsin(e∗ + ) − ω(q, c)) + σa2 + γ
π
2

+ Θ(2 )] − Θ(l−1 ) + σa2 + γ
= σv2 [arcsin(e∗ ) + p
π
1 − (e∗ )2

2
= e∗ (σv2 + σa2 ) + (e∗ + )p + σv2 p
+ Θ(2 ) − Θ(l−1 )
π 1 − (e∗ )2

2
e∗ (p − p − σv2 − σa2 ) = p − p + σv2 p
+ Θ(2 ) − Θ(l−1 )
π 1 − (e∗ )2
2

e∗ σv2 (Vφ(q) − 1) = p − p + σv2 p
+ Θ(2 ) − Θ(l−1 )
π 1 − (e∗ )2
1
2
1
 = (e∗ σv2 (1 − Vφ(q)) + Θ(2 ) − Θ(l−1 ) + (p + σv2 p
))
p
π 1 − (e∗ )2
(e∗ + )p = σv2 (

= Θ(l−3/2 ) + (1 − δ (l) /l)

where
l 2
1
2
(σ Vφ(q) + σa2 − σv2 p
) + Θ(/l)
p v
π 1 − (e∗ )2
1
2
)/(σv2 + σa2 ) + Θ(/l)
= (1 + Θ(l−1/2 ))(σv2 (1 − Θ(l−1/2 )) + σa2 − σv2 p
π 1 − (e∗ )2

δ (l) =

= δ ∗ + O(l−1/2 ),
where δ ∗ := 1− π2 √
∗

σv2
1
2
2 , which is positive by Lemma C.24.
1−(e∗ )2 σv +σa

By taking the δ of Lemma C.11

∗

to be δ + ε or δ − ε respectively for lower and upper bounding the dynamics of (l) , the solution
∗
∗
(l) is Ω(l−δ −ε ) and O(l−δ +ε ) for any ε > 0 since 12 > δ ∗ .

C.6.2

Backward Dynamics

Theorem B.13. Assume φ = tanh in an FRN.
• If σw = 0, χ(m) = χ(l) for all l, m.
• If σw > 0, then for l ≥ m ≥ 0,
√
√
log(χ(m) /χ(l) ) = A( l − m) + B(log l − log m) + O(1)
44

where
r

2 σv2 σw
p
π σv2 + σa2


4
3
σv4
2
B=
− σw
9π σv2 + σa2 σv2 + σa2

4
A=
3

Proof. The σw = 0 case is obvious. We will assume σw > 0 from here on.
As in the proof of Thm B.6,
−1/2

−1
log(χ(m) /χ(l) ) = 2BDσw
b0

√
√
( l − m)

−3/2

−1
− (BDσw
b0
q
2
where B = σv2 σw
, D = 23 π2 ,

−2 −1 −1
b1 2−1 + B 2 D2 σw
b0 2 )(log l − log m) + O(1)

b0 = σv2 + σa2
−2Cσ 2 σ −1
b1 = p v w
σv2 + σa2
b2 =
with C =

q

2
π.

−2
−C 2 σv4 σw
.
2
2
(σv + σa )2

This simplifies to the desired form.

Theorem B.14. Suppose φ = tanh in an FRN.
• If σw = 0, then
(l)

χb = σv2 χ(L) Vφ̇(σb2 )
2 (L)
χ(l)
Vφ̇(σb2 )((σv2 Vφ(σb2 ) + σa2 )(l − 1) + p(0) )
w = σv χ
(L)
χ(l)
Vφ(σb2 )
v =χ
(L)
χ(l)
.
a =χ

• If σw > 0, then for l ≥ m ≥ 0,
√
√
(l)
/χb ) = A( l − m) + Bb (log l − log m) + O(1)
√
√
(l)
log(χ(m)
m) + Bw (log l − log m) + O(1)
w /χw ) = A( l −
√
√
(m)
(l)
log(χa /χa ) = A( l − m) + B(log l − log m) + O(1)
√
√
(l)
log(χ(m)
m) + B(log l − log m) + O(1)
v /χv ) = A( l −
q


σ2 σ
σv4
4
3
2
where A = 43 π2 √ v2 w 2 and B = 9π
−
σ
2
2
2
2
w are as in Thm B.13 and
σ +σ
σ +σ
(m)

log(χb

σv +σa

Bb = B +

1
2

v

a

v

a

and Bw = B − 12 .

Proof. Similar to Thm B.7.
C.7

α-ReLU: Full Residual Network

The following can be checked readily
Lemma B.15. If α > − 12 , then Vψα (q) = cα q α , where cα =
Since ψ̇α = αψα−1 , we have as a corollary,
45

√1 2α−1 Γ
π

α+

1
2



.

0.11
0.10
0.09
0.08
0.07
0.06
0.05
0.04
1

p (l) =l 1 ¡ ®

0.03

1

[¾v2 ¾w2® c® (1 ¡ ®)] 1 ¡ ®

0.02
0

2000

4000

6000

8000

10000

Figure C.14: Verification of leading term of Thm C.28 for α = 0.55.

Lemma C.26. If α > 21 , then Vψ̇α (q) = α2 cα−1 q α−1 .
As a special case, when α = 1, cα = 12 .
The following is a trivial computation, but useful for many simplifications.
Lemma C.27. cα+1 /cα = 2α + 1.
C.7.1

Forward Dynamics

2
Theorem C.28. Suppose we have the nonlinearity φ = ψ1 . Then p(l) = Θ((1 + σv2 σw
/2)l ), with
the hidden constant depending on the initial condition.

Proof. We have
1 2 2
σ (σ p + σb2 ) + σa2 + p
2 v w
1
1
2
+ 1)p + (σv2 σb2 + σa2 ).
= ( σv2 σw
2
2
By the standard method of characteristic equation, we get that
p=

p(l) = A + CB l
where A = −

2
σa
+σb2 σv2
,
2
σv2 σw

B =1+

2
σv2 σw
2 ,

and C is a coefficient determined by initial conditions.

Theorem C.29. Suppose α < 1. We have the following asymptotic expansion
1

p(l) = K1 l 1−α + R(l)
where the remainder term

α

−K2 l 1−α log l
R(l) ∼ (C − K2 )l log l

 C(1−α) l
1−2α
1

if α >
if α =
if α <

1
2
1
2
1
2

and K2 6= C

1

α

2α
2α 1−α
where K1 = [σv2 σw
cα (1 − α)] 1−α , K2 = 21 [σv2 cα σw
]
(1 − α) 1−α −1 α and C = σa2 .

Fig. C.14 verifies the leading coefficient and the exponent of the leading term.
Proof. The difference equation governing the evolution of p is
p − p = A(p + B)α + C
2α
2
where A = σv2 cα σw
, B = σb2 /σw
, and C = σa2 . Then Lemma C.15 yields the result.

Thm C.29 combined with Thm C.28 gives the following result.
46

f(c)

f(c)

0

1.0

1
6

0.8

1

0.6

1

3

2

2
0.4

3

5
6

0.2

1
0.2

0.4

0.6

0.8

1.0

c

f(c)

-5

1.00

-6



-7

1
2

2

-8

3

1

0.99

2

2
3

0.98

5

5

-9

6

6

0.97

1

1

-10

c

0.96

-11

c
0.96

0.97

0.98

0.99

1.00

c

0.96

0.97

0.98

0.99

1.00

c

Figure C.15: (a) Jα for different αs and the identity function. From this plot, it looks like Jα (c) ≥ c and
J̇α (c) ≤ 1 for all α ∈ ( 12 , 1] with equality iff c = 1, but this is misleading. (b) shows |Jα (c) − c| in log scale.
Where the curves dip below the x-axis indicate points where Jα (c) = c. We see that in fact every Jα has a
solution Jα (c) = c for a c < 1, when α < 1. (c) Furthermore, at each such c, J̇α < 1.
f(c)
1.0

0.8

1

0.6

c
0.4

0.2

0.2

0.4

0.6

0.8

1.0

c

Figure C.16: J1 vs identity

Theorem B.16. Suppose we have the nonlinearity φ = ψα . The in an FRN: If α = 1, then
2
p(l) = Θ((1 + σv2 σw
/2)l ), with the hidden constant depending on the initial condition. If 0 < α < 1,
1
1
1
2α
then p(l) = Θ(l 1−α ). More precisely, liml→∞ p/l 1−α = [σv2 σw
cα (1 − α)] 1−α .
By [2], we know that Wψα (q, qc) = Vψα (q)Jα (c), where Jα (c) = Jα (arccos c) and
Z π/2
1
dη cosα η
Jα (θ) :=
(sin θ)2α+1 Γ(α + 1)
.
2πcα
(1 − cos θ cos η)1+α
0

(4)

Note that Jα (c) ∈ (−∞, ∞) for α ∈ (−1, ∞) and any c ∈ (0, 1), even though Vψα is only defined
for α > −1/2.
Fig. C.15 shows a comparison of Jα for different αs along with the identity function. By
[3, Lemma 11], Jα is an increasing and convex function as long as ψα2 is Gaussian-integrable,
which is precisely when α > −1/2. We can compute Jα (1) = Wψα (q, q)/Vψα (q) = 1, and
Γ( α + 1 )2

2
2
Jα (0) = Wψα (q, 0)/Vψα (q) = Vψα/2 (q)2 /Vψα (q) = c2α/2 /cα = 2√1 π Γ(α+
. We record these
1
2)
observations as a lemma.
Lemma C.30. Jα (c) is an increasing and convex function for each α > −1/2 on c ∈ [0, 1].

Jα (1) = 1 and Jα (0) =

α
1 2
1 Γ( 2 + 2 )
√
.
2 π Γ(α+ 21 )

For α = 1, Cho and Saul [2] computed
1 p
J1 (c) = ( 1 − c2 + (π − arccos(c))c).
π
Fig. C.16 shows a plot of J1 vs identity. It has derivative J̇1 (c) = 1 − π1 arccos c, which shows
that J̇1 (c) < 1 with equality iff c = 1, and consequently J1 (c) ≥ c with equality iff c = 1. At the
same time, J̇1 (c) ≥ 0 with equality
iff c = −1, so J1 is increasing on [−1, 1]. It has an asymptotic
√
2 2 3/2
expansion J1 (1 − ε) = 1 − ε + 3π  + Θ(5/2 ) at 1.
R∞
The zeroth Bessel function of the second kind is defined by K0 (z) = 1 e−zx (x2 − 1)−1/2 dx. It
is one of the fundamental solutions to the homogeneous differential equation x2 ẏ + xẏ − x2 y = 0.
The following lemma shows that Jα can be expressed in terms of K0 .
R∞
1
Lemma C.31. For any α > −1, Jα (θ) = 2πc
sin2α+1 θ 0 dxK0 (x)ex cos θ xα
α
Proof. Cho and Saul [2] gave the expression
Z ∞
Z
2πcα Jα (θ) = csc θ
du
0

∞

2

dve−(u

0

47

+v 2 −2uv cos θ)/2 sin2 θ α α

u v .

Note that the integrand is symmetric in u and v. Thus, if V = {(u, v) : u, v ≥ 0 & v ≥ u}, then
Z
2
2
2
2πcα Jα (θ) = 2 csc θ du dve−(u +v −2uv cos θ)/2 sin θ uα v α .
V

√
Now make the change of variables from V to {(p, q) : q ≥ 2 p}:

p = uv
q=u+v
dp dq = (v − u) du dv

dp = v du + u dv
dq = du + dv

du dv = (q2 − 4p)−1/2 dp dq

so that we have
Z
2πcα Jα (θ) = 2 csc θ
0

∞

dpep(1+cos θ) csc

2

θ

pα

Z

∞

√
2 p

dqe−q

2

csc2 θ

(q2 − 4p)−1/2 .

The inner integral in q can be expressed in terms of K0 by a change of variable x = q2 /2
Z ∞
2
2
1
dpep(1+cos θ) csc θ pα e−p csc θ K0 (p csc2 θ)
2πcα Jα (θ) = 2 csc θ
2
0
Z ∞
2
= csc θ
dpK0 (p csc2 θ)ep cos θ csc θ pα
0
Z ∞
2α+1
= sin
θ
dxK0 (x)ex cos θ xα

√

p:

0

Define Lα (θ) = 2πcα Jα (θ) csc2α+1 θ =
Lemma C.32. If α > 1, then

R∞
0

dxK0 (x)ex cos θ xα .

Lα (θ) = csc2 θ[(2α − 1) cos θLα−1 (θ) + (α − 1)2 Lα−2 (θ)].
Proof. We will prove this claim for θ < 1, and by continuity this also proves the case θ = 1. As
remarked above, K0 (z) = K̈0 (z) + z −1 K̇0 (z). Thus
Z ∞
Lα (θ) =
dx(K̈0 (x) + x−1 K̇0 (x))ex cos θ xα
0
x cos θ α−1 ∞
= K̇0 ex cos θ xα |∞
x
|0
0 + K0 e
Z
− dx[cos θex cos θ xα + αex cos θ xα−1 ]K̇0
Z
− dx[cos θex cos θ xα−1 + (α − 1)ex cos θ xα−2 ]K0

p π −z
Asymptotically, K0 (z) ∼
as z → ∞ and K0 (z) ∼ − ln(z) as z & 0, and K̇0 (z) ∼
2z e
p π −z
− 2z e as z → ∞ and K̇0 (z) ∼ −z −1 as z & 0. Thus, as α > 1,
p
K̇0 ex cos θ xα |∞
π/2e−x(1−cos θ) xα−1 + lim ex cos θ xα−1 = 0
0 = − lim
x→∞
x&0
p
x cos θ α−1 ∞
−x(1−cos θ) α−2
K0 e
x
|0 = − lim π/2e
x
+ lim ex cos θ xα−1 ln x = 0
x→∞

x&0

So
Z
Lα (θ) = − cos θLα−1 (θ) − (α − 1)Lα−2 (θ) −

dx[cos θex cos θ xα + αex cos θ xα−1 ]K̇0

Via another integration by parts, the integral on the right is
x cos θ α−1
cos θex cos θ xα K0 |∞
x
K0 |∞
0 + αe
0
Z
2
x cos θ α
x cos θ α−1
− dx[cos θe
x + 2α cos θe
x
+ α(α − 1)ex cos θ xα−2 ]K0

= −[cos2 θLα (θ) + 2α cos θLα−1 (θ) + α(α − 1)Lα−2 (θ)]
48

where the evaluation terms vanish just like before. Altogether, we have
Lα (θ) = cos2 θLα (θ) + (2α − 1) cos θLα−1 (θ) + (α − 1)2 Lα−2 (θ)
= csc2 θ[(2α − 1) cos θLα−1 (θ) + (α − 1)2 Lα−2 (θ)]

As a corollary we get
Lemma C.33. Suppose α > 1. Then
Jα (θ) = cos θJα−1 (θ) + (α − 1)2 (2α − 1)−1 (2α − 3)−1 sin2 θJα−2 (θ)
Jα (c) = cJα−1 (c) + (α − 1)2 (2α − 1)−1 (2α − 3)−1 (1 − c2 )Jα−2 (c)
The derivative of Jα (θ) turns out to be quite simple.
Lemma C.34. Suppose α > 0. Then
J˙α (θ) = −α2 (2α − 1)−1 Jα−1 (θ) sin θ
J̇α (c) = α2 (2α − 1)−1 Jα−1 (c)
Proof. We will prove the first formula. The second follows from chain rule. By Lemma C.31,
Z
1
sin2α+1 θ dxK0 (x)ex cos θ xα
Jα (θ) =
2πcα
Z
1
2α
˙
Jα (θ) =
[(2α + 1) sin θ cos θ dxK0 (x)ex cos θ xα
2πcα
Z
− sin2α+2 θ dxK0 (x)ex cos θ xα+1 ]
cα+1
csc θJα+1 (θ)
cα
= (2α + 1) csc θ[cos θJα (θ) − Jα+1 (θ)].
= (2α + 1) cot θJα (θ) −

As α + 1 > 1, by Lemma C.33, this is
− (2α + 1) csc θ[(α − 1)2 (2α + 1)−1 (2α − 1)−1 sin2 θJα−1 (θ)]
= −(α − 1)2 (2α − 1)−1 sin θJα−1 (θ).

Thus J̇α (1) = α2 (2α − 1)−1 Jα−1 (1) = α2 (2α − 1)−1 for any α > 0 by Lemma C.30. For
1/2 < α ≤ 1, J̇α (1) ≥ 1 with equality iff α = 1, and for α = 1/2, J̇α (1) = ∞ > 1 by continuity
of J̇α (c) in α. Because for α > −1/2, Jα is increasing and convex on [0, 1] and Jα (0) > 0 by
Lemma C.30, Jα intersects identity at a unique point away from 1 when α ∈ [1/2, 1). We record this
as a theorem.
Theorem C.35. For α ∈ [1/2, 1), Jα (c) = c has two solutions: an unstable solution at 1 (”unstable”
meaning J̇α (1) > 1) and a stable solution in e∗ ∈ (0, 1) (”stable” meaning J̇α (e∗ ) < 1).
This result confirms that pictures presented in Fig. C.17b,c are qualitatively correct, that there are
indeed stable fixed points of Jα away from 1.
2 −1
Theorem B.17. Suppose φ = ψ1 . Then in an FRN, e(l) → 1 and 1 − e(l) ∼ [ 14 σv2 σw
B U l]−2 for
√
2
B = 1+σv2 σw
/2 and U = 23π2 . As a result, s(l) = (1−e(l) )p(l) = Θ(l−2 exp(Θ(l))) = exp(Θ(l)).
Proof. If e < 1, then
c=

2
σw
γ + σb2
≥e
2 p + σ2
σw
b

J1 (c) ≥ J1 (e)
σ 2 cα qα J1 (c) + σb2
e= v 2
≥ J1 (e)
σv cα qα + σb2
49

f(c)

0

1.0

1
6

0.8

1

0.6

1

3

2

2
0.4

3

5
6

0.2

1
0.2

0.4

0.6

0.8

1.0

f(c)

f(c)
-5

1.00

-6

1

-7

3

2

2
3

0.98

5

5

-9

6

6

0.97

1

1

-10

c

0.96
-11

c

c

1

0.99

2

2

-8

0.96

0.97

0.98

0.99

1.00

c

0.96

0.97

0.98

0.99

1.00

c

Figure C.17: Left-to-right: (a) Jα for different αs and the identity function (black, dashed line). J1 is highlighted
in red. From this plot, it looks like Jα (c) ≥ c and J̇α (c) ≤ 1 for all α ∈ ( 12 , 1] with equality iff c = 1, but
this is misleading. (b) shows |Jα (c) − c| in log scale. Where the curves dip below the x-axis indicate points
where Jα (c) = c. We see that in fact every Jα has a solution Jα (c) = c for a c < 1, when α < 1. (c)
Furthermore, at each such c, J̇α < 1. (b) and (c) demonstrate the existence of stable fixed points away from 1
for Jα , α ∈ (1/2, 1), which is confirmed rigorously by Thm C.35.

but e ≥ J1 (e) > e as noted above. Thus by monotone convergence e converges, and e∗ = 1 is the
only possible fixed point.
By Lemma C.1, c = e(1 + Θ(p−1 )) = 1 −  + Θ(p−1 ) = 1 − u where u := 1 − Θ(p−1 ). Using
the asymptotic expansion J1 (1 − ) = 1 −  + U 3/2 + Θ(5/2 ), we have
q
(1 − )p = σv2 J1 (1 − u) + σa2 + (1 − )p
2
q
−p = σv2 (J1 (1 − u) − 1) − p
2
2q
= σv [−u + U u3/2 3/2 + Θ(u5/2 5/2 )] − p
2
1
q
 =  [p + σv2 (u − U u3/2 1/2 + Θ(u5/2 3/2 ))]
p
2
1
q
2
=  [p − σa + σv2 (Θ(p−1 ) − U u3/2 1/2 + Θ(u5/2 3/2 ))]
p
2
2
2q
−σa + σv 2 (Θ(p−1 ) − U u3/2 1/2 + Θ(u5/2 3/2 ))
]
= [1 +
p
−σa2 q−1 + 21 σv2 (Θ(p−1 ) − U u3/2 1/2 + Θ(u5/2 3/2 ))
= [1 +
]
pq−1
2
Let the content of the bracket on the RHS be ℵ. We have pq−1 = (1 + o(1))B/σw
. If  = O(p−1 ),
−1
then ℵ = 1 − O(p ), but because p is exponentially decreasing, this means  = Θ(1) and does not
converge to 0 — this is a contradiction. Therefore,  = ω(p−1 ), and

1
2
U 1/2 (1 + o(1))]
 = [1 − B −1 σv2 σw
2
1
2
 −  = − B −1 σv2 σw
U 3/2 (1 + o(1))
2
2 −1
Using Lemma C.14 to upper and lower bound our dynamics, we get that (l) ∼ [ 14 σv2 σw
B U l]−2 .

Lemma C.36. Let φ be any nonlinearity. Suppose Wφ(r, rd) = Vφ(r)K(d) for some twice differentiable function K(d) independent of q, where K(1) = 1 naturally. Suppose further that
• K(d) = d has a solution d = e∗ > 0 where K̇(e∗ ) = δ < 1;
• K(d) > d for all d < e∗ and K(d) < d for all 1 > d > e∗ ; and
• K is nondecreasing.
Let (l) := e(l) − e∗ and suppose e(0) < 1. If γ (l) → ∞ and Vφ(q(l) ) → ∞, then (l) → 0 and
satisfies


σ 2 + (1 − δ + O())σv2 Vφ(q)
= 1− a
+ Vφ(q)Θ(γ −1 p−1 ).
p
50

Proof. First we note that because e∗ is the only stable fixed point of the dynamics x 7→ K(x), with the
basin of attraction [0, 1), we can show e(l) → e∗ as in the proof of Thm B.11 (using Lemma C.23).
Write V (l) := Vφ(q(l) ). We first show that e(l) → e∗ . When l is large,
c=
e=

2
σw
γ + σb2
= e(1 + O(γ −1 ))
2
σw p + σb2

σv2 V K(c) + σa2
= K(c)(1 + O(V −1 K(c)−1 )).
σv2 V + σa2

If γ (l) is bounded for all l, then e → 0 because p(l) → ∞. Since K(c) > 0 for c ∈ [0, 1]
and V (l) → ∞, we have that in the limit l → ∞, liml→∞ e = 0 = K(liml→∞ e) = K(0) (by
the continuity of K), which is impossible by our assumptions. Thus γ (l) → ∞, and we have
liml→∞ e = K(liml→∞ e). By our assumptions, e∗ is the only stable fixed point of K with basin of
attraction [0, 1), so this shows that e → e∗ as desired.
Now we derive the equation in question. Note that c = e(1 + Θ(γ −1 )) because e∗ < 1. We use the
Taylor expansion K(e∗ + ) = e∗ + δ + O(2 ).

(e∗ + )p = σv2 V K (e∗ + )(1 + Θ(γ −1 )) + σa2 + (e∗ + )p
= σv2 V (e∗ + δ( + Θ(γ −1 )) + O(2 )) + σa2 + (e∗ + )p
p = σv2 V (δ( + Θ(γ −1 )) + O(2 )) + p
 = (1 −

σa2 + (1 − δ + O())σv2 V
) + Θ(V γ −1 p−1 )
p

Theorem B.18. Suppose φ = ψα for 0 < α < 1 in an FRN. Then e converges to the unique nonunit
fixed point e∗ of Jα , and |e∗ − e(l) | is Θ̌(l−µ ), where µ = (1 − J̇α (e∗ ))/(1 − α). Additionally,
s(l) = Θ(p(l) ) = Θ(l1/(1−α) ).
Proof. We apply Lemma C.36. We first check the conditions of the lemma, with K = Jα . The
following conditions were already verified.
• Jα has a fixed point e∗ less than but very close to 1, where its slope is υ := J̇α (e∗ ) < 1.
(Thm C.35)
• Jα (d) > d for all d < e∗ and Jα (d) < d for all d > e∗ . (By the convexity shown in
Lemma C.30)
• Jα is nondecreasing (Lemma C.30). Furthermore, from its integral formula (Eq. (4)), we
see easily that Jα is smooth at e∗ < 1.
We also proved the following
1

1

2α
• p(l) ∼ [σv2 σw
cα (1 − α)] 1−α l 1−α (Thm C.29) and γ (l) is asymptotically a constant fraction
(l)
of p (Lemma C.36), so both go to ∞.
2
• Vψα (q) = cα qα = cα (σw
p + σb2 )α = Θ(lα/(1−α) ), so goes to ∞. (Lemma B.15)

Thus, for υ = J̇(e∗ ),
2α
σa2 + (1 − υ + O())σv2 Vφ(q)
(1 − υ)σv2 σw
cα
∼
1−α
p
p
= l−1 (1 − υ)/(1 − α).

51

1

Now, Vφ(q)γ −1 p−1 = Θ(l− 1−α −1 ). By using the dynamics of Lemma C.11 to upper and lower
bound our dynamics, we have (l) = Ω(l−µ− ), O(l−µ+ ) for any  > 0, where µ = min((1 −
υ)/(1 − α), 1/(1 − α)) = (1 − υ)/(1 − α).

C.7.2

Backward Dynamics

Lemma C.37. Suppose random variable X ∼ N (0, σ 2 ), and Y = ψ−β (X) for some β > 0, where
ψα is α-ReLU. Then for ξ > 0, Y has density
Pr[Y ∈ [ξ, ξ + dξ]] =

β

√

1

1

2πσ 2

ξ − β −1 e−ξ

−2/β

/2σ 2

.

At ξ = 0, Y has density given by a Dirac delta of mass 21 .
Furthermore, Y has finite second moment iff β < 12 .
Proof. We have
Pr[Y ∈ [ξ, ∞)] = Pr[X ∈ [0, ξ −1/β ]]
Z ξ−1/β
2
2
1
=√
e−x /2σ dx.
2
2πσ 0
Differentiating the RHS against ξ using Leibniz’s rule, we get
d Pr[Y ∈ [ξ, ∞)]/dξ = √
=

1
2πσ 2

e−ξ

−2/β

/2σ 2

d −1/β
ξ
dξ

−2/β
1
−1
/2σ 2
√
ξ − β −1 e−ξ
.
2
β 2πσ

Negating both sides gives the density fY of Y for ξ > 0. For ξ = 0, observe that limξ→0 fY (ξ) = 0
1
−2/β
/2σ 2
because, while ξ − β −1 blows up polynomially, e−ξ
blows up exponentially. Thus the
contribution of Y ’s mass at Y = 0 from X > 0 is 0. On the other hand, all X < 0 gets mapped to
Y = 0, so fY (0) = 12 δ0 , where δ0 is the Dirac delta.
For the second assertion, observe that
fY (ξ) ∼
Thus, ξ 2 fY (ξ) is integrable iff 2 −

1
β

√

1

1

β 2πσ 2

ξ − β −1 as ξ → ∞.

− 1 < −1 ⇐⇒ β < 21 .

Theorem B.19. Suppose we have the nonlinearity ψα in an FRN. Var(ψ̇α (ζ)2 ) diverges for any
Gaussian variable ζ with mean 0 if α ≤ 34 but is finite if α > 43 .
Proof. Note that ψ̇α ∝ ψα−1 , so it suffices to show that Var(ψα−1 (ζ)2 ) = Var(ψ2α−2 (ζ)) is
infinite for ζ ∼ N (0, σ 2 ). By Lemma C.37 with β = 2 − 2α, ψ2α−2 (ζ) has finite variance iff
β < 12 ⇐⇒ α > 34 .
Theorem B.20. Suppose we have the nonlinearity ψα in an FRN. If α = 1, then χ(l−m) =
m
α2
2
χ(l) 12 σv2 σw
,
+ 1 . If α ∈ ( 43 , 1), then χ(l−m) = Θ(1)χ(l) (l/(l − m))R for R = (1−α)(2α−1)
where the constants in Θ(1) do not depend on l or m.
Proof. If α = 1, then

So χ(l−m) /χ(l) = Θ(1)B m

1
2
χ = χ(1 + σv2 σw
).
2
2
for B = 1 + 12 σv2 σw
.
52

If

1
2

< α < 1, then χ/χ − 1 is
2
σv2 σw
Vφ̇(q)
2 2
= σv2 σw
α cα−1 qα−1
2 2
2
= σv2 σw
α cα−1 (σw
p)α−1 + Θ(pα−2 )
α

1

α−2

α

2α 2
= σv2 σw
α cα−1 (K1 l 1−α − K2 l 1−α log l + o(l 1−α log l))α−1 + Θ(l 1−α )

=
=

2α 2
σv2 σw
α cα−1 [K1α−1 l−1 + Θ(l−2 log l)]
2α 2
σv2 σw
α cα−1 K1α−1 l−1 + Θ(l−2 log l)
−1
−2

= Rl

+ Θ(l

+ O(l

−3

by Thm C.29

)

log l)

2α 2
where R = σv2 σw
α cα−1 K1α−1 =

α2
(1−α)(2α−1)

1

2α
and K1 = [σv2 σw
cα (1 − α)] 1−α . So

χ = χ exp(Rl−1 + Θ(l−2 log l))

R
l
(l−m)
(l)
χ
= Θ(1)χ
l−m
as desired.
Theorem B.21. If φ = ψ1 in an FRN, then for l ≥ m ≥ 0,
(l−m)

χb

= Θ(1)χ(l) B m ,

χ(l−m)
= Θ(1)χ(l) B l ,
w

χ(l−m)
= Θ(1)χ(l) B l ,
v

χ(l−m)
= Θ(1)χ(l) B m .
a

2
/2.
where B = 1 + σv2 σw

If φ = ψα in an FRN, for α < 1, then for l ≥ m ≥ 0,
(l−m)

χb

α

χ(l−m)
= Θ(1)χ(l) lR (l − m) 1−α −R ,
w

= Θ(1)χ(l) lR (l − m)−R−1 ,
α

χ(l−m)
= Θ(1)χ(l) lR (l − m) 1−α −R ,
v

χ(l−m)
= Θ(1)χ(l) (l/(l − m))R .
a

Proof. The proof is similar to that of Thm B.7.

53

