On Continuous-Time Gaussian Channels ∗

arXiv:1704.02569v2 [] 25 Nov 2017

Xianming Liu
Huazhong University of Science and Technology
email: xmliu@hust.edu.cn

Guangyue Han
The University of Hong Kong
email: ghan@hku.hk

November 28, 2017

Abstract
A continuous-time white Gaussian channel can be formulated using a white Gaussian noise, and a conventional way for examining such a channel is the sampling approach based on the Shannon-Nyquist sampling theorem, where the original continuoustime channel is converted to an equivalent discrete-time channel, to which a great variety of established tools and methodology can be applied. However, one of the key
issues of this scheme is that continuous-time feedback and memory cannot be incorporated into the channel model. It turns out that this issue can be circumvented
by considering the Brownian motion formulation of a continuous-time white Gaussian channel. Nevertheless, as opposed to the white Gaussian noise formulation, a
link that establishes the information-theoretic connection between a continuous-time
channel under the Brownian motion formulation and its discrete-time counterparts has
long been missing. This paper is to fill this gap by establishing causality-preserving
connections between continuous-time Gaussian feedback/memory channels and their
associated discrete-time versions in the forms of sampling and approximation theorems, which we believe will play important roles in the long run for further developing
continuous-time information theory.
As an immediate application of the approximation theorem, we propose the socalled approximation approach to examine continuous-time white Gaussian channels
in the point-to-point or multi-user setting. It turns out that the approximation approach, complemented by relevant tools from stochastic calculus, can enhance our
understanding of continuous-time Gaussian channels in terms of giving alternative and
strengthened interpretation to some long-held folklore, recovering “long known” results
from new perspectives, and rigorously establishing new results predicted by the intuition that the approximation approach carries. More specifically, using the approximation approach complemented by relevant tools from stochastic calculus, we first derive
the capacity regions of continuous-time white Gaussian multiple access channels and
broadcast channels, and we then analyze how feedback affects their capacity regions:
feedback will increase the capacity regions of some continuous-time white Gaussian
broadcast channels, while it will not increase capacity regions of continuous-time white
Gaussian multiple access channels.

Index Terms: continuous-time channel, Gaussian channel, feedback, memory, sampling
theorem, network information theory, capacity, capacity region, mutual information
∗

Results in this paper have been partially presented in the 2014 IEEE ISIT [52].

1

1

Introduction

Continuous-time Gaussian channels were considered at the very inception of information theory. In his celebrated paper [66] birthing information theory, Shannon studied the following
point-to-point continuous-time white Gaussian channels:
Y (t) = X(t) + Z(t),

t ∈ R,

(1)

where X(t) is the channel input with average power limit P , Z(t) is the white Gaussian
noise with flat power spectral density 1 and Y (t) are the channel output. Shannon actually
only considered the case that the channel has bandwidth limit ω, namely, the channel input
X and the noise Z, and therefore the output Y all have bandwidth limit ω (alternatively,
as in (9.54) of [14], this can be interpreted as the original channel (1) concatenated with
an ideal bandpass filter with bandwidth limit ω). Using the celebrated Shannon-Nyquist
sampling theorem [54, 67], the continuous-time channel (1) can be equivalently represented
by a parallel Gaussian channel:
Yn(ω) = Xn(ω) + Zn(ω) ,

n ∈ Z,

(2)

(ω)

where the noise process {Zn } is i.i.d. with variance 1 [14]. Regarding the “space” index n
as time, the above parallel channel can be interpreted as a discrete-time Gaussian channel
associated with the continuous-time channel (1). It is well known from the theory of discretetime Gaussian channels that the capacity of the channel (2) can be computed as


P
(ω)
.
(3)
C = ω log 1 +
2ω
Then, the capacity C of the channel (1) can be computed by taking the limit of the above
expression as ω tends to infinity:
C = lim C (ω) = P/2.
ω→∞

(4)

The sampling approach consisting of (1)-(4) as above, which serves as a link between the
continuous-time channel (1) and the discrete-time channel (2), typifies a conventional way
to examine continuous-time Gaussian channels: convert them into associated discrete-time
Gaussian channels, for which we have ample ammunition at hands. Note that when P tends
to 0, using the fact that


P
P
= ,
lim ω log 1 +
P →0
2ω
2
one also reaches (4), which roughly explains the following long-held folklore within the information theory community:
a continuous-time infinite-bandwidth Gaussian channel without feedback or memory is “equivalent” to a discrete-time Gaussian channel with low signal-to-noise
ratio (SNR).
(A)

2

Moments of reflection, however, reveals that the sampling approach for the channel capacity
(with bandwidth limit or not) is heuristic in nature: For one thing, a bandwidth-limited
signal cannot be time-limited, which renders it infeasible to define the data transmission
rate if assuming a channel has bandwidth limit. In this regard, rigorous treatments coping
with this issue and other technicalities can be found in [75, 22]; see also [69] for a relevant
in-depth discussion. Another issue is that, even disregarding the above technical nuisance
arising from the bandwidth limit assumption, the sampling approach only gives a lower
bound for the capacity of (1): it shows that P/2 is achievable via a class of special coding
schemes, but it is not clear that why transmission rate higher than P/2 cannot be achieved
by other coding schemes. The capacity of (1) was rigorously studied in [21, 8], and a complete
proof establishing P/2 as its de facto capacity can be found in [3, 4].
Alternatively, the continuous-time white Gaussian channel (1) can be examined [38] under
the Brownian motion formulation:
Z t
Y (t) =
X(s)ds + B(t),
(5)
0

where, slightly abusing the notation, we still use Y (t) to denote the output corresponding
to the input X(s), and B(t) denotes the standard Brownian motion (Z(t) can be viewed as
a generalized derivative of B(t)); equivalently, the channel (5) can be seen as the original
channel (1) concatenated with an integrator circuit. As opposed to white Gaussian noises,
which only exist as generalized functions [60], Brownian motions are well-defined stochastic
processes and have been extensively studied in probability theory. Here we remark that, via
a routine orthonormal decomposition argument, both of the two channels are equivalent to
a parallel channel consisting of infinitely many Gaussian sub-channels [5].
An immediate and convenient consequence of such a formulation is that many notions in
discrete time, including mutual information and typical sets, carry over to the continuoustime setting, which will rid us of the nuisances arising from the bandwidth limit assumption.
Indeed, such a framework yields a fundamental formula for the mutual information of the
channel (5) [15, 42] and a clean and direct proof [42] that the capacity of (5) is P/2; moreover,
as evidenced by numerous results collected in [38] and some recent representative work [71,
72] on point-to-point Gaussian channels, the use of Brownian motions elevates the level of
rigorousness of our treatment, and equip us with a wide range of established techniques and
tools from stochastic calculus. Here we remark that Girsanov’s theorem, one of the most
important theorems in stochastic calculus, lays the foundation of our rigourous treatment;
for those who are interested in the technical details in our proofs, we refer to [51, 38],
where Girsanov’s theorem (and its numerous variants) and its wide range of applications in
information theory are discussed in great details.
Furthermore, as elaborated in Remark 3.8, the Brownian motion formulation is also
versatile enough to accommodate feedback and memory; in particular, the point-to-point
continuous-time white Gaussian memory/feedback channel can be characterized by the following stochastic differential equation:
Z t
Y (t) =
g(s, W0s, Y0s )ds + B(t), t ∈ [0, T ],
(6)
0

where g is a function from [0, T ] × C[0, T ] × C[0, T ] to R. Note that (6) can be interpreted
3

1) either as a feedback channel, where W0s can be rewritten as M, interpreted as the
message to be transmitted through the channel, and g(s) can be rewritten as X(s),
interpreted as the channel input, which depends on M and Y0s , the channel output up
to time s that is fed back to the sender,
2) or as a memory channel, where W0s can rewritten as X0s , interpreted as the channel
input, g is “part” of the channel, and Y (t), the channel output at time t, depends on
X0t and Y0t , the channel input and output up to time t that are present in the channel
as memory, respectively.
Note that, strictly speaking, the third parameter of g in (6) should be Y0s− , which, however,
can be equivalently replaced by Y0s due to the continuity of sample paths of {Y (t)}. Note
that, with the presence of feedback/memory, the existence and uniqueness of Y is in fact a
tricky mathematical problem, however, we will in this paper simply assume that the input
X is appropriately chosen such that Y uniquely exists. For more detailed discussion about
the Brownian motion formulation, we refer the reader to [38].
As opposed to the white Gaussian noise formulation, under the Brownian motion formulation, memory and feedback can be naturally translated to the discrete-time setting:
the pathwise continuity of a Brownian motion allows the inheritance of temporal causality
when the channel is sampled (see Section 2) or approximated (see Section 3). On the other
hand, the white Gaussian noise formulation is facing inherent difficulty as far as inheriting
(w)
temporal causality is concerned: in converting (1) to (2), while Xn are obtained as “time”
(w)
samples of X(t), Zn are in fact “space” samples of Z(t), as they are merely the coefficients
of the (extended) Karhunen-Loeve decomposition of Z(t) [23, 32, 33]; see also [44] for an
in-depth discussion on this.
On the other hand though, as opposed to the white Gaussian noise formulation, a link
that establishes the information-theoretic connection between the continuous-time channel
(6) and its discrete-time counterparts has long been missing, which may explain why discretetime and continuous-time information theory (under the Brownian motion formulation) have
largely gone separate ways with little interaction for the past several decades. In this paper,
we will fill this gap by establishing causality-preserving connections between the channel (5)
and its associated discrete-time versions in the forms of sampling and approximation theorems, which we believe will serve as the above-mentioned missing links and play important
roles in the long run for further developing continuous-time information theory, particularly
for the communication scenarios when feedback/memory is present.
As an immediate application of the approximation theorem, we propose the approximation approach to examine continuous-time Gaussian feedback channels with the average
power constraint and infinite bandwidth (again, by comparison, the conventional sampling
approach cannot handle feedback). It turns out that this approach, when complemented by
relevant tools from stochastic calculus, can greatly enhance our understanding of continuoustime Gaussian channels in terms of giving alternative and strengthened interpretations to the
low SNR equivalence in (A), recovering “long known” results (Theorems 5.1 and 5.4) from
new and rigorous perspectives, and deriving new results (Theorems 5.6 and 5.7) inspired by
the intuition that the approximation approach carries.
Below, we summarize the contributions of this paper in greater details.
In Section 2, we prove Theorems 2.1 and 2.3, sampling theorems for a continuous-time
4

Gaussian feedback/memory channel, which naturally connect such a channel with their sampled discrete-time versions. And in Section 3, we prove Theorems 3.1 and 3.2, the so-called
approximation theorems, which connect a continuous-time Gaussian feedback/memory channel with its approximated discrete-time versions (in the sense of the Euler-Maruyama approximation [34]). Roughly speaking, a sampling theorem says that a time-sampled channel is
“close” to the original channel if the sampling is fine enough, and an approximation theorem
says that an approximated channel is “close” to the original channel if the approximation is
fine enough, both in an information-theoretic sense. Note that, as elaborated in Remark 3.7,
certain version of the approximation theorem boils down to the sampling theorem when there
is no memory and feedback in the channel.
Apparently a sampling theorem, whose spirit is in line with the Shannon-Nyquist sampling theorem, is of practical and theoretical value due to the fact it deals with the “real”
values of the channel output; and, as will be elaborated later, approximation theorems seem
to be surprisingly useful in a number of respects despite the fact it only deals with the “approximated” values of the channel output: it can certainly provide alternative rigorous tools
in translating results from discrete time to continuous time; more importantly, as elaborated
in Section 4, it lays the foundation for the approximation approach, which gives us the intuition in the point-to-point continuous-time setting, which will further help us to deliver
rigorous treatments of multi-user continuous-time Gaussian channels in Section 5.
More specifically, in Section 5, guided by the intuition gleaned from the approximation
approach (see Remark 5.3), we rigorously derive the capacity regions of a continuous-time
white Gaussian multiple access channel (Theorem 5.1) and a continuous-time white Gaussian
broadcast channel (Theorem 5.4). Here, we note that when there is no feedback, as discussed
in Remark 5.2, the results above are “long known” in the sense that they are roughly suggested by the conventional sampling approach, or alternatively, the low SNR equivalence in
(A). However, to the best of our knowledge, explicit formulations and statements of such
results are missing in the literature and their rigourous proofs are non-trivial (for instance,
when establishing Theorem 5.4, we have to resort to the continuous-time I-MMSE relationship [24], which has been established only recently). By comparison, the presence of feedback
necessitates the use of the approximation approach, which help us to connect relevant results and proofs in discrete time to analyze how feedback affects the aforementioned capacity
regions: feedback will increase the capacity regions of some continuous-time Gaussian broadcast (Theorem 5.7), while it will not increase capacity regions of a continuous-time physically
degraded Gaussian broadcast channel (Theorem 5.6) and a continuous-time Gaussian multiple access channels (Theorem 5.1).

2

Sampling Theorems

A very natural question is whether, similarly for the white Gaussian noise formulation,
sampling theorems hold for continuous-time white Gaussian channels under Brownian motion
formulation. In this section, we will establish sampling theorems for the channel (6), which
naturally connect such channels with their discrete-time versions obtained by sampling.
Consider the following regularity conditions for channel (6):
(a) The solution {Y (t)} to the stochastic differential equation (6) uniquely exists;
5

(b)
P

Z

T

g
0

2

(t, W0t , Y0t )dt

<∞



=P

Z

0

T

g

2

(t, W0t , B0t )dt

<∞



= 1;

(c)
Z

T
0

E[|g(t, W0t, Y0t )|]dt < ∞.

Note that all the three above conditions are rather weak: Condition (a) is necessary for the
channel to be meaningful, and Conditions (b) and (c) are very mild integrability assumptions.
Now, for any n ∈ N, choose sampling times tn,0 , tn,1, . . . , tn,n ∈ R such that
0 = tn,0 < tn,1 < · · · < tn,n−1 < tn,n = T,
and let ∆n , {tn,0 , tn,1 , . . . , tn,n }. Sampling the channel (6) over the time interval [0, T ] with
respect to ∆n , we obtain its sampled discrete-time version as follows:
Z tn,i
Y (tn,i ) =
g(s, W0s, Y0s )ds + B(tn,i ), i = 0, 1, . . . , n.
(7)
0

For any sampling time sequence ∆n , we will use δ∆n to denote its minimal stepsize, namely,
δ∆n , max (tn,i − tn,i−1 ).
i=1,3,...,n

∆n is said to be equidistant if tn,i −tn,i−1 = T /n for all feasible i, and we will use the shorthand
notation δn to denote its stepsize, i.e., δn , tn,1 − tn,0 = T /n. Apparently, equidistant
sampling time sequences are natural candidates with respect to which a continuous-time
Gaussian channel can be sampled.
We are primarily concerned with the mutual information for the channel (6), whose
standard definition (see, e.g., [59, 38]) is given below:
i
( h
dµW Y
WY
, if dµdµ
exists ,
E
log
dµW ×µY
W ×µY
(8)
I(W0T ; Y0T ) =
∞,
otherwise ,
where the subscripted µ denotes the measure induced on C[0, T ] or C[0, T ] × C[0, T ] by
the corresponding stochastic process and dµW Y /dµW × µY denotes the Randon-Nikodym
derivative of µW Y with respect to dµW × µY .
Roughly speaking, the following sampling theorem states that for any sequence of “increasingly refined” samplings, the mutual information of the sampled discrete-time channel
(7) will converge to that of the original channel (6).
Theorem 2.1. Assume Conditions (a)-(c). Suppose that ∆n ⊂ ∆n+1 for all n and that
δ∆n → 0 as n tends to infinity. Then, we have
lim I(W0T ; Y (∆n )) = I(W0T ; Y0T ),

n→∞

where Y (∆n ) , {Y (tn,0 ), Y (tn,1 ), . . . , Y (tn,n )}.
6

Proof. The proof is rather technical and thereby postponed to Appendix A.
Regarding the assumptions of Theorem 2.1, as mentioned before, Conditions (a)-(c) are
very weak, but the condition that “∆n ⊂ ∆n+1 for all n” is somewhat restrictive, which,
in particular, is not satisfied by the set {∆n } of all equidistant sampling time sequences.
We next show that this condition can be replaced by some extra regularity conditions: The
same theorem holds as long as the stepsize of the sampling tends to 0, which, in particular,
is satisfied by the set of all equidistant sampling sequences.
Below and hereafter, defining the distance kU0s − V0t k between U0s and V0t with 0 ≤ s ≤ t
as
kU0s − V0t k , sup |U(r) − V (r)| + sup |U(s) − V (r)|,
(9)
r∈[0,s]

r∈[s,t]

we may assume the following three regularity conditions for the channel (6):
(d) Uniform Lipschitz condition: There exists a constant L > 0 such that for any
0 ≤ s1 , s2 , s3 , t1 , t2 , t3 ≤ T , any U0T , V0T , Y0T and Z0T ,
|g(s1, U0s2 , Y0s3 ) − g(t1 , V0t2 , Z0t3 )| ≤ L(|s1 − t1 | + kU0s2 − V0t2 k + kY0s3 − Z0t3 k);

(e) Uniform linear growth condition: There exists a constant L > 0 such that for any
W0T and any Y0T ,
|g(t, W0t , Y0t )| ≤ L(1 + kW0t k + kY0t k),
where

kW0t k = sup |W (r)|,
r∈[0,t]

kY0t k = sup |Y (r)|;
r∈[0,t]

(f) Regularity conditions on W : There exists ε > 0 such that
T 2

E[eεkW0 k ] < ∞,
and for any K > 0, there exists ε′ > 0 such that
2

E[eK sup|s−t|≤ε′ (W (s)−W (t)) ] < ∞,
and there exists a constant L > 0 such that for any ε′′ > 0,
E[sup|s−t|≤ε′′ (W (s) − W (t))2 ] ≤ Lε′′ .
The following lemma, whose proof is postponed to Appendix B, says that Conditions
(d)-(f) are stronger than Conditions (a)-(c). We however remark that Conditions (d)-(f) are
still rather mild assumptions: The uniform Lipschitz condition, uniform linear growth condition and their numerous variants are typical assumptions that can guarantee the existence
and uniqueness of the solution to a given stochastic differential equation. In theory, these
two conditions are considered mild in the sense there are examples that the corresponding
stochastic differential equation may not have solutions at all if these two conditions are not
satisfied (see, e.g., [53]). Note that the third condition is a mild integrability condition; as a
matter of fact, for a feedback channel where W is interpreted as the message, this condition
is trivially satisfied. All three conditions above will be taken for granted in most practical
communication situations: as might be expected, the signals employed in practice will be
much better-behaving.
7

Lemma 2.2. Assume Conditions (d)-(f ). Then, there exists a unique strong solution of (6)
with initial value Y (0) = 0. Moreover, there exists ε > 0 such that
T k2

E[eεkY0

] < ∞,

(10)

which immediately implies Conditions (b) and (c).
Roughly speaking, the following sampling theorem states that if the stepsizes of the
samplings tend to 0, the mutual information of the channel (7) will converge to that of the
channel (6). Note that in this theorem, we do not need the assumption that “∆n ⊂ ∆n+1
for all n”, which is required in Theorem 2.1.
Theorem 2.3. Assume Conditions (d)-(f ). For any sequence {∆n } with δ∆n → 0 as n tends
to infinity, we have
lim I(W0T ; Y (∆n )) = I(W0T ; Y0T ).
n→∞

Proof. The proof is rather technical and lengthy, and thereby postponed to Appendix C.

3

Approximation Theorems

In this section, we will establish approximation theorems for the channel (6), which naturally
connect such channels with their discrete-time versions obtained by approximation. As
elaborated in later sections, the approximation theorem will underpin the approximation
approach that will be introduced in Section 4.
An application of the Euler-Maruyama approximation [34] with respect to ∆n to (6) will
yield a discrete-time sequence {Y (n) (tn,i ) : i = 0, 1, . . . , n} and a continuous-time process
{Y (n) (t) : t ∈ [0, T ]}, a linear interpolation of {Y (tn,i )}, as follows: Initializing with Y (n) (0) =
0, for each i = 0, 1, . . . , n − 1, we recursively define
Z tn,i+1
t
(n),t
(n)
(n)
Y (tn,i+1 ) = Y (tn,i ) +
g(s, W0 n,i , Y0 n,i )ds + B(tn,i+1 ) − B(tn,i ),
(11)
tn,i

which is immediately followed by the linear interpolation below:
Y (n) (t) = Y (n) (tn,i ) +

t − tn,i
(Y (n) (tn,i+1 ) − Y (n) (tn,i )),
tn,i+1 − tn,i

tn,i ≤ t ≤ tn,i+1 .

We are now ready to prove the following theorem:
Theorem 3.1. Assume Conditions (d)-(f ). Then, we have
lim I(W0T ; Y (n) (∆n )) = I(W0T ; Y0T ).

n→∞

Proof. The proof is rather technical and lengthy, and thereby postponed to Appendix D.

8

(12)

For any {∆n }, let W (n) (t) denote the piecewise linear version of W0T with respect to ∆n ;
more precisely, for any i = 0, 1, . . . , n, W (n) (tn,i ) = W (tn,i ), and for any tn,i−1 < s < tn,i
with s = λtn,i−1 + (1 − λ)tn,i where 0 < λ < 1, W (n) (s) = λW (tn,i−1) + (1 − λ)W (tn,i ).
The following modified Euler-Maruyama approximation with respect to ∆n applied to the
channel (6) yields a discrete-time sequences {Y (n) (tn,i ) : i = 0, 1, . . . , n} and a continuoustime processes {Y (n) (t) : t ∈ [0, T ]} as follows: Initializing with Y (n) (0) = 0, for each
i = 0, 1, . . . , n − 1, we recursively define
Z tn,i+1
(n),t
(n),t
(n)
(n)
Y (tn,i+1 ) = Y (tn,i ) +
g(s, W0 n,i , Y0 n,i )ds + B(tn,i+1 ) − B(tn,i ),
(13)
tn,i

which is immediately followed by a linear interpolation below:
Y (n) (t) = Y (n) (tn,i ) +

t − tn,i
(Y (n) (tn,i+1 ) − Y (n) (tn,i )),
tn,i+1 − tn,i

tn,i ≤ t ≤ tn,i+1 .

(14)

Now, using a parallel argument in the proof of Theorem 3.1, we have the following approximation theorem.
Theorem 3.2. Assume Conditions (d)-(f ). Then, we have
lim I(W (n) (∆n ); Y (n) (∆n )) = I(W0T ; Y0T ).

n→∞

Remark 3.3. When the channel (6) is interpreted as a feedback channel, both W (n) and
W are precisely M. When the channel (6) is interpreted as a memory channel, Theorem 3.2
states that the mutual information between its input and output is the limit of that of
its approximated input and output (in the sense of the above-mentioned modified EulerMaruyama approximation).
Other variants of the Euler-Maruyama approximation can also be applied to the channel
to yield variants of the approximation theorem. For instance, under Conditions (d)-(f), for
the following variant of the Euler-Maruyama approximation,
Z tn,i+1
t
(n),t
(n)
(n)
Y (tn,i+1 ) = Y (tn,i ) +
X(tn,i , W0 n,i , Y0 n,i )ds + B(tn,i+1 ) − B(tn,i ),
(15)
tn,i

a parallel argument as in the proof of Theorem 3.1 will give the following variant of Theorem 3.1:
Theorem 3.4. Assume Conditions (d)-(f ). Then, we have
lim I(W0T ; Y (n) (∆n )) = I(W0T ; Y0T ).

n→∞

(16)

Moreover, for
Y

(n)

(tn,i+1 ) = Y

(n)

(tn,i ) +

Z

tn,i+1
tn,i

(n),tn,i

X(tn,i , W0

we have the following variant of Theorem 3.2:
9

(n),tn,i

, Y0

)ds + B(tn,i+1 ) − B(tn,i ),

(17)

Theorem 3.5. Assume Conditions (d)-(f ). Then, we have
lim I(W (n) (∆n ); Y (n) (∆n )) = I(W0T ; Y0T ).

(18)

n→∞

Regarding the approximation theorem and its variants, we make the following several
remarks.
Remark 3.6. Continuous-time directed information has been defined in [72] for continuoustime white Gaussian channels with positively delayed feedback. In this remark, we show that
our approximation theorem can be used to give an alternative definition of continuous-time
directed information, even for the case that the feedback is instantaneous.
Consider the following continuous-time Gaussian feedback channel:
Z t
Y (t) =
X(s, M, Y0s )ds + B(t), t ∈ [0, T ].
(19)
0

For any ∆n , we define X̃ (n) (·) as follows: for any t with tn,i ≤ t < tn,i+1 ,
X̃

(n)

(t) =

i−1 Z
X
j=0

(n),t

Writing X̃(t, M, Y0

tn,j+1
tn,j

(n),t
X(s, M, Y0 n,j )ds

+

Z

t

tn,i

(n),tn,i

X(s, M, Y0

)ds,

) as X̃ (n) (t) for simplicity, (11) can be rewritten as

Y (n) (tn,i+1 ) = Y (n) (tn,i ) + X̃ (n) (tn,i+1) − X̃ (n) (tn,i ) + B(tn,i+1 ) − B(tn,i ),
for which it can be readily checked that
I(X̃ (n) (∆n ) → Y (n) (∆n )) = I(M; Y (n) (∆n )).

(20)

Theorem 3.1 and the above observation can be used to define continuous-time directed
mutual information. To be more precise, the continuous-time directed information from X0T
to Y0T of the channel (19) can be defined as
I(X0T → Y0T ) , lim I(X̃ (n) (∆n ) → Y (n) (∆n )).
n→∞

(21)

Consider the following continuous-time Gaussian channel with possibly delayed feedback:
Z t
Y (t) =
X(s, M, Y0s−D )ds + B(t), t ∈ [0, T ],
(22)
0

where D ≥ 0 denotes the delay of the feedback. In [72], the notion of continuous-time
directed information from X0T to Y0T is defined as follows:
ID (X0T

→

Y0T )

= inf
∆n

n
X
i=1

t

t

t

n,i−1
n,i
n,i
|Ytn,0
).
I(Xtn,0
; Ytn,i−1

(23)

It is proven that for the case D > 0, using this notion, a connection between information
theory and estimation theory can be established as follows:
Z
1 T
T
T
E[(X(t) − E[X(t)|Y0t ])2 ]dt.
(24)
ID (X0 → Y0 ) =
2 0
10

On the other hand though, it is easy to see that for the case D = 0, i.e., there is no delay in
the feedback as in (19), the definition in (23) and the equality as in (24) may run into some
problems: Consider the extreme scenario and choose X(t) = −Y (t) for any feasible t, then
clearly the right hand side of (24) should be equal to 0. On the other hand though, for the
left hand side, each small interval in (23) will yield
t

t

t

t

t

t

t

t

n,i+1
n,i+1
n,i
n,i+1
n,i+1
n,i
n,i+1
n,i+1
ID=0 (Xtn,0
; Ytn,i
|Ytn,0
) = ID=0 (Ytn,i
; Ytn,i
|Ytn,0
) = ID=0 (Ytn,i
; Ytn,i
|Y (tn,i )),

where for the last equality, we have used the fact that under the assumption that X(t) =
−Y (t), {Y (t)} is an Ornstein-Uhlenbeck process, which is a Gaussian Markov process. Noting that given Y (ti ) = y(ti ), the Randon-Nikodym derivative dµY ti+1 Y ti+1 /dµY ti+1 × µY ti+1
ti

does not exist, we conclude, by referring to the definition in (8), that
t

t

t

t

ti

ti

ti

t

n,i+1
n,i+1
n,i+1
n,i
n,i+1
|Ytn,0
) = ∞,
ID=0 (Ytn,i
; Ytn,i
|Y (tn,i )) = ∞, and thereby, ID=0 (Xtn,0
; Ytn,i

which further implies that, ID=0 (X0T → Y0T ), the left-hand side of (24) at D = 0 is infinite,
a contradiction. On the other hand, be it the case D > 0 or D = 0, with the definition in
(21), Theorem 3.1 however promises:
I(X0T

→

Y0T )

=

I(M0T ; Y0T )

1
=
2

Z

0

T

E[(X(t) − E[X(t)|Y0t ])2 ]dt.

Remark 3.7. When there is no feedback or memeory, Theorem 3.1 boils down to Theorem 2.3: obviously we will have
Y (n) (t) = Y (t),
which means that Theorem 3.1 actually states
lim I(W0T ; Y (∆n )) = I(W0T ; Y0T ),

n→∞

which is precisely the conclusion of Theorem 2.3. And moreover, by Remark 3.6, we also
have
lim I(X̃ (n) (∆n ); Y (∆n )) = lim I(X̃ (n) (∆n ) → Y (∆n )) = I(W0T ; Y0T ).
n→∞

n→∞

Remark 3.8. In this remark, we briefly discuss the possible applications of our sampling
and approximation theorems, both of which we believe will important roles in the long run
for further developing continuous-time information theory, particularly for scenarios where
feedback and memory are present.
Taking advantage of the pathwise continuity of a Brownian motion, our sampling theorems, Theorems 2.1 and 2.3, naturally connect continuous-time Gaussian memory/feedback
channels with their discrete-time counterparts, whose outputs are precisely sampled outputs
of the original continuous-time Gaussian channel. In discrete time, the Shannon-McMillanBreiman theorem provides an effective way to approximate the entropy rate of a stationary ergodic process, and numerical computation and optimization of mutual information
of discrete-time channel using the Shannon-McMillan-Breiman theorem and its extensions
have been extensively studied (see, e.g., [25, 26] and references therein), which suggests our
sampling theorems may well serve as a bridge to capitalize on relevant results in discrete
11

time to numerically compute and optimize the mutual information of continuous-time Gaussian channels. In short, despite numerous technical barriers that one needs to overcome,
we believe that in the long run the sampling theorems can help us in terms of numerically
computing the mutual information and capacity of continuous-time Gaussian channels.
By comparison, our approximation theorems, Theorems 3.1 and 3.2, are somewhat “artificial” in the sense that the outputs of the associated discrete-time channels are only approximated outputs of the original continuous-time channels. Nonetheless, as the EulerMaruyama approximation of a continuous-time channel yields the form of a discrete-time
channel typically takes, our approximation theorems allow a smooth translation from the
results and ideas from the discrete-time setting to the continuous-time setting. As a result,
the approximation theorems underpin the so-called approximation approach (to be introduced in Section 4) and readily yield results for continuous-time Gaussian channels in the
multi-user setting, which will be elaborated in the following sections.

4

The Approximation Approach

Consider the following continuous-time white Gaussian channel with feedback
Z t
Y (t) =
X(s, M, Y0s )ds + B(t), t ≥ 0,

(25)

0

satisfying the power constraint: there exists P > 0 such that for any T , with probability 1,
Z T
X 2 (s, M, Y0s )ds ≤ P T.
(26)
0

As mentioned in Section 1, it is well-known that the capacity of the above channel is P/2
(The same result can be established under alternative power constraints; see, e.g., [38]).
As elaborated in Section 1, when there is no feedback in the channel, i.e., the channel (25)
is actually equivalent to (1), and one can “derive” the non-feedback capacity heuristically
using the conventional sampling approach as in (1)-(4). But this approach is unable to
tackle feedback since an application of the Shannon-Nyquist sampling theorem will destroy
the temporal causality.
In this section, we use our approximation theorems to give an alternative way to “derive”
the capacity of (25), which will be referred to as the approximation approach in the remainder
of the paper. Compared to the sampling approach, the approximation approach can handle
feedback due to the fact the Euler-Maruyama approximation preserves temporal causality.
Below we briefly explain this new approach, which will be further developed and used,
either heuristically or rigorously, in Section 5, where multiple users may be involved in a
communication system.
For fixed T > 0, consider the equidistant sequence ∆n with stepsize δn = T /n. Applying
the Euler-Maruyama approximation (15) to the channel (25) over the time window [0, T ],
we obtain
Z tn,i+1
(n),t
(n)
(n)
Y (tn,i+1 ) = Y (tn,i ) +
X(tn,i , M, Y0 n,i )ds + B(tn,i+1 ) − B(tn,i ).
(27)
tn,i

12

By Theorem 3.4, we have
I(M; Y0T ) = lim I(M; Y (n) (∆n )).

(28)

n→∞

Our strategy is to “establish” the capacity for the discrete-time channel (27) first, and then
the capacity for the continuous-time channel (25) using the “closeness” between the two
channels, as claimed by approximation theorems.
For the converse part, we first note that
I(M; Y

(n)

(∆n )) =
≤

n
X

i=1
n
X

H(Y
H(Y

i=1

(n)

(n)

(tn,i ) − Y

(n)

(tn,i ) − Y

(n)

(n),t
(tn,i−1 )|Ytn,0 n,i−2 )

(tn,i−1 )) −

n
X
i=1

−

n
X
i=1

H(B(tn,i ) − B(tn,i−1 ))

H(B(tn,i ) − B(tn,i−1 )).

It then follows from the fact
V ar(Y (n) (tn,i ) − Y (n) (tn,i−1 )) = E[(Y (n) (tn,i ) − Y (n) (tn,i−1 ))2 ]

= E[δn2 (X (n) (tn,i−1 ))2 ] + E[(B(tn,i ) − B(tn,i−1 ))2 ]

= E[δn2 (X (n) (tn,i−1 ))2 ] + δn ,
that
n

(n)
I(M; Y∆n )

1X
log(1 + δn (X (n) (tn,i ))2 )
≤
2 i=0

(29)

n

≤

1X
δn (X (n) (tn,i ))2 ,
2 i=0

(30)

which, by (28), immediately yields
I(M; Y0T )

1
≤
2

Z

T

0

X 2 (s)ds ≤

PT
,
2

(31)

which establishes the converse part.
For the availability part, note that if we assume all X (n) (tn,i−1 ) are independent of the
Brownian motion B with E[X (n) (tn,i−1 )] = P , then the inequalities in (29) and (30) will
become equalities. The part then follows from a usual random coding argument with codes
generated by the distribution of X (n) (or more precisely, a linear interpolation of X (n) ). It
is clear that as n tends to infinity, the process X (n) behaves more and more like a white
Gaussian process. This observation echoes Theorem 6.4.1 in [38], whose proof rigorously
shows that an Ornstein-Uhlenbeck process that oscillates “extremely” fast will achieve the
capacity of (25).
Roughly speaking, similar to the conventional sampling approach, the above approximation approach establishes a continuous-time Gaussian feedback channel as the limit of the
associated discrete-time channels as the SNR for each channel use shrink to zero proportionately (note that in the above arguments, the SNR for each channel use is δn ). In other
words, we have strengthened the low SNR equivalence in (A) as follows:
13

a continuous-time infinite-bandwidth Gaussian feedback channel is “equivalent”
to a discrete-time Gaussian feedback channel with low SNR.
(B)
We remark however that for the purpose of deriving the capacity of (25) though, the approximation approach, like the conventional sampling approach, is heuristic in nature: Theorem 3.1 does require Conditions (d)-(f), which are much stronger than the power constraint (26). Nevertheless, this approach is of fundamental importance to our treatment
of continuous-time Gaussian channels: as elaborated in Section 5, not only can it channel
the ideas and techniques in discrete time to rigorously establish new results in continuous
time, more importantly, it can also provide insights and intuition for our rigorous treatments
where we will employ established tools and develop new tools in stochastic calculus.

5

Continuous-Time Multi-User Gaussian Channels

Extending Shannon’s fundamental theorems on point-to-point communication channels to
general networks with multiple sources and destinations, network information theory aims
to establish the fundamental limits on information flows in networks and the optimal coding schemes that achieve these limits. The vast majority of researches on network information theory to date have been focusing on networks in discrete time. In a way, this
phenomenon can find its source from Shannon’s original treatment of continuous-time pointto-point channels, where such channels were examined through their associated discrete-time
versions. This insightful viewpoint has exerted major influences on the bulk of the related
literature on continuous-time Gaussian channels, oftentimes prompting a model shift from
the continuous-time setting to the discrete-time one right from the beginning of a research
attempt.
The primary focus of this section is to illustrate the possible applications of the approximation approach: 1) Guided by this approach, we will rigorously derive the capacity
regions of families of continuous-time multi-user one-hop white Gaussian channels, including
continuous-time multi-user white Gaussian multiple access channels (MACs) and broadcast
channels (BCs). To deliver the rigourous proofs of our results, we will directly work within
the continuous-time setting, employing established tools and developing new tools (see Theorems E.1 and F.1) in stochastic calculus to complement the approximation approach. 2)
We can also rigorously apply this approach to examine how feedback affects the capacity
regions of the above-mentioned channels via translations of results and techniques in discrete time. It turns out that some results can be translated from the discrete-time setting to
the continuous-time setting, such as that feedback increases the capacity region of Gaussian
BCs, and that feedback does not increase the capacity region of physically degraded BCs.
Nevertheless, there is a seeming “exception”: as opposed to discrete-time Gaussian MACs,
feedback does not increase the capacity region some of continuous-time Gaussian MACs,
which, somewhat surprisingly, can also be explained by the approximation approach as well.
Below, we summarize the results in this section. To put our results into a relevant context,
we will first list some related results in discrete time, and for obvious reasons, we can only
list those that are most relevant to ours.
Gaussian MACs. When there is no feedback, the capacity region of a discrete-time
memoryless MAC is relatively better understood: a single-letter characterization has been
14

established by Ahlswede [2] and the capacity region of a Gaussian MAC was explicitly derived
in Wyner [76] and Cover [12]. On the other hand, the capacity region of MACs with feedback
still demands more complete understanding, despite several decades of great effort by many
authors: Cover and Leung [13] derived an achievable region for a memoryless MAC with
feedback. In [73], Willems showed that Cover and Leung’s region is optimal for a class of
memoryless MACs with feedback where one of the inputs is a deterministic function of the
output and the other input. More recently, Bross and Lapidoth [9] improved Cover and
Leung’s region, and Wu et al. [74] extended Cover and Leung’s region for the case where
non-causal state information is available at both senders. An interesting result has been
obtained by Ozarow [56], who derived the capacity region of a memoryless Gaussian MAC
with two users via a modification of the Schalkwijk-Kailath scheme [63]; moreover, Ozarow’s
result showed that in general, the capacity region for a discrete memoryless MAC is increased
by feedback.
In Section 5.1, guided by the approximation approach, we first establish Lemma E.1,
a key lemma which roughly says that “other users can be simply treated as noises”, and
we then employ established tools from stochastic calculus to derive the capacity region of a
continuous-time white Gaussian MAC with m senders and with/without feedback. It turns
out that for such a channel, feedback does not increase the capacity region, which, at first
sight, may seem at odds with the aforementioned Ozarow’s result and the conclusion of our
approximation theorems. This however can be roughly explained by the well-known fact that
“>” may become “=” when taking the limit (indeed, an > bn does not necessarily imply
limn→∞ an > limn→∞ bn ); see Remark 5.3 for a more detailed explanation.

Gaussian BCs. The capacity regions of discrete-time Gaussian BCs without feedback
are well known [11, 7]. And it has been shown by El Gamal [18] that feedback cannot increase
the capacity region of a physically degraded Gaussian BC. On the other hand, it was shown
by Ozarow and Leung [57] that feedback can increase the capacity of stochastically degraded
Gaussian BCs, whose capacity regions are far less understood.
In Section 5.2, we first establish a continuous-time version of entropy power inequality
(Theorem F.1) and then derive the capacity region of a continuous-time Gaussian nonfeedback BC with m receivers. Employing the approximation approach, we use a modified
argument in [18] to show that feedback does not increase the capacity region of a physically
degraded continuous-time Gaussian BC, and on the other hand, a translated version of
the argument in [57] to show that feedback does increase the capacity region of certain
continuous-time Gaussian BC.

Here we remark that the above-mentioned capacity results for the non-feedback case
(Theorems 5.1 (on non-feedback capacity) and 5.4) are “long known” in the sense that they
are “predicted” by the conventional sampling approach and their proofs follow from the
usual framework. On the other hand though, explicit formulations and statements of these
results and their rigorous and complete proofs, to the best of our knowledge, do not exist
in the literature. The reason, we believe, is that there are a number of technical difficulties
that one has to overcome to prove such results: Lemma E.1 and Lemma F.1 (which is based
on the I-MMSE relationship that has only been established in [24]) are newly developed in
this work and their proofs are non-trivial.
In contrast, the approximation approach can be applied to continuous-time Gaussian
feedback channels, either heuristically or rigorously. More specifically, it can be heuristically

15

applied to “explain” Theorems 5.1 (on feedback capacity) and 5.4) and give us intuition (as
elaborated in Remark 5.3, it helps “predict” the optimal channel input distribution, which
has been made rigorous in Lemma E.1), and it can also be applied to establish Theorems 5.6
and 5.7.

5.1

Gaussian MACs

Consider a continuous-time white Gaussian MAC with m users, which can be characterized
by
Z t
Z t
Z t
s
s
Y (t) =
X1 (s, M1 , Y0 )ds+
X2 (s, M2 , Y0 )ds+· · ·+
Xm (s, Mm , Y0s )ds+B(t), t ≥ 0,
0

0

0

(32)
where Xi is the channel input from sender i, which depends on Mi , the message sent from
sender i, which is independent of all messages from other senders, and possibly on the
feedback Y0s , the channel output up to time s.
For T, R1 , . . . , Rm , P1 , . . . , Pm > 0, a (T, (eT R1 , . . . , eT Rm ), (P1 , . . . , Pm ))-code for the MAC
(32) consists of m sets of integers Mi = {1, 2, . . . , eT Ri }, the message alphabet for user i,
i = 1, 2, . . . , m, and m encoding functions, Xi : Mi → C[0, T ], which satisfy the following
power constraint: for any i = 1, 2, . . . , m, with probability 1,
Z
1 T 2
Xi (s, Mi , Y0s )ds ≤ Pi ,
(33)
T 0
and a decoding function,
g : C[0, T ] → M1 × M2 × · · · × Mm .
The average probability of error for the above code is defined as
Pe(T ) =

1

Pm
eT ( i=1 Ri )

X

(M1 ,M2 ,...,Mm )∈M1 ×M2 ×···×Mm

P {g(Y0T ) 6= (M1 , M2 , . . . , Mm ) | (M1 , M2 , . . . , Mm ) sent}.

A rate tuple (R1 , R2 , . . . , Rm ) is said to be achievable for the MAC if there exists a sequence
(T )
of (T, (eT R1 , . . . , eT Rm ), (P1 , . . . , Pm ))-codes with Pe → 0 as T → ∞. The capacity region
of the MAC is the closure of the set of all the achievable (R1 , R2 , . . . , Rm ) rate tuples.
The following theorem, whose proof is postponed to Appendix E, gives an explicit characterization of the capacity region of (32).
Theorem 5.1. Whether there is feedback or not, the capacity region of the continuous-time
white Gaussian MAC (32) is
{(R1 , R2 , . . . , Rm ) ∈ Rm
+ : Ri ≤ Pi /2,

i = 1, 2, . . . , m}.

Remark 5.2. When there is no feedback, Theorem 5.1 can be heuristically explained using
the sampling approach as in (2)-(4) (this heuristical approach in this example should be
well-known; see, e.g., Exercise 15.26 in [14]).
16

For simplicity only, we consider the following continuous-time white Gaussian multiple
access channel with two senders:
Y (t) = X1 (t) + X2 (t) + Z(t),

t ∈ R,

(34)

where Xi , i = 1, 2, is the input from the i-th user with average power limit Pi . Similarly as
before, consider its associated discrete-time version corresponding to bandwidth limit ω:
(ω)

(ω)

Yn = X1,n + X2,n + Zn(ω) ,

n ∈ Z.

Then, it is well known [19] that the outer bound on the capacity region can be computed as





P2
P1
2
, R2 ≤ W log 1 +
,
(R1 , R2 ) ∈ R+ : R1 ≤ W log 1 +
2ω
2ω
and the inner bound as


(R1 , R2 ) ∈

R2+



P1
: R1 ≤ ω log 1 +
2ω







P2
P1 + P2
, R2 ≤ ω log 1 +
, R1 + R2 ≤ ω log 1 +
.
2ω
2ω

(Here, it is known [76, 12] that the outer bound can be tightened to coincide with the inner
bound, which, however, is not needed for this example.) It is easy to verify that the two
bounds also collapse into the same region as ω tends to infinity:

(R1 , R2 ) ∈ R2+ : R1 ≤ P1 /2, R2 ≤ P2 /2 ,

which is “expected” to be the capacity region of (34); or alternatively, one can apply the low
SNR equivalence in (A) and take the limit as P tends to 0, reaching the same conclusion.
Note that similar arguments hold for more than two senders as well through a parallel
extension.
Remark 5.3. When the feedback is present in the channel, the approximation approach,
rather than the conventional sampling approach, is necessary to explain Theorem 5.1.
Again, for simplicity only, we consider the following continuous-time Gaussian MAC with
two senders:
Z t
Z t
s
Y (t) =
X1 (s, M1 , Y0 )ds +
X2 (s, M2 , Y0s )ds + B(t), t ≥ 0,
(35)
0

0

with the power constraints: there exist P1 , P2 > 0 such that for all T ,
Z T
Z T
s
2
X1 (s, M1 , Y0 )ds ≤ P1 T,
X22 (s, M2 , Y0s )ds ≤ P2 T.
0

(36)

0

Applying the Euler-Maruyama approximation to the above channel over the time window
[0, T ] with respect to the equidistant ∆n with δn = T /n, we obtain
Z tn,i+1
Z tn,i+1
(n),tn,i
(n),t
(n)
(n)
Y (tn,i+1 ) = Y (tn,i )+
X1 (s, M1 , Y0
)ds+
X2 (s, M2 , Y0 n,i )ds+B(tn,i+1)−B(tn,i ).
tn,i

tn,i

17

Now, straightforward computations and a usual concavity argument then yields that for
large n,
I(M1 ; Y (n) (∆n )|M2 ) = H(Y (n) (∆n )|M2 ) − H(Y (n) (∆n )|M1 , M2 )
n
n
X
X
(n),t
(n),tn,i−1
(n)
H(Y (n) (tn,i )|Ytn,0 n,i−1 , M1 , M2 )
H(Y (tn,i )|Ytn,0
, M2 ) −
=
i=1

i=1

≤
≤
=
≤
≈
≤

n
X
i=1



1
log E
2

n
X
1

i=1
n
X

2

log

i=1 i=1
n
n X
X
i=1 i=1

Z

tn,i
tn,i−1

tn,i

tn,i−1

1
log
2

i=1
n X
n
X

Z

Z

(n),tn,i−1

X1 (s, M1 , Y0

(n),t
EX1 (s, M1 , Y0 n,i−1 )2 ds

tn,i

Z

tn,i

1
2

Z

tn,i

tn,i−1

tn,i−1

!

(n),tn,i−1 2

tn,i−1

1
2

)ds

!2

EX1 (s, M1 , Y0



+ δn  −

δn + δn

!

1
log(δn )
2
−

1
log(δn )
2

!

) ds + 1

(n),tn,i−1 2

EX1 (s, M1 , Y0

) ds

EX1 (s, M1 , Y0s )2 ds

P1 T
.
2

A completely parallel argument will yield that
I(M2 ; Y (n) (∆n )|M1 ) ≤

P2 T
.
2

It then follows from Theorem 3.1 the region below give an outer bound of the capacity region:
{(R1 , R2 ) : 0 ≤ R1 ≤ P1 /2,

0 ≤ R2 ≤ P2 /2}.

(37)

To see that this outer bound can be achieved, set X1 (s), X2 (s), tn,i ≤ s ≤ tn,i+1 , in (35)
to be independent Gaussian random variables with variances P1 , P2 , respectively. Then, one
verifies that for large n,
I(M1 ; Y (∆n )) = H(Y (∆n )) − H(Y (∆n )|M1 )
n
n
X
X
H(Y (tn,i )|M1 )
=
H(Y (tn,i )) −
i=1

=
=

i=1

n
X
1
i=1
n
X
i=1

2

log(P1 δn2 + P2 δn2 + δn ) −



1
P1 δn2
log 1 +
2
P2 δn2 + δn

≈ P1 T /2,

1
log(P2 δn2 + δn )
2

(38)
(39)
(40)

18

where we have used the fact that δn is “close” to 0 for large enough n for (40), which, parallelly
as in Remark 5.2, can be alternatively explained by taking P1 to 0 and then applying the
low SNR equivalence in (B). With a similar argument, one can prove that
I(M2 ; Y (∆n )) ≈ P2 T /2.
It then follows that the outer bound in (37) can be achieved.
Here we remark that similarly as in Section 4, for n large enouch, the constructed processes X1 and X2 behave like “fast-oscillating” Ornstein-Uhlenbeck processes, and moreover,
from (38) and (39), one can tell that for one user to achieve the maximum transmission rate,
the other user can simply be ignored. Predicting the optimal channel input, these facts echo
Remark E.2 and give another explanation to Lemma E.1, a key lemma in our rigourous proof
of Theorem 5.1 in Appendix E.

5.2

Gaussian BCs

In this section, we consider a continuous-time white Gaussian BC with m receivers, which
is characterized by: for i = 1, 2, . . . , m,
Z t
√
X(s, M1 , M2 , . . . , Mm )ds + Bi (t), t ≥ 0,
(41)
Yi (t) = snri
0

where X is the channel input, which depends on Mi , the message sent from sender i, which
is uniformly distributed over a finite alphabet Mi and independent of all messages from
other senders, snri is the signal-to-noise ratio in the channel for user i, Bi (t) are (possibly
correlated) standard Brownian motions.
For T, R1 , R2 , . . . , Rm , P > 0, a (T, (eT R1 , . . . , eT Rm ), P )-code for the BC (41) consists of
m set of integers Mi = {1, 2, . . . , eT Ri }, the message set for receiver i, i = 1, 2, . . . , m, and
an encoding function, X : M1 × M2 × · · · × Mm → C[0, T ], which satisfies the following
power constraint: with probability 1,
Z
1 T 2
X (s, M1 , M2 , . . . , Mm )ds ≤ P,
(42)
T 0

Pe(T )

and m decoding functions, gi : C[0, T ] → Mi , i = 1, 2, . . . , m.
The average probability of error for the (T, (eT R1 , eT R2 , . . . , eT Rm ), P )-code is defined as
X
1
P {gi (Y0T ) 6= Mi , i = 1, 2, . . . , m | (M1 , M2 , . . . , Mm ) sent}.
= T (Pm R )
i=1 i
e
(M1 ,M2 ,...,Mm )∈M1 ×M2 ×···×Mm

A rate tuple (R1 , R2 , . . . , Rm ) is said to be achievable for the BC if there exists a sequence
(T )
of (T, (eT R1 , eT R2 , . . . , eT Rm ), P )-codes with Pe → 0 as T → ∞. The capacity region of
the BC is the closure of the set of all the achievable (R1 , R2 , . . . , Rm ) rate tuples.
The following theorem explicitly characterizes the capacity region of the above BC, whose
proof is postponed to Appendix F.
Theorem 5.4. The capacity region of the continuous-time white Gaussian BC (41) is


R2
Rm
P
R1
m
+
+···+
≤
.
(R1 , R2 , . . . , Rm ) ∈ R+ :
snr1 snr2
snrm
2
19

Remark 5.5. Theorem 5.4 can be heuristically derived using a similar argument employing
the approximation approach as in Remark 5.3.
We are now ready to use the approximation approach to analyze the effects of feedback
on continuous-time Gaussian BCs.
The following theorem says that feedback does not help physically degraded Gaussian
BCs, whose proof is inspired by the ideas in Section 4 and parallels the argument in [18].
Theorem 5.6. Consider the following continuous-time physically degraded Gaussian broadcast channel with one sender and two receivers:
Z t
p
s
s
Y1 (t) =
X(s, M1 , Y1,0
, Y2,0
)ds + N1 B1 (t),
0

Y2 (t) =

t

Z

0

s
s
X(s, M2 , Y1,0
, Y2,0
)ds +

p
p
N1 B1 (t) + N2 B2 (t),

where N1 , N2 > 0, and B1 , B2 are independent standard Brownian motions, and the channel
input X(s) is assumed to satisfy Conditions (d)-(f ). Then, feedback does not increase the
capacity region of the above channel.
Proof. Let X be a (T, (eT R1 , eT R2 ), P )-code. By the code construction, for i = 1, 2, it is
T
possible to estimate the messages Mi from the channel output Yi,0
with an arbitrarily low
probability of error. Hence, by Fano’s inequality, for i = 1, 2,
T
H(Mi |Yi,0
) ≤ T Ri Pe(T ) + H(Pe(T ) ) = T εi,T ,

where εi,T → 0 as T → ∞. It then follows that
T
T R1 = H(M1 ) = H(M1 |M2 ) ≤ I(M1 ; Y1,0
|M2 ) + T ε1,T ,
T
T R2 = H(M2 ) ≤ I(M2 ; Y2,0
) + T ε2,T .

Now the Euler-Maruyama approximation with respect to the equidistant ∆n of stepsize
δn = T /n applied to the continuous-time physically degraded Gaussian BC yields:
(n)
Y1 (tn,i )

(n)

(n)

(n)
− Y1 (tn,i−1 )

Y2 (tn,i )−Y2 (tn,i−1 ) =

Z

tn,i
tn,i−1

=

Z

tn,i

tn,i−1

(n),t

(n),t

X(s, M, Y1,tn,0n,i−1 , Y2,tn,0n,i−1 )ds +

(n),t

(n),t

X(s, M, Y1,tn,0n,i−1 , Y2,tn,0n,i−1 )ds+

Then, by Theorem 3.1, we have

p

p

N1 B1 (tn,i )−

N1 B1 (tn,i ) −

p

(n)

n→∞

(n)

= lim I(M2 ; ∆Y2 (∆n ))
n→∞

(n)

(n)

= lim H(∆Y2 (∆n )) − H(∆Y2 (∆n )|M2 ),

20

N1 B1 (tn,i−1 ),

N1 B1 (tn,i−1 )+

T
I(M2 ; Y2,0
) = lim I(M2 ; Y2 (∆n ))

n→∞

p

p

N2 B2 (tn,i )−

p

N2 B2 (tn,i−1 ).

(n)

(n)

(n)

where ∆Y2 (∆n ) , {Y2 (tn,i ) − Y2 (tn,i−1 ) : i = 1, 2, · · · , n}. Note that
(n)
H(∆Y2 (∆n ))

≤

n
X

log(2πe(P δn2 + N2 δn )),

i=1

and
(n)
H(∆Y2 (∆n )|M2 )

=
≥
=

n
X

i=1
n
X

(n)

(n),t

p
p
H( N2 B2 (tn,i ) − N2 B2 (tn,i−1 ))

i=1

n
X

(n)

H(Y2 (tn,i ) − Y2 (tn,i−1)|Y2,tn,0n,i−1 , M2 )

log(2πeN2 δn ),

i=1

which implies that there exists an α ∈ [0, 1] such that
(n)

H(∆Y2 (∆n )|M) =

n
X
n
i=1

2

log(2πe(αP δn2 + N2 δn )).

It then follows from Theorem 3.1 that
n

T
I(M2 ; Y2,0
)

X
P δn2 + N2 δn
(1 − α)P T
1
log
=
.
≤ lim
2 n→∞ i=1
αP δn2 + N2 δn
2N2

Next we consider
(n)

(n)

(n)

I(M1 ; Y1 (∆n )|M2 ) = H(Y1 (∆n )|M2 ) − H(Y1 (∆n )|M1 , M2 )
n
X
(n),t
(n)
(n)
= H(Y1 (∆n )|M2 ) −
H(Y1 (tn,i )|M1 , M2 , Y1,tn,0n,i−1 )
i=1

(n)

≤ H(Y1 (∆n )|M2 ) −

n
X

(n)

(n),t

(n),t

H(Y1 (tn,i )|M1 , M2 , Y1,tn,0n,i−1 , Y1,tn,0n,i−1 )

i=1

n

(n)

= H(Y1 (∆n )|M2 ) −

1X
log(2πeN1 δn ).
2 i=1

Now, using Lemma 1 in [18] (an extension of the entropy power inequality), we obtain
H(Y (n) (∆n )|M2 ) ≥

(n)
n
log(22H(Y1 (∆n ))|M2 )/n + 2πe(N2 − N1 )δn ),
2

which immediately implies that
n

(n),t
H(Y1,tn,0n,i |M2 )

1X
log(2πe(αP δn2 + N1 δn ))
≤
2 i=1
21

and furthermore, by Theorem 3.1,
n

n

1X
1X
log(2πe(αP δn2 + N1 δn )) −
log(2πeN1 δn )
n→∞ 2
2
i=1
i=1


n
X
αP δn
1
log 1 +
= lim
n→∞ 2
N1
i=1

T
I(M1 ; Y1,0
|M2 ) ≤ lim

=

αP T
.
2N1

Now, by Theorem 5.4, we conclude that feedback capacity region is exactly the same nonfeedback capacity region; in other words, feedback does not increase the capacity region of
a physically degraded continuous-time Gaussian BC.
The following theorem says that feedback does help some stochastically degraded Gaussian BCs, whose proof, instead of directly employing the approximation theorem, uses the
connections between continuous-time and discrete-time Gaussian channels and the notion
of continuous-time directed information in Remark 3.6, both of which can find their source
from the approximation theorem. We only provide the sketch of the proof, since it is largely
based on a translated version of the argument in [57].
Theorem 5.7. Feedback increases the capacity region of certain continuous-time stochastically degraded Gaussian broadcast channel.
Proof. Consider the following symmetric continuous-time Gaussian broadcast channel:
Z t
Y1 (t) =
X(s)ds + B1 (t),
0

Y2 (t) =

Z

t

X(s)ds + B2 (t),

0

where B1 , B2 are independent standard Brownian motions, and X satisfies the average power
constraint P . By Theorem 5.4, without feedback, the capacity region is the set of rate pairs
(R1 , R2 ) such that
P
(43)
R1 + R2 ≤ .
2
With feedback, one can use the following variation [57] of the Schalkwijk-Kailath coding
scheme [63] over [0, T ] at discrete time points {tn,i } that form an equidistant ∆n of stepsize
δn : For the channel input, after some proper initialization, at time t ∈ [tn,i , tn,i+1 ), we send
(n)
(n)
X (n) (t) = X1 (t) + X2 (t), where
(n)

(n)

(n)

(n)

X1 (t) = γi (X1 (tn,i−1 ) − E[X1 (tn,i−1 )|Y1 (tn,i−1 )]),
(n)

(n)

(n)

(n)

X2 (t) = −γi (X2 (tn,i−1 ) − E[X2 (tn,i−1 )|Y2 (tn,i−1 )]),

where γi is chosen so that E[Xi2 (t)] = P for each i; and for the channel outputs, we have,
for any t ∈ [tn,i , tn,i+1 ],
Z t
(n)
(n)
Y1 (t) = Y1 (tn,i ) +
X (n) (tn,i )ds + B1 (t) − B1 (tn,i ),
(44)
tn,i

22

and
(n)
Y2 (t)

=

(n)
Y2 (tn,i )

+

Z

t

tn,i

X (n) (tn,i )ds + B2 (t) − B2 (tn,i ).

(45)

Going through a completely parallel argument as in [57] and capitalizing on the fact that
the SNR in the channels (44) and (45) tend to 0 as n tends to infinity, we derive that


P δn (1+ρ∗ )/2
n
P (1 + ρ∗ )
1 X X log 1 + (1+P δn (1−ρ∗ ))/2
(n)
=
,
lim I(X (n) (∆n ) → Y1 (∆n )) =
n→∞
2 n→∞ i=1
T
4
and parallelly,
(n)

lim I(X (n) (∆n ) → Y2 (∆n )) =

n→∞

P (1 + ρ∗ )
,
4

∗

where ρ > 0 satisfies the condition
P (P + 2)(1 − ρ∗ )
.
2

ρ∗ (1 + (P + 1)(1 + P (1 − ρ∗ )/2)) =
Note that, by Remark 3.6, we have
(n),T

I(X0

(n),T

→ Y1,0

(n)

) ≥ I(X (n) (∆n ) → Y1 (∆n )),

(n),T

I(X0

(n),T

→ Y2,0

(n)

) ≥ I(X (n) (∆n ) → Y2 (∆n )),

which immediately implies that
R1 = R2 =

P (1 + ρ∗ )
4

(46)

are achievable. The claim that feedback strictly increases the capacity region then follows
from (43), and (46) and the fact that ρ∗ > 0.

6

Conclusions and Future Work

For a continuous-time white Gaussian channel without feedback, the classical ShannonNyquist sampling theorem can convert it to a discrete-time Gaussian channel, however such
a link has long been missing when feedback/memory is present in the channel. In this paper,
we establish sampling and approximation theorems as the missing links, which we believe
will play important roles in the long run for further developing continuous-time information
theory, particularly for the communication scenarios where feedback/memory is present.
As an immediate application of our approximation theorem, we propose the approximation approach, an analog of the conventional sampling approach, for Gaussian feedback
channels. It turns out that, like its non-feedback counterpart, the approximation approach
can bring insights and intuition to investigation of continuous-time Gaussian channels with
possible feedback, and moreover, when complemented with relevant tools from stochastic
calculus, can deliver rigorous treatments in the point-to-point or multi-user setting.
On the other hand though, there are many questions that remain unanswered and a
number of directions that need to be further explored. Below we list a number of research
directions that look promising in the near future.
23

1) The first direction is to strengthen and generalize our sampling and approximation
theorems.
Note that both Theorem 2.3 and Theorem 3.1 require Conditions (d)-(f), which are
stronger than the typical average power constraint. While Conditions (d)-(f) are rather
mild for practical considerations, the stronger assumptions in our theorems will narrow their
reach in some theoretical situations. For instance, despite the fact that our approximation
theorem gives intuitive explanations to the rigorous treatment of continuous-time multi-user
Gaussian channels in Section 5, it fails to rigorously establish Theorems 5.1 and 5.4. The
stochastic calculus approach employed in Section 5 requires only the power constraints, which
can be loosely explained by the fact that Girsanov’s theorem (or, more precisely, its several
variants) only requires as weak conditions. It is certainly worthwhile to explore whether the
assumptions in our sampling and approximation theorems can be relaxed either in general
or for some special settings.
Another topic in this direction is the rate of convergence in the sampling and approximation theorems. While the current versions of our theorems have merely established some
limits, the rate of convergence will certainly yield a more quantitative description of how
fast those limits will be approached.
One can also consider generalizing these two theorems to general Gaussian channels [35,
29]. For this topic, note that there exist in-depth studies [30, 41, 31, 36, 40, 37, 6, 39] on
continuous-time point-to-point general Gaussian channels with possible feedback, for which
information-theoretic connections with the discrete-time setting are somehow lacking. A first
step in this direction can be establishing sampling or approximation theorems for stationary
Gaussian processes. Obviously, such theorems for stationary Gaussian processes can connect
continuous-time stationary Gaussian channels to their discrete-time counterparts, for which
the variational formulation of discrete-time stationary Gaussian feedback capacity in [46]
proves to rather effective.
2) The second direction is to further explore the possible applications of our sampling
and approximation theorems in the following respects.
We have shown that feedback may increase the capacity region of some continuoustime Gaussian BC, but the capacity regions of such channels remain unknown in general.
An immediate problem is to explicitly find the exact capacity regions of continuous-time
Gaussian BCs using the approach employed in this work, as we have done for continuoustime Gaussian MACs. Of course, further topics also include exploring whether the ideas
and techniques in this paper can be applied to other families of continuous-time multi-user
Gaussian channels with possible feedback.
So far we have implicitly assumed infinite bandwidth and average power constraints, but
our theorems can certainly go beyond these assumptions. For instance, one can consider
examining continuous-time Gaussian channels with both bandwidth limit and peak power
constraint, which are more reasonable assumptions for many practical communication scenarios as they give a more accurate description of the limitations of the communication
system. Little is known about the capacity of continuous-time Gaussian channels with such
constraints except some upper and lower bounds established in [58, 64]. In stark contrast,
discrete-time peak power constrained channels (including, but not limited to Gaussian channels) have been better investigated: there has been a series of work on their capacity, such
as [70, 65, 1, 10, 61, 68, 20, 17], which feature relatively thorough discussions about different

24

aspects of channel capacity including capacity achieving distribution, bounds and asymptotics of capacity, and numerical computation of capacity. An immediate question is to
explore whether the approximation approach can translate the aforementioned existing results in discrete time, or more probably, help channel the ideas and techniques therein to
the continuous-time setting. A next question is to explore whether there exists any randomized algorithm for computation of the capacity of such a channel, for which, as discussed in
Remark 3.8, we believe our sampling theorems can be particularly helpful in terms of numerically computing and optimizing the mutual information of a continuous-time Gaussian
channel with bandwidth limit and peak power constraint.
Acknowledgement. We would like to thank Ronit Bustin, Jun Chen, Young-Han
Kim, Haim Permuter, Shlomo Shamai, Tsachy Weissman and Wenyi Zhang for insightful
suggestions and comments, and for pointing out relevant references.

Appendices
A

Proof of Theorem 2.1

First of all, an application of Theorem 7.14 of [51] with Conditions (b) and (c) yields that
Z T

2
t
t
t
P
E [g(t, W0 , Y0 )|Y0 ]dt < ∞ = 1.
(47)
0

Then one verifies that the assumptions of Lemma 7.7 of [51] are all satisfied (this lemma is
stated under very general assumptions, which are exactly Conditions (b), (c) and (47) when
restricted to our settings), which implies that for any w,
µY ∼ µY |W =w ∼ µB ,
where “∼” is the standard notation for two measures being equivalent (i.e., one is absolutely
continuous with respect to the other and vice versa), and moreover, with probability 1,
dµY |W T
1
(Y0 ) =
,
RT
R
1 T
2
dµB
E[e− 0 g(s)dY (s)+ 2 0 g(s) ds |Y0T , W0T ]

1
dµY
(Y0T ) =
,
RT
R
1 T
2
dµB
E[e− 0 g(s)dY (s)+ 2 0 g(s) ds |Y0T ]
(48)

where
we have rewritten
g(s, W0s, Y0s ) as g(s) for notational
simplicity.
Here we remark that
R
R
R
RT
1 T
2
T
T
− 0T g(s)dY (s)+ 21 0T g(s)2 ds
− 0 g(s)dY (s)+ 2 0 g(s) ds
|Y0 , W0 ] is in fact equal to e
, but we keep it
E[e
the way it is as above for an easy comparison.
Note that it follows from E[dµB /dµY (Y0T )] = 1 that
E[e−

RT
0

g(s)dY (s)+ 12

RT
0

g(s)2 ds

] = 1,

E[e−

RT

g(s)dB(s)− 12

RT

g(s)2 ds

] = 1.

which is equivalent to

0

25

0

Then, a parallel argument as in the proof of Theorem 7.1 of [51] further implies that for any
∆n , with probability 1,
dµY |W
1
(Y (∆n )) =
,
R
RT
1 T
2
dµB
E[e− 0 g(s)dY (s)+ 2 0 g(s) ds |Y (∆n ), W0T ]

1
dµY
(Y (∆n )) =
,
R
RT
1 T
2
dµB
E[e− 0 g(s)dY (s)+ 2 0 g(s) ds |Y (∆n )]
(49)

where we have defined
B(∆n ) , {B(tn,0 ), B(tn,1 ), . . . , B(tn,n )},
and moreover,
dµY |W
dµY (∆n )|W
(Y (∆n )),
(Y (∆n )) ,
dµB
dµB(∆n )

dµY (∆n )
dµY
(Y (∆n )).
(Y (∆n )) ,
dµB
dµB(∆n )

Then, by definition, we have
I(W0T ; Y





dµY |W
dµY
(∆n )) = E log
(Y (∆n )) − E log
(Y (∆n )) .
dµB
dµB
RT

1

RT

2 ds

Notice that it can be easily checked that e− 0 g(s)dY (s)+ 2 0 g(s)
with the fact that ∆n ⊂ ∆n+1 for all n, further implies that
E[e−

RT
0

g(s)dY (s)+ 12

RT
0

g(s)2 ds

|Y (∆n ), W0T ],

E[e−

RT
0

integrable, which, together

g(s)dY (s)+ 12

RT
0

g(s)2 ds

|Y (∆n )]

are both martingales, and therefore, by Doob’s martingale convergence theorem [16],
dµY |W
dµY |W T
(Y (∆n )) →
(Y0 ),
dµB
dµB

dµY
dµY T
(Y (∆n )) →
(Y ), a.s.
dµB
dµB 0

Now, by Jensen’s inequality, we have
 Z T

Z
R
RT
1 T
1 T
2
2
T
E −
g(s)dY (s) +
g(s) ds Y (∆n ), W0 ≤ log E[e− 0 g(s)dY (s)+ 2 0 g(s) ds |Y (∆n ), W0T ],
2 0
0
(50)
and, by the fact that log x ≤ x for any x > 0, we have
log E[e−

RT
0

g(s)dY (s)+ 12

RT
0

g(s)2 ds

|Y (∆n ), W0T ] ≤ E[e−

RT
0

g(s)dY (s)+ 21

RT
0

g(s)2 ds

|Y (∆n ), W0T ]. (51)

It then follows from (50) and (51) that
−

log E[e

RT
0

g(s)dY (s)+ 21

RT
0

g(s)2 ds

|Y

(∆n ), W0T ]



≤ E −
+ E[e−

Z

RT
0

T
0

1
g(s)dY (s) +
2

g(s)dY (s)+ 12

RT
0

g(s)2 ds

Z

T
2

g(s) ds Y

0

(∆n ), W0T

|Y (∆n ), W0T ].

Applying the general Lebesgue dominated convergence theorem (see, e.g., Theorem 19 on
Page 89 of [62]), we then have




R
R
dµY |W
dµY |W T
T
T
− 0T g(s)dY (s)+ 21 0T g(s)2 ds
lim E log
(Y (∆n )) = E[log E[e
|Y0 , W0 ]] = E log
(Y0 ) .
n→∞
dµB
dµB
26



A completely parallel argument yields that




R
R
dµY T
dµY
T
− 0T g(s)dY (s)+ 12 0T g(s)2 ds
|Y0 ]] = E log
lim E log
(Y (∆n )) = E[log E[e
(Y ) .
n→∞
dµB
dµB 0
So, with the definition
I(W0T ; Y0T )




dµY |W T
dµY T
(Y0 ) − E log
(Y ) ,
= E log
dµB
dµB 0


we conclude that
lim

n→∞

B

I(W0T ; Y




dµY |W T
dµY T
(∆n )) = E log
(Y0 ) − E log
(Y ) = I(W0T ; Y0T ).
dµB
dµB 0


Proof of Lemma 2.2

With Conditions (d)-(f), the proof of the existence and uniqueness of the solution to (6) is
somewhat standard; see, e.g., Section 5.4 in [53]. So, in the following, we will only prove
(10).
For the stochastic differential equation (6), applying Condition (e), we deduce that there
exists L1 > 0 such that
Z T
T
kY0 k ≤
L1 (1 + kW0t k + kY0t k)dt + kB0T k
0
Z T
T
T
≤ L1 T + L1 T kW0 k + kB0 k +
L1 kY0t kdt.
0

Then, applying the Gronwall inequality followed by a straightforward bounding analysis, we
deduce that there exists L2 > 0 such that
RT

kY0T k ≤ (L1 T + L1 T kW0T k + kB0T k)e

0

L1 dt

= eL1 T (L1 T + L1 T kW0T k + kB0T k)

= L2 + L2 kW0T k + L2 kB0T k.

Now, for any ε > 0, applying Doob’s submartingale inequality, we have
T k2

E[eεkY0

T

2

T

] ≤ E[eε(L2 +L2 kW0 k+L2 kB0 k) ]
2

2

T 2 +L2 kB T k2 )
2
0

≤ E[e3ε(L2 +L2 kW0 k
3εL22

=e

3εL22

=e

]E[e

3εL22 kW0T k2

]E[sup0≤t≤T e3εL2 B(t) ]

E[e
E[e
2

]

3εL22 kW0T k2

2

3εL22 kB0T k2

T 2

]
2

2

2

≤ 4e3εL2 E[e3εL2 kW0 k ]E[e3εL2 B(T ) ],
which, by Condition (f), is finite provided that ε is small enough.

27

2

C

Proof of Theorem 2.3

We proceed in the following steps.
Step 1. In this step, we establish the theorem assuming that there exists C > 0 such
that for all w0T ∈ C[0, T ] and all y0T ∈ C[0, T ],
Z T
(52)
g 2(s, w0s , y0s )ds < C.
0

By the definition of mutual information, (48) and (49), we have
I(W0T ; Y




dµY |W
dµY
(Y (∆n )) − E log
(Y (∆n ))
(∆n )) = E log
dµB
dµB


RT

1

= −E[log E[e− 0 g(s)dY (s)+ 2
= −E[Fn ] + E[Gn ],

RT

g(s)2 ds

0

|Y (∆n ), W0T ]] + E[log E[e−

RT
0

g(s)dY (s)+ 12

where, for notational simplicity, we have rewritten g(s, W0s, Y0s ) as g(s).
Step 1.1. In this step, we prove that as n tends to infinity,
Z T
Z
1 T
Fn → −
g(s)dY (s) +
g(s)2 ds,
2
0
0

RT
0

g(s)2 ds

|Y (∆n )]]

(53)

in probability.
Let Ȳ∆Tn ,0 denote the piecewise linear version of Y0T with respect to ∆n ; more precisely,
for any i = 0, 1, . . . , n, Ȳ∆n (tn,i ) = Y (tn,i ), and for any tn,i−1 < s < tn,i with s = λtn,i−1 +
(1 − λ)tn,i for some 0 < λ < 1, Ȳ∆n (s) = λY (tn,i−1 ) + (1 − λ)Y (tn,i ). Let ḡ∆n (s, W, Ȳ∆s n ,0 )
denote the piecewise “flat” version of g(s, W, Ȳ∆s n,0 ) with respect to ∆n ; more precisely, for
t
t
).
any tn,i−1 ≤ s < tn,i , ḡ∆n (s, W0s , Ȳ∆s n,0 ) = g(tn,i−1, W0 n,i−1 , Ȳ∆n,i−1
n ,0
s
s
Rewriting ḡ∆n (s, W0 , Ȳ∆n,0 ) as ḡ∆n (s), we have
Fn = − log E[e−

= − log E[e−

RT
0

RT

RT

0

g(s)dY (s)+ 12

RT

ḡ∆n (s)dY (s)+ 12
1

g 2 (s)ds

0

RT

RT

0

|Y (∆n ), W0T ]

2 (s)ds−
ḡ∆
n

2

RT
0

(g(s)−ḡ∆n (s))dY (s)+ 12

RT

RT
0

1

2 (s))ds
(g 2 (s)−ḡ∆
n

RT

2

|Y (∆n ), W0T ]

= − log e− 0 ḡ∆n (s)dY (s)+ 2 0 ḡ∆n (s)ds E[e− 0 (g(s)−ḡ∆n (s))dB(s)− 2 0 (g(s)−ḡ∆n (s)) (s)ds |Y (∆n ), W0T ]
Z
Z T
RT
R
1 T
1 T 2
2
ḡ∆n (s)ds − log E[e− 0 (g(s)−ḡ∆n (s))dB(s)− 2 0 (g(s)−ḡ∆n (s)) ds |Y (∆n ), W0T ],
=−
ḡ∆n (s)dY (s) −
2 0
0

where we have used the fact that
E[e−

RT
0

ḡ∆n (s)dY (s)+ 12

RT
0

2 (s)ds
ḡ∆
n

|Y (∆n ), W0T ] = e−

RT
0

ḡ∆n (s)dY (s)+ 12

RT
0

2 (s)ds
ḡ∆
n

,

since ḡ∆n (s) is a function depending only on W0T and Y (∆n ).
We now prove the following convergence:
" Z
2 #
  Z T
Z
Z
T
1 T 2
1 T 2
ḡ (s)ds − −
g (s)ds
→ 0,
g(s)dY (s) −
E
−
ḡ∆n (s)dY (s) −
2 0 ∆n
2 0
0
0
(54)
28

which will imply that
Z
Z
Z T
Z T
1 T 2
1 T 2
ḡ (s)ds → −
g (s)ds
g(s)dY (s) −
−
ḡ∆n (s)dY (s) −
2 0 ∆n
2 0
0
0
in probability. Apparently, to prove (54), we only need to prove that
" Z
2 #
Z
T
1 T
E −
(g(s) − ḡ∆n (s))dB(s) −
→ 0.
(g(s) − ḡ∆n (s))2 ds
2
0
0

(55)

To establish (55), notice that, by the Itô isometry [55], we have
"Z
2 #
Z T

T
2
E
=E
(g(s) − ḡ∆n (s))dB(s)
(g(s) − ḡ∆n (s)) ds ,
0

0

which means we only need to prove that as n → ∞,
Z T

2
E
(g(s) − ḡ∆n (s)) ds → 0.

(56)

0

To see this, we note that, by Conditions (d) and (e), there exists L1 > 0 such that for any
s ∈ [0, T ] with tn,i−1 ≤ s < tn,i ,
|g(s, W0s, Ȳ∆s n ,0 ) − ḡ∆n (s, W0s , Ȳ∆s n,0 )|

t

t

= |g(s, W0s, Ȳ∆s n,0 ) − g(tn,i−1, W0 n,i−1 , Ȳ∆n,i−1
)|
n ,0
t

t

≤ L1 (|s − tn,i−1 | + kW0s − W0 n,i−1 k + kȲ∆s n ,0 − Ȳ∆n,i−1
k)
n ,0
t

≤ L1 (|s − tn,i−1 | + kW0s − W0 n,i−1 k + |Y (tn,i ) − Y (tn,i−1 )|)
≤ L1 δ∆n + L1 supr∈[tn,i−1 ,tn,i ] |W (r) − W (tn,i−1)|

+ L1 δ∆n + L1 δ∆n kW0T k + L1 δ∆n kY0T k + |B(tn,i ) − B(tn,i−1 )|.

(57)
(58)

Moreover, by Lemma 2.2 and Condition (f), both kY0T k and kW0T k are quadratically integrable. And furthermore, by Condition (f), we deduce that for any tn,i−1 ≤ s < tn,i ,
E[supr∈[tn,i−1 ,tn,i ] (W (r) − W (tn,i−1 ))2 ] ≤ L2 δ∆n ,

(59)

for some L2 > 0, and by the Itô isometry, we deduce that
E[(B(tn,i ) − B(tn,i−1 ))2 ] = (tn,i − tn,i−1 ) ≤ δ∆n .

(60)

It can be readily checked that (58), (59) and (60) imply (56), which in turn implies (54), as
desired.
We now prove that as n tends to infinity,
E[|E[e−

RT
0

(g(s)−ḡ∆n (s))dB(s)− 12

RT
0

(g(s)−ḡ∆n (s))2 ds

|Y (∆n ), W0T ] − 1|] → 0,

which will imply that
log E[e−

RT
0

(g(s)−ḡ(s))dB(s)− 12

RT
0

(g(s)−ḡ(s))2 ds

29

|Y (∆n ), W0T ] → 0

(61)

in probability and furthermore (53). To establish (61), we first note that
E[|E[e−

RT
0

(g(s)−ḡ∆n (s))dB(s)− 12

≤ E[E[|e−

RT
0

RT
0

(g(s)−ḡ∆n (s))2 ds
RT

(g(s)−ḡ∆n (s))dB(s)− 12

RT

1

0

|Y (∆n ), W0T ] − 1|]

(g(s)−ḡ∆n (s))2 ds

RT

2

− 1||Y (∆n ), W0T ]]

= E[|e− 0 (g(s)−ḡ∆n (s))dB(s)− 2 0 (g(s)−ḡ∆n (s)) ds − 1|]

 Z T
Z
R
R
1 T
2
|− 0T (g(s)−ḡ∆n (s))dB(s)− 12 0T (g(s)−ḡ∆n (s))2 ds|
(g(s) − ḡ∆n (s)) ds e
(g(s) − ḡ∆n (s))dB(s) −
≤E −
2 0
" Z0
#
Z
2
i
h
T
R
RT
1 T
1 T
2
≤E −
(g(s) − ḡ∆n (s))dB(s) −
E e2|− 0 (g(s)−ḡ∆n (s))dB(s)− 2 0 (g(s)−ḡ∆n (s)) ds| .
(g(s) − ḡ∆n (s))2 ds
2 0
0

By (54), we have that as n tends to infinity,
" Z
#
Z
2
T
1 T
E −
(g(s) − ḡ∆n (s))dB(s) −
→ 0.
(g(s) − ḡ∆n (s))2 ds
2
0
0
It then follows that, to prove (61), we only need to prove that if δ∆n is small enough,
h
i
RT
R
1 T
2
E e2|− 0 (g(s)−ḡ∆n (s))dB(s)− 2 0 (g(s)−ḡ∆n (s)) ds| < ∞.
(62)
Since

h
i
R
R
2|− 0T (g(s)−ḡ∆n (s))dB(s)− 12 0T (g(s)−ḡ∆n (s))2 ds|
E e
i
h RT
i
h
R
R
R
2( 0 (g(s)−ḡ∆n (s))dB(s)+ 12 0T (g(s)−ḡ∆n (s))2 ds)
2(− 0T (g(s)−ḡ∆n (s))dB(s)− 12 0T (g(s)−ḡ∆n (s))2 ds)
+E e
,
≤E e
(63)
we only have to prove that the two terms in the above upper bound are both finite provided that δ∆n is small enough. Note that for the first term, applying the Cauchy-Schwarz
inequality, we have
E[e2(−

RT
0

(g(s)−ḡ∆n (s))dB(s)− 12

RT
0

(g(s)−ḡ∆n (s))2 ds)

RT

] = E[e

0

2(g(s)−ḡ∆n (s))dB(s)−

RT

≤ E[e

0

RT

4(g(s)−ḡ∆n (s))dB(s)−

0

RT
0

4(g(s)−ḡ∆n (s))2 ds+3
8(g(s)−ḡ∆n (s))2 ds

RT
0

(g(s)−ḡ∆n (s))2 ds

]E[e6

RT
0

(g(s)−ḡ∆n (s))2 ds

It is well known that an application of Fatou’s lemma yields that
RT

E[e

0

4(g(s)−ḡ∆n (s))dB(s)−

RT

8(g(s)−ḡ∆n (s))2 ds

0

] ≤ 1,

(64)

and by (58), we deduce that there exists L3 > 0 such that
E[e6

RT
0

(g(s)−ḡ∆n (s))2 ds

2

δ∆
n k2

] ≤ eL3 δ∆n E[eL3 kB0

T k2

2

]E[eL3 δ∆n kY0

2

T 2

L3 sup|s−t|≤δ

]E[eL3 δ∆n kW0 k ]E[e

∆n

|W (s)−W (t)|2

Note that it follows from Doob’s submartingale inequality that if δ∆n is small enough,
δ∆
n k2

E[eL3 kB0

] < ∞,

and by Lemma 2.2, we also deduce that if δ∆n is small enough,
2

T k2

E[eL3 δ∆n kY0

30

] < ∞,

]

].

].

which, together with Condition (f), yields that for the first term in (63)
E[e2(−

RT
0

(g(s)−ḡ∆n (s))dB(s)− 12

RT
0

(g(s)−ḡ∆n (s))2 ds)

] < ∞.

(65)

A completely parallel argument will yield that for the second term in (63)
i
h RT
R
1 T
2
E e(2 0 (g(s)−ḡ∆n (s))dB(s)+ 2 0 (g(s)−ḡ∆n (s)) ds) < ∞,

which, together with (65), immediately implies (62), which in turn implies (61), as desired.
Step 1.2. In this step, we prove that as n tends to infinity,
Gn → log E[e−

RT
0

g(s)dY (s)+ 12

RT

g(s)2 ds

0

|Y0T ],

(66)

in probability.
First, note that by Theorem 7.23 of [51], we have,
Z
dµY |W =w T
dµY T
(Y0 ) =
(Y0 )dµW (w),
dµB
dµB
where

RT
R
dµY |W =w T
1 T 2
s
s
(Y0 ) = e 0 g(w0 )dY (s)− 2 0 g (w0 )ds ,
dµB

where we have rewritten g(s, w0s, Y0s ) as g(w0s) for notational simplicity. It then follows from
(49) that
Z
R
R
dµY |W =w T
T
− 0T g(s)dY (s)+ 12 0T g 2 (s)ds
(Y0 )dµW (w)
|Y0 ] = − log
log E[e
dµB
Z R
R
T
1 T 2
s
s
= − log e 0 g(w0 )dY (s)− 2 0 g (w0 )ds dµW (w).
Similarly, we have
dµY
(Y (∆n )) =
dµB
=

dµY |W =w
(Y (∆n ))dµW (w)
dµB
Z
1
Z

E[e−

RT
0

g(s)dY (s)+ 12

RT
0

g 2 (s)ds

|Y (∆n ), W ]|W =w

dµW (w).

It then again follows from (49) that
Gn = log E[e−
Z
= − log

RT
0

g(s)dY (s)+ 12

E[e−

RT
0

RT
0

g 2 (s)ds

g(s)dY (s)+ 12

RT
0

31

|Y (∆n )]
1

g 2 (s)ds

|Y (∆n ), W ]|W =w

dµW (w).

Z

Now, we consider the following difference:
Z
R
RT
1
s )dY (s)− 1 T g 2 (w s )ds
g(w
0
0
2 0
i
h RT
dµW (w) −
e0
R
T
− 0 g(w0s )dY (s)+ 12 0 g 2 (w0s )ds
Y (∆n ), W
E e
Z R
RT
R
R
T
1 T 2
1 T
s
s
s 2
s
= e 0 g(w0 )dY (s)− 2 0 g (w0 )ds − e 0 ḡ∆n (w0 )dY (s)− 2 0 ḡ∆n (w0 ) ds dµW (w)
Z R
R
T
1 T
s
s 2
+ e 0 ḡ∆n (w0 )dY (s)− 2 0 ḡ∆n (w0 ) ds
RT

×

E[e−(

0

g(w0s )dY (s)− 12

RT
0

R
R
2 (w s )ds)
g 2 (w0s )ds)+( 0T ḡ∆n (w0s )dY (s)− 21 0T ḡ∆
0
n

dµW (w)
W =w

|Y (∆n ), W ]|W =w − 1

R
R
R
R
2 (w s )ds)
−( 0T g(w0s )dY (s)− 12 0T g 2 (w0s )ds)+( 0T ḡ∆n (w0s )dY (s)− 21 0T ḡ∆
0
n

E[e
= In + J n .

|Y (∆n ), W ]|W =w

dµW (w)

Applying the inequality that for any x, y ∈ R,
|ex − ey | = |ey (ex−y − 1)| ≤ ey (|x − y|ex−y + |x − y|ey−x ) = |x − y|(ex + e2y−x ),

(67)

we have
h RT s
i
R
RT
R
1 T 2
1 T 2
s
s
s
E[|In |] ≤ E e 0 g(w0 )dY (s)− 2 0 g (w0 )ds − e 0 ḡ∆n (w0 )dY (s)− 2 0 ḡ∆n (w0 )ds dµW (w)
Z
Z Z T
1 T 2 s
2
s
s
g (w0 ) − ḡ∆
(w0s )ds
≤ E
g(w0) − ḡ∆n (w0 )dY (s) −
n
2
0
i
 R T s0
R
R
RT
R
R
1 T 2
s )dY (s)− T ḡ 2 (w s )ds)−( T g(w s )dY (s)− 1 T g 2 (w s )ds)
s )ds
(w
ḡ
g
(w
(2
g(w
)dY
(s)−
∆
0
0
0
0
0
∆n
0
0
2 0
2 0
+e 0 n 0
dµW (w)
× e0
Z Z T
Z T
1
1
s
s
≤ E
(g(w0 ) − ḡ∆n (w0 ))dB(s) +
(g(w0s ) − ḡ∆n (w0s ))(g(s) − g(w0s ) − ḡ∆n (w0s ))ds
2
2
0
i
 R T s0
RT
RT 2
RT
R
R
1 T 2
1 T 2
s
s
s
s
s
× e 0 g(w0 )dY (s)− 2 0 g (w0 )ds + e(2 0 ḡ∆n (w0 )dY (s)− 0 ḡ∆n (w0 )ds)−( 0 g(w0 )dY (s)− 2 0 g (w0 )ds) dµW (w)
Z "Z T
≤ E
(g(w0s) − ḡ∆n (w0s ))dB(s) + (Lδ∆n + L sup |w(s) − w(t)| + Lδ∆n
Z

|s−t|≤δ∆n

0

Z

T


1
1
s
s
+ sup |B(s) − B(t)|)
+
g(s) − g(w0 ) − ḡ∆n (w0 ) ds
2
2
|s−t|≤δ∆n
0
i
 RT s
R
R
R
R
R
T
T
T
T
2 (w s )ds)−(
g(w0s )dY (s)− 12 0T g 2 (w0s )ds)
(2 0 ḡ∆n (w0s )dY (s)− 0 ḡ∆
g(w0 )dY (s)− 12 0 g 2 (w0s )ds
0
0
0
n
dµW (w).
+e
× e
+Lδ∆n kw0T k

Lδ∆n kY0T k

Now, using (58), Condition (f) and the Itô isometry, we deduce that as n → ∞,
#
Z "Z
2
T

E

0

and as n tends to infinity,
Z

g(w0s) − ḡ∆n (w0s )dB(s)

E[(Lδ∆n + L

sup
|s−t|≤δ∆n

32

dµW (w) → 0,

|w(s) − w(t)| + Lδ∆n

(68)

+ Lδ∆n kw0T k + Lδ∆n kY0T k +

sup
|s−t|≤δ∆n

|B(s) − B(t)|)2 ]dµW (w) → 0.

(69)

Now, using a similar argument as above with (52) and Lemma 2.2, we can show that for any
constant K,
RT

E[e

0

2 (s)ds
K ḡ∆
n

RT

] = E[e

0

K(ḡ∆n (s)−g(s)+g(s))2 ds

RT

] = E[e

0

K(2(ḡ∆n (s)−g(s))2 +2g 2 (s))ds

] < ∞,

(70)

provided that n is large enough, which, coupled with a similar argument as in the derivation
of (65), proves that for n large enough,
Z  R

R
R
RT
R
R
T
s )dY (s)− 1 T g 2 (w s )ds
s )dY (s)− T ḡ 2 (w s )ds)−( T g(w s )dY (s)− 1 T g 2 (w s )ds) 2
g(w
ḡ(w
(2
0
0
0
0
0
0
0
0
2 0
2 0
E e0
dµW (w) < ∞,
+e 0
(71)

and furthermore
Z Z

T

0

1
1
g(s) − g(w0s ) − ḡ∆n (w0s ) ds
2
2

2

 RT s
2
RT
RT 2
RT
R
R
1 T 2
1 T 2
s
s
s
s
s
× e 0 g(w0 )dY (s)− 2 0 g (w0 )ds + e(2 0 ḡ∆n (w0 )dY (s)− 0 ḡ∆n (w0 )ds)−( 0 g(w0 )dY (s)− 2 0 g (w0 )ds) dµW (w) < ∞,
(72)
which further implies that as n tends to infinity,
E[|In |] → 0.

(73)

Now, using the shorthand notations An , A for
RT
R
1 T
s
g(w
)dY
(s)
−
g(w0s )2 ds respectively, we have
0
0
2 0
E[|Jn |] = E
=E
≤E

Z

Z
Z

Z

RT
0

ḡ∆n (w0s )dY (s) −

E[e−A+An |Y (∆n ), W ]|W =w − 1
dµW (w)
eAn
E[e−A+An |Y (∆n ), W ]|W =w

E[e−A+An − 1|Y (∆n ), W ]|W =w
dµW (w)
E[e−A |Y (∆n ), W ]|W =w

E[|e−A+An − 1||Y (∆n ), W ]|W =w
dµW (w)
E[e−A |Y (∆n ), W ]|W =w
|A−An |

A

RT
0

ḡ∆n (w0s )2 ds,





|Y (∆n ), W ]|W =w E[e |Y (∆n ), W ]|W =w dµW (w)
 



dµY |W T
dµY T
|A−An |
A
(Y ) /
(Y0 )
= E E[|A − An |e
|Y (∆n ), W ]E[e |Y (∆n ), W ]
dµB 0
dµB


 


dµY |W T
dµY T
|A−An |
A
= E |A − An |e
E[e |Y (∆n ), W ]E
(Y ) /
(Y0 ) |Y (∆n ), W .
dµB 0
dµB
≤E

E[|A − An |e

1
2

Now, a similar argument as in (68)-(72), together with the well-known fact (see, e.g., Theorem
6.2.2 in [38]) that
R
RT
1 T 2
dµY T
(Y0 ) = e 0 ĝ(s)dY (s)− 2 0 ĝ (s)ds ,
dµB

R
RT
dµY |W T
1 T 2
(Y0 ) = e 0 g(s)dY (s)− 2 0 g (s)ds ,
dµB

33

where ĝ(s) = E[g(s)|Y0T ], yields that
E[|Jn |] → 0.
Now, we are ready to conclude that as n tends to infinity,
E[e−

RT
0

g(s)dY (s)+ 12

RT
0

g 2 (s)ds

|Y (∆n )] → E[e−

RT
0

g(s)dY + 12

RT
0

g 2 (s)ds

|Y0T ]

in probability and furthermore (66), as desired.
Step 1.3. In this step, we show the convergence of {E[Fn ]} and {E[Gn ]} and further
establish the theorem under the condition (52).
Now, using the concavity of the log function and the fact that log x ≤ x, we can obtain
the upper bounds and lower bounds of Fn and Gn as follows:

 Z T
Z
h RT
i
R
1 T 2
T
− 0 g(s)dY (s)+ 12 0T g 2 (s)ds
T
g (s)ds Y∆n , W0 +E e
Y∆n , W0 ,
Fn ≤ E −
g(s)dY (s) +
2 0
0

 Z T
Z
h RT
i
R
1 T 2
T
− 0 g(s)dY (s)+ 12 0T g 2 (s)ds
T
g (s)ds Y∆n , W0 −E e
Fn ≥ − E −
g(s)dY (s) +
Y∆n , W0 ,
2 0
0
and
 Z T

Z
h RT
i
R
1 T 2
− 0 g(s)dY (s)+ 12 0T g 2 (s)ds
Gn ≤ E −
g(s)dY (s) +
g (s)ds Y∆n + E e
Y ∆n ,
2 0
0
 Z T

Z
i
h RT
R
1 T 2
− 0 g(s)dY (s)+ 21 0T g 2 (s)ds
g(s)dY (s) +
Y ∆n .
g (s)ds Y∆n − E e
Gn ≥ − E −
2 0
0
And furthermore, using a similar argument as in Step 1.1, we can show that as n tends to
infinity,
 Z T
  Z T

Z
Z
1 T 2
1 T 2
T
E −
g(s)dY (s) +
g (s)ds Y (∆n ), W0 = −
ḡ (s)ds
ḡ∆n (s)dY (s) +
2 0
2 0 ∆n
0
0

 Z T
Z
1 T 2
2
T
(g (s) − ḡ∆n (s))ds Y (∆n ), W0
×E −
(g(s) − ḡ∆n )(s)dY (s) +
2 0
0
 Z T

Z
1 T 2
→ −
g(s)dY (s) +
g (s)ds
2 0
0
and
E[e−

RT
0

g(s)dY (s)+ 21

= E[e−
= e−

RT

→ e−

0

RT
0

0

g 2 (s)ds

|Y (∆n ), W0T ]

ḡ∆n (s)dY (s)+ 12

ḡ∆n (s)dY (s)+ 12

RT
0

RT

g(s)dY (s)+ 12

RT

RT
0

0

RT
0

2 (s)ds−
ḡ∆
n

2 (s)ds
ḡ∆
n

g 2 (s)ds

RT
0

E[e−

(g(s)−ḡ∆n (s))dY (s)+ 12

RT
0

RT
0

(g(s)−ḡ∆n (s))dB(s)− 12

2 (s))ds
(g 2 (s)−ḡ∆
n

RT
0

|Y (∆n ), W0T ]

(g(s)−ḡ∆n (s))2 ds

|Y (∆n ), W0T ]

.

It then follows from the general Lebesgue dominated convergence theorem that

 Z T
Z
1 T
2
g(s) ds .
lim E[Fn ] → E −
g(s)dY (s) +
n→∞
2 0
0
34

A parallel argument can be used to show that
lim E[Gn ] = E[log E[e−

RT
0

g(s)dY (s)+ 12

RT
0

g(s)2 ds

n→∞

|Y0T ]].

So, under the condition (52), we have shown that
lim I(W0T ; Y (∆n )) = I(W0T ; Y0T ).

n→∞

Step 2. In this step, we will use the convergence in Step 1 and establish the theorem
without the condition (52).
Following Page 264 of [51], we define, for any k,
(
Rt
RT
inf{t ≤ T : 0 g 2 (s, W0s , Y0s )ds ≥ k}, if 0 g 2 (s, W0s , Y0s )ds ≥ k
RT
τk =
(74)
∞, if 0 g 2(s, W0s , Y0s )ds < k.
Then, we again follow [51] and define a truncated version of g as follows:
g(k) (t, γ0t , φt0 ) = g(t, γ0t , φt0 )1R t g2 (s,γ t ,φs )ds<k .
0

0

Now, define a truncated version of Y as follows:
Z t
Y(k) (t) = ρ
g(k) (s, W0s , Y0s )ds + B(t),
0

0

t ∈ [0, T ],

which, as elaborated on Page 265 in [51], can be rewritten as
Z t
s
Y(k) (t) = ρ
g(k) (s, W0s , Y(k),0
)ds + B(t), t ∈ [0, τk ∧ T ].

(75)

0

Note that for fixed k, the system in (75) satisfies the condition (52), and so the theorem
holds true. To be more precise, note that




dµτk ,Y |W τk
dµτk ,Y τk
τk
T
(Y0 ) − E log
(Y ) ,
I(W0 ; Y0 ) = E log
dµτk ,B
dµτk ,B 0
where µτk ,Y and µτk ,B respectively denote the truncated versions of µY and µB (from time 0
to time τn ). Applying Theorem 7.10 in [51], we obtain
R τk
R
dµτk ,Y |W τn
1 τk 2
(Y0 ) = e 0 g(s)dY (s)− 2 0 g (s)ds ,
dµτk ,B

and

R τk
R
1 τk 2
dµτk ,Y τk
(Y0 ) = e 0 ĝ(s)dY (s)− 2 0 ĝ (s)ds ,
dµτk ,B

where
ĝ(s) = E[g(s, W0s , Y0s )|Y0s ].

35

It then follows that
I(W0T ; Y0τk )

1
= E
2

Z

τk

0

2



(g(s) − ĝ(s)) ds .

Notice that it can be easily verified that τk → T as k tends to infinity, which, together with
the monotone convergence theorem, further yields that monotone increasingly,

Z τk
Z T

1
1
τk
T
2
T
T
2
I(W0 ; Y0 ) = E
(g(s) − ĝ(s)) ds → I(W0 ; Y0 ) = E
(g(s) − ĝ(s)) ds ,
2
2
0
0
as k tends to infinity. By Step 1, for any fixed ki ,
τk

lim I(W0T ; Y (∆n ∩ [0, τki ])) = I(W0T ; Y0 i ),

n→∞

which means that there exists a sequence {ni } such that, as i tends to infinity, we have,
monotone increasingly,
I(W0T ; Y (∆ni ∩ [0, τki ])) → I(W0T ; Y0T ).
Since, by the fact that Y0τk coincides with Y0T on the interval [0, τk ∧ T ], we have
I(W0T ; Y (∆ni )) ≥ I(W0T ; Y (∆ni ∩ [0, τki ])).
Now, using the fact that
I(W0T ; Y (∆ni )) ≤ I(W0T ; Y0T ),

we conclude that as i tends to infinity,

lim I(W0T ; Y (∆ni )) = I(W0T ; Y0T ).

i→∞

A similar argument can be readily applied to any subsequence of {I(W0T ; Y (∆n ))}, which
will establish the existence of its further subsubsequence that converges to I(W0T ; Y0T ), which
implies that
lim I(W0T ; Y (∆n )) = I(W0T ; Y0T ).
n→∞

The proof of the theorem is then complete.
Remark C.1. The arguments in the proof of Theorem 2.3 can be adapted to yield a sampling
theorem for continuous-time minimum mean square error (MMSE), a quantity of central
importance in estimation theory.
More precisely, consider the following continuous-time Gaussian feedback channel under
the assumptions of Theorem 2.3:
Z t
Y (t) =
X(s, M, Y0s )ds + B(t), t ∈ [0, T ].
0

The MMSE is the limit of the MMSE based on the samples with respect to ∆n , namely,
Z T
Z T
T 2
E[(X(s) − E[X(s)|Y0 ]) ]ds = lim
E[(X(s) − E[X(s)|Y (∆n ))2 ]ds.
0

n→∞

36

0

To see this, note that the above-mentioned convergence follows from the fact that
" R
2 #
s
s
T
X(s,
m
,
Y
)dµ
(Y
)/dµ
dµ
(m)
B
M
Y
|M
=m
0
0
0
,
E[E2 [X(s)|Y0T ]] = E
dµY (Y0T )/dµB
and
E[E2 [X(s)|Y (∆n )]] = E

" R

X(s, ms0 , Y0s )dµY |M =m (Y (∆n ))/dµB dµM (m)
dµY (Y (∆n ))/dµB

2 #

,

and the proven fact that dµY (Y (∆n ))/dµB and dµY |M (Y (∆n ))/dµB respectively converge to
dµY (Y0T )/dµB and dµY |M (Y0T )/dµB and a parallel argument as in establishing the convergence of {E[Fn ]} and {E[Gn ]} in the proof of Theorem 2.3.
Similarly, we can also conclude that under the assumptions of Theorem 2.3, the causal
MMSE is the limit of the sampled causal MMSE, namely,
Z T
Z T
s 2
E[(X(s) − E[X(s)|Y0 ]) ]ds = lim
E[(X(s) − E[X(s)|Y (∆n ∩ [0, s])])2 ]ds.
n→∞

0

D

0

Proof of Theorem 3.1

In this section, we give the detailed proof of Theorem 3.1.
We will first need the following lemma, which is parallel to Lemma 2.2.
Lemma D.1. Assume Conditions (d)-(f ). Then, there exists ε > 0 and a constant C > 0
such that for all n,
(n)
2
E[eε sup0≤t≤T (Y (t)) ] < C.
(76)
Proof. A discrete-time version of the proof of Lemma 2.2 implies that there exists ε > 0 and
a constant C > 0 such that for all n
E[eε supi∈{0,1,...,n} (Y

(n) (t
2
n,i ))

] < C,

which, together with (12), immediately implies (76).
We also need the following lemma, which is parallel to Theorem 10.2.2 in [34].
Lemma D.2. Assume Conditions (d)-(f ). Then, there exists a constant C > 0 such that
for all n,
E[ sup |Y (n) (t) − Y (t)|2 ] ≤ Cδ∆n .
0≤t≤T

Proof. Note that for any n, we have
Z
Y (tn,i+1 ) = Y (tn,i ) +

tn,i+1

tn,i

g(s, W0s, Y0s )ds + B(tn,i+1 ) − B(tn,i ),

and
Y

(n)

(tn,i+1 ) = Y

(n)

(tn,i ) +

Z

tn,i+1

tn,i

t

(n),tn,i

g(s, W0 n,i , Y0
37

)ds + B(tn,i+1 ) − B(tn,i ).

It then follows that
Y (tn,i+1 ) − Y

(n)

(n)

(tn,i+1 ) = Y (tn,i ) − Y

(tn,i ) +

tn,i+1

Z

tn,i

t

(n),tn,i

(g(s, W0s, Y0s ) − g(s, W0 n,i , Y0

))ds.

(77)
Now, for any t, choose n0 such that tn,n0 ≤ t < tn,n0 +1 . Now, a recursive application of (77),
coupled with Conditions (d) and (e), yields that for some L > 0,
Y (t) − Y

(n)

(t) =
≤
+

n0 Z
X

tn,i+1

i=0 tn,i
n0 Z tn,i+1
X

i=0
Z t

tn,i

tn,n0 +1

(g(s, W0s , Y0s ) −

t
(n),t
g(tn,i , W0 n,i , Y0 n,i ))ds
t

+

t

Z

tn,n0 +1

(n),s

L|s − tn,i | + LkW0s − W0 n,i k + LkY0s − Y0
tn,n0 +1

L|s − tn,n0 +1 | + LkW0s − W0

tn,n0 +1

(g(s, W0s , Y0s ) − g(tn,i , W0
(n),s

k + LkY0

(n),s

k + LkY0s − Y0

(n),tn,i

− Y0

(n),s

k + LkY0

(n),tn,n0 +1

, Y0

kds

(n),tn,n0 +1

− Y0

kds.

Noticing that for any s with tn,i ≤ s < tn,i+1 , we have
(n),t
(n),s
kY0 −Y0 n,i k2

≤

(n),t
(n),t
kY0 n,i+1 −Y0 n,i k2

≤2

Z

tn,i+1

tn,i

2
t
(n),t
g(s, W0 n,i , Y0 n,i )ds

+2|B(tn,i+1)−B(tn,i )|2 ,

which, together with Condition (e) and the fact that for all n and i,
E[|B(tn,i+1 ) − B(tn,i )|2 ] = O(δ∆n ),

(78)

implies that
(n),s

E[kY0

(n),tn,i 2

k ] = O(δ∆n ).

− Y0

(79)

Noting that the constants in the two terms O(δ∆n ) in (78) and (79) can be chosen uniform
over all n, a usual argument with the Gronwall inequality and Condition (f) applied to
(n),t
E[kY0t − Y0 k2 ] completes the proof of the theorem.
We are now ready for the proof of Theorem 3.1.
Proof of Theorem 3.1. We proceed in two steps.
Step 1. In this step, we establish the theorem assuming that there exists a constant
C > 0 such that for all w0T ∈ C[0, T ] and all y0T ∈ C[0, T ],
Z T
(80)
g 2(s, w0s , y0s )ds < C.
0

We first note that straightforward computations yield
fY (n) (∆n )|W (y (n) (∆n )|w0T ) =

n
Y
i=1

=

n
Y
i=1

(n)

(n),tn,i−1

f (ytn,i |ytn,0

p

t

, w0n,i−1 )


1
exp −
2π(tn,i − tn,i−1 )

(n)

(n)

(ytn,i − ytn,i−1 −

38

R tn,i

t

(n),tn,i−1

g(s, w0n,i−1 , y0
tn,i−1

2(tn,i − tn,i−1 )

)ds)2



,

))ds

(n)

(n)

(here we have used the shorter notations ytn,i , ytn,i−1 for y (n) (tn,i ), y (n)(tn,i−1 ), respectively)
and


R tn,i
tn,i−1
(n),tn,i−1
(n)
(n)
Z Y
2
n
(y
−
y
−
g(s,
w
,
y
)ds)
tn,i
tn,i−1
0
0
1
tn,i−1
 dµW (w),
p
exp −
fY (n) (∆n ) (y (n) (∆n )) =
2(t
−
t
)
2π(t
−
t
)
n,i
n,i−1
n,i
n,i−1
i=1
which further lead to

fY (n) (∆n )|W (Y (n) (∆n )|W0T ) =

n
Y
i=1

and
fY (n) (∆n ) (Y (n) (∆n )) =

Z Y
n
i=1



1
p
exp −
2π(tn,i − tn,i−1 )


1
p
exp −
2π(tn,i − tn,i−1 )

(n)

(n)

(n)

(Ytn,i − Ytn,i−1 −

(n)

(Ytn,i − Ytn,i−1 −

R tn,i

t

R tn,i

(n),tn,i−1

g(s, W0 n,i−1 , Y0
tn,i−1

2(tn,i − tn,i−1 )

t

)ds)2

(81)
(n),tn,i−1

g(tn,i−1 , w0n,i−1 , Y0
tn,i−1
2(tn,i − tn,i−1 )

With (81) and (82), we have

)ds)2

(82)



,



 dµW (w).

I(W0T ; Y (n) (∆n )) = E[log fY (n) (∆n )|W (Y (n) (∆n )|W0T )] − E[log fY (n) (∆n ) (Y (n) (∆n ))]



R tn,i
R tn,i
t
(n),t
t
(n),t
(n)
(n)
n
g(s, W0 n,i−1 , Y0 n,i−1 )ds)2
−2 tn,i−1
g(s, W0 n,i−1 , Y0 n,i−1 )ds (Ytn,i − Ytn,i−1 ) + ( tn,i−1
Y

exp −
= E log
2(tn,i − tn,i−1 )
i=1




R tn,i
R tn,i
t
(n),t
t
(n),t
(n)
(n)
Z Y
n
g(s, w0n,i−1 , Y0 n,i−1 )ds)2
−2 tn,i−1
g(s, w0n,i−1 , Y0 n,i−1 )ds (Ytn,i − Ytn,i−1 ) + ( tn,i−1
 dµW (w)
− E log
exp −
2(tn,i − tn,i−1 )
i=1



R tn,i
R tn,i
t
(n),t
t
(n),t
(n)
(n)
n
g(s, W0 n,i−1 , Y0 n,i−1 )ds)2
−2 tn,i−1
g(s, W0 n,i−1 , Y0 n,i−1 )ds (Ytn,i − Ytn,i−1 ) + ( tn,i−1
X
−

= E
2(tn,i − tn,i−1 )
i=1




R tn,i
R tn,i
t
t
(n),t
(n),t
(n)
(n)
Z
n
X
−2 tn,i−1
g(s, w0n,i−1 , Y0 n,i−1 )ds (Ytn,i − Ytn,i−1 ) + ( tn,i−1
g(s, w0n,i−1 , Y0 n,i−1 )ds)2
 dµW (w) .
−
− E log exp
2(tn,i − tn,i−1 )
i=1

On the other hand, it is well known (see, e.g., [38]) that




dµY |W T
dµY T
T
I(W ; Y0 ) = E log
(Y0 ) − E log
(Y )
dµB
dµB 0

Z T

Z
1 T 2
s
s
s
s
= E log exp
g(s, W0 , Y0 )dY (s) −
g (s, W0 , Y0 )ds
2 0
0

 Z
Z T
Z
1 T 2
s
s
s
s
g (s, w0 , Y0 )dsdµW (w)
− E log exp
g(s, w0, Y0 )dY (s) −
2 0
0
Z T

Z
1 T 2
s
s
s
s
=E
g(s, W0 , Y0 )dY (s) −
g (s, W0 , Y0 )ds
2 0
0

 Z
Z T
Z
1 T 2
s
s
s
s
g (s, w0 , Y0 )dsdµW (w) .
− E log exp
g(s, w0, Y0 )dY (s) −
2 0
0
39

Now, we compute
R tn,i
t
(n),t
(n)
(n)
Z T
n
X
g(s, W0 n,i−1 , Y0 n,i−1 )ds(Ytn,i − Ytn,i−1 )
t
n,i−1
g(s, W0s, Y0s )dY (s) −
tn,i − tn,i−1
0
i=1
Z T
n
X
t
(n),t
(n)
(n)
=
g(s, W0s , Y0s )dY (s) −
g(tn,i−1, W0 n,i−1 , Y0 n,i−1 )(Ytn,i − Ytn,i−1 )
0

−

n
X
i=1

i=1
tn,i−1
(n),tn,i−1
(g(s, W0
, Y0
)
tn,i−1

R tn,i

t

(n),tn,i−1

− g(tn,i−1 , W0 n,i−1 , Y0
tn,i − tn,i−1

(n)

(n)

))ds(Ytn,i − Ytn,i−1 )

.

It can be easily checked that the second term of the right hand side of the above equality
converges to 0 in mean. For the first term, we have
Z T
n
X
t
(n),t
(n)
(n)
s
s
g(s, W0 , Y0 )dY (s) −
g(tn,i−1, W0 n,i−1 , Y0 n,i−1 )(Ytn,i − Ytn,i−1 )
0

=
+

i=1

n Z tn,i
X

i=1
n
X

tn,i−1

g(s, W0s, Y0s )dY
t

(s) −

(n),tn,i−1

g(tn,i−1 , W0 n,i−1 , Y0

i=1

=
+

n Z
X

i=1
n
X

tn,i
tn,i−1

g(s, W0s, Y0s )dY
t

(n),tn,i−1

i=1

=

n Z
X
i=1

+

n
X
i=1

tn,i
tn,i−1

t

i=1

(n)

n Z
X
i=1

n

i=1

(n),tn,i−1

(n)

t

(n),tn,i−1

(n)

)dY (s)

(n)

)((Ytn,i − Ytn,i−1 ) − (Ytn,i − Ytn,i−1 ))
t

t

)(Ytn,i − Ytn,i−1 )

g(tn,i−1, W0 n,i−1 , Y0

(n),tn,i−1

(g(s, W0s , Y0s ) − g(tn,i−1, W0 n,i−1 , Y0

g(tn,i−1 , W0 n,i−1 , Y0

(n),tn,i−1

g(tn,i−1, W0 n,i−1 , Y0

)((Ytn,i − Ytn,i−1 ) − (Ytn,i − Ytn,i−1 ))

(s) −

g(tn,i−1 , W0 n,i−1 , Y0

n
X

))dY (s)
(n)

(n)

)((Ytn,i − Ytn,i−1 ) − (Ytn,i − Ytn,i−1 )).

It then follows from Conditions (d) and (e), Lemmas 2.2, D.1 and D.2 that


R tn,i
tn,i−1
(n),tn,i−1
(n)
(n)
Z T
n
X
,
Y
)ds(Y
g(s,
W
−
Y
)
1
tn,i
tn,i−1
0
0
tn,i−1
−
E
g(s, W0s, Y0s )dY (s)  = O(δ∆2 n ).
tn,i − tn,i−1
0
i=1
(83)

And using a similar argument as above, we deduce that


R
Z T
n ( tn,i g(s, W tn,i−1 , Y (n),tn,i−1 )ds)2
X
1
0
0
1
1
tn,i−1
−
g(s, W0s, Y0s )2 ds  = O(δ∆2 n ).
E
2 i=1
tn,i − tn,i−1
2 0

(84)

It then follows from (83) and (84) that as n tends to infinity,



R tn,i
R tn,i
t
(n),t
t
(n),t
(n)
(n)
n
X
−2 tn,i−1
g(s, W0 n,i−1 , Y0 n,i−1 )ds(Ytn,i − Ytn,i−1 ) + ( tn,i−1
g(s, W0 n,i−1 , Y0 n,i−1 )ds)2

−
E
2(t
−
t
)
n,i
n,i−1
i=1
40

−

Z

T
0

g(s, W0s , Y0s )dY

1
(s) +
2

Z

T

g(s, W0s , Y0s )2 ds

0

We now establish the following convergence:


E log

Z

(n)

exp A



1

= O(δ∆2 n ).

(85)


 Z

(w)dµW (w) → E log exp A(w)dµW (w) .

(86)

where
A(n) (w) =

n
X
i=1

and let



−

−2

R tn,i

t

(n),tn,i−1

g(s, w0n,i−1 , Y0
tn,i−1

A(w) =

(n)

(n)

)ds(Ytn,i − Ytn,i−1 ) + (
2(tn,i − tn,i−1 )

T

Z

g(s, w0s, Y0s )dY

0

1
(s) −
2

Z

R tn,i

t

(n),tn,i−1

g(s, w0n,i−1 , Y0
tn,i−1

)ds)2

T
0

g 2(s, w0s , Y0s )ds.

Note that using a parallel argument as the derivation of (85), we can establish
Z
E
A(n) (w) − A(w) dµW (w) → 0,

(87)

as n tends to infinity; and similarly as in the derivation of (73), from Conditions (d), (e) and
(f), Lemmas 2.2, D.1 and D.2, we deduce that
Z

(n)
E
exp A (w) − exp A(w) dµW (w) → 0
(88)
as n tends to infinity. And note that we always have
Z
Z
Z
(n)
(n)
A(n) (w)dµW (w) .
log exp A (w)dµW (w) ≤ exp A (w)dµW (w) +

(89)

So, by the general Lebesgue dominated convergence theorem with (87), (88) and (89), we
have
 Z

 Z

(n)
E log exp A (w)dµW (w) → E log exp A(w)dµW (w) .

So, under the condition (80), we have established the theorem.
Step 2. In this step, we will use the convergence in Step 1 and establish the theorem
without the condition (80).
Defining the stopping τk , g(k) and Y(k) as in the proof of Theorem 2.3, we again have:
Y(k) (t) = ρ

Z

0

t
s
g(k) (s, W0s , Y(k),0
)ds + B(t),

t ∈ [0, τk ∧ T ].

For any fixed k, applying the Euler-Maruyama approximation as in (11) and (12) to the
(n)
above channel with respect to ∆n , we obtain the process Y(k) (·).

41



.

Now, by the fact that
I(W0T ; Y (n) (∆n )) = E[log fY (n) (∆n )|W (Y (n) (∆n )|W0T )] − E[log fY (n) (∆n ) (Y (n) (∆n ))]
 Z

(n)
(n)
= E[A (W )] − E log exp A (w)dµW (w)
≥ 0,

we deduce that
E[log fY (n) (∆n ) (Y (n) (∆n ))]
≤ E[log fY (n) (∆n )|W (Y (n) (∆n )|W0T )]



R tn,i
R tn,i
tn,i−1
(n),tn,i−1
tn,i−1
(n),tn,i−1
(n)
(n)
2
n
X
, Y0
)ds)
−2 tn,i−1 g(s, W0
, Y0
)ds (Ytn,i − Ytn,i−1 ) + ( tn,i−1 g(s, W0

−
= E
2(t
−
t
)
n,i
n,i−1
i=1


R
n ( tn,i g(s, W tn,i−1 , Y (n),tn,i−1 )ds)2
X
0
0
tn,i−1

= E
2(t
−
t
)
n,i
n,i−1
i=1


R
R
n ( tn,i g(s, W tn,i−1 , Y (n),tn,i−1 )2 ds) tn,i ds
X
0
(k),0
tn,i−1
tn,i−1

≤ E
2(t
−
t
)
n,i
n,i−1
i=1
" n Z
#
X tn,i
tn,i−1
(n),tn,i−1 2
=E
g(s, W0
, Y(k),0
) ds
i=1
T

=E

Z

0

tn,i−1

(n),⌊s⌋∆n 2
⌊s⌋
) ds
g(s, W0 ∆n , Y0



,

where ⌊s⌋∆n denote the unique number n0 such that tn,n0 ≤ s < tn,n0 +1 . Now, using the
easily verifiable fact that
R

1
= E[exp(−A(n) (W ))|Y0T ],
exp A(n) (w)dµW (w)

and Jensen’s inequality, we deduce that




1
(n)
T
=
E
log
E[exp(−A
(W
))|Y
]
≤ log E[exp(−A(n) (W ))] ≤ 0,
E log R
0
(n)
exp A (w)dµW (w)
where for the last inequality, we have applied Fatou’s lemma as in deriving (64). It then
follows that
" n Z
#
X tn,i
tn,i−1
(n),tn,i−1 2
(n)
0 ≤ E[log fY (n) (∆n ) (Y (∆n ))] ≤ E
g(s, W0
, Y0
) ds ,
i=1

tn,i−1

which further implies that
I(W0T ; Y (n) (∆n )) ≤ E

" n Z
X
i=1

tn,i
tn,i−1

42

t

(n),tn,i−1 2

g(s, W0 n,i−1 , Y0

#

) ds .

(n)

Now, using the fact that Y (n) and Y(k) coincide over [0, τk ∧ T ], one verifies that for any
ε > 0,
Z T

(n),⌊s⌋∆n 2
⌊s⌋∆n
(n)
T
(n)
T
I(W0 ; Y (∆n )) − I(W0 ; Y(k),∆n ) ≤ E
) ds
, Y0
g(s, W0
τk
Z T

Z T

(n),⌊s⌋∆n 2
⌊s⌋∆n
(n),⌊s⌋∆n 2
⌊s⌋∆n
≤E
) ds; T − τk ≤ ε + E
, Y0
g(s, W0
) ds; T − τk > ε
, Y0
g(s, W0
τk
τk
Z T

Z T
i
h
(n),⌊s⌋∆n 2
⌊s⌋∆n
(n),⌊s⌋∆n 2
⌊s⌋∆n
≤
) ds + E
, Y0
E g(s, W0
) ds; T − τk > ε .
, Y0
g(s, W0
T −ε

τk

Using the easily verifiable fact that {τk } converges to T in probability uniformly over all n
and the fact that ε can be arbitrarily small, we conclude that as k tends to infinity, uniformly
over all n,
(n)
I(W0T ; Y(k) (∆n )) → I(W0T ; Y (n) (∆n )).
(90)
Next, an application of the monotone convergence theorem, together with the fact that
τk → T as k tends to infinity, yields that monotone increasingly

Z τn
Z T

1
1
τn
2
T
T
2
T
(g(s) − ĝ(s)) ds → I(W0 ; Y0 ) = E
(g(s) − ĝ(s)) ds
I(W0 ; Y0 ) = E
2
2
0
0

as n tends to infinity. By Step 1, for any fixed ki ,
τk

(n)

lim I(W0T ; Y(ki ) (∆n )) = I(W0T ; Y0 i ),

n→∞

which means that there exists a sequence {ni } such that, as i tends to infinity,
(n )

I(W ; Y(ki)i (∆n )) → I(W ; Y0T ).
Moreover, by (90),
(n )

lim I(W0T ; Y(ki )i (∆ni )) = lim I(W0T ; Y (ni ) (∆n )),

i→∞

i→∞

which further implies that
lim I(W0T ; Y (ni ) (∆n )) = I(W0T ; Y0T ).

i→∞

The theorem then follows from a usual subsequence argument as in the proof of Theorem 2.3.
Remark D.3. Parallel to Remark C.1, the arguments in the proof of Theorem 3.1 can be
adapted to yield an approximation theorem in estimation theory.
More precisely, consider the following continuous-time Gaussian feedback channel under
the assumptions in Theorem 3.1:
Z t
Y (t) =
X(s, M, Y0s )ds + B(t), t ∈ [0, T ].
0

43

The MMSE is the limit of the approximated MMSE, namely,
Z T
Z T
T 2
E[(X(s) − E[X(s)|Y0 ]) ]ds = lim
E[(X (n) (s) − E[X (n) (s)|Y (n) (∆n )])2 ]ds.
n→∞

0

0

In more detail, the above-mentioned convergence follows from the fact that
Z T
Z T
(n)
2
E[(X (s)) ]ds →
E[(X(s))2 ]ds
0

0

and the fact that
2

E[E

[X(s)|Y0T ]]

=E

" R

and the fact that
E[E2 [X (n) (s)|Y (n) (∆n )]] = E

X(s, ms0 , Y0s ) exp(A(m))dµM (m)
R
exp(A(m))dµM (m)

" R

2 #

,

X (n) (s, ms0 , Y0s ) exp(A(n) (m))dµM (m)
R
exp(A(n) (m))dµM (m)

2 #

.

Then, using a similar argument as in the proof of Theorem 3.1, we can show
lim E[E2 [X (n) (s)|Y (n) (∆n )]] = E[E2 [X(s)|Y0T ]],

n→∞

which implies the claimed convergence.
Similarly, we can also conclude that with the assumptions in Theorem 3.1, the causal
MMSE is the limit of the approximated causal MMSE, namely,
Z T
Z T
s 2
E[(X(s) − E[X(s)|Y0 ]) ]ds = lim
E[(X (n) (s) − E[X (n) (s)|Y (n) (∆n ∩ [0, s])2 ]ds.
n→∞

0

E

0

Proof of Theorem 5.1

In the section, we give the proof of Theorem 5.1. For notational convenience only, we will
assume m = 2, the case with a generic m being completely parallel. We will first need
the following lemma, which is a key component in our treatment of both continuous-time
Gaussian MACs.
Lemma E.1. For any ǫ > 0, there exist two independent Ornstein-Uhlenbeck processes
{Xi (s) : s ≥ 0}, i = 1, 2, satisfying the following power constraint:
Z
1 t
E[Xi2 (s)]ds = Pi ,
(91)
for i = 1, 2, there exists Pi > 0 such that for all t > 0,
t 0
such that for all T ,
and

|IT (X1 , X2 ; Y )/T − (P1 + P2 )/2| ≤ ǫ,
|IT (X1 ; Y |X2 )/T − P1 /2| ≤ ǫ,

|IT (X2 ; Y |X1 )/T − P2 /2| ≤ ǫ,
44

(92)
(93)

moreover,
where

|IT (X1 ; Y )/T − P1 /2| ≤ ǫ,
Y (t) =

Z

t

X1 (s)ds +

0

Z

|IT (X2 ; Y )/T − P2 /2| ≤ ǫ,

(94)

t

X2 (s)ds + B(t),

0

t ≥ 0.

(95)

Here (and often in the remainder of the paper) the subscript T means that the (conditional)
mutual information is computed over the time period [0, T ].
Proof. For a > 0, consider the following two independent Ornstein-Uhlenbeck processes
Xi (t), i = 1, 2, given by
Z t
p
e−a(t−s) dBi (s),
Xi (t) = 2aPi
−∞

where Bi , i = 1, 2, are independent standard Brownian motions. Obviously, for Xi defined
as above, (91) is satisfied. A parallel version of the proof of Theorem 6.2.1 of [38] yields that
Z
1 T
IT (X1 , X2 ; Y ) = IT (X1 + X2 ; Y ) =
E[(X1 (t) + X2 (t) − E[X1 (t) + X2 (t)|Y0t ])2 ]dt.
2 0
It then follows from Theorem 6.4.1 in [38] (applied to the Ornstein-Uhlenbeck process X1 (t)+
X2 (t) that as a → ∞,
IT (X1 , X2 ; Y )/T = IT (X1 + X2 ; Y )/T → (P1 + P2 )/2,
uniformly in T , which establishes (92).
For i = 1, 2, define
Z t
Ỹi (t) =
Xi (s)ds + B(t),

t > 0.

0

As in the proof of Theorem 6.4.1 in [38], we deduce that for i = 1, 2, IT (Xi ; Ỹi )/T tend to
Pi /2 uniformly in T . Now, since X1 and X2 are independent, we have for any fixed T ,
IT (X1 ; Y |X2 ) = IT (X1 ; Ỹ1 |X2 ) = IT (X1 ; Ỹ1 ),
and
IT (X2 ; Y |X1 ) = IT (X2 ; Ỹ2 |X1 ) = IT (X2 ; Ỹ2 ),

which immediately implies (93).
Now, by the chain rule of mutual information,

IT (X1 , X2 ; Y ) = IT (X1 ; Y ) + IT (X2 ; Y |X1 ) = IT (X2 ; Y ) + IT (X1 ; Y |X2 ),
which, together with (92) and (93), implies (94).
Remark E.2. With Xi , i = 1, 2, regarded as channel inputs, (95) can be reinterpreted as a
white Gaussian MAC. For i 6= j, I(Xi ; Y ), the reliable transmission rate of Xi when Xj is
not known can be arbitrarily close to I(Xi ; Y |Xj ), the reliable transmission rate of Xi when
Xj is known. In other words, for white Gaussian MACs, knowledge about other user’s inputs
45

will not help to achieve faster transmission rate, and therefore, they can be simply treated as
noises. An more intuitive explanation of this result is as follows: for the Ornstein-Uhlenbeck
process Xi as specified in the proof, its power spectral density can be computed as
fi (λ) =

2aPi
,
2π(λ2 + a2 )

which is “negligible” compared to that of the white Gaussian noise (which is the constant
1) as a tends to infinity. Lemma E.1 is a key ingredient for deriving the capacity regions of
white Gaussian MACs.
We also need some result on the information stability of continuous-time Gaussian processes. Let (U, V ) = {(U(t), V (t)), t ≥ 0} be a continuous Gaussian system (which means
U(t), V (t) are pairwise Gaussian stochastic processes). Define
(T )

ϕ(T ) (u, v) =
(T )

(T )

dµU V
(T )

(T )

dµU × µV

(u, v),

(u, v) ∈ C[0, T ] × C[0, T ],

(T )

where µU , µV and µU V denote the probability distributions of U0T , V0T and their joint
(T )
distribution, respectively. For any ε > 0, we denote by Tε the ε-typical set:


1
(T )
(T )
Tε = (u, v) ∈ C[0, T ] × C[0, T ]; | log ϕ (u, v) − IT (U, V )| ≤ ε .
T
The pair (U, V ) is said to be information stable [59] if for any ε > 0,
(T )

lim µU V (Tε ) = 1.

T →∞

The following theorem is a rephrased version of Theorem 6.6.2. in [38].
Lemma E.3. The Gaussian system (U, V ) is information stable provided that
IT (U; V )
= 0.
T →∞
T2
lim

Lemma E.3 will be used in the proof of Theorem 5.1 to establish, roughly speaking, that
almost all sequences are jointly typical.
With Lemmas E.1 and E.3, Theorem 5.1 largely follows from a lengthy yet almost routine
argument, which is included below due to a number of technical challenges in the proof.
Proof of Theorem 5.1. The converse part. In this part, we will show that for any sequence
(T )
of (T, (eT R1 , eT R2 ), (P1 , P2 ))-codes with Pe → 0 as T → ∞, the rate pair (R1 , R2 ) will have
to satisfy
R1 ≤ P1 /2,
R2 ≤ P2 /2.

Fix T and consider the above-mentioned (T, (eT R1 , eT R2 ), (P1 , P2 ))-code. By the code
construction, it is possible to estimate the messages (M1 , M2 ) from the channel output Y0T

46

with a low probability of error. Hence, the conditional entropy of (M1 , M2 ) given Y0T must
be small; more precisely, by Fano’s inequality,
H(M1 , M2 |Y0T ) ≤ T (R1 + R2 )Pe(T ) + H(Pe(T ) ) = T εT ,
where εT → 0 as T → ∞. Then, we have
H(M1 |Y T ) ≤ H(M1 , M2 |Y T ) ≤ T εT ,

H(M2 |Y T ) ≤ H(M1 , M2 |Y T ) ≤ T εT .

Now, we can bound the rate R1 as follows:
T R1 = H(M1 )
= I(M1 ; Y0T ) + H(M1 |Y0T )

≤ I(M1 ; Y0T ) + T εT

≤ H(M1 ) − H(M1 |Y0T ) + T εT

≤ H(M1 |M2 ) − H(M1 |Y0T , M2 ) + T εT
= I(M1 ; Y0T |M2 ) + T εT .

Conditioning on M2 and applying Theorem 6.2.1 in [38], we have
Z T

1
2
T
E[(X1 (t) + X2 (t) − X̂1 (t) − X̂2 (t)) |M2 ]dt
I(M1 ; Y0 |M2 ) = E
2
0
Z
1 T
E[(X1 (t) + X2 (t) − X̂1 (t) − X̂2 (t))2 ]dt,
=
2 0
where X̂i (t) = E[Xi (t)|Y0T , M2 ], i = 1, 2. Noticing that X2 = X̂2 , we then have
I(M1 ; Y0T |M2 )

1
=
2

Z

T
0

E[(X1 (t) − X̂1 (t))2 ]dt,

which, together with (33), implies that R1 ≤ P1 /2. A completely parallel argument will
yield that R2 ≤ P2 /2.
The achievability part. In this part, we will show that as long as (R1 , R2 ) satisfying
0 ≤ R1 < P1 /2,

0 ≤ R2 < P2 /2,

(96)
(T )

we can find a sequence of (T, (eT R1 , eT R2 ), (P1 , P2 ))-codes with Pe → 0 as T → ∞. The
argument consists of several steps as follows.
Codebook generation: For a fixed T > 0 and ε > 0, assume that X1 and X2 are independent Ornstein-Uhlenbeck processes over [0, T ] with respective variances P1 −ε and P2 −ε, and
that (R1 , R2 ) satisfying (96). Generate eT R1 independent codewords X1,i , i ∈ {1, 2, . . . , eT R1 },
of length T , according to the distribution of X1 . Similarly, generate eT R2 independent codewords X2,j , j ∈ {1, 2, . . . , eT R2 }, of length T , according to the distribution of X2 . These
codewords (which may not satisfy the power constraint in (33)) form the codebook, which
is revealed to the senders and the receiver.
47

Encoding: To send message i ∈ M1 , sender 1 sends the codeword X1,i . Similarly, to send
j ∈ M2 , sender 2 sends X2,j .
(T )
Decoding: For any fixed ε > 0, let Tε denote the set of jointly typical (x1 , x2 , y) sequences, which is defined as follows:
Tε(T ) = {(x1 , x2 , y) ∈ C[0, T ] × C[0, T ] × C[0, T ] : | log ϕ1 (x1 , x2 , y) − IT (X1 , X2 ; Y )| ≤ T ε,
| log ϕ2 (x1 , x2 , y) − IT (X1 ; X2 , Y )| ≤ T ε, | log ϕ3 (x1 , x2 , y) − IT (X2 ; X1 , Y )| ≤ T ε},

where

ϕ1 (x1 , x2 , y) =
ϕ2 (x1 , x2 , y) =

dµX1 X2 Y
(x1 , x2 , y),
dµX1 X2 × µY
dµX1 X2 Y
(x1 , x2 , y),
dµX1 × µX2 Y

dµX1 X2 Y
(x1 , x2 , y).
dµX2 × µX1 Y
Here we remark that it is easy to check that the above Randon-Nykodym derivatives are all
well-defined; see, e.g., Theorem 7.7 of [51] for sufficient conditions for their existence. Based
on the received output y ∈ C[0, T ], the receiver chooses the pair (i, j) such that
ϕ3 (x1 , x2 , y) =

(x1,i , x2,j , y) ∈ Tε(T ) ,
if such a pair (i, j) exists and is unique; otherwise, an error is declared. Moreover, an error
will be declared if the chosen codeword does not satisfy the power constraint in (33).
Analysis of the probability of error: Now, for fixed T, ε > 0, define
Eij = {(X1,i , X2,j , Y ) ∈ Tε(T ) }.
By symmetry, we assume, without loss of generality, that (1,1) was sent. Define π (T ) to be
the event that
Z T
Z T
2
(X1,1 (t)) dt > P1 T,
(X2,1 (t))2 dt > P2 T.
0

0

(T )
P̂e ,

Then,
the error probability for the above coding scheme (where codewords violating
the power constraint are allowed), can be upper bounded as follows:
[
c
∪(i,j)6=(1,1) Eij )
P̂e(T ) = P (π (T ) ∪ E11
c
≤ P (π (T ) ) + P (E11
)+

So, for any i, j 6= 1, we have

X

P (Ei1 ) +

i6=1,j=1

X

P (E1j ) +

i=1,j6=1

X

P (Eij ).

i6=1,j6=1

c
) + eT R1 P (Ei1 ) + eT R2 P (E1j ) + eT R1 +T R2 P (Eij )
P̂e(T ) ≤ P (π (T ) ) + P (E11

Using the well-known fact that an Ornstein-Uhlenbeck process is ergodic [50, 47], we deduce
that P (π (T ) ) → 0 as T → ∞. And by Lemma E.3 and Theorem 6.2.1 in [38], we have
c
lim P ((X1,1 , X2,1 , Y ) ∈ Tε(T ) ) = 1 and thus lim P (E11
) = 0.

T →∞

T →∞

48

Now, we have for any i 6= 1,
P (Ei1 ) = P ((X1,i , X2,1 , Y ) ∈ Tε(T ) )
Z
=
dµX1 (x1 )dµX2 Y (x2 , y)
(T )
(x1 ,x2 ,y)∈Tε
Z
1
=
dµX1 X2 Y (x1 , x2 , y)
(T ) ϕ (x , x , y)
1 1
2
Tε
Z
≤
e−IT (X1 ;X2 ,Y )+εT dµX1 X2 Y (x1 , x2 , y)
(T )

Tε
−IT (X1 ;Y |X2 )+εT

=e

,

where we have used the independence of X1 and X2 , and the consequent fact that
IT (X1 ; X2 , Y ) = IT (X1 ; X2 ) + IT (X1 ; Y |X2 ) = IT (X1 ; Y |X2 ).
Similarly, we have, for j 6= 1,
P (E1j ) ≤ e−IT (X2 ;Y |X1 )+εT ,
and for i, j 6= 1,

P (Eij ) ≤ e−IT (X1 ,X2 ;Y )+εT .

It then follows that
c
P̂e(T ) ≤ P (π (T ) )+P (E11
)+eT R1 +εT −IT (X1 ;Y |X2 ) +eT R2 +εT −IT (X2 ;Y |X1 ) +eT R1 +T R2 +εT −IT (X1 ,X2 ;Y ) .

By Lemma E.1, one can choose independent OU processes X1 , X2 such that IT (X1 ; Y |X2 )/T →
(P1 − ǫ)/2, IT (X2 ; Y |X1 )/T → (P2 − ǫ)/2 and IT (X1 , X2 ; Y )/T → (P1 + P2 − 2ǫ) uniformly in
(T )
T . This implies that with ǫ chosen sufficiently small, we have P̂e → 0, as T → ∞. In other
words, there exists a sequence of good codes (which may not satisfy the power constraint)
with low average error probability. Now, from each of the above codes, we delete the worse
half of the codewords (any codeword violating the power constraint will be deleted since it
must have error probability 1). Then, with only slightly decreased transmission rate, the
remaining codewords will satisfy the power constraint and will have small maximum error
(T )
probability (and thus small average error probability Pe ), which implies that the rate pair
(R1 , R2 ) is achievable.
Remark E.4. The achievability part can be proven alternatively, which will be roughly
described as follows: for arbitrarily small ǫ > 0, by Lemma E.1, one can choose independent Ornstein-Uhlenbeck processes Xi with respective variances Pi − ǫ, i = 1, 2, such that
IT (Xi ; Y )/T approaches (Pi −ǫ)/2. Then, a parallel random coding argument with Xj , j 6= i,
being treated as noise at receiver i shows that the rate pair ((P1 − ǫ)/2, (P2 − ǫ)/2) can be
approached, which yields the achievability part.

49

F

Proof of Theorem 5.4

One of the important tools that plays a key role in discrete-time network information theory is
the entropy power inequality [14, 19], which can be applied to compare information-theoretic
quantities involving different users. The following lemma, which, despite its strikingly different form, serves the typical function of a discrete-time entropy power inequality.
Lemma F.1. Consider a continuous-time white Gaussian channel characterized by the following equation
Z t
√
X(s)ds + B(t), t ≥ 0,
Y (t) = snr
0

where snr ≥ 0 denotes the signal-to-noise ratio in the channel and M is the message to
be transmitted through the channel. Then, for any fixed T , IT (M; Y )/snr is a monotone
decreasing function of snr.
Proof. For notational convenience, in this proof, we write IT (M; Y ) as IT (snr). By Theorem
6.2.1 in [38], we have
snr
IT (snr) =
2

Z

T
0

E[(X(s) − E[X(s)|Y0s ])2 ]ds,

and Theorem 6 in [24], we have (the derivative is with respect to snr)
IT′ (snr)

1
=
2

Z

T
0

E[(X(s) − E[X(s)|Y0T ])2 ]ds.

It then follows that

′


IT (snr)
IT (snr)
1
′
I (snr) −
=
snr
snr T
snr
Z T

Z T
1
T 2
s 2
E[(X(s) − E[X(s)|Y0 ]) ]ds −
E[(X(s) − E[X(s)|Y0 ]) ]ds ≤ 0,
=
2snr
0
0
which immediately implies the lemma.
We are now ready for the proof of Theorem 5.4.
Proof of Theorem 5.4. For notational convenience only, we prove the case when n = 2, the
case when n is generic being parallel.
The converse part. Without loss of generality, we assume that
snr1 ≥ snr2 .
(T )

We will show that for any sequence of (T, (eT R1 , eT R2 ), P ) codes with Pe
the rate pair (R1 , R2 ) will have to satisfy
R2
P
R1
+
≤ .
snr1 snr2
2
50

→ 0 as T → ∞,
(97)

Fix T and consider the above-mentioned (T, (eT R1 , eT R2 ), P )-code. By the code construcT
tion, for i = 1, 2, it is possible to estimate the messages Mi from the channel output Yi,0
with an arbitrarily low probability of error. Hence, by Fano’s inequality, for i = 1, 2,
T
H(Mi |Yi,0
) ≤ T Ri Pe(T ) + H(Pe(T ) ) = T εi,T ,

where εi,T → 0 as T → ∞. It then follows that
T
T R1 = H(M1 ) = H(M1 |M2 ) ≤ I(M1 ; Y1,0
|M2 ) + T ε1,T ,

(98)

T
T R2 = H(M2 ) ≤ I(M2 ; Y2,0
) + T ε2,T .

(99)

By the chain rule of mutual information, we have
T
T
T
T
I(M1 , M2 ; Y2,0
) = I(M2 ; Y2,0
) + I(M1 ; Y2,0
|M2 ) ≥ I(M2 ; Y2,0
)+

snr2
T
I(M1 ; Y1,0
|M2 ), (100)
snr1

where, for the inequality above, we have applied Lemma F.1. Now, by Theorem 6.2.1 in [38],
we have
Z
Z
snr2 T
snr2 T
T
s
2
I(M1 , M2 ; Y2,0 ) =
E[(X(s) − E[X(s)|Y2,0]) ]ds ≤
E[X 2 (s)]ds,
2
2
0
0
which, together with (98), (99), (100) and (42), immediately implies the converse part.
The achievability part. We only sketch the proof of this part. For an arbitrarily small
ǫ > 0, by Theorem 6.4.1 in [38], one can choose an Ornstein-Uhlenbeck processes X̃ with
variance P − ǫ, such that IT (X̃; Yi )/T approaches snri (P − ǫ)/2. For any 0 ≤ λ ≤ 1, let
√
√
X(t) = λX1 (t) + 1 − λX2 (t), t ≥ 0,
where X1 and X2 are independent copies of X̃. Then, by a similar argument as in the
proof of Lemm E.1, we deduce that IT (X1 ; Y1 )/T, IT (X2 ; Y2 )/T approach snr1 λ(P − ǫ)/2,
snr2 (1 − λ)(P − ǫ)/2, respectively. Then, a parallel random coding argument as in the proof
of Theorem 5.1 such that
• when encoding, Xi only carries the message meant for receiver i;
• when decoding, receiver i treats Xj , j 6= i, as noise,
shows that the rate pair (snr1 λ(P − ǫ)/2, snr2 (1 − λ)(P − ǫ)/2) can be approached, which
immediately establishes the achievability part.
Remark F.2. For the achievability part, instead of using the power sharing scheme as in
the proof, one can also employ the following time sharing scheme: set X to be X1 for λ
fraction of the time, and X2 for 1 − λ fraction of the time. Then, it is straightforward to
check this scheme also achieves the rate pair (snr1 λ(P − ǫ)/2, snr2 (1 − λ)(P − ǫ)/2). This,
from a different perspective, echoes the observation in [49] that time sharing achieves the
capacity region of a white Gaussian BC as the bandwidth limit tends to infinity.

51

References
[1] I. Abou-Faycal, M. Trott and S. Shamai. The capacity of discrete-time memoryless
Rayleigh-fading channels. IEEE Trans. Info. Theory, vol. 47, no. 4, pp. 1290-1301,
2001.
[2] R. Ahlswede. Multi-way communication channels. IEEE ISIT, 1973.
[3] R. Ash. Capacity and error bounds for a time-continuous Gaussian channel. Information
and Control, vol. 6, pp. 14-27, 1963.
[4] R. Ash. Further discussion of a time-continuous Gaussian channel. Information and
Control, vol. 7, pp. 78-83, 1964.
[5] R. Ash. Information Theory, Wiley Interscience, New York, 1965.
[6] R. Baker and S. Ihara. Information capacity of the stationary Gaussian channel. IEEE
Trans. Info. Theory, vol. 37, no. 5, pp. 1314-1326, 1991.
[7] P. Bergmans. Random coding theorem for broadcast channels with degraded components. IEEE Trans. Info. Theory, vol. 19, pp. 197-207, 1973.
[8] P. Bethoux. Test et estimations concernant certaines functions aleatoires en particulier
Laplaciennes. Ann. Inst. Henri Poincare, vol. 27, pp. 255-322, 1962.
[9] S. Bross and A. Lapidoth. An improved achievable region for the discrete memoryless
two-user multiple-access channel with noiseless feedback. IEEE Trans. Info. Theory,
vol. 51, no. 3, pp. 811-833, 2005.
[10] T. Chan, S. Hranilovic and F. Kschischang. Capacity-Achieving probability measure for
conditionally Gaussian channels with bounded inputs. IEEE Trans. Info. Theory, vol.
51, no. 6, pp. 2073-2088, 2005.
[11] T. Cover. Broadcast channels. IEEE Trans. Info. Theory, vol. 18, pp. 2-14, 1972.
[12] T. Cover. Some advances in broadcast channels. Advances in Communication Systems,
Academic Press, San Francisco, vol. 4, pp. 229-260, 1975.
[13] T. Cover and C. Leung. An achievable rate region for the multiple-access channel with
feedback. IEEE Trans. Info. Theory, vol. 27, no. 3, pp. 292-298, 1981.
[14] T. Cover and J. Thomas. Elements of Information Theory, 2nd ed., Wiley Interscience,
New York, 2006.
[15] T. Duncan. On the calculation of mutual information. SIAM Journal on Applied Mathematics, vol. 19, no. 1, pp. 215-220, 1970.
[16] R. Durrett. Probability: Theory and Examples, 4th ed., Cambridge Series in Statistical
and Probabilistic Mathematics, Cambridge University Press, Cambridge, 2010.

52

[17] A. Dytso, M. Goldenbaum, S. Shamai and V. Poor. Upper and lower
bounds on the capacity of amplitude-constrained MIMO channels. Preprint,
https://www.princeton.edu/˜mg27/globecom17proofs.pdf
[18] A. El Gamal. The capacity of the physically degraded Gaussian broadcast channel with
feedback. IEEE Trans. Info. Theory, vol. 27, no. 4, pp. 508-511, 1981.
[19] A. El Gamal and Y. Kim. Network Information Theory, Cambridge University Press,
New York, 2011.
[20] J. Fahs and I. Abou-Faycal. Using Hermite bases in studying capacity-achieving distributions over AWGN channels. IEEE Trans. Info. Theory, vol. 58, no. 8, pp. 53025322,
2012.
[21] R. Fortet. Hypothesis testing and Estimation for Laplacian Functions. Fourth Berkeley
Symposium on Mathematical Statistics and Probability, vol. 1, pp. 289-305, 1961.
[22] R. Gallager. Information Theory and Reliable Communication, Wiley, New York, 1968.
[23] A. Gelfand and I. Yaglom. Calculation of the amount of information about a random
function contained in another such function. Uspekhi Mat. Nauk, vol. 12, pp. 3-52, 1957.
Trans. in Am. Math. Sot. Trans., Ser. 2, vol. 12, pp. 199-247, 1959.
[24] D. Guo, S. Shamai and S. Verdu. Mutual information and minimum mean-square error
in Gaussian channels. IEEE Trans. Info. Theory, vol. 51, no. 4, pp. 1261-1282, 2005.
[25] G. Han. Limit theorems in hidden Markov models. IEEE Transactions on Info. Theory,
vol. 59, no. 3, pp. 1311-1328, 2013.
[26] G. Han. A randomized algorithm for the capacity of finite-state channels. IEEE Transactions on Info. Theory, vol. 61, no. 7, pp. 3651-3669, 2015.
[27] G. Han and J. Song. Extensions of the I-MMSE relationship to Gaussian channels with
feedback and memory. IEEE Trans. Info. Theory, vol. 62, no. 10, pp. 5422-5445, 2016.
[28] T. Hida. Canonical representations of Gaussian processes and their applications, Memoirs of the College of Science, University of Kyoto, Series A: Mathematics, vol. 33, no.
1, pp. 109-155, 1960.
[29] T. Hida and M. Hitsuda. Gaussian processes, vol. 120, American Mathematical Soc.,
1993.
[30] M. Hitsuda. Mutual information in Gaussian channels. J. Multivariate Anal., vol. 4, pp.
66-73, 1974.
[31] M. Hitsuda and S. Ihara. Gaussian channels and the optimal coding. J. Multivariate
Anal., vol. 5, pp. 106-118, 1975.
[32] R. Huang and R. Johnson. Information capacity of time-continuous channels. IEEE
Trans. Info. Theory, IT-8, pp. l91-l98, 1962.
53

[33] R. Huang and R. Johnson. Information transmission with time-continuous random processes. IEEE Trans. Info. Theory, vol. 9, no. 2, pp. 84-94, 1963.
[34] P. Kloeden and E. Platen. Numerical Solution of Stochastic Differential Equations,
Stochastic Modelling and Applied Probability, vol. 23, Springer-Verlag, Berlin Heidelberg, 1992.
[35] I. Ibragimov and Y. Rozanov. Gaussian Random Processes, Springer-Verlag, New York,
1978.
[36] S. Ihara, S. On the capacity of the continuous time Gaussian channel with feedback. J.
Multivariate Anal., vol. 10, pp. 319-331, 1980.
[37] S. Ihara. Capacity of mismatched Gaussian channels with and without feedback. Probability Theory Rel. Fields, vol. 84, pp. 453-471, 1990.
[38] S. Ihara. Information Theory for Continuous Systems, World Scientific, Singapore, 1993.
[39] S. Ihara. Coding theorems for a continuous-time Gaussian channel with feedback. IEEE
Trans. Info. Theory, vol. 40, no. 6, pp. 2014-2045, 1994.
[40] S. Ihara. Mutual information in stationary channels with additive noise. IEEE Trans.
Info. Theory, vol. 31, no. 5, pp. 602-606, 1985.
[41] B. Jacob, M. Zakai and J. Ziv. On the ε-entropy and the rate-distortion function of
certain non-Gaussian processes. IEEE Trans. Info. Theory, vol. 20, no. 4, pp. 517-524,
1974.
[42] T. Kadota, M. Zakai and J. Ziv. Mutual information of the white Gaussian channel
with and without feedback. IEEE Trans. Info. Theory, vol. 17, pp. 368-371, 1971.
[43] I. Karatzas and S. Shreve. Brownian Motion and Stochastic Calculus, New York:
Springer-Verlag, 1991.
[44] Y. Kim. Gaussian Feedback Capacity, Ph.D. Thesis, Stanford University, 2006.
[45] Y. Kim. Feedback capacity of the first-order moving average Gaussian channel. IEEE
Trans. Info. Theory, vol. 52, no. 7, pp. 3063-3079, 2006.
[46] Y. Kim. Feedback capacity of stationary Gaussian channels. IEEE Trans. Info. Theory,
vol. 56, no. 1, pp. 57-85, 2010.
[47] Y. Kutoyants. Statistical inference for ergodic diffusion processes, Springer, London,
2004.
[48] A. Lapidoth. On the asymptotic capacity of stationary Gaussian fading channels. IEEE
Trans. Info. Theory, vol. 51, no. 2, pp. 437-446, 2005.
[49] A. Lapidoth, E. Telatar and R. Urbanke. On wide-band broadcast channels. IEEE
Trans. Info. Theory. vol. 49, no. 12, pp. 3250-3258, 2003.
54

[50] A. Leon-Garcia. Probability, Statistics, and Random Processes for Electrical Engineering, Pearson/Prentice Hall, N.J., 2008
[51] R. Liptser and A. Shiryaev. Statistics of random processes (I): General theory, 2nd
edition, Springer-Verlag, Berlin, 2001.
[52] X. Liu and G. Han. Recent results in continuous-time network information theory. IEEE
ISIT, pp. 2421-2425, 2014.
[53] X. Mao. Stochastic Differential Equations and Applications, Horwood, 1997.
[54] H. Nyquist. Certain factors affecting telegraph speed. The Bell System Technical Journal, pp. 324, 1924.
[55] B. Oksendal. Stochastic Differential Equations: An Introduction with Applications,
Springer, Berlin, 1995.
[56] L. Ozarow. The capacity of the white Gaussian multiple access channel with feedback.
IEEE Trans. Info. Theory, vol. 30, no. 4, pp. 623-628, 1984.
[57] L. Ozarow and S. Leung. An achievable region and outer bound for the Gaussian broadcast channel with feedback. IEEE Trans. Info. Theory, vol. 30, no. 4, pp. 667-671, 1984.
[58] L. Ozarow, A. Wyner and J. Ziv. Achievable rates for constrained Gaussian channel.
IEEE Trans. Info. Theory, vol. 35, no. 5, pp. 1079-1084, 1985.
[59] M. Pinsker. Information and Information Stability of Random Variables and Processes,
Holden-Day, San Francisco, 1964.
[60] H. Poor. An introduction to signal detection and estimation, Springer-Verlag, New York,
1994.
[61] M. Raginsky. On the information capacity of gaussian channels under small peak power
constraints. Proceedings of the 46th Allerton Conf. on Commun., Control, & Comp.,
pp. 286-293, 2008.
[62] H. Royden. Real analysis, 4th edition, Prentice Hall, Boston, 2010.
[63] J. Schalkwijk and T. Kailath. Coding scheme for additive noise channels with feedback
I: No bandwidth constraint. IEEE Trans. Info. Theory, vol. 12, no. 2, pp. 172-182, 1966.
[64] S. Shamai and I. Bar David. Upper bounds on capacity for a constrained Gaussian
channel. IEEE Trans. Info. Theory, vol. 35, no. 5, pp. 1079-1084, 1989.
[65] S. Shamai and I. Bar-David. The capacity of average and peak-power-limited quadrature
Gaussian channels. IEEE Trans. Info. Theory, vol. 41, no. 4, pp. 1060-1071, 1995.
[66] C. Shannon. A mathematical theory of communication. The Bell System Technical Journal, vol. 27, pp. 379-423, 1948.

55

[67] C. Shannon. Communication in the presence of noise. Proc. IRE, vol. 37, pp. 10-21,
1949.
[68] N. Sharma and S. Shamai. Transition points in the capacity-achieving distribution for
the peak-power limited AWGN and free-space optical intensity channels. Probl. Inf.
Transm., vol. 46, no. 4, pp. 283-299, 2010.
[69] D. Slepian. On Bandwidth. Proc. IEEE, vol. 64, no. 3, 1976.
[70] J. Smith. The information capacity of amplitude- and variance-constrained scalar Gaussian channels. Inform. Control, vol. 18, pp. 203-219, 1971.
[71] T. Weissman. The relationship between causal and non-causal mismatched estimation
in continuous-time AWGN channels. IEEE Trans. Info. Theory, vol. 56, no. 9, pp. 42564273, 2010.
[72] T. Weissman, Y. Kim and H. Permuter. Directed information, causal estimation, and
communication in continuous time. IEEE Trans. Info. Theory, vol. 59, no. 3, pp. 12711287, 2013.
[73] F. Willems. The feedback capacity region of a class of discrete memoryless multiple
access channels. IEEE Trans. Info. Theory, vol. 28, no. 1, pp. 93-95, 1982.
[74] W. Wu, S. Vishwanath and A. Arapostathis. On the capacity of multiple access channels with state information and feedback. IEEE Trans. Info. Theory, submitted for
publication.
[75] A. Wyner. The capacity of the band-limited Gaussian channel, The Bell System Technical Journal, pp. 359-395, 1966. Also reprinted in Key Papers in the Development of
Information Theory, edited by D. Slepian, IEEE Press, New York, pp. 190-193, 1974.
[76] A. Wyner. Recent results in the Shannon theory. IEEE Trans. Info. Theory, vol. 20, no.
1, pp. 2-10, 1974.

56

