1

On the Design of Channel Shortening
Demodulators for Iterative Receivers in Linear
Vector Channels
arXiv:1506.07331v4 [] 21 Dec 2017

Sha Hu and Fredrik Rusek

Abstract
We consider the problem of designing demodulators for linear vector channels with memory that
use reduced-size trellis descriptions for the received signal. We assume an overall iterative receiver,
and use interference cancellation (IC) based on the soft information provided by the outer decoder, to
mitigate the parts of the signal that are not covered by the reduced-size trellis description. In order to
reach a trellis description, a linear filter is applied as front-end to compress the signal structure into
a small trellis. This process requires three parameters to be designed: (i) the front-end filter, (ii) the
feedback filter through which the IC is done, and (iii) a target response which specifies the trellis.
Demodulators of this form have been studied before under then name channel shortening (CS), but
the interplay between CS, IC and the trellis-search process has not been adequately addressed in the
literature. In this paper, we analyze two types of CS demodulators that are based on the Forney and
Ungerboeck detection models, respectively. The parameters are jointly optimized based on a generalized
mutual information (GMI) function. We also introduce a third type of CS demodulator that is in general
suboptimal, but has closed-form solutions. Moreover, signal to noise ratio (SNR) asymptotic properties
are analyzed and we show that the third CS demodulator asymptotically converges to the optimal CS
demodulator in the sense of GMI-maximization.

Index Terms
Channel shortening, intersymbol interference, prefilter, front-end filter, feedback filter, target response, generalized mutual information, achievable information rate, Forney model, Ungerboeck model.

The authors are with the Department of Electrical and Information Technology, Lund University, Lund, Sweden (email:
firstname.lastname@eit.lth.se).
This paper has been presented in part [53] at the IEEE 26th annual international symposium on personal, indoor and mobile
radio communications (PIMRC), Hongkong, Sep. 2015.

2

I. I NTRODUCTION
Channel shortening (CS) demodulators have a long and rich history, see [1]–[12], [61]. For
intersymbol interference (ISI) channels, Forney [13] showed that the Viterbi Algorithm (VA)
[14] implements maximum likelihood (ML) detection. However, the complexity of the VA is
exponential in the memory of the channel which prohibits its use in many cases of interest. As
a remedy, Falconer and Magee proposed in 1973 the concept of CS [1]. The concept is to filter
the received signal with a prefilter so that the effective channel has much shorter duration than
the original channel, and then apply the VA to the shorter effective channel.
Traditionally, CS demodulators have been optimized from a minimum mean square error
(MMSE) perspective [2]–[10]. Two exceptions from this are the papers [11] and [12]. In [11],
the authors attempt to minimize the error probability of an uncoded system which leads to a new
notion of posterior equivalence between the target response and the filtered channel. However,
since [11] works with uncoded error probabilities, the analysis in [11] does not adequately
address the case of coded systems and Shannon capacity properties. The first paper that works
with capacity-related cost measures is [12]. In [12] the authors consider the achievable rate, in
the form of generalized mutual information (GMI) [15]–[19], that the transceiver system can
achieve if a CS demodulator is adopted. However, [12] is limited to ISI channels only, and the
design method in [12] of the CS demodulator is in fact not always possible to execute. The
limitations of [12] were first dealt with in [18], which extended the CS concept to any linear
vector channel and resulted in a closed-form optimization procedure.
Iterative receivers such as turbo equalization [51]–[56] followed as a natural extension to
turbo codes as an iterative technique for detection and decoding of forward error correction
(FEC) protected data that is transmitted over dispersive channel. However, when it comes to
turbo equalization, common settings of the equalizer are [51] the maximum a posterior (MAP)
demodulator [23] and its suboptimal variants such as dimension-reduction and subspace based
detections [63], [64], and MMSE based approaches [52], [55], [56], [65] that replace the MAP
demodulator with a linear equalizer or a decision feedback equalizer (DFE) to reduce the
prohibitive complexity of the MAP demodulator. One important open problem in the area of
turbo equalization is the development of other non-trellis-based detection methods that provide
performance between that of MAP and MMSE performance [43], [51]. Instead of fully removing
the trellis-based detection, another possible approach is to reduce the memory size of the original

3

linear vector channel through an interference cancellation (IC) based prefiltering. To the best of
our knowledge, there is only limited literature [54], [62] on such a design of demodulator that
combines both IC based prefiltering and a memory-size shortened BCJR in iterative receiver
design. A closely related concept is delayed-decision-feedback-sequence-estimation (DDFSE)
[21], [57], which also reduces the number of states in the BCJR. However, in DDFSE the IC is
done within a single iteration, and not between the iterations of an iterative receiver.
In this paper, we generalize the idea in [18] of GMI-maximization based CS demodulators to
iterative receivers. With iterative receivers it is reasonable to expect that better performance can
be reached by allowing the parameters of the CS demodulator to change in each iteration. The
CS demodulator in [18] does not take the prior information into account, rendering its design
static in all iterations. We aim at constructing a CS demodulator that takes soft information
provided by the outer decoder into account so that the parameters of the CS demodulator are
designed for a particular level of prior knowledge. This procedure includes an IC mechanism
to deal with the signal part that can not be handled by the trellis-search. Preliminary results for
CS demodulators in iterative receivers are available in [20], but this paper non-trivially advances
the state-of-the-art.
Although the trellis-search based detection is still utilized in the CS demodulator, the memory
size ν of the linear vector channel has been reduced which results in significant complexity
reduction compared to the MAP demodulator. Meanwhile, with different values of ν, the CS
demodulator provides trade-off between the performance of MMSE and MAP. As will become
clear later, the CS demodulator is closely related to the concept of linear MMSE receiver with
parallel interference cancellation (LMMSE-PIC) [26]–[28], which cooperates the soft information
into the filter coefficients and interference cancellation process. With setting ν = 0, the CS
demodulator is identical to the LMMSE-PIC demodulator whose trellis-search process is trivial
since different symbols are assumed to be independent after the front-end filtering. The CS
demodulator can also be viewed as an extension of the LMMSE-PIC to include a trellis-search,
where the parameters of the front-end filter, IC, and trellis-search are jointly optimized. On the
other hand, by setting ν to be equal to the original memory size of the linear vector channel, the
CS demodulator is identical to MAP. Therefore, the CS demodulator is a generalized framework
that includes both the MAP and LMMSE-PIC in iterative receiver design.
The rest of the paper is organized as follows: The linear vector channel model and the iterative
receiver structure are introduced in Sec. II, while the general form of the CS demodulators and

4

the GMI are described in Sec. III. In Sec. IV we analyze three types of CS demodulators for finite
length linear vector channels. In Sec. V we deal with ISI channels as asymptotic versions of the
results established in Sec. IV. The signal to noise ratio (SNR) asymptotic of the CS demodulators
are discussed in Sec. VI. Empirical results are provided in Sec. VII, and Sec. VIII summarizes
the paper. For improved readability, we have deferred some long proofs and derivations to
Appendices A-K.
Notation: Throughout the paper, a capital bold letter such as A represents a matrix, a lower
case bold letter a represents a vector, and a capital letter A represents a number. The expression
A ≺ 0 means matrix A is negative definite, while A  0 means A is positive definite. Matrix
I represents the identity matrix and in general the dimension will be omitted; when it cannot
be understood from the context, we let I K represent a K ×K identity matrix. Our superscripts
have the following meanings: ( )∗ is complex conjugate, ( )T is matrix transpose, ( )† denotes the
conjugate transpose of a matrix, ( )−1 is matrix inverse. In addition, ∝ means proportional to,
E[ ] is the expectation operator, Tr( ) takes the trace of a matrix, R{ } returns the real part of a
variable, ⊗ is the Kronecker multiplication operator, vec(A) is a column vector containing the
columns of matrix A stacked on top of each other, and [A, B] is the set of integers {k : A ≤ k ≤ B}.
Furthermore, we say that a matrix A is banded within diagonals [−ν1 , ν2 ] (ν1 , ν2 ≥ 0), if the
(k, `)th element A(k, `) satisfies1
A(k, `) = 0, ` − k > ν1 or k − ` > ν2 .
Moreover, we define two matrix operators [ ]ν and [ ]\ν such that A = [A]ν +[A]\ν , with [A]ν
banded within diagonals [−ν, ν] where [A]\ν is constrained to zero.
II. S YSTEM M ODEL
We consider linear vector channels according to
y = Hx+n

(1)

where y is an N ×1 vector of received signal, x is a K×1 vector comprising unit energy coded
symbols that belong to a constellation X , H is an N×K matrix representing the communication
channel which is perfectly known to the receiver and n is zero-mean complex Gaussian noise
1

Note that ν1 refers to the number of upper diagonals of A that are nonzero. We have this convention in order to subsequently

follow standard notation for Toeplitz matrices [41].

5

vector with covariance matrix N0 I. Model (1) may represent many different communication
systems, such as for example multi-input multi-output (MIMO) or ISI channels. In the MIMO
case, the variables N and K are finite while they grow without bounds in the ISI case. For the
former case, a block fading model is assumed, where the coherence time is infinite. The block
fading model allows us to perform an analysis for a single symbol period.
Denote xk as the kth element of x and hk as the kth column vector of H, (1) can be rewritten
as
y=

K−1
X

hk xk +n.

(2)

k=0

In an iterative receiver, the feedback from the outer decoder can be utilized in the demodulator
to improve the performance. As the outer decoder provides the demodulator with a posteriori
probability (APP) and extrinsic information (in terms of bit log-likelihood ratios (LLRs)) [22],
[60], side information is present about the symbols x and we represent this by the probability
mass function pk (s) = P(xk = s), (0 ≤ k ≤ K − 1). Note that the side-information does not
consider the dependency among the symbols, but are symbol-wise marginal probabilities. This
reflects the situation encountered in iterative receivers with perfect interleaving. In those cases,
the prior probabilities provided from previous iterations are assumed independent, i.e., P(x =
Q
s) = pk (s). Due to the perfect interleaving assumption, the demodulator can compute x̂ =
Ep(x) [x] = [ x̂0 , x̂1 , · · · , x̂K−1 ]T in a per-entry fashion as
X
spk (s),
x̂k =
s∈X

where the expectations are computed with respect to the prior distribution pk (s).
With soft information x̂, we define a K ×K diagonal matrix P as follows. For finite length
linear vector channels, P equals


P = ET Ep(x) [xx̂† ] = ET [x̂x̂† ],

(3)

where the exception “ET ” is taken over the transmitted blocks of x under the block fading
assumption. For ISI case, as the whole data block experiences the same channel, we let
P = αI,

(4)

K−1
1 X
|x̂k |2 .
α=
K k=0

(5)

where the scalar

6

CS Parameter
Opmizaon

Interference
Cancelaon

BCJR

Extrinsic in

LLR

+

Decoder

APP out

So
Informaon
Generaon

Extrinsic out

+

CS Demodulator

Fig. 1.

Iterative receiver structure with CS demodulator and outer decoder. The target of the CS demodulator is to maximize

the GMI through jointly optimizing the parameters V , R and G, which are referred to as the front-end filter, IC matrix and
trellis representation matrix, respectively.

The variable P in (3) can alternatively be written as




P = ET [x̂x̂† ] = ET Ep(x) [xx† ] − ET cov(x) .

(6)

Under the natural assumption of soft information that satisfies


ET Ep(x) [xx† ] = I,

(7)

it follows that 0  P  I, and the same also holds for ISI case. The variable P reflects the
accuracy of the side information. That is, when there is no soft information available, we have
P = 0, while with perfect feedback we get P = I.
The task of the demodulator is to generate soft information about the symbols in x given the
observable y and the side information {pk (s)}. The optimal demodulator is the MAP demodulator
[23], [24] which evaluates the posterior probabilities P(xk = s|y). However, the number of leaves
of the search tree corresponding to the MAP demodulator is in general |X |K which is prohibitive
for most practical applications. The purpose of the CS demodulator is to force the signal model
to be an lower triangular matrix with only ν +1 (0 ≤ ν < K −1) nonzero diagonals by means
of a linear filter2 , where ν is referred to as the memory size of the CS demodulator. Then, a
BCJR [25] demodulator can be applied over a trellis with |X |ν states. Moreover, since there is
side information present about x, the parts of H that are outside the memory of the BCJR can
be partly eliminated by means of IC through the prior mean x̂.
2

For finite length linear vector channels such as MIMO channel, “filtering” means matrix multiplication.

7

The structure of an iterative receiver utilizing a CS demodulator is depicted in Fig. 1. The
extrinsic information from the outer decoder is used to compute an estimate x̂ and a matrix P
that indicates the feedback quality. Based on the updated P in each iteration, the optimal CS
parameters are found by maximizing the GMI. A prefiltering and IC process are then implemented
on y with optimal V and R to obtain the signal ŷ, which is sent to a memory ν BCJR module
specified by an optimal G. Moreover, the extrinsic information iteratively exchanged between
the BCJR and the outer decoder is also used as a priori information for the transmitted symbols.
Note that if we set ν = K − 1, the search space of the CS demodulator is no longer a trellis
but corresponds to the original tree and is therefore equivalent to MAP, and LMMSE-PIC is a
special case of the CS demodulation with ν = 0.
III. T HE G ENERAL F ORM OF THE CS D EMODULATOR
We state two lemmas that will be useful later, and Lemma 2 can be verified straightforwardly.
Lemma 1. Let A1 and A2 be two K ×K matrices, where A1 is invertible and banded within
diagonals [−ν, ν]. If [A−1
1 ]ν = [A2 ]ν , then


Tr A1 A2 = Tr I .
Proof. Let A3 = A2 −A−1
1 , then [A3 ]ν = 0 and A3 = [A3 ]\ν . As A1 = [A1 ]ν , the elements along


the main diagonal of A1 A3 are zero. Therefore Tr A1 A2 = Tr A1 (A−1

1 +A3 ) = Tr(I).
Lemma 2. Let A1 and A2 be two K ×K matrices that are banded within diagonals [−ν1 , ν2 ]
and [−ν3 , ν4 ], respectively. Then the product A1 A2 is banded within diagonals [max(−(ν1 +
ν3 ), 1 − K), min(ν2 +ν4 , K − 1)].
A. System Model of the CS Demodulator
The CS demodulators that we investigate operate on the basis of the mismatched3 function
p̃(y|x) = exp 2R{x† (V y−Rx̂)}−x† Gx
3



(8)

By “mismatched” we mean that p̃(y|x) may not be a valid probability distribution function and in general differs from the

true conditional probability distribution function p(y|x) even with x̂ = 0, but such a “mismatched” property is for the purpose
of reducing the size of trellis description in the BCJR.

8

instead of the true conditional probability


1
ky−Hxk2
p(y|x) =
.
exp −
(πN0 )N
N0

(9)

The matrices V , R and G are the front-end filter, IC matrix, and trellis representation matrix,
respectively. Without loss of generality, we have absorbed N0 into V , R, and G. Models (8)
and (9) are equivalent for demodulation if we set V = H † /N0 , R = 0, and G = H † H/N0 , in
which case the CS demodulator represents the MAP demodulator.
The detection model (8) has its roots in Falconer and Magee’s paper [1] with adding an IC
step, where the system model of the demodulator is described as
T̃ (y|x) = exp −kW y − T x̂ − F xk2



(10)

By setting T = 0, we obtain the same system model as in [1]. If identifying V = F † W , R = F † T ,
and G = F † F , model (10) is equivalent to (8) since

T̃ (y|x) ∝ exp 2R{x† (F † W y − F † T x̂)}−x† F † F x

= exp 2R{x† (V y − Rx̂)} − x† Gx .
The detection model (10) is usually denoted as “Forney” model [1] due to its Euclidean-distance
form, while the more general model (8) is called “Ungerboeck” model [32], [36], [37]. An
advantage of the Ungerboeck model over the Forney model is that the parameter optimization
through GMI-maximization is simpler [18]. However, as both models can be viewed as “natural”
CS demodulators, we shall investigate both in CS demodulator design for iterative receivers.
In order to optimize (V , R, G), we choose to work with the GMI which is an achievable rate
for a receiver that operates on the basis of a mismatched version of the channel law. The GMI
in nats/channel is defined as
IGMI = −Ep(y) [log p̃(y)]+Ep(y,x) [log p̃(y|x)]

(11)

R
where p̃(y) = (1/π K ) p̃(y|x) exp(−kxk2 )dx and the expectation is taken over the true statistics
p(y) and p(y, x). Although finite constellations X are almost always used in practice, they are
hard to analyze. In order to obtain a mathematically tractable problem, here we use a zero-mean,
unit variance, complex Gaussian constellation for each entry of x. With Gaussian inputs, the
trellis discussed earlier has no proper meaning as the number of states is infinite even for finite
ν. However, the Gaussian assumption is only made in order to design the receiver parameters.
We first state Theorem 1 which shows the calculation of the GMI for model (8).

9

Theorem 1. The GMI for the detection model (8) equals


IGMI (V , R, G) = log det(I +G) −Tr(G)+2R Tr(V H −RP )


−Tr (I +G)−1 V (N0 I +HH † )V † −2R V HP R† +RP R† .

(12)

The proof of Theorem 1 is given in Appendix A. Here we make the same assumption as in
[18] that I + G is positive definite, otherwise the GMI is not well defined. With any parameters
(V , R, G), the GMI can be calculated in (12), although they may not be optimal in the sense
GMI-maximization. We illustrate Theorem 1 with two examples.
Example 1. Extended Zero-Forcing filter (EZF). We extend the zero-Forcing filter [30] to only
partly invert the channel so that a trellis-search is necessary after the EZF front-end filter. In
view of the CS demodulator, we can select the parameters in (8) as:
V = (I +G)(H † H)−1 H † , R = 0,
and then optimize (12) over G. To satisfy the constraint of having a trellis with |X |ν states, we
should have G = [G]ν . The optimal G, in the sense of maximizing (12), will be shown (Theorem
2) to satisfy
[(I +G)−1 ]ν = N0 [(H † H)−1 ]ν .
Utilizing Lemma 1, the GMI in (12) for the optimal G equals



IGMI = log det(I +G) +Tr I −N0 (H † H)−1 (I +G) = log det(I +G) .
Example 2. Truncated Matched filter (TMF). As previously mentioned, the MAP demodulator
(9) can be written in the form (8) by setting V = H † /N0 , R = 0 and G = H † H/N0 . The
front-end is in this case a matched filter [31] and the BCJR needs to be implemented over the
Ungerboeck model [32]. To reach a trellis with |X |ν states, we can truncate G to its center
2ν +1 diagonals, i.e., we can use the following parameters in (8):
V = H † /N0 , R = 0, and G = [H † H/N0 ]ν .
With these choices, the GMI in (12) equals

−1

IGMI = log det(I +[H † H/N0 ]ν ) −Tr H † H N0 I +[H † H]ν [H † H]\ν .

10

H ,N0 , P

y , xˆ

Fig. 2.

Parameter
Optimization
and
Interference
Cancelation

G

yˆ  Vy  Rxˆ

BCJR

CS demodulator that maximizes the GMI based on the tuple (ŷ, x).

B. Constraints on the Parameter R for the CS Demodulator
As mentioned earlier, optimization of the demodulator will be made on the basis of GMI which
is evaluated for the statistical model of the tuple (x, ŷ). As illustrated in Fig. 2, our approach to
design a CS demodulator consists of two steps:
•

Construction of a signal ŷ = V y−Rx̂ based on the received signal y and prior mean x̂;

•

BCJR demodulation of ŷ operating on a reduced number of states |X |ν .

This procedure is fully analogous to LMMSE-PIC demodulator which first subtracts the
interference, applies a Wiener filter, and concludes by a BCJR that operates with a diagonal
matrix G. The statistical behavior of (ŷ, x) may be superior to that of the original (y, x) as the
former tuple corresponds to a statistically different channel than the true one. As what will be
shortly shown in Example 3, the GMI obtained with tuple (ŷ, x) based on perfect feedback x̂ can
be infinitely large, which exceeds the channel capacity with the original tuple (y, x). Therefore,
the computed value of GMI may have little relevance for the performance of the transceiver
system. In order for GMI to have bearing on performance, it is critical to put constraints on R
as the next example will show.
Example 3. Let the system model be
y =x+n
with noise density N0 , and y, x, n are K×1 vectors. Assume perfect feedback information, i.e.,
x̂ = x. The demodulator parameters are taken as V = 0, R = −(1+β)I, and G = βI, β an
arbitrary positive real value, then the statistical model for ŷ is
ŷ = V y − Rx̂ = (1+β)x.
The GMI in (12) for the tuple (x, ŷ) is

IGMI (V , R, G) = K 1+log(1+β) .

11

 1

0
.
.
.

K   1

0

0

0

0

.
.
.

0

0
0
 1

0

0

.
.
.

(a)

0

0

.
.
.

0
. 0
.
.
0

.
.
.

0

 1
(b)

(c)

Fig. 3. Three different types of shape of matrix R, where ν is the memory size of F or G, i.e., the memory size of the BCJR.

In order to maximize the GMI, the demodulator will choose β → ∞ to make IGMI infinite.
This is because, except for using the feedback information for IC, the demodulator uses the
prior mean x̂ as a signal energy via R. A demodulator equipped with these parameters will
have significant error propagation and does not have much operational meaning for an iterative
receiver. Thus, we conclude that unless constraints are put on R, the GMI value is not relevant.
Three typical shapes of R are specified in Fig. 3. All three have in common that rather than
adding signal energy, the rationale of R should be to remove interference. Therefore at the very
minimum the diagonal elements of R should be constrained to zero, so that the demodulation of
each symbol in x does not rely on its own prior mean x̂. Such a constraint is perfectly aligned
with the operations of LMMSE-PIC, where x̂` is not used for demodulation of x` . Furthermore,
the rationale of the constraints we impose on R is to follow the principle of extrinsic information:
The BCJR module should not rely on the prior information x̂` when demodulating x` (this
requires more than just the diagonal of R to be zero).
We point out that the fact that the GMI can exceed the channel capacity is a consequence of
our choice not to include the side information as a prior distribution on x when evaluating the
GMI. If we did, then the GMI is decaying with increasing quality of the side information (due to
the mutual information I(x, y|x̂) goes to 0 as x̂ becomes perfect). Finally, we acknowledge the
fact that a permutation of the columns of H can boost the performance of the CS demodulator
whenever 0 < ν < K − 1 for finite length linear vector channels. However, minimum-phase
conversions of ISI channels are not beneficial as we will solve for the optimal front-end filter.

12

IV. PARAMETER O PTIMIZATION FOR F INITE L ENGTH L INEAR V ECTOR C HANNEL
In this section, we elaborate the parameter optimization for finite length linear vector channels.
We introduce three different methods, Method I, Method II and Method III. We start with the
classical Forney model (10) based demodulator, i.e., Method I, and then extend the demodulation
model into the Ungerboeck model (8), i.e., Method II. As both Method I and Method II need
gradient-based approach for the optimization of target response, by carefully examining the
properties of the CS demodulator with Ungerboeck model, we propose a suboptimal Method III
which has an explicit construction based on an LMMSE-PIC and all parameters are in closedforms.
A. Method I
In Method I, the CS demodulator is based on detection model (10) and the following structures
of the CS parameters (W , T , F ) are imposed:
•

W is a K ×N matrix with no constraints.

•

F is a K ×K lower triangular matrix where only the main diagonal and the first ν lower
diagonals are nonzero, i.e., F is banded within diagonals [0, ν] (0 ≤ ν < K −1), where ν
is denoted as the memory size of F . Moreover, the main diagonal of F is constrained to
only contain positive real values.

•

T is a K ×K matrix that is constrained to be zero wherever F can take nonzero values.

The constraint of F is to shorten the memory for the trellis-search in BCJR, while the purpose
of the constraint on T is to cancel the signal part that F can not handle. From Theorem 1, and
by identifying V = F † W , R = F † T , and G = F † F , the GMI in (12) of Method I equals



IGMI (W , T , F ) = log det(I +F † F ) −Tr(F † F )+2R Tr F † (W H −T P )

−Tr (I +F † F )−1 L1

(13)

where

L1 = F † W (N0 I +HH † )W † F −2R F † W HP T † F +F † T P T † F .
With the aforementioned constraints on F and T , the matrix R = F † T has a form of shape
(a) in Fig. 3. That is, all diagonal elements are zero as well as the lower triangular part of the
(ν +1)×(ν +1) small matrix at the right bottom corner.

13

In order to optimize (13) over (W , T , F ), we first introduce an S ×K 2 indication matrix Ω
only consisting of ones and zeros4 , having a single 1 in each row, and S equals the number
of elements in T that are allowed to be nonzero. Let I(vec(T )) be a vector that contains the
positions where the vector vec(T ) is allowed to be nonzero. Then the value of the kth entry in
I(vec(T )) gives the column where row k of Ω is 1. That is, the S×1 vector Ωvec(T ) stacks the
columns of T on top of each other but with all elements that are constrained to zero removed.
With such a definition of Ω, and define two K ×K matrices as,
M = H † (N0 I +HH † )−1 H −I,

(14)

M̃ = P (I +M )P −P ,

(15)

the GMI for the optimal W and T is given in Proposition 1 and the proof is in Appendix B.

Proposition 1. Define an S×K 2 matrix D = Ω (P M ∗ )⊗I K , the optimal W maximizing the
GMI in (13) is
W opt = F −† (I +F † F +F † T P )H † (N0 I +HH † )−1 ,

(16)

and when P 6= 0, the optimal T maximizing the GMI is given by
 −1
∗
vec(T opt ) = −ΩT Ω M̃ ⊗ F (I +F † F )−1F † ΩT Dvec(F ).

(17)

With the optimal W and T , the GMI reads,

 I (F ),
P =0
1
IGMI (W opt , T opt , F ) =
 I1 (F ) + δ1 (F ), P =
6 0

(18)

where the functions I1 (F ) and δ1 (F ) are defined as


I1 (F ) = K +log det(I +F † F ) +Tr M (I +F † F ) ,

 −1
∗
δ1 (F ) = −vec(F )†D † Ω M̃ ⊗ F (I +F † F )−1 F † ΩT Dvec(F ).

(19)
(20)

Remark 1. With the definitions in (14) and (15), M is the negative of the MSE matrix and
M̃  0 holds. Hence δ1 (F ) ≥ 0 represents the GMI increments from the soft feedback.
Before discussing the GMI-maximization of (18), we first state Theorem 2 that deals with a
general maximization problem.

4

For instance, assuming T = 

0

1

2

0





, then the indication matrix Ω = 

0

1

0

0

0

0

1

0





, and the vector Ωvec(T ) = 

2
1


.

14

Theorem 2. Define a scalar function I with respect to a K ×K matrix G as


I(G) = K +log det(I +G) +Tr M (I +G)

(21)

where G satisfies G = [G]ν . Then the optimal G maximizing I is the unique solution that satisfies
[(I +Gopt )−1 ]ν = −[M ]ν .

(22)

With Gopt , the maximal I equals

I(Gopt ) = log det(I +Gopt ) .

(23)

Proof. Taking the first order differential of I with respect to G and noticing that G is banded
within diagonals [−ν, ν], yields (22) after some manipulations. The existence and uniqueness
of such an optimal solution for (22) is proved in [34, Theorem 2] and also illustrated in [18,

Proposition 2]. By Lemma 1, Tr [I + Gopt ]−1 M = −K from (22), and then (23) follows. 
Optimizing over F in (18) when P 6= 0 is difficult and cannot be carried out in closedform. In Appendix C we show by an example that (18) is in general non-concave. Therefore, a
gradient based numerical optimization procedure is utilized to search for the optimal F . In the
ith iteration, we construct
F (i) = F (i−1) +∇F ∗ IGMI W opt , T opt , F (i−1)



where ∇F ∗ IGMI (W opt , T opt , F ) is the conjugate of the gradient of the GMI with respect to (the
nonzero part of) F , and is given in Appendix D.
With P = 0, if replacing F † F by G, (19) has the same form as (21), and Gopt is in closedform as stated in Theorem 2. If Gopt  0, the optimal F then equals the Cholesky decomposition
of Gopt . Whenever it is not, a gradient based numerical optimization procedure is utilized to
optimize (19), and Gopt from Theorem 2 is used to initialize the starting point of F for any P ,
which has been observed to be highly reliable.
Next we establish a connection between the front-end filter W and IC matrix T in Method I.
Proposition 2. For P 6= 0, and with the optimal W and T , the matrix F † (W opt H − T opt ) is
banded within diagonals [−ν, K −1].
Proof. Noting that ΩT Ωvec(T opt ) = vec(T opt ) and ΩΩT = I, from (17) and (81), it holds that


∗
Ω M̃ ⊗ F (I +F † F )−1F † ΩT Ωvec(T opt ) = Ωvec F (I +F † F )−1F † T opt M̃
= −Ωvec(F M P ),

(24)

15

which shows that, the elements of the matrix ∆ = F (I +F † F )−1F † T opt M̃ +F M P are zero
wherever T can be nonzero. Hence ∆ is banded within diagonals [0, ν]. On the other hand, with
the optimal W given in (16) and M , M̃ defined in (14) and (15), we have
F † (W opt H −T opt )−(I +F † F ) = (I +F † F )F −1 ∆P −1 .

(25)

Note that, F −1 is lower triangular, I + F † F is banded within diagonals [−ν, ν], and P is
diagonal. Utilizing Lemma 2, the r.h.s in (25) is banded within diagonals [−ν, K−1]. Therefore
F † (W opt H − T opt ) is also banded within diagonals [−ν, K −1].



Proposition 2 reveals an interesting and somewhat surprising fact that, although the BCJR
only has a memory size ν, the interference outside the memory size ν shall not be perfectly
canceled with the optimal CS demodulator in Method I. As will be shown later, such a property
also holds for the other two designs of CS demodulator, i.e., Method II and III.
B. Method II
Method II origins from Ungerboeck’s 1974 paper [32]. Different from Method I, an Ungerboeck detection model (8) instead of the Forney model (10) is applied. The Ungerboeck model
has been extensively discussed in [35]–[37]. The system model (8) has the following constraints:
•

V is a K ×N matrix with no constraints.

•

G is a K×K Hermitian matrix satisfying G = [G]ν and I+G  0, where ν is the memory
size of G.

•

R is a K ×K matrix where the shape can be specified.

Instead of optimizing (W , T , F ), in Method II we optimize (V , R, G) for (12). The same
definition of the indication matrix Ω is used as in Method I, but now Ω corresponds to R
instead of T . We continue to let S denote the number of elements that are allowed to be
nonzero in R. That is, the S×1 vector Ωvec(R) stacks the columns of R on top of each other
but with all elements that are constrained to zero removed. In Method II, we have Proposition
3 that shows the GMI calculation with optimal V and R.
Proposition 3. Define an S×1 vector d = Ωvec(M P ), the optimal V for the GMI in (12) is
V opt = (I + G + Ropt P )H † (HH † + N0 I)−1 ,

(26)

and when P 6= 0, the optimal R maximizing the GMI is given by,
 −1
∗
vec(Ropt ) = −ΩT Ω M̃ ⊗(I + G)−1 ΩT d.

(27)

16

With the optimal V and R, the GMI in (12) equals

 I (G),
P =0
2
IGMI (V opt , Ropt , G) =
 I2 (G) + δ2 (G), P =
6 0

(28)

where the functions I2 (G) and δ2 (G) are defined as,


I2 (G) = K +log det(I +G) +Tr M (I +G) ,
 −1
∗
δ2 (G) = −d† Ω M̃ ⊗(I +G)−1 ΩT d.

(29)
(30)

The proof is given in Appendix E. Similar to δ1 (F ) in Method I, δ2 (G) ≥ 0 represents the
GMI increment from the soft information.
When P 6= 0, the optimization over G in (28) also uses a gradient based numerical optimization, and the gradient of IGMI (V opt , Ropt , G) with respect to (the nonzero part of) G is provided
in Appendix F. The closed-from G from Theorem 2 with P = 0 is still used as the starting point
for P 6= 0. However, different from Method I, the optimization procedure is concave and the
proof is given in Appendix G.
Although the optimal R is solved for in closed-form as in (27), we shall specify the constraint
(reflected by Ω) on it. We consider two types of R in Method II. Firstly, as we are interested
in the comparison between Method I and Method II, we also consider the shape (a) in Fig. 3,
which has the same shape as for R = F † T in Method I. Secondly, we consider a band-shaped
R with memory size νR , where shape (b) and (c) in Fig. 3 are typical cases with νR = 0 and
νR = ν, respectively. With shape (b), we only limit the diagonal elements of R to be zero and
intend to eliminate the interference as much as possible. With shape (c), we limit R to have
the opposite form of G, that is, the elements of R are constrained to be zero wherever G is
nonzero. The intention is to only cancel the interference that the BCJR represented by G cannot
handle. Shape (c) is based on the same idea as Method I, but now operates on the Ungerboeck
model.
The connection between the optimal front-end filter V and IC matrix R in Method II is now
established in Proposition 4.
Proposition 4. For P 6= 0 and the optimal V and R,
[V opt H]\(ν+νR ) = [Ropt ]\(ν+νR ) .

(31)

17

That is, the elements of V opt H and Ropt are equal outside the center 2(ν +νR )+1 diagonals
for any G that is banded within diagonals [−ν, ν], where νR = 0 for R with both shape (a) and
(b), while νR = ν for R with shape (c).
Proof. Following similar steps as in the proof of Proposition 2, (27) can be rewritten as,

Ωvec (I + G)−1 Ropt M̃ = −Ωvec(M P ).

(32)

It shows that, the elements of the matrix ∆ = (I + G)−1 Ropt M̃ P −1 +M are zero wherever R
can be nonzero. On the other hand, with the optimal V in (26) we have
V opt H −Ropt −(I +G) = (I +G)∆.

(33)

As I +G is banded within diagonals [−ν, ν], utilizing Lemma 2 (R with shape (a) is slightly
different, but it can be verified straightforwardly), and with the three shapes of R in Fig. 3, it
can be shown that the r.h.s in (33) is banded within diagonals [−(ν + νR ), ν + νR ], where νR = 0
for the shape (a) and (b), and νR = ν fot the shape (c). Therefore, V opt H −Ropt on the l.h.s in
(33) is banded within diagonals [−(ν + νR ), ν + νR ].



The same as Proposition 2 for Method I, Proposition 4 shows that the signal part that is
not considered in G (the BCJR) shall not be perfectly canceled inside the center 2(ν +νR )+1
diagonals for Method II, instead of the center 2ν +1 diagonals where G is constrained to be
nonzero. With LMMSE-PIC, we have ν = νR = 0 and Proposition 4 is natural and frequently
used. However, when νR > 0, a more general property is revealed that, V opt H and R are only
equal outside the center 2(ν +νR )+1 diagonals.
C. Method III
So far we have discussed two types of CS demodulators based on Forney and Ungerboeck
detection models, respectively. One disadvantage of them is that, in general both methods need an
numerical optimization to obtain the optimal target response. Next, we construct a third method
that has closed-form solutions for all CS parameters, although its GMI is suboptimal in general.
Method III relies on the same operations as Method II for P = 0. By inserting V opt in (26)
into (8) and setting R = 0, the demodulator actually operates on the mismatched function

p̃(y|x) = exp 2R{x† V opt y}−x† Gx


= exp 2R x† (I +G)x̌ −x† Gx

(34)

18

where x̌ = H † (HH †+N0 I)−1 y is the LMMSE estimate. As can be seen from (34), the BCJR is
based on x̌. With soft feedback, we can therefore replace x̌ by LMMSE-PIC estimates x̃. That
is, instead of (34) we operate on


p̃(y|x, x̃) = exp 2R x† (I +G)x̃ −x† Gx

(35)

where G has the same banded-shape as the first two methods, but optimized according to x̃.
The estimate x̃ is constructed as follows. As we prefer to handle the interference through the
trellis-search process, the IC should not be present within the memory size ν. In other words,
the signal vector after the IC that is used to form the kth symbol of x̃ is denoted as ỹ k and
X
ỹ k = y−
hn x̂n
(36)
n∈Ak



where Ak = 0 ≤ n ≤ K−1 : n ∈
/ [max(0, k−ν), min(k+ν, K−1)] . Denote pn as the nth diagonal
element of P , the Wiener filtering coefficients [38] for the kth symbol are calculated through
†

ŵk = hk (H † C k H +N0 I

−1

where C k is a diagonal matrix with the nth diagonal element defined as

 1−p , k ∈A
n
k
Ck (n) =
 1,
otherwise.

(37)

(38)

The estimate x̃ is then obtained through

T
x̃ = ŵ1 ỹ 1 ŵ2 ỹ 2 · · · ŵK ỹ K = Ŵ y− Ĉ x̂

(39)

where the coefficient matrix Ŵ and IC matrix Ĉ defined as
T
T T
Ŵ = [ ŵT
] ,
1 ŵ 2 · · · ŵ K

Ĉ = [Ŵ H]\ν .

(40)
(41)

Inserting x̃ in (39) back into (35), the detection model we operate on reads



p̃(y|x, x̂) = exp 2R x† (I +G)Ŵ y−(I +G)Ĉ x̂ −x† Gx .
Note that, (42) is a also special case of (8) by identifying
V = (I +G)W̃ ,
R = (I +G)C̃.

(42)

19

x̂2
x̂3

Interference
Cancelation
eq.(36)

y0

Wiener Filtering
eq.(39)

x0

ŵ0
x̂3

Interference
Cancelation
eq.(36)

y1

Wiener Filtering
eq.(39)

x1

ŵ1
x̂0

Interference
Cancelation
eq.(36)

y2

Wiener Filtering
eq.(39)

Update MSE
eq.(44)

x

M̂

[18, Prop.2]

x2

G

ŵ2
x̂0
x̂1
y

Interference
Cancelation
eq.(36)

y3

Wiener Filtering
eq.(39)

BCJR
eq.(42)

LLR

x3

ŵ3

Fig. 4.

An graphical overview of Method III with K = 4 and ν = 1.

The GMI in (12) in this case reads, after some manipulations,


IGMI (G) = K +log det(I +G) +Tr M̂ (I +G)

(43)

with M̂ (the updated M in Method II) defined as
†

†

†

M̂ = Ŵ HP Ĉ + Ŵ H −P Ĉ + Ŵ HP Ĉ + Ŵ H −P Ĉ
†

† †

†

−Ŵ (HH † +N0 I)Ŵ − ĈP Ĉ −I,

(44)

which can be shown to be the negative of the MSE matrix since




M̂ = −E (x− x̃)(x− x̃)† = −E (x− Ŵ y+ Ĉ x̂)(x− Ŵ y+ Ĉ x̂)† .
The optimal G for (43) is then obtained from Theorem 2, and the optimal GMI reads

IGMI (Gopt ) = log det(I +Gopt ) .
An graphical overview of Method III for K = 4 and ν = 1 is illustrated in Fig. 4. For any G
with memory size ν, the IC matrix (I+G)C̃ is zero along the main diagonal, which guarantees
that the extrinsic information will not be used for current symbols in the IC process. In GMI
sense, Method III will not outperform Method II with a shape (b) R, but it may outperform the
GMI of Method II with a shape (c) R, as it can be verified that a shape (c) R has zeros at the
positions where (I +G)Ĉ are also zeros.
Remark 2. As Ŵ H−Ĉ = [Ŵ H]ν , by Lemma 2 (I+G)(Ŵ H − Ĉ) is banded within diagonals
[−2ν, 2ν], which shows that, [(I +G)Ŵ H]\2ν = [(I +G)Ĉ]\2ν . Therefore, Proposition 4 also
holds for Method III with νR = ν.

20

V. PARAMETER O PTIMIZATION FOR ISI C HANNEL

In this section, we extend the CS demodulators to ISI channels where the matrix P = αI and
the block length K is infinitely large. The formulas for the achievable rates in (12), (13) and
(43) can be directly applied to (1), but as the achievable rate IGMI (as a function of the specified
CS parameters) is then dependent on the block length K, we are interested in asymptotic rate
1
IGMI .
I¯ = lim
K→∞ K
Ideally, in the ISI case the front-end matrix V and IC matrix R correspond to linear filtering
operations and the filters are infinitely long, but in practice filters with finite tap lengths are
used. Therefore, we analyze the properties of V , R (and W , T ) with a finite number of taps
and approximate them by band-shaped Toeplitz matrices. Furthermore, the trellis representation
matrix G (and F ), and channel matrix H are also band-shaped Toeplitz matrices. Therefore, in
the ISI case all matrices we consider are assumed to be band-shaped Toeplitz matrices, and the
band size can be arbitrary and sufficiently large so that we can analyze the asymptotic properties.
In [39] a complete theoretic machinery for ISI channels is derived and a result is that, as K → ∞
the linear convolution in (1) can be replaced with a circular convolution.
In the following, we denote the Fourier series associated to a band-shaped Toeplitz matrix E
with infinitely large dimensions by E(ω), where E is constrained to be zero outside the middle
2NE +1 diagonals, and NE is referred to as the tap length of E(ω). The series E(ω) defined as
E(ω) =

NE
X

ek exp(jkω)

k=−NE

is specified by a vector e = [ e−NE . . . e−1 e0 e1 . . . eNE ], where e0 is the element on the main
diagonal and ek is the element on kth lower (k > 0) or upper (k < 0) diagonal. As all quantities
are evaluated as the block length K grows large, E(ω) approaches the eigenvalue distribution
of E (see [40], [41] for a precise statement of this result). We first state Theorem 3, which is
an asymptotic version of Theorem 2 for ISI channels.
Theorem 3. Assume that two band-shaped Toeplitz matrices G and M with infinitely large
dimensions satisfying [G]\ν = 0, I +G  0 and M ≺ 0. Define a scalar function
Z

1 π
¯
log(1+G(ω))+M (ω)(1+G(ω)) dω.
I = 1+
2π −π

(45)

21

Then, the optimal G(ω) that maximizes I¯ is
Gopt (ω) = |u0 + ûϕ(ω)|2 −1,
where the 1×ν vector ϕ(ω) = [ exp(jω) exp(j2ω) . . . exp(jνω) ]T , and
1
u0 = q
,
† −1
τ 1 τ 2 τ 1 −τ0
û = −u0 τ †1 τ −1
2 .
The real scalar τ0 , ν ×1 vector τ 1 , and ν ×ν matrix τ 2 are defined as
Z
1 π
τ0 =
M (ω)dω,
2π −π
Z
1 π
M (ω)ϕ(ω)dω,
τ1 =
2π −π
Z
1 π
τ2 =
M (ω)ϕ(ω)ϕ(ω)† dω.
2π −π
Furthermore, with Gopt (ω) the optimal I¯ reads
I¯ = 2 log(u0 ).

(46)

(47)

(48)

Proof. As I + G  0, we assume that 1 + G(ω) = |U (ω)|2 , with U (ω) = u0 + ûϕ(ω) and û =
[ u1 u2 . . . uν ]. Then I¯ in (45) can be rewritten as
Z

1 π
¯
M (ω) u20 +2R{u0 ûϕ(ω)}+ ûϕ(ω)ϕ† (ω)û† dω.
(49)
I = 1+2 log(u0 )+
2π −π
Taking the first order differentials with respect to u0 and û and optimizing them directly results
in the optimal solution (46). Inserting (46) back into (49) and after some manipulations, the
optimal asymptotic rate is then in (48).



A. Method I
The structures of (W , T , F ) are the same as in Section IV-A, except that now the matrices
have infinite dimensions. Applying Szegő’s eigenvalue distribution theorem [40] to (13), the
asymptotic rate reads

1
I¯ W (ω), T (ω), F (ω) = lim
IGMI (W , T , F )
K→∞ K

Z 

1 π
L1 (ω)
2
2
=
log 1+|F (ω)| −|F (ω)| −
dω
2π −π
1+|F (ω)|2
Z

1 π  ∗
+
R F (ω) W (ω)H(ω)−αT (ω) dω
π −π

(50)

22

where

L1 (ω) = |F (ω)W (ω)|2 N0 +|H(ω)|2 +α|F (ω)T (ω)|2
−2α|F (ω)|2R{H(ω)W (ω)T ∗ (ω)} .
Note that, the Fourier series associated to M and M̃ in (14) and (15) are
M (ω) =

|H(ω)|2
−1,
N0 +|H(ω)|2

M̃ (ω) = α2 (M (ω) + 1)−α.

(51)
(52)

Further, define a (2NT −ν)×1 vector




 T
φ(ω) = exp −jNT ω . . . exp −j(ν +1)ω exp j(ν +1)ω . . . exp jNT ω
,
a (2NT −ν)×1 vector ε1 , and a (2NT −ν)×(2NT −ν) Hermitian matrix ε2 as
Z
α π
M (ω)F ∗ (ω)φ(ω)dω,
ε1 =
2π −π
Z
1 π M̃ (ω)|F (ω)|2 φ(ω)φ(ω)†
ε2 =
dω,
2π −π
1+|F (ω)|2

(53)

(54)

where NT is the tap length of T (ω), and ν +1 is the band size where matrix T is constrained
to zero. Then, we have Proposition 5 with the proof5 given in Appendix H.
Proposition 5. The optimal W (ω) for the asymptotic rate in (50) is
Wopt (ω) =


H ∗ (ω)
2
∗
1+|F
(ω)|
+αF
(ω)T
(ω)
,
opt
F ∗ (ω)(N0 +|H(ω)|2 )

(55)

and when 0 < α ≤ 1, the optimal T (ω) reads
Topt (ω) = −ε†1 ε−1
2 φ(ω).
With the optimal W (ω) and T (ω), the asymptotic rate equals

  I¯1 (F (ω)),
α=0
I¯ Wopt (ω), Topt (ω), F (ω) =
 I¯1 (F (ω)) + δ̄1 (F (ω)), 0 < α ≤ 1.
5

(56)

(57)

Proposition 5 is the same as [62, Theorem 1] which has been derived for hard feedback symbols. For completeness, we

restate the proof in Appendix H.

23

The functions I¯1 (F (ω)) and δ̄1 (F (ω)) are defined as6 ,
Z


1 π
2
2
¯
log 1+|F (ω)| +M (ω) 1+|F (ω)| dω,
I1 (F (ω)) = 1 +
2π −π
δ̄1 (F (ω)) = −ε†1 ε−1
2 ε1 .

(58)
(59)

In the ISI case, Method I is still not concave an example is also provided in Appendix C, and
a gradient based optimization is used to optimize F (ω) with the optimal solution of Gopt (ω)
from Theorem 3 is used to initialize the starting point.
The connection between the optimal front-end filter W (ω) and the IC filter T (ω) in Proposition
2 also holds for ISI channels. An asymptotic version of Proposition 2 is stated in Proposition 6.
Proposition 6. When 0 < α ≤ 1, ak = bk holds for k < −(ν +1), where
Z

1 π ∗
ak =
F (ω)Wopt (ω)H(ω)exp −jkω dω
2π −π
Z

1 π ∗
bk =
F (ω)Topt (ω)exp −jkω dω.
2π −π
Proof. In Appendix H, the optimal t̃ in (95) satisfies t̃opt ε2 = −ε†1 . With the definitions of ε1 ,
ε2 in (54), this is equivalent to
Z
Z
α π
1 π M̃ (ω)|F (ω)|2 Topt (ω)φ(ω)†
dω = −
F (ω)M (ω)φ(ω)† dω.
2
2π −π
1 + |F (ω)|
2π −π

(60)

On the other hand, with Wopt in (55) and M (ω), M̃ (ω) defined in (51) and (52), we have
Z


1 π ∗
(F (ω)Wopt (ω)H(ω)−F ∗ (ω)Topt − 1+|F (ω)|2 exp −jkω dω
2π −π
Z



1 π  M̃ (ω)F ∗ (ω)Topt (ω)
2
=
+ 1+|F (ω)| M (ω) exp −jkω dω.
(61)
2π −π
α
Transforming (60) and (61) back into matrix forms, we have that (24) and (25) hold. Following
the same arguments as in the proof of Proposition 2, F † (W opt H − Ropt ) is banded within
diagonals [−ν, K −1]. Therefore we have
Z


1 π ∗
F (ω)Wopt (ω)H(ω)−F ∗ (ω)Topt (ω) exp −jkω dω = 0
2π −π
whenever k < −(ν +1), which proves Proposition 6.
6



Similar to finite length linear vector channels, δ̄1 (F (ω)) in (59) is only defined for α 6= 0 which represents the rate increment

with soft information. The same holds for δ̄2 (G(ω)) in (69) for Method II.

24

B. Method II
The matrices (V , R, G) have the same constraints as in Section IV-B while the dimensions of
these matrices are infinitely large. However, as the shape (a) of R in Fig. 3 is not meaningful as
N, K → ∞, it is not considered for ISI case. Applying Szegő’s eigenvalue distribution theorem
to (12), the asymptotic rate of Method II reads
¯ (ω), R(ω), G(ω)) = lim 1 IGMI (V , R, G)
I(V
K→∞ K

Z 

1 π
L2 (ω)
=
dω
log 1+G(ω) −G(ω)−
2π −π
1+G(ω)
Z

1 π 
R V (ω)H(ω)−αR(ω) dω
+
π −π

(62)

where


L2 (ω) = |V (ω)|2 N0 +|H(ω)|2 +α|R(ω)|2 −2αR H(ω)V (ω)R∗ (ω) .
Define a 2(NR −νR )×1 vector




 T
ψ(ω) = exp −jNR ω . . . exp −j(νR +1)ω exp j(νR +1)ω . . . exp jNR ω ,
a 2(NR −νR )×1 vector ζ 1 , and a 2(NR −νR )×2(NR −νR ) Hermitian matrix ζ 2 as
Z
α π
ζ1 =
M (ω)ψ(ω)dω,
2π −π
Z
1 π M̃ (ω)ψ(ω)ψ(ω)†
ζ2 =
dω,
2π −π
1+G(ω)

(63)

(64)

where NR denotes the tap length of Ropt (ω), 2νR +1 is the band size where R is constrained to
zero, and M (ω) and M̃ (ω) are in (51) and (52). Then, we have Proposition 6 with the proof in
Appendix I, where we also show that R(ω) is real and R has Hermitian symmetry.
Proposition 7. The optimal V (ω) for (62) is,
Vopt (ω) =


H ∗ (ω)
1+G(ω)+αR
(ω)
,
opt
N0 +|H(ω)|2

(65)

and when 0 < α ≤ 1, the optimal R(ω) reads
Ropt (ω) = −ζ †1 ζ −1
2 ψ(ω).
With the optimal V (ω) and R(ω), the asymptotic rate equals

  I¯2 (G(ω)),
α=0
I¯ Vopt (ω), Ropt (ω), G(ω) =
 I¯2 (G(ω)) + δ̄2 (G(ω)), 0 < α ≤ 1.

(66)

(67)

25

The functions I¯1 (G(ω)) and δ̄2 (G(ω)) are defined as,
Z


1 π
¯
log 1+G(ω) +M (ω) 1+G(ω) dω,
I2 (G(ω)) = 1+
2π −π
δ̄2 (G(ω)) = −ζ †1 ζ−1
2 ζ 1.

(68)
(69)

For 0 < α ≤ 1, it still needs a gradient based optimization to find the optimal G(ω) for (68),
and the closed-form solution in Theorem 3 is utilized as the starting point. The asymptotic rate
¯ opt (ω), Ropt (ω), G(ω)) is also concave with respect to G(ω), which is shown in Appendix J.
I(V
Proposition 8. When 0 < α ≤ 1, ak = bk holds for |k| > ν +νR , where
Z

1 π
ak =
Vopt (ω)H(ω)exp −jkω dω,
2π −π
Z

1 π
Ropt (ω)exp −jkω dω.
bk =
2π −π

(70)
(71)

The connection of the optimal V (ω) and R(ω) stated in Proposition 8 is an asymptotic version
of Proposition 4, and the proof follows the similar approach as Proposition 6. We show an
example in Fig. 5 to illustrate Proposition 8 with Method II and ν = νR = 1. The Proakis-C [42]
channel is tested at an SNR of 10 dB and α equals 0.1, 0.4 and 0.8, respectively. As νR = 1,
bk as defined in (71) is constrained to zero for 0 ≤ k ≤ 1. As can be seen, ak as defined in (70)
equals bk only for |k| > 2, and when |k| = 2, ak and bk are not identical. This shows that with
the optimal V (ω) and R(ω), the signal part along the second upper and lower diagonals that
is not considered in G(ω) shall not be perfectly canceled out. This behavior cannot be seen in
[43] which treats LMMSE-PIC for ISI channels, due to ν = νR = 0.
C. Method III
In Method III, from (43) the asymptotic rate reads
Z


1
1 π
¯
IGMI (G) = 1+
log 1+G(ω) + M̂ (ω)(1+G(ω)) dω
I(G(ω)) = lim
K→∞ K
2π −π

(72)

where according to (44),

M̂ (ω) = 2R αŴ (ω)H(ω)Ĉ ∗ (ω)+ Ŵ (ω)H(ω)
−αĈ ∗ (ω) −

|Ŵ (ω)|2
−α|Ĉ(ω)|2 −1.
2
N0 +|H(ω)|

Replacing M (ω) by M̂ (ω), the optimal G(ω) and asymptotic rate I¯ follow from Theorem 3.

26

8
7
6
5
4
3
2
1
0
-10

Fig. 5.

-5

0

5

10

Comparison between ak and bk for Method II under Proakis-C channel h = [ 0.227 0.46 0.688 0.46 0.227 ].

Remark 3. Proposition 8 also holds for Method III with νR = ν, due to the fact that [(I +
G)(Ŵ H − Ĉ)]\2ν = 0.
VI. SNR A SYMPTOTICS
In this section, we analyze asymptotic properties of the CS demodulators, and show that, as
N0 goes to 0 and ∞, Method III and Method II are asymptotically equivalent. As Method I is
inferior to Method II in GMI sense, we limit our investigations to Method II and Method III,
and start the analysis for finite length linear vector channels first. The following limits can be
verified straightforwardly:
lim M /N0 = −(H † H)−1 ,

N0 →0

lim N0 (I +M ) = H † H.

N0 →∞

(73)

Moreover, it also holds that
lim M̃ = P 2 − P ,

N0 →0

lim M̃ = −P .

N0 →∞

As M̃ should be invertible from the definition of δ2 (G) in (30), we restrict that P ≺ I.

(74)

27

Lemma 3. When N0 → 0 and ∞, the optimal G for (28) in Method II satisfies (22), and the
following limits hold,


lim (N0 (I +Gopt ))−1 ν = [(H † H)−1 ]ν ,

N0 →0

lim [N0 Gopt ]ν = [H † H]ν .

N0 →∞

(75)
(76)

Proof. When P = 0, from Theorem 2 the optimal G for (28) satisfies (22). From (73), when
N0 → 0, M → 0 and N0 → ∞, M → −I. Therefore, by the definition of Ω,
lim d = Ωvec(M P ) = 0.

N0 →0,∞

This implies that the gradient dG (δ2 ) in (90) (Appendix F) converges to zero. Hence the
differentials of IGMI (V opt , Ropt , G) in (28) with P 6= 0 converges to the differentials with
P = 0. From (22) and (73), the limit (75) follows.
Next, since
lim

N0 →∞



N0 I −(I +Gopt )−1 ν = lim [N0 (I +M )]ν = [H † H]ν ,
N0 →∞

(77)

it shows that I−(I+Gopt )−1 → 07 as N0 → ∞. By the matrix inversion lemma, I−(I+Gopt )−1 →
Gopt as N0 → ∞, and combining this with (77) proves the limit (76).



Lemma 4. In Method II, with the optimal G, when N0 → 0 the GMI increment δ2 (G) in (30)
converges to zero with speed O(1/N0 )8 and when N0 → ∞ the GMI increment δ2 (G) converges
to zero with speed O(N02 ).
Proof. As N0 → 0, from (73) we have

lim d/N0 = lim Ωvec(M P /N0 ) = −Ωvec (H † H)−1 P .

N0 →0

N0 →0

Based on (74) and Lemma 3, the below equalities hold,
d†   ∗ (I +Gopt )−1  T −1 d
Ω M̃ ⊗
Ω
= O(N0 ).
δ2 (Gopt ) = N0
N0
N0
N0
On the other hand, as N0 → ∞, and by the definition of Ω, from (73) we also have

lim N0 d = lim Ωvec(N0 M P ) = lim Ωvec N0 (I +M )P = Ωvec(H † HP ).

N0 →∞

N0 →∞

N0 →∞

7

A matrix A → B or a vector a → b means the nonzero elements of A−B or a−b converges to zero.

8

Two scalars A and B as functions of a variable n converging to each other with speed O(n) means that, there exists a

constant C such that lim n|A − B| < C.
n→∞

28

Again utilizing (74) and Lemma 3, the below equalities hold,
δ2 (Gopt ) =

 T −1
1
∗
†
−1
Ω
(N0 d) = O(1/N02 ).
(N
d
)
Ω
M̃
⊗(I
+G
)
0
opt
N02

Therefore, Lemma 4 holds.



Lemma 5. When N0 → 0 and ∞, the optimal GMI in Method III is independent of P and
converges to the optimal GMI with P = 0. Moreover, (75) and (76) hold.
The proof is given in Appendix K. Combining Lemmas 3-5, and using the fact that Method
III and Method II are equivalent with P = 0, we have the following Theorem 4.
Theorem 4. Assume that P ≺ I, when N0 → 0 and ∞, the optimal GMI in Method III converges
to the optimal GMI in Method III with P = 0. Moreover, the optimal GMI in Method II also
converges to the optimal GMI in Method III with P = 0, with speed O(1/N0 ) when SNR increase
and O(N02 ) when SNR decreases. The optimal G for both methods has the limits (75) and (76).
From Theorem 4 we know that, except for the case where one of the elements in the diagonal
matrix P is 1, the soft feedback information becomes asymptotically insignificant for the design
of the CS parameters. The reason is that, when N0 → 0, x̂ is overwhelmed by the noise, while
when N0 → ∞, the optimal front-end filter will null out x̂ since the filter can perfectly reconstruct
the transmitted symbols without using the side information.
Remark 4. When N0 → 0 and ∞, the optimal CS demodulator is the EZF demodulator defined
in Example 1, and the TMF defined in Example 2, respectively.
With ISI channels, as the same constraint P = αI ≺ I shall hold, we make the restriction
that α < 1. The asymptotic properties for ISI channels are presented in Corollary 1, which is
an asymptotic version of Theorem 4 when the channel matrix H and CS parameters are bandshaped Toeplitz matrices with infinite dimensions. The detailed proof is following the same
analysis as for the finite linear vector channels and omitted.
Corollary 1. Assume that 0 ≤ α < 1, when N0 → 0 and ∞, the optimal GMI in Method III
converges to the optimal GMI in Method III with α = 0. Moreover, the optimal GMI in Method
II also converges to the optimal GMI in Method III with α = 0, with speed O(1/N0 ) when SNR

29

increase and O(N02 ) when SNR decreases. The optimal G for both methods has the following
asymptotic properties hold for |k| ≤ ν:
Z π
Z π
1
1
exp(−jkω)dω =
exp(−jkω)dω,
lim
2
N0 →0 −π N0 (1 + Gopt (ω))
−π |H(ω)|
Z π
Z π
N0 Gopt (ω) exp(−jkω)dω =
|H(ω)|2 exp(−jkω)dω.
lim
N0 →∞

−π

−π

VII. E MPIRICAL R ESULTS
In this section, we provide empirical results to show the behaviors of CS demodulators in
an iterative detection and decoding receiver designs. With the considered MIMO channels, all
channel elements are assumed to be independent identically distributed (IID) complex Gaussian
with zero-means, and the received signal power at each receive antenna is normalized to unity.
For ISI case, we test with the typical Proakis-C channel as in Fig. 5.
A. GMI Evaluation
We first evaluate the GMI under 5 × 5 MIMO channels with memory size ν = 1 for all
CS demodulators. We simulate 10000 channel realizations for each SNR point. The GMIs are
compared with that of the static CS demodulator [18], which is equivalent to Method II with
P = 0. The channel capacity is also presented for comparison. The results of GMI are shown
in Fig. 6. As the quality of soft information improves beyond P = 0, Method II with νR = 0
performs the best among all CS demodulators, as it has the most degrees of freedom (DoF)
in R. Method II with νR = ν is the worst among Method I and Method II, while Method I is
slightly worse than Method II with R of shape (a) in Fig. 3, which is because although the IC
matrix R is shape (a) in both cases, R in Method II is more general than in Method I which is
constrained to R = F † T . The GMI of Method III is inferior to Method II as expected.
The results show consistent GMI increments for all CS demodulators when the feedback
quality improves. When P increases from P = 0 to the ideal case P = I, the channel capacity
becomes inferior to the GMI as the pair (x, ŷ) is superior to (x, y) for information transfer.
B. SNR Asymptotic of the GMI
Next, we evaluate the asymptotic properties of the GMI described in Theorem 4 under 5×5
MIMO channels. As shown in Fig. 7, the GMIs of Method II and Method III both converge to
Method III with P = 0. Moreover, the GMI of the CS demodulators converge to the EZF in

30

16

16

14

14

12

12

10

10

8

8

6

6
8

10

12

14

16

16

8

10

12

14

16

8

10

12

14

16

20

14
15

12
10

10

8
6

5
8

Fig. 6.

10

12

14

16

GMI of CS demodulators under 5×5 MIMO IID complex Gaussian channels with ν = 1.

Example 1 at high SNR, and the TMF in Example 2 at low SNR, respectively, which are well
aligned with Theorem 4.
C. EXIT Charts of CS Demodulators
In order to predict the dynamics of iterative receivers, we use the tool of extrinsic information
transfer (EXIT) charts invented by ten Brink [59], [60] for analysis of iterative receiver behavior.
For EXIT analysis, the CS demodulator and the decoder measure the output extrinsic information
IE based on a sequence of observations y and a priori information IA into a new sequences.
In Fig. 8, we evaluate the EXIT charts for CS demodulators under 4 × 6 MIMO channels with
ν = 2 for F and G at an SNR of 10dB. With Method II, we test different values of νR . As can
be seen, when νR > ν, the demodulation performance is inferior to νR ≤ ν. This is because, the
interference outside memory size ν and inside memory size νR is neither considered in the IC
process nor in the BCJR module. Moreover, with νR ≤ ν, the CS demodulators with Method
II performs quite close to each other as well as Method I and III. For Method II with νR < ν,
the interference inside memory size ν and outside νR are considered both in the IC and BCJR
processes. However, an interesting observation is that, with large a priori input IA , Method II
with νR = 0 is inferior to νR = 1 and νR = 2. Therefore, a conservative approach with Method II
is to set νR = ν such that the interference is either removed in IC process or dealt with in the

31

40
35
30
25
20
15
10
5
0
-20

Fig. 7.

-10

0

10

20

30

40

SNR asymptotic under 5×5 MIMO IID complex Gaussian channels with ν = 1.

BCJR module, to get rid of potential error propagation caused by redundant processings of the
same part of interference.
In Fig. 9, we show the iterative detection and decoding trajectories for CS demodulators under
Proakis-C channel with ν = 2 and at an SNR of 10dB. We use an [7, 5] convolutional code
[51] with a coded block-length K = 2004, and a random permutation is applied to the coded
bits. As can be seen, the CS demodulators with Method II and Method III are superior to the
LMMSE-PIC demodulator, and the iterative detection and decoding trajectories are well aligned
with the measured EXIT charts.
D. Link Performance
We next turn to link-level simulations with turbo codes [44] where the outer decoder uses 8
internal iterations. A single code-block over all transmit symbols is used. At each SNR point
20000 data blocks are simulated and the block-error-ratio (BLER) is measured. In all simulations,
at most three global iterations are used between the demodulators, and the decoder the tap length
of the front-end and IC filters are all set to 8L, and νR = ν for Method II.
In Fig. 10, we evaluate the BLER under Proakis-C channel with QPSK symbols and ν = 2 for
all CS demodulators. A (1064, 1600) turbo code is used. Note that, at the first iteration when
there is no soft information, Method II and III overlap with each other. With CS demodulators,
the gap to the MAP demodulator is less than 1 dB, while the LMMSE-PIC has a gap to the
MAP that is up to 10 dB. Moreover, Method II performs slightly better than Method I, and

32

0.95
0.9
0.85
0.8
0.75
0.7
0.65
0.6
0.55
0

Fig. 8.

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

EXIT charts under 4×6 IID complex Gaussian MIMO channels with ν = 2 and different values of νR .
1

0.8

0.6

0.4

0.2

0
0

Fig. 9.

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Iterative detection and decoding trajectories under Proakis-C channel at an SNR of 10 dB. The outer code is an [7,

5] convolutional code with generator polynomials g0 (D) = 1 + D2 and g1 (D) = 1 + D + D2 . A random permutation of the
code block is used and the black curve is the decoding EXIT chart. The dashed lines are the iterative detection and decoding
trajectories for LMMSE-PIC, Method III and II, respectively.

Method III is slightly inferior to both methods. However, Method III has the advantage of less
computational complexity than the others since all parameters are in closed-forms.
In Fig. 11, we evaluate the BLER under 4×6 MIMO channels with QPSK symbols and ν = 3
for all CS demodulators. A (1064, 1800) turbo code is used. As N < K, the LMMSE-PIC fails
[46] at the first iteration due to the lack of receive diversity. However, the CS demodulators with
ν = 3 significantly improve the performance and with less than 1 dB gap at 10% BLER to the

33

100

10-1

10-2

10-3

10-4

Fig. 10.

8

9

10

11

12

13

14

15

16

17

18

19

BLER performance of the LMMSE-PIC, Method I-III, and MAP under Proakis-C channel with QPSK modulation.
100

10-1

10-2

10-3

Fig. 11.

9

9.5

10

10.5

11

11.5

12

12.5

13

BLER performance of the LMMSE-PIC, Method II, Method III, and MAP under 4×6 MIMO channels with QPSK

modulation.

MAP. CS demodulators with ν = 1 after three iterations is less than 2 dB away from the MAP.
With less computational cost, Method III still performs close to Method II.
Finally we remark that, for the sake of complexity savings, both for finite linear vector channels
and ISI channels, the parameters of CS demodulators do not need to be updated through all
iterations. Once the feedback information quality is good enough and the parameter P or α are
close to ideal, the CS parameters can be kept unchanged in successive iterations.

34

VIII. S UMMARY
In this paper we considered the design of channel shortening (CS) demodulators for linear
vector channels that use a trellis representation of the received signal in combination with
interference cancellation (IC) of the signal part that is not appropriately modeled by the trellis. In
order to reach a trellis representation, a linear filter is applied as front-end. It is an extension of
the well studied CS demodulators to iterative receivers and a generalization of the LMMSE-PIC
demodulator to cooperate with trellis-search in turbo equalization.
We analyzed the properties of three different approaches for designing such optimal CS
demodulators as all of them may come across as natural “CS” demodulators. In the used
framework, there are three parameters that need to be optimized. Based on a generalized mutual
information (GMI) cost function, two of these are solved for in closed-form, while the third
needs to be numerically optimized except for Method III where we constructed it explicitly at
the cost of a small performance loss. A simple gradient based optimization is used and turns
out to perform well.
Numerical results are provided to illustrate the behavior of the proposed CS demodulators.
In general, Method II based on the Ungerboeck model is superior to Method I that is based on
the Forney model. Method II has the advantage over Method I that the optimization procedure
is concave. The suboptimal Method III performs close to Method I and Method II, and it has
all parameters in closed-forms. An interesting result is that the interference cancellation of the
CS demodulators should not cancel the effective channel perfectly outside the memory size ν,
a property that cannot be seem in LMMSE-PIC demodulator as ν = 0. Moreover, we have also
analyzed asymptotic properties of the CS demodulators and showed that, Method III converges
to Method II asymptotically when the noise density goes to zero or infinity.
A PPENDIX A: D ERIVATION OF THE GMI
By making the eigenvalue decomposition QΛQ† = G and letting s = Q† x. As x is assumed to
be zero mean complex Gaussian random vector with covariance matrix I, we can write p̃(y|x)
in (8) as


p̃(y|x) = exp 2R s† d −s† Λs ,

(78)

35

where d = Q† (V y−Rx̂). We can now evaluate
Z
p̃(y) = p̃(y|x)p(x)dx
Z


1
= K exp 2R s† d −s† Λs exp −s† s)ds
π


K−1
Y 1
|dk |2
exp
.
=
1+λk
1+λk
k=0
where λk is the kth diagonal element of Λ and dk is the kth entry of d. Taking the expectation
over y gives


−Ep(y) [log(p̃(y))] = log det(I +G) −Tr L(I +G)−1
where the matrix L = Ep(y) [Qdd† Q† ] is given by
L = V (N0 I +HH † )V † −V HP R† −RP H † V † +RP R† .

(79)

On the other hand, we have

−Ep(y,x) [log(p̃(y|x))] = Tr(G)−2R Tr(V H −RP ) .
Combining the two expectations, the GMI reads,



IGMI (V , R, G)=log det(I +G) −Tr L(I +G)−1 −Tr(G)+2R Tr(V H −RP )


=log det(I +G) −Tr(G)+2R Tr(V H −RP )


−Tr (I +G)−1 V [HH † +N0 I]V † −2R V HP R† +RP R† .

A PPENDIX B: T HE P ROOF OF P ROPOSITION 1
As the formula of GMI in (13) is quadratic in W and no constraints apply to W , taking the
gradient of IGMI (W , T , F ) with respect to W and setting it to zero, the optimal W is given
in (16). Inserting W opt into (13) gives, after some manipulations,


IGMI (W opt , T , F ) = K +log det(I +F † F ) +Tr T † F (I +F † F )−1 F † T M̃



+Tr M (I +F † F ) +2R Tr P M F † T .
where M and M̃ are defined in (14) and (15). If P = 0, (80) equals


I1 (F ) = K +log det(I +F † F ) +Tr M (I +F † F ) .

(80)

36

In this case, there is no soft information available and the matrix T is not included in the
formula. When P 6= 0, the terms of IGMI in (80) related to T are



f (T ) = Tr T †F (I +F † F )−1 F † T M̃ +2R Tr P M F † T .
Let tk denote the kth column of T , but all elements in rows [k, min(k+ν, K−1)] removed, and
T
T ]T , then by the definition of the indication matrix
define the column vector t = [ tT
0 t1 . . . tK−1

Ω, we have
t = Ωvec(T ).
Similarly, let z k denote the kth column of the matrix F M P but with all elements in rows
T ]T , then we have
T
[k, min(k+ν, K −1)] removed, and define a row vector z = [ z T
0 z 1 . . . z K−1


z = Ωvec(FMP ) = Ω (PM ∗ )⊗I K vec(F ).

∗
Finally, defining a Hermitian matrix B̂ 1 = Ω M̃ ⊗ F (I +F † F )−1F † ΩT , and with that we
can rewrite f (T ) as f (T ) = t† B̂ 1 t+2R{z † t}. Taking the gradient with respect to t and setting
it to zero yields,
−1

topt = −B̂ 1 z.

(81)

Transferring topt back into T opt given the optimal T in (81) and inserting this into f (T ) gives
−1

f (T opt ) = −z † B̂ 1 z.
Thus, with the optimal W and T , when P 6= 0 the GMI equals


IGMI (W opt , T opt , F ) = K +log det(I +F † F ) +Tr M (I +F † F )

 T −1
∗
†
†
†
−1 †
−vec(F ) D Ω M̃ ⊗ F (I +F F ) F Ω
Dvec(F ).

where D = Ω (PM ∗ )⊗I K .
A PPENDIX C: N ON -C ONCAVITY E XAMPLES OF M ETHOD I
We give examples to demonstrate the non-concavity of Method I for MIMO and ISI channels
with assuming that P = I and α = 1, respectively. The memory size ν = 1 and the noise density
N0 equals 1 in both cases. A 5×5 MIMO channel and the Proakis-C channel are used.

37

11.5

0.58

0.575

11

0.57
10.5
0.565
10
0.56
9.5
0.555
9
0.55
8.5

0.545

8
0

Fig. 12.

0.2

0.4

0.6

0.8

1

0.54
0

0.2

0.4

0.6

0.8

1

Non-concaveness of Method I under 5×5 MIMO channel (left figure) and Proakis-C ISI channel (right figure).

Example 4. MIMO case:






2 0 −3 5 4
4.94 4.45 0
0
0
2.03 6.17 0
0
0






−5 2 −1 0 2
 0 0.21 3.85 0
 0 5.22 3.56 0


0
0

















H =  2 −4 3 3 3, F 1 =  0
0 5.56 1.76 0 , F 2 =  0
0 7.43 0.73 0 
.






−1 −5 −4 1 2
 0
 0


0
0
0.61
7.10
0
0
4.98
4.32






0 −2 0 5 5
0
0
0
0 2.79
0
0
0
0 10.11
Example 5. ISI case:
h
i
h
i
h
i
h = 0.227 0.460 0.688 0.460 0.227 , f 1 = 0.1606 0.9009 , f 2 = 0.2230 0.2035 .
The IGMI (W opt , T opt , F ) given in (18) as a function F is plotted on the left in Fig. 12, while
¯ opt (ω), Topt (ω), F (ω)) given in (57) as a function of F (ω) is plotted on the right. If
the I(W
¯ opt (ω), Topt (ω), F (ω)) are concave or convex, the blue curves lie
IGMI (W opt , T opt , F ) and I(W
above or below the black curves, which clearly does not hold in our examples.
A PPENDIX D: T HE G RADIENT IN M ETHOD I

FOR

F INITE L INEAR V ECTOR C HANNEL

In this section we derive the first order differential of the GMI given in (18) with respect to F .
In order to utilize the differential with respect to a matrix, we use the α-differential as defined in
[47]. Assume a matrix Y N,K with dimension N×K and a matrix X M,S with dimension M×S,

38

define dX Y as the α-differential of Y with respect to X. Furthermore, defining y` and x` as
[ y1 y2 · · · yNK ] = vec(Y )T and [ x1 x2 · · · xMS ] = vec(X)T , the α-differential dX Y is


∂y1
∂y1
∂y1
·
·
·
∂x2
∂xMS

 ∂x1
∂y2
∂y2
∂y2 

·
·
·
∂vec(Y )
 ∂x1
∂x2
∂xMS 
dX Y =
.
= .
.
.
.. 
T

 ..
..
..
∂vec(X)
.


∂yNK
∂yNK
∂yNK
· · · ∂xMS
∂x1
∂x2
The reason for adopting the α-differential is because it keeps the chain rule and the product
rule. We introduce an N K × N K permutation matrix Z N,K , which satisfies the condition
vec(Y T ) = Z N,K vec(Y ). It is easy to verify that Z −1
N,K = Z K,N , and when N = 1 or K = 1, Y
is a vector and vec(Y T ) = vec(Y ), hence Z N,1 = I N and Z 1,K = I K . Furthermore, by definition

we have dF (F ) = dF (vec(F )) = I, and dF (F † ) = dF vec(F † ) = 0. We start by reviewing a few
properties [47], [48] of α-differential below that will be used later, where both matrices X and
Y are functions of F and the dimensions are specified by the subscripts associated to them:
−T
−1
dF (X −1
K,K ) = −(X K,K ⊗X K,K )dF X K,K

dF (Y N,K X K,S ) = (X T
K,S ⊗I N )dF Y N,K + (I S ⊗Y N,K )dF X K,S
T
dF (log(det(X K,K ))) = vec(X −T
K,K ) dF X K,K

dF (Y N,K ⊗X M,S ) = (I K ⊗Z S,N ⊗I M )(I NK ⊗vec(X)dF Y N,K
+(I K ⊗Z S,N ⊗I M )(vec(Y )⊗I MS )dF X M,S .
The α-differential of I1 (F ) with respect to F is
dF (I1 ) = vec((I +F † F )−T )T (I K ⊗F † )+vec(F ∗M T )T
= vec(FM +F (I +F † F )−† )† .

(82)
∗

Defining a K ×K matrix B = F (I +F † F )−1 F † and an S ×S matrix Π = Ω(M̃ ⊗B)ΩT

−1

,

the α-differential of δ1 (F ) with respect to F is

dF (δ1 ) = −vec(F )† D † (vec(F )T D T )⊗I S dF (Π)−vec(F )† D † ΠD,

(83)

where
∗

dF (Π) = −(ΠT ⊗Π)dF (Ω(M̃ ⊗B)ΩT )

∗
= − (ΠT Ω)⊗(ΠΩ) (I K ⊗Z K,K ⊗I K )(vec(M̃ )⊗I K 2 )dF B

(84)

39

and

dF (B) = dF I −(I +F F † )−1


= (I +F F † )−T ⊗ (I +F F † )−1 (F ∗ ⊗I K )
= F ∗ (I +F F † )−T )⊗(I −B).

(85)

Then, defining a K ×K matrix F̃ = (I +F † F )−1 F † and a K 4 ×K 2 matrix

∗
Ψ = (I K ⊗Z K,K ⊗I K ) vec(M̃ )⊗I K 2 ,

(86)

and by combing (82)-(86), we finally have when P 6= 0,
dF IGMI (W opt , T opt , F )



= dF (I1 )+dF (δ1 )
† †
= vec FM + F̃ −vec(F )† D † ΠD


T
+vec(F )† D † (ΩT ΠDvec(F ))T ⊗ ΠΩ) Ψ F̃ ⊗(I −B) .
A PPENDIX E: T HE P ROOF OF P ROPOSITION 3
As the formula of GMI in (12) is quadratic in V and no constraints apply to V , taking the
gradient of IGMI (V , R, G) with respect to V and setting it to zero, yields the optimal V given
in (26). Inserting V opt into (12) gives, after some manipulations


IGMI (V opt , R, G) = K +log det(I +G) +2R Tr(P M R)


+Tr M (I +G) +Tr (I +G)−1RM̃ R†

(87)

where M and M̃ are defined in (14) and (15). If P = 0, (87) equals


I2 (G) = K +log det(I +G) +Tr M (I +G) .
When P 6= 0, the terms of IGMI (V opt , R, G) in (87) related to R are


g(R) = 2R Tr(P M R) +Tr (I +G)−1 RM̃ R† .
Let r k denote the kth column of R, but where all elements in rows [max(0, k −νR ), min(k +
T
T ]T , then we
νR , K − 1)] are removed, and define the column vector r = [ r T
0 r 1 . . . r K−1

have r = Ωvec(R). Moreover, let dk denote the kth column of the matrix M P but with
all elements in rows [max(0, k − νR ), min(k + νR , K − 1)] are removed and define the vector

40

T ]T . From the definition of d, we have d = Ωvec(M P ). Defining a
T
d = [ dT
0 d1 . . . dK−1

Hermitian matrix B̂ 2 as

∗
B̂ 2 = Ω M̃ ⊗(I +G)−1 ΩT ,
we can write f (R) as g(R) = r † B̂ 2 r+2R{d† r}. Therefore, the optimal r is
−1

r opt = −B̂ 2 d.

(88)

Transferring r opt back into Ropt gives the optimal R in (27) and inserting this into g(R) gives
−1

g(Ropt ) = −d† B̂ 2 d.
Thus, with the optimal V and R, when P 6= 0 the GMI equals


IGMI (V opt , Ropt , G) = K +log det(I +G) +Tr M (I +G)
 −1
∗
−d† Ω M̃ ⊗(I +G)−1 ΩT d.

A PPENDIX F: T HE G RADIENT IN M ETHOD II

FOR

F INITE L INEAR V ECTOR C HANNEL

Now we calculate the α-differential of IGMI (V opt , Ropt , G) given in (28) with respect to G
when P 6= 0. Taking the α-differential of I2 (G) with respect to G yields,
dG (I2 ) = vec((I +G)−1 +M )† .
(89)
 −1
∗
and taking the α-differential
Define an S×S Hermitian matrix Φ = Ω M̃ ⊗(I + G)−1 ΩT
of δ2 (G) with respect to G yields,
dG (δ2 ) = −(dT ⊗d† )dG (Φ)
 
∗
= (dT ⊗d† )(ΦT ⊗Φ)dG Ω M̃ ⊗(I + G)−1 ΩT


∗
= (dT ΦT )⊗(d† Φ) (Ω⊗Ω)dG M̃ ⊗(I +G)−1


= (dT ΦT Ω)⊗(d† ΦΩ) ΨdG (I +G)−1


= − (dT ΦT Ω)⊗(d† ΦΩ) Ψ (I +G)−T ⊗(I +G)−1
where Ψ is defined in (86). Combining (89) and (90), we can obtain

dG IGMI (V opt , Ropt , G) = dG (I2 )+dG (δ2 )
†
= vec (I +G)−1 +M


− (dT ΦT Ω)⊗(d† ΦΩ) Ψ (I +G)−T ⊗(I +G)−1 .

(90)

41

A PPENDIX G: T HE C ONCAVITY P ROOF OF M ETHOD II WITH F INITE L INEAR V ECTOR
C HANNELS


When P = 0, as log det(I+G) is concave [49] and Tr M (I+G) is linear in G, the function
I2 (G) in (29) is concave with respect to G whenever I +G is positive definite.
The concavity when P 6= 0 can be deduced from the composition theorem in [49, Chpater
3.6]. For a positive definite matrix X, d† X−1 d is convex and non-increasing (with respect to the
generalized inequality for positive definite Hermitian matrices, see [49], [50]) for any column
vector d. Furthermore, since I +G is positive definite, (I +G)−1 is convex. As M̃ ≺ 0 X =
 −1

∗
∗
Ω M̃ ⊗(I+G)−1 ΩT is concave in G. By the composition theorem, d† Ω M̃ ⊗[I+G]−1 ΩT d
is convex, and δ2 (G) is then concave. Therefore the function IGMI (V opt , Ropt , G) in (28) is
concave with respect to G whenever I +G is positive definite.
A PPENDIX H: T HE P ROOF OF P ROPOSITION 5
The Fourier series associated to the Toeplitz matrix W is
W (ω) =

∞
X

wk exp(jkω) ,

k=−∞

¯ (ω), T (ω), F (ω)) in (50) with respect to wk (where ω is fixed) is
and the differential of I(W

Z
∂ I¯
1 π |F (ω)|2 N0 + |H(ω)|2 W ∗ (ω)
=−
exp(jkω) dω
∂wk
2π −π
1 + |F (ω)|2
Z

1 π ∗
α|F (ω)|2 H(ω)T ∗ (ω) 
+
F (ω)H(ω) +
exp jkω dω.
(91)
2
π −π
1 + |F (ω)|
Since (91) should equal zero for all k, the optimal W (ω) is given in (55). Inserting Wopt (ω)
back into (50) yields,
Z
α π  ∗
¯
I(Wopt (ω), T (ω), F (ω)) = 1+
R F (ω)T (ω)M (ω) dω
π −π
Z

 M̃ (ω)|T (ω)F (ω)|2
1 π
2
2
+
log 1+|F (ω)| +
+M (ω) 1+|F (ω)| dω.
2π −π
1+|F (ω)|2

(92)

where M (ω) and M̃ (ω) are defined in (51) and (52). When α = 0, the GMI in (92) equals (58),
and when 0 < α ≤ 1, the terms related to T (ω) in (92) are
Z
Z
α π  ∗
1 π M̃ (ω)|T (ω)F (ω)|2
R F (ω)T (ω)M (ω) dω+
f (T (ω)) =
dω.
π −π
2π −π
1+|F (ω)|2

(93)

42

As the elements of the main diagonal and the first ν lower diagonals of matrix T are constrained
to zero, we define the vector t̃ that specifies the Toeplitz matrix T as
t̃ = [ t−NT . . . t−1 tν+1 . . . tNT ],
and with φ(ω) defined in (53), the Fourier series T (ω) with a finite tap length NT is
X

T (ω) =
tk exp jkω = t̃φ(ω).

(94)

−NT ≤k≤NT ,k∈[0,ν]
/

Furthermore, with ε1 and ε2 defined in (54), (93) can be rewritten as

†
f (T (ω)) = t̃ε2 t̃ +2R t̃ε1 .
Therefore, the optimal t̃ is
t̃opt = −ε†1 ε−1
2 .

(95)

¯ (ω), T (ω), F (ω)) for
Putting t̃opt back into (92)-(94), the optimal T (ω) is given in (56) and I(W
the optimal W (ω) and T (ω) is given in (57).
A PPENDIX I: T HE P ROOF OF P ROPOSITION 7
∞
P
vk exp(jkω) and the
The Fourier series associated to the Toeplitz matrix V is V (ω) =
k=−∞

¯ (ω), R(ω), G(ω)) in (62) with respect to vk (where ω is fixed) is
differential of I(V

Z
∂ I¯
1 π N0 +|H(ω)|2 V ∗ (ω)
=−
exp(jkω) dω
∂vk
2π −π
1+G(ω)
Z

αH(ω)R∗ (ω) 
1 π
H(ω)+
+
exp jkω dω.
π −π
1+G(ω)

(96)

Since (96) shall equal zero for all k, the optimal V (ω) is given in (65). Putting Vopt (ω) in (65)
back into (62) yields,
¯ opt (ω), R(ω), G(ω)) = 1+ α
I(V
π

Z

π


1
R M (ω)R(ω) dω+
2π
−π

Z

π


log 1+G(ω)

−π

2


M̃ (ω)|R(ω)|
+
+M (ω) 1+G(ω) dω,
1+G(ω)

(97)

where M (ω) and M̃ (ω) are defined in (51) and (52). When α = 0, the GMI in (97) equals (68),
¯ opt (ω), R(ω), G(ω)) related to R(ω) in (97) are
and when 0 < α ≤ 1, the terms of I(V
Z
Z π
α π 
1
M̃ (ω)|R(ω)|2
g(R(ω)) =
R M (ω)R(ω) dω+
dω.
(98)
π −π
2π −π 1+G(ω)

43

Define the vector r̃ that specifies the Toeplitz matrix R as
r̃ = [ r−NR . . . r−νR −1 rνR +1 . . . rNR ],
and with ψ(ω) defined in (63), the Fourier series R(ω) with a finite tap length NR is
X

R(ω) =
rk exp jkω = r̃ψ(ω)

(99)

−NR ≤k≤NR ,k∈[−ν
/
R ,νR ]

where 2νR +1 is the band size that R is constrained to zero. With ζ 1 and ζ 2 defined in (64),
(98) can be written as g(R(ω)) = r̃ζ 2 r̃ † +2R {r̃ζ 1 }. Therefore, the optimal r̃ is
r̃ opt = −ζ †1 ζ −1
2 .

(100)

This shows that r̃ opt has Hermitian symmetry as G(ω), M (ω) and M̃ (ω) are all real valued,
thus Ropt (ω) is real. Putting r̃ opt back into (97)-(99), the optimal R(ω) is given in (66) and
¯ (ω), R(ω), G(ω)) for the optimal V (ω) and R(ω) is given in (67).
I(V
A PPENDIX J: T HE C ONCAVITY P ROOF OF M ETHOD II WITH ISI C HANNELS
¯ opt (ω), Ropt (ω), G(ω)) in (67) is concave with respect to G(ω), it is sufficient to
To prove I(V
† −1
prove that ζ †1 ζ −1
2 ζ 1 is convex with respect to G(ω). For a positive definite matrix ζ 2 , ζ 1 ζ 2 ζ 1 is

convex and non-increasing (with respect to a generalized inequality for positive definite Hermitian
matrices) in G(ω) for any vector ζ 1 and with arbitrary finite tap length NR . As matrix M̃ is
negative definite, ζ 2 in (64) is concave with respect to G(ω) under the constraint that I +G is
positive definite. Hence ζ †1 ζ −1
2 ζ 1 is convex in G(ω) by the composition theorem [49].
A PPENDIX K: T HE P ROOF OF L EMMA 5
From Theorem 2, the optimal G in Method III satisfies [(I +Gopt )−1 ]ν = −[M̂ ]ν . Note that
when P = 0, Method III and Method II are equivalent as M̂ = M . Hence, in order to prove
Lemma 4, it is sufficient to show that [M̂ ]ν converges to [M ]ν as N0 → 0 and ∞. When P ≺ I,
C k in (38) is positive definite, and as N0 → 0,
†
−1 −1
†
H † (HC k H † +N0 I)−1 H = C −1
k (H H +N0 C k ) H H

−1
†
−1
= C −1
I
−
N
C
(H
H)
+ O(N02 ).
0
k
k

Therefore with Ŵ and Ĉ defined in (37)-(41),
Ŵ H = I − N0 (H † H)−1 + O(N02 ),
Ĉ = [Ŵ H]\ν = −N0 [(H † H)−1 ]\ν + O(N02 ).

(101)

44

With (101) and M̂ in (44), it can be verified that lim [M̂ /N0 ]ν = −[(H † H)−1 ]ν . On the other
N0 →0

hand, when N0 → ∞, from (37)-(44) we have
N0 Ŵ = H † (HC k H † /N0 +I)−1 = H † + O(1/N0 ),
N0 Ĉ = [Ŵ H]\ν = [H † H]\ν + O(1/N0 ),

(102)

With (102) and M̂ defined in (44), it can be verified that lim [N0 (I+M̂ )]ν = [H † H]ν . Hence,
N0 →∞

from (73) [M̂ ]ν converges to [M ]ν as N0 → 0 and ∞, which completes the proof.
R EFERENCES
[1] D. D. Falconer, and F. R. Magee, “Adaptive channel memory truncation for maximum likelihood sequence estimation,”
The Bell Syst. Tech. J., vol.51, no. 9, pp. 1541-1562, Nov. 1973.
[2] S. A. Fredricsson, “Joint optimization of transmitter and receiver filter in digital PAM systems with a Viterbi detector,”
IEEE Trans. Inf. Theory, vol. IT-22, no. 2, pp. 200-210, Mar. 1976.
[3] C. T. Beare, “The choice of the desired impulse response in combined linear-Viterbi algorithm equalizers,” IEEE Trans.
Commun., vol. 26, pp. 1301-1307, 1978.
[4] N. Sundström, O. Edfors, P. Ödling, H. Eriksson, T. Koski, and P. O. Börjesson, “Combined linear-Viterbi equalizers-a
comparative study and a minimax design,” Proc. IEEE Veh. Technol. Conf. (VTC), Stockholm, Sweden, Jun. 1994, vol.
2, pp. 1263-1267.
[5] N. Al-Dhahir, and J. M. Cioffi, “Efficiently computed reduced-parameter input-aided MMSE equalizers for ML detection:
A unified approach,” IEEE Trans. Inf. Theory, vol. 42, pp. 903-915, Apr. 1996.
[6] M. A. Lagunas, A. I. Perez-Neia, and J. Vidal, “Joint beamforming and Viterbi equalizer in wireless communications,”
Proc. Asilomar Conf. Signals, Syst. & Comput. (ACSSC), Pacific Grove, (CA) , Nov.1997, vol. 1, pp. 915-919.
[7] S. A. Aldosari, S. A. Alshebeili, and A. M. Al-Sanie, “A new MSE approach for combined linear-Viterbi equalizers,”
Proc. IEEE Veh. Technol. Conf. (VTC), Tokyo, Japan, May 2000, vol. 3, pp. 1707-1711.
[8] R. Venkataramani and S. Sankaranarayanan, “Optimal channel shortening equalization for MIMO ISI channels,” Proc.
IEEE Global Telecommun. Conf. (GLOBECOM), New Orleans, (LO), Dec. 2008.
[9] A. Shaheem, Iterative detection for wireless communications, Ph.D. thesis, School of Electrical, Electronic and Computer
Engineering, University of Western Australia, 2008.
[10] U. L. Dang, W. H. Gerstacker, and D. T. M. Slock, “Maximum SINR prefiltering for reduced state trellis based equalization,”
Proc. IEEE Int. Conf. Commun. (ICC), Kyoto, Japan, Jun. 2011.
[11] R. Venkataramani and M. F. Erden, “A posteriori equivalence: A new perspective for design of optimal channel shortening
equalizers,” arXiv preprint: 0710.3802v1.
[12] I. Abou-Faycal and A. Lapidoth, “On the capacity of reduced complexity receivers for intersymbol interference channels”,
Proc. Conf. Inf. Sciences and Systems (CISS), Princeton University, Mar. 2000, pp. WA4 32-37.
[13] G. D. Forney Jr., “Maximum likelihood sequence estimation of digital sequences in the presence of intersymbol
interference,” IEEE Trans. Inf. Theory, vol. 18, no. 3, pp. 363-378, May 1972.
[14] A. J. Viterbi, “Error bounds for convolutional codes and an asymptotically optimum decoding algorithm,” IEEE Trans.
Inf. Theory, vol. IT-13, no. 2, pp. 260-269, Apr. 1967.

45

[15] N. Merhav, G. Kaplan, A. Lapidoth and S. Shamai, “On information rates for mismatched decoders,” IEEE Trans. Inf.
Theory, Nov. 1994.
[16] A. Ganti, A. Lapidoth, and I. E. Telatar, “Mismatched decoding revisited: General alphabets, channels with memory, and
the wide-band limit,” IEEE Trans. Inf. Theory, vol. 46, pp. 2315-2328, Nov. 2000.
[17] M. R. McKay, I. B. Collings, and A. M. Tulino, “Achievable sum rate of MIMO MMSE receivers: A general analytic
framework,” IEEE Trans. Inf. Theory, vol. 56, no. 1, Jan. 2010.
[18] F. Rusek and A. Prlja, “Optimal channel shortening of MIMO and ISI channels,” IEEE Trans. Wireless Commun., vol. 11,
no. 2, pp. 810-818, Feb. 2012.
[19] H. Weingarten, Y. Steinberg, and S. Shamai, “Gaussian codes and weighted nearest neighbor decoding in fading multipleantenna channels,” IEEE Trans. Inf. Theory, vol. 50, no. 8, pp. 1665-1686, Aug. 2004.
[20] F. Rusek, N. Al-Dhahir, and A. Gomaa, “A rate-maximizing channel-shortening detector with soft feedback side
information,” Proc. IEEE Global Telecommun. Conf. (GLOBECOM), Anaheim, (CA), Dec., 2012, pp. 1-6.
[21] A. Duel-Hallen and C. Heegard, “Delayed decision-feedback sequence estimation,” IEEE Trans. Commun., vol. 37, no. 5,
pp. 428-436, May 1989.
[22] J. Hagenauer, “Source-controlled channel decoding,” IEEE Trans. Commun., vol. 43, no. 9, pp. 2449-2457, Sep. 1995.
[23] S. M. Kay, “Fundamentals of statistical signal processing, volume I: Estimation theory,” Prentice Hall, Apr. 1993.
[24] B. M. Hochwald and S. ten Brink, “Achieving near-capacity on a multiple-antenna channel,” IEEE Trans. Commun., vol.
51, no. 3, pp. 389-399, Mar. 2003.
[25] L. Bahl, J. Cocke, F. Jelinek, and J. Raviv, “Optimal decoding of linear codes for minimizing symbol error rate,” IEEE
Trans. Inf. Theory, vol. IT-20, no. 2, pp. 284-287, Mar. 1974.
[26] C. Studer, S. Fateh, and D. Seethaler, “ASIC Implementation of soft-input soft-output MIMO detection using parallel
interference cancellation,” IEEE J. Solid-State Circuits, vol. 46, no. 7, pp. 1754-1765, Jul. 2011.
[27] M. Witzke, S. Bro, F. Schreckenbach, and J. Hagenauer, “Iterative detection of MIMO signals with linear detectors,” Proc.
Asilomar Conf. Signals, Syst. and Comput. (ACSSC), Monterey, (CA), Nov. 2002, pp. 289-293.
[28] J. Zhang, H. Nguyen, and G. Mandyam, “LMMSE-based iterative and turbo equalization methods for CDMA downlink
channels,” Proc. IEEE 6th Workshop Signal Process. Advances Wireless Commun., Jun. 2005, pp. 231-235.
[29] F. Rusek, D. Fertonani, “Bounds on the information rate of intersymbol interference channels based on mismatched
receivers,” IEEE Trans. Inf. Theory, vol. 58, no. 3, pp. 1470-1482, 2012.
[30] G. Caire and S. Shamai, “On the achievable throughput of a multiantenna Gaussian broadcast channel,” IEEE Trans. Inf.
Theory, vol. 49, no. 7, Jul. 2003.
[31] G. L.Turin, “An introduction to digital matched filters,” Proc. IEEE, vol. 64, pp. 1092-1112, Jul. 1972.
[32] G. Ungerboeck, “Adaptive maximum likelihood receiver for carrier-modulated data-transmission systems,” IEEE Trans.
Commun., vol. 22, pp. 624-636, May 1974.
[33] G. H. Golub and C. F. Van Loan, “Matrix computations,” third edition, Baltimore, MD: Johns Hopkins, 1996.
[34] A. Kavčić and J. M. F. Moura, “Matrix with banded inverses: algorithms and factorization of Gauss-Markov processes,”
IEEE Trans. Inf. Theory, vol. 46, no. 4, pp. 1495-1509, Jul. 2000.
[35] G. Colavolpe and A. Barbieri, “On MAP symbol detection for ISI channels using the Ungerboeck observation model,”
IEEE Commun. Lett., vol. 9, no. 8, pp. 720-722, Aug. 2005.
[36] F. Rusek, G. Colavolpe, and, C. Sundberg, “40 years with the Ungerboeck model: a Look at its potentialities [Lecture
Notes],” Signal Process. Mag., vol. 32, pp. 156-161, May 2015.
[37] F. Rusek, M. Loncar and A. Prlja, “A comparison of Ungerboeck and Forney models for reduced complexity lSI
equalization,” Proc. IEEE Global Telecommun. Conf. (GLOBECOM), Washington D.C., Dec. 2007, pp. 1431-1436.

46

[38] O. Edfors, M. Sandell, J. J. Van de Beek, S. K. Wilson and P. O. Borjesson, “OFDM channel estimation by singular value
decomposition,” IEEE Trans. Commun., vol. 46, no. 7, pp. 931-939, Jul. 1998.
[39] W. Hirt, Capacity and information rates of discrete-time channels with memory, Ph.D thesis, no. ETH 8671, Inst. Signal
and Inf. Process., Swiss Federal Inst. Technol., Zürich, 1988.
[40] U. Grenander and G. Szegő, Toeplitz forms and their applications, University of Calif. Press Berkeley and Los Angeles,
1958.
[41] R. M. Gray, “Toeplitz and circulant matrices: A review,” Foundations and Trends in Commun. and Inf. Theory, vol. 2, no.
3, pp 155-239, 2006.
[42] J. G. Proakis and M. Salehi, Digital communications, the fifth edition, McGraw-Hill, 2008.
[43] M. Tüchler, A. Singer and R. Kötter, “Minimum mean squared error (MMSE) equalization using priors,” IEEE Trans.
Signal Process., vol. 50, pp. 673-683, 2000.
[44] 3GPP TS 36.212: Evolved universal terrestrial radio access (E-UTRA); Multiplexing and channel coding, Release 12,
v12.4.0, Mar. 2015.
[45] P. Kabal and S. Pasupathy, “Partial-response signaling,” IEEE Trans. Commun., vol. COM-23, pp. 921-934, Sep. 1975.
[46] F. Rusek and O. Edfors, “An information theoretic characterization of channel shortening receivers,” Asilomar Conf.
Signals, Syst. and Comput. (ACSSC), Pacific Grove, (CA), Nov. 2013, pp. 2108-2112.
[47] J. R. Magnus, “On the concept of matrix derivative,” J. Multivariate Anal., vol. 101, no. 9, pp. 2200-2206, Oct. 2001.
[48] P. L. Fackler, “Notes on Matrix Calculus,” North Carolina State University, Sep. 2005.
[49] S. Boyd and L. Vandenberghe, Convex optimization, Cambridge University Press, 2004.
[50] C. Davis, “Notions generalizing convexity for functions defined on spaces of matrices,” Proc. Symp. Pure Math., vol. 7,
pp. 187-201, 1963.
[51] M. Tüchler and A. C. Singer, “Turbo Equalization: An Overview,” IEEE Trans. Inf. Theory., vol. 57, no. 2, pp. 920-952,
Feb. 2011.
[52] S. J. Lee, A. C. Singer, and N. R. Shanbhag, “Linear turbo equalization analysis via BER transfer and EXIT charts,” IEEE
Trans. Signal Process., vol. 53, no. 8, pp. 2883-2897, Aug. 2005.
[53] S. Hu and F. Rusek, “On the design of reduced state demodulators with interference cancellation for iterative receivers,”
IEEE Annual Int. Symp. Personal, Indoor, and Mobile Radio Commun. (PIMRC), Sep. 2015, pp. 981-985.
[54] A. Shaheem, H. Zepernick, and M. Caldera, “Enhanced channel shortened turbo equalization,” Int. Conf. Advanced Technol.
Commun. (ATC), 6-9 Oct. 2008, pp. 8-11.
[55] A. Glavieux, C. Laot, and J. Labat, “Turbo equalization over a frequency selective channel,” Proc. Int. Symp. Turbo Codes
Related Topics, Brest, France, Sep. 1997, pp. 96-102.
[56] R. Lopes and J. Barry, “The soft-feedback equalizer for turbo equalization of highly dispersive channels,” IEEE Trans.
Commun., vol. 54, no. 5, pp. 783-788, May 2006.
[57] A. Berthet, R. Visoz, and P. Tortelier, “Sub-optimal turbo-detection for coded 8-PSK signals over ISI channels with
application to EDGE advanced mobile system,” Proc. IEEE Veh. Technol. Conf. (VTC), Sep. 2000.
[58] N. Al-Dhahir, “FIR channel-shortening equalizers for MIMO ISI channels,” IEEE Trans. Commun., vol. 49, pp. 213-218,
Feb. 2001.
[59] S. ten Brink, “Convergence of iterative decoding,” Electron. Lett., vol.35, pp. 806-808, May 1999.
[60] S. ten Brink, “Convergence behavior of iteratively decoded parallel concatenated codes,” IEEE Trans. Commun., vol. 40,
pp. 1727-1737, Oct. 2001.
[61] D. Darsena and F. Verde, “Minimum-mean-output-energy blind adaptive channel shortening for multicarrier SIMO
transceivers,” IEEE Trans. Signal Process., vol. 55, no. 12, pp. 5755-5771, Dec. 2007.

47

[62] S. Hu, H. Kröll, Q. Huang, and F. Rusek, “An Optimal channel shortener design for reduced-state soft-output viterbi
equalizer in single-carrier systems,” IEEE Trans. Commun., vol. 65, no. 6, pp. 2568-2582, Jun. 2017.
[63] J. W. Choi, B. Shim, A. C. Singer, and N. I. Cho, “Low-complexity decoding via reduced dimension maximum-likelihood
search,” IEEE Trans. Signal Process., vol. 58, no. 3, pp. 1780-1794, Mar. 2010.
[64] J. W. Choi, B. Lee, and B. Shim, “Iterative group detection and decoding for large MIMO systems,” J. Commun. and
Netw., vol 17, no. 6, pp. 609-621, Dec. 2015.
[65] J. W. Choi, A. C. Singer, J. W. Lee, and N. I. Cho, “Improved linear soft-input soft-output detection via soft feedback
successive interference cancellation,” IEEE Trans. Commun., vol. 58, no. 3, pp. 986-996, Mar. 2010.

