DEEP LEARNING FOR ACCELERATED ULTRASOUND IMAGING
Yeo Hun Yoon and Jong Chul Ye ∗

arXiv:1710.10006v1 [cs.CV] 27 Oct 2017

Bio Imaging & Signal Processing Lab., Dept. of Bio and Brain Engineering, KAIST
291 Daehak-ro, Yuseong-gu, Daejeon, Republic of Korea
ABSTRACT
In portable, 3-D, or ultra-fast ultrasound (US) imaging systems, there is an increasing demand to reconstruct high quality images from limited number of data. However, the existing solutions require either hardware changes or computationally expansive algorithms. To overcome these limitations,
here we propose a novel deep learning approach that interpolates the missing RF data by utilizing the sparsity of the RF
data in the Fourier domain. Extensive experimental results
from sub-sampled RF data from a real US system confirmed
that the proposed method can effectively reduce the data rate
without sacrificing the image quality.
Index Terms— Deep learning, ultrasound imaging, lowrank Hankel matrix
1. INTRODUCTION
Due to the the excellent temporal resolution with reasonable
image quality and minimal invasiveness, ultrasound imaging
has been adopted as a golden-standard for many disease diagnosis in heart, liver, etc. Accordingly, there have been many
research efforts to extend the US imaging to new applications
such as portable imaging in emergency care, 3-D imaging,
ultra-fast imaging, etc. However, the common technical hurdle in these applications is to reconstruct high resolution images from significantly reduced radio-frequency (RF) data acquired with an US transducer.
While the compressed sensing approach has been investigated to address this issue [1], accurate modeling of wave
propagation is usually required. Recently, Wagner et al [2]
models a scan line profile as finite rate of innovation, and
proposed a specially-designed hardware architecture that enables high resolution scan line reconstruction using sparsity
driven recovery algorithm [2]. Another recent proposal is to
use a low rank Hankel matrix completion approach [3]. In
particular, thanks to the strong correlation between scan lines
and adjacent temporal frames, Jin et al [4] showed that a rank
deficient Hankel structured matrix can be obtained from the
reordered RF measurement data. Accordingly, the missing
∗ This work is supported by National Research Foundation of Korea
(NRF-2016R1A2B3008104).

data can be accurately reconstructed using a low rank Hankel matrix completion algorithm [3]. However, the algorithm
is computationally very expensive, which is not suitable for
routine diagnostic applications.
One of the most important contributions of this paper is
therefore to show that the low-rank interpolations problem of
RF data can be solved using deep convolutional neural network (CNN), which enables very fast run-time reconstruction.
In particular, inspired by recent finding that a CNN can be
interpreted as a deep convolutional framelets obtained from
a decomposition of Hankel matrix [5], we construct a CNN
that performs direct interpolation of RF data. Thus, the final
reconstruction can be done using the standard delay-and-sum
(DAS) beamformer without changing any hardware/software
structures. Compared to an image domain CNN that attempts
to learn acquisition geometry-dependent artifacts, one of the
most important advantages of the proposed RF domain CNN
is its universality. Specifically, although an image domain
deep learning requires many data set from various acquisition
geometry and body parts [6], our CNN can be trained using
a RF data measured by, for example, the linear array transducer for a particular organ, which can be then used for other
type of transducers and/or different body parts. Therefore, the
proposed systems is very practical in real applications.
Extensive experimental results using RF data from a real
US system confirmed that the proposed method has significant potential for accelerated US systems.
2. THEORY
2.1. Sub-sampled dynamic aperture US system
As a proof-of-concept, we consider a sub-sampled dynamic
aperture US system [4]. Specifically, a B-mode ultrasound
scans the body and displays the 2-D image as shown in Fig.1.
In Fig.1, SC, Rx and DAS denote scan line for each ultrasound beam, receivers of the transducer, and delay-and-sum
(DAS) beamformer, respectively. After a focused ultrasound
beam is illuminated along the scan line as shown in Fig.1(a),
ultrasound beam is reflected at some tissue boundaries and
the reflected US data are recorded by receivers as RF data
(see Fig.1(b)). Thus, for each scan line (SC), a Depth-Rx coordinate RF data is obtained, and this is repeated for each scan

(a) Beam emission

(b) RFs record
Fig. 2. Extraction of Rx-SC data for each depth direction.
when the Fourier component of the Rx-SC image is given by
fb, then we can find an annihilating function b
h in the spectral
domain such that their multiplication becomes zero:
fb b
h=0

(1)

This is equivalent to the convolutional relationship in the RxSC domain:
f ~ h = 0,
(c) conventional data acquisitions
Fig. 1. B-mode ultrasound imaging flow.
line to obtain a 3-D cube of RF data in Depth-Rx-SC coordinates. Then, a DAS beamformer uses the 3-D stacked data to
generate one temporal frame as shown in Fig.1(c). Because
of huge dat set, a hardware beamformer is usually incorporated to a transducer and the generated image is transferred
to an image workstation. On the other hand, recent softwarebased US systems directly transfer RF data to the image workstation, where a software-based beamformer reconstructs images. However, for many portable or high spatio/temporal
applications, online reconstruction or huge data transfer may
not be a viable option.
In this paper, we thus consider a randomly sub-sampled
dynamic aperture US system that acquires only partial data
as shown in Fig.3. Because hardware is not changed, this
method can be easily used in any conventional ultrasound system. Also, it can relieve the frame rate limitation caused by
pulse repetition frequency.
2.2. Redundancy of RF data and low-rank Hankel matrix
Recall that B-mode ultrasound measurements are acquired by
point-by-point scanning for each scan line. Since the scan line
only changes incrementally, the acquired Rx data along detectors do not change rapidly for each scan line. This implies
that there exists some level of skewed redundancy in Rx-SC
coordinate data, which can be also observed in Fig. 2.
This type of redundancy can be easily seen as a sparsity
in the Fourier domain as demonstrated in Fig. 2. Therefore,

(2)

which can be represented in a matrix form :
H(f )h = 0,

(3)

where H(f ) is a Hankel structured matrix constructed from
f . This implies that the Hankel matrix constructed from reordered raw data is rank-deficient. Furthermore, its rank is
determined by sparsity level as theoretically proven in [3]. In
fact, Jin et al [4] utilized this idea to interpolate missing RF
data using low-rank Hankel matrix completion. However, the
main limitation of [4] is its computational complexity, which
is prohibitive in real applications.
2.3. Deep Convolutional Framelets
Recently, it was shown that CNN is closely related to Hankel
matrix decomposition [5]. Specifically, for a given Hankel
matrix H(f ) ∈ Rn×d , let Φ and Φ̃ ∈ Rn×m (resp. Ψ and
Ψ̃ ∈ Rd×q ) are frame and its dual frame, respectively, satisfying the frame condition: Φ̃Φ> = I, ΨΨ̃> = I such that it
satisfies the perfect reconstruction condition:
H(f ) = Φ̃Φ> H(f )ΨΨ̃> .

(4)

One of the most important discoveries in [5] is to show that
(4) can be equivalently represented in the signal domain using
so-called deep convolutional framelets expansion, where the
decoder-part convolution is given by


f =
Φ̃C ~ τ (Ψ̃)
(5)
where C is the framelet coefficient matrix obtained from the
encoder part of convolution:

C = Φ> f ~ Ψ .
(6)

Fig. 3. Network architecture for B-mode ultrasound reconstruction
Here, the local filters for the encoder and decoder part convolution are given by
 
ψ̃1


1 . 
d×q
, τ (Ψ̃) :=  ..  ∈ Rdq .
Ψ := ψ 1 · · · ψ q ∈ R
d
ψ̃q
where v represents a flipped version of a vector v. Moreover,
for a given matrix input Z ∈ Rn×p , we have the following
convolutional framelet expansion [5]:
Z

=

(ΦC) ~ τ (Ψ̃)

where the framelet coefficients C is given by

C = Φ> Z ~ Ψ
Here, the encoder and decoder filters are defined by
 1

1
ψ1 · · · ψq
 .
dp×q
.. 
..

Ψ := 
.
 ..
. ∈R
p
ψ1

τ (Ψ̃)

:=

 1
ψ̃1
1 .
 .
d .
ψ̃q1

···

···
..
.
···

(7)

(8)

(9)

p
ψq


ψ̃1p
..  ∈ Rdq×p
. 

(10)

ψ̃qp

The simple convolutional framelet expansion using (5), (7),
(6) and (8) is so powerful that a CNN with the encoderdecoder architecture emerges from them by inserting the pair
(8) and (7) between the pair (6) and (5). Moreover, a deep
CNN training can be interpreted to learn the basis matrix Ψ
for a given basis Φ such that maximal energy compaction can
be achieved. For more detail, see [5].
In brief, due to the redundancy in Rx-SC domain data,
the associated Hankel matrix is low-ranked, which makes the
convolutional framelet coefficients sparse. This implies that
a directly interpolation of Rx-SC domain RF data is feasible
using deep CNN.

3. METHOD
US data in the RF domain were acquired by E-CUBE 12R US
system (Alpinion Co., Korea). Real RF data were acquired by
a linear array transducer (L3-12H) with the center frequency
of 8.48MHz and a convex array transducer (SC1-4H) with the
center frequency of 3.2MHz. The sampling frequencies were
40MHz. We first acquired RF data from nine subjects using a linear array transducer. The size of each Rx-SC planes
is 64×384. 15000 Rx-SC planes of the seven people data
sets are randomly selected for generating training data set,
and 3000 Rx-SC planes of the another person data sets are
randomly selected for generating validation data set. The remaining one person data sets are used as test set. In addition,
we acquired RF data of liver from one subject using a convex
array transducer. This data set was used to verify the universality of the algorithm. The proposed CNN is composed
of convolution layers, batch normalization, ReLU and bypass
connection with concatenation as shown in Fig. 3. Specifically, the network consists of 28 convolution layers composed
of a batch normalization and a ReLU except for the last convolution layer. The first 27 convolution layers use 3×3 kernel,
and the last convolution layer use 1×1 kernel. Four bypass
connections with concatenation exist. As input data, we used
randomly sub-sampled data at the downsampling ratio of 4.
Because our proposed method assumes a dynamic aperture
US system, the RF data are sub-sampled along the receivers.
And the full data of Rx-SC plane was used as the label data.
For network training, the number of epoch was 400. The regularization parameter was 10−4 . The network was trained by
the stochastic gradient descent. The learning rate started from
10−7 and gradually decreased down to 10−9 . The network
was implemented using MatConvNet.
4. EXPERIMENTAL RESULTS
Fig. 4 illustrates B-mode ultrasound images acquired from
linear array transducer. The proposed CNN-based interpola-

Fig. 4. Linear transducer US images from (first row) full RF
data, (second row) x4 downsampled RF data, and (last row)
proposed CNN-based interpolation from x4 sub-sampled RF
data.

Fig. 5. Convex transducer US images from (first row) full RF
data, (second row) x4 downsampled RF data, and (last row)
proposed CNN-based interpolation from x4 sub-sampled RF
data.
important platform for accelerated US imaging.

tion successfully reconstructed the missing RF data from only
25% of RF data such that the beamformer removes the blurring artifacts and generates near artifact-free images. The average of PSNR for the reconstruction B-mode image is around
30.92, which is about 10dB improvement compared to the Bmode images from sub-sampled RF data.
Next, we use the network trained using the RF data from
linear array transducer to interpolate the missing RF data from
a convex array probe. The reconstruction results from a DAS
beamformer are shown in in Fig.5. Because the Rx-SC data of
linear array transducer and convex array transducer are similar, the DAS beamformer provided very accurate reconstruction results without any line artifacts or blurring from CNNbased RF interpolation. It is also remarkable that the accurate reconstruction was obtained for the liver region, which
was never seen by the network trained using the linear array
transducer data. The results confirmed the universality of the
algorithm.
5. CONCLUSIONS
In this paper, we proposed a novel deep learning approach
for accelerated B-mode ultrasound imaging. To exploit the
redundancy in the RF domain, the proposed CNN was applied
to Rx-SC domain. Compared with the methods that need to
change the hardware, the proposed method does not need any
hardware change and can be applied to any B-node ultrasound
system or any transducer. Therefore, this method can be an

6. REFERENCES
[1] Hervé Liebgott, Rémy Prost, and Denis Friboulet, “Prebeamformed RF signal reconstruction in medical ultrasound using compressive sensing,” Ultrasonics, 2012.
[2] Noam Wagner, Yonina C Eldar, and Zvi Friedman,
“Compressed beamforming in ultrasound imaging,”
IEEE Trans. Signal Process., vol. 60, no. 9, pp. 4643–
4657, 2012.
[3] Jong Chul Ye, Jong Min Kim, Kyong Hwan Jin, and
Kiryung Lee, “Compressive sampling using annihilating
filter-based low-rank interpolation,” IEEE Transactions
on Information Theory, vol. 63, no. 2, pp. 777–801, Feb.
2017.
[4] Kyong Hwan Jin, Yo Seob Han, and Jong Chul
Ye, “Compressive dynamic aperture b-mode ultrasound
imaging using annihilating filter-based low-rank interpolation,” in 2016 IEEE 13th International Symposium on
Biomedical Imaging (ISBI). IEEE, 2016, pp. 1009–1012.
[5] Jong Chul Ye and Yo Seob Han, “Deep convolutional
framelets: A general deep learning for inverse problems,”
arXiv preprint arXiv:1707.00372, 2017.
[6] Eunhee Kang, Jaejun Yoo, and Jong Chul Ye, “Wavelet
residual network for low-dose CT via deep convolutional
framelets,” arXiv preprint arXiv:1707.09938, 2017.

