Synthesizing CT from Ultrashort Echo-Time MR Images via Convolutional Neural Networks
Snehashis Roy John A. Butman Dzung L. Pham








CT Synthesis from UTE via CNN




S. Roy et. al.

1 
Center for Neuroscience and Regenerative Medicine, Henry Jackson Foundation, USA

2 Radiology and Imaging Sciences, Clinical Center, National Institute of Health, USA



Synthesizing CT from Ultrashort Echo-Time MR Images via Convolutional Neural Networks
    Snehashis Roy1 
Support for this work included funding from the Department of Defense in the
Center for Neuroscience and Regenerative Medicine.
John A. Butman2 
Dzung L. Pham1

    December 30, 2023
====================================================================================================================================================================================





With the increasing popularity of PET-MR scanners in clinical applications, synthesis of CT images 
from MR has been an important research topic. Accurate PET image reconstruction requires attenuation 
correction, which is based on the electron density of tissues and can be obtained from CT images. 
While CT measures electron density information for x-ray photons, MR images convery information 
about the magnetic properties of tissues.  Therefore, with the advent of PET-MR systems, the 
attenuation coefficients  need to be indirectly estimated from MR images. In this paper, we propose 
a fully convolutional neural network (CNN) based method to synthesize head CT from ultra-short 
echo-time (UTE) dual-echo MR images. Unlike traditional T_1-w images which do not have any bone 
signal, UTE images show some signal for bone, which makes it a good candidate for MR to CT 
synthesis. A notable advantage of our approach is that accurate results were 
achieved with a small training data set. Using an atlas of a single CT and dual-echo UTE pair, we 
train a deep neural network model to learn the transform of
MR intensities to CT using patches. We compared our CNN based model with a state-of-the-art 
registration based as well as a Bayesian model based CT synthesis method, and showed that the 
proposed CNN model outperforms both of them. We also compared the proposed model when only T_1-w 
images are available instead of UTE, and show that UTE images produce better synthesis 
than using just T_1-w images.



 




§ INTRODUCTION


Accurate PET (positron emission tomography) image reconstruction requires correction for the 
attenuation of γ photons by tissue. The attenuation coefficients, called μ-maps, can be 
estimated from CT images, which are x-ray derived estimates of electron densities in tissues. 
Therefore PET-CT scanners 
are well suited for accurate PET reconstruction. In recent years, PET-MR scanners have become more 
popular in clinical settings. This is because of the fact that unlike CT, MRI (magnetic resonance 
imaging) does not impart any radiation, and MR images have superior soft tissue contrast. However, 
an MR image voxel contains information about the magnetic properties of the tissues at that voxel, 
which has no direct relation to its electron density. Therefore synthesizing CT from MRI is an 
active area of research.

Several MR to CT synthesis methods for brain images have been proposed. Most of them can be 
categorized into two classes – segmentation based and atlas based. CT image intensities
represent quantitative Hounsfeld Units (HU) and their standardized values are usually known for air, 
water, bone, and other brain tissues such as fat, muscle, grey matter (GM), white matter (WM), 
cerebro-spinal fluid (CSF) etc. Segmentation based methods 
<cit.> first segment a T_1-w MR image of the whole head into 
multiple classes, such as bone, air, GM, and WM. Then each of the segmented classes are replaced 
with the mean HU for that tissue class, or the intensity at a voxel is obtained from the 
distribution of HU for the tissue type of that voxel.


Most segmentation based approaches rely on accurate multi-class segmentation of T_1-w images. 
However, traditional T_1-w images do not produce any signal for bone. As bone has the highest 
average HU compared to other soft tissues, accurate segmentation of bone is crucial for accurate PET 
reconstruction. 
Atlas based methods <cit.> can overcome this limitation via registration. An atlas 
usually consists of an MR and a co-registered CT pair. For a new subject, multiple atlas MR images 
can be deformably registered to the subject MR; then the deformed atlas CT images are combined using 
voxel based label fusion <cit.> to generate a synthetic subject CT. It has 
been shown that atlas based methods generally outperform segmentation based methods 
<cit.>, because they do not need accurate segmentation of tissue classes, which can be 
difficult because it becomes indistinguishable from background, tissues with short T_1, and 
tissues whose signal may be suppressed, such as CSF. 

One disadvantage of registration based methods is that a large number of atlases is needed for
accurate synthesis. For example, 40 atlases were used in <cit.>, leading to 
significantly high computational cost with such a large number of registrations. To alleviate this 
problem, atlas based patch matching methods have been proposed <cit.>. For a 
particular patch on a subject MR, relevant matching patches are found from atlas MR images. The 
atlases only need to be rigidly registered to the subject <cit.>. The matching atlas MR 
patches can either be found from a neighborhood of that subject MR patch <cit.>, or from 
any location within the head <cit.>. Once the matching patches are found, their 
corresponding CT patches are averaged with weights based on the patch similarity to form a 
synthetic CT. The advantage of patch matching is that deformable registration is not needed, thereby 
decreasing the computational burden and increasing robustness to differences in the anatomical 
shape. These type of methods also require fewer atlases (e.g., 10 in <cit.> and 1 in 
<cit.>).

Recently, convolutional neural networks (CNN), or deep learning <cit.>, has been 
extensively used in many medical imaging applications, such as lesions and tumor segmentation
<cit.>, brain segmentation, image synthesis, and skull stripping. Unlike traditional 
machine learning algorithms, CNN models do not need hand-crafted features, and are therefore 
generalizable to a variety of problems. They can accommodate whole images or much larger patches 
(e.g., 17^3 in <cit.>) compared to smaller sized patches used in most patch based 
methods (e.g., 3^3 in <cit.>), thereby introducing better neighborhood 
information. A CNN model based on U-nets <cit.> has been recently proposed to 
synthesize CT from T_1-w images <cit.>. In this paper, we propose a synthesis method based 
on fully convolutional neural networks to generate CT images from dual-echo UTE images. We 
compare with two leading CT synthesis methods, one registration based <cit.> and one 
patch based <cit.>, and show that our CNN model produces more accurate results compared to 
both of them. We also show  that better synthesis can be obtained using UTE images rather than
only T_1-w images.





§ DATA DESCRIPTION

MR images were acquired on 7 patients on a 3T Siemens Biograph mMR. The MR acquisition includes
T_1-w dual-echo UTE and MPRAGE images. The specifications of UTE images are as follows,
image size 192 × 192 × 192, resolution 1.56 mm3, repetition time TR=11.94s, 
echo time TE=70μs and 2.46ms, flip angle 10^∘. MPRAGE images were acquired 
with the following parameters, resolution 1.0 mm3, TR=2.53s, TE=3.03ms, flip angle 
7^∘. CT images were acquired on a Biograph 128 Siemens PET/CT scanner with a tube voltage of 
120 kVp, with dimensions of 512 × 512 × 149, and resolution of 0.58 × 0.58 
× 1.5 mm3. MPRAGE and CT were rigidly registered <cit.> to the second UTE 
image. All MR images were corrected for intensity inhomogeneities by N4 <cit.>. The 
necks were then removed from the MPRAGE images using FSL's  <cit.>. 
Finally, to create a mask of the whole head, background noise was removed from the MPRAGE using 
Otsu's threshold <cit.>. UTE and CT images were masked by the headmask obtained from the 
corresponding MPRAGE. Note that the choice of MPRAGE to create the headmask is arbitrary, CT could 
also be used as well. The headmask was used for two purposes.


  * Training patches were obtained within the headmask, so that the center voxel of a patch 
contains either skull or brain.

  * Error metrics between synthetic CT and the original CT were computed only within the headmask.






§ METHOD


We propose a deep CNN model to synthesize CT from UTE images. Although theoretically the model can 
be used with whole images, we used patches due to memory limitations. Many CNN architectures have 
previously been proposed. In this paper, we adopt Inception blocks <cit.>, that have 
been successfully used in many image classification and recognition problems in natural image 
processing  via GoogleNet. The rationale for using this architecture over U-net 
is discussed in Sec. <ref>. The proposed CNN architecture is shown in 
Fig. <ref>. 

Convolutions and pooling are two basic building layers of any CNN model. Traditionally they 
are used in a linear manner, e.g. in text classification <cit.>. The primary innovation
of the Inception module <cit.> was to use them in a parallel fashion. In an Inception 
module, there are two types of convolutions, one with traditional n^3 (n>1) filter banks, and one 
with 1^3 filter banks. It is noted that 1^3 filters are downsampling the number of channels. The 
1^3 filters are used to separate initial number of channels (128) into multiple smaller sets 
(96, 16, and 64). Then the spatial correlation is extracted via n^3 (n>1) filters. The 
downsampling of channels and parallelization of layers reduce the total number of parameters to be 
estimated,which in turn introduces more non-linearity, thereby improving classification accuracy
<cit.>. Note that the proposed model is fully convolutional, as we did not 
use a fully connected layer.


During training, 25 × 25× 5 patches around each voxel within the headmask are extracted 
from the UTE images with stride 1. Then the patches from each UTE image are first convolved with 
128 filters of size 3× 3× 3. Such a filter is denoted by 128@3^3 in 
Fig. <ref>. The outputs of the
filters are processed through separate Inception blocks. The outputs of these Inception blocks are 
then concatenated through their channel axis (which is same as the filter axis)
and processed through another Inception block and a 3^3 filter. The coefficients 
of all the filters are computed by minimizing mean squared errors between the CT patch and the 
output of the model via stochastic gradient descent. Note that every convolution is followed by a 
ReLU (rectified linear unit), module, which is not shown in the figure. The pooling is performed by 
replacing each voxel of a feature map by the maximum of its 3× 3× 3 neighbors.







We used Adam <cit.> as the 
optimizer to estimate the filter weights. Adam has been shown to produce much faster convergence
than comparable optimizers. While training, we used 75% of the total atlas patches as training 
set and 25% as the validation set. To obtain convergence, 25 epochs were used. The filter
parameters were initialized by randomly choosing numbers from a zero-mean Gaussian distribution
with standard deviation of 0.001. A batch size of 64 was empirically chosen and found to 
produce sufficient convergence without requiring much GPU memory. The model was implemented in Caffe 
<cit.>. Anisotropic 25× 25× 5 patches were used because larger size isotropic 
patches requires more GPU memory, while the patch size was empirically estimated. To compensate for 
the fact that patches are anisotropic, the atlas was reoriented in three 
different orientations – axial, coronal, and sagittal. Training was performed separately for
each oriented atlas to generate three models, one for each orientation. Then for a new subject, the 
models were applied on the corresponding reoriented versions of the subject, and then averaged to 
generate a mean synthetic CT. Training on a TITAN X GPU with 12GB memory takes about 6 hours. 
Synthesizing a CT image from a new subject takes about 30 seconds, where approximately 10
seconds is needed to predict one orientation. Although the training is
performed using 25× 25× 5 patches, the learnt models are able to predict a whole 2D 
slice of the image by applying the convolutions on every slice. Each of the three learnt models
were used to predict every 2D slice of the image in each of the three orientations. Then the 3
predicted images were averaged to obtain the final synthetic CT.



§ RESULTS


We compared our CNN based method to two algorithms, GENESIS <cit.> and intensity fusion
<cit.>. GENESIS uses dual-echo UTE images and generates a synthetic CT based on another 
pair of UTE images as atlases. While GENESIS is a patch matching method which does not need any 
subject to atlas registration, the intensity fusion method (called “Fusion") registers atlas 
T_1-w images to a subject T_1-w image, and combines the registered atlas CT images based on 
locally normalized correlation. In our implementation of Fusion, the second echo of an UTE image
pair was chosen as the subject image and was registered to the second echo UTE images of the 
atlases. The second echo was chosen for registration as its contrast closely 
matches the regular T_1-w contrast used in <cit.>.
Similar to <cit.> which proposed a CNN model only using T_1-w images, we also 
compared the proposed model with both channels as the MPRAGE.


One patient was arbitrarily chosen to be the “atlas" for both GENESIS and the proposed CNN model
with both UTE and MPRAGE as inputs. The trained CNN models are applied to the other 6 subjects. 
Since Fusion requires multiple atlas registrations, the validation is computed in a leave-one-out 
manner only for Fusion. GENESIS was also trained on the same atlas and evaluated on the remaining 
6.

Fig. <ref> shows examples of two subjects, one healthy volunteer and one with a large
lesion in the left frontal cortex. For the healthy volunteer, all of the three methods perform 
similarly, while Fusion shows some diffused bone. It is because the deformable registrations can be
erroneous, especially in presence of skull. CNN with MPRAGE shows some artifacts near ventricles
(yellow arrow), while CNN with UTE images provide the closest representation to the original CT. For 
the subject with a brain lesion, Fusion can not successfully reproduce the lesion, as none of the 
atlases have any lesion in that region. CNN with MPRAGE shows artifacts where CSF is misrepresented 
as bone (blue arrow). This can be explained by the fact that both CSF and MPRAGE have low signal on 
MPRAGE. Synthetic CT from CNN with UTE shows the closest match to the CT, followed by GENESIS, which 
is noisier.




To quantitatively compare the competing methods, we used PSNR and linear correlation coefficient
between the original CT and the synthetic CTs. PSNR is defined as a measure of mean squared error 
between original CT 𝒜 and a synthetic CT ℬ as, 
PSNR=10log_10(MAX_𝒜^2/||𝒜-ℬ||^2),
where MAX_𝒜 denotes the maximum value of the image 𝒜. Larger PSNR indicates
better matching between 𝒜 and ℬ. Table <ref> shows the PSNR 
and correlation for Fusion, GENESIS, the proposed CNN model with only MPRAGE and with dual-echo UTE 
images. The proposed model with UTE images produces the largest PSNR and correlation compared
to both GENESIS and Fusion, as well as CNN with MPRAGE. A Wilcoxon signed rank test showed a
p-value of 0.0312 comparing CNN with UTE with the other three for both PSNR and correlation,
indicating significant improvement in CT synthesis. Note that we used only 6 
atlases for our implementation of Fusion, although the original paper <cit.> recommended 
40 atlases. Better performance would likely have been achieved with additional 
atlases.  Nevertheless, the proposed model outperforms it with only one atlas.



§ DISCUSSION


We have proposed a deep convolutional neural network model to synthesize CT from dual-echo UTE
images. The advantage of a CNN model is that 
prediction of a new image takes less than a minute. This efficiency is especially useful in clinical 
scenarios when using PET-MR systems, where PET attenuation correction is immediately 
needed after MR acquisition. Another advantage of the CNN model is that no atlas registration is 
required. Although adding multiple atlases can increase the training time linearly, the prediction
time (∼ 30 seconds) is not affected by the number of atlases. This is significant in comparison 
with patch based <cit.> and registration based approaches <cit.> 
(∼ 1 hr), where adding more atlases increases the prediction time linearly.

The primary limitation of the proposed, or in general, any CNN model is that it requires 
large amount of training data because the number of free parameters to estimate is usually large. In 
our case, by using only 3 Inception modules, the total number of free parameters are approximately 
29,000. We used all patches inside the headmask which was about 500,000 for the 1.56 mm3 
UTE images. By adding more Inception modules, as done in GoogleNet <cit.>, the number of 
free parameters grow exponentially, which needs more training data.  An important 
advantage of the proposed model over the U-net in <cit.> is that only a single UTE image 
pair was used as atlas. Since we used patches instead of 2D slices <cit.> 
for training, the number of training samples is not limited by the number of slices in an atlas. 
One atlas with 256 slices was used to generate 500,000 training samples, which was 
sufficient to produce better results than competing methods. In clinical applications, it can be
difficult to obtain UTE and high resolution CT images for many subjects. Therefore using patches 
instead of slices give exponentially more training samples.



The patch size (25× 25× 5 ) is an important parameter of the model which was chosen 
empirically to make best practical use of the available GPU memory. Although CNN models do not need 
hand-crafted features, it was observed that using bigger patches usually increases accuracy. 
However, there lies a trade-off between patch size and available memory. Future work includes
optimization of patch size and number of atlases, as well as exploring further CNN architectures.


splncs03




