Interpatient Respiratory Motion Model Transfer for Virtual Reality Simulations of Liver Punctures
Andre Mastmeyer Matthias Wilms Heinz Handels
cs.CV


[@twocolumnfalse

Spectral sequences of Type Ia supernovae. I. Connecting normal and sub-luminous SN Ia and the presence of unburned carbon
    [
    
=========================================================================================================================


numberstyle

Current virtual reality (VR) training simulators of liver punctures often rely on static 3D patient data and use an unrealistic (sinusoidal) periodic animation of the respiratory movement.
Existing methods for the animation of breathing motion support simple mathematical or patient-specific, estimated breathing models. However with personalized breathing models for each new patient, a heavily dose relevant or expensive 4D data acquisition is mandatory for keyframe-based motion modeling.
Given the reference 4D data, first a model building stage using linear regression motion field modeling takes place. Then the methodology shown here allows the transfer of existing reference respiratory motion models of a 4D reference patient to a new static 3D patient. This goal is achieved by using non-linear inter-patient registration to warp one personalized 4D motion field model to new 3D patient data.
This cost- and dose-saving new method is shown here visually in a qualitative proof-of-concept study. 




 §.§ Keywords

Virtual Reality, Liver Puncture Training, 4D Motion Models, Inter-patient Registration of Motion Models



]



§ INTRODUCTION




The virtual training and planning of minimally invasive surgical interventions with virtual reality simulators provides an intuitive, visuo-haptic user interface for the risk-sensitive learning and planning of interventions. 
The simulation of liver punctures has been an active research area for years <cit.>.



Obviously first, the stereoscopic visualization of the anatomy of the virtual patient body is important <cit.>. Second, the haptic simulation of the opposing forces through the manual interaction, rendered by haptic input and output devices, with the patient is key <cit.>. Third in recent developments, the simulation of the appearance and forces of the patient's breathing motions is vital <cit.>.

The previously known VR training simulators usually use time invariant 3D patient models. A puncture of the spinal canal can be simulated sufficiently plausibly by such models. In the thoracic and upper abdominal region, however, respiratory and cardiac movements are constantly present. In the diaphragm area at the bottom of the lungs just above the liver, breathing movement differences in the longitudinal z direction of up to 5 cm were measured <cit.>. Now for 4D animation, the necessary data consists of a single 3D CT data set and a mathematical or personalized animation model.
Our 
aim here is to incorporate these physiological-functional movements into realistic modeling in order to offer the user a more realistic visuo-haptic VR puncture simula­tion. This means also to take into account the intra- and intercycle variability (hysteresis, variable amplitude during inhalation / exhalation).

A major interest and long term goal of virtual and augmented reality is the planning <cit.> and intra-operative navigation assistance <cit.>. However, in these works breathing motion is not incorporated or applicability limits by neglecting breathing motion in terms of minimal tumor size are given <cit.>.
Published approaches from other groups <cit.> model only a sinusoidal respiratory motion without hysteresis and amplitude variation. First steps in the direction of a motion model building framework were taken by our group <cit.>. Accurate simulation of respiratory motion depending on surrogate signals is relevant e.g. in fractionated radiotherapy.
However, since a patient-specific 4D volume data set is required for personalized breathing model building and its acquisition is associated with a high radiation dose with 4D-CT (≥ 20-30 mSv (eff.)), our approach is
the transfer of existing 4D breathing models to new 3D patient data.
For comparison, the average natural background radiation is approximately 2.1 mSv (eff.) per year[http://www.bfs.de/EN/topics/ion/environment/natural-radiation-exposure/natural-radiation-exposure_node.htmlIntercontinental flight max. 0.11 mSv (eff.)].

On the other hand, there is no medical indication to acquire 4D CT data just for training purposes and model building from 4D MR data to be included is unjustifiable for cost reasons.

In this paper, we present a feasibility study with first qualitative results for the transfer of an existing 4D breathing model 
<cit.> to static 3D patient data, in which only a 3D CT covering chest and upper abdomen at maximum inhalation is necessary (approximately 2- 13 mSv (eff.))[https://static.healthcare.siemens.com/siemens_hwem-hwem_ssxa_websites-context-root/wcm/idc/groups/public/@global/@imaging/@ct/documents/download/mdaw/mtm1/ edisp/ct_somatom_definition_as_brochure-00032845.pdfSiemens Somatom Definition AS].



§ RECENT SOLUTION

The existing solution requires a full 4D data set acquisition for each new patient.
In <cit.>, concepts for a 3D VR simulator and efficient patient modeling for the training of different punctures (e.g. liver punctures) have already been presented, see Fig. <ref>. A Geomagic Phantom Premium 1.5 HighForce is used for the manual control and haptic force feedback of virtual surgical instruments.  Nvidia shutter glasses and a stereoscopic display provide the plausible rendering of the simulation scene. This system uses time invariant 3D CT data sets as a basis for the patient model.
In case of manual interaction with the model, tissue deformation due to acting forces of the instruments are represented by a direct visuo-haptic volume rendering method.

New developments of VR simulators
<cit.> allow a time-variant 4D-CT data set to be used in real time for the visualized patient instead of a static 3D CT data set. The respiratory movement can be visualized visuo-haptically as a keyframe model using interpolation or with a flexible linear regression based breathing model as described below.



§ PROPOSED SOLUTION

The new solution requires only a 3D data set acquisition for each new patient. 


 §.§ Modeling of Breathing Motion


Realistic, patient-specific modeling of breathing motion in <cit.>  relies on a 4D CT data set covering one breathing cycle. It consists of N_phases phase 3D images indexed by j. Furthermore, a surrogate signal (for example, acquired by spirometry) to parametrize patient breathing in a low-dimensional manner is necessary.

We use a measured spirometry signal v (t) [ml] and its temporal derivative in a composite surrogate signal: (v (t), v '(t))^T.
This allows to describe different depths of breathing and the distinction between inhalation and exhalation (respiratory hysteresis). We assume linearity between signal and motion extracted from the 4D data. First, we use the 'sliding motion'-preserving approach from <cit.> for N_phases-1 intra-patient inter-phase image registrations to a selected reference phase j_ref:

    φ^pat4D_j=
       φargmin(D_NSSD[I^pat4D_j, I^pat4D_j_ref∘φ]
    +α_S · R_S(φ)),
                 j∈{1,..,j_ref-1,j_ref+1,..,N_phases},

where a distance measure D_NSSD (normalized sum of voxel-wise squared differences <cit.>) and a specialized regularization R_S establishes smooth voxel correspondences except in the pleural cavity where discontinuity is a wished feature <cit.>.
Based on the results, the coefficients a ^ pat4D_1..3 are estimated as vector fields over the positions 𝐱. The personalized breathing model then can be stated as a linear multivariate regression <cit.>:

    φ̂^pat4D(𝐱,t)=
       a^pat4D_1(𝐱)· v(t) +
       a^pat4D_2(𝐱)· v'(t) +
       a^pat4D_3(𝐱),  𝐱∈Ω_pat4D.

Thus, a patient's breathing state can be represented by a previously unseen breathing signal: Any point in time t corresponds to a shifted reference image I^pat4D_j_ref∘φ̂^pat4D(𝐱,t).
Equipped with a real-time capable rendering technique via ray-casting with bent rays (see <cit.> for full technical details), the now time variant model-based animatable CT data I^pat4D_j_ref can be displayed in a new variant of the simulator and used for training. The rays are bent by the breathing motion model and this conveys the impression of an animated patient body, while being very time efficient (by space-leaping and early ray-termination) compared to deforming the full 3D data set for each time point and linear ray-casting afterwards <cit.>. 



 §.§ Transfer of Existing Respiratory Models to new, static Patient Data


Using the method described so far, personalized breathing models can be created, whose flexibility is sufficient to approximate the patients' breathing states, which are not seen in the observation phase of the model formation.

However, the dose-relevant or expensive acquisition of at least one 4D data set has thus far been necessary for each patient.

Therefore, here we pursue the idea to transfer a readily built 4D reference patient breathing model to new static patient data pat3D and to animate it in the VR simulator described in Sec. <ref>.

For this purpose, it is necessary to correct for the anatomical differences between the reference patient with the image data I^pat4D_j_ref and the new patient image data I^pat3D_ref based on a similar breathing phase. This is achieved, for example, by a hold-breath scan (ref) in the maximum inhalation state, which corresponds to a certain phase j_ref in a standardized 4D acquisition protocol. A nonlinear inter-patient registration φ (𝐱): Ω_pat3D→Ω_pat4D with minimization of a relevant image distance D ensures the necessary compensation <cit.>:

    φ^pat3D → pat4D_j_ref=
       φargmin(D_SSD[I^pat3D_ref, I^pat4D_j_ref∘φ]
    +α_D· R_D(φ)),
 
where a distance measure D_SSD (sum of squared voxel-wise differences) and a diffusive non-linear regularization R_D establishes smooth inter-patient voxel correspondences.
On both sides, the breathing phase 3D image of maximum inhalation is selected as the reference phase (ref).
The distance measurement can be selected according to the modality and quality of the image data.
The transformation φ^pat3D → pat4D_j_ref, which is determined in the nonlinear inter-patient registration, can now be used to warp the intra-patient inter-phase deformations of the reference patient φ^pat4D_j as a plausible estimate φ ^ pat3D_j (j ∈{1, …, n }; ∘: right to left):

    φ^pat3D_j =
       (
    φ^pat3D → pat4D_j_ref)^-1∘φ^pat4D_j∘φ^pat3D → pat4D_j_ref.

The approach for estimating the respiratory motion for the new patient can now be applied analogously to the reference patient (see Sec. <ref>). With a efficient regression method <cit.>, the breathing movement of virtual patient models, which are only based on a comparatively low dose of acquired 3D-CT data, can be plausibly approximated:

    φ̂^pat3D(𝐱,t)=   a^pat3D_1(𝐱)· v(t) +
       a^pat3D_2(𝐱)· v'(t) +
       a^pat3D_3(𝐱),  𝐱∈Ω_pat3D.

Optionally, simulated surrogate signals v (t) 
can be used for the 4D animation of 3D CT data. Simple alternatives are to use the surrogate signal of the reference patient or also a (scaled) signal of the new patient pat3D, which can simply be recorded with a spirometric measuring device without new image acquisition.


 








§ EXPERIMENTS AND RESULTS


We performed a qualitative feasibility study, results are animated in the 4D VR training simulator <cit.>.

For the 4D reference patient, a 4D-CT data set of the thorax and upper abdomen with 14 respiratory phases (512^2 × 462 voxel to 1^3 mm) and a spirometry signal v (t) were used (Fig. <ref>).
The new patient is represented only by a static 3D CT data set (512^2 × 318 voxel to 1^3 mm).

All volume image data was reduced to a size of 256^3 voxel due to the limited graphics memory of the GPU used (Nvidia GTX 680 with 3 GB RAM).

According to Eq. <ref> we first perform the intra-patient inter-phase registrations to a chosen reference phase j_ref.

The registrations from Eqs. <ref> and <ref> use weights α_S = 0.1 and α_D=1 for the regularizers R_S and R_D. In both registration processes, the phase with maximum inhalation is used as the reference respiratory phase j_ref and for the training of the breathing model.

The respiratory signal used for model training is shown in Fig. <ref>, gray curve. We show the areas with plausible breathing simulation and use the unscaled respiratory signal of pat3D with larger variance to provoke artifacts (Fig. <ref>, blue curve). The model training according to Eqs. <ref> and <ref> is very efficient using matrix computations.

We use manual expert segmentations of the liver and lungs, available for every phase of the 4D patient, to mainly assess the quality of the inter-patient registration in Eq. <ref>. Via the availabe inter-phase registrations φ^pat4D_j  (Eq. <ref>) to the 4D reference phase, we first warp the phase segmentation masks accordingly. After applying the inter-patient registration to pat3D, we have the segmentation masks of pat4D in the space of the targeted 3D patient. Now for this patient, also a manual expert segmentation is availabe for comparison.
Quantitatively, the DICE coefficients of the transferred segmentation masks (liver, lungs) can be given to classify the quality of the registration chain of the reference respiratory phases (single atlas approach). 
Qualitatively, we present sample images for four time instants and a movie.

The mean DICE coefficients of the single-atlas registration of the liver and lung masks to the new static patient pat3D yield satisfying values of 0.86±0.12 and 0.96±0.09. Note the clearly different scan ranges of the data sets (Fig. <ref>a).
The animation of the relevant structures is shown as an example in Fig. <ref>, using a variable real breathing signal of the target patient pat3D (Fig. <ref>b). In the puncture-relevant liver region, the patient's breathing states are simulated plausibly for the 4D reference patient (Fig. <ref>) and, more importantly, the 3D patient (Figs. <ref>, <ref>), to which the motion model of pat4D was transferred[https://goo.gl/DVVYzwDemo movie, click here].




§ DISCUSSION, OUTLOOK AND CONCLUSION

For interested readers, the basic techiques for 4D breathing motion models have been introduced in <cit.> by our group. However there, the motion model is restricted to the inside of the lungs and by design a mean motion model is built from several 4D patients. The mean motion model is artificial to some degree, more complex and timely to build.
The method described here for the transfer of retrospectively modeled respiratory motion of one 4D reference patient to a new 3D patient data set is less complex and extends to a larger body area. It already allows the plausible animation of realistic respiratory movements in a 4D-VR-training-simulator with visuo-haptic interaction. Of course in the future, we want to build a mean motion model for the whole body section including (lower) lungs and the upper abdomen, too.


In other studies, we found α_D=1 in Eq. <ref> robust (compromise between accuracy and smoothness) for inter-patient registration with large shape variations  <cit.>. In Eq. <ref> for intra-patient inter-phase registration, we use α_S=0.1 to allow more flexibility for more accuracy as the shape variation between two phases of the same patient is much smaller <cit.>.

We achieve qualitatively plausible results for the liver area in this feasibility study. In the upper thorax especially at the rib cage in neighborhood to the dark lungs stronger artifacts can occur (Fig. <ref>). They are due to problems in the inter-patient registration that is a necessary step for the transfer of the motion model. The non-linear deformation sometimes is prone to misaligned ribs.
The same is true for the lower thorax with perforation first of the liver and then diaphragm (Fig. <ref>). Further optimization have to be carried out as artifacts can appear on the high contrast lung edge (diaphragm, ribs) with a small tidal volume.
For liver punctures only, the artifacts of smeared ribs are minor as can be seen in Fig. <ref>.

Summing up, the previous assumption from Sec. <ref> of a dose-relevant or expensive acquisition of a 4D-CT data set for each patient, can be mitigated for liver punctures by the presented transfer of an existing 4D breathing model.

Future work will deal with the better adaptation and simulation of the breathing signal.
Further topics are the optimization of the inter-patient registration and the construction of alternatively selectable mean
4D reference breathing models. As in <cit.>, the authors plan to perform usability studies with medical practitioners.

To conclude, the method allows VR needle puncture training in the hepatic area of breathing virtual patients based on a low-risk and cheap 3D data acquisition for the new patient only.
The requirement of a dose-relevant or expensive acquisition of a 4D CT data set for each new patient can be mitigated by the presented concept.
Future work will include the reduction of artifacts and building mean reference motion models.



§ ACKNOWLEDGEMENT

Support by grant: DFG HA 2355/11-2.
wscg-alpha
0





