Multi-scale Image Fusion Between Pre-operative Clinical CT and X-ray Microtomography of Lung Pathology
Holger R. Roth Kai Nagara Hirohisa Oda Masahiro Oda Tomoshi Sugiyama Shota Nakamura Kensaku Mori













Multi-scale Image Fusion Between Pre-operative Clinical CT 
 and X-ray Microtomography of Lung Pathology
    Holger R. Rotha, Kai Nagarab, Hirohisa Odab, 
 
Masahiro Odab, Tomoshi Sugiyamac, Shota Nakamurac, Kensaku Moria,b


aInformation & Communications, Nagoya University, Japan

bGraduate School of Information Science, Nagoya University, Japan

cNagoya University Graduate School of Medicine, Japan


    December 30, 2023
============================================================================================================================================================================================================================================================================================================







Computational anatomy allows the quantitative analysis of organs in medical images. However, most analysis is constrained to the millimeter scale because of the limited resolution of clinical computed tomography (CT). X-ray microtomography () on the other hand allows imaging of ex-vivo tissues at a resolution of tens of microns. In this work, we use clinical CT to image lung cancer patients before partial pneumonectomy (resection of pathological lung tissue). The resected specimen is prepared for  imaging at a voxel resolution of 50 μm (0.05 mm). This high-resolution image of the lung cancer tissue allows further insides into understanding of tumor growth and categorization. For making full use of this additional information, image fusion (registration) needs to be performed in order to re-align the  image with clinical CT. We developed a multi-scale non-rigid registration approach. After manual initialization using a few landmark points and rigid alignment, several levels of non-rigid registration between down-sampled (in the case of  ) and up-sampled (in the case of clinical CT) representations of the image are performed. Any non-lung tissue is ignored during the computation of the similarity measure used to guide the registration during optimization. We are able to recover the volume differences introduced by the resection and preparation of the lung specimen. The average (± std. dev.) minimum surface distance between  and clinical CT at the resected lung surface is reduced from 3.3 ± 2.9 (range: [0.1, 15.9]) to 2.3 mm ± 2.8 (range: [0.0, 15.3]) mm. This is a significant improvement with p < 0.001 (Wilcoxon Signed Rank Test). The alignment of clinical CT with  will allow further registration with even finer resolutions of  (up to 10 μm resolution) and ultimately with histopathological microscopy images for further macro to micro image fusion that can aid medical image analysis.




image fusion, non-rigid registration, computed tomography (CT), X-ray microtomography ()





§ INTRODUCTION


Clinical computed tomography (CT) is used for the diagnostic imaging of the living human (in-vivo imaging). As a result, most computational analysis is constrained to the millimeter scale because of the limited resolution of clinical CT. At this millimeter scale, pulmonary blood vessels and lung lobes can be observed. However, finer detailed anatomy is not observable. X-ray microtomograph () on the other hand allows imaging of ex-vivo tissues at a resolution of tens of microns. At this μm-scale, the alveoli and bronchiole regions can be clearly observed <cit.>.

The prospective study of pre-operative imaging together with the high-resolution image analysis of resected tissue after surgery may provide us with unique opportunities to verify and potentially improve imaging protocols for cancer diagnostics <cit.>. 

In this work, we use clinical CT to image lung cancer patients before partial pneumonectomy (resection of pathological lung tissue). The resected specimen is then prepared for  imaging allowing the imaging of μm-scale anatomy. This study investigates the use of non-rigid intensity-based registration in order to establish a scale-seamless registration between clinical CT and , with the ultimate aim of allowing a seamless navigation between anatomical scales inside the human body <cit.>. Related work is the registration of in-vivo and ex-vivo MRI of surgically resected specimens by <cit.>.




§ METHOD


The resected specimen is prepared for  imaging at a voxel resolution of 50 μm (0.05 mm). This high-resolution image of the lung cancer tissue allows further insides into understanding of tumor growth and categorization. For making full use of this additional information, image fusion (registration) needs to be performed in order to re-align the  image with clinical CT. We developed a multi-scale non-rigid registration approach. 



 §.§ Establishing scale-seamless registration between  and clinical CT


We use a non-rigid registration method to align the  specimen with the pre-operative clinical CT. This method is driven by the intensity similarity between the images. Any non-lung tissue is ignored during computation of the similarity measure 𝒮 used to guide the registration during optimization. This is achieved via simple thresholding and morphological operations to extract the lung region in clinical CT and  as pre-processing step. 

A coarse-to-fine approach is proposed in order to capture first the largest deformations and then the smaller differences between both images. This is achieved with a four-level multi-resolution pyramid as illustrated in Fig. <ref>. In order to compensate for the large differences in resolution between the  (in the case of  ) and clinical CT images, we use a down-sample and up-sample (in the case of clinical CT) pyramid and perform the registration optimizations at each level in a coarse-to-fine fashion.



A good initialization for the registration algorithm is important. We use a few (3∼5) manual point in order to provide affine alignment of the specimen with the pre-operative CT image. After affine initialization, we establish non-rigid alignment using the B-spline registration method, also known as 3D free form deformation based registration of <cit.> with the implementation provided by <cit.>.

A 3D cubic B-Splines deformation model uses a lattice of control points {ϕ⃗}. The spacing between each control point is uniform and denoted as δ_x, δ_y, and δ_z along the x-, y-, and z-axis respectively. For each voxel x⃗ in the domain Ω of the target image. the deformation 𝐓(x⃗) can be computed as:

    𝐓(x⃗) = ∑_i,j,kβ^3(x/δ_x - i) ×β^3(y/δ_y - j) ×β^3(y/δ_z - k) ×ϕ⃗_ijk,
 
where β^3 represents the cubic B-Spline function.

The images are aligned by finding the transformation which maximizes the following objective function:

    𝒪(I_p,I_s(𝐓);{ϕ⃗}) = 	(1 - α - β - γ)×𝒮
     
    	- α×𝒞_smooth(𝐓) - β×𝒞_volpres(𝐓) -γ×𝒞_inconsistency(𝐓)

which combines a similarity measure, 𝒮, and three penalty constraint terms, 𝒞_smooth, 𝒞_volpres, and 𝒞_inconsistency. Each term is weighted against each other by user-defined weights α, β, and γ. 






The similarity measure used between the reference (R) and floating image (F) is normalized mutual information (NMI): 

    𝒮≡𝐍𝐌𝐈 =    H(R)  + H(F(𝐓))  /   H(R, F(𝐓))
 
where H(R) and H(F(𝐓)) the two marginal entropies, and H(R, F(𝐓)) is the joint entropy. Its computation requires a joint histogram which is filled by using a Parzen Window (PW) approach <cit.>.

The three constraint terms are used to encourage realistic deformations. The bending energy describes the smoothness of the deformation and is defined as:

    𝒞_smooth =
    	 1/N∑_x⃗∀Ω(
    	 |∂^2𝐓(x⃗)/∂x^2|^2 +
            |∂^2𝐓(x⃗)/∂y^2|^2 + 
            |∂^2𝐓(x⃗)/∂z^2|^2 
    
    + 2×[|∂^2𝐓(x⃗)/∂xy|^2 +
                    |∂^2𝐓(x⃗)/∂yz|^2 +
                    |∂^2𝐓(x⃗)/∂xz|^2 
         ])
    .
 
The volume-preserving penalty term discourages large expansions/contractions, and is defined as:

    𝒞_volpres = 1/N∑_x⃗∀Ω[
                              log(
                                              (
                                                                Jac(𝐓(x⃗))
                                                       )
                                                       )
                 ]^2

In addition we prevent the occurrence of folding in the transformation using a folding correction scheme <cit.>. For each transformed voxel that would cause a negative Jacobian determinant, its influence on its neighborhood control points is computed. The control point positions is then changed until the determinant value is positive. 

In an ideal case, the transformations from F to R (forward) and R to F (backward) are the inverse of each other, e.g.  =^-1 and =^-1 <cit.>. Hence, we include a penalty term that encourages inverse consistency of both transformations. We follow the approach of <cit.> using compositions of  and  and add 

    𝒞_inconsistency = 
    		∑_x⃗∀Ω((x⃗))^2 
    
    	+ ∑_x⃗∀Ω((x⃗))^2

The following parameters were found empirically by visual examination of the registration results. We use a four-level multi-resolution pyramid with a maximum of 500 iterations per level. Both the image and B-spline control point grid resolutions are doubled with increasing resolution levels. The final control point spacing between voxels is 5. The objective function weights are set to α = 10^-4, β = 10^-12, and γ = 0.1. These parameters were found to recover the majority of the deformation between the two images, while preventing unrealistic deformations from occurring. We used the open-source software[<http://sourceforge.net/projects/niftyreg>] by <cit.> for this study.




§ RESULTS


We collected pre-operative clinical CTs from two patients before partial pneumonectomy. The clinically acquired CT images have dimensions of [512, 512, 435∼554], and voxel spacings of [0.625, 0.625, 0.6] mm. The  images used for registration had dimensions of [278∼512, 278∼512, 346∼538] with  isotropic voxel spacings of 0.111∼0.127 mm. Note that we downsampled the original  images (circa 50μm resolution) by a factor of two for the registration experiments. The tube voltage for  was 90kVp, and tube current was 110 μA.

3∼5 corresponding points where chosen by an expert clinician (SN) in order to provide an initial affine alignment of the specimen with the pre-operative CT image. We then crop the clinical CT image to the extent of the aligned  image for subsequent non-rigid registration.

Figure <ref> shows the alignment of  lung specimen with pre-operative clinical CT before and after non-rigid registration for one case. A qualitatively better alignment of  with the target clinical CT regions can be observed. 



In order to quantitatively evaluate the registration performance, we measure the average minimum surface distance (AvgDist) between the lung surface extracted from clinical CT and  before and after non-rigid registration. Figure <ref> shows the extracted surfaces used for measurement before and after non-rigid alignment. The AvgDist measures are given in Table <ref>. A reduction from 3.3 ± 2.9 (range: [0.1, 15.9]) to 2.3 mm ± 2.8 (range: [0.0, 15.3]) mm on average can be observed. This is a significant improvement with p < 0.001 (Wilcoxon signed rank test). 








§ CONCLUSIONS


We presented a method for non-rigid alignment between images of pre-operative clinical CT and x-ray microtomography () of lung pathology. After manual initialization using a few landmark points and affine alignment, several levels of non-rigid registration between down-sampled (in the case of  ) and up-sampled (in the case of clinical CT) representations of the image are performed. This allows us to recover the volume differences introduced by the resection and preparation of the lung specimen. The proposed multi-scale image fusion approach will allow further registration with even finer resolutions of  (up to 10 μm resolution) <cit.> and ultimately with histopathological microscopy images for further macro to micro image fusion that can aid medical image analysis.






§ ACKNOWLEDGEMENTS

This paper was supported by MEXT KAKENHI (26108006 and 15H01116).






ieeetr  


