We propose to tackle the problem of multiview 2D 3D rigid registration for intervention via a Point-Of-Interest Network for Tracking and Triangulation (POINT^2). POINT^2 learns to establish 2D point-to-point correspondences between the pre- and intra-intervention images by tracking a set of random POIs. The 3D pose of the pre-intervention volume is then estimated through a triangulation layer. In POINT^2, the unified framework of the POI tracker and the triangulation layer enables learning informative 2D features and estimating 3D pose jointly. In contrast to existing approaches, POINT^2 only requires a single forward-pass to achieve a reliable 2D 3D registration. As the POI tracker is shift-invariant, POINT^2 is more robust to the initial pose of the 3D pre-intervention image. Extensive experiments on a large-scale clinical cone-beam CT (CBCT) dataset show that the proposed POINT^2 method outperforms the existing learning-based method in terms of accuracy, robustness and running time. Furthermore, when used as an initial pose estimator, our method also improves the robustness and speed of the state-of-the-art optimization-based approaches by ten folds.
@cite_18: Generation of digitally reconstructed radiographs (DRRs) is computationally expensive and is typically the rate-limiting step in the execution time of intensity-based two-dimensional to three-dimensional (2D-3D) registration algorithms. We address this computational issue by extending the technique of light field rendering from the computer graphics community. The extension of light fields, which we call attenuation fields (AFs), allows most of the DRR computation to be performed in a preprocessing step; after this precomputation step, DRRs can be generated substantially faster than with conventional ray casting. We derive expressions for the physical sizes of the two planes of an AF necessary to generate DRRs for a given X-ray camera geometry and all possible object motion within a specified range. Because an AF is a ray-based data structure, it is substantially more memory efficient than a huge table of precomputed DRRs because it eliminates the redundancy of replicated rays. Nonetheless, an AF can require substantial memory, which we address by compressing it using vector quantization. We compare DRRs generated using AFs (AF-DRRs) to those generated using ray casting (RC-DRRs) for a typical C-arm geometry and computed tomography images of several anatomic regions. They are quantitatively very similar: the median peak signal-to-noise ratio of AF-DRRs versus RC-DRRs is greater than 43 dB in all cases. We perform intensity-based 2D-3D registration using AF-DRRs and RC-DRRs and evaluate registration accuracy using gold-standard clinical spine image data from four patients. The registration accuracy and robustness of the two methods is virtually identical whereas the execution speed using AF-DRRs is an order of magnitude faster.
@cite_4: The registration of pre-operative volumetric datasets to intra-operative two-dimensional images provides an improved way of verifying patient position and medical instrument location. In applications from orthopedics to neurosurgery, it has great value in maintaining up-to-date information about changes due to intervention. We propose a mutual information-based registration algorithm which establishes the proper alignment via a stochastic gradient ascent strategy. Our main contribution lies in estimating probability density measures of image intensities with a sparse histogramming method which could lead to potential speedup over existing registration procedures and deriving the gradient estimates required by the maximization procedure. Experimental results are presented on fluoroscopy and CT datasets of a real skull, and on a CT-derived dataset of a real skull, a plastic skull and a plastic lumbar spine segment.
@cite_1: We present a framework for robustly estimating registration between a 3D volume image and a 2D projection image and evaluate its precision and robustness in spine interventions for vertebral localization in the presence of anatomical deformation. The framework employs a normalized gradient information similarity metric and multi-start covariance matrix adaptation evolution strategy optimization with local-restarts, which provided improved robustness against deformation and content mismatch. The parallelized implementation allowed orders-of-magnitude acceleration in computation time and improved the robustness of registration via multi-start global optimization. Experiments involved a cadaver specimen and two CT datasets (supine and prone) and 36 C-arm fluoroscopy images acquired with the specimen in four positions (supine, prone, supine with lordosis, prone with kyphosis), three regions (thoracic, abdominal, and lumbar), and three levels of geometric magnification (1.7, 2.0, 2.4). Registration accuracy was evaluated in terms of projection distance error (PDE) between the estimated and true target points in the projection image, including 14 400 random trials (200 trials on the 72 registration scenarios) with initialization error up to ±200 mm and ±10°. The resulting median PDE was better than 0.1 mm in all cases, depending somewhat on the resolution of input CT and fluoroscopy images. The cadaver experiments illustrated the tradeoff between robustness and computation time, yielding a success rate of 99.993 in vertebral labeling (with 'success' defined as PDE <5 mm) using 1,718 664 ± 96 582 function evaluations computed in 54.0 ± 3.5 s on a mid-range GPU (nVidia, GeForce GTX690). Parameters yielding a faster search (e.g., fewer multi-starts) reduced robustness under conditions of large deformation and poor initialization (99.535 success for the same data registered in 13.1 s), but given good initialization (e.g., ±5 mm, assuming a robust initial run) the same registration could be solved with 99.993 success in 6.3 s. The ability to register CT to fluoroscopy in a manner robust to patient deformation could be valuable in applications such as radiation therapy, interventional radiology, and an assistant to target localization (e.g., vertebral labeling) in image-guided spine surgery.
@cite_6: This paper presents work towards a system for intra-operative registration of 3D CT data to 2D X-ray radiographs. The registration procedure involves iterative comparison of Digitally Reconstructed Radiographs (DRRs) with X-ray images acquired during surgery. A new data structure called a Transgraph permits rapid generation of DRRS, and greatly speeds up the registration process. The underlying data structures are described, and the registration algorithm is evaluated for application to an existing image guided radiosurgery system.
@cite_0: Recent years have seen exciting advances in Computer Assisted Surgery (CAS). CAS systems are currently in use which provide data to the surgeon, provide passive feedback and motion constraint, and even automate parts of the surgery by manipulating cutters and endoscopic cameras. For most of these systems, accurate registration between the patient's anatomy and the CAS system is crucial: if the position of the surgical target is not known with sufficient accuracy, therapies cannot be applied precisely, and treatment efficacy falls. This thesis presents a system for recovering the position and orientation of the target anatomy in 3D space based on iterative comparison of 2D planar radiographs with preoperative CT data. More specifically, this system uses X-ray images acquired at the time of treatment, and iteratively compares them with synthetic images, known as Digitally Reconstructed Radiographs (DRRs), in order to estimate the position and orientation of the target anatomy. An intermediate data representation called a Transgraph is presented. The Transgraph is similar to the Lumigraph, or Light Field, and extends the computer graphics field called image-based rendering to transmission imaging. This representation speeds up computation of DRRs by over an order of magnitude compared to ray-casting techniques, without the use of special graphics hardware. A hardware based volume rendering technique is also presented. This approach is based on new texture mapping techniques which enable DRR generation using off the shelf consumer grade computer graphics hardware. These techniques permit computation of full resolution (512 x 512) DRRs based on 256 x 256 x 256 CT data in roughly 70 ms. The registration system is evaluated for application to frameless stereotactic radiosurgery, and phantom studies are presented demonstrating accuracy comparable to state of the art immobilization systems. Additional phantom studies are presented in which the registration system is used to measure implant orientation following total hip replacement surgery, improving on current practice by a factor of two.
@cite_2: Current noninvasive or minimally invasive methods for evaluating in vivo knee kinematics are inadequate for accurate determination of dynamic joint function due to limited accuracy and or insufficient sampling rates. A three-dimensional (3-D) model-based method is presented to estimate skelet al motion of the knee from high-speed sequences of biplane radiographs. The method implicitly assumes that geometrical features cannot be detected reliably and an exact segmentation of bone edges is not always feasible. An existing biplane radiograph system was simulated as two separate single-plane radiograph systems. Position and orientation of the underlying bone was determined for each single-plane view by generating projections through a 3-D volumetric model (from computed tomography), and producing an image (digitally reconstructed radiograph) similar (based on texture information and rough edges of bone) to the two-dimensional radiographs. The absolute 3-D pose was determined using known imaging geometry of the biplane radiograph system and a 3-D line intersection method. Results were compared to data of known accuracy, obtained from a previously established bone-implanted marker method. Difference of controlled in vitro tests was on the order of 0.5 mm for translation and 1.4 spl deg for rotation. A biplane radiograph sequence of a canine hindlimb during treadmill walking was used for in vivo testing, with differences on the order of 0.8 mm for translation and 2.5 spl deg for rotation.
@cite_5: X-rayimages are often used to guide minimally invasive procedures in interventional radiology. The use of a preoperatively obtained 3D volume can enhance the visualization needed for guiding catheters and other surgical devices. However, for intraoperative usefulness, the 3D dataset needs to be registered to the 2D x-rayimages of the patient. We investigated the effect of targeting subvolumes of interest in the 3D datasets and registering the projections with C-arm x-rayimages. We developed an intensity-based 2D 3D rigid-body registration using a Monte Carlo-based hybrid algorithm as the optimizer, using a single view for registration. Pattern intensity (PI) and mutual information (MI) were two metrics tested. We used normalization of the rays to address the problems due to truncation in 3D necessary for targeting. We tested the algorithm on a C-arm x-rayimage of a pig's head and a 3D dataset reconstructed from multiple views of the C-arm. PI and MI were comparable in performance. For two subvolumes starting with a set of initial poses from + ∕ − 15 mm in x , from + ∕ − 3 mm (random), in y and z and + ∕ − 4 deg in the three angles, the robustness was 94 for PI and 91 for MI, with accuracy of 2.4 mm (PI) and 2.6 mm (MI), using the hybrid algorithm. The hybrid optimizer, when compared with a standard Powell’s direction set method, increased the robustness from 59 (Powell) to 94 (hybrid). Another set of 50 random initial conditions from [ + ∕ − 20 ] mm in x , y , z and [ + ∕ − 10 ] deg in the three angles, yielded robustness of 84 (hybrid) versus 38 (Powell) using PI as metric, with accuracies 2.1 mm (hybrid) versus 2.0 mm (Powell).
@cite_13: Intraoperative patient registration may significantly affect the outcome of image-guided surgery (IGS). Image-based registration approaches have several advantages over the currently dominant point-based direct contact methods and are used in some industry solutions in image-guided radiation therapy with fixed X-ray gantries. However, technical challenges including geometric calibration and computational cost have precluded their use with mobile C-arms for IGS. We propose a 2D 3D registration framework for intraoperative patient registration using a conventional mobile X-ray imager combining fiducial-based C-arm tracking and graphics processing unit (GPU)-acceleration. The two-stage framework 1) acquires X-ray images and estimates relative pose between the images using a custom-made in-image fiducial, and 2) estimates the patient pose using intensity-based 2D 3D registration. Experimental validations using a publicly available gold standard dataset, a plastic bone phantom and cadaveric specimens have been conducted. The mean target registration error (mTRE) was 0.34±0.04 mm (success rate: 100 , registration time: 14.2 s) for the phantom with two images 90° apart, and 0.99±0.41 mm (81 , 16.3 s) for the cadaveric specimen with images 58.5° apart. The experimental results showed the feasibility of the proposed registration framework as a practical alternative for IGS routines.
@cite_25: The efficacy of radiation therapy treatment depends on the patient setup accuracy at each daily fraction. A significant problem is reproducing the patient position during treatment planning for every fraction of the treatment process. We propose and evaluate an intensity based automatic registration method using multiple portal images and the pre-treatment CT volume. We perform both geometric and radiometric calibrations to generate high quality digitally reconstructed radiographs (DRRs) that can be compared against portal images acquired right before treatment dose delivery. We use a graphics processing unit (GPU) to generate the DRRs in order to gain computational efficiency. We also perform a comparative study on various similarity measures and optimization procedures. Simple similarity measure such as local normalized correlation (LNC) performs best as long as the radiometric calibration is carefully done. Using the proposed method, we achieved better than 1 mm average error in repositioning accuracy for a series of phantom studies using two open field (i.e., 41 cm 2 ) portal images with 90� vergence angle.
@cite_20: Reproducible positioning of the patient during fractionated external beam radiation therapy is imperative to ensure that the delivered dose distribution matches the planned one. In this paper, we expand on a 2D-3D image registration method to verify a patient’s setup in three dimensions (rotations and translations) using orthogonal portal images and megavoltage digitally reconstructedradiographs (MDRRs) derived from CT data. The accuracy of 2D-3D registration was improved by employing additional image preprocessing steps and a parabolic fit to interpolate the parameter space of the cost function utilized for registration. Using a humanoid phantom, precision for registration of three-dimensional translations was found to be better than 0.5 mm (1 s.d.) for any axis when no rotations were present. Three-dimensional rotations about any axis were registered with a precision of better than 0.2 ° (1 s.d.) when no translations were present. Combined rotations and translations of up to 4 ° and 15 mm were registered with 0.4 ° and 0.7 mm accuracy for each axis. The influence of setup translations on registration of rotations and vice versa was also investigated and mostly agrees with a simple geometric model. Additionally, the dependence of registration accuracy on three cost functions, angular spacing between MDRRs, pixel size, and field-of-view, was examined. Best results were achieved by mutual information using 0.5 ° angular spacing and a 10 × 10 cm 2 field-of-view with 140 × 140 pixels . Approximating patient motion as rigid transformation, the registration method is applied to two treatment plans and the patients’ setup errors are determined. Their magnitude was found to be ⩽ 6.1 mm and ⩽ 2.7 ° for any axis in all of the six fractions measured for each treatment plan.
Optimization-Based Approaches. Optimization-based approaches usually suffer from high computational cost and is sensitive to the initial estimate. To reduce the computational cost, many works have been proposed to improve the efficiency in hardware-level @cite_0 @cite_25 @cite_13 or software-level @cite_18 @cite_4 @cite_6 . Although these works have successfully reduced the DRR generation time to a reasonable range, the overall registration time is still non-negligible @cite_18 @cite_13 and the registration accuracy might be compromised for faster speed @cite_4 @cite_18 . For better initial pose estimation, many attempts have been made by either sampling better initial position @cite_20 @cite_5 , using multistart strategies @cite_2 @cite_1 , or a carefully designed objective function that is less sensitive to the initial position selection @cite_13 . However, these methods usually achieve a more robust registration at the cost of longer running time as more locations, and the corresponding DRRs need to be sampled and generated, respectively, to avoid being trapped in the local extrema.