i3PosNet: Instrument Pose Estimation from X-Ray in temporal bone surgery
David Kügler Jannik Sehring Andrei Stefanov Igor Stenin Julia Kristin Thomas Klenzner Jörg Schipper Anirban Mukhopadhyay



i3PosNet: Instrument Pose Estimation from X-Ray in temporal bone surgery












D. Kügler German Center for Neuro-degenerative Diseases, Bonn, Germany

	                work done at Department of Computer Science, TU Darmstadt, Darmstadt, Germany

	david.kuegler@dzne.de           
	                J. Sehring A. Mukhopadhyay Department of Computer Science, Technischer Universität Darmstadt, Darmstadt, Germany

	
	
	
	           I. Stenin J. Kristin T. Klenzner J. Schipper ENT Clinic, University Düsseldorf, Germany





    David Kügler Jannik Sehring Andrei Stefanov Igor Stenin Julia Kristin Thomas Klenzner Jörg Schipper Anirban Mukhopadhyay

    Received: date / Accepted: date
=============================================================================================================================






 Purpose:
Accurate estimation of the position and orientation (pose) of surgical instruments is crucial for delicate minimally invasive temporal bone surgery.
Current techniques lack in accuracy and/or line-of-sight constraints (conventional tracking systems) or expose the patient to prohibitive ionizing radiation (intra-operative CT).
A possible solution is to capture the instrument with a c-arm at irregular intervals and recover the pose from the image.

Methods:
i3PosNet infers the position and orientation of instruments from images using a pose estimation network.
Said framework considers localized patches and outputs pseudo-landmarks.
The pose is reconstructed from pseudo-landmarks by geometric considerations.


Results:
We show i3PosNet reaches errors less than 0.05.
It outperforms conventional image registration-based approaches reducing average and maximum errors by at least two thirds.
i3PosNet trained on synthetic images generalizes to real x-rays without any further adaptation. 

Conclusion:
The translation of Deep Learning based methods to surgical applications is difficult, because large representative datasets for training and testing are not available.
This work empirically shows sub-millimeter pose estimation trained solely based on synthetic training data.









§ INTRODUCTION





Current clinical practice in temporal bone surgery for cochlear implantation (CI) and 
vestibular schwannoma removal is still centered on a conventional and open operation setting. 
One fundamental challenge in moving to less traumatic minimally-invasive procedures 
is to satisfy the required navigation accuracy. 

To ensure delicate risk structures such as the facial nerve and chorda tympani are not damaged by surgical tools, the clinical navigation accuracy must exceed 0.5 <cit.>.
Recent efforts have used force-feedback <cit.>, optical tracking systems (OTSs) <cit.> and neuro-monitoring for CI <cit.>, but each one of these strategies have drawbacks in the minimally-invasive setting.
For example OTSs require line-of-sight and a registration between patient and tracking system.
Electromagnetic tracking <cit.>, force-feedback or neuro-monitoring on the other hand, feature limited accuracy. 

None of these methods can be used to navigate next-generation flexible instruments that follow non-linear paths <cit.>.
X-ray imaging, on the other hand, is precise and not constrained by line-of-sight.

However, similar to OTS, fiducials used for patient registration significantly impact tracking accuracy of surgical instruments.
The small size of fiducials, 
low contrast to anatomy alongside high anatomy-to-anatomy contrast 
and rotational symmetry characterize challenges specific to pose estimation of surgical tools in temporal bone surgery. 



Unlike previous methods, Deep Learning allows instrument pose estimation to break into submillimeter accuracy at acceptable execution times <cit.>.

Previous non-Deep Learning pipelines based on 2D/3D registration <cit.> and template matching <cit.> achieve submillimeter accuracy for simple geometries.
However, such techniques do not scale, do not generalize to more complex instruments and/or require full-head pre-operative CT scans.
In addition, these solutions are usually customized to a specific application, e.g. screws in pedicle screw placement <cit.>, fiducials <cit.> or guide-wires and instruments <cit.>.
The recent shift to Deep Neural Networks offers better accuracy at near real-time execution speed for implants and instruments <cit.>.
However, no such solution has been proposed for temporal bone surgery.




We propose i3PosNet, a Deep-Learning-powered Iterative Image-guided Instrument Pose Estimation method to provide high-precision estimation of poses.


We focus on pose estimation from x-ray images showing a fiducial (screw) placed on the skull close to the temporal bone, because screws are commonly used as fiducials in robotic CI <cit.>. 
For optimum performance, i3PosNet implements a modular pipeline consisting of 1) region of interest normalization, 
2) 2D pose estimation and 
3) 3D pose reconstruction to determine the pose.
To this effect, we patchify and normalize a small region around the expected fiducial position.
We design a convolutional neural network (CNN) to predict six pseudo-landmarks on two axes with subpixel accuracy.
A geometric model reconstructs 3D poses from their landmark coordinates.
The division of the pipeline into three modular steps reduces complexity, increases performance and significantly boosts the angle estimation performance of i3PosNet.

As part of this work we publish three Datasets in addition to the source code[Find additional information at <https://i3posnet.david-kuegler.de/>]. 
Based on these three Datasets, we show i3PosNet i) generalizes to real x-ray while only training on synthetic images, 
ii) generalizes to two surgical instruments in pilot evaluations, and 
iii) outperforms state-of-the-art registration methods as well as the end-to-end variation of i3PosNet.
As no public datasets with ground truth poses 
are available for training and evaluation of novel pose estimation methods, we believe these new Datasets will foster further developments in this area.


Dataset A consists of synthetic radiographs with a medical screw for training and evaluation.

In Dataset B the screw is replaced by either a drill or a robot.
Dataset C features real images of micro-screws placed on a head phantom. 
Images are acquired with a c-arm and manually annotated in 2D.







§ RELATED WORK



Pose estimation using radiographs of instruments in the temporal bone has received little scientific attention.
Most published research uses other tracking paradigms, most notably optical tracking <cit.>.
While some Deep-Learning based approaches directly <cit.> or indirectly <cit.> extract instrument poses from x-ray images, neither address temporal bone surgery.
In this Section, we give a brief overview of instrument tracking for temporal bone surgery.

Robotic solutions for minimally invasive cochlear implantation demand accurate tracking solutions.
The small size of critical risk structures limits the acceptable navigation accuracy to which the tracking accuracy is a significant contributor <cit.>.
In response, robotic cochlear implantation solutions rely on high precision optical <cit.> and electromagentic tracking <cit.>.
However, electromagnetic tracking is not suitable to guide a robot because of accuracy and metal-distortion constraints <cit.>.
Optical tracking cannot directly measure the instrument pose due to the occluded line-of-sight.
Instead, the tool base is tracked adding the base-to-tip registration as a significant error source <cit.>.
Adding additional, redundant approaches based on
pose reconstruction from bone density and drill forces <cit.> as well as a neuro-monitoring-based fail-safe 
<cit.>,
Rathgeb <cit.> report a navigation accuracy of 0.22 ±0.1 at the critical facial nerve / 0.11 ±0.08 at the target.

Earlier work on x-ray-imaging-based instrument pose estimation centers on traditional registration <cit.>, segmentation <cit.> and template-matching <cit.>.
These methods are employed for various applications from pedicle screw placement to implant and guide-wire localization.
Temporal bone surgery has only been addressed by Kügler <cit.>.
Recent work introduces Deep Learning methods for instrument pose estimation using segmentation as intermediate representations <cit.> and directly estimating the 3D pose <cit.>.
While Miao <cit.> employ 974 specialized neural networks, Bui 's work <cit.> extends the PoseNet architecture, but does not feature anatomy.
Bier <cit.> proposed an anatomical landmark localization method.
However instruments sizes are significantly smaller in temporal bone surgery impacting x-ray attenuation and therefore image contrast.
For the pose estimation of surgical instruments on endoscopic images <cit.> Deep Learning is prevalent technique, but sub-pixel accuracy is not achieved – in part because the manual ground truth annotation do not allow it.

No Deep Learning-based pose estimation method addresses temporal bone surgery or its challenges such as very small instruments and low contrast.


§ DATASETS



This paper introduces three new Datasets: 
two synthetic Digitally Rendered Radiograph (DRR) Datasets (Dataset A for a screw and Dataset B for two surgical instruments), and 
a real x-ray Dataset (Dataset C for manually labeled screws).
All datasets include annotations for the pose with a unified file format.



 §.§ Dataset A: Synthetic



This dataset shows a CT scan of a head and a screw rendered as x-ray. 
We balance it w.r.t. anatomical variation and projection geometry by implementing a statistical sampling method, which we describe here.
In consequence, the dataset is ideal for method development and the exploration of design choices. 


Anatomy and Fiducial:
To account for the variation of patient-specific anatomy, we consider three different conserved human cadaver heads captured by a SIEMENS SOMATOM Definition AS+.
The slices of the transverse plane are centered around the otobasis and include the full cross-section of the skull.
A small medical screw is virtually implanted near the temporal bone similar to the use for tool registration of the pre-operative CT in robotic cochlear surgery <cit.>.
The screw geometry is defined by a CAD mesh provided by the screw manufacturer (c.f. Dataset C).
Its bounding box diagonal is 6.5.




Method for Radiograph Generation:

Our DRR Generation pipeline is fully parameterizable and tailored to the 
surgical pose estimation use-case.
We use the Insight Segmentation and Reconstruction Toolkit 
and 
the Registration Toolkit 
to modify and project the CT anatomy and the fiducial into 2D images.
The pipeline generates projections and corresponding 
ground truth poses from a CT anatomy, a mesh of the fiducial
and a parameter definition, where most parameters can be defined statistically. 

Since the CT data only includes limited sagittal height, we require all projection rays that pass through an approx. 5 sphere around the fiducial to be uncorrupted by regions of missing CT data.
For this, missing data is delineated by polygons.
We export the pose  (eqn:position) of the instrument for later use in training and evaluation.





Parameters of Generation:
The generation process is dependent on the projection geometry and the relative position and orientation of the fiducial w.r.t. the anatomy.
In specific, we manually choose 10 realistic fiducial poses w.r.t. the anatomy per subject and side.
To increase variety, we add noise to these poses (position  and orientation ): 
The noise magnitude for testing is lower than for training to preserve realism and increase generalization.
The projection geometry on the other hand describes the configuration of the mesh w.r.t. x-ray source and detector.
These parameters are encoded in the  Source-Object-Distance, the displacement orthogonal to the projection direction and the rotations around the object.
We derive these from the specification of the c-arm, which we also use for the acquisition of real x-ray images.

In total, the dataset contains 18,000 images across three anatomies. Three thousand of them are generated with the less noise setting for testing.


 §.§ Dataset B: Surgical Tools


To show generalization to other instruments, we introduce a second synthetic dataset.
This dataset differs from the first two-fold: instead of a screw we use a medical drill or a prototype robot (see fig:projection); these are placed at realistic positions inside of the temporal bone instead of on the bone surface.
This dataset includes the same number of images per instrument as Dataset A.

Drill:
Despite the drill's length (drill diameter 3), for the estimation of the tip's pose only the tip is considered to limit the influence of drill bending. 

Prototype Robot:
The non-rigid drilling robot consists of a spherical drilling head and two cylinders connected by a flexible joint. 
By flexing and expanding the joint in coordination with cushions on the cylinders, the drill-head creates non-linear access paths. 
With a bounding box diagonal of the robot up to the joint of 13.15, its dimensions are in line with typical MIS and temporal bone surgery applications. 










 §.§ Dataset C: Real x-rays



For Dataset C we acquire real x-ray images and annotations.
The dataset includes 540 images with diverse acquisition angles, distances and positions.

Preparation: 
The experimental setup is based on a realistic x-ray head phantom featuring a human skull embedded in tissue equivalent material.
To not damage the head phantom for the placement of the fiducial (screw), we attach the medical titanium micro-screw with modeling clay as the best non-destructive alternative.
The setup is then placed on the carbon fiber table and supported by x-ray-translucent foam blocks.


Image Acquisition:
We capture multiple x-ray images with a Ziehm c-arm certified for clinical use before repositioning the screw on the skull.
These images are collected from multiple directions per placement.



Manual Annotation:
In a custom annotation tool, we recreate the projection geometry from c-arm specifications replacing x-ray source and detector with camera and x-ray respectively.
The screw is rendered as an outline and interactively translated and rotated to match the x-ray.
We remove images, where the screw is not visible to the annotater (approx. 1% of images).
Finally, the projected screw position is calculated to fit the projection geometry.


§ METHODS




Our approach breaks the task of surgical pose estimation down into three steps: reduction of image variety (region of interest appearance normalization), a convolutional neural network for information extraction and finally pose reconstruction from pseudo-landmarks.
We describe a formalized problem definition in addition to these individual three steps in this section.



 §.§ Problem Definition


Surgical Pose estimation is the task of extracting the pose from a x-ray image.
We define the pose  to be the set of coordinates, , projection angle and distance to the x-ray source (depth). 
It is defined w.r.t. the detector and the projection geometry (see fig:posedefinition).

    =  (  x , y , ,  ,  )^T

The   indicates the angle between the instrument's main rotational axis projected onto the image plane and the horizontal axis of the image.
The projection angle  quantifies the tilt of the instrument w.r.t. the detector plane.
The depth  represents the distance on the projection normal from the source (focal point) to the instrument (c.f. Source-Object-Distance).

We assume an initial pose with accuracy Δ x_initial≤2.5 and Δα_initial≤30 is available. 
The initial pose can be manually identified, adopted from an independent low-precision tracking system, previous time points or determined from the image by a localization U-Net <cit.>.

Given a new predicted pose, the initial guess is updated and the steps described in Sections <ref> to <ref> iteratively repeated. 



 §.§ Appearance Normalization


Since megapixel x-ray images (10241024 pixels) are dominated by regions unimportant to the pose estimation, 
the direct extraction of subpixel accuracy from megapixel images oversaturates the deep learning strategy with data. 
It is therefore crucial to reduce the size of the image and the image variety based on prior knowledge.

To reduce image variety, the appearance normalization creates an image patch that is rotated, translated and cut to the estimated object position.
Additionally, we normalize the intensity in the patch of 9248 pixels.
Based on this normalization step and given perfect knowledge of the pose, the object will always appear similar w.r.t. the in-plane pose components (position and ).
We define this as standard pose – the object positioned at a central location and oriented in the direction of the x-axis of the patch.


Pseudo-landmarks are generated from the pose annotation.
Their geometric placement w.r.t. the fiducial is described by the pair of x- and y-coordinates (x_i, y_i).
The independence from the fiducial's appearance motivates the term “pseudo-landmark”.
They are placed 15 apart in a cross-shape centered on the instrument (see fig:landmarks).
Two normalized support vectors define the legs of the cross: instruments rotational axis (x-direction, 3+1 landmarks), and its cross product with the projection direction (y-direction, 2+1 landmarks).
eqn:keypoint2image formalizes the transformation of the pose to point coordinates in the image plane dependent on the projection geometry (c_d2p = Δ_ds / d_SDD with  Source-Detector-Distance d_SDD and Detector-Pixel-Spacing Δ_ds).
{(x^LP_i , y^LP_i)}_i=1^6 describe the local placement (LP) of pseudo-landmarks w.r.t. the support vectors.
Finally, landmark positions are normalized w.r.t. maximum values.

    (x_i,y_i)^T = (x,y)^T +  1  c_d2p·  R() (x^LP_i · cos(), y^LP_i)^T


To construct prior knowledge for training, we generate random variations of the in-plane components of the pose effecting both image and annotation:

    =1pt
    [              (R, β) ∼ (0Δ x_initial,  0360);                    Δ∼   [0](1/3Δ_initial)^2 ]

By drawing the variation of the position  in polar coordinates (R,  β) (𝒰: uniform distribution) and the  Δ from a normal distribution (𝒩), 
we skew the training samples towards configurations similar to the standard pose.
This skew favors accuracy based on good estimates over more distant cases, similar to class imbalance.
In effect, this appearance normalization increases the effectiveness (performance per network complexity) of the Deep Neural Network through the utilization of data similarity.

The patch appearance is dominated by the difference between the actual and the standard pose, i.e. the error of the prior knowledge.


 §.§ Pseudo-Landmark Prediction



Based on the 9248-normalized greyscale patch, a convolutional neural network (CNN) extracts the pose information.
Our analysis shows pseudo-landmarks outperform direct pose prediction.
While we explain the design of the pseudo-landmark estimation here, the direct pose estimation follows a similar strategy.

The CNN is designed after a VGG-fashion <cit.> with 13 weight layers. 
We benchmark the CNN on multiple design 
dimensions including the number of convolutional layers and blocks, the pooling layer type, 
the number of fully connected layers and the regularization strategy. 
In this context, we assume a block consists of multiple convolutional layers and ends in a 
pooling layer shrinking the layer size by a factor of 22.

All layers use ReLU activation. 
We double the number of channels after every block, starting with 32 for the first block. 

We use the Mean Squared Error as loss function.
For optimizers, we evaluated both Stochastic Gradient Descent with Nesterov Momentum 
update and Adam including different parameter combinations.



 §.§ Pose Reconstruction


We reconstruct the pose from the pseudo-landmarks based on their placement in a cross-shape (x^LP_i = 0 or y^LP_i = 0).
It enables us to invert eqn:keypoint2image geometrically by fitting lines through two subsets of landmarks.
The intersection yields the position = (x,y)^T and the slope the  .
The depth  and projection angle  are determined by using eqn:depthreconstruction and <ref> on the same landmark subsets. 


    = c_d2p^-1·|y^LP_i - y^LP_j|  |(x_i,y_i)^T - (x_j,y_j)^T|_2, i  j , x^LP_i/j = 0


    cos () = c_d2p·|(x_i,y_i)^T - (x_j,y_j)^T|_2  |x^LP_i - x^LP_j|, i  j, y^LP_i/j = 0







§ EXPERIMENTS & RESULTS




We performed a large number of different experiments with independent analysis of the position ,  , projection angle  and depth .
In this Section, we present the common experimental setup, evaluation metrics and multiple evaluations of i3PosNet.
We group in-depth evaluations in three blocks: <ref> general, <ref> modular design evaluation and <ref> limitations.
To streamline the description of the training and evaluation pipeline, we only present differences to the initially described common experimental setup.



 §.§ Common Experimental Setup

Generalization to unseen anatomies is a basic requirement to any CAI methodology, therefore we always evaluate on an unseen anatomy.
Following this leave-one-anatomy-out evaluation strategy, individual training runs only include 10k images, since 5k images are available for training per anatomy.

Training: 
Based on the training dataset (from Dataset A), 
we create 20 variations of the prior pose knowledge (see sec:appearancenormalization).
In consequence, training uses 20 different image patches per 10241024 image for all experiments (200k patches for training).
We train the convolutional neural network to minimize the Mean-Squared-Error of the Pseudo-Landmark regression with the Adam optimizer and standard parameters.
Instead of decay, we use a learning rate schedule that uses fixed learning rates for different training stages of 5 · 10^-3 initially and decrease the exponent by one every 35 epochs.
We stop the training after 80 epochs and choose the best-performing checkpoint by monitoring the loss on the validation set.

Testing:
The testing dataset (from Dataset A) features the unseen third anatomy, which we combine with 10 randomly drawn initial poses per image.
We use the strategy presented in sec:appearancenormalization to draw initial in-plane poses.
For testing, no out-of-plane components are required a priori.
The prediction is iterated three times, where the prior knowledge is updated with pose predictions each iteration.
Images with projection angles || > 80 are filtered out, because the performance significantly degrades biasing results.
This leads to on average 7864 tests per trained model.
This degradation is obvious given the ambiguity that arises, if projection direction and fiducial main axis (almost) align.
We provide an in-depth analysis of this limitation in sec:limitations.

Metrics:
We evaluated the in-plane and out-of-plane components of the predicted pose independently using 5 error measures. 
During annotation, we experienced,
out-of-plane components are much harder to recover from single images so we expect it much harder to recover for neural networks as well.

In-plane components: 
The Position Error (also reprojection distance <cit.>) is the Euclidean Distance in a plane through the fiducial and orthogonal to the projection normal.
It is measured in pixel (in the image) or millimeter (in the world coordinate system).
The relationship between Pixel and Millimeter Position Error is image-dependent because of varying distances between source and object.
The Forward Angle Error is the angle between estimated and ground truth orientation projected into the image plane in degrees, i.e. the in-plane angle error.

Out-of-plane components: 
For the Projection Angle Error, we consider the tilt of the fiducial out of the image plane in degrees.
Since the sign of the projection angle is not recoverable for small fiducials  (cos() = cos (-)), we only compare absolute values for this out-of-plane angle error.
The rotation angle is not recoverable at all for rotationally symmetric fiducials specifically at this size.
Finally, the Depth Error (millimeter) considers the distance between x-ray-source and fiducial (also known as the target registration error in the projection direction <cit.>).










 §.§ Evaluation

Comparison to registration:

Due to long execution times of the registration (>30 per image), the evaluation was performed on a 25-image subset of one (unseen) anatomy with two independent estimates image.
We limited the number of DRRs generated online to 400. 
At that point the registration was always converged.
Both i3PosNet and i2PosNet metrics represent distributions from 4 independent trainings to cover statistical training differences.
Comparing i3PosNet to two previously validated registration methods <cit.>, i3PosNet outperforms these by a factor of 5 (see fig:comparison).
The errors for i3PosNet and i2PosNet are below 0.5 Pixel (0.1) for all images.
At the same time i3PosNet reduces the single-image prediction time to 57.6 on one GTX 1080 at 6% utilization.

Real x-ray image evaluation:
Because of the significant computation overhead (projection), we randomly choose 25 images from anatomy 1 in Dataset A and performed two pose estimations from randomly sampled deviations from the initial estimate.
Four i3PosNet-models were independently trained for 80 epochs and evaluated for 3 iterations (see tab:results).

Generalization:
i3PosNet also generalizes well to other instruments.
Training and evaluating i3PosNet with corresponding images from Dataset B (drill and robot) shows consistent results across all metrics.





 §.§ Evaluation of design decisions


To emphasize our reliance on geometric considerations and our central design decision, 
we evaluated the prediction of forward angles (end-to-end) in comparison to the usage of pseudo-landmarks and 2D pose reconstruction (modular).
Comparing our modular solution to the end-to-end setup, we found the latter  
to display significantly larger errors for the , especially for relevant cases of bad initialization (see fig:mvse2e).



 §.§ Limitations to projection parameters


We evaluate the dependence on projection angles  imposed especially for drill-like instruments (strong rotational symmetry) (see fig:forwardanglebyprojangle).
We observe a decreasing quality starting at 60 with instabilities around 90 motivating the exclusion of images with || > 80 from the general experiments.



§ DISCUSSION & CONCLUSION



We estimate the pose of three surgical instruments using a Deep-Learning based approach.
By including geometric considerations into our method, we are able to approximate the non-linear properties of rotation and projection.

The accuracy provided by i3PosNet improves the ability of surgeons to accurately determine the pose of instruments, even when the line of sight is obstructed.
However, the transfer of the model trained solely based on synthetic data significantly reduces the accuracy, a problem widely observed in learning for CAI <cit.>.
As a consequence, while promising on synthetic results, i3PosNet closely misses the required tracking accuracy for temporal bone surgery on real x-ray data.
Solving this issue is a significant CAI challenge and requires large annotated datasets mixed into the training <cit.> or novel methods for generation <cit.>.

In the future, we also want to embed i3PosNet in a multi-tool localization scheme, where fiducials, instruments etc. are localized and their pose estimated without the knowledge of the projection matrix.
To increase the 3D accuracy, multiple orthogonal x-rays and a proposal scheme for the projection direction may be used.
Through this novel navigation method, surgeries previously barred from minimally-invasive approaches are opened to new possibilities with an outlook of higher precision and reduced patient surgery trauma.




§ COMPLIANCE WITH ETHICAL STANDARDS


Disclosure of potential conflicts of Interest: 
This research was partially funded by the German Research Foundation under Grant FOR 1585. 


Research involving Human Participants and/or Animals: This article does not contain any studies with human participants or animals. 


Informed consent: This articles does not contain patient data.














spbasic      





