Noname manuscript No.
(will be inserted by the editor)

Fast 5DOF Needle Tracking in iOCT

arXiv:1802.06446v1 [] 18 Feb 2018

Jakob Weiss · Nicola Rieke ·
Mohammad Ali Nasseri · Mathias
Maier · Abouzar Eslami · Nassir Navab

Abstract Purpose. Intraoperative Optical Coherence Tomography (iOCT) is
an increasingly available imaging technique for ophthalmic microsurgery that
provides high-resolution cross-sectional information of the surgical scene. We
propose to build on its desirable qualities and present a method for tracking the
orientation and location of a surgical needle. Thereby, we enable direct analysis of instrument-tissue interaction directly in OCT space without complex
multimodal calibration that would be required with traditional instrument
tracking methods. Method. The intersection of the needle with the iOCT scan
is detected by a peculiar multi-step ellipse fitting that takes advantage of the
directionality of the modality. The geometric modelling allows us to use the
ellipse parameters and provide them into a latency aware estimator to infer the
5DOF pose during needle movement. Results. Experiments on phantom data
and ex-vivo porcine eyes indicate that the algorithm retains angular precision
especially during lateral needle movement and provides a more robust and
consistent estimation than baseline methods. Conclusion. Using solely crosssectional iOCT information, we are able to successfully and robustly estimate
a 5DOF pose of the instrument in less than 5.4 ms on a CPU.
Jakob Weiss (Email: Jakob.Weiss@tum.de), Nicola Rieke
Computer Aided Medical Procedures, Technische Universität München,
Boltzmannstr. 3, 85748 Garching, Germany
Mohammad Ali Nasseri, Mathias Maier
Augenklinik und Poliklinik, Klinikum rechts der Isar der Technische Universit München,
81675 München, Germany
Abouzar Eslami
Carl Zeiss Meditec AG, 81379 München, Germany
Nassir Navab
Computer Aided Medical Procedures, Technische Universität München, Germany and
Johns Hopkins University, Baltimore, MD, USA
Conflict of Interest: The authors declare that they have no conflict of interest.

2

J. Weiss et al.

1 Introduction and Related Work
In Ophthalmic Microsurgery, the surgeon manipulates surgical instruments at
micron-scale precision while relying on visual feedback from the microscope.
Although state-of-the-art devices provide high resolution stereo view, depth
perception is still challenging due to the enface view. This makes it especially
challenging to estimate the distance between the utilized surgical instrument
and the anatomical surface [1]. Intraoperative Optical Coherence Tomography (iOCT) [2] has been introduced as an additional interventional imaging
modality, which provides high-resolution cross-sectional 2D images (B-Scan)
and therefore depth information in real time. The high spatial resolution provided by iOCT combined with its ability to image tissue structures below the
surface without requiring direct contact could make it a natural choice for
many computer-assisted ophthalmic procedures. Information about the location of the surgeon’s instrument inside the operating area is of essential importance for many assistance applications. However, prior work focused mainly on
the use of microscopic RGB images. In this context, Richa et al. [3] presented
tool tracking based on weighted mutual information between stereo images.
If the 3D CAD model of the tool is known, the instrument pose can be recovered by projective contour modelling [4]. Sznitman et al. [5] classified each
pixel as either background or tool part using a multiclass ensemble classifier.
The precise localisation of different tool parts is subsequently obtained by a
weighted averaging on the response scores. Allan et al. [6] estimate the full 3D
pose based on a level-set algorithm incorporating optical flow. Rieke et al. [7]
propose to combine a fast color-based tracker with a robust HoG feature-based
2D pose estimator via a dual Random Forest. An offline learning with online
adaption approach further increased the generalisation regarding unseen backgrounds and instruments [8]. Supervised Deep Learning based approaches [9,
10, 11] require an extensive annotated dataset to capture the wide range of
image distortions such as blur, specular reflections and limited focused field of
view.
Despite the recent advances in instrument tracking based on microscopic
RGB video, none of the methods can tackle a major inherent disadvantage:
Even if the tracking precision is perfect in the microscope image, it cannot
yield precise depth information through its projective imaging geometry. Acquiring information from the high resolution OCT at the 3D location of the
instrument would still not be feasible if the accurate spatial mapping between
the microscopic image and the iOCT is unknown. Although optical microscopy
and iOCT can share the same optical path in a device, this alignment requires
complex calibration routines. For the same reason, traditional navigation solutions such as optical tracking or electromagnetic tracking are not applicable
as they usually have an accuracy in the range of 200 to 1400µm. The intraoperative OCT on the other hand has an axial resolution of 5 to 10µm, which
is close to histopathology. Therefore, we propose to track the 5DOF pose of
the surgical instrument directly in the iOCT B-Scans and by that completely
avoid the bottleneck of calibration. OCT is a fundamentally different imaging

Fast 5DOF Needle Tracking in iOCT

3

Real-Time view of
OCT B-Scans
Microscope view and
augmented OCT
B-Scan locations
Ocular and Microscope

d0
27G needle and
eye phantom

n0

p0

y

Micromanipulator

z

p1

p2

x

p3

p4

l

l0

Fig. 1 Test scenario and coordinate system. Left: We use an OPMI Lumera 700
surgical microscope together with a Rescan 700 iOCT system and a Callisto Eye assistance
system, all from Carl Zeiss Meditec, Oberkochen. Right: Explanatory sketch describing
notation and spatial relationships between B-Scan direction and needle.

modality than conventional microscopic imaging: It measures echo delay and
intensity of back reflected near-infrared light waves [2]. Instead of an RGB enface view of the surgical scene, the B-Scans provide grayscale, cross-sectional
information. Due to the underlying physics, conventional surgical instruments
appear hyperreflective and consequently any signal beneath them is lost (c.f.
Fig. 3(b)). A first step towards instrument tracking in this type of data was recently proposed by Zhou et al. [12] in terms of instrument segmentation based
on a fully convolution network. The method however requires a volumetric
dataset, is not applicable to real-time applications and is restricted to static
instruments.
In this paper, we present a novel real-time 5DOF needle tracking method
in 2D iOCT images based on geometric modelling. The contribution of our
work is as follows. We show how the shadows and distinct reflection caused
by the surgical instrument in the B-Scans are related to the incident angle
of the tool. In a second step, we integrate this information from several - not
necessarily parallel - scans to infer the axis of the instrument. In order to tackle
the latency between OCT scan lines, we derive an application specific Kalman
filter that models the instrument movement between two acquired B-scans.
We do not track the needle tip explicitly, as this is impossible from single
B-Scans only unless the needle tip is exactly in plane with one OCT scan.
Nonetheless, the needle axis is already useful for many applications such as
computing the projected tissue intersection point or OCT repositioning. One
of the main advantages of the method is that it is applicable to general iOCT
scanning patterns such as parallel, crossing or volumetric patterns. With a
frame rate of more than 180 FPS, our method is easily capable of supporting
real-time applications. Throughout experimental results on a phantom and
ex-vivo porcine eyes, we demonstrate how the proposed method is able to
withstand the iOCT specific noise and remarkably improves the robustness to
instrument movement between iOCT scans. Furthermore, we exemplarily show

4
(a)

J. Weiss et al.
(b)

t-1

t

Fig. 2 Geometric Modelling. (a)The cross-section of the tool on a single B-Scan allows
to determine its 3D axis ` up to one ambiguity. (b) The Kalman filter resolves this ambiguity
and relates between time steps: A linear motion model of time step t−1 gives a line prediction
`ˆt ; this is corrected by the ellipse parameters measured at time t and leads to final estimate
`t . The blurred positions in image planes ρ indicate the estimated error covariances.

the potential impact on clinical practice by employing the 5DOF instrument
tracking for injection guidance.
2 Method
In this section, we derive the proposed method. First, we describe the setup
and specify our notations. Second, we demonstrate how to robustly detect the
elliptical cross-section of the needle in OCT B-Scans and its relation to the
needle pose. Finally, we integrate this noisy and incomplete data into a latency
aware filter and infer the 5DOF pose of the needle.
Geometric Setup: OCT allows real-time cross-sectional imaging of tissues by using a low-coherence light source and measuring the light reflected at
tissue interfaces with different indices of refraction. Based on the interference
pattern generated from superimposing the reflected light and the light traveled through a reference arm, OCT reconstructs an intensity profile along the
direction of the sampling laser (A-Scan). Standard iOCT engines have a galvanometer which allows vertical and horizontal deflection of the sampling laser.
The mirror is usually moved in a repeating pattern and the reconstructed signal is transferred after scanlines have been completed. For interventional OCT
imaging, usually a fixed pattern of several parallel and/or orthogonal scanlines
is used to provide the surgeon with different cross-sectional views (B-Scans) of
the working volume. From the known layout of this pattern, a transformation
to 3D space can be computed for each pixel. Commercially available iOCT
devices typically have a fixed A-Scan rate of 27-32 kHz, resulting in a B-Scan
update rate of 27-32 Hz if 1000 A-Scans per B-Scan are assumed. The number
and placement of B-Scans is determined by scanning patterns which can be
flexibly interchanged during an intervention.
To set a reference coordinate system, we define the axis of positive horizontal deflection as our x axis and the vertical deflection as the y axis. As
the projective field of view for small B-Scan lengths is narrow, we can neglect
the slight projectivity of the system and assume a euclidean coordinate system instead, thus assuming that the z axis is parallel to our A-scan direction.

Fast 5DOF Needle Tracking in iOCT
a

5
b

c

d

𝒅𝒎𝒊𝒏

e

f

g

𝒙𝒍

𝒙𝒓

Fig. 3 Ellipse parameters and detection. (a) Schematic view showing the relationship
between ellipse parameters and the needle. (b-d) Steps during ellipse detection: (b) Input
image. (c) Candidate points (violet + red) and fitted tissue layer (violet) with inlier margin.
(d) Tool candidate points p∗tool (yellow) obtained by filtering and thresholding d(p). (e)
RANSAC-fitted tool ellipse and inliers (green). (f ) Final ellipse obtained by non-linear
optimization (purple) and parameters. (g) Example that our method is still able to detect
the ellipse even if it is touching the tissue.

Figure 1 illustrates the geometric relationships. The plane corresponding to
a B-scan in 3D that is reconstructed from each scanline is parametrized as
ρ : ( #»
x − #»
p ) · n̄ = 0 where #»
p corresponds to the top-left corner of the B-Scan
image and n̄ is the plane normal. As a simplification, we model the tracked needle as an idealized cylinder with known diameter dn and an axis parametrized
as


sin θ cos φ
x + τ ¯l, τ ∈ R,
` : #»
x (τ ) = #»
x + τ  sin θ sin φ  = #»
cos θ
where θ and φ are azimuth and polar angle of the axis direction.
Ellipse Detection for Orientation Estimation: To calculate the axis of
the needle in 3D, we use the cross-section of the tool that is visible in an OCT
B-Scan B = (br,c ) ∈ Rnr ×nc with nr rows and nc columns. It can be shown
that the intersection between a cylinder and a plane in the non-degenerate
case forms an ellipse, which we parametrize through the center position in
image coordinates (Cx , Cy ) ∈ R2 , the length of the long (λ1 ) and short (λ2 )
axis and the angle α that is formed between the longer axis of the ellipse and
the negative y-axis of the image (c.f. Fig. 3(a)). Similar to acoustic shadowing
in ultrasound imaging, only the hyper-reflective surface of the metal needle is
visible on an OCT frame, while everything below is shadowed. Our algorithm
for ellipse detection and determination of its parameters is performed through
the following steps, illustrated in Fig. 3 (b - f ).
1. Candidate Points are defined as the pixels with maximum intensity along
each A-Scan (column) of the B-Scan image: pccand = (argmaxr (br,c ), c). The
set of all candidate points over all colums is pcand . These generally correspond to either the tool’s surface (ptool ), noise (pnoise ) or an anatomical
layer (peye ) such as the corneal surface in anterior segment or retinal pigment epithelium in posterior segment images.

6

J. Weiss et al.

2. A Tissue Layer representing the global eye shape is fitted based on these
candidate points by using the RANSAC algorithm. For anterior surgery
and posterior surgery with non-degenerate RPE, a circular model is used.
In cases where a circular model is not suitable, we use a polynomial of
order 4 to approximate the anatomical layer more closely.
3. Tool Candidate Points ptool ∗ can now be separated from the anatomical
surface peye by computing the distance of every candidate pointto the tissue
model d(p). To remove isolated candidate points that belong to pnoise or
not-hyperreflective tissue layers, we apply a 1D morphological opening,
closing, followed by median filtering each with a 15 px kernel and obtain
a filtered distance list d∗ (p). The tool candidates are then defined as the
set ptool ∗ = {p ∈ pccand |d∗ (p) > dmin } where dmin is a threshold to remove
points close to the surface.
4. (optional) For suspected pathologies (for example due to preoperative
imaging), we iteratively exclude from ptool ∗ all points which are closer than
dmin to an already excluded point, effectively removing all points which are
connected to the tissue layer.
5. A First Ellipse Estimate is computed by fitting an ellipse to the set of
tool candidate points ptool ∗ using RANSAC, provides an inlier set ptool .
6. Ellipse Refinement is performed by minimizing the geometric distance
between the ellipse and the points ptool . To reduce the number of parameters to optimize, we directly assign the center Cx = 0.5 ∗ (xr + xl ) where
xl , xr are the x coordinate of the leftmost and rightmost point of ptool .
Furthermore, we suppose a known needle diameter dn which corresponds
to λ2 .
7. Tool center line `t can then be related to the ellipse parameters by:

cos α = z̄ I − n̄n̄T ¯l = cos θ,
λ2
cos β
= n̄ · ¯l
λ1 =

(1)

where α is the angle between the major ellipse axis and the z axis and β
is the angle between the cylinder axis and the plane normal (c.f. Fig. 3
(a)). The first relationship follows by considering
the projection of ¯l into

T ¯
the B-Scan plane, computed as I − n̄n̄ l. Since α is the angle between
the A-Scan direction z̄ and this projected vector, its cosine is equal to
their dot product, which directly simplifies to cos θ. The second equality
is developed from considering the inset view of Fig. 3(a): From the right
triangle shown, sin γ = λλ21 follows, and sin γ = sin π2 − β = cos β = n̄ · ¯l
from trigonometry and the dot product definition.
Extended Kalman Filter: From the ellipse parameters in one single cross
section, it is not possible to uniquely reconstruct the pose of the cylindrical
needle. We use a Kalman filter [13] to fuse the noisy measurements from frames
at different time points and infer the current pose of the needle in each frame.
This section develops the state and measurement transition of the extended
Kalman Filter (EKF) that nonlinearly filters our measurements.

Fast 5DOF Needle Tracking in iOCT

7

The state and knowledge about our system are represented by the state

T
T
vector x = #»
x T , θ, φ, ẋT , θ̇, φ̇
and control vector u = n̄T , #»
p T , ∆t , where
#»
x , θ and φ model the needle axis and ẋ, θ̇ and φ̇ model its current velocity and
angular velocities, respectively. The control vector contains the parameters
defining the iOCT plane of the next measurement as well as the time since the
last measurement.
State Transition: We assume that our tool is influenced by unknown accelerT
ations and angular accelerations a = ( #»
a T#»
x , aθ , aφ ) drawn from a zero-mean
Gaussian distribution. Our preliminary state update is thereby defined as

 #»∗  
xt
1 0 0 ∆t 0 0
 θt  0 1 0 0 ∆t 0 

  
 φt  0 0 1 0 0 ∆t
 ∗ = 

(2)
 ẋt  0 0 0 1 0 0  · xt−1 + wt ,

  
 θ̇t  0 0 0 0 1 0 
000 0 0 1
φ̇t
2
2
2
a T#» , ∆t a , ∆t a , ∆t #»
a #» , ∆ta , ∆ta )T ∼ N (0, Q).
where w = ( ∆t #»
t

2

x

2

θ

2

φ

x

θ

φ

However, as the ellipse measurement is related to a different image plane
at the next timestep, we move the base point of the line xt to that plane (c.f.
Fig. 2 (b)). Therefore, we determine the final update for #»
x t by intersecting the
predicted tool center line `ˆt defined by #»
x ∗t , θt and φt with the image plane ρt−1
of the next measurement which is defined by the control vector ut−1 . Solving
the intersection between `ˆt and ρt results in a nonlinear state transition for #»
x:
( #»
p t−1 − #»
x ∗t ) · n̄t−1 ¯
#»
x t = `ˆt ∩ ρt−1 = #»
x ∗t +
· lt
(3)
¯lt · n̄t−1
With Eq. (2) and (3), our state transition function f : xt 7→ f (xt−1 , ut−1 , wt )
is fully specified. Re-basing the line onto the next OCT plane allows the
Kalman Filter to retain low error covariance for the position of the line due
to the resulting simple measurement transition, as opposed to letting the base
point of the tool line be arbitrary and performing the line-plane intersection as
part of the measurement transition. The more complex update of the position
implies that the new position is non-linearly dependent on the process noise
variable at . Therefore, we use the formulation of the EKF with non-linear
noise to update the predicted error covariance matrix as
T
P t|t−1 = F t−1 P t−1|t−1 F >
t−1 +Lt−1 QLt−1

(4)

where the P t−1|t−1 is the estimated error covariance matrix of the previous
time step and the Jacobian matrices of the state transition function F t−1 =
∂f
∂f
and Lt−1 = ∂w
are derived analytically.
∂x
x̂t−1|t−1 ,ut−1

x̂t−1|t−1 ,ut−1

Measurement Transition: From a single B-scan we find the parameters of the
cross-sectional ellipse as described above. We define the measurement transformation
λ2
ẑt = h(xt ) + vt = ( #»
c Tt , cos α, )T + vt
(5)
λ1

8

J. Weiss et al.

where ct are the 3D coordinates of the measured ellipse center and the measurement noise vt ∼ N (0, R) is assumed as additive Gaussian noise. Above
equation is motivated by Eq. 1, together with the fact that #»
c t is on the tool
axis and we can thus set #»
c t = #»
x t . The other components of vt are not directly
related to ẑt . The standard EKF innovation equation is St = Ht Pt|t−1 HTt +
t
Mt RMTt . Again, we are able to determine the Jacobian Ht = ∂ẑ
∂x x̂t and
∂h
Mt = ∂v x̂
= I analytically.
t|t−1
Due to our choice of representation and parametrization, two mathematical
singularities arise: Elements of Ft−1 and Ht tend to infinity for θt → kπ, k ∈ Z
as well as for `t · nt−1 → 0. The first implies that the needle is parallel to
the A-Scan direction while the second represents the needle parallel to the
OCT plane, so for both cases we are not able to see elliptical cross sections
in the OCT frame. We avoid numerical instabilities in our predictions by only
updating the predicted state if an ellipse has been detected and increasing the
time step ∆t for the next frame when no ellipse could be detected.

3 Experiments and Results
In this section, we evaluate the performance of our algorithm regarding different aspects: We first discuss computational performance and how we estimated
the measurement noise covariance matrix. Then we analyse the algorithm in a
series of experiments on both anterior and posterior segment in phantom eyes
and ex-vivo porcine eyes in terms of movement stability and robustness to
pathologies. Finally, we demonstrate an application example of our algorithm
which consists of an injection guidance application.
Choice of Parameters: For the minimum distance of candidate points
to the tissue layer, set dm in to 20px, which correspondes to 50microns. The
measurement noise covariance matrix R is determined by analyzing the covariance of the ellipse parameters across several data sets where the needle is at
an unknown but fixed angle with respect to the B-Scans. Based on the physical interpretation of the process noise as an unknown acceleration induced
2
by the surgeon(c.f. Sec. 2), we empirically choose values for #»
a #»
x = 3.0mm/s ,
2
aθ = aφ = 60deg/s and derive Q based on the definition of wt . These parameters are used across all experiments.
Computational Performance: The prediction and estimation steps for
the EKF reduce to matrix operations on matrices not larger than 10x10 elements, which can be implemented very efficiently. Therefore, the most computationally intensive part is the processing of each B-Scan to detect the ellipse
and find its parameters. Our CPU-based native implementation (C++) with
circle fitting and without pathology handling is able to process 1024x1024px
B-Scans in under 5.4 ms (186 FPS), on a Notebook with an Intel Core i76820HQ CPU @ 2.70 GHz and 16GiB RAM. We are therefore are easily able
to process the OCT framerates of current iOCT engines, which range around
27-32 FPS at this resolution.

Fast 5DOF Needle Tracking in iOCT

9
(c)

3 ours [°]

? ours [°]

3 base [°]

? base [°]

xours [10 -1 mm]

3 ours

(a)

100

? ours

3 base

? base

?

130

(b)

160

120
110

140
80

angle [°]

100
120

angle [°]

60

40

100

80
70

80
20

90

60
50

60

40
0

40
0

5

10

t [s]

15

20

0

5

10

15

3 ours

3 base

?ours

?base

t [s]

Fig. 4 Phantom Evaluation. Parameters with subscript base are from line fitting while
ours indicate the proposed method. (a) Needle movement lateral to the B-Scan direction.
Due to the fixed needle direction, θ and φ are expected to be constant. The baseline method
exhibits significant higher variation which is correlated with the lateral movement direction, while our method retains a stable orientation. (b) Needle rotation around the z-axis
simulated by rotation of the OCT scanning pattern (green). The known rotation angle φ is
recovered robustly while our method shows better stability regarding the expected constant
angle θ. (c) Box plot of estimated angles during axial movement. Our method shows much
reduced variation and therefore better results regarding the reconstructed orientation.

Movement Stability Evaluation: We evaluate the movement stability
of the proposed method on both phantom and ex-vivo porcine eyes. Since a
comparison to optical tracking methods or other traditional methods is not feasible, we employ a mechanical micromanipulator for generating ground truth
in terms of known, precise 3-DOF movements along an axis with fixed direction
for the surgical tool. To evaluate the quality of our estimation, we compare our
method to line fitting through the ellipse centers of two subsequent images.
Phantom Experiments: A first set of experiments was performed with a 27G
needle in an otherwise empty field of view with an OCT scanning pattern of
five parallel B-Scans to assess the stability of our algorithm to different kinds of
movements. With the needle fixed at a constant orientation, we move it only
along one axis of the micromanipulator in order to determine the influence
of translations on the pose estimation. Figs. 4 (a) show the effect of lateral
motion on the estimated direction of the needle. It can be seen that direct
line fitting exhibits systematic variations in the estimated polar angle φlin .
These are correlated with the transverse velocity because the linear fit cannot
distinguish between a rotated needle and a needle that has moved between
B-Scans. Our method generally produces more stable results and correctly
identifies lateral movement. The same effect can be seen for needle movement
along the Z-Axis (c.f. Fig. 4 (c)), where the movement influences the azimuthal
angle θ of the baseline estimate. As we cannot produce precise rotation with
the mechanical micromanipulator, we instead record an image sequence during
which the OCT scan pattern is rotated around the z axis and then manipulate
the metadata to ignore the known rotation, yielding an image sequence that

10

J. Weiss et al.
(a)

0.1

(b)
3ø

3 ours

?ø

3 ø,path

? ø,path

? ours

0.09

100

3 base

3 ø,skip

? ø,skip

? ~,enh

3 ~,enh

(d)

? base

0.08

80
0.07

60

Var [rad 2 ]

0.06

0.05

(c)

40

0.04
20
0.03

0

0.02

0.01
-20
0
0

5

10

Data Set Number

15

10

11

12

13

14

15

16

17

18

19

20

t [s]

Fig. 5 Ex-Vivo Evaluation. Reconstruction of the needle orientation during lateral
movement. Movement stability: (a) Analysis of the variance of the estimated orientation
during lateral movement with fixed orientation. Our method shows significantly reduced
variance for both angles in all data sets. Robustness to irregular tissue or ellipse detection
failure: (b) Robustness to failing ellipse detection is verified by simulating failed detection
in B-Scans marked as red. (c) Polynomial fit and additional pathology detection (Step
4) can distinguish pathology points (blue) from ellipse points (green). (d) While the basic
circle fit (orange) performs worse for pathologies when compared to same movement without
pathology (blue), a polynomial tissue model with pathology handling can still reconstruct
the needle axis (purple). Needle axis stability is also maintained when ellipse can only be
detected in three of five B-scans due to the needle touching the tissue in the other scans
(yellow).

is equivalent to rotating the tool around the z axis. The analysis of the fitted
orientation in Fig. 4 (b) shows that our algorithm is able to reliably reconstruct
this rotation while being less susceptible to noise in the ellipse estimation of
each frame. A slightly delayed angular adaptation of our method is noticeable
due to the smoothing property of the Kalman filter. However, we argue that
a rotation as strong as in this data set rarely occurs in ophthalmic surgery,
where needle movement is generally very slow and controlled.
Ex-vivo experiment: To evaluate the transfer towards real scenarios, we
performed a similar experiment on enucleated porcine eyes. We acquired a
series of 15 anterior data sets from 5 different eyes, each with the same setup
with a fixed needle angle and lateral movement. Fig. 5 (a) shows that our
algorithm can still robustly estimate the translation and greatly reduce the
variance in the estimated angle.
Irregular tissue and ellipse detection: To investigate the robustness of our
method in challenging cases, we have tested the following cases on one of the
ex-vivo data sets with lateral needle movement (c.f. Fig. 5(b-d)): To simulate
a needle being too close to the tissue to be found by our ellipse detection, we
force the ellipse detection to fail in two of five B-Scans. It can be seen that
our method is able to retain stable tracking. To validate robustness against
pathologies, we add a pathology to the B-Scans by shifting the candidate points

Fast 5DOF Needle Tracking in iOCT

11
(b)

3 ours [°]

? ours [°]

3 base [°]

D [10-1 mm]

? base [°]

(a)

100

80

60

40

20

0
0

1

2

3

4

5

6

7

t [s]

Fig. 6 Freehand movement in ex-vivo experiment. (a) Analysis of freehand needle
movement in posterior segment while scanning a Cross-Pattern of two perpendicular BScans. D (green line) is the distance between estimated tool axis and intersecting line of
the two B-Scans. Linear fitting fails to compute a reliable pose while our method can still
provide a stable tracking. (b) Microscope view with OCT scanning location overlaid in blue.
Yellow circles indicate the centers of the detected ellipse from the two B-Scans. Orange line
is the estimated line from the baseline method. Green line is our estimated, highlighting the
benefit of using the ellipse shape for more stable tracking.

pcand to resemble an irregularly shaped retina (Fig. 5(c)). Our experiment
shows that the basic circular fit fails while our enhanced pathology handling
successfully recovers stable tracking results, at the cost of slightly increased
per-frame processing time of 7.1ms.
Pattern comparison and Freehand movement: We performed an
experiment with freehand movement of the needle inside the OCT region while
scanning with a pattern consisting of only two perpendicular B-Scans. Fig. 6
(a) shows the orientation of the sequence once again compared to the baseline
method. This highlights the limitations of the baseline method, which is unable
to provide a meaningful estimate when the intersection points are too close
together, which is the case when the needle moves closer to the intersection of
the two B-Scans (Fig. 6 (b)). It can be seen that our method is susceptible to
bad initialization by the linearly fitted line through the first frames, however
it is able to converge to a stable tracking after a few seconds and retain this
pose even when the needle is close to the center. This demonstrates that our
estimator can still infer the needle orientation from the ellipse shape when the
ellipse centers alone do not provide enough information.
Injection Guidance Application: As an example application, we have
designed an assistance application that provides injection guidance during subretinal injection by showing the surgeon the projected intersection point of
the tracked needle with the target layer. During an actual injection, the OCT
would be optimally placed through this injection point to give the surgeon a
good impression of the current needle depth. Manual repositioning is however
not feasible. We use the proposed algorithm to track the injection needle and
show the projected injection point overlayed on the microscope image. To estimate the intersection point, we first reconstruct the target surface by using

12

J. Weiss et al.
Guidance in Microscope View

Based on Instrument Tracking in OCT

Fig. 7 Screenshot of the injection guidance application. Left: Augmented view of
the surgical scene, showing the camera view with the overlaid OCT scanning locations as
well as the projected intersection point with the RPE layer. Current and last B-Scan are
marked with white and blue bars for illustrative purposes. Right: Schematic view of the
3D relationships between B-Scans (blue), current needle estimate (green), and intersection
point with the target surface (red). These relationships cannot easily be inferred from a
simple 2D microscope image.

the tissue surface points peye of the ellipse detection stage of each B-Scan,
which correspond to pixels on the RPE for posterior images (c.f. Fig. 3, b).
We reproject the points peye from several B-Scans to 3D space and fit a sphere
using RANSAC. The intersection points of the tracked tool axis with the estimated sphere are projected to the camera coordinate system using the 2D
calibration provided by the manufacturer, which is valid in the current microscope focus plane. We draw a circle corresponding to the needle thickness
to indicate the injection point. Thus, we provide distance perception without
explicitly tracking the needle tip, as the surgeon can infer the distance of the
needle to the surface by the distance of the projected intersection point and
the instrument tip visible in the camera image.

4 Conclusion
We presented a novel algorithm for tracking a surgical needle in 3D space
solely using the high-resolution, cross-sectional OCT view. The method makes
no assumptions on the layout of the scanning pattern and is therefore easily integratable into existing systems with dynamically changing scanning patterns.
We avoid expensive computations by geometric modelling and consequently,
our method is able to update at more than 180 FPS, which easily matches
the high framerates provided by today’s OCT engines. In experimental evaluation both on phantom and ex-vivo porcine eyes, show that the method is able
to compensate needle movement between subsequent B-Scans, thus showing
increased robustness towards B-Scan latency and generating a more stable estimate compared to line fitting. We demonstrate the usefulness of our tracking
algorithm by providing a simple augmented reality scenario for subretinal in-

Fast 5DOF Needle Tracking in iOCT

13

jection. As our ellipse detection fails for B-Scans where the needle is inside the
tissue, we currently rely on the needle to be visible above the tissue in at least
one B-Scan to maintain our tracking. A possible future extension is to extend
the ellipse detection to support cases where the needle is already penetrating
tissue to improve tracking accuracy during this critical phase. Furthermore,
the needle guidance application can be extended by a more robust target surface reconstruction to provide a more precise intersection point visualization,
and perform a validation study on the benefits of such a system.
Ethical statement: All procedures performed in studies involving animals were in accordance with the ethical standards of the institution or practice at which the studies were
conducted. This article does not contain patient data.

References
1. Roodaki, H., Amat di San Filippo, C., Zapp, D., Navab, N., Eslami, A.: A surgical guidance system for big-bubble deep anterior lamellar keratoplasty. In: MICCAI, Springer
(2016) 378–385
2. Ehlers, J.: Intraoperative optical coherence tomography: past, present, and future. Eye
30(2) (2016) 193–201
3. Richa, R., Balicki, M., Meisner, E., Sznitman, R., Taylor, R., Hager, G.: Visual tracking
of surgical tools for proximity detection in retinal surgery. In: IPCAI, pp. 55–66 (2011)
4. Baek, Y.M., Tanaka, S., Kanako, H., Sugita, N., Morita, A., Sora, S., Mochizuki, R.,
Mitsuishi, M.: Full state visual forceps tracking under a microscope using projective
contour models. In: Proc. of IEEE ICRA, pp. 2919–2925 (2012)
5. Sznitman, R., Becker, C., Fua, P.: Fast part-based classification for instrument detection
in minimally invasive surgery. In: Golland, P., Hata N., Barillot C., Hornegger J., Howe
R. (eds) MICCAI 2014. LNCS, vol. 8673, pp. 692–699. Springer, Heidelberg (2014)
6. Allan, M., Chang, P.L., Ourselin, S., Hawkes, D.J., Sridhar, A., Kelly, J., Stoyanov, D.:
Image based surgical instrument pose estimation with multi-class labelling and optical
flow. In: MICCAI, Springer (2015) 331–338
7. Rieke, N., Tan, D.J., Amat di San Filippo, C., Tombari, F., Alsheakhali, M., Belagiannis,
V., Eslami, A., Navab, N.: Real-time localization of articulated surgical instruments in
retinal microsurgery. Medical Image Analysis 34 (2016)
8. Rieke, N., Tan, D.J., Tombari, F., Page Vizcaı́no, J., Amat di San Filippo, C., Eslami,
A., Navab, N.: Real-time online adaption for robust instrument tracking and pose
estimation. In: International Conference on Medical Image Computing and ComputerAssisted Intervention, Springer (2016) 422–430
9. Garcia Peraza Herrera, L., Li, W., Gruijthuijsen, C., Devreker, A., Attilakos, G., Deprest, J., Vander Poorten, E., Stoyanov, D., Vercauteren, T., Ourselin, S.: Real-time
segmentation of non-rigid surgical tools based on deep learning and tracking. In: CARE
workshop at International Conference on Medical Image Computing and ComputerAssisted Intervention, Springer (2016)
10. Laina, I., Rieke, N., Rupprecht, C., Vizcaı́no, J.P., Eslami, A., Tombari, F., Navab, N.:
Concurrent segmentation and localization for tracking of surgical instruments. In: International Conference on Medical Image Computing and Computer-Assisted Intervention
2017, Springer (2017) 664–672 The first two authors contributed equally to this paper.
11. Kurmann, T., Neila, P.M., Du, X., Fua, P., Stoyanov, D., Wolf, S., Sznitman, R.: Simultaneous recognition and pose estimation of instruments in minimally invasive surgery.
In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer (2017) 505–513
12. Zhou, M., Roodaki, H., Eslami, A., Chen, G., Huang, K., Maier, M., Lohmann, C., Knoll,
A., Nasseri, M.: Needle Segmentation in Volumetric Optical Coherence Tomography
Images for Ophthalmic Microsurgery. Applied Sciences 7(8) (jul 2017) 748
13. Jazwinski, A.H.: Stochastic processes and filtering theory. Courier Corporation (2007)

