A Framework for Evaluating 6-DOF Object Trackers
Mathieu Garon Denis Laurendeau Jean-François Lalonde





headings
18SubNumber982  

 


A Framework for Evaluating 6-DOF Object Trackers

Mathieu Garon, Denis Laurendeau and Jean-François Lalonde


Université Laval[]


A Framework for Evaluating 
 6-DOF Object Trackers
    Mathieu Garon 0000-0003-1811-4156, Denis Laurendeau 0000-0003-2858-5955 and Jean-François Lalonde 0000-0002-6583-2364
    December 30, 2023
=========================================================================================================================




We present a challenging and realistic novel dataset for evaluating 6-DOF object tracking algorithms. Existing datasets show serious limitations—notably, unrealistic synthetic data, or real data with large fiducial markers—preventing the community from obtaining an accurate picture of the state-of-the-art. Using a data acquisition pipeline based on a commercial motion capture system for acquiring accurate ground truth poses of real objects with respect to a Kinect V2 camera, we build a dataset which contains a total of 297 calibrated sequences. They are acquired in three different scenarios to evaluate the performance of trackers: stability, robustness to occlusion and accuracy during challenging interactions between a person and the object. We conduct an extensive study of a deep 6-DOF tracking architecture and determine a set of optimal parameters. We enhance the architecture and the training methodology to train a 6-DOF tracker that can robustly generalize to objects never seen during training, and demonstrate favorable performance compared to previous approaches trained specifically on the objects to track. 







§ INTRODUCTION









With the recent emergence of 3D-enabled augmented reality devices, tracking 3D objects in 6 degrees of freedom (DOF) is a problem that has received increased attention in the past few years. As opposed to SLAM-based camera localization techniques—now robustly implemented on-board various commercial devices—that can use features from the entire scene, 6-DOF object tracking approaches have to rely on features present on a (typically small) object, making it a challenging problem. Despite this, recent approaches have demonstrated tremendous performance both in terms of speed and accuracy <cit.>. 

Unfortunately, obtaining an accurate assessment of the performance of 6-DOF object tracking approaches is becoming increasingly difficult since accuracy on the main dataset used for this purpose has now reached closed to 100%. Introduced in 2013 by Choi and Christensen <cit.>, their dataset consists of 4 short sequences of purely synthetic scenes. The scenes are made of unrealistic, texture-less backgrounds with a single colored object to track, resulting in noiseless RGBD images (see fig. <ref>-(a)). The object is static and the camera rotates around it in wide motions, occasionally creating small occlusions (at most 20% of the object is occluded). While challenging at first, the dataset has now essentially been solved for the RGBD case. For example, the method of Kehl et al. <cit.> (2017) reports an average error in translation/rotation of 0.5mm/0.26^∘, which is an improvement of 0.3mm/0.1^∘ over the work of Tan et al. (2015) <cit.>, who have themselves reported a 0.01mm/1^∘ improvement to the approach designed by Krull et al. (2014) <cit.>. The state of the art on the dataset has reached a near-perfect error of 0.1mm/0.07^∘ <cit.>, which highlights the need for a new dataset with more challenging scenarios.






Another dataset, introduced by Garon and Lalonde <cit.>, includes 12 sequences of real objects captured with real sensors. While a significant improvement over the synthetic dataset of <cit.>, dealing with real data raises the issue of providing accurate ground truth pose of the object at all times. To obtain this ground truth information, their strategy (also adopted in 6-DOF detection datasets <cit.>) is to use calibration boards with fiducial markers. While useful to accurately and easily determine an object pose, this has the unfortunate consequence of constraining the object to lie on a large planar surface (fig. <ref>-(b)). 

In this paper, we present a novel dataset allowing the systematic evaluation of 6-DOF tracking algorithms in a wide variety of real scenarios without requiring calibration boards (fig. <ref>-(c)). Our dataset is one order of magnitude larger than the previous work: it contains 297 sequences of 11 real objects. The sequences are split into 3 different scenarios, which we refer to as stability, occlusion, and interaction. The stability scenario aims at quantifying the degree of jitter in a tracker. The object is kept static and placed at various angles and distances from the camera. The occlusion scenario, inspired by <cit.>, has the object rotating on a turntable and being progressively occluded by a flat panel. Occlusion ranges from 0% (unoccluded) to 75%, thereby testing trackers in very challenging situations. Finally, in the interaction scenario, a person is moving the object around freely in front of the camera (fig. <ref>-(c)), creating occlusions and varying object speed. 

In addition, we also introduce two new 6-DOF real-time object trackers based on deep learning. The first, trained for a specific object, achieves state-of-the-art performance on the new dataset. The second, trained without a priori knowledge of the object to track, is able to achieve an accuracy that is comparable with previous work trained specifically on the object. These two trackers rely on the same deep learning architecture and only differ in the training data. Furthermore, both of our trackers have the additional significant advantage of requiring only synthetic training data (i.e. no real data is needed for training). We believe this is an exciting first step in the direction of training generic trackers which do not require knowledge of the object to track at training time. 

In summary, this paper brings 3 key contributions to 6-DOF object tracking: 


  * A novel dataset of real RGBD sequences for the systematic evaluation of 6-DOF tracking algorithms that is one order of magnitude larger than existing ones, and contains 3 challenging scenarios; 

  * A real-time deep learning architecture for tracking objects in 6-DOF which is more stable and more robust to occlusions than previous approaches; 

  * A generic 6-DOF object tracker trained without knowledge of the object to track, achieving performance on par with previous approaches trained specifically on the object. 













§ RELATED WORK



There are two main relevant aspects in 6-DOF pose estimation: single frame object detection and multi-frame temporal trackers. The former has received a lot of attention in the literature and benefits from a large range of public datasets. The most notorious dataset is arguably Linemod <cit.>, which provide 15 objects with their mesh models and surface colors. To obtain the ground truth object pose, a calibration board with fiducial markers is used. Since then, many authors created similar but more challenging benchmarks <cit.>. However these datasets do not contain temporal and displacement correlation between each frame, which makes them inadequate for evaluating temporal trackers. 




In the case of temporal tracking, only a few datasets exist to evaluate the approaches. As mentioned in the introduction, the current, widely used standard dataset is the synthetic dataset of Choi and Christensen <cit.>, which contains 4 sequences with 4 objects rendered in a texture-less virtual scene. Another available option is the one provided by Akkaladevi et al. <cit.> who captured a single sequence of a scene containing 4 different objects with a Primesense sensor. However, the 3D models are not complete and do not include training data that could be exploited by learning-based methods. Finally, recent work by Garon and Lalonde <cit.> proposed a public dataset of 4 objects containing 4 sequences with clutter and an additional set of 8 sequences with controlled occlusion on a specific object. Fiducial markers are used to generate the ground truth pose of the model, which limits the range of displacements that can be achieved. In contrast, we propose a new method to collect ground truth pose data that makes the acquisition simpler without the need for fiducial markers. 

There is an increasing interest in 6-DOF temporal trackers since they were shown to be faster and more robust than single frame detection methods. In the past, geometric methods based on ICP <cit.> were used for temporal tracking, but they lack robustness for small objects and are generally computationally expensive. Data-driven approaches such as the ones reported in <cit.> can learn more robust features and the use of the Random Forest regressor <cit.> decreases the computing overhead significantly. Other methods show that the contours of the objects in RGB and depth data provide important cues for estimating pose <cit.>. While their optimization techniques can be accurate, many assumptions are made on the features which restrict the type of object and the type of background that can be dealt with. Recently, Garon and Lalonde <cit.> proposed a deep learning framework which can learn robust features automatically from data. They use a feedback loop by rendering the 3D model at runtime at the previous pose, and regress the pose difference between the rendered object and the real image. While their method compares to the previous work with respect to accuracy, their learned features are more robust to higher level of occlusion and noise. A downside is that their method needs a dataset of real images and a specific network has to be trained for each object which can be time consuming. We take advantage of their architecture but introduce novel ideas to provide a better performing tracker that can be trained entirely on synthetic data. In addition, our network can be trained to generalize to previously unseen objects.




§ DATASET CAPTURE AND CALIBRATION




Building a dataset with calibrated object pose w.r.t the sensor at each frame is a challenging task since it requires an accurate method to collect ground truth object pose. Until now, the most practical method to achieve this task was to use fiducial markers and calibrate the object pose w.r.t these markers <cit.>. However, this method suffers from two main drawbacks. First, the object cannot be moved independently of the panel so this restricts the camera to move around the object of interest. Second, the scene always contains visual cues (the markers) which could involuntarily “help” the algorithms. 

Our approach eliminates these limitations. A Vicon™  MX-T40 motion capture system is used to collect the ground truth pose of the objects in the scene. The retroreflective Vicon markers that must be used are very small in size (3mm diameter) and can be automatically removed in a post-processing step. In this section, we describe the capture setup and the various calibration steps needed to align the object model and estimate its ground truth pose. The resulting RGBD video sequences captured using this setup are presented in sec. <ref>. 






 §.§ Capture setup


The motion capture setup is composed of a set of 8 calibrated cameras that track retro-reflective markers of 3mm in diameter installed on the objects of interest in a 3×3×3m^3 work area. Vicon systems can provide a marker detection accuracy of up to 0.15 mm on static objects and 2mm on moving objects according to <cit.>. A Kinect V2 is used to acquire the RGBD frames, and is calibrated with the Vicon to record the ground truth pose of the objects in the Kinect coordinate system. The actual setup used to capture the dataset is shown in fig. <ref>-(a). 








 §.§ Calibration



With an RGB-D sensor such as the Kinect V2, color and depth values are projected onto two different planes. We define the Kinect reference frame (“knt”) as the origin of its RGB camera, and align the depth data by reprojecting it to the color plane using the factory calibration parameters. We calibrate the depth correction as in Hodan et al. <cit.>. In this section, the notation ab is used to denote a rigid transformation from reference frame “a” to “b”.

We aim to recover the pose of the object in the Kinect reference frame objknt (fig. <ref>-(b)). To do so, we first rely on the Vicon motion capture system, which has its own reference frame “vcn”. The set of retroreflective markers installed on the object define the local reference frame “objm”. Similarly, the set of markers placed on the Kinect define the local reference frame “kntm”. The Vicon system provides the transformations objmvcn and kntmvcn directly, that is, the mappings between the object and Kinect markers and the Vicon reference frame respectively. The transformation between the object markers and the Kinect markers is obtained by chaining the previous transformations:


    objmkntm = (kntmvcn)^-1objmvcn .


The pose objknt is recovered with the transformations between the local frames defined by the markers and the object/Kinect reference frames objobjm and kntkntm: 


    objknt = (kntkntm)^-1objmkntmobjobjm .


The calibration procedures needed to obtain these two transformations, also shown in gray in fig. <ref>-(b), are detailed next. 





  §.§.§ Kinect calibration


In order to find the transformation kntmknt between the local frame defined by the markers installed on the Kinect and its RGB camera, we rely on a planar checkerboard target on which Vicon markers are randomly placed. Then, the position of each corner of the checkerboard is determined with respect to the markers with the following procedure. A 15cm-long pen-like probe that has a 1cm Vicon marker attached at one end was designed for this purpose. The sharp end is placed on the corner to be detected, and the probe is moved in a circular motion around that point. A sphere is then fit (using least-squares) to the resulting marker positions (achieving an average radius estimation error of 0.7mm), and the center of the sphere is kept as the location for the checkerboard corner. The checkerboard target was then moved in the capture volume and corners were detected by the Kinect RGB camera, thereby establishing 2D-3D correspondences between these points. The perspective-n-points algorithm <cit.> was finally used to compute kntmknt. 





  §.§.§ Object calibration


To estimate the transformation between the local frame defined by the markers placed on the object and its mesh coordinate system objmobj, we rely on the Kinect pose calibrated with the method described previously. As a convention, we define the origin of the object local coordinate system at the center of mass of the markers, the same convention is used for the mesh by using the center of mass of the vertices. We roughly align the axis and use ICP to refine its position (based on the Kinect depth values). Finally, with the help of a visual interface where a user can move and visualize the aligment of the object, fine-scale adjustments can be performed manually from several viewpoints to minimize the error between the observed object and the reprojected mesh.








  §.§.§ Synchronization



In addition to spatial calibration, precise temporal alignment must be achieved to synchronize Vicon and Kinect frames. Unfortunately, the Kinect does not offer hardware synchronization capabilities, therefore we adopt the following software solution. We assume that the sequences are short enough to neglect clock drift. We also assume a stable sampling of the Vicon system on a high bandwidth closed network. In this setup, synchronization can be achieved by estimating the (constant) time difference δ t between the Vicon and the Kinect frame timestamps. By moving the checkboard of sec.<ref> with varying speed, we estimate the δ t that minimizes the reprojection error between the checkerboard corners from sec. <ref> and the Vicon markers.




  §.§.§ Removing the markers



The 3mm markers used to track the object are retro-reflective and, despite their small size and their low number (7 per object on average), they nevertheless create visible artifacts in the depth data measured by the Kinect, see fig. <ref>. We propose a post-processing algorithm for automatically removing them in all the sequences. First, to ensure that the marker can be observed by the Kinect we reproject the (known) marker positions onto the depth image and compute the median distance between the depth in a small window around the reprojected point and its ground truth depth. If the difference is less than 1cm, the point is considered as not occluded, and will be processed. Finally, we render the depth values of the 3D model at the given pose and replace the 10×10 pixel window from the original image with the rendered depth values. For more realism, a small amount of gaussian noise is added. Pixels from the background are simply ignored. On average, only 3.4% of the object pixels are corrected. We also minimize the chances of affecting the geometric structure of the object by placing the markers on planar surfaces. Fig. <ref> shows a comparison of the error between a Kinect depth image captured with markers, and another image of the same scene with markers that have been corrected with our algorithm. The RMSE of the pixel patches around the markers is 139.8 mm without the correction, and 4.7 mm with the correction.

















§ DATASET SCENARIOS, METRICS, AND STATISTICS







This section defines novel ways to evaluate 6-DOF trackers using calibrated sequences captured with the setup presented in sec. <ref>. We provide an evaluation methodology that will reflect the overall performance of a tracker in different scenarios. To attain this objective, we captured 297 sequences of 11 different objects of various shapes in 3 scenarios: stability, occlusion, and interaction. We also provide quantitative metrics to measure the performance in each scenario. Our dataset and accompanying code is available at <http://www.jflalonde.ca/projects/6dofObjectTracking>. 






 §.§ Performance metrics


Before we describe each scenario, we first introduce how we propose to evaluate the difference between two poses 𝐏_1 and 𝐏_2. Here, a pose 𝐏 = [[ 𝐑 𝐭 ]] is described by a rotation matrix 𝐑 and a translation vector 𝐭. Previous works consider the average of each axis component in translation and rotation separately. The side effect of this metric is that a large error on a single component is less penalized. To overcome this limitation, the translation error is simply defined as the L2 norm between the two translation vectors: 

    δ_𝐭(𝐭_1, 𝐭_2) = ||𝐭_1 - 𝐭_2||_2  .


The distance between two rotation matrices is computed using:


    δ_𝐑(𝐑_1, 𝐑_2) = arccos( (𝐑_1^T𝐑_2) - 1/2)  ,


where (·) denotes the matrix trace.




 §.§ Scenarios



  §.§.§ The stability scenario



In this first scenario, we propose to quantify the degree of pose jitter when tracking a static object. To evaluate this, we captured 5-second sequences of the object under 4 different viewpoints and with 3 configurations: at a distance of 0.8m from the sensor (“near”), of 1.5m from the sensor (“far”), and of 0.8m from the sensor, but this time with distractor objects partly occluding the object of interest (“occluded”).

To measure the stability, Tan et al. <cit.> use the standard deviation of the pose parameters on a sequence. We propose a different metric inspired from <cit.> that penalizes variation from frame to frame instead of the general distribution across the sequence. We compute the distance between poses 𝐏_i-1 and 𝐏_i at time i. In other words, we report the distribution of δ_𝐭(𝐭_i-1, 𝐭_i) and δ_𝐑(𝐑_i-1, 𝐑_i) for all frames of the stability scenario.











  §.§.§ The occlusion scenario



To evaluate the robustness to occlusion, we follow <cit.> and place the object on a turntable at 1.2m from the sensor, and a static occluder is placed in front of the object in a vertical and horizontal position. We compute the amount of occlusion based on the largest dimension of the object, and provide sequences for each object from 0% to 75% occlusion in 15% increments, which results in a total of 11 sequences per object. 

Here, we compute errors by comparing the pose 𝐏_i at time i with the ground truth 𝐏^*_i for that same frame, i.e., δ_𝐭(𝐭^*_i, 𝐭_i) and δ_𝐑(𝐑^*_i, 𝐑_i). Temporal trackers may lose tracking on difficult frames. This can affect the overall score depending on the moment where the tracker fails. To bypass this limitation, we initialize the tracker at the ground truth pose 𝐏^*_i every 15 frames as in <cit.>. 



  §.§.§ The interaction scenario






In this last scenario, the experimenter holds the object in his hands and manipulates it in 4 different ways: 1) by moving the object around but without rotating it (“translation-only”); 2) by rotating the object on itself without translating it (“rotation-only”); 3) by freely moving and rotating the object around at low speeds (“free-slow”); and 4) by freely moving and rotating the object at higher speeds and by voluntarily generating more occlusions (“free-hard”). In all situations but the “free-hard”, we reset the tracker every 15 frames and we report δ_𝐭(𝐭^*_i, 𝐭_i) and δ_𝐑(𝐑^*_i, 𝐑_i) as in sec. <ref>. 

Since the object speed varies, we also compute the translational and rotational inter-frame displacement (δ_𝐭(𝐭^*_i-1, 𝐭^*_i), δ_𝐑(𝐑^*_i-1, 𝐑^*_i)) and report the performance metric above as a function of that displacement. In addition, it is also informative to count the number of times the tracker has failed. We consider a tracking failure when either δ_𝐭(𝐭^*_i, 𝐭_i)> 3 cm or δ_𝐑(𝐑^*_i, 𝐑_i) > 20^∘ for more than 7 consecutive frames. When a failure is detected, the tracker is reset at the ground truth pose 𝐏^*. We report these failures on the “free-hard” sequences only. 







 §.§ Dataset statistics
 

We selected 11 different objects to obtain a wide variety of object geometries and appearance, as shown in fig. <ref>. To obtain a precise 3D model of each object in the database, each of them was scanned with a Creaform GoScan™ handheld 3D scanner at a 1mm voxel resolution. The scans were manually cleaned using Creaform VxElements™ to remove background and spurious vertices.

Overall, the dataset contains 297 sequences: 27 sequences for each object. The breakdown is the following: 12 sequences for stability (4 viewpoints, 3 configurations: “near”, “far”, “occluded”); 11 sequences for occlusion (0% to 75% in 15% increment for both horizontal and vertical occluders); and 4 sequences for interaction (“rotation-only”, “translation-only”, “free-slow”, “free-hard”).
It also contains high resolution textured 3D models for each object.







§ ANALYZING A DEEP 6-DOF TRACKER WITH OUR DATASET



As a testbed to evaluate the relevance of the new dataset, we borrow the technique of Garon and Lalonde <cit.> who train a 6-DOF tracker using deep learning, but propose changes to their architecture and training methodology. We evaluate several variants of the network on our dataset and show that it can be used to accurately quantify the performance of a tracker in a wide variety of scenarios. 




 §.§ Training an object-specific tracker







































We propose improvements over the previous work of <cit.> by adding 5 main changes: 2 to the network architecture, and 3 to the training procedure. The new proposed network architecture is shown in fig. <ref>. As in <cit.>, the network accepts two inputs: an image of the object rendered at its predicted position (from the previous timestamp in the video sequence) 𝐱_pred, and an image of the observed object at the current timestamp 𝐱_obs. The last layer outputs the 6-DOF (3 for translation, 3 for rotation in Euler angles) representing the pose change between the two inputs. We first replace convolution layers by the “fire” modules proposed in <cit.>. The second change, inspired by DenseNet <cit.>, is to concatenate the input features of each layers to the outputs before being maxpooled. Our improvement requires the same runtime as <cit.>, which is 6 ms on a Nvidia GTX-970M. As in <cit.>, the loss used is the MSE between the predicted and ground truth pose change. Note that we experimented with the reprojection loss <cit.>, but found it did not help in our context.

We also propose changes to the training procedure of <cit.>. Their approach consists in generating pairs of synthetic views of the object with random pose changes between them. To sample the random pose changes, they proposed to independently sample a random translation t_x,y,z∼ U(20mm,20mm) and rotation r_α,β,γ∼ U(-10^∘ ,10^∘) in Euler angle notation, with U(a,b) referring to a uniform distribution on interval [a,b]. Doing so unfortunately biases the resulting pose changes. For example, small amplitude translations are quite unlikely to be generated (since this requires all three translation components to be small simultaneously). Our first change is to sample a random translation vector and magnitude separately. The translation vector 𝐯_t is sampled in spherical coordinates (θ_t,ϕ_t), where θ_t ∼ U(-180^∘,180^∘) and ϕ_t = cos^-1(x) with x ∼ U(-1,1). The translation magnitude m_t is drawn from a Gaussian distribution m_t ∼𝒩(0, Δ t). The same process is repeated for rotations, where the rotation axis 𝐯_r and angle m_r ∼𝒩(0, Δ r) are sampled similarly. Here, we intentionally parameterize the translation magnitude m_t and rotation angle m_r distributions with Δ t and Δ r, since the range of these parameters may influence the behavior of the network. Our second change is to downsample the depth channel to better match the resolution of the Kinect V2. Our third change consists in a data augmentation method for RGBD images where we randomly set a modality (depth or RGB) to zero during training, which has the effect of untangling the features of both modalities. With these changes, we can now rely purely on synthetic data to train the network (in <cit.> a set of real frames was required to fine-tune the network). 













 §.§ Training a generic tracker



To train a generic 6-DOF object tracker, we experimented with two ways of generating a training dataset, using the same network architecture, loss, and training procedure described in sec. <ref>. First, we generate a training set of images that contain all 11 objects from our dataset, as well as 30 other objects. These other objects, downloaded from 3D Warehouse[Available at: <https://3dwarehouse.sketchup.com>.] and from “Linemod” <cit.>, show a high diversity in geometry and texture and are roughly of the same size. We name the network trained on this dataset the “multi-object” network. Second, we generate a training set of images that contain only the 30 other objects—the actual objects to track are not included. We call this network “generic”, since it never saw any of the objects in our dataset during training. Note that all these approaches require the 3D model of the object to track at test time, however. 








§ EXPERIMENTS



In this section, we perform an exhaustive evaluation of the various approaches presented in sec. <ref> using our novel dataset and framework proposed in sec. <ref>. First, we analyze the impact of varying the training data generation hyper-parameters Δ t and Δ r for the object-specific case. Then, we proceed to compare our object-specific, “multi-object”, and “generic” trackers with two existing methods: Garon and Lalonde <cit.> and Tan et al. <cit.>. 





 §.§ Analysis to training data generation parameters



We now apply the evaluation methodology proposed in sec. <ref> on the method presented above and evaluate the influence of the Δ r and Δ t hyper-parameters on the various metrics and sequences from our dataset. We experiment by varying Δ t ∈{10, 20, 30, 40, 50}mm and Δ r ∈{15, 20, 25, 30, 35}^∘ one at a time (the other parameter is kept at its lowest value). For each of these parameters, we synthesize 200,000 training image pairs per object using <cit.> and the modifications proposed in sec. <ref>. We then train a network for each object, for each set of parameters, and evaluate each network on our dataset. A subset of the results of this analysis is shown in tab. <ref>. Note that, for the interaction scenario, the “free-hard” sequences (sec. <ref>) were left out since they are much harder than the others and would bias the results. In particular, we show the impact that varying Δ t has on δ_𝐭, as well as that of varying Δ r has on δ_𝐑 for all 3 scenarios. Here, we drop the parentheses for the δ_{𝐭, 𝐑} error metrics for ease of notation (see sec. <ref> for the definitions). The figure reveals a clear trend: increasing Δ r (tab. <ref>-(b)) systematically results in worse performance in rotation. This is especially visible for the high occlusion cases (45% and 60%), where the rotation error δ_𝐑 increases significantly as a function of Δ r. The situation is not so simple when Δ t is increased (tab. <ref>-(a)). Indeed, while increasing Δ t negatively impacts δ_𝐭 in the stability and occlusion scenarios, performance actually improves when the object speed is higher, as seen in the interaction scenario. Therefore, to achieve a good balance between stability and accuracy at higher speeds, a value of Δ t = 30mm seems to be a good trade-off. The remainder of the plots for this analysis, as well as plots evaluating the impact of the resolution of the crop and the size of the bounding box w.r.t the object are shown in the supplementary material.








 §.§ Comparison with previous work


Our trackers yields a 1.7mm/0.6^∘ error on the 4 sequences of <cit.> which is slightly above <cit.> who obtain 0.81mm/0.37^∘. However, as reported in tab. <ref>, more interesting differences between the trackers can be observed when using our new dataset. We compare with object-specific versions of the work of Garon and Lalonde <cit.> as well as the Random Forest approach of Tan et al. <cit.>. For <cit.>, we use the training parameters reported in their paper. For our trackers, the Δ m and Δ r hyper-parameters were obtained with leave-one-out cross-validation to ensure no training/test overlap. As before, the “free-hard” sequences were left out for the interaction experiments. 

Overall, as can be observed in tab. <ref>, the proposed deep learning methods perform either on par or better than the previous work. The “object-specific” networks outperform almost all the other techniques, except for the case of translational error in the interaction scenario. It performs remarkably well at predicting rotations, and is on par with the other methods for translation. In comparison, <cit.> performs well at low occlusions, but fails when the occlusion level is 30% or greater (particularly in rotation). <cit.> shows improved robustness to occlusions, but still achieves high rotation errors at 45% occlusion, and is also much less stable (esp. in rotation) than our “object-specific” networks. Interestingly, our “generic” tracker, which has seen none of these objects in training, performs similarly to the previous works that were trained specifically on these objects. Indeed, it shows a stability, robustness to occlusions and behavior at higher speeds that is similar to <cit.> and <cit.>, demonstrating that learning generic features that are useful for tracking objects can be achieved. Finally, we use the “free-hard” interaction sequences to count the number of times the tracking is lost (sec. <ref>). In this case, the “object-specific” and “generic” networks outperforms the other methods. Qualitative videos showing side-by-side comparisons of these methods are available in the supplementary material.





§ DISCUSSION



The recent evolution in 6-DOF tracking performance on the popular dataset of Choi et al. <cit.> highlights the need for a new dataset containing real data and more challenging scenarios. In this paper, we provide such a dataset, which we hope will spur further research in the field. Our dataset contains 297 sequences containing 11 objects of various shapes and textures. The sequences are grouped into 3 scenarios: stability, occlusion, and interaction. The dataset and companion evaluation code is released publicly[<http://www.jflalonde.ca/projects/6dofObjectTracking>]. Additionally, we build on the framework of <cit.> with an improved architecture and training procedure which allows the network to learn purely from synthetic data, yet generalize well on real data. In addition, the architecture allows for training on multiple objects and test on different objects it has never seen in training. To the best of our knowledge, we are the first to propose such a generic learner for the 6-DOF object tracking task. Finally, our approach is extensively compared with recent work and is shown to achieve better performance. 

A current limitation is that the Vicon markers must be removed in a post-processing step, which may leave some artifacts behind. While the markers are very small (3mm) and the resulting marker-free images have low error (see fig. <ref>), there might still be room for improvement. Finally, our “generic” tracker is promising, but it still does not perform quite as well as “object-specific” models, especially for rotations. In addition, a 3D model of the object is still required at test time, so exploring how this constraint can be removed would make for an exciting future research direction. 






§ ACKNOWLEDGEMENTS

The authors wish to thank Jonathan Gilbert for his help with data acquisition and Sylvain Comtois for the Vicon setup. This work was supported by the NSERC/Creaform Industrial Research Chair on 3D Scanning: CREATION 3D. We gratefully acknowledge the support of Nvidia with the donation of the Tesla K40 and Titan X GPUs used for this research.

splncs






