Probabilistic Regression for Visual Tracking
Martin Danelljan Luc Van Gool Radu Timofte
cs.CV cs.LG









Probabilistic Regression for Visual Tracking
    Martin Danelljan Luc Van Gool Radu Timofte


Computer Vision Lab, D-ITET, ETH Zürich, Switzerland

    
======================================================================================================






	Visual tracking is fundamentally the problem of regressing the state of the target in each video frame. While significant progress has been achieved, trackers are still prone to failures and inaccuracies. It is therefore crucial to represent the uncertainty in the target estimation. Although current prominent paradigms rely on estimating a state-dependent confidence score, this value lacks a clear probabilistic interpretation, complicating its use.

In this work, we therefore propose a probabilistic regression formulation and apply it to tracking. Our network predicts the conditional probability density of the target state given an input image. Crucially, our formulation is capable of modeling label noise stemming from inaccurate annotations and ambiguities in the task. The regression network is trained by minimizing the Kullback-Leibler divergence. When applied for tracking, our formulation not only allows a probabilistic representation of the output, but also substantially improves the performance. Our tracker sets a new state-of-the-art on six datasets, achieving 59.8% AUC on LaSOT and 75.8% Success on TrackingNet.
The code and models are available at .








§ INTRODUCTION



Visual object tracking is the task of estimating the state of a target object in each frame of a video sequence. Most commonly, the state is represented as a bounding box encapsulating the target. Different flavors of the problem arise from  the type of given prior information about the scenario, such as object class <cit.> or static camera <cit.>. In its most general form, however, virtually no prior knowledge is assumed and the initial state of the target is given during inference. This imposes severe challenges, since the method must learn a model of the target during tracking itself.

Along with a myriad of other computer vision tasks, such as object detection <cit.>, pose estimation <cit.>, and keypoint detection <cit.>, visual tracking can fundamentally be formulated as a regression problem. In this general view, the goal is thus to learn a model, typically a deep neural network, capable of predicting the target state in each frame. While current and past approaches apply a variety of techniques to address this problem, most of the successful methods have a crucial aspect in common. Namely, that the task of regressing the target state y^* in a frame x is achieved by learning to predict a confidence value s(y,x) for any given state y. The target state is then estimated by maximizing the predicted confidence y^* = _y s(y,x).




The aforementioned confidence-based regression strategy is shared by the previously dominant Discriminative Correlation Filter (DCF) paradigm <cit.> and the more recent Siamese trackers <cit.>. Both employ a convolutional operation to predict a target confidence s(y,x) at each spatial position y, in order to localize the target. Recent work <cit.> also demonstrated the effectiveness of training a network branch to predict the confidence s(y,x) of the entire target box y to achieve highly accurate bounding box regression. Due to the vast success of these confidence-based regression techniques, we first set out to unify much of the recent progress in visual tracking under this general view.

One distinctive advantage of confidence-based regression is its ability to flexibly represent uncertainties, encoded in the predicted confidence values s(y,x). In contrast, a direct regression strategy y=f(x) forces the network to commit to a single prediction y, providing no other information. However, the confidence value s(y,x) itself has no clear interpretation, since it simply acts as a quantity to be maximized. The range of values and characteristic of the predicted confidence largely depends on the choice of loss and strategy for generating the corresponding pseudo labels for training. This provides severe challenges when designing strategies for estimating and reasoning with the uncertainty in the prediction. Such measures are highly relevant in tracking, to decide whether to update, if the target is lost, or how uncertain the output is (see Figure <ref>). We aim to address these limitations by taking a probabilistic view.

Contributions
We propose a formulation for learning to predict the conditional probability density p(y|x) of the target state y given an input image x. Unlike the confidence value s(y,x), the density p(y|x) has a clear and direct interpretation, allowing the computation of absolute probabilities. We assume no particular family of distributions, such as Gaussian, instead letting p(y|x) be directly parametrized by the network architecture itself. Specifically, the density p(y|x) is represented by the continuous generalization of the SoftMax operation, previously employed in energy-based models <cit.> and recently in <cit.>. In contrast to these previous works, we also model the uncertainty in the annotations themselves. This is shown to be crucial in visual tracking to counter noise in the annotations and ambiguities in the regression task itself. The network is trained by minimizing the Kullback-Leibler divergence between the predicted density and the label distribution.

We demonstrate the effectiveness of our general approach by integrating it into the recent state-of-the-art tracker DiMP <cit.>. Our resulting tracker does not only allow a fully probabilistic representation p(y|x) of the predicted target state. Comprehensive experiments on seven benchmark datasets show that our probabilistic representation and training significantly improves the performance of the tracker. Our Probabilistic DiMP (PrDiMP) outperforms previous state-of-the-art by a large margin, particularly on available large-scale datasets, including LaSOT (+2.9% AUC) and TrackingNet (+1.8% Success).



§ REGRESSION BY CONFIDENCE PREDICTION


In machine learning, regression is fundamentally the problem of learning a mapping f_θ: 𝒳→𝒴 from an input space 𝒳 to a continuous output space 𝒴, given a set of example pairs {(x_i,y_i)}_i ⊂𝒳×𝒴. For our purposes, 𝒳 constitutes the space of images. The most straight-forward take on regression is to directly learn the function f_θ, parametrized as a deep neural network with weights θ, by minimizing a loss L(θ) = ∑_i ℓ(f_θ(x_i), y_i). Here, the function ℓ measures the discrepancy between the prediction f_θ(x_i) and corresponding ground-truth value y_i. While the choice of loss ℓ is highly problem dependent, popular alternatives include the L^p family, ℓ(y, y') = y - y'_p^p.



 §.§ General Formulation

While direct regression is successfully applied for many computer vision problems, including optical flow <cit.> and depth estimation <cit.>, it has proven less suitable for other vision tasks. Examples of the latter include visual tracking <cit.>, object detection <cit.> and human pose estimation <cit.>. In these problems, networks are often trained to instead predict a confidence score, which is then maximized in order to achieve the final estimate. Confidence prediction has prevailed over standard direct regression in these circumstances thanks to two key advantages. First, confidence prediction can capture the presence of uncertainties, multiple hypotheses and ambiguities in the output space 𝒴. The network does not have to commit to a single estimate f_θ(x) = y. Second, the network can more easily exploit symmetries shared by 𝒳 and 𝒴, such as translational invariance in the case of image 2D-coordinate regression tasks, which are particularly suitable for CNNs.


Formally, we define confidence-based regression as learning a function s_θ: 𝒴×𝒳→ that predicts a scalar confidence score s_θ(y,x) ∈ given an output-input pair (y,x). The final estimate f(x) = y^* is obtained by maximizing the confidence to y,

    f(x) = _y∈𝒴 s_θ(y,x)  .

The regression problem is thus transformed to learning the function s_θ from the data {(x_i,y_i)}_i. This is generally performed by defining a function a: 𝒴×𝒴→ for generating a pseudo label a(y,y_i), acting as the ground-truth confidence value for the prediction s_θ(y,x_i). The confidence prediction network can then be trained by minimizing the loss L = ∑_i L(θ; x_i, y_i), where

    L(θ; x_i, y_i) = ∫_𝒴ℓ(s_θ(y,x_i), a(y,y_i))  y  .

The function ℓ : ×→ now instead measures the discrepancy between the predicted confidence value s_θ(y,x_i) and the corresponding label value a(y,y_i). In practice, a variety of losses ℓ and pseudo label functions  a are employed, depending on the task at hand. In the next section, some more popular examples are studied, where our discussion focuses on the visual tracking problem in particular.




 §.§ In Visual Tracking


In visual tracking, the task is to regress the state of the target object in each frame of the video, given its initial location. The state is most often represented as an axis-aligned bounding box y ∈^4. Compared to other vision tasks, this problem is particularly challenging since an example appearance of the target object is only provided at test-time. The tracker must thus learn a model based on this first example in order to locate the object in each frame.

Due to this challenging nature of the problem, the majority of approaches until very recently, focused on regressing the center 2D image coordinate y ∈^2 of the target object, and then optionally using this model to estimate the one-parameter scale factor by a multi-scale search. This class of methods include the widely popular Discriminative Correlation Filter (DCF) approaches <cit.>, most of the more recent Siamese networks <cit.> and other earlier approaches <cit.>. The formulation (<ref>), (<ref>) is explicitly utilized in the theory of Structural SVMs <cit.>, employed in the well-known Struck tracker <cit.>. In DCF-based methods, a convolutional layer is trained online, during tracking, to predict a target confidence score

    s_θ(y,x) = (w_θϕ(x))(y)  .

Here, w_θ is the convolution kernel and ϕ(x) are the features extracted from the image x, typically by a CNN with frozen weights <cit.>. The result of the convolution (<ref>) is evaluated at the spatial location y to obtain the confidence s_θ(y,x). The DCF paradigm adopts a squared loss ℓ(s,a) = (s - a)^2 on the confidence predictions, which enables efficient optimization of (<ref>) w_θ in the Fourier domain <cit.>. Nearly all DCF methods employ a Gaussian confidence pseudo label a(y,y_i) = e^-y - y_i^2/2σ^2 centered at the target position y_i in frame x_i.

In contrast to DCF, Siamese trackers <cit.> aim to fully learn the parameters θ of the network in an offline training stage. This is performed by learning an embedding space ϕ_θ in which similarities between a target template z and frame x can be computed as a correlation,

    s_θ(y,x) = (ϕ_θ(z) ϕ_θ(x))(y)  .

Siamese methods often employ a binary cross entropy loss 

    ℓ(s,a) = alog(1 + e^-s) + (1-a)log(1 + e^s)

in (<ref>) to train the network parameters θ. That is, target localization is treated as a dense binary classification problem, where the pseudo label a(y,y_i) ∈ [0,1] represents the target/background class, or more generally, a Bernoulli distribution. It is commonly set to a(y,y_i) = 1 in the target vicinity y - y_i < r and a(y,y_i) = 0 otherwise <cit.>.

To achieve an accurate prediction of the full target bounding box, a few recent trackers <cit.> have achieved remarkable performance by separating the tracking problem into two parts. First, the object is coarsely localized using techniques reminiscent of the aforementioned approaches, that are robust to similar background objects, clutter and occlusions. In the second stage, a separate network branch is employed for regressing the target bounding box. 
For this purpose, the ATOM tracker <cit.> employs an IoU-Net <cit.> based network head s_θ(y,x), which scores any input bounding-box y∈^4. This head is trained in an offline learning stage to predict the Intersection-over-Union (IoU) overlap a(y,y_i) = IoU(y,y_i) using the squared error ℓ(s,a) = (s - a)^2 in (<ref>). In this case, the integral (<ref>) is approximated by sampling bounding boxes during training. During tracking, the optimal box (<ref>) is achieved by gradient-based maximization of the predicted confidence. 

More recently, Bhat  <cit.> proposed the DiMP tracker by designing a meta-learning based network architecture which predicts discriminative target model weights w_θ = ψ_θ({(ϕ_θ(z_j), y_j)}_j) in (<ref>) from a set of sample pairs {(z_j, y_j)}_j. The predicted weights are then employed for the first-stage robust target localization, and updated during tracking through a learned recurrent optimization process. The target model predictor ψ_θ is learned end-to-end using a robust version of the squared error ℓ and Gaussian confidence labels a(y,y_i). For the second stage, it adopts the bounding-box regression technique proposed in ATOM.




§ METHOD



We propose a probabilistic regression model that integrates all advantages of confidence-based regression. However, unlike the aforementioned confidence-based models, our approach generates a predictive probability distribution p(y|x_i, θ) as output. The network is trained by minimizing the KL divergence between the predictive density p(y|x, θ) and the conditional ground-truth distribution p(y|y_i), which models label noise and ambiguities in the task itself. During inference, a point estimate of the regressed value is obtained by maximizing the predicted density.

Our approach possesses a few important advantages compared to the confidence-based regression methods.
In the latter, the prediction s_θ(y,x) can be difficult to interpret, and its value largely depend on the pseudo label function a and employed loss ℓ. In contrast, the probabilistic nature of our method allows reasoning about the uncertainty in the output. Moreover, in our approach, the pseudo label function a is replaced by the label-conditional distribution p(y|y_i), which models noise and uncertainty in the annotation y_i. Lastly, in contrast to confidence-based regression, our approach does not require a choice of loss ℓ. Instead, we directly minimize the Kullback-Leibler (KL) divergence between the predictive distribution and the ground-truth. Next, we provide a general formulation of the proposed regression model, and apply it to tracking in Section <ref>.





 §.§ Representation

In this section, we formulate an approach for effectively training the network to predict a probability distribution p(y|x, θ) of the output y given the input x. The density itself is represented using the formulation previously employed in probabilistic energy-based deep learning <cit.> and the recent deep conditional target densities <cit.>,

    p(y|x, θ) = 1/Z_θ(x) e^s_θ(y,x) ,  Z_θ(x) = ∫ e^s_θ(y,x) y  .

As for the confidence-based methods described in Section <ref>, s_θ: 𝒴×𝒳→ is a deep neural network mapping the output-input pair (y,x) to a scalar value. The expression (<ref>) converts this value to a probability density by exponentiation and division by the normalizing constant Z_θ(x). In fact, it should be noted that (<ref>) is a direct generalization of the SoftMax operation to an arbitrary output space 𝒴. 

Since the output of the network represents a probability density over 𝒴, we can learn the network parameters θ by applying techniques for fitting a probability distribution to data. Given training sample pairs {(x_i, y_i)}_i, the simplest approach is to minimize the negative log-likelihood,

    -log p(y_i|x_i, θ) = log(∫ e^s_θ(y,x_i) y) - s_θ(y_i,x_i)  .

This strategy was recently successfully applied for a number of computer vision tasks <cit.>, including bounding-box regression in visual tracking.
One advantage of the negative log-likelihood loss (<ref>) is that it only employs the training sample (x_i,y_i) itself, without further assumptions. However, this brings an important limitation, discussed next.





 §.§ Label Uncertainty and Learning Objective



Compared to the negative log-likelihood loss (<ref>), the confidence-based paradigm described in Section <ref> enjoys a certain flexibility stemming from the pseudo label function a(y,y_i). In practice, the design of a(y,y_i) has been shown to be critical for tracking performance <cit.>. We believe that this is mostly due to the inherent ambiguity of the task and the uncertainty in the label y_i itself.
Most tracking approaches focus on regressing the center coordinate y ∈^2 of the target in the image. However, for most objects, this is an inherently ambiguous and ill-defined task. While the center coordinate can be defined as the center of mass of the target bounding box, this is hardly a visually intuitive definition for a human, or a tracking algorithm for that matter.

Consider the example in Figure <ref>. When the target dog in the video raises its tail, the center of mass changes radically and ends up at a background pixel. On the other hand, the appearance and location of the object is almost unchanged. The tracker would thus naturally predict a similar target center location as before. This demonstrates that the definition of the target center is largely ambiguous and that the center of mass is often confusing for the tracker. The pseudo label function a(y,y_i) can encapsulate this ambiguity by having a wider high-confidence peak, which has been shown beneficial for training tracking models <cit.>.
Another source of uncertainty is label noise. Accurate bounding box annotation is a difficult task, especially in the presence of occlusions, motion blur, and for small objects, as shown in Figure <ref>. In other words, multiple annotators would naturally disagree for a given object, with some level of variation. This variation, or noise, in the annotation is most often ignored when training the network.

We propose to probabilistically model label noise and task ambiguities for the regression problem as a conditional ground-truth distribution p(y|y_i). It characterizes the probability density of the ground-truth output value y, given the annotation y_i. Instead of the negative log-likelihood (<ref>), we train the network to minimize the KL divergence to p(y|y_i),

    (p(·|y_i),p(·|x_i,θ)) = ∫ p(y|y_i) logp(y|y_i)/p(y|x_i,θ) y 
       ∼log(∫ e^s_θ(y,x_i) y) - ∫ s_θ(y,x_i) p(y|y_i)   y  .

Here, ∼ denotes equality up to a constant term. The second line in (<ref>) corresponds to the cross entropy between the two distributions and the discarded constant term is the negative entropy ∫ p(y|y_i) log p(y|y_i)  y of the label distribution. See Appendix <ref> for a detailed derivation.

The loss (<ref>) naturally integrates information about the uncertainty p(y|y_i) in the annotated sample (x_i,y_i). Unlike the pseudo label function a(y|y_i) employed in confidence-based regression, p(y|y_i) has a clear interpretation as a probability distribution. In fact, p(y|y_i) could be empirically estimated by obtaining multiple annotations for a small subset of the data. In the case of a Gaussian model p(y|y_i) = (y|y_i,σ^2), the variance σ^2 can be estimated as the empirical variance of these annotations. In this work, we simply consider σ^2 a hyper-parameter.



 §.§ Training



In this section, we consider strategies for training the network parameters θ based on the loss (<ref>). In practice, this requires approximating the two integrals in (<ref>). We consider two techniques for this purpose, namely grid sampling and Monte Carlo integration with importance sampling. 

Grid Sampling
For 2D image coordinate regression problems, the case of regressing the center of the tracked target, y ∈𝒴⊂^2 represents a location in the image. In this case, translational invariance is efficiently exploited by parametrizing s_θ(y,x) = f_θ(x)(y), where f_θ is a Convolutional Neural Network (CNN). s_θ(y,x) is thus obtained by evaluating the output of the CNN at image coordinate y. Let {y^(k)}_k=1^K ⊂𝒴 be the set of uniform grid locations evaluated by the CNN f_θ(x) when applied to an image sample x. Further, let A be the area of a single grid cell. The uniform grid sampling automatically provided by the CNN generates the following approximation of the loss (<ref>),

    L_i  = log(A∑_k=1^K e^s_θ(y^(k),x_i)) - A∑_k=1^K s_θ(y^(k),x_i) p(y^(k)|y_i).

The final loss is then obtained by averaging L_i over all samples i in the mini-batch. 

Monte Carlo Integration
For the more general regression problems, grid sampling does not necessarily provide any computational benefits. On the contrary, it scales poorly to higher dimensions and may induce a sampling bias due to the rigid grid. In the more general case, we therefore adopt the Monte Carlo (MC) based sampling strategy proposed in <cit.>. Specifically, we draw samples y_i^(k)∼ q(y|y_i) from a proposal distribution q(y|y_i) during training. The same samples are employed to approximate both integrals in (<ref>),

    L_i =log(1/K∑_k=1^Ke^s_θ(y_i^(k),x_i)/q(y_i^(k)|y_i)) - 1/K∑_k=1^K s_θ(y_i^(k),x_i) p(y_i^(k)|y_i)/q(y_i^(k)|y_i).

To accurately approximate the original loss (<ref>), the proposal distribution q(y|y_i) should ideally cover the label distribution p(y|y_i) as well as regions with high predicted density p(y|x_i,θ). In <cit.> it was shown that a simple Gaussian mixture centered at the annotation y_i sufficed for a variety of tasks, including bounding box regression.

The loss (<ref>) requires multiple evaluations of the network s_θ(y_i^(k),x_i). In practice, however, computer vision architectures popularly employ deep backbone feature extractors ϕ_θ(x), such as ResNet <cit.>, generating a powerful representation of the image. The output value y can be fused at a late stage, such that s_θ(y,x) = f_θ(y, ϕ_θ(x)). This allows the computationally demanding feature extraction ϕ_θ(x_i) to be shared among all samples y_i^(k). Specifically for our purpose, such architectures have been successfully employed for bounding box regression in object detection and visual tracking problems <cit.>. 



§ TRACKING APPROACH



We apply the general probabilistic regression formulation introduced in Section <ref> for the challenging and diverse task of visual target tracking.



 §.§ Baseline Tracker: DiMP


We employ the recent state-of-the-art tracker DiMP <cit.> as our baseline. As briefly discussed in Section <ref>, the DiMP model contains two output branches.

Target Center Regression (TCR)
The center regression branch aims to coarsely localize the target in the image by only regressing its center coordinate. This branch emphasizes robustness over accuracy. It consists of a linear convolutional output layer, who's weights w_θ are predicted by the network as an unrolled optimization process that minimizes an L^2-based discriminative learning loss. This allows the tracker to robustly differentiate the target object from similar objects in the background. The target center confidence at location y^tc∈^2 in frame x is predicted similarly to (<ref>), s_θ^tc(y^tc,x) = (w_θϕ_θ(x))(y^tc), where ϕ_θ is the backbone feature extractor. This branch is trained in a meta-learning setting, with a confidence-based objective (<ref>) using Gaussian pseudo labels a^tc and a robust L^2 loss,

    ℓ(s,a) = 
    (s - a)^2  ,     a > T 
    max(0, s)^2  ,     a ≤ T
     .

During tracking, the target center is regressed by densely computing confidence scores s_θ^tc(y^tc,x) within a wide search region in the frame x. We refer to <cit.> for details.

Bounding Box Regression (BBR)
The BBR branch adopts the target conditional IoU-Net-based <cit.> architecture proposed in <cit.>. As discussed in Section <ref>, this branch predicts a confidence score s_θ^bb(y^bb,x) for a given box . It is trained using the bounding box IoU as pseudo label a^bb(y^bb,y^bb_i) and the standard L^2 loss ℓ in (<ref>). During tracking, the BBR branch is applied to fit an accurate bounding box to the target using gradient-based maximization of s_θ^bb(y^bb,x) y^bb. We refer to <cit.> for details.






 §.§ Our Tracker: Probabilistic DiMP


We introduce a tracking approach with fully probabilistic output representations, obtained by integrating our regression formulation into both branches of the baseline DiMP. Example predicted densities are visualized in Figure <ref>.

Target Center Regression
We represent the predicted distribution of the target center coordinate p(y^tc|x,θ) by applying (<ref>) to the network output s_θ^tc(y^tc,x). Since this branch is fully convolutional, we approximate the KL-divergence loss (<ref>) for training using the grid sampling strategy (<ref>). The conditional ground-truth density is set to a Gaussian p(y^tc|y^tc_i)=(y^tc;y^tc_i,σ_tc^2) with the same variance parameter σ_tc^2 used for the corresponding pseudo label function a^tc in the baseline DiMP (1/4th of the target size). For the optimization module, which predicts the convolution weights w_θ for the center regression branch, we use the KL-divergence loss (<ref>) with an added L^2 regularization term. We modify the steepest descent based architecture <cit.> to employ a second order Taylor expansion, since the original Gauss-Newton approximation is limited to least-squares objectives. Our approach benefits from the fact that the resulting objective (<ref>) is convex in w_θ for the linear predictor s_θ^tc(y^tc,x) = (w_θϕ_θ(x))(y^tc), and thanks to efficient analytic expressions of the gradient and Hessian. See Appendix <ref> for a detailed description of the optimizer module.

Bounding Box Regression
We use the same architecture s_θ^bb(y^bb,x) as in <cit.> and apply it in our probabilistic formulation (<ref>). We follow the work of <cit.>, which extended the same ATOM BBR module <cit.> to the probabilistic setting using the negative log-likelihood loss (<ref>) and an MC-based approximation. In this work, we further integrate the label distribution p(y^bb|y^bb_i) to model the noise and uncertainty in the bounding box annotations, and minimize the KL-divergence (<ref>) using MC sampling (<ref>). Specifically, we use an isotropic Gaussian distribution p(y^bb|y^bb_i) = (y^bb;y^bb_i,σ_bb^2) and set σ_bb = 0.05. For a fair comparison, we use the same proposal distribution q(y^bb|y^bb_i)=1/2(y^bb;y^bb_i,0.05^2)+1/2(y^bb;y^bb_i,0.5^2) and bounding box parametrization as in <cit.>. 

Details
Our entire network is trained jointly end-to-end using the same strategy and settings as for the original DiMP <cit.>, by integrating it into the publicly available  framework <cit.>. The training splits of the LaSOT <cit.>, GOT10k <cit.>, TrackingNet <cit.> and COCO <cit.> are used, running 50 epochs with 1000 iterations each. We also preserve the tracking procedure and settings in DiMP, only performing minimal changes, which are forced by the probabilistic output representation provided by our model. Due to different scaling of the network output, we accordingly change the threshold for which the target is reported missing and the gradient step length used for bounding box regression. We refer to <cit.> for detailed description of training and inference settings. Our code is available at  <cit.>.


§ EXPERIMENTS



We perform a detailed analysis of our approach along with comprehensive comparisons with state-of-the-art on seven datasets. The ResNet-18 and 50 versions of our tracker operates at about 40 and 30 FPS respectively. 




 §.§ Comparison of Regression Models



We first analyze the impact of different regression formulations for tracking. In each case, we train a separate version of the DiMP baseline, with the ResNet-18 backbone. We compare four different approaches. L2: Standard squared loss, used in the baseline DiMP for Bounding Box Regression (BBR). R-L2: The Robust L2 loss (<ref>), employed in the baseline DiMP for the Target Center Regression (TCR) branch. NLL: The probabilistic Negative Log-Likelihood formulation (<ref>) proposed in <cit.>. Ours: Trained using the KL-divergence (<ref>) as described in Section <ref>. 

Following <cit.>, we perform this analysis of our approach on the combined OTB-100 <cit.>, UAV123 <cit.> and NFS <cit.> dataset, totaling 323 diverse videos, and report the average over 5 runs. We report the Overlap Precision OP_T, percentage of frames with an IoU overlap larger than T, along with the main metric AUC=∫_0^1 OP_T (see <cit.>).




Complete Tracker
We first analyze the performance when each regression model is applied to the entire tracker, for both the BBR and TCR branch. The results are reported in the left section of Table <ref>. The R-L2, which corresponds to standard DiMP, improves 0.8% AUC over the standard L2 loss. Our model outperforms the R-L2 baseline with a gain of 1.7% in AUC. 

Bounding Box Regression (BBR)
We exclusively analyze the impact of each model for BBR by using the baseline DiMP R-L2 formulation for TCR in all cases. Thus, only the BBR branch is affected. Results are provided in the center section of Table <ref>. The baseline <cit.>, employing L2 loss to predict IoU, achieves 63.8 AUC. The NLL formulation <cit.> gives a substantial 1.2% gain. By modeling uncertainty in the annotation and using the KL loss (<ref>), our approach achieves an additional improvement of 0.8%.

Target Center Regression (TCR)
Similarly, we compare the models for TCR by employing our approach for the BBR branch in all cases. The results, reported in the right section of Table <ref>, show that probabilistic NLL is not suitable for TCR. This is likely due to the inherent ambiguity of the problem, which is not accounted for. By explicitly modeling label uncertainty p(y|y_i), our formulation achieves a 2.3% gain in AUC, further outperforming the standard L2 model. The R-L2 model, which is specifically designed for TCR <cit.>, achieves a marginally better performance. However, note that this result is achieved when combined with our formulation for BBR.

In conclusion, our probabilistic formulation outperforms the baseline DiMP, employing L2 and Robust L2 for TCR and BBR respectively, by a large 1.7% margin in AUC, while achieving the best overall performance for all regression models. In the subsequent experiments, we employ our formulation for both the BBR and TCR branch.






 §.§ Analysis of Label Uncertainty



In this section, we further analyze the importance of our ability to model the uncertainty p(y|y_i) in the annotation y_i. This is probed by investigating the impact of the standard deviation parameter σ for our Gaussian model of the label noise p(y|y_i) = (y;y_i,σ^2) (see Section <ref>). This is individually performed for the bounding box (σ_bb) and target center (σ_tc) regression branch. We use the same experiment setup as described in the previous section, reporting the average of five runs over the combined OTB100-NFS-UAV123 dataset, and using the ResNet-18 backbone.

Figure <ref> shows the tracking performance in AUC when varying the standard deviations σ_bb and σ_tc over a wide range of values. The baseline DiMP-18 performance is shown in red for reference. A similar overall trend is observed in both cases. A too large standard deviation σ leads to poor results, since the over-estimation in the label uncertainty forces the tracker to output too uncertain predictions. However, a small σ causes overfitting and over-confident predictions, leading to sub-optimal performance. Properly modeling the label uncertainty is thus critical for visual tracking, due to the ambiguities in the regression tasks and annotation noise. We also note that by outperforming the baseline DiMP over a wide range of σ_bb and σ_tc values, our approach is not sensitive to specific settings.



 §.§ State-of-the-Art



We compare our approach, termed PrDiMP, on seven tracking benchmarks. We evaluate two versions, PrDiMP18 and PrDiMP50, employing ResNet-18 and ResNet-50 respectively as backbone. The same settings and parameters are used for one-pass evaluation on all datasets. To ensure the significance in the results, we report the average over 5 runs for all datasets, unless the specific protocol requires otherwise. Additional results are provided in Appendix <ref>.

LaSOT <cit.>
We first compare on the large-scale LaSOT dataset. The test set contains 280 long videos (2500 frames in average), thus emphasizing the robustness of the tracker, along with its accuracy. The success plot in Figure <ref>, showing the overlap precision OP_T as a function of the threshold T, is computed as the average over 5 runs. Trackers are ranked their AUC score, shown in the legend. Our approach outperforms the previous best tracker, DiMP, by a large margin of 3.2% and 2.9% in AUC when using ResNet-18 and ResNet-50 respectively. The improvement in  OP_T is most prominent for T>0.3, demonstrating superior accuracy achieved by our bounding box regression.

TrackingNet <cit.>
This is a large-scale tracking dataset with high variety in terms of classes and scenarios. The test set contains over 500 videos without publicly available ground-truth. The results, shown in Table <ref>, are obtained through an online evaluation server. Both our versions outperform all previous approaches in the main Success (AUC) metric by a significant margin. With the same ResNet-50 backbone, our approach achieves a gain of 2.5% and 1.8% in AUC over SiamRPN++ <cit.> and DiMP <cit.> respectively.

VOT2018 <cit.>
Next, we evaluate on the 2018 edition of the Visual Object Tracking challenge. We compare with the top methods in the challenge <cit.>, as well as more recent methods. The dataset contains 60 videos. Trackers are restarted at failure by the evaluation system. The performance is then decomposed into accuracy and robustness, defined using IoU overlap and failure rate respectively. The main EAO metric takes both these aspects into account. The results, computed over 15 repetitions as specified in the protocol, are shown in Table <ref>. Our PrDiMP50 achieves the best overall performance, with the highest accuracy and competitive robustness compared to previous methods.




GOT10k <cit.>
This dataset contains 10,000 sequences for training and 180 for testing. We follow the defined protocol <cit.> and only train on the specified GOT10k training set for this experiments, while keeping all other settings the same. By having no overlap in object classes between training and testing, GOT10k also benchmarks the generalizability of the tracker to novel objects. The results in Table <ref>, obtained through a evaluation server, are reported in terms of SR_T and AO, which are equivalent to OP_T and AUC <cit.>. Compared to DiMP, our method achieve large gains of 3.3% and 2.3% in terms of the main AO metric, when using ResNet-18 and -50 respectively.

UAV123 <cit.>
This challenging dataset, containing 123 videos, is designed to benchmark trackers for UAV applications. It features small objects, fast motions, and distractor objects. The results are shown in Figure <ref>, where OP_T is plotted over IoU thresholds T and AUC is shown in the legend. ATOM <cit.> and DiMP50 obtain 64.2% and 65.3% respectively. Our PrDiMP50 achieves a remarkable 68.0%, surpassing previous methods by a significant margin.

OTB-100 <cit.>
For reference, we report results on the OTB-100 dataset. While this dataset has served an important role in the development of trackers since its release, it is known to have become highly saturated over recent years, as shown in Table <ref>. Still, our approach performs similarly to the top correlation filter methods, such as UPDT <cit.>, while on par with the end-to-end trained SiamRPN++.

NFS <cit.>
Lastly, we report results on the 30 FPS version of the Need for Speed (NFS) dataset, containing fast motions and challenging distractors. As shown in Table <ref>, our approach achieves a substantial improvement over the previous state-of-the-art on this dataset.





§ CONCLUSIONS


We propose a probabilistic regression formulation, where the network is trained to predict the conditional density p(y|x,θ) of the output y given the input x. The density is parametrized by the architecture itself, allowing the representation of highly flexible distributions. The network is trained by minimizing the KL-divergence to the label distribution p(y|y_i), which is introduced to model annotation noise and task ambiguities. When applied for the tracking task, our approach outperforms the baseline DiMP <cit.> and sets an new state-of-the-art on six datasets.

Acknowledgments:
This work was partly supported by the ETH Zürich Fund (OK), a Huawei Technologies Oy (Finland) project, an Amazon AWS grant, and Nvidia.

ieee_fullname











	Appendix



This Appendix provides detailed derivations and results.
Appendix <ref> derive the employed loss in (<ref>) from the KL divergence. We then provide details about the target center regression module in Appendix <ref>. Lastly, we report more detailed results in Appendix <ref> and provide additional visualizations of the predicted distributions in Appendix <ref>.



§ DERIVATION OF KL DIVERGENCE LOSS


Here, we derive the loss (<ref>) from the KL divergence between the predicted distribution p(y|x_i,θ) and the ground-truth density p(y|y_i). Starting from the definition of the KL divergence and inserting our formulation (<ref>), we achieve,

    (p(·|y_i),p(·|x_i,θ)) = ∫ p(y|y_i) logp(y|y_i)/p(y|x_i,θ) y 
        = ∫ p(y|y_i) logp(y|y_i)/e^s_θ(y,x_i) / Z_θ(x_i) y 
        = ∫ p(y|y_i) ( log p(y|y_i) - log e^s_θ(y,x_i) + log Z_θ(x_i) )  y 
        = ∫ p(y|y_i) log p(y|y_i)  y 
                + log Z_θ(x_i) - ∫ p(y|y_i) s_θ(y,x)  y 
       ∼log(∫ e^s_θ(y,x_i) y) - ∫ s_θ(y,x_i) p(y|y_i)   y  .

In the last row, we have discarded the first term (the negative entropy of p(y|y_i)) and substituted the definition of the partition function Z_θ(x_i) from (<ref>).



§ TARGET CENTER REGRESSION MODULE



In this section, we give a detailed description and derivation of the optimization module employed for target center regression in our PrDiMP tracker. 
The goal of the optimizer module is to predict the weights w_θ of the target center regression component,

    s_θ^tc(y^tc,x) = (w_θϕ_θ(x))(y^tc)  .

Here, x is an input image, y^tc∈^2 is an image coordinate and ϕ_θ is the backbone feature extractor (see Section <ref>). The weights w_θ are learned from a set of training (support) images {z_j}_j=1^n and corresponding target bounding box annotations {ỹ_j^bb}_j=1^n. As in the baseline DiMP <cit.>, these images are sampled from an interval within each sequence during training. During tracking, the images z_j are obtain by performing augmentations on the first frame, and by gradual update of the memory.

In DiMP, the optimizer module is derived by applying the steepest descent algorithm to a least-squares objective. In our case however, the employed loss function does not admit a least-squares formulation. In this work, we therefore replace the Gauss-Newton approximation with the quadratic Newton approximation in order to compute the step-length necessary for the steepest descent algorithm. We derive closed form solutions of all operations, ensuring simple integration of the optimizer module as a series of deep neural network layers.

Similarly to our offline training objective, we let the optimizer module minimize the KL-divergence based learning loss (<ref>). We also add an L^2 regularization term to benefit generalization to unseen frames. The loss is thus formulated as follows,

    L(w_θ) = ∑_j=1^nγ_j L_CE(z̃_j  w_θ;p_j) + λ/2w_θ^2  .

The non-negative scalars λ and γ_j control the impact of the regularization term and sample z_j respectively. We also make the following definitions for convenience,



0.2em
rl@l@
z̃_j    = ϕ_θ(z_j)     Extracted image features.  

p_j^(k)    = p(y^tc,(k)|y_j^tc)     Label density at location k.  

s_j^(k)     = (z̃_j  w_θ)(y^tc,(k))     Target scores at location k.  

p̂_j^(k)    = exp(s_j^(k))/∑_l=1^K exp(s_j^(l))     Spatial SoftMax of  s_j . 


Note that we use superscript k ∈{1,…,K} to denote spatial grid location y^tc,(k)∈^2. In the following, the quantities in (<ref>)-(<ref>) are either seen as vectors in ^K or 2D-maps ^H× W (with K=HW), as made clear from the context. 

In (<ref>), the per-sample loss L_CE is the grid approximation (<ref>) of the original KL-divergence objective. Without loss of generality, we may assume A=1, obtaining

    L_CE(s;p)    = log( ∑_k=1^K e^s^(k)) - ∑_k=1^K s^(k) p^(k)
       = log(1 e^s) - p s .

Here, 1=[1, …, 1] denotes a vector of ones. Note that the grid approximation thus corresponds to the SoftMax-Cross Entropy loss, commonly employed for classification. 






To derive the optimization module, we adopt the steepest descent formulation <cit.>, but employ the Newton approximation discussed above. This results in the following optimization strategy,



    w_θ^(i+1)   = w_θ^(i) - α^(i)∇ L(w_θ^(i)) 
    α^(i)   = ∇ L(w_θ^(i))∇ L(w_θ^(i))/∇ L(w_θ^(i)) H(w_θ^(i)) ∇ L(w_θ^(i)) .


Here, ∇ L(w_θ^(i)) and H(w_θ^(i)) is the gradient and Hessian of L (<ref>), evaluated at the current estimate w_θ^(i) of the weights. To implement (<ref>), we efficiently compute these quantities by deriving closed-form expressions of both. 

First, we can first easily compute the gradient and hessian of (<ref>) s as,

	 

    ∇_s L_CE(s;p)    = p̂ - p 
    ∂^2/∂ s^2 L_CE(s;p)    = (p̂) - p̂p̂


Here, p̂ is the SoftMax of s as defined in (<ref>). By applying (<ref>) together with the chain rule, we can compute the gradient of (<ref>) as,

    ∇ L(w_θ)    = ∑_j=1^nγ_j [∂ s_j/∂ w_θ]∇_s L_CE(s_j;p_j) + λ w_θ
       = ∑_j=1^nγ_j z̃_j(p̂_j - p_j) + λ w_θ .

We denote the transpose of the linear convolution operator w ↦z̃ w as z̃, which corresponds to the transpose of the Jacobian ∂ s_j/∂ w_θ=z̃. By another differentiation w_θ, using (<ref>), the chain rule and the linearity of s_j in w_θ, we obtain the Hessian of (<ref>) as

    H(   w_θ) =  ∂^2/∂ w_θ^2 L(w_θ) 
       = ∑_j=1^nγ_j [∂ s_j/∂ w_θ][∂/∂^2 s L_CE(s_j;p_j)] [∂ s_j/∂ w_θ] + λ I 
       = ∑_j=1^nγ_j z̃_j((p̂_j) - p̂_jp̂_j)z̃_j + λ I  .

Here, I denotes the identity matrix. We further obtain a simple expression of the denominator of the step-length in (<ref>) by evaluating the product,

    g   H(w_θ)g = 
       = ∑_j=1^nγ_j (z̃_j  g) ((p̂_j) - p̂_jp̂_j) (z̃_j  g) + λ g g 
       = ∑_j=1^nγ_j v_j(p̂_j · (v_j - p̂_j v_j)) + λ g g  ,   v_j = z̃_j  g  .

Here, · denotes element-wise multiplication. We summarize the full optimization module in Algorithm <ref>.



§ DETAILED RESULTS



We provide more detailed results from the state-of-the-art comparison performed in Section <ref>.






 §.§ LaSOT

In addition to the success plot shown in Section <ref>, we here provide the normalized precision plot over the LaSOT <cit.> test set, containing 280 videos. The normalized precision score NPr_D is computed as the percentage of frames where the normalized distance (relative to the target size) between the predicted and ground-truth target center location is less than a threshold D. NPr_D is plotted over a range of thresholds D ∈ [0,0.5]. The trackers are ranked using the area under this curve, which is shown in the legend (see <cit.> for details). The normalized precision plot is shown in Figure <ref>. For the ResNet-18 and 50 versions of our PrDiMP, we report the average result over 5 runs. We compare with state-of-the-art trackers DiMP <cit.>, ATOM <cit.>, SiamRPN++ <cit.>, MDNet <cit.>, VITAL <cit.>, and SiamFC <cit.>. Our approach outperforms previous state-of-the-art by a large margin. Compared to the ResNet-50 based SiamRPN++ <cit.> and DiMP-50 <cit.>, our PrDiMP50 version achieves absolute gains of 11.9% and 3.8% respectively. As demonstrated by the plot, this gain is obtained by an improvement in both accuracy (small thresholds) and robustness (large thresholds).




 §.§ GOT10k

The success plot over the 180 videos in the GOT10k <cit.> test set is shown in Figure <ref>. It displays the overlap precision OP_T as a function of the IoU threshold T. Overlap precision OP_T itself is defined as the percentage of frames with a ground-truth IoU overlap larger than a threshold T. The final area-under-the-curve (AUC) score is shown in the legend. We compare our approach with trackers with available results: DiMP <cit.>, ATOM <cit.>, SiamFCv2 (CFNet) <cit.>, SiamFC <cit.>, GOTRUN <cit.>, CCOT <cit.>, ECO <cit.>, and CF2 (HCF) <cit.>. Our PrDiMP versions outperform previous methods, in particular for large overlap thresholds. This demonstrates the superior accuracy of our probabilistic bounding box regression formulation. 



 §.§ OTB-100

We provide the success plot over the 100 videos in the OTB dataset <cit.> in Figure <ref>. We compare with state-of-the-art trackers UPDT <cit.>, SiamRPN++ <cit.>, ECO <cit.>, DiMP <cit.>, CCOT <cit.>, MDNet <cit.>, ATOM <cit.>, and DaSiamRPN <cit.>. Despite the highly saturated nature of this dataset, our tracker performs among the top methods, providing a significant gain over the baseline DiMP.



 §.§ NFS

Figure <ref> show the success plot over the 30 FPS version of the challenging NFS dataset <cit.>. We compare with top trackers with available results:  DiMP <cit.>, ATOM <cit.>, UPDT <cit.>, CCOT <cit.>, ECO <cit.>, and MDNet <cit.>. In this case, both our ResNet-18 and 50 versions achieve similar results, significantly outperforming the previous state-of-the-art DiMP-50.







§ VISUALIZATION OF PREDICTED DISTRIBUTIONS



In this section, we provide additional visualizations of the predicted probability distributions for target center and bounding box regression branches in our tracker. We visualize the output as in Figure <ref>. Several example frames for two challenging sequences are shown in Figure <ref>. As during standard tracking, the predicted distribution p(y^tc|x,θ) for the target center regression (second column) is computed by applying the fully convolutional center regression branch on the search region centered at the previous target location.

To visualize the probability distribution p(y^bb|x,θ) predicted by the bounding box regression branch, we evaluate the density in a grid. Note that the bounding box y^bb∈^4 is 4-dimensional, and we therefore cannot visualize the full distribution as a 2-dimensional heatmap. We therefore plot two slices of this density, showing the variation in the bounding box location and size as follows. Our bounding box is parametrized as y^bb = (c_x/w_0, c_y/h_0, log w, log h), where (c_x,c_y) is the center position, (w,h) is the size (width and height), and (w_0,h_0) is a constant reference size. The latter is set to the current estimate of the target size. The distribution of the bounding box center (third column in Figure <ref>) is obtained by predicting the density value p(y^bb|x,θ) ∼exp(s_θ^bb(y^bb,x)) in a dense grid of center coordinates (c_x,c_y), while keeping the size (w,h) constant at the current target estimate. To visualize the distribution over the bounding box size (fourth column), we conversely evaluate p(y^bb|x,θ) ∼exp(s_θ^bb(y^bb,x)) in a dense grid of log-size coordinates (log w, log h), while keeping the box centered at the current target estimate (c_x,c_y). 

In Figure <ref>, the target center density is visualized in the range ± 2.5√(wh) relative to the previous target location. The bounding box center density is plotted within the range c_x ± w and c_y ± h. The distribution over the bounding box size is plotted from 1/3 to 3 times the estimated target size (w,h), ±log 3. Outputs for two challenging sequences are visualized in Figure <ref>. The left part shows a cat and its mirror image. Due to their similarity and proximity, it is in many frames difficult to predict the exact bounding box of the cat. In these cases, the predicted distribution captures this uncertainty. For example, in the fourth row, two clear modes are predicted, which correspond to aligning the box with the real cat or with the right edge of the reflection. Moreover, the last row shows a failure case, where the box briefly expands. However, note that the size probability distribution (right column) is highly uncertain. This information could thus be used to indicate low reliability of the estimated bounding box size.

The right part of Figure <ref> depicts a challenging sequence with multiple distractors. The target center regression, which has a wider view of the scene, captures the presence of distractors in uncertain cases. In the last row, the tracker briefly fails by jumping to a distractor object. However, there is a strong secondary mode in the target center regression distribution that indicates the true target. Thus, the estimated distribution accurately captures the uncertainty in this ambiguous case. Moreover, in row 4-7, the target object is small with many nearby similar-looking objects. This makes bounding box regression extremely hard, even for a human. Our network can predict flexible distributions reflecting meaningful uncertainties for both bounding box position and size, when encountered with these difficulties. 



