\section{Introduction}

Similar to the role of object detection in image-based visual analysis, object tracking is an essential component for visual understanding in videos~\cite{tapaswi2019video}. Its outputs are subsequently used in visual attribute analysis~\cite{kanade2000comprehensive,shen2017learning} and instance recognition~\cite{deng2019arcface,li2014deepreid}. 
The major goals of a multi-object tracker are: 1) to distinguish different objects into different tracks; 2) to keep the same object in a single track. 
Despite the rapid advance of generic object tracking algorithms ~\cite{bergmann2019tracking,feichtenhofer2017detect,zhang2016tracking}, designing a multi-object tracking approach to achieve these goals still remains a difficult problem.
Besides, due to the strict latency requirement of the video processing task, an multi-object tracking approach is also expected to have: 1) fast and  stable runtime speed; 2) accurate and robust localization accuracy. 3) The capability for online processing. 

\begin{figure}[t]
      \centering
      \includegraphics[width=1\linewidth]{figures/eccv_fig/teaser_eccv.png}
      \centering
      \caption{The proposed SMOT  consists of two stages. The first stage generates temporally consecutive tracklets by exploring the temporal and spatial correlations from previous frame. The second stage performs online linking of the tracklets to generate a face track for each person (better view in color).}
      \label{fig:teaser}
\end{figure}

Previous multi-object tracking algorithms~\cite{lin2018prior,zhang2016tracking,wu2013simultaneous,weng2020gnn3dmot,braso2020learning,zhang2019robust,pang2020tubetk,yin2020unified}rely on a three-stage scheme ( detection, tracklet generation and linking), also known as \emph{tracking by detection}. The first stage exercises a \emph{per-frame object detector} to localize object bounding boxes in each frame. The second stage, \emph{tracklet generation}, merges detection results to create a set of tracklets, \emph{i.e.}, short tracks, based on short-term cues. These short-term cues include motion~\cite{wojke2017simple}, appearance features~\cite{Wojke2018deep}, and geometrical affinity~\cite{sun2019deep}. Then the third stage, \emph{long-term tracklets linking}, links the tracklets in the whole video to form long-term tracking outputs. The three-stage tracking-by-detection scheme is known to provide reasonable tracking quality. However, there are several limitations in this scheme. Firstly, the division of object detection and tracklet generation makes the tracking quality highly reliant on the detector's capability in accurately detecting objects against all video-specific nuances, such as motion blurriness, large pose variations, and video compression artifacts. These detectors~\cite{ren2015faster,liu2016ssd}, however, are mostly designed for still-images and operated in a per-frame manner. Without adaptation, it is hard to utilize temporal context to overcome these nuances. Secondly, it is impossible to recover an object bounding box through temporal context once it is missed by the detector. Thirdly, low-cost matching techniques (\emph{e.g.} IoU matching in~\cite{wojke2017simple}) used in the tracklet generation process are prone to false matches, especially in occluded cases thus creating tracks with multi objects.

In this work, we propose the single-shot multi object tracker (SMOT), a new framework for multi object tracking, to address the above issues. It is composed of two cascaded stages, the \emph{tracklet generation} stage and the \emph{tracklet linking} stage, in contrast to the three stages in previous mentioned tracking by detection scheme. We design the tracklet generation stage following the ideas of tracking by re-detection, proposed in~\cite{bergmann2019tracking}, which emphasizes performing detection and tracking simultaneously. We adapt this idea on the SSD detectors for constant runtime regardless of the number of objects in each frame. 

The tracklet generation stage starts with detecting objects in the first frame and propagating the detection results to the following frames to \textbf{re-detect} these objects. 
Assuming no significant motion exists between two consecutive frames, the method can establish tracklets without additional matching.
Unlike in~\cite{bergmann2019tracking}, where Faster R-CNN ~\cite{ren2015faster} is used and re-detection is implemented by passing the previous frame detection as additional \textbf{tracking proposals} for the next frame, we implement the idea of redetection with single shot face (SSD) detectors. 
This choice is based on the low-latency requirement of video face understanding. 
However, this choice proves to be non-trivial due to an obvious issue. That is, in SSD detectors, there is no proposal classification and regression stage as in two-stage detectors (\emph{e.g.} Faster R-CNN~\cite{ren2015faster}) where the proposals can be passed through.
We solve this problem by proposing a \textbf{tracking anchor assignment} module that emulates the process of adding tracking proposals to two stage detectors. It is inspired by the fact that SSD detectors implement the concept of proposals in a sliding window manner, \emph{i.e.} the anchors. We utilize the fact that a detected bounding box in one frame can still be approximated by a combination of one or multiple anchors, and thus be used for re-detection in the next frame.
For the tracklet linking stage we propose to use a simple online linking algorithm that links a new tracklet to existing tracklets according to its appearance features. 
Thanks to the high-quality tracklets generated by the first module, we found this simple linker can work well in experiments, while at the cost of minimal increase in overall runtime of the SMOT tracker.


We benchmark the proposed SMOT~ method on three multi object tracking datasets: MusicVideos~\cite{zhang2016tracking}, Hannah~\cite{ozerov2013evaluating}, and MOT17~\cite{milan1603mot16}.
Experimental results show that the SMOT~can perform fast, accurate, and robust long-term tracking of objects in videos. Especially it significantly reduced the number of transfers~\cite{ristani2016performance}, \emph{i.e.}, the cases of mis-matching multi objects into one track, which is important when there are downstream tasks like recognition relying on the tracking results. 
To summarize, the contributions of this work are:
1) we propose a compact design of multi-object tracking approach that unifies single-shot multi-box (SSD) detector and the tracking-by-redetection scheme;
2) we propose the tracking anchor assign technique to enable propagation of temporal information in SSD detectors;
3) extensive experiments demonstrate that the SMOT~tracker assisted with a light-weight tracklet linking module can achieve comparable or better performance to the state-of-the-art trackers. 


\begin{figure*}[thpb]
      \centering
      \includegraphics[width=0.9\linewidth]{figures/pipeline.png}
      \centering
      \caption{Given a frame $F_t$, we first use motion estimator to align existing anchor box $A_i^{t-1}$ associating with the identity $i$ to a new location in the corresponding single shot detector feature map. The regressor and classifier is then used to generate bounding box prediction $B_i^t$ and confidence score $C_i^t$. The $C_i^t$ is used to kill potentially occluded tracks and the $B_i^t$ is used to update the anchor box $A_i^t$. At the same time, the face detector provides a set of detections. We initialize a new track, if a detection has no substantial intersection over union with any bounding box in the set of active tracks. (view in color is suggested)}
      \label{pipeline}
\vspace{-15pt}
\end{figure*}

\section{Related Works}
\textbf{Deep learning-based object detection}
Object detection is a basic computer vision problem. Based on whether using explicit detection proposals, the deep learning detectors can be divided into two main groups: two-stage detectors ~\cite{lin2017feature,ren2015faster,girshick2015fast,he2017mask}  and one stage detectors ~\cite{liu2016ssd,redmon2016you,redmon2018yolov3}. The two-stage detectors first generate detection proposals and then conduct classification as well as location regression. One known issue for the two stage detectors is that the runtime increase dramatically when there is multi
instances to detect, affecting the practical usage in real scenario. While single-shot detectors like SSD ~\cite{liu2016ssd} conduct the proposal generation in an implicit manner. Anchors are generated directly from feature maps and then regressed to the ultimate detection results. The anchor-based regression is agnostic to instance numbers so it could be well applied in scenes where multi-target detection is required like face detection ~\cite{deng2019retinaface}. 

\textbf{Multi-object tracking}
Benefiting from the advance of object detection, we saw the emergence of many multi-object tracking approaches ~\cite{voigtlaender2019mots,xu2019deepmot,bochinski2017high,insafutdinov2017arttrack,ma2018trajectory}. ~\cite{wang2019exploit} proposed a multi-scale TrackletNet to merge tracklets generated through CNN features, IoU scores and  epipolar geometry.  ~\cite{feichtenhofer2017detect} combined the tracking and detection, providing a simultaneous approach as detection shows the power in regressing accurate position of the targets. The author developed the D2T framework by adding a correlation layer to a Siamese R-FCN \cite{dai2016r} detector. Global tracklets fusion are employed later to generate final tracking results. Compared to these batch based tracking methods, online multi-target tracking has drawn more and more attention due to its run-time potential in practice. ~\cite{Wojke2018deep} provides a simple but effective online multi-target solution. To realize online multi-target tracking, the author combined the deep learning based detection, feature embedding with traditional MOT framework and proposed deep feature \& IoU merging based two-gate tracking methods. Different from these tracking by detection methods where complete detection are employed at the first stage, ~\cite{bergmann2019tracking} builds the connection between tracking and detection in a more entangled way. It saves detection results and transforms as new detector proposal to regress target positions in the new frame. This approach has a strong assumption that the target does not have large motion, and thus the new regressed position and the proposal corresponds to the same object. This means the pair of bounding boxes automatically form a tracklet without any additional association efforts.


\section{Pipeline Overview}
Before diving into details of each stage in the SMOT~, we first review the overall object tracking pipeline.
The input to the SMOT~is a video, represented by a consecutive sequence of $T$ frames, $\{F_i\}_{t=1}^T$, and the output is a series of tracklets $\{S_i\}$ representing object locations from the input sequence.
Here one tracklets is represented by a set of object bounding boxes $B_i^t$ from frames with the same pathing ID $i$. 
And each bounding box is in the format of: top-left corner $(x_i^t, y_i^t)$, height $h_i^t$, and width $w_i^t$.

SMOT~operates with a cascade of two components, as illustrated in Fig.~\ref{fig:teaser}.
The first stage is called \emph{tracklet generator}, in the sense that it focuses on generating tracklets from the video frames.
A tracklet $L_j$ is a short-term consecutive bounding box sequence, which encapsulates a series of bounding boxes of the same object instance in several consecutive frames. 
In SMOT~ the tracketlet generator is converted from a normal SSD detector. 
At timestep $t$, the tracklet generator observes the current frame $F_t$.
It also carries the tracklets that were present in the previous frame. 
We detect the known objects again in the new frame by propagating their information from the previous frames to extend the tracklets. We call this operation \emph{re-detection}.
Re-detected objects are used to extend the existing tracklets by inheriting their identity. 
Existing tracklets failed to be re-detected will be terminated and released from the tracklet generator. 
The tracket generator also performs normal detection on the new frame. 
The result bounding boxes will first be merged with existing tracklets to avoid duplication, where the remaining standalone bounding boxes will be then used to initiate new tracklets.
The second stage is called \emph{tracklet linking}, which works in an online manner to link tracklets into long-term ones. 
The tracklet linking extracts appearance embedding feature for a newly formed tracklet and matches it with the existing tracklets. 
The matched tracklets are then linked to the corresponding tracks with the same track ID. 
Non-match tracklets will start as new tracks with their own new track ID.
In the following sections, we will go through the details of the two stages.

\section{Tracklet Generator}\label{sec:tracklet_generator}
The trackelet generator is based on the idea of tracking by redetection, which performs detection and tracking simultaneously.
This scheme is recently proposed by~\cite{bergmann2019tracking} for the task of person bounding box pathing. 
In this work, the basic motivation is that the bounding box regressors in two-stage object detectors, such as faster R-CNN, has the capacity of rectifying the locations and shapes of rough object proposals. 
Thus by propagating the rough locations of the tracked objects in the previous frames, we can re-detect these objects and naturally extend the tracklets overtime.
In~\cite{bergmann2019tracking}, this idea is implemented on a faster R-CNN~\cite{ren2015faster} person detector trained on the COCO dataset~\cite{lin2014microsoft}.
The bounding boxes of the tracked objects in the previous frame are propagated to the next frame as additional ``tracking proposals'' for classification and regression. 
It is proven to be effective for the person tracking task. 
However, there is a downside of this implementation, that its runtime will grow linearly with the number of objects, which becomes a problem for multi-object tracking in crowded scenes where a fast, constant runtime is required.
In SMOT, we design the tracklet generator following the tracking by redetection scheme, but using a single shot face detector ~\cite{liu2016ssd} (SSD) as the backbone. 
The goal is to have an almost constant runtime regardless of the number of objects in the frames. However, it is non-trivial to integrate the tracking by redection scheme to SSD detectors, which have no proposal classification and regression process as in Faster-RCNNs~\cite{ren2015faster}, where we can insert the tracking proposals . 
To solve this issue, we propose a novel \textbf{tracking anchor assignment} module, which enables the process of propagating tracked object locations between frames without explicit proposals. 


\subsection{Redetection with Tracking Anchors}\label{sec:redect}
Unlike Faster-RCNN, SSD detectors do not have an explicit concept of proposals. Instead, they operate by classifying and rectifying the locations of a set of predefined location hypothesis, \emph{i.e.}, anchors.
Therefore, another way to view single shot detectors is that they actually have a fixed number of proposals in the form of anchors with prior locations and shapes.
The network then, based on the image features, makes prediction for each proposal. 
This gives us a way to reinstate in the current frame the bounding boxes from tracked object in previous frames. 
Instead of inserting tracked object locations as proposals, we can use a set of predefined anchors to approximate the rough locations of the tracked objects. 
We call this set of anchors the \textbf{tracking anchors} of an object. 
By aggregating the detection output of the tracking anchors, we can then estimate the redetection result as if the tracked object locations were inserted as proposals.

Formally, assuming for a frame $F_{t}$, we have detected several object bounding boxes assigned to tracklet $\{L_j^t\}$.
For frame $F_{t+1}$, we adopt the following steps to re-detect the objects.
For each bounding box $B_j^t$ of the trackle $\{L_j^t\}$, we predict its location and shape at $t+1$ by function $\hat{B}_j^{t+1} = H(B_j^t)$.
The simplest form of $H$ can be the identity function, which means 
$\hat{B}_j^{t+1} = B_j^t$.
From all the detector anchor candidates, we extract a set of $K$ anchors $\{a_{jk}\}_{k=1}^{K}$ which are close to $B_j^t$ in both scale and location spaces. 
This set of anchors will serve as the tracking anchors of $B_j^t$. They each carries a weight coefficient $w_{jk} = A(a_{jk},\hat{B}_j^t) $ determined by the tracking anchor assignment function $A$. 
The redetection can then be performed by aggregating the detection output of these anchors the bounding boxes shapes $\{B_{jk}^{t+1}\}$ and the detection confidence scores $\{c_{jk}^{t+1}\}$ on frame $F_{t+1}$, defined as following: 
\begin{align}\label{eq:redetect}
  B_j^{t+1} = \sum_{k=1}^{K}\frac{w_{jk}^{t+1} B_{jk}^{t+1}}{w_{jk}^{t+1} },\quad
  c_j^{t+1} = \sum_{k=1}^{K}\frac{w_{jk}^{t+1} c_{jk}^{t+1}}{w_{jk}^{t+1} },
\end{align}
where $B_j^{t+1}$ is the redetected bounding box shape of the tracked object $j$ in frame $F_{t+1}$ and $c_j^{t+1}$ is its confidence score.
Determined by a tracking specific threshold $\sigma_{active}$, there are two possible outcomes for these anchors:
a) if $c_j^{t+1} \geq \sigma_{active}$, this is a valid redetection. The bound box $B_j^{t+1}$ is considered as the new presence of the tracked object $j$;
b) if $c_j^{t+1} < \sigma_{active}$, this is a failed redetection. The tracklet is considered terminated, and released from the tracklet generator.

\noindent\textbf{Tracking anchor assignment strategy}.
In the SMOT~framework, we use strategies to build the tracking anchor assignment module. The simplest strategy is when $K=1$, we just take the anchor ${a_j^{t+1}}$ that has the highest intersection over union (IoU) with $\hat{B}_j^{t+1}$. In experiments, we found that this simple strategy, known as ``single-assignment'', performs well for the task like face or person tracking.
When $K > 1$, we can take the top-$K$ highest IoU anchors with $\hat{B}_j^{t+1}$ and use their IoU values as the weights. We call this strategy ``multi-assignment''.
It is also possible to use other strategies such as learned weighting, which we leave as future work.  

\noindent\textbf{Is the prediction by the tracking anchors reliable?}
To answer this question, we conduct a toy experiment on a small set of images. 
We look at the predictions of the anchors that are similar in terms of intersection over union (IoU) to the ground truth bounding boxes. 
We found most of these anchors' predicted bounding boxes point to the groundtruth bounding boxes.
Quantitatively, we demonstrated in Fig.~\ref{fig_aug_ratio} how the initial shift of a bounding box affect the ability to correctly detect an object.
Using the $w,h$ of the bounding box in $F_{i}$ as a scale, we added multi ratios of shift to the image for re-detection. Fig.~\ref{fig_aug_ratio} shows the re-detection results. We found that the detector can correctly localize the object position even with a relative shift ratio of 0.33. This means if there is only a small motion between two consecutive frames, we can accurately track the objects in the new frames.
Similar to their counterpart in faster RCNN, the bounding box regressors in SSD also have strong power to rectify anchors, producing tight bounding boxes of the objects.
If the tracking anchors are close to the rough locations of the object to track, there is a high chance to correctly find its new location, so we can rely on the aggregation of these tracking anchors' predictions for redetection.


\noindent\textbf{The workflow of tracklet generation}. In summary, we can describe the workflow of our tracklet generator as following:

(1) For the first frame $F_0$: run the single shot object detector and get all object detection results $\{B_i^0\}$. Create one tracklet ID for each bounding box;

(2) Predict the locations and shapes of these tracked objects $\{\hat{B}_j^{1}\}$ in frame $F_{1}$ . Obtain the tracking anchors $\{a_{ik}^0\}$ and their weights $\{w_{ik}^0\}$ for each object $B_j^1$ through the tracking anchor assignment module;

(3) Run the detector on frame $F_1$. Obtain the redetection results according to Eq.~\ref{eq:redetect}, decide whether to extend or halt the tracklet according to the tracking specific threshold $\sigma_{active}$;

(4) Perform non-maximal suppression on the re-detection bounding boxes to merge overlapping tracklets, by suppressing new detection results that have high IoU to the tracked bounding boxes.

(6) For remaining new detection results, create a new tracklet for each of them.

(7) Repeat the steps 2-6 for the remaining frames consecutively. 


\begin{figure}[thpb]
      \centering
      \includegraphics[width=1\linewidth]{figures/eccv_fig/hannah_anchor_test_eccv.png}
      \centering
      \caption{Robustness test of the anchor regression (a-d corresponds to shift ratio of 0.1, 0.25, 0.33, 0.45, respectively). Red denotes the ground truth, yellow denotes the shifted box and green the re-detected target position (better view in color ).}
      \label{fig_aug_ratio}
\end{figure}


\subsection{Motion Assisted Re-Detection}
Another important module in the SMOT~tracklet generator is the function $H$ that predicts the rough location of a tracked object in the new frame. The simplest form of $H$ is the identity function, which implies a small motion assumption. 
When the video contains large motion, this assumption breaks and it becomes easy for the tracklet generator to lose the target. 
Therefore, it is important to consider motion in bounding box prediction. 

Motion estimation from two consecutive frames is a widely investigated problem~\cite{ilg2017flownet,ilg2018occlusions}. 
Dense optical flow algorithms provide pixel-wise displacement prediction between two frames, which can be used to predict the bounding box movement. 
We propose to use the dense optical flow field to build the function $H$ for each tracked object.
Particularly, given the optical flow field $D_t$ between $F_{t}$ and $F_{t+1}$ and a two element displacement vector (horizontal and vertical) $\Vec{d}_{(h, w)}$ on each pixel location, we can predict the new location of the bounding box $B_j^{t}$ in frame $F_{t+1}$ as
\begin{align}
    \hat{B}_j^{t+1} = H_{motion}(B_j^{t}) = B_j^{t} \oplus \frac{\sum_{(h, w) \in B_j^{t}} \Vec{d}_{(h, w)}}{\mathrm{area}(B_j^{t})},
\end{align}
where the operator $\oplus(B, \Vec{d})$ shifts the center of the bounding box $B$ by the vector $\Vec{d}$ and the function $\mathrm{area}(B)$ gives the area of the bounding boxes in square-pixels.
We call the tracklet generation process aided with this form of $H$ as motion-assisted redetection.
In experiments, we compare two different dense flow estimation approaches, a traditional FarneBack flow~\cite{farneback2003two} and another deep learning based PWCNet flow~\cite{sun2018pwc}.
We found that the motion assisted redetection can benefit from both approaches even though their optical flow estimation benchmark numbers are quite different. 


\begin{figure}[thpb]
      \centering
      \includegraphics[width=1\linewidth]{figures/eccv_fig/linking_eccv.png}
      \centering
      \caption{Online updating of the feature bank for tracklet-track linking. $\phi_i^t$ denotes the identity embedding of track $i$ at time $t$. We store a bank of the identity embeddings where each track has one feature. When we have a new tracklet, we either match it to an existing identity in the bank or add a new one to the bank (better view in color).}
      \label{linking}
\end{figure}
\vspace{-15pt}
\section{Online Linking of the Tracklets}
Since the output of the tracklet generator are short tracklets, which are usually halted due to occlusion and temporary disappearance of objects in videos,
we design an online tracklet linking module to group the tracklets into final object tracks.

The linker carries a set of existing tracks $\{S_i\}$, where each track is represented by an identity embedding $\phi(S_i)$ produced by averaging the output of a embedding model $\phi$ on objects in this track.
At a certain time step $t$, there are two possible states of a track: 1) this track is linked to an active tracklet in the current frame $F_t$; 2) this track has no active tracklet. Fig. ~\ref{linking} illustrates the identity linking process.
For each new tracklet $L_j^t$ appearing in $F_t$, we try to match it with tracks in state 2). 
To perform the matching, we first extract the appearance embedding from the object in $L_j^t$, then we calculate the pairwise match cost between $L_j^t$ and the tracks. Once a track has a new tracklet linked to it, it goes to state 1) in the next frame. By solving the bipartite matching problem we can link the matching tracklet to the tracks. For those new tracklets that are not matched to any existing tracks, generate a new track for each of them. 


This tracklet linker handles the cases of occlusion and temporal disappearance across time. 
On the other hand it requires extracting appearance embedding using an embedding model, usually a deep neural network like~\cite{deng2019arcface,shen2017learning}, which adds to the runtime cost.
However, this additional cost can be minimal due to two reasons:
1) the linking is only performed for newly created tracklets, once a tracklet is linked we do not need to re-extract its embedding.
2) the extraction of appearance embedding for the track do not need to be run on each frame, as in videos the appearance of objects usually change slowly.



\section {Experiments}
We evaluate the proposed SMOT~tracker on three multi object tracking datasets: the Hannah dataset~\cite{ozerov2013evaluating}, the Music Videos dataset~\cite{zhang2016tracking}, and the MOT17~\cite{milan1603mot16} dataset.
The approach is assessed in term of standard multi object tracking metrics, runtime speed, and metrics related to downstream tasks such as video face recognition or person identification.
We also provide qualitative analysis of the approach's ability in handling common nuance factors to object tracking.
\subsection{Implementation details}
In our implementation of SMOT, we implement the single shot object detector with ResNet34~\cite{he2016deep} 
~as the backbone architecture for both person and face tracking tasks. Feature pyramid network~\cite{lin2017feature} is adopted to enhance the context modeling ability.
We set the confidence threshold $\sigma_{det}$ to $0.9$ for detection and $\sigma_{tk}$ to $0.4$.for tracking.
We set the NMS threshold for redetection as $0.6$ and the IoU threhold between tracks and new detection as $0.3$.
In motion assisted re-detection, we set optical flow field size for the Farneback algorithm to $256\times 256$ pixels and $1024\times1024$ for the PWCNet~\cite{sun2018pwc}, respectively.
We use the face embedding network with 128-dimensional feature vectors proposed in~\cite{deng2019arcface} for tracklet linking module and the threshold distance of face verification is 0.97. No appearance model and tracklet linker is used for person tracking. The tracklet generator in the person tracking task uses $0.9$ as detection threshold and $0.4$ as tracking threshold to keep high precision of our bounding box quality, as well as better recall capability for tracking regression.
\subsection{Dataset}
 To evaluate the performance of our approach as generic multi-object tracking, we test on two common use cases: person tracking and face tracking. We test our approach in the MOT17 dataset to see how it performs in person tracking, while on Hannah and MusicVideo dataset for face tracking scenario.

\noindent\textbf{MOT17} ~\cite{milan1603mot16} is a challenging dataset designed for multi object tracking. The dataset includes 7 video sequences for training and 7 videos for testing, covering indoor and outdoor person tracking cases. There are lots of person occlusion, fast motion, making the tracking even more challenging. 

\noindent\textbf{Hannah dataset}~\cite{ozerov2013evaluating} is a public available face tracking dataset from the Hollywood movie \textit{Hannah and her sisters}. There are in total 153825 frames and 245 shots. As a multi face tracking dataset, it contains 254 people and 53 of whom are named characters. Overall 2002 tracks are annotated, ranging from 1 to 500 frames in length. 

\noindent\textbf{Music video dataset} ~\cite{zhang2016tracking} is a widely used face tracking benchmark with $8$ high resolution music videos from YouTube ranging from four to six minutes in duration. The videos contain
edited scenes with singing and dancing movement. Each video contains around 3000 to 5000 frames with 1 to 10 people in the video. It is a challenging benchmark for face tracking due to fast motion, shot changing and crowd occlusions.



\vspace{-15pt}
\subsection{Evaluation Metric}
For the multi object trackings tasks we report the performance metrics as discussed in~\cite{ristani2016performance}. 
In particular, we report multi-object tracking accuracy (\textbf{MOTA}), min-cost matched f1 scores (\textbf{IDF1})), recall, precision, f1 score (\textbf{F1}), number of transfers (\textbf{Transfer}), and id switches (\textbf{IDsw.}).


\subsection{Baseline approaches} We implemented the widely used DeepSORT tracker~\cite{Wojke2018deep} as one of our baseline approaches. It is based on the standard tracking by detection scheme. 
As shown in ~\cite{wojke2017simple}, the performance of the DeepSORT tracker is highly dependent on the quality of its object detector and re-ID features.
For face tracking, we equip this approach with the state-of-the-art image based face detector trained on the WIDER Face~\cite{yang2016wider} dataset and the state-of-the-art face embedding~\cite{deng2019arcface} for re-identification features, same to our approach for fair comparison. We refer to this method as \textbf{DFS} (Deep Face Sort).
Another baseline method we employed is the Tacktor++ described~\cite{bergmann2019tracking}. 
For face tracking benchmarks, we use a face detector based on faster-RCNN+FPN and trained on the WIDERFace~\cite{yang2016wider} dataset for fair comparison.
For person tracking, our model detector is trained on COCO ~\cite{lin2014microsoft} and finetuned on MOT17 ~\cite{milan2016mot16} training split. And then benchmark the result on MOT17 under the same experiment protocol with Tracktor++~\cite{bergmann2019tracking}.


\begin{figure}[thpb]
      \centering
      \includegraphics[width=.9\linewidth]{figures/eccv_fig/annotation_eccv.png}
      \centering
      \caption{Fig. $(a)$ shows the different bounding box styles of Hannah dataset and WiderFace, WiderFace has tight bounding box covering the face region while Hannah style bounding box is wider horizontally including ears but shorter in vertical which starts from forehead center to jaw center. Fig. $(b)$ illustrates the rare case where shot changes but new face appear at the same region, this leads to identity miss match in tracking}
      \label{fig_dataset}
\end{figure}

\vspace{-15pt}
\subsection{Comparison with State of the Arts}
Table ~\ref{tab:tab_overall} shows the evaluation results on the three benchmarks. We compare our basic tracklet generator, motion-assisted tracklet generator and the full SMOT~tracker with the state-of-the-art approaches. We can observe that our basic tracklet generator shows much better performance in number of transfers and recall rate compared to the DFS methods, which means it miss-matches less different objects and detects more object boxes with the help of temporal propagation.
We also compared with the tracktor++ baseline results, we find that our tracklet generator shows comparable or better tracking accuracy than tracktor++. Furthermore for the setting of online tracking, the SMOT outperforms the the state-of-the-art-methods. Even compared with other offline approaches where more spatial and temporal information are fused to improve tracking \cite{lin2018prior} \cite{zhang2016tracking}, the method has comparable performance and shows significant improvement in metric like transfer.   
This demonstrates the effectiveness of our solution, employing tracking anchor assignment and aggregation in single-stage tracking. 


\vspace{-15pt}
\subsection{Ablation Study}

\noindent\textbf{Detection and tracking thresholds.}
We first examine the impact of the two confidence thresholds in the tracklet generator, namely, the detection threshold and the tracking threshold $\sigma_{active}$. 
The results on both datasets are shown in Table~\ref{tab_anchor} where $d$ and $t$ stands for detection and tracking confidence threshold, respectively.
We can observe that generally a higher detection threshold leads to fewer false positive detections and thus improves the precision rate, while 
the recall rates are less affected, so we can choose a high detection threshold for the tracklet generator. 
The number of transfers can be further reduced by tuning the tracking threshold $\sigma_{active}$ .
We recommend the combination of $0.9$ and $0.4$ for the detection and tracking thresholds.

\noindent\textbf{Tracking Anchor Assignment Strategies}
In sec~\ref{sec:redect}, we introduced the tracking anchor assignment module. It is flexible with multi assignment strategies. Here we experimented with two of them. The first is the single-assignment where only one anchor $i$ is assigned for each tracked object. The second is the multi-assignment strategy, where a set of $K$ anchors is used and their IoU scores with the tracked objects are used as aggregation weights. 
The results are shown in Table~\ref{tab_anchor}. 
We observe that even with one single tracking anchor, the tracklet generator can still work quite well. 
With $K$ increasing, the tracklet generator is able to generate longer tracklets with fewer breaks, which is illustrated by the increasing IDF1 and the decreasing ID switches.

\noindent\textbf{Dense optical flow algorithms.}
The motion assisted redetection involves computation of dense optical flow. 
We experimented with two types of algorithms , the first is FarneBack~\cite{farneback2003two} algorithm, and the second is the PWCNet~\cite{sun2018pwc}. Here the baseline is the basic tracklet generator without motion assistance.
We found that involving motion can significantly improve the performance. And the more accurate PWCNet leads to better tracking accuracy. This suggests that with better learned motion prediction module, the tracklet generator could be further improved.


\begin{table}
\centering
\tiny
\caption{SMOT~ tracker performance evaluation}
\label{tab:tab_overall}
\begin{tabular}{ |c|c|c|c|c|c|c|c|c|c|c| }

\hline
&\multicolumn{10}{c|}{MOT17} \cr
\hline
      Methods & MOTA & IDF1 &ID Precision &ID Recall & Transfer & F1& Precision &Recall & ID switch & Online?\\
\hline

  Tracktor++ ~\cite{bergmann2019tracking} &0.588 &0.623 &- &- &- &0.761&0.99 &0.626 &1425  & \checkmark \\
  SMOT &0.529 &0.477 &0.639 &0.382 &690 &0.721 &0.965 &0.576 &2491 & \checkmark\\
  SMOT w/ Farneback& 0.554 &0.538 &0.731 &0.426 &89 &0.721 &0.979 &0.571 &743 &\checkmark \\
  SMOT w/ PWCNet& 0.56 &0.539 &0.721 &0.43 &69 &0.728 &0.974 &0.581 &620  &\checkmark \\

\hline


\hline
&\multicolumn{10}{c|}{Hannah} \cr
\hline
      Methods & MOTA & IDF1 &ID Precision &ID Recall & Transfer & F1& Precision &Recall & ID switch & Online?\\
\hline
  DFS ~\cite{Wojke2018deep}& 0.642 &0.671 & 0.604 &0.753 &71 &0.842 &0.759 &0.946 &788 & \checkmark \\ 
  Tracktor++ ~\cite{bergmann2019tracking} &0.542 &0.552 &0.484 &0.642 &46 &0.807 &0.708 &0.938&2042  & \checkmark \\
  SMOT & 0.609 &0.569 &0.507 &0.646 &20 &0.784 &0.698 &0.896 &3449 & \checkmark\\
  SMOT w/ Farneback& 0.726 &0.580 &0.569 &0.590 &10 &0.870 &0.853 &0.885 &1499 &\checkmark \\
  SMOT w/ PWCNet& 0.716 &0.565 &0.555 &0.574 &8 &0.865 &0.851 &0.880 &1805  &\checkmark \\
  SMOT w/ online linking& 0.728 &0.654 &0.642 &0.667 &165 &0.869 &0.853 &0.885 &1122 &\checkmark \\
\hline
&\multicolumn{10}{c|}{MusicVideo} \cr
\hline
  Siamese~\cite{zhang2016tracking} & 0.623 &- &- &- &986 &0.795 &0.894 &0.715 &- &  x \\ 
  SymTriplets~\cite{zhang2016tracking}& 0.638 &- &- &- &699 &0.798 &0.897 &0.718 &- & x \\ 
  Prior-Less~\cite{lin2018prior} &0.692 &- &- &- &624 &0.853 &0.902 &0.817 &- & x \\ 
  DFS~\cite{Wojke2018deep}&0.614 &0.141 &0.141 &0.142 &400 &0.84 &0.842 &0.837 &5641 &\checkmark \\ 
  Tracktor++~\cite{bergmann2019tracking}&0.417 &0.117 &0.109 &0.127 &230 &0.765 &0.709 &0.830 &6537 & \checkmark \\
  SMOT &0.536 &0.114 &0.106 &0.123 &38 &0.831 &0.772 &0.900 &8866 & \checkmark\\
  SMOT w/ Farneback&0.571 &0.120 &0.118 &0.121 &44 &0.821 &0.809 &0.834 &5693 &\checkmark \\
  SMOT w/ PWCNet&0.552 &0.111 &0.099 &0.127 &49 &0.803 &0.813 &0.824 &5894 &\checkmark \\
  SMOT w/ online linking&0.615 &0.244 &0.232 &0.259 &545 &0.853 &0.808 &0.904 &6525 &\checkmark \\
\hline

\end{tabular}
\vspace{-15pt}
\end{table}


\begin{table}
\centering
\tiny
\caption{SMOT~ tracker performance with different detection \& tracking confidence thresholds.}
\label{tab_anchor}
\begin{tabular}{ |c|c|c|c|c|c|c|c|c|c|c|c|c| }

\hline
&\multicolumn{6}{c|}{Hannah} &\multicolumn{6}{c|}{MusicVideo} \cr
\hline
      $\sigma_{det}$ $\sigma_{tk}$  & MOTA & IDF1 & Transfer & F1&Precision & ID switch& MOTA & IDF1 & Transfer & F1&Precision & ID switch \\
\hline
0.5 0.1 & 0.468 & 0.542  & 22 & 0.786 & 0.671  & 3122 & 0.416 & 0.108 & 51  & 0.698 & 0.784   & 8400 \cr  
0.65 0.3 & 0.552 & 0.541  & 13 & 0.819& 0.716  & 4994 & 0.493 & 0.109 & 32  & 0.751 & 0.824   & 10673 \cr  
0.8 0.2 & 0.573 & 0.559  & 20 & 0.823 & 0.725  & 3558 & 0.511 & 0.112 & 38  & 0.756 & 0.823   & 9066 \cr 
0.8 0.3 & 0.565 & 0.513  & 15 & 0.837 & 0.746  & 12106 & 0.465 & 0.110 & 32  & 0.773 & 0.834   & 15629 \cr 
0.8 0.4 & 0.590 & 0.532  & 12 & 0.837 & 0.745  & 7330 & 0.512 & 0.111 & 25  & 0.779 & 0.841   & 12722 \cr 
0.9 0.2 & 0.609 & 0.569  & 20 & 0.836 & 0.746  & 3449 & 0.536 & 0.114 & 38  & 0.772 & 0.831   & 8866 \cr 
0.9 0.3 & 0.607 & 0.524  & 15 & 0.851 & 0.771  & 11579 & 0.493 & 0.112 & 32  & 0.791 & 0.841   & 14988 \cr 
0.9 0.4 & 0.627 & 0.543  & 11 & 0.849 & 0.768  & 6885 & 0.538 & 0.113 & 25  & 0.796 & 0.848   & 3122 \cr  \hline
\end{tabular}

\vspace{-5pt}
\end{table}


\begin{table}
\centering
\tiny
\caption{SMOT~ tracker performance v.s. numbers of anchor aggregation}
\begin{tabular}{ |c|c|c|c|c|c|c|c|c|c|c| }

\hline
&\multicolumn{5}{c|}{Hannah} &\multicolumn{5}{c|}{MusicVideo} \cr
\hline
      Num of anchor & MOTA & IDF1 & Transfer & F1 & ID switch&MOTA & IDF1 & Transfer & F1 & ID switch \\
\hline
1 & 0.612 & 0.573  & 13 & 0.837  & 3239 &  0.563  & 0.116 & 27 & 0.845 & 8821  \\ 
5 & 0.607 & 0.581  & 17 & 0.835  & 3040 &  0.553  & 0.117 & 34 &0.837 & 8361 \\
10 & 0.609 &0.569   & 20 & 0.836  & 3449 & 0.536 & 0.114 & 38 & 0.831  & 8884 \\
20 & 0.608 & 0.581  & 17 & 0.835  & 3040 &  0.553  & 0.117& 34 & 0.837  & 7495 \\
\hline
\end{tabular}
\vspace{-5pt}
\end{table}

\begin{table}[h!]
\centering
\tiny
\caption{SMOT~tracker runtime v.s. performance}
\begin{tabular}{ |c|c|c|c|c|c|c|c|c|c|c|c| }

\hline
& &\multicolumn{5}{c|}{Hannah} &\multicolumn{5}{c|}{MusicVideo} \\
\hline
      Model &FPS& MOTA & IDF1 & Transfer & F1 & ID switch&MOTA & IDF1 & Transfer & F1 & ID switch \\
\hline
Tracktor ++ 1024& 9 & 0.542 & 0.552  & 46 & 0.807  & 2042 &  230  & 0.117 & 230 & 0.765 & 6537\\
SMOT 1024&17 & 0.609 & 0.569  & 20 & 0.836  & 3449 &  0.537  & 0.114 & 38 & 0.832 & 8866  \\ 
SMOT 768&22 & 0.612 & 0.568  & 21 & 0.837  & 3443 &  0.536  & 0.114 & 39 &0.829 & 8746 \\
SMOT 512&27 & 0.621 &0.566   & 20 & 0.840  & 3464 & 0.529 & 0.114 & 36 & 0.826  & 8884 \\
SMOT 256&40 & 0.708 & 0.536  & 11 & 0.878  & 3477 &  0.506  & 0.114& 35 & 0.798  & 7495 \\
\hline
\end{tabular}
\vspace{-15pt}
\label{runtime_exp}
\end{table}



\vspace{-15pt}
\subsection{Runtime Analysis}
One of the main advantages of SMOT is its fast speed. We conducted a comparison experiment to explore the runtime efficiency. In the experiment setting, we resize the input image to $1024\times 1024$, $768\times768$, $512\times512$, $256\times256$ respectively. With one NVIDIA V100 GPU on the different resolution settings, the SMOT can achieve 17, 22, 27, 40 frame per second as shown in Table ~\ref{runtime_exp}, respectively. We also don't find significant change in runt-time when the resolution changes or the number of tracked object increases. This shows the SMOT is fast in runtime and robust in image resolution changes, which makes it possible for real-time processing in different application scenarios. Note that tracklet generation module is independent to the backbone detector structures. It can be applied to any single shot detector. All the current components can be replaced with faster versions in order to achieve real-time performance in much less powerful machines, like edge devices.  



\vspace{-15pt}
\section{Conclusion}
In this paper, we present a multi object tracking approach, named Single-shot Multi Object Tracker (SMOT),  that is able to simultaneously generate detection and tracking outputs. It is composed of a tracklet generator and a tracklet linker. By applying the tracking by re-detection idea to single shot object detectors, the tracklet generator runs at close to constant speed w.r.t. the number of targets. Motion assisted redetection makes it robust to large motion in video tracking cases. A simple but effective tracklet linker merges the tracklets in an online manner and further improves the tracking result.
Extensive experiments shows the SMOT method achieves state-of-the-art accuracy when comparing with other online object tracking approach, with comparable performance even to the state-of-the-art offline methods. 

{
\bibliographystyle{splncs04}

\section{Introduction}

Similar to the role of face detection in image-based face analysis, face tracking is an essential component for face understanding in videos~\cite{tapaswi2019video}. Its outputs are subsequently used in facial expression analysis~\cite{kanade2000comprehensive}, face attributes detection~\cite{shen2017learning}, and face recognition~\cite{deng2019arcface}. 
One important tenet of a multi-face tracker is to distinguish faces of different persons into different tracks.
Despite the rapid advance of tracking algorithms with the help of deep learning~\cite{bergmann2019tracking, feichtenhofer2017detect, zhang2016tracking}, designing a face tracker to achieve this goal still remains a difficult problem.
Besides, due to the high volume size of the video data, the face tracker is also expected to have: 1) fast runtime speed; 2) accurate and robust face localization and 3) the capacity for online processing. 

\begin{figure}[t]
      \centering
      \includegraphics[width=1\linewidth]{figures/teaser.png}
      \centering
      \caption{The proposed FAR tracker consists of two stages. The first stage generates temporally consecutive tracklets by exploring the temporal and spatial correlations from previous frame. The second stage performs online linking of the tracklets to generate a face track for each person (view in color is suggested).}
      \label{fig:teaser}
\end{figure}

Most existing multi-face tracking algorithms~\cite{lin2018prior, zhang2016tracking, wu2013simultaneous} rely on a three-stage scheme, known as \emph{tracking by detection}. The first stage exercises a \emph{per-frame face detector} to localize face bounding boxes in each frame. The second stage, \emph{tracklet generation}, merges detection results to create a set of tracklets, \emph{i.e.}, short tracks, based on short-term cues. These short-term cues include motion~\cite{wojke2017simple}, appearance features~\cite{Wojke2018deep}, and geometrical affinity~\cite{sun2019deep}. Then the third stage, \emph{long-term tracking}, links the tracklets in the whole video to form long-term tracking outputs. The three-stage tracking-by-detection scheme is known to provide reasonable tracking quality. However, there are several limitations in this scheme. Firstly, the division of face detection and tracklet makes the tracking quality highly reliant on the quality of face detectors. These detectors, mostly designed for still-images and operated in a per-frame manner, are not able to utilize temporal context and thus are not robust against video-specific nuance factors, such as motion blurring, large pose variations, and video compression artifacts. Secondly, it is impossible to recover a face bounding box through temporal context once it is missed by the detector. Thirdly, the low-cost matching used in the tracklet creation process is prone to cause false matches, thus creating tracks with multiple persons' faces in it.

In this work, we propose a new tracker architecture to address the issues above based on the idea of perform detection and tracking simultaneously. This tracker is composed of two stages, first for generating tracklets and second for linking them in long-term tracking. 

For the first stage we design a tracklet generation model that creates tracklet directly from a sequence of input frames, without the need for grouping. This tracklet generator can be transformed from an existing face detector with a single-shot multibox detector (SSD)~\cite{liu2016ssd}~design. It runs at an almost constant frame per second (FPS) speed regardless of the number of faces in each frame. As we will later show in the experimental analysis, it is also less prone to cause false matches in tracklets. Our tracklet generation model also gives much higher recall of faces comparing with other divided face detection and grouping approaches, due to the ability to utilize temporal context for detecting faces. The second stage is a simple tracklets linking module based on averaging face identity embedding features of the tracklets, which can be implemented in an online manner. The proposed multi face tracker outperforms other online state--the-art multi-face trackers~\cite{Wojke2018deep} and can achieve comparable results with the global optimization methods~\cite{lin2018prior, zhang2016tracking}.

To build the tracklet generation module that detects and tracks faces simultaneously, we took inspiration from the work in~\cite{bergmann2019tracking}. In~\cite{bergmann2019tracking}, a faster-RCNN~\cite{ren2015faster} model for human detection is converted to multi-person tracker by passing the detection results in the previous frame as additional object proposals for the current frame. This operation is based on the observation that the bounding box regression modules in fast RCNN ~\cite{ren2015faster} has a good capacity in correcting slightly shifted object bounding boxes. It is then used to \textbf{re-detect} the one object tracked in the previous frame from its estimated location in the current frame, so the track identity can also be carried over to this re-detection output. 
We adopt this simple but effective strategy and apply it to the case of SSD-based face detectors. We choose SSD-based detector since the face detection has high runtime efficiency requirement which can be verified in recent face detection algorithms~\cite{zhang2019refineface, zhang2019accurate, deng2019retinaface, li2019pyramidbox++}.
But one question arises for this adoption, that is the SSD-based face detector does not have a explicit proposal stage. Thus it is not straightforward how to convey the detection results between frames.
We solve this problem by reckoning the fact that SSD detector just uses sliding window proposals and designing a process that locate the \textbf{proposal} for re-detection. 
Using the SSD-based architecture enables us to run the model at a constant speed w.r.t. number of faces in each frame. 

Overall, we propose a new face tracker that is composed of a novel tracklet generation module and a simple online linker. We call this tracker \emph{FAR} tracker in the sense that it provides fast, accurate and robust long-term tracking of faces in videos. 
The contributions of this work are:
1) we propose a compact design of multi-face tracker that unites face detection and tracklet creation into a single model followed by a light-weight linkage post-processing algorithm;
2) we implement the tracklet generation by carefully adapting an SSD-based face detector to perform detection and tracking simultaneously;
3) In experiments, the simple yet effective FAR tracker lead to superior results than other state-of-the-art methods while operating in an online manner. 

\begin{figure*}[thpb]
      \centering
      \includegraphics[scale=0.35]{figures/pipeline.png}
      \centering
      \caption{Given a frame $F_t$, we first use motion estimator to align existing anchor box $A_i^{t-1}$ associating with the identity $i$ to a new location in the corresponding single shot detector feature map. The regressor and classifier is then used to generate bounding box prediction $B_i^t$ and confidence score $C_i^t$. The $C_i^t$ is used to kill potentially occluded tracks and the $B_i^t$ is used to update the anchor box $A_i^t$. At the same time, the face detector provides a set of detections. We initialize a new track, if a detection has no substantial intersection over union with any bounding box in the set of active tracks. (view in color is suggested)}
      \label{pipeline}
\end{figure*}

\section{Related Works}

\textbf{Deep learning-based object detection}
Object detection is a basic computer vision problem. Based on whether using explicit detection proposals, the deep learning detectors can be divided into two main groups: two-stage detectors ~\cite{lin2017feature,ren2015faster, girshick2015fast, he2017mask}  and one-shot detectors ~\cite{liu2016ssd, redmon2016you, redmon2018yolov3}. The two-stage detectors first generate detection proposals and then conduct classification as well as position regression. One known issue for the two stage detectors is that the runtime increase dramatically when there is multiple
instances to detect, affecting the practical usage in real scenario. While single-shot detectors like SSD ~\cite{liu2016ssd} conduct the proposal generation in an implicit manner. Anchors are generated directly from feature map and then regressed to the ultimate detection results. The anchor-based regression is agnostic to instance numbers so it could be well applied in scenes multi-target detection is required like face detection ~\cite{deng2019retinaface}. 

\textbf{Multi-target tracking}
Benefiting from the advance of object detection, we saw the emergence of many multi-object tracking approaches ~\cite{voigtlaender2019mots, xu2019deepmot, bochinski2017high, insafutdinov2017arttrack, ma2018trajectory}. ~\cite{wang2019exploit} proposed a multi-scale TrackletNet to merge tracklets generated through CNN feature, IoU score and  epipolar geometry.  ~\cite{feichtenhofer2017detect} combined the tracking and detection, providing a simultaneous approach as detection shows the power in regressing accurate position of targets. The author developed the D2T framework by adding correlation layer to a Siamese R-FCN ~\cite{dai2016r} detector. Global tracklets fusion are employed later to generate final tracking results. Comparing to these batch based tracking methods, online multi-target tracking draw more and more attention because of the potential usage in practical. ~\cite{Wojke2018deep} provide a simple but effective online multi-target solution. To realize online multi-target tracking, the author combined the deep learning based detection, feature embedding with tradition MOT framework and proposed deep feature \& IoU merging based two-gate tracking methods. Different from these tracking by detection methods where complete detection are employed at the first step, ~\cite{bergmann2019tracking} build the connection between tracking and detection in a more entangled way. The author save detection results and transform as new detector proposal to regress target positions in new frame. The most interesting thing of this approach is that, given the assumption that the target does not have large motion, the new regressed position and the \textbf{proposal} corresponding to the same object. This means the pair of bounding box automatically form a tracklet without any additional association efforts.

\textbf{Multi-face tracking with deep learning}
As a special case of multi-target tracking, multi-face tracking also benefits from the overall advance in object detection. In ~\cite{zhang2016tracking}, the author proposed a multi-stage approach to realize multi-face tracking in the wild. The approach first employed a shot detector and face detector to extract face bounding boxes within small shots. Tracklets are then formed through the similirary comparison of face appearance and positions. Improved Triplet loss~\cite{schroff2015facenet} are conducted to generate better face embedding feature for tracklets linking. ~\cite{lin2018prior} provided a multi-model and multi-stage based multi-face tracking solution. In order to generate better face tracklets, the author combined multiple body parts detector to sustain face detection in the scene of large pose, occlusion and low resolution. Then in the tracklets linking stage, recursive association are conducted to finetune the tracking results and Gaussian Process model is used to detect and re-assign outlier tracklets. The additional body parts detection model and recursive association process enrich the tracklets information and thus achieve the state-of-the-art performance.





\section{Pipeline Overview}
The proposed FAR tracker is based on the idea of two stage tracking.
Before diving into details of each stage, we first review the overall tracking pipeline.
The input to the tracker is a video, represented by a consecutive sequence of $T$ frames, $\{F_i\}_{t=1}^T$. 
The tracker is expected to output a series of face tracks $\{S_i\}$ that capture the faces of different persons in multiple frames.
Here one face tracks is represented by a series of rectangular face bounding boxes $B_i^t$ in different frames with the same track ID $i$. 
Each bounding box is represented by the image space coordinates of its top-left corner $(x_i^t, y_i^t)$, heigth $h_i^t$, and width $w_i^t$.

Our tracker operates with a cascade of two stages, as illustrated in Fig.~\ref{fig:teaser}.
The first stage is called \emph{tracklet generator}, in the sense that it focuses on generating tracklets from the video frames.
Tracklets $L_j$ are short-term tracks that are consecutive in time. 
In particular, it encapsules a series of face bounding boxes of the same person in several consecutive frames. 
The tracklet genreator simultaneously perform bounding box detection and tracking. 
At a timestep $t$, it observes the current frame $F_t$.
It also carries the tracklets that are present in the previous frame. 
To extend the tracklets, it tries to detect the current frame but with tracking information absorbed from previous frame, we call this operation re-detection, means detecting new frames with remembering old information.
Re-detected faces are used to extend the existing tracklets by inheriting their identity. 
Existing tracklets failed to be re-detected will be released from the tracklet generator. 
The newly detected bounding boxes  will be considered as the start of a new tracklet.
The second stage is called \emph{tracklet linker}.
It also works in an online manner to link tracklets into long-term tracks. 
The tracklet linker extract face identity embedding for a newly formed tracklet and matches it against the tracks in the linker's memory. 
The matched tracklets are then linked to the corresponding tracks and obtain the track ID. 
Non-match tracklets will start as new tracks with their own new track ID.
In the following sections, we go through the details of the two stages.



\section{Tracklet Generator}\label{sec:tracklet_generator}
The trackelet generator is based on the idea of performing detection and tracking simultaneously, by trying to re-detect the  existing faces within current frame. 
This idea is not new, a similar approach has been introduced in~\cite{bergmann2019tracking} for the task of person bounding box tracking. 
The basic motivation of this approach is that object detectors bounding box regression has good capability to rectify the locations and shapes if giving a not accurate initial proposal. 
Thus by reusing the previous frame's detection results as prior information, we can re-detect these objects and can propagate the same track ID to the re-detected bounding boxes.
In~\cite{bergmann2019tracking}, this idea is realized on a fasterRCNN~\cite{ren2015faster} person detector trained on the COCO dataset~\cite{lin2014microsoft}.
FasterRCNN is by itself a two-stage detector architecture, which first generates proposals then classifies each proposal.
Thus its runtime will grow linearly with the number of objects it is tracking, which is not desired.
In this work, we starting the design of tracklet generator from a single-stage face detector, namely, single-short multibox detector (SSD)~\cite{liu2016ssd}. 
This way, the runtime will not grow no matter how many faces the tracklet generator is tracking.
This is an ideal property especially in crowded scenes.
But re-purposing SSD for the task of tracklet generation with re-detection is not straightforward.
Below we describe how this idea can be implemented.


\subsection{Tracking by Re-Detection with SSD}
Unlike faster-RCNN, single-stage detectors do not have an explicit concept of proposals. They operate by classifiying and rectifying the locations of a set of predefined location hypothesis, \emph{i.e.}, anchors.
Another way to view single-stage detectors, is that, they actually have a huge amount of sliding window proposals. 
These proposals have their prior locations and shapes.
The network then, based on the image features it extracts, makes prediction for each proposal. 
This provides a way to reinstate detected bounding boxes in the detection process of the current frame. 

Given the frame $F_{t}$ and we have detected several face bounding boxes assigned to tracklets $\{L_j^t\}$.
For frame $F_{t+1}$, we adopt the following steps to re-detect the faces in frame $F_{t+1}$.
For each bounding box $B_j^t$ of the tracklet $\{L_j^t\}$, we predict its location and shape at $t+1$ by function $\hat{B}_j^{t+1} = H(B_j^t)$.
The simplest form of $H$ can be identity function, which means 
$\hat{B}_j^{t+1} = B_j^t$.
Of all the detector anchor candidates, the closest one to $\hat{B}_j^{t+1}$ will be the one have largest IoU with $\hat{B}_{t+1}^j$. In this way, we obtained the seed anchor for re-detection.
We directly look at the network predictions the anchors matched to the $\{B_j^t\}$.
There are two possible outcomes for these anchors:
a) the anchor is classified as a valid detection. Then the re-detection is successful. The detected bound box is considered the new appearance of the corresponding tracklet;
b) the anchor is classified as background. Then the re-detection has failed. The tracklet is considered halted, and released from the tracklet generator.
This process is illustrated in Fig.~\ref{fig_anchor_prop}
Though this process looks reasonable, one fundamental question needs to be answered for it to work.

\begin{figure}[thpb]
      \centering
      \includegraphics[width=1\linewidth]{figures/anchor-to-img2.png}
      \centering
      \caption{We show the re-detection results of a set of anchor boxes $A^t$ which are spatially closed. The corresponding bounding box $B^t$ is indicated using the same color with the anchor box on the feature map. The ground truth bounding box is represented using green color (view in color is suggested).}
      \label{fig_anchor_prop}
\end{figure}

\textbf{Is the prediction on the matched anchor always reliable?}
This question is critical to the success of the approach described above. To verify it, we conduct experiments on a small set of images. 
We look at the predictions of SSD on the anchors that are similar in terms of IoU to the ground truth bounding boxes. 
We find these nearby anchors' final prediction position all pointing to the final target as shown in Fig.~\ref{fig_anchor_prop}, the color pixels represents neiboring anchors, the dot bounding box represents the re-detected target position generated by the neighboring anchors.
Quantitative, we demonstrate in Fig.~\ref{fig_aug_ratio} how the initial shift of bounding box affect the ability correctly re-detect a face target.
Using the $w,h$ of the bounding box in $F_{i}$ as a scale, we added multiple ratios of shift to the Image for re-detection. Fig.~\ref{fig_aug_ratio} shows the re-detection results. We found that the detector can correctly localize the face position even with a 0.33 ratio of relative shift. This means if two consecutive frames only has a small motion, we can accurately track the faces in new frames.
We can observe that, similar to their counterpart in faster RCNN, the bounding box regressors in SSD also have strong power to rectify anchors to produce tight bounding boxes of the faces.
This means if we start from the anchor that are close the predict shape and location of a face, there is high chance of re-detecting this face and produce precise localization.

With the above question answered, we can formally describe the workflow of our tracklet generator as follow:
With the above question answered, we can formally describe the workflow of our tracklet generator as follow:

With the above question answered, we can formally describe the workflow of our tracklet generator as follow:

(1) For the first frame $F_0$: run the SSD detector and get all face detection results $\{B_i^0\}$. Create one tracklet ID for each bounding box.

(2) Predict the locations and shape of the bounding boxes in frame $F_{1}$. Find the closest anchors to $\{B_i^0\}$ by IoU-based bi-party matching. Mark these anchors as seed anchors for re-detection.

(3) Run the SSD detector on  frame $F_1$. Based on the prediction on the marked anchors, decide whether to extend or halt the tracklet. 

(4) Suppress all detection results that have high IoU to the tracked bounding boxes.

(5) For remaining detection results without tracklet, create a new tracklet for each of them.

(6) Repeat the steps 2-5 for the remaining frames one by one.





\begin{figure}[thpb]
      \centering
      \includegraphics[width=1\linewidth]{figures/hannah_anchor_test.png}
      \centering
      \caption{Detector anchor regression robustness test (a-d corresponds to shift ratio of 0.1, 0.25, 0.33, 0.45). Red is ground truth, yellow is the shifted box and green is the re-detected target position (view in color is suggested).}
      \label{fig_aug_ratio}
\end{figure}








\subsection{Motion Assisted Re-Detection}
The simplest form of our tracklet generator predicts the location of boxes in the next frame by identity mapping. 
This effectively relies on a static motion assumption.
When the video contains large motion, this assumption breaks and it becomes easy for the tracklet generator to loose the target. 
Thus, it is important to consider motion in bounding box prediction. 

Luckily, motion prediction from two consecutive frames is a widely investigated problem~\cite{ilg2017flownet, ilg2018occlusions}. 
Dense optical flow algorithms provide pixel-wise displacement prediction between two frames, which can be used to predict the bounding box movement. 
In this work we adopt the state of the art optical flow network, PWCNet~\cite{sun2018pwc} to prediction optical flow between two frames.
It uses deep network with correlation layers and warping layers to generate dense optical flow fields.
With the flow field, we predict a bounding box's location and shape using the following steps:


Run dense flow estimation for a pair of input images $F_{t-1}$ and $F_{t}$ and obtain the dense field, note as  $P_{t-1, x}, P_{t-1, y}$.

Obtain the ROI dense flow by cropping the dense flow field with current tracked face bounding box $B_j^t$, $b_{x}, b_{y}$ as the center of $B_j^t$.

Average the cropped dense flow, obtain position change $(\Delta{x}, \Delta{y} )$, this will be the difference to new bounding box center.

Compute the average vertical change of left/right half, horizontal change of upper/bottom half of the cropped dense flow, namely $P_{t-1, x| x<c_{x}}$, $P_{t-1, x| x>c_{x}}$, $P_{t-1, y| y<c_{y}}$ and $P_{t-1, y| y>c_{y}}$, this will be the left, right, top and bottom boundary change, noted as $\Delta b_{t}^j$.

Apply the boundary change to original bounding box, obtain the motion assisted bounding box estimation $B_j^t + \Delta b_{t}^j$. And use the new estimated bounding box to find seed anchor. 
We call this approach \emph{motion assisted tracklet generation}. In experiments, we empirically compare this approach with the static motion based tracklet generator and find it significantly improve the tracking performance.




\begin{figure}[thpb]
      \centering
      \includegraphics[width=1\linewidth]{figures/linking.png}
      \centering
      \caption{Online feature bank updating for tracklet-track linking. $\phi_i^t$ denotes identity embedding of track $i$ at time $t$. We store a bank of the identity embeddings where each track has one feature item. When we have a new tracklet, we either match it to an existing identity in the bank or add a new item to the bank (view in color is suggested).}
      \label{linking}
\end{figure}

\section{Online Linking of the Tracklets}
Output of tracklet generator are short tracklets, which are sensitive to occlusion and temporary disappearance of faces in videos. 
We design an online tracklet linker module to group the tracklets into the output of the FAR tracker, face tracks.

The linker carries a set of existing tracks $\{S_i\}$.
Each track is represented by an identity embedding $\phi(S_i)$ produced by averaging the output of a face embedding model $\phi$ on faces in this track.
At a certrain time step $t$, there are two possible states of a track: 1) this track is linked to an active tracklet in the current frame $F_t$; 2) this track has no active tracklet in the current frame. Fig. ~\ref{linking} illustrates the identity linking process.
For each new tracklet $L_j^t$ appearing in $F_t$, we try to match it with tracks in state 2). 
To perform the matching, we first extract the face identity embedding from the face in $L_j^t$, then we calculate the pairwise match cost between $L_j^t$ and the tracks. Once a track has a new tracklet linked to it, it goes to state 1) in the next frame. By solving the bi-party matching problem we can link the matching tracklet to the tracks. For those new tracklets that are not matched to any existing track, generate a new track for each of them. 


This tracklet linker handles the cases of occlusion and temporal disappearance by linking tracklet across time. 
On the other hand it requires extracting face embedding using an embedding model, usually a deep neural network like~\cite{deng2019arcface}, which adds to the runtime cost.
However, this additional cost can be minimal due to two reasons:
1) the linking is only performed for newly created tracklets, once a tracklet is linked we do not need to extract it embedding.
2) the extraction of face embedding for the track do not need to be run on each frame, as in videos the appereance of faces usually change slowly.





\section {Experiments}

We evaluate the proposed FAR tracker on two benchmark datasets: the Hannah dataset~\cite{ozerov2013evaluating} and the Music Videos dataset~\cite{zhang2016tracking}.
The tracker is assessed in term of standard multiple object tracking metrics, runtime speed, and metrics concerned by downstream tasks such as video face recognition.
We provide qualitative analysis of the tracker's ability in handling common nuance factors to face trackers.
\subsection{Implementation details}

In our implementation of FAR Tracker, we first employ a face detector which is based on Resnet34 as backbone. Feature pyramid network~\cite{lin2017feature} is adopted to enhance the detection performance. We customize the structure of the detector to allow marking anchors for redetection. So input of the detector will be an input image and a list of anchor indices. We choose detection confidence score $0.3$ as the confidence threshold for re-detection and $0.6$ as the confidence threshold for new detection.
In motion assisted re-detection, we use PWCNet ~\cite{sun2018pwc} to compute dense optical flow field. The resolution of the image for flow computing is $256$x$320$ (height vs. width). We choose this relatively small scale based on trade off between speed and prediction accuracy.  
In the identity linking, we employ a face embedding network~\cite{deng2019arcface} which generates 128-dimensional feature embedding. We threshold the euclidean distance between embedding on $0.97$ in the tracklet linker.

\subsection{Dataset}

Hannah dataset~\cite{ozerov2013evaluating} is a public dataset originally from the Hollywood movie \textit{Hannah and her sisters}. There are in total 153825 frames and 245 shots. As a muti-face tracking dataset, it contains 254 people and 53 of which are named characters. There are 2002 tracks annotated, ranging from 1 to 500 frames in length. In the Hannah dataset, face bounding box contains ears in the horizontal direction, and the bounding box starts from center of forehead and ends at center of the jaw in the vertical direction.

Music video dataset ~\cite{zhang2016tracking} is a widely used face tracking benchmark with $8$ high resolution music videos from YouTube. The videos are of high quality, the scenes are mainly music video or live concert recordings. This dataset is challenging for it contains a lot of fast moving humans, facial appearance changes, crowd occlusions.

IQIYI single face tracking dataset is a face tracking benchmark we create on top of the IQIYI Celebrity Video dataset~\cite{liu2018iqiyi}. IQIYI Celebrity Video dataset consists of 600K video clips. The dataset contains 5,000 labeled celebrities and 84,759 unknown identities (distractors for face recognition task) and the each video clip is labeled with one identity from the 5,000 celebrities. The dataset only clip-level label and no bounding box annotation. 
We manually annoate the bounding boxes in 2595 videos from the evaluation set of IQIYI datasets using Amazon Mechanical Turk. In the annotation process, we provide a pre-computed bounding box in WiderFace ~\cite{yang2016wider} format. Annotators were asked to to adjust bounding box location, add missing bounding box and remove false detection. Note that in this dataset each video contains exactly one person. 

\subsection{Evaluation Metric}
We treat multi-face tracking as a special case of multiple object tarcking (MOT), so the standard MOT metrics including Recall, Precision, F1, MOTA, MOTP, MT, Fragments are used for evaluation. 
Because one of the major downstream tasks for face tracking is face recognition, we design an additional evaluation metrics, called \textbf{identity miss-match} (ID-missmatch).
It counts how many times in the tracking results faces from different persons are falsely tracked in one track.
This metric is important for face recognition in the sense that the video recognition usually assumes faces in a track are from the same person.
For IQIYI dataset, since each video only contains one person's faces, we use the Recall, Precision, MT/PT/ML/GT, and MOTA, MOTP  as the evaluation metrics.

\subsection{Evaluation and analysis}
\textbf{Baseline approach} We implemented the widely used DeepSORT tracker~\cite{Wojke2018deep} as our baseline approach. It is based on the standard tracking by detection scheme. 
As shown in ~\cite{wojke2017simple}, the performance of the DeepSORT tracker is high dependent on the quality of its object detector and re-ID features.
We equip this tracker with state of the art image based face detector trained on the WIDERFace~\cite{yang2016wider} dataset and the state of the art face embedding~\cite{deng2019arcface} for re-identification features.
This makes it a strong baseline to compete against.
This tracker is called deep face SORT (DFS) in experimental results.


\begin{figure}[thpb]
      \centering
      \includegraphics[width=1\linewidth]{figures/annotation.png}
      \centering
      \caption{Fig. $(a)$ shows the different bounding box style of Hannah dataset and WiderFace, WiderFace have tight bounding box covering the face region while Hannah style have loosen bounding box in horizontal including ears but shorter in vertical which starts from forehead center to haw center. Fig. $(b)$ illustrates the rare case where shot changes but new face appear at the same region, this leads to identity miss match in tracking}
      \label{fig_dataset}
\end{figure}

\begin{table*}[t]

\begin{center}
\caption{Multi-face tracking evaluation results}
\label{tab:tab_hannah_eval}
\addtolength{\tabcolsep}{-1pt}
\begin{tabular}{c l c c c c c c c c c}
\hline
& Method & Recall$\uparrow$ & Precision$\uparrow$ & F1$\uparrow$ & MT/GT$\uparrow$ & ID missmatch$\downarrow$ & Frag$\downarrow$ & MOTA$\uparrow$ & MOTP$\uparrow$  & Online\\
\hline
\multirow{4}{*}{\rotatebox{90}{Hannah}}&DFS ~\cite{Wojke2018deep} &0.666 & 0.822  &0.732 & 499/864 & 94 & 807&0.27 &0.74 & Y \\
 &FAR &0.894 & 0.687 & 0.776 &604/864  &15  & 1349 &0.49 &0.75 & Y\\
 &FAR-flow & \textbf{0.894} & 0.686 & 0.775 & \textbf{607/864} & \textbf{15}& 1324 & 0.50& 0.75 & Y\\
 &FAR-link &0.842 & \textbf{0.820} & \textbf{0.830} & 481/864 & 35 & \textbf{602} & \textbf{0.66} & \textbf{0.78} & Y\\
\hline
\multirow{7}{*}{\rotatebox{90}{Musicvideo}} &Siamese ~\cite{zhang2016tracking}  & 0.715 & 0.894 & 0.795 & 18/44 & 986 & 2512 & 0.623 & 0.640 & N \\  
 &SymTriplets ~\cite{zhang2016tracking} & 0.718 & 0.897 & 0.798 & 19/44 & 699 & 2563 & 0.628 &0.643 & N\\
 &Prior-Less ~\cite{lin2018prior} & 0.817& \textbf{0.902} & 0.853 & 32/44 & 624 & \textbf{1645} & \textbf{0.692}& \textbf{0.860}& N \\
 &DFS~\cite{Wojke2018deep} & \textbf{0.816} & 0.873 & \textbf{0.843} &38/44  &362  &6548  &0.571 & 0.617& Y\\
 &FAR &0.788 &0.810  &0.798  &36/44  & 54 & 7081 &0.580& 0.683& Y\\
 &FAR-flow& 0.774 & 0.802 & 0.797 & \textbf{41/44}& \textbf{47}& 6985& 0.601 &0.699 & Y\\ 
 &FAR-link& 0.792 & 0.835 & 0.812 & 35/44& 208& 4038 & \textbf{0.668} & \textbf{0.727} & Y\\
 \hline
 \multirow{3}{*}{\rotatebox{90}{IQIYI}}& SiamMask ~\cite{wang2019fast} & 0.354& 0.368 &- & 473/3001 &- & -& -0.253 & 0.739 & N \\
 & DFS ~\cite{wojke2017simple} &0.927 & 0.898 & -& 1913/3001 &- &- &0.825 &0.873 & Y\\
 & FAR & \textbf{0.980} & \textbf{0.981} & -& \textbf{2646/3001} & -&- & \textbf{0.964}& \textbf{0.91} & Y\\
\hline
\end{tabular}
\end{center}

\end{table*}
\textbf{Bounding box transformation} Both Musicvideo and Hannah dataset have their own annotation styles for face bounding boxes. In face detection scenario, many of current face detector follow the annotation styles induced by the WIDERFace ~\cite{yang2016wider} data. In this format, the bounding box tightly contains forehead, cheek, chin. While in Hannah dataset, the bounding box contain looser bounding box including ears, shorter bounding box in the middle of forehead and middle of jaw. In Musicvideo dataset, the bounding box is tighter than WiderFace format in forehead boundaries. The difference between annotation styles is illustrated in figure ~\ref{fig_dataset}
In IoU based evaluation, which is standard for MOT benchmarks, this difference leads to undesired performance drop.
To make fair comparison between algorithms, in evaluaion we obtain landmark localization results of each output face bounding box and rebuild the bounding boxes according to the dataset specification.

\textbf{Results on the Hannah dataset}
Table ~\ref{tab:tab_hannah_eval} shows the Hannah dataset evaluation results based on baseline DFS and our proposed FAR tracklets generation, optical flow assisted tracklets and tracklets\& identity linking results. We found that even the raw tracklets generation results have better performance compared to the DFS methods. We can see that with a powerful face detector and face embedding extractor, the DFS can have a relative good face detection results as shown by the precision and recall number. While the approach have much larger miss matching number compared with the FAR series methods. We also see that applying optical flow did not change the original detection too much, but adding flow based motion compensation, the mostly tracked number and MOTP both increased, This means the optical flow do help the FAR tracker find more accurate target positions. We find that adding the identity link increase the miss match number, which is acceptable as merging tracklets will introduce miss matches due to false positive verification. But the MOTA and fragments metric have obvious improvements. 
We saw the significant improvement in identity miss match compared with DFS ~\cite{Wojke2018deep}, this means within a given tracklet, the FAR tracker can provide a set of face images with less incorrect id. This is very important in face recognition as outlier identity will generate different feature and harm the recognition. It is mainly contributed by the Re-detection mechanism of FAR tracker, the given dataset have a relative high frame rate so the assumption that most consecutive frames have small motion stands. This also explained why the flow assisted motion do not leads to much boost in detection quality. During the evaluation we also found that the FAR tracker is robust to shot change most of the time, it is rare to have a consecutive face across shots. But we do find some error cases, as shown in Fig. \ref{fig_dataset}$(b)$. This might be even problematic when there are crowds of face where it is easy to have different faces at same location across frames. Finding the correct corresponding faces in the challenging case will be the focus of our future work. 



\textbf{Results on the Musicvideo dataset}
We conduct the evaluation  on MusicVideo dataset. Musicvideo datasets are used for evaluation in ~\cite{zhang2016tracking, lin2018prior}'s work. We tested DFS~\cite{Wojke2018deep}, and FAR series methods on the MusicVideo datasets and show the result in table \ref{tab:tab_hannah_eval}.  


As shown in table \ref{tab:tab_hannah_eval}, compared to the state of the art multi-face tracker which multi-stage detection and tracklets association are conducted, our online methods provide comparable tracking results. In \cite{lin2018prior}'s work, multiple body parts detection leads to better detection quality as especially in recall metrics. Our proposed FAR tracker shows much less miss match while more fragments. This is mainly due to the global multi-step tracklets association mentioned in \cite{lin2018prior, zhang2016tracking} which tends to find most tracklets and merge to the same tracklets. In our proposed approaches, we use a very strict tracklets merging threshold. This results in a large fragment number in tracking, because we are trying to make sure multiple identity will not merge into one tracklet as we believe this is a very important requirement in video based face recognition. It is also important to notice that additional information is useful in improving the recall of video face detection.  







\textbf{IQIYI single face tracking evaluation}
To evaluate our approach, we compared the proposed FAR tracker with DFS \cite{Wojke2018deep} and the state of the art single object tracker SiamMask ~\cite{wang2019fast}. 
Table ~\ref{tab:tab_hannah_eval} shows the evaluation results on IQIYI dataset. The SiamMask~\cite{wang2019fast} single object tracker fails in most metrics. The main reason is lack of domain knowledge in face detection. Due to the task type, there is no multi-face linking problems need to be considered by DFS. This show the detector's basic performance in video detection. FAR tracker is running without flow assistance and identity linking. To our surprise, the FAR tracker can pick up most of the detection missed by the detector, improving recall from 92.7\% to 98.0\%. This also help the tracker generate more Mostly tracked tracklets. We also see that benefiting from the prior knowledge of face bounding box in previous frame, precision improves from 89.8\% to 98.1\% and mota/motp is boosted from 0.825/0.873 to 0.964/0.91 respectively. 

\subsubsection{Runtime analysis}

One of the main advantage of FAR is that it is an online multiple face tracker. To see if it is possible to make the FAR tracker running in realtime, we conduct a breakdown of the components runtime. All the benchmark is based on Nvidia V100 graphic card. With a input image size 1024 by 1024, the main components detection, tracker update, flow computing, mbedding computing take 126 $ms$, 9$ms$, 15$ms$, 14 $ms$ respectively. We can see that the detection take up the major part of the runtime. The current pipeline can achieve around 6 FPS in tracking. Note that tracklets generation module is independent to the based detector structures. It can be applied to any single-stage detector. All the current components can be replaced with a faster module in order to achieve realtime performance.  





\section{Conclusion}
In this paper, we present a two-stage face tracker that is able to generate fast, accurate and robust tracking results from videos. By applying the tracking by detection idea to single shot face detectors, its tracklet generator runs at constant speed w.r.t. the number of faces it is tracking. With motion assisted redetection, the tracker is robust to large motion in videos.
A simple but effective tracklet linkers links the tracklets into tracks in an online manner. 
In experiments, our FAR tracker achieves state-of-the-art accuracy and is robust to multiple nuance factors. 
Particularly, it is able detect and track the faces where the image based detector in three-stage trackers will fail.
Its high accuracy and robustness validates the effectiveness of our designing principle: simultaneous performing detection and tracking leads to better detection and better tracking.

{\small
\bibliographystyle{ieee}
