bibliography: plos_bib.bib

# Abstract

Human languages evolve continuously, and a puzzling problem is how to reconcile the apparent robustness of most of the deep linguistic structures we use with the evidence that they undergo possibly slow, yet ceaseless, changes. Is the state in which we observe languages today closer to what would be a dynamical attractor with statistically stationary properties or rather closer to a non-steady state slowly evolving in time? Here we address this question in the framework of the emergence of shared linguistic categories in a population of individuals interacting through language games. The observed emerging asymptotic categorization, which has been previously tested - with success - against experimental data from human languages, corresponds to a metastable state where global shifts are always possible but progressively more unlikely and the response properties depend on the age of the system. This aging mechanism exhibits striking quantitative analogies to what is observed in the statistical mechanics of glassy systems. We argue that this can be a general scenario in language dynamics where shared linguistic conventions would not emerge as attractors, but rather as metastable states.

# Introduction

A wide open question about the emergence and the evolution of shared linguistic conventions concerns the role of timescales . Phonetic, morphological, semantic, syntactic features of language vary over time. A fair degree of variability of words and grammatical structures can be observed in the diachronic study of a given language , and the proportions of different linguistic variants used by individuals within a population are not constant, but rather shift within time . An interesting case study is offered by linguistic categories, the classical and prototypical example being that of basic color terms , or simply "colors". For example, from Old to Middle English the meaning of color terms shifted progressively from a brightness meaning sense to the present-day hue sense , and similar shifts have been documented for a wide array of languages . Also the variability existing across different languages is an evidence of the continuous change of linguistic categories, as pointed out by the data gathered in the World Color Survey (WCS) . At the same time, statistical analysis over a large number of languages have shown that different color naming schemes share universal patterns , and, after a long debate, the existence of universalities is nowadays widely accepted .

General principles of categorization  have been claimed to be sufficient to account for the observed universality . For example, it has been suggested  that the simple principle according to which categories are constructed as to maximize similarity within categories and to minimize it across categories could be responsible for cross-linguistic similarity, and a quantitative analysis based on this intuition has confirmed its validity . Alternatively, it has been hypothesized that weak perceptual biases could do the job , and recent numerical simulations have shown that this might well be the case . In any case, both hypotheses have to deal with the existence of a large variability, which is reflected in the acknowledgment of the existence of non-optimal categorizations  and in the consideration of the weakness of the advocated perceptual biases . But if optimality is the leading principle, how can languages get stuck in suboptimal categorization schemes? Or what does it mean that perceptual biases are "weak"? i.e., why are they not able to drive the evolution towards the very same end? And, more in general, which is the mechanism that allow languages to appear static while they are yet evolving?

Here we focus on the emergence of shared categorization patterns in the framework of the so-called Category Game (CG) , a language game  through which a population of individuals establishes a shared categorization that quantitatively reproduces the average correlation among different human languages as measured in the World Color Survey experiment . A detailed analysis of the CG dynamics reveals that the physics of glassy systems  can be the proper framework to formalize the intuition that languages change at the same time because of and notwithstanding the fact that they are the outcome of a collective behavior . Languages are described as metastable states of global agreement, reconciling the evidence that they do continuously evolve  and they are at the same time stable enough to be intelligible across a population.

The physics of a so-called glass-forming liquid is such that when rapidly undercooled under its melting temperature, it looses it ability to flow on experimental time-scales and freezes in an amorphous state with huge rheological times, while the most stable state - the crystal - is never reached . This slowing down process can be quantified through the so-called relaxation time which turns out to be proportional to the viscosity of the fluid. Despite languages and glassy systems stand apparently very far apart, it is very intriguing to explore the analogy between a linguistic system and a dynamical system featuring glassy properties.

Within this perspective, we study the dynamics of the CG model with the tools of glass theory. In particular, we focus on the main aspects which are peculiar to these physical systems, i.e., the scaling of relaxation times and correlation functions with the population size as well as with the age of the system. In particular, the larger is the time over which one observes the system, formally known as the waiting time, the slower will be its response, i.e., its ability to undergo large-scale changes. From this perspective, the Category Game exhibits a glassy behavior and constitutes a first quantitative evidence of a very interesting link between cognitive science and the physics of glassy systems.

# Results

The Category Game  (see Materials and Methods for the details) describes the emergence of a hierarchical category structure made of two distinct levels: a basic layer, responsible for fine discrimination of the environment (perceptual categories), and a shared linguistic layer that groups together perceptions to guarantee communicative success (linguistic categories). At each time step a pair of individuals (one will be denoted as the speaker and the other as the hearer) is randomly selected from the population to play a language game that allow them to co-evolve the structure of their categories as well as their form-meaning inventories. Fig.  depicts a typical long-time configuration of the emerging category structure. While the number of perceptual categories (separated by short bars in fig.  ) is tuned by a parameter of the model (see Materials and Methods) and can be arbitrarily large, the number of linguistic categories (separated by long bars in fig.   and grouping together several perceptual categories sharing the same word) turns out to be finite and small, as observed in natural languages (for instance, like the basic color names across languages).

In the following, we report three sets of measures to establish the emergence of aging in the Category Game.

## Persistence of the Linguistic Categories

We start by investigating the dynamics of the number of linguistic categories emerging in the repertoire of each individual in a population. Two regimes are clearly distinguished (fig. a): initially, corresponding to a series of uncorrelated games, the average number of linguistic categories per individual $n_{ling}$ exhibits a rapid growth due to the pressure of discrimination (for a detailed description of CG we refer to the Materials and Methods section), followed by a rapid drop due to the onset of consensus and the merging of perceptual categories. A second regime is characterized by a quasi-arrested dynamics signaled by a "plateau" region, corresponding to a value of the average number of linguistic categories of the order of ten . Interestingly, the dependence of the number of linguistic categories on the population size $N$ is different in the two regimes. In the first one, the average number of linguistic categories scales with $\log{N}$ (see inset of fig. a), while in the second regime the dependence of the height of the plateau on the population size is extremely weak (O($\sqrt{\log{\log{(N)}}}$)): the average number of linguistic categories in the population remains limited to a small value (of the order of $20 \pm 10$) even for very large population sizes (up to billions of individuals). Furthermore, in the first regime we recover a time dependence on the population size of order $N^{3/2}$ (fig. a), with a similar behaviour as in the Naming Game , while the length of the plateau features a much stronger dependence on $N$, reaching a practically infinite value for large populations. At very large times, when the population is finite, the average number of linguistic categories starts to drop. We shall come back to this finite-size effect later on in the article. Most importantly, at the onset of the plateau region we observe a slowing down of the dynamics signaled by the divergence of the persistence time (fig. b). The plateau region is thus the interesting regime describing the persistence and evolution of the category system, and we will next describe its properties by looking at a more sophisticated dynamical quantity.

## Autocorrelation function and metastability

A system is said to be in dynamical equilibrium when it shows invariance under time translations; if this holds, any observable comparing the system at time $t_w$ with the system at time $t_w+t$ does not depend on $t_w$. In contrast, a system undergoing aging is not invariant under time translation, i.e., time is not homogeneous. This property can be revealed by measuring correlations of the system at different times. Here we consider a suitably defined autocorrelation function, which we term $C(t, t_w)$: at time $t_w$ we save a copy of the configuration of all the agents in the population and subsequently, at time instances greater than $t_w$, we compute the linguistic overlap of each agent with its copy saved at $t_w$; finally, we average this quantity over all agents (see Materials and Methods for detailed definitions). Results are presented in fig.  for two different population sizes. We recognize two different time scales, which we can associate to local or individual (fast) and collective or population-related (slow) dynamics. In particular, for $(t-t_w)/N<
10^3$, $C(t, t_w)$ depends (almost) only on $t-t_w$ (see inset of fig. a). This phenomenon corresponds to what is known in the physics of glassy systems as the $\beta$-relaxation regime. This fast dynamics corresponds to the microscopic dynamics of the boundaries between linguistic categories at the individual level.

On the other hand, for $(t-t_w)/N> 10^3$, $C(t, t_w)$ reveals aging, corresponding to the so-called $\alpha$-relaxation regime in glassy systems. This slow dynamics corresponds to the collective dynamics of the boundaries between linguistic categories at the population level. We find, in particular, for a fixed population size, a dependence on $t_w$ of the form:

$$C(t,t_w)=\tilde{C}(t^{1-\mu}-t_w^{1-\mu}) \, ,$$

with $\mu=0.75$ (see fig. b). Note that the same type of dependence on $t$ and $t_w$ is also found in correlation functions of real glasses , making quantitative the analogy of the CG dynamics with the dynamics of real out-of-equilibrium physical systems that exhibit sub-aging behavior. Let us simply recall that in the limit $\mu \rightarrow 1$ one recovers a pure aging behaviour since $t^{1-\mu}-t_w^{1-\mu}
\rightarrow log(t/t_w)$. In the opposite limit, $\mu \rightarrow 0$, one recovers invariance under time translations.

Fig.  also reveals that the dependence of the length of the plateau on the population size is of the order of $N^{\beta}$ with $\beta\simeq 4.5$. This suggests that the attractor of the dynamics, where a single linguistic category spreads over the whole interval, is practically never reached for large enough population sizes, and metastable states with a limited number of linguistic categories last for a practically infinite time.

## Finite-size effects

In this section, we consider finite-size effects. We focus, in particular, on the behaviour of the average number of linguistic categories as a function of time (as observed in fig. a). A careful observation reveals that for very long times the plateau behaviour leaves room for a bending of the curves leading to a reduction in the average number of linguistic categories. This bending occurs earlier for small populations, i.e., for small system sizes. Fig.  shows the collapse of the curves for the average number of linguistic categories for different system sizes from $N=100$, for which the bending is stronger, to $N=800$. The collapse is aimed at superimposing only the bending region. It turns out that one collapses the curves after a rescaling of the abscissa as $t \rightarrow
b(t/N^{\alpha} - \tau/N^{\beta})^{c}$, where the term $\tau/N^{\beta}$ allows to superimpose the onset of the bending region while the term $t/N^{\alpha}$ is the time rescaling well inside the bending region and $b$ is a constant. The value of $\beta\simeq 4.5$ is consistent with what is observed in the collapse of the correlation function (see fig. ) and confirms the idea that the length of the plateau region is scaling with a large power of the system size. On the other hand the bending region exhibits a characteristic time scaling as $N^{\alpha}$ with $\alpha\simeq 3$ and the overall behaviour is well fitted by a stretched exponential function with an exponent $c$ close to $1/3$.

The onset of the bending region is marked by a clear phenomenon occurring in the structure of the perceptual and linguistic categories. In the plateau region discrimination keeps taking place, though at a slow pace, while at the onset of the bending region discrimination ceases and one is left with a pure dynamics of the boundaries between linguistic categories (see fig. ). A detailed description of the dynamics of the domain boundaries in the plateau and in the bending region is out of the scope of the present paper and it will be presented elsewhere. It is nevertheless interesting to mention that one can describe the dynamics of the domain boundaries between linguistic categories in terms of correlated random walkers. The crucial difference between the plateau region (where aging occurs) and the bending region is that when the system ages the number of perceptual categories, which represent the underlying lattice where the random walkers can diffuse, is an increasing function of time while it is a constant when finite-size effects start.

# Discussion

In summary, our *in silico* experiment demonstrates a strong analogy between the slow dynamics of the Category Game and that of a suddenly quenched glass-former, in many different crucial aspects. The relaxation time of the number of linguistic categories shows a huge increase, similar to a singularity, at a finite number of categories: the system traps itself in a metastable state and the dynamics appears arrested even though the final state (only one category) is far from being reached. The dependence of the number of linguistic categories on the population size appears to be extremely weak (slower than a logarithm), and this could account for the universality of the number of color names (between $2$ and $11$) among different languages. The decorrelation of the system, apart from being slow, is also age-dependent, thus suggesting a possible explanation for the existence of more stable conserved properties of a language.

Taking a wider perspective, our results suggest that glassy behavior could provide important conceptual and technical tools to address the general problem of language change from a new perspective. Metastability, for example, allows to unfold the seemingly paradoxical nature of language change, according to which languages evolve because they are spoken by a large number of speakers but the evolution is frustrated since the speakers are indeed many. These intuitions are of course not a novelty (see for instance ), but for the first time they have been properly quantified, in a numerical model closely connected to experimental data, and addressed in a well established framework. Future work could take into account crucial phenomena like language contact or more general cultural evolution processes, thus paving the way for a comparison with true historical data.

# Materials and Methods

### The Category Game

The computational model used for this study, introduced in , investigates how a population of individuals can develop a shared repertoire of linguistic categories, i.e., co-evolve their own system of symbols and meanings, without any pre-defined categorization and by only means of elementary language games . A population of $N$ artificial agents is considered and to each agent (or individual) a continuous perceptual space (e.g., the visible light spectrum) is associated which, without any loss of generality, is assumed to be the interval $[0, 1)$. A categorization pattern refers to a partition of this interval into sub-intervals, or perceptual categories. Each individual has a dynamical form-meaning repertoire linking perceptual categories (meanings) to words (forms) representing their linguistic counterpart. The perceptual categories and the words associated with them co-evolve through a sequence of elementary language games among the agents. Initially, all individuals have a single perceptual category $[0, 1)$ and no name associated to it.

At each time step a pair of individuals (one will be denoted as the speaker and the other as the hearer) is randomly selected from the population and based on the success or failure of communication, both rearrange their form-meaning inventories. Both the speaker and the hearer are presented with a scene made of $M \ge 2$ [^1] stimuli (objects), where a stimulus is a real number in the interval $[0, 1)$. Any two objects in the scene cannot appear at a distance closer than $d_{min}$: this is the only parameter of the model, fixing a minimal length scale which encodes a non infinite resolving power of any perception, for instance, the human Just Noticeable Difference (see below) in the case of colors. One of the objects is randomly selected to be the *topic* of the game and is known only to the speaker. The speaker checks whether the topic is the unique stimulus corresponding to one of its perceptual categories. If the two stimuli fall in one perceptual category, then the category is divided into two new categories by a barrier located in the center of the segment connecting the two stimuli. Both the new categories inherit the words associated to the original category plus a new word; this process is termed as *discrimination*. Subsequently, the speaker utters the most relevant name of the category containing the topic, where the most relevant name corresponds to either the last name used in a winning game or the new name in case the category has just been created. If the hearer does not have a category with this name, the game is a failure. If the hearer recognizes the name and has any object in one or more categories associated with that name, then it picks randomly one of these objects. If the object picked is the topic, then the game is a success; otherwise, it is a failure. In case of failure, the hearer learns the name used by the speaker for the category corresponding to the topic. In case of success, that name becomes the most relevant for that category and all other competing names are removed from the inventory associated with the category for both the players. An example of the evolving dynamics is shown in fig. .

It is worth to remark that in the framework of the CG model it was recently possible to reproduce the outcomes of the World Color Survey (WCS) . This is a first evidence that the model can account for the *universality* of categorization patterns across cultures by means of only weak constraints on the perceptual space of the individuals. In , universal categorization patterns have been identified among populations whose individuals are endowed with the human Just Noticeable Difference (JND) function, describing the resolution power of the human eye to variations in the wavelength of the incident light . In the simulations presented here, for the sake of simplicity, we adopt a constant $d_{min}$ equal to the average human JND ($0.0143$), after having checked that the dynamical properties do not depend on eventual modulations of the JND function.

### A fast implementation of the Category Game

In this paper, for the investigation of the long-time dynamics of the model we devised and adopted a fast version of the Category Game that we briefly describe here, referring to a forthcoming paper for further details. The central idea behind the fast algorithm we implement is that of avoiding all the unnecessary games without outcomes (i.e., without changes in the configuration of either the speaker or the hearer or both) by forcing that each two players' game has an outcome and rescaling time accordingly. In this way, we do not alter the original dynamics since we conserve the playing order of the pairs speaker-hearer as well as the probability of playing in a given region.

More concretely, at each step we extract one pair speaker-hearer according to the probability $P_{out}$ that an outcome will result if the pair plays, where an outcome is defined as any change in the repertoire of the speaker and/or the hearer[^2]. Furthermore, the region for placing the topic is extracted according to the probability that this choice will produce an outcome, and the object is extracted consistently. After each game time is increased by $1/P_{out}$. In this way time becomes a dependent variable. We checked that this fast implementation of the Category Game features the same dynamical properties of the original model for all the quantities of interest.

### Dynamical properties of the Category Game

In the Category Game dynamics it is possible to distinguish two different phases. In the first regime, the number of perceptual categories increases due to the pressure of discrimination, and at the same time many different words are used by different agents for naming similar perceptual categories. This kind of synonymy reaches a peak and then drops  in a fashion similar to the well-known Naming Game . A second phase starts when most of the perceptual categories are associated with only one word (see fig. ). During this phase, words are found to expand their dominion across adjacent perceptual categories. In this way, sets of contiguous perceptual categories sharing the same words are formed, giving raise to what we define as *linguistic categories* (see fig. ). An important outcome thus is the emergence of a hierarchical category structure made of two distinct levels: a basic layer, responsible for fine discrimination of the environment, and a shared linguistic layer that groups together perceptions to guarantee communicative success. Remarkably, the emergent number of linguistic categories in this phase turns out to be finite and small , as observed in natural languages, even in the limit of an infinitesimally small length scale $d_{min}$, as opposed to the number of the underlying perceptual categories which is of order $1/d_{min}$.

### Linguistic overlap and autocorrelation function

The autocorrelation function $C(t, t_w)$ is defined as the average in the population of the individual linguistic autocorrelation, which, in turn, is defined as the overlap of the linguistic categories  of the considered individual at time $t_w$, with itself at a later time $t$. It then reads: $$\label{eq:ov}
  C(t,t_w) = \sum_{i} \frac{O_{i}(t,t_w)}{N} \textrm{ with }
  O_{i}(t,t_w) = \frac{2\sum_{c_{i(t)}^{i(t_w)}} {(l_{c_{i(t)}^{i(t_w)}})}^2}
  {\sum_{c_{i(t)}} {(l_{c_{i(t)}})}^2 + \sum_{c_{i(t_w)}} {(l_{c_{i(t_w)}})}^2}$$ where $l_c$ is the width of the linguistic category $c$, $c_{i(t)}$ is a linguistic category of the $i^{th}$ agent at time $t$ and $c_{i(t)}^{i(t_w)}$ is the generic category of the intersection set containing all of the linguistic category boundaries of the agent $i$ at time $t$ and its previous image saved at $t_w$. The function $O_{i} (t,t_w)$ returns a value proportional to the degree of alignment of the two category inventories reaching its maximum unitary value when they are perfectly aligned.

# Acknowledgments

A. Baronchelli acknowledges support from the Spanish Ministerio de Ciencia e Innovación through the Juan de la Cierva program, as well as from project FIS2007-66485-C02-01 (Fondo Europeo de Desarrollo Regional), and from the Junta de Andalucía project P09-FQM4682. A. Puglisi acknowledges support by the Italian MIUR under the FIRB-IDEAS grant RBID08Z9JE.

# Figures and Legends

[^1]: Without any loss of generality we will use in all the simulations $M=2$.

[^2]: In the considered dynamics, we found that the probabilities $P_{out}$ of all the pairs, at each given time, followed a peaked distribution, so that we could ultimately randomly extract one pair at each step without altering the results and significantly reducing the computational complexity.


abstract: The common internal structure and algorithmic organization of object detection, detection-based tracking, and event recognition facilitates a general approach to integrating these three components. This supports multidirectional information flow between these components allowing object detection to influence tracking and event recognition and event recognition to influence tracking and object detection. The performance of the combination can exceed the performance of the components in isolation. This can be done with linear asymptotic complexity.
author: Andrei Barbu, Aaron Michaux, Siddharth Narayanaswamy, and Jeffrey Mark Siskind
bibliography: arxiv2012a.bib
date: Purdue University
  School of Electrical and Computer Engineering
  465 Northwester Avenue
  West Lafayette IN 47907-2035 USA
  `firstname.lastname@example.com,{amichaux,snarayan,email@example.com`
title: Simultaneous Object Detection, Tracking, and Event Recognition

# Introduction

Many common approaches to event recognition classify events based on their motion profile. This requires detecting and tracking the event participants. Adaptive approaches to tracking , e.glet@token.. Kalman filtering , suffer from three difficulties that impact their utility for event recognition. First, they must be initialized. One cannot initialize on the basis of motion since many event participants move only for a portion of the event, and sometimes not at all. Second, they exhibit drift and often must be periodically reinitialized to compensate. Third, they have difficulty tracking small, deformable, or partially occluded objects as well as ones whose appearance changes dramatically. This is particularly of concern since many events, e.glet@token.. picking things up, involve humans interacting with objects that are sufficiently small for humans to grasp and where such interaction causes appearance change by out-of-plane rotation, occlusion, or deformation.

Detection-based tracking is an alternate approach that attempts to address these issues. In detection-based tracking an object detector is applied to each frame of a video to yield a set of candidate detections which are composed into tracks by selecting a single candidate detection from each frame that maximizes temporal coherency of the track. However, current object detectors are far from perfect. On the PASCAL VOC Challenge, they typically achieve average precision scores of 40% to 50% . Directly applying such detectors on a per-frame basis would be ill-suited to event recognition. Since the failure modes include both false positives and false negatives, interpolation does not suffice to address this shortcoming. A better approach is to combine object detection and tracking with a single objective function that maximizes temporal coherency to allow object detection to inform the tracker and vice versa.

One can carry this approach even further and integrate event recognition with both object detection and tracking. One way to do this is to incorporate coherence with a target event model into the temporal coherency measure. For example, a top-down expectation of observing a *pick up* event can bias the object detector and tracker to search for event participants that exhibit the particular joint motion profile of that event: an object in close proximity to the agent, the object starting out at rest while the agent approaches the object, then the agent touching the object, followed by the object moving with the agent. Such information can also flow bidirectionally. Mutual detection of a *baseball bat* and a *hitting* event can be easier than detecting each in isolation or having a fixed direction of information flow.

The common internal structure and algorithmic organization of current object detectors , detection-based trackers , and HMM-based approaches to event recognition facilitates a general approach to integrating these three components. We demonstrate an approach to integrating object detection, tracking, and event recognition and show how it improves each of the these three components in isolation. Further, while prior detection-based trackers exhibit quadratic complexity, we show how such integration can be fast, with linear asymptotic complexity.

# Detection-based tracking

The methods described in sections , , and extend a popular dynamic-programming approach to detection-based tracking. We review that approach here to set forth the concepts, terminology, and notation that will be needed to describe the extensions.

Detection-based tracking is a general framework where an object detector is applied to each frame of a video to yield a set of candidate detections which are composed into tracks by selecting a single candidate detection from each frame that maximizes temporal coherency of the track. This general framework can be instantiated with answers to the following questions:

We answer questions  and  by taking a detection to be a scored axis-aligned rectangle (box), such as produced by the object detectors, though our approach is compatible with any method for producing scored axis-aligned rectangular detections. If $b^t_j$ denotes the $j$th detection in frame $t$, $f(b^t_j)$ denotes the score of that detection, $T$ denotes the number of frames, and $\mathbf{j}=\langle j_1,\ldots,j_T\rangle$ denotes a track comprising the $j_t$th detection in frame $t$, we answer question  by formulating temporal coherency of a track $\mathbf{j}=\langle j_1,\ldots,j_T\rangle$ as: $$\max_{j_1,\ldots,j_T}
  \sum_{t=1}^T f(b^t_{j_t})+
  \sum_{t=2}^T g(b^{t-1}_{j_{t-1}},b^t_{j_t})
\label{eq:track}$$ where $g$ scores the local temporal coherency between detections in adjacent frames. We take $g$ to be the negative Euclidean distance between the center of $b^t_{j_t}$ and the center of $b^{t-1}_{j_{t-1}}$ projected forward one frame, though, as discussed below, our approach is compatible with a variety of functions discussed by . The forward projection internal to $g$ can be done in a variety of ways including optical flow and the Kanade-Lucas-Tomasi (KLT) feature tracker. We answer question  by observing that Eq.  can be optimized in polynomial time with the Viterbi algorithm : $$\begin{array}[t]{@{}l@{}}
    \textbf{for}\;j=1\;\textbf{to}\;J_1\;
    \textbf{do}\;\delta^1_j\mathop{\mathrm{:\!=}}f(b^1_j)\\
    \textbf{for}\;t=2\;\textbf{to}\;T\\
    \textbf{do}\;\begin{array}[t]{@{}l@{}}
         \textbf{for}\;j=1\;\textbf{to}\;J_t\\
         \textbf{do}\;\delta^t_j\mathop{\mathrm{:\!=}}
                   f(b^t_j)
                   +\displaystyle
                \max_{j'=1}^{J_{t-1}} g(b^{t-1}_{j'},b^t_j)
                           +\delta^{t-1}_{j'}
         \end{array}
  \end{array}
  \label{eq:viterbi}$$ where $J_t$ is the number of detections in frame $t$. This leads to a lattice as shown in Fig. .

Detection-based trackers exhibit less drift than adaptive approaches to tracking due to fixed target models. They also tend to perform better than simply picking the best detection in each frame. The reason is that one can allow the detection source to produce multiple candidates and use the combination of the detection score $f$ and the adjacent-frame temporal-coherency score $g$ to select the track. The essential attribute of detection-based tracking is that $g$ can overpower $f$ to assemble a more coherent track out of weaker detections. The nonlocal nature of Eq.  can allow more-reliable tracking with less-reliable detection sources.

A crucial practical issue arises: *How many candidate detections should be produced in each frame?* Producing too few may risk failing to produce the desired detection that is necessary to yield a coherent track. In the limit, it is impossible to construct any track if even a single frame lacks any detections.[^1] The current state-of-the-art in object detection is unable to simultaneously achieve high precision and recall and thus it is necessary to explore the trade-off between the two . A detection-based tracker can bias the detection source to yield higher recall at the expense of lower precision and rely on temporal coherency to compensate for the resulting lower precision. This can be done in at least three ways. First, one can depress the detection-source acceptance thresholds. One way this can be done with the Felzenszwalb et allet@token.. detectors is to lower the trained model thresholds. Second, one can pool the detections output by multiple detection sources with complementary failure modes. One way this can be done is by training multiple models for people in different poses. Third, one can use adaptive-tracking methods to project detections forward to augment the raw detector output and compensate for detection failure in subsequent frames. This can be done in a variety of ways including optical flow and KLT. The essence of our paper is a more principled collection of approaches for compensating for low recall in the object detector.

A practical issue arises when pooling the detections output by multiple detection sources. It is necessary to normalize the detection scores for such pooled detections by a per-model offset. One can derive an offset by computing a histogram of scores of the top detection in each frame of a video and taking the offset to be the minimum of the value that maximizes the between-class variance when bipartitioning this histogram and the trained acceptance threshold offset by a small but fixed amount.

The operation of a detection-based tracker is illustrated in Fig. . This example demonstrates several things of note. First, reliable tracks are produced despite an unreliable detection source. Second, the optimal track contains detections with suboptimal score. Row (b) demonstrates that selecting the top-scoring detection does not yield a temporally-coherent track. Third, forward-projection of detections from the second to third column in row (c) compensates for the lack of raw detections in the third column of row (a).

```latex
\begin{figure*}[t]\begin{center}
    \begin{tabular}{@{}c@{\hspace*{10pt}}c@{\hspace*{2pt}}c@{\hspace*{2pt}}c@{}}
      \raisebox{15pt}{(a)}&
      \includegraphics[width=0.3\textwidth]{images/viterbi-a-1}&
      \includegraphics[width=0.3\textwidth]{images/viterbi-a-2}&
      \includegraphics[width=0.3\textwidth]{images/viterbi-a-3}\\
      \raisebox{15pt}{(b)}&
      \includegraphics[width=0.3\textwidth]{images/viterbi-b-1}&
      \includegraphics[width=0.3\textwidth]{images/viterbi-b-2}&
      \includegraphics[width=0.3\textwidth]{images/viterbi-b-3}\\
      \raisebox{15pt}{(c)}&
      \includegraphics[width=0.3\textwidth]{images/viterbi-c-1}&
      \includegraphics[width=0.3\textwidth]{images/viterbi-c-2}&
      \includegraphics[width=0.3\textwidth]{images/viterbi-c-3}\\
      \raisebox{15pt}{(d)}&
      \includegraphics[width=0.3\textwidth]{images/viterbi-d-1}&
      \includegraphics[width=0.3\textwidth]{images/viterbi-d-2}&
      \includegraphics[width=0.3\textwidth]{images/viterbi-d-3}
    \end{tabular}
  \end{center}
  %
  \caption{The operation of a detection-based tracker.
    %
    (a)~Output of the detection sources, biased to yield false positives.
    %
    (b)~The top-scoring output of the detection source.
    %
    (c)~Augmenting the output of the detection sources with forward-projected
    detections.
    %
    (d)~The optimal tracks selected by the Viterbi algorithm.}
  %
  \label{fig:tracker}
\end{figure*}
```

Detection-based tracking runs in time $O(TJ^2)$ on videos of length $T$ with $J$ detections per frame. In practice, the run time is dominated by the detection process and the dynamic-programming step. Limiting $J$ to a small number speeds up the tracker considerably while minimally impacting track quality. We further improve the speed of the detectors when running many object classes by factoring the computation of the HOG pyramid.

# Evaluation of detection-based tracking

We evaluated detection-base tracking using the year-one (Y1) corpus produced by DARPA for the Mind's Eye program. These videos are provided at 720p@30fps and range from 42 to 1727 frames in length, with an average of 438.84 frames, and depict people interacting with a variety of objects to enact common English verbs.

Four Mind's Eye teams (University at Buffalo, , Stanford Research Institute, , University of California at Berkeley, , and University of Southern California, ) independently produced human-annotated tracks for different portions of Y1. We used these sources of human-annotated tracks to evaluate the performance of detection-based tracking by computing human-human intercoder agreement between all pairs of the four sources of human-annotated tracks and human-machine intercoder agreement between a detection-based tracker and all four of these sources. Since each team annotated different portions of Y1, each such intercoder agreement measure was computed only over the $N$ videos shared by each pair, as reported in Table (a). One team (University at Buffalo, ) annotated detections as clusters of quadrilaterals around object parts. These were converted to a single bounding box.

Different teams labeled the tracks with different class labels. It was possible to determine from these labels whether the track was for a person or nonperson by assuming that the labels 'person' and 'human', and only those labels, denoted person tracks, but it was not possible to automatically make finer-grained class comparisons. Thus we independently compared person tracks with person tracks and nonperson tracks with nonperson tracks. When comparing an annotation $u$ of a video $n$ containing $l^u_n$ person tracks with an annotation $v$ of that same video containing $l^v_n$ person tracks, we compared $\lfloor l^u_n,l^v_n\rfloor$ person tracks. We selected the best over all $\lfloor l^u_n,l^v_n\rfloor!$ permutation mappings $\rho_n$ between person tracks in $u$ and person tracks in $v$. A permutation mapping was preferred when it had higher average overlap score among corresponding boxes across the tracks and the frames in a video, where the overlap score was that used by the PASCAL VOC Challenge , namely the ratio of the area of their intersection to the area of their union. Different tracks could annotate different frames of a video. When comparing such, we considered only the shared frames.

For every pair of teams, we computed the mean and standard deviation of the overlap score across all shared frames in all tracks in the best permutation mappings for all shared videos. The averaging process used both to determine the best permutation mapping for each video pair and to determine overall mean and standard deviation measures weighted each overlap score equally. More precisely, if $1\leq n\leq N$, $1\leq l\leq\lfloor l^u_n,l^v_n\rfloor$ denotes a shared track for video $n$, $T^l_n$ denotes the set of shared frames for that shared track $l$ in video $n$, $U^n_t$ and $V^n_t$ denote the vector of boxes for frame $t$ in video $n$ for annotations $u$ and $v$ respectively, and $O$ denotes the overlap measure, we score a permutation mapping $\rho_n$ for video $n$ as: $$\cfrac{1}{\displaystyle\sum_{l=1}^{\lfloor l^u_n,l^v_n\rfloor}\lvert T^l_n\rvert}
  \sum_{l=1}^{\lfloor l^u_n,l^v_n\rfloor}\sum_{t\in T^l_n}
  O(\rho_n (U^n_t)[l],V^n_t[l])$$ and computed the mean overlap for a pair of teams as: $$\cfrac{1}{\displaystyle
    \sum_{n=1}^N\sum_{l=1}^{\lfloor l^u_n,l^v_n\rfloor}\lvert T^l_n\rvert}
  \sum_{n=1}^N\sum_{l=1}^{\lfloor l^u_n,l^v_n\rfloor}\sum_{t\in T^l_n}
  O(\rho_n (U^n_t)[l],V^n_t[l])$$ with an analogous computation for standard deviation and nonperson tracks.

The overall mean and standard deviation measures, reported in Table (b,c), indicate that the mean human-human overlap is only marginally greater than the mean human-machine overlap by about one standard deviation. This suggests that improvement in tracker performance is unlikely to lead to significant improvement in action recognition performance and sentential description quality.

```latex
\begin{table*}
    \begin{center}
      \scalebox{0.98}{
      \begin{tabular}{@{}ccc@{}}
    \begin{tabular}{l|rrrr}
      $N$ & UB & SRI  & UCB  & USC \\\hline
      UB  &    & 8    & 20   & 8   \\
      SRI & 8  &      & 1201 & 95  \\
      UCB & 20 & 1201 &      & 204 \\
      USC & 8  & 95   & 204  &     \\\hline
      us  & 48 & 1254 & 1829 & 360 \\
    \end{tabular}&
    \begin{tabular}{l|rrrr}
      $\mu$ & UB   & SRI  & UCB  & USC  \\\hline
      UB    &      & 0.76 & 0.68 & 0.59 \\
      SRI   & 0.76 &      & 0.55 & 0.59 \\
      UCB   & 0.68 & 0.55 &      & 0.48 \\
      USC   & 0.59 & 0.59 & 0.48 &      \\\hline
      us    & 0.54 & 0.40 & 0.35 & 0.43 \\
    \end{tabular}&
    \begin{tabular}{l|rrrr}
      $\sigma$ & UB   & SRI  & UCB  & USC  \\\hline
      UB       &      & 0.06 & 0.14 & 0.10 \\
      SRI      & 0.06 &      & 0.27 & 0.16 \\
      UCB      & 0.14 & 0.27 &      & 0.23 \\
      USC      & 0.10 & 0.16 & 0.23 &      \\\hline
      us       & 0.26 & 0.24 & 0.23 & 0.20 \\
    \end{tabular}\\
    (a) & (b) & (c)\\
      \end{tabular}}
    \end{center}
  %
  \caption{(a)~The number of videos in common, (b)~the mean overlap, and (c)~the
    standard deviation in overlap between each pair of annotation sources. }
  %
  \label{tab:intercoder-agreement}
\end{table*}
```

# Combining object detection and tracking

While detection-based tracking is resilient to low precision, it requires perfect recall; it cannot generate a track through a frame that has no detections and it cannot generate a track through a portion of the field of view which has no detections regardless of how good the temporal-coherence of the resulting track would be. This brittleness means that any detection source employed will have to significantly over-generate detections to achieve near-perfect recall. This has a downside. While the Viterbi algorithm has linear complexity in the number of frames, it is quadratic in the number of detections per frame. This drastically limits the number of detections that can reasonably be processed leading to the necessity of tuning the thresholds on the detection sources. We have developed a novel mechanism to eliminate the need for a threshold and track every possible detection, at every position and scale in the image, in time linear in the number of detections and frames. At the same time our approach eliminates the need for forward projection since every detection is already present. Our approach involves simultaneously performing object detection and tracking, optimizing the joint object-detection and temporal-coherency score.

Our general approach is to compute the distance between pairs of detection pyramids for adjacent frames, rather than using $g$ to compute the distance between pairs of individual detections. These pyramids represent the set of all possible detections at all locations and scales in the associated frame. Employing a distance transform makes this process linear in the number of location and scale positions in the pyramid. Many detectors, e.glet@token.. those of Felzenszwalb et allet@token.., use such a scale-space representation of frames to represent detections internally even though they might not output such. Our approach requires instrumenting such a detector to provide access to this internal representation.

At a high-level, the Felzenszwalb et allet@token.. detectors learn a forest of HOG filters for each object class along with their characteristic displacements. Detection proceeds by applying each HOG filter at every position in an image pyramid followed by computing the optimal displacements at every position in that image pyramid, thereby creating a new pyramid, the detection pyramid. Finally, the detector searches the detection pyramid for high-scoring detections and extracts those above a threshold. The detector employs a dynamic-programming algorithm to efficiently compute the optimal part displacements for the entire image pyramid. This algorithm is very similar to the Viterbi algorithm. It is made tractable by the use of a generalized distance transform that allows it to scale linearly with the number of image pyramid positions. Given a set $\mathcal{G}$ of points (which in our case denotes an image pyramid), a distance metric $d$ between pairs of points $p$ and $q$, and an arbitrary function $\phi:\mathcal{G}\rightarrow\Re$, the generalized distance transform $D_{\phi}(q)$ computes: $$D_{\phi}(q)=\min_{p\in\mathcal{G}}(d(p,q)+\phi(q))$$ in linear time for certain distance metrics including squared Euclidean distance.

Instead of extracting and tracking just the thresholded detections, one can directly track all detections in the entire pyramid simultaneously by defining a distance measure between detection pyramids for adjacent frames and performing the Viterbi tracking algorithm on these pyramids instead of sets of detections in each frame. To allow comparison between detections at different scales in the detection pyramid, we convert the detection pyramid to a rectangular prism by scaling the coordinates of the detections at scale $s$ by $\pi(s)$, chosen to map the detection coordinates back to the coordinate system of the input frame. We define the distance between two detections, $b$ and $b'$, in two detection pyramids as a scaled squared Euclidean distance: $$d(b_{xys},b'_{x'y's'})=
  \begin{array}[t]{ll}
    (\pi(s)x-\pi(s')x')^2\\
    {}+(\pi(s)y-\pi(s')y')^2\\
    {}+\alpha(s-s')^2
  \end{array}
  \label{eq:distance}$$ where $x$ and $y$ denote the original image coordinates of a detection center at scale $s$. Nominally, detections are boxes. Comparing two such boxes involves a four-dimensional distance metric. However, with a detection pyramid, the aspect ratio of detections is fixed, reducing this to a three-dimensional distance metric. The coefficient $\alpha$ in the distance metric weights a difference in detection area differently than detection position.

The above amounts to replacing detections $b^t_j$ with $b^t_{xys}$, lattice values $\delta^t_j$ with $\delta^t_{xys}$, and Eq.  with: $$\begin{array}[t]{@{}l@{\hspace*{-10pt}}}
    \textbf{for}\;x=1\;\textbf{to}\;X\\
    \textbf{do}\;\begin{array}[t]{@{}l@{}}
         \textbf{for}\;y=1\;\textbf{to}\;Y\\
         \textbf{do}\;\begin{array}[t]{@{}l@{}}
                  \textbf{for}\;s=1\;\textbf{to}\;S\;
                  \textbf{do}\;\delta^1_{xys}\mathop{\mathrm{:\!=}}
                  f(b^1_{xys})
                  \end{array}
         \end{array}\\
    \textbf{for}\;t=2\;\textbf{to}\;T\\
    \textbf{do}\;\begin{array}[t]{@{}l@{}}
         \textbf{for}\;x=1\;\textbf{to}\;X\\
         \textbf{do}\;\begin{array}[t]{@{}l@{}}
                  \textbf{for}\;y=1\;\textbf{to}\;Y\\
                  \textbf{do}\;\begin{array}[t]{@{}l@{}}
                       \textbf{for}\;s=1\;\textbf{to}\;S\\
                       \textbf{do}\;
                       \delta^t_{xys}\mathop{\mathrm{:\!=}}
                       \begin{array}[t]{@{}l@{}}
                       f(b^t_{xys})\\
                       +\displaystyle
                       \max_{x',y',s'}
                       g(b^{t-1}_{x'y's'},b^t_{xys})
                       +\delta^{t-1}_{x'y's'}
                       \end{array}
                       \end{array}
                  \end{array}
         \end{array}
  \end{array}
  \label{eq:felzenszwalb-viterbi}$$

The above formulation allows us to employ the generalized distance transform as an analog to $g$ in Eq. , although it restricts consideration of $g$ to be squared Euclidean distance rather than Euclidean distance. We avail ourselves of the fact that the generalized distance transform operates independently on each of the three dimensions $x$, $y$, and $s$ in order to incorporate $\alpha$ into Eq. . While linear-time use of the distance transform restricts the form of $g$, it places no restrictions on the form of $f$.

One way to view the above is that the vector of $\delta^t_j$ for all $1\leq j\leq J_t$ from Eq.  is being represented as a pyramid and the loop: $$\begin{array}[t]{@{}l@{}}
    \begin{array}[t]{@{}l@{}}
      \textbf{for}\;j=1\;\textbf{to}\;J_t\\
      \textbf{do}\;\delta^t_j\mathop{\mathrm{:\!=}}
      f(b^t_j)
      +\displaystyle
      \max_{j'=1}^{J_{t-1}} g(b^{t-1}_{j'},b^t_j)
      +\delta^{t-1}_{j'}
    \end{array}
  \end{array}$$ is being performed as a linear-time construction of a generalized distance transform rather than a quadratic-time nested pair of loops. Another way to view the above is that we generalize the notion of a detection pyramid from representing per-frame detections $b_{xys}$ at three-dimensional pyramid positions $\langle x,y,s\rangle$ to representing per-video detections $b^t_{xys}$ at four-dimensional pyramid positions $\langle x,y,s,t\rangle$ and finding a sequence of per-video detections for $1\leq~t\leq~T$ that optimizes the following variant of Eq. : $$\max_{\substack{x_1,\ldots,x_T\\y_1,\ldots,y_T\\s_1,\ldots,s_T}}
  \sum_{t=1}^T f(b^t_{x_ty_ts_t})+
  \sum_{t=2}^T g(b^{t-1}_{x_{t-1}y_{t-1}s_{t-1}},b^t_{x_ty_ts_t})
  \label{eq:fv}$$ This combination of the detector and the tracker is performing simultaneous detection and tracking integrating the information between the two. Before, the tracker was affected by the detector but the detector was unaffected by the tracker: potential low-scoring but temporally-coherent detections would not even be generated by the detector despite the fact that they would yield good tracks. Because now, the detector no longer chooses which detections to produce but instead scores all detections at every position and scale, the tracker is able to choose among any possible detection. Such tight integration of higher- and lower-level information will be revisited when integrating event models into this framework.

# Combining tracking and event detection

It is popular to use Hidden Markov Models (HMMs) to perform event recognition . When doing so, the log likelihood of a video conditioned on an event model is: $$\log\sum_{k_1,\ldots,k_T}
  \exp\sum_{t=1}^Th(k_t,b^t_{j^{*}_t})+\sum_{t=2}^Ta(k_t,k_{t-1})$$ where $k_t$ denotes the state of the HMM for frame $t$, $h(k,b)$ denotes the log probability of generating a detection $b$ conditioned on being in state $k$, $a(k,k')$ denotes the log probability of transitioning from state $k$ to $k'$, and $j^{*}_t$ denotes index of the detection produced by the tracker in frame $t$. This log likelihood can be computed with the forward algorithm which is analogous to the Viterbi algorithm. Maximum likelihood (ML), the standard approach to using HMMs for classification, selects the event model that maximizes the likelihood of an observed event. One can instead select the model with the maximum *a posteriori* (log) probability (MAP). $$\max_{k_1,\ldots,k_T}
  \sum_{t=1}^Th(k_t,b^t_{j^{*}_t})+
  \sum_{t=2}^Ta(k_t,k_{t-1})
  \label{eq:map}$$ This can be computed with the Viterbi algorithm. The advantage of doing so is that one can combine the Viterbi algorithm used for detection-based tracking with the Viterbi algorithm used for event classification.

One can combine Eq.  with Eq.  to yield a unified cost function: $$\max_{j_1,\ldots,j_T}
  \max_{k_1,\ldots,k_T}
  \begin{array}[t]{l}
  \displaystyle\sum_{t=1}^T f(b^t_{j_t})
  +\displaystyle\sum_{t=2}^T g(b^{t-1}_{j_{t-1}},b^t_{j_t})\\
  +\displaystyle\sum_{t=1}^Th(k_t,b^t_{j_t})
  +\displaystyle\sum_{t=2}^Ta(k_t,k_{t-1})
  \end{array}
  \label{eq:unified}$$ that computes the joint MAP of the best possible track and the best possible state sequence by replacing $j^{*}_t$ with $j_t$ inside nested quantification. This too can be computed with the Viterbi algorithm, taking the lattice values $\delta^t_{jk}$ to be indexed by the detection index $j$ and the state $k$, forming the cross product of the tracker lattice nodes and the event lattice nodes: $$\begin{array}[t]{@{}l@{\hspace*{-20pt}}}
    \textbf{for}\;j=1\;\textbf{to}\;J_1\\
    \textbf{do}\;\begin{array}[t]{@{}l@{}}
         \textbf{for}\;k=1\;\textbf{to}\;K\;
         \textbf{do}\;\delta^1_{jk}\mathop{\mathrm{:\!=}}f(b^1_j)+h(k,b^1_j)
         \end{array}\\
    \textbf{for}\;t=2\;\textbf{to}\;T\\
    \textbf{do}\;\begin{array}[t]{@{}l@{}}
         \textbf{for}\;j=1\;\textbf{to}\;J_t\\
         \textbf{do}\;\begin{array}[t]{@{}l@{}}
                  \textbf{for}\;k=1\;\textbf{to}\;K\\
                  \textbf{do}\;\begin{array}[t]{@{}l@{}}
                       \delta^t_{jk}\mathop{\mathrm{:\!=}}
                       \begin{array}[t]{l}
                       f(b^t_j)+h(k,b^t_j)\\
                       {}+\displaystyle
                       \max_{j'=1}^{J_{t-1}}
                       \max_{k'=1}^K
                       \begin{array}[t]{l}
                       g(b^{t-1}_{j'},b^t_j)
                       +a(k,k')\\
                       {}+\delta^{t-1}_{j'k'}
                       \end{array}
                       \end{array}
                       \end{array}
                  \end{array}
         \end{array}
  \end{array}
  \label{eq:viterbisquared}$$ This finds the optimal path through a graph where the nodes at every frame represent the cross product of the detections and the HMM states.

Doing so performs simultaneous tracking and event classification. Before, the event classifier was affected by the tracker but the tracker was unaffected by the event classifier: potential low-scoring tracks would not even be generated by the tracker despite the fact that they would yield a high MAP estimate for some event class. Because now, the tracker no longer chooses which tracks to produce but instead scores all tracks, the event classifier is able to choose among any possible track. This amounts to a different kind of track-coherence measure that is tuned to specific events. Such a measure might otherwise be difficult to achieve without top-down information from the event classifier. For example applying this method to a video of a running person along with an event model for running, will be more likely to compose a track out of person detections that has high velocity and low change in direction.

Processing each frame $t$ with the algorithm in Eq.  is quadratic in $J_tK$. This can be problematic since $J_tK$ can be large. As before, we can make this linear in $J_t$ using a generalized distance transform. One can make this linear in $K$ for suitable state-transition functions $a$ .

Two practical issues arise when applying the above method. First, one can factor Eq.  as Eq. : $$\displaystyle\max_{j'=1}^{J_{t-1}}
  \displaystyle\max_{k'=1}^K
  \left(g(b^{t-1}_{j'},b^t_j)+a(k,k')+\delta^{t-1}_{j'k'}\right)
  \label{eq:before}$$ $$\displaystyle\max_{j'=1}^{J_{t-1}}
  \left(g(b^{t-1}_{j'},b^t_j)+
  \displaystyle\max_{k'=1}^K
  \left(a(k,k')+\delta^{t-1}_{j'k'}\right)\right)
  \label{eq:after}$$ This is important because the computation of $g(b^{t-1}_{j'},b^t_j)$ might be expensive as it involves a projection of $b^{t-1}_{j'}$ forward one frame (e.glet@token.. using optical flow or KLT). Second, when applying this method to multiple event models, the same factorization can be extended to cache the computation of $g(b^{t-1}_{j'},b^t_j)$ across different event models as this term does not depend on the event model.

# Combining object detection, tracking and event detection

One can combine the methods of Sections  and  to optimize a cost function: $$\hspace{-0.5ex}
  \max_{\substack{x_1,\ldots,x_T\\y_1,\ldots,y_T\\s_1,\ldots,s_T\\k_1,\ldots,k_T}}
  \begin{array}[t]{l@{\hspace*{-10pt}}}
  \displaystyle\sum_{t=1}^T f(b^t_{x_ty_ts_t})+h(k_t,b^t_{x_ty_ts_t})\\
  +\displaystyle\sum_{t=2}^T g(b^{t-1}_{x_{t-1}y_{t-1}s_{t-1}},b^t_{x_ty_ts_t})
  +a(k_t,k_{t-1})
  \end{array}
  \label{eq:allthree}$$ that combines Eq.  with Eq.  by forming a large Viterbi lattice with values $\delta^t_{xysk}$.

One practical issue arises when applying the above method. In Eq. , $h$ is a function of $b^t_{x_ty_ts_t}$, the detection in the current frame. This allows the HMM event model to depend on static object characteristics such as position, shape, and pose. However, many approaches to event recognition using HMMs use temporal derivatives of such characteristics to provide object velocity and acceleration information . Having $h$ also be a function of $b^{t-1}_{x_{t-1}y_{t-1}s_{t-1}}$, the detection in the previous frame, requires incorporation $h$ into the generalized distance transform and thus restricts its form.

The above combination performs simultaneous object detection, tracking, and event classification, integrating information across all three. Without such information integration, the object detector is unaffected by the tracker which is in turn unaffected by the event model. With such integration, the event model can influence the tracker and both can influence the object detector.

This is important because current object detectors cannot reliably detect small, deformable, or partially occluded objects. Moreover, current trackers also fail to track such objects. Information from the event model can focus the object detector and tracker on those particular objects that participate in a specified event. An event model for recognizing an agent picking an object up can bias the object detector and tracker to search for an object that exhibits a particular profile of motion relative to the agent, namely where the object is in close proximity to the agent, the object starts out being at rest while the agent approaches the object, then the agent touches the object, followed by the object moving with the agent.

A traditional view of the relationship between object and event detection suggests that one recognizes a *hammering* event, in part, because one detects a *hammer*. Our unified approach inverts the traditional view, suggesting that one can recognize a *hammer*, in part, by detecting a *hammering* event. Furthermore, a strength of our approach is that such relationships are not encoded explicitly, do not have to be annotated in the training data for the event models, and are learned automatically as part of learning the parameters of the different event models. This is to say that the relationship between a person and the objects they manipulate can be learned from the co-occurrence of tracks in the training data, rather than from manually annotated symbolic relationships.

# Experimental results

Figure demonstrates improved performance of simultaneous object detection and tracking (c) over object detection (a) and tracking (b) in isolation. This happens for different reasons: motion blur, even for large objects, can lead to poor detection results and hence poor tracks, small objects are difficult to detect and track, and integration can improve detection and tracking of deformable objects, such as a person transitioning from an upright pose to sitting down.

```latex
\begin{figure*}[p]\begin{center}
    \begin{tabular}{@{}c@{\hspace*{2pt}}c@{\hspace*{2pt}}c@{}}
      \includegraphics[width=0.32\textwidth]{images/viterbi-pedro-1-1}&
      \includegraphics[width=0.32\textwidth]{images/viterbi-pedro-1-2}&
      \includegraphics[width=0.32\textwidth]{images/viterbi-pedro-1-3}\\
      \includegraphics[width=0.32\textwidth]{images/viterbi-pedro-2-1}&
      \includegraphics[width=0.32\textwidth]{images/viterbi-pedro-2-2}&
      \includegraphics[width=0.32\textwidth]{images/viterbi-pedro-2-3}\\
      \includegraphics[width=0.32\textwidth]{images/viterbi-pedro-3-1}&
      \includegraphics[width=0.32\textwidth]{images/viterbi-pedro-3-2}&
      \includegraphics[width=0.32\textwidth]{images/viterbi-pedro-3-3}\\
      \includegraphics[width=0.32\textwidth]{images/viterbi-pedro-4-1}&
      \includegraphics[width=0.32\textwidth]{images/viterbi-pedro-4-2}&
      \includegraphics[width=0.32\textwidth]{images/viterbi-pedro-4-3}\\
      (a)&(b)&(c)\\
    \end{tabular}
  \end{center}
  %
  \caption{Improved performance of simultaneous object detection and tracking.
    %
    (a)~Output of the Felzenszwalb {et al}\ifx\@let@token.\else.\null\fi\ detector.
    %
    (b)~Tracks produced by detection-based tracking.
    %
    (c)~Tracks produced by simultaneous object detection and tracking.}
  %
  \label{fig:objectsandtracking}
\end{figure*}
```
```latex
\begin{figure*}[p]\begin{center}
    \begin{tabular}{@{}c@{\hspace*{2pt}}c@{\hspace*{2pt}}c@{}}
      \includegraphics[width=0.32\textwidth]{images/viterbi-event-1-1}&
      \includegraphics[width=0.32\textwidth]{images/viterbi-event-1-2}&
      \includegraphics[width=0.32\textwidth]{images/viterbi-event-1-3}\\
      \includegraphics[width=0.32\textwidth]{images/viterbi-event-2-1}&
      \includegraphics[width=0.32\textwidth]{images/viterbi-event-2-2}&
      \includegraphics[width=0.32\textwidth]{images/viterbi-event-2-3}\\
      \includegraphics[width=0.32\textwidth]{images/viterbi-event-3-1}&
      \includegraphics[width=0.32\textwidth]{images/viterbi-event-3-2}&
      \includegraphics[width=0.32\textwidth]{images/viterbi-event-3-3}\\
      \includegraphics[width=0.32\textwidth]{images/viterbi-event-4-1}&
      \includegraphics[width=0.32\textwidth]{images/viterbi-event-4-2}&
      \includegraphics[width=0.32\textwidth]{images/viterbi-event-4-3}\\
      (a)&(b)&(c)
    \end{tabular}
  \end{center}
  %
  \caption{Improved performance of simultaneous tracking and event recognition.
    %
    (a)~Output of the Felzenszwalb {et al}\ifx\@let@token.\else.\null\fi\ detector.
    %
    (b)~Tracks produced by detection-based-tracking.
    %
    (c)~Tracks produced by simultaneous tracking and event recognition.}
  %
  \label{fig:objectsandtrackingandevents}
\end{figure*}
```
```latex
\begin{figure*}
  \begin{center}
    \begin{tabular}{@{}c@{\hspace*{2pt}}c@{\hspace*{2pt}}c@{}}
      \includegraphics[width=0.32\textwidth]{images/viterbi-unified-1-1}&
      \includegraphics[width=0.32\textwidth]{images/viterbi-unified-1-2}&
      \includegraphics[width=0.32\textwidth]{images/viterbi-unified-1-3}\\
      \includegraphics[width=0.32\textwidth]{images/viterbi-unified-2-1}&
      \includegraphics[width=0.32\textwidth]{images/viterbi-unified-2-2}&
      \includegraphics[width=0.32\textwidth]{images/viterbi-unified-2-3}\\
      (a)&(b)&(c)
    \end{tabular}
  \end{center}
  %
  \caption{Improved performance of simultaneous object detection, tracking, and
    event recognition.
    %
    (a)~Output of the Felzenszwalb {et al}\ifx\@let@token.\else.\null\fi\ detector.
    %
    (b)~Tracks produced by detection-based-tracking.
    %
    (c)~Tracks produced by simultaneous object-detection, tracking, and event
    recognition.}
  %
  \label{fig:allthree}
\end{figure*}
```

Figure demonstrates improved performance of simultaneous tracking and event recognition (c) over tracking (b) in isolation. These results were obtained with object and event models that were trained independently.[^2] The object models were trained on isolated frames using the standard Felzenszwalb training software. The event models were trained using tracks produced by the detection-based tracking method described in Section . It is difficult to track the person running with detection-based tracking alone due to articulated appearance change and motion blur. Imposing the prior of detecting *running* biases the tracker to find the desired track.

Figure demonstrates improved performance of simultaneous object detection, tracking, and event recognition (c) over object detection (a) and tracking (b) in isolation. As before, these results were obtained with object and event models that were trained independently.

# Conclusion

Detection-base tracking using dynamic programming has a long history , as do motion-profile-based approaches to event recognition using HMMs . Moreover, there have been attempts to integrate object detection and tracking , tracking and event recognition , and object detection and event recognition . However, we are unaware of prior work that integrates all three and does so in a fashion that efficiently finds a global optimum to a simple unified cost function.

We have demonstrated a general framework for simultaneous object detection, tracking, and event recognition Many object detectors can naturally be transformed into trackers by introducing time into their cost functions, thus tracking every possible detection in each frame. Furthermore, the distance transform can be used to reduce the complexity of doing so from quadratic to linear. The common internal structure and algorithmic organization of object detection, detection-based tracking, and event recognition further allows an HMM-based approach to event recognition to be incorporated into the general dynamic-programming approach. This facilitates multidirectional information flow where not only can object detection influence tracking and, in turn, event recognition, event recognition can influence tracking and, in turn object detection.

# Acknowledgments

This work was supported, in part, by NSF grant CCF-0438806, by the Naval Research Laboratory under Contract Number N00173-10-1-G023, by the Army Research Laboratory accomplished under Cooperative Agreement Number W911NF-10-2-0060, and by computational resources provided by Information Technology at Purdue through its Rosen Center for Advanced Computing. Any views, opinions, findings, conclusions, or recommendations contained or expressed in this document or material are those of the author(s) and do not necessarily reflect or represent the views or official policies, either expressed or implied, of NSF, the Naval Research Laboratory, the Office of Naval Research, the Army Research Laboratory, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes, notwithstanding any copyright notation herein.

[^1]: One can ameliorate this somewhat by constructing a lattice that skips frames . This increases the asymptotic complexity to be exponential in the number of frame skips allowed.

[^2]: It would appear possible to co-train object and event models by combining Baum-Welch with the training procedure for object models .


abstract: Transformer, an attention-based encoder-decoder architecture, has not only revolutionized the field of natural language processing (NLP), but has also done some pioneering work in the field of computer vision (CV). Compared to convolutional neural networks (CNNs), the Vision Transformer (ViT) relies on excellent modeling capabilities to achieve very good performance on several benchmarks such as ImageNet, COCO, and ADE20k. ViT is inspired by the self-attention mechanism in natural language processing, where word embeddings are replaced with patch embeddings.
  .
  This paper reviews the derivatives in the field of ViT and the cross-applications of ViT with other fields.
author: Zujun Fu
title: Vision Transformer: Vit and its Derivatives

# Pyramid Vision Transformer

To overcome the quadratic complexity of the attention mechanism, the Pyramid Vision Transformer (PVT) uses a variant of self-attention called Spatial-Reduced Attention (SRA). It is characterized by spatial reduction of keys and values, similar to Linformer attention in the NLP field.

By applying SRA, the feature space dimension of the whole model is slowly reduced and the concept of order is enhanced by applying positional embedding in all transformer blocks.PVT has been used as a backbone network for object detection and semantic segmentation to process high resolution images.

Later on, the research team further improved their PVT model named PVT-v2, with the following major improvements.

-   overlapping patch embedding

-   convolutional feedforward networks

-   linear-complexity self-attention layers.

Overlapping patches is a simple and general idea to improve ViT, especially for dense tasks (e.g. semantic segmentation).By exploiting overlapping regions/patch, PVT-v2 can obtain more local continuity of image representations.

Convolution between fully connected layers (FC) eliminates the need for fixed size positional encoding in each layer. The 3x3 deep convolution with zero padding (p=1) is designed to compensate for the removal of positional encoding from the model (they are still present, but only in the input). This process allows more flexibility to handle multiple image resolutions.

Finally, using key and value pooling(p=7), the self-attentive layer is reduced to a complexity similar to that of a CNN.

# Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Swin Transformer aims to build the idea of locality from the standard NLP transformer, i.e. local or window attention:

In the Swin Transformer, local self-attention is used for non-overlapping windows. The next layer of window-to-window communication produces hierarchical representation by progressively merging windows.

As shown in **Figure 3**, the left shows the regular window partitioning scheme in the first layer, where self-attention is computed within each window. The window partitioning in the second layer on the right is shifted by 2 image patches, resulting in crossing the boundary of the previous window.

The local self-attention scales linearly with image size $O(M*N)$ instead of $O(N^2)$ in the window size used for sequence length $N$ and $M$.

By merging and adding many local layers, there is a global representation. In addition, the spatial dimensions of the feature maps has been significantly reduced. The authors claim to have achieved promising results on both ImageNet-1K and ImageNet-21K.

# Scaling Vision Transformer

Deep learning and scale are related. In fact, scale is a key component in pushing the state-of-the-art. In this study, the authors from Google Brain Research trained a slightly modified ViT model with 2 billion parameters and achieved a top-1 accuracy of 90.45% on ImageNet. This over-parameterized generalized model was tested on few-shot learning, with only 10 examples per class. A top-1 accuracy of 84.86% was achieved on ImageNet.

Few-shot learning refers to fine-tuning a model with an extremely limited number of samples. The goal of few-shot learning is to motivate generalization by slightly adapting the acquired pre-trained knowledge to a specific task. If large models are successfully pre-trained, it makes sense to perform well with a very limited understanding of the downstream task (provided by only a few examples).

The following are some of the core contributions and main results of this paper.

-   Representation quality can be bottlenecked by model size, given that you have enough data to feed it;

-   Large models benefit from additional supervised data, even over 1B images.

**Figure 5** depicts the effect of switching from a 300M image dataset (JFT-300M) to 3 billion images (JFT-3B) without any further scaling. Both the medium (B/32) and large (L/16) models benefit from adding data, roughly by a constant factor. Results are obtained by few-shot(linear) evaluation throughout the training process.

-   Larger models are more sample efficient, achieving the same level of error rate with fewer visible images.

-   To save memory, they remove class tokens (cls). Instead, they evaluated global average pooling and multi-head attention pooling to aggregate the representations of all patch tokens.

-   They use different weight decay for the head and the rest of the layers called 'body'. The authors demonstrate this well in the **Figure 6**. The box values are few-shot accuracy, while the horizontal and vertical axes indicate the weight decay for the body and the head, respectively. Surprisingly, the stronger decay of the head produces the best results. The authors speculate that a strong weight decay of the head leads to representations with a larger margin between classes.

This is perhaps the most interesting finding that can be more widely applied to pre-training ViT.

They used a warm-up phase at the beginning of training and a cool-down phase at the end of training, where the learning rate linearly anneals to zero. In addition, they used the Adafactor optimizer, which has a memory overhead of 50% compared to traditional Adam.

# Replacing self-attention: independent token + channel mixing methods

It is well known that self-attention can be used as an information routing mechanism with fast weights. So far, 3 papers tell the same story: replacing self-attention with 2 information mixing layers; one for mixing token (projected patch vector) and one for mixing channel/feature information.

## MLP-Mixer

The MLP-Mixer contains two MLP layers: the first applied independently to the image patches (i.e., " mixing" the features at each location) and the other across the patches (i.e., " mixing" the spatial information).MLP Mixer architecture is shown in **Figure 7**.

## XCiT: Cross-Covariance Image Transformers

The other is the recent architecture XCiT, which aims to modify the core building block of ViT: self-attention applied to the token dimension.XCiT architecture is shown in **Figure 8**.

**XCA**: For information mixing, the authors propose a cross-covariance attention (XCA) function that operates on the feature dimension of a token rather than on its own. Importantly, this method is only applicable to the L2-normalized set of queries, keys, and values. the L2 norm is denoted by the hat above the letters K and Q. The result of multiplication is also normalized to \[-1,1\] before softmax.

**Local Patch Interaction**: To achieve explicit communication between the patches, the researchers added two depth-wise 3×3 convolutional layers with Batch Normalization and GELU non-linearity in between, as shown in **Figure 9**. Depth-wise convolution was applied to each channel (here the patch) independently.

## ConvMixer

Self-attention and MLP are theoretically more general modeling mechanisms, as they allow for larger receptive fields and content-aware behaviour. Nevertheless, the inductive bias of convolution has undeniable results in computer vision tasks.

Motivated by this, researchers have proposed another variant based on convolutional networks called ConvMixer, as shown in **Figure 10**. the main idea is that it operates directly on the patches as input, separating the mixing of spatial and channel dimensions and maintaining the same size and resolution throughout the network.

More specifically, depthwise convolution is responsible for mixing spatial locations, while pointwise convolution (1x1x channel kernel) for mixing channel locations, as shown in the **Figure 11**.

Mixing of distant spatial locations can be achieved by selecting a larger kernel size to create a larger receptive field.

# Multiscale Vision Transformers

The CNN backbone architecture benefits from the gradual increase of channels while reducing the spatial dimension of the feature map. Similarly, the Multiscale Vision Transformer (MViT) exploits the idea of combining a multi-scale feature hierarchies with a Vision Transformer model. In practice, the authors start with an initial image size of 3 channels and gradually expand (hierarchically) the channel capacity while reducing the spatial resolution.

Thus, a multi-scale feature pyramid is created, as shown in **Figure 12**. Intuitively, the early layers will learn high-spatial with simple low-level visual information, while the deeper layers are responsible for complex high-dimensional features.

# Video classification: Timesformer

After a successful image task, the Vision Transformer is applied to video recognition. Two architectures are presented here,as shown in **Figure 13**.

-   Right: Reducing the architecture level. The proposed method applies a spatial Transformer to the projection image patches and then has another network responsible for capturing time correlations. This is similar to the winning strategy of CNN+LSTM based on video processing.

-   Left: Space-time attention that can be implemented at the self-attention level, with the best combination in the red box. Attention is applied sequentially in the time domain by first treating the image frames as tokens. Then, the combined space attention of the two spatial dimensions is applied before the MLP projection. **Figure 14** is the t-SNE visualization of the method.

In **Figure 14**, each video is visualized as a point. Videos belonging to the same action category have the same color. A TimeSformer with split space-time attention learns more separable features semantically than a TimeSformer with only space attention or ViT.

# ViT in semantic segmentation: SegFormer

NVIDIA has proposed a well-configured setup called SegFormer. SegFormer has an interesting design component. First, it consists of a hierarchical Transformer encoder that outputs multi-scale features. Second, it does not require positional encoding, which can deteriorate performance when the test resolution is different from the training.

SegFormer ,as shown in **Figure 15**, uses a very simple MLP decoder to aggregate the multi-scale features of the encoder. Contrary to ViT, SegFormer uses small image patches, such as 4 x 4, which are known to favor intensive prediction tasks. The proposed Transformer encoder outputs 1/4, 1/8, 1/16, 1/32 multi-scale features at the original image resolution. These multi-level features are provided to the MLP decoder to predict the segmentation mask.

Mix-FFN in **Figure 15**: In order to mitigate the impact of positional encoding, the researchers use zero-padding 3 × 3 convolutional layers to leak location information.Mix-FFN can be expressed as follows. $$x_{out}=MLP(GELU(Conv(MLP(x_{in}))))+x_{in}$$

Efficient self-attention is proposed in PVT, which uses a reduction ratio to reduce the length of the sequence. The results can be measured qualitatively by visualizing the effective receptive field (ERF) as shown in **Figure 16**.

# Vision Transformers in Medical imaging: Unet + ViT = UNETR

Although there are other attempts in medical imaging, UNETR provides the most convincing results. In this approach, ViT is applied to 3D medical image segmentation. It was shown that a simple adaptation is sufficient to improve the baselines for several 3D segmentation tasks.

Essentially, UNETR uses the Transformer as an encoder to learn the sequence representation of the input audio, as in **Figure 17**. Similar to the Unet model, it aims to efficiently capture global multi-scale information that can be passed to the decoder through long skip connections, forming skip connections at different resolutions to compute the final semantic segmentation output.