Question:
represent the article for finding a single-sentence tldr

Fewshots:
'We study model recovery for data classification, where the training labels are generated from a one-hidden-layer fully -connected neural network with sigmoid activations, and the goal is to recover the weight vectors of the neural network. We prove that under Gaussian inputs, the empirical risk function using cross entropy exhibits strong convexity and smoothness uniformly in a local neighborhood of the ground truth, as soon as the sample complexity is sufficiently large. This implies that if initialized in this neighborhood, which can be achieved via the tensor method, gradient descent converges linearly to a critical point that is provably close to the ground truth without requiring a fresh set of samples at each iteration. To the best of our knowledge, this is the first global convergence guarantee established for the empirical risk minimization using cross entropy via gradient descent for learning one-hidden-layer neural networks, at the near-optimal sample and computational complexity with respect to the network input dimension. Neural networks have attracted a significant amount of research interest in recent years due to the success of deep neural networks BID18 in practical domains such as computer vision and artificial intelligence BID24 BID15 BID27. However, the theoretical underpinnings behind such success remains mysterious to a large extent. Efforts have been taken to understand which classes of functions can be represented by deep neural networks BID7 BID16 BID0 ), when (stochastic) gradient descent is effective for optimizing a non-convex loss function BID8, and why these networks generalize well BID1.One important line of research that has attracted extensive attention is a model-recovery setup, i.e., given that the training samples (x i, y i) ∼ (x, y) are generated i.i.d. from a distribution D based on a neural network model with the ground truth parameter W, the goal is to recover the underlying model parameter W, which is important for the network to generalize well BID22. Previous studies along this topic can be mainly divided into two types of data generations. First, a regression problem, for example, assumes that each sample y is generated as y = 1 K K k=1 φ(w k x), where w k ∈ R d is the weight vector of the kth neuron, 1 ≤ k ≤ K, and the input x ∈ R d is Gaussian. This type of regression problem has been studied in various settings. In particular, BID28 studied the single-neuron model under ReLU activation, BID38 ) studied the onehidden-layer multi-neuron network model, and BID19 ) studied a two-layer feedforward networks with ReLU activations and identity mapping. Second, for a classification problem, suppose each label y ∈ {0, 1} is drawn under the conditional distribution P(y = 1|x) = 1 K K k=1 φ(w k x), where w k ∈ R d is the weight vector of the kth neuron, 1 ≤ k ≤ K, and the input x ∈ R d is Gaussian. Such a problem has been studied in BID21 in the case with a single neuron. For both the regression and the classification settings, in order to recover the neural network parameters, all previous studies considered (stochastic) gradient descent over the squared loss, i.e., qu (W ; x, y) = DISPLAYFORM0 Furthermore, previous studies provided two types of statistical guarantees for such model recovery problems using the squared loss. More specifically, BID38 showed that in the local neighborhood of the ground truth, the Hessian of the empirical loss function is positive definite for each given point under independent high probability event. Hence, their guarantee for gradient descent to converge to the ground truth requires a fresh set of samples at every iteration, thus the total sample complexity will depend on the number of iterations. On the other hand, studies such as BID21 BID28 establish certain types of uniform geometry such as strong convexity so that resampling per iteration is not needed for gradient descent to have guaranteed linear convergence as long as it enters such a local neighborhood. However, such a stronger statistical guarantee without per-iteration resampling have only been shown for the squared loss function. In this paper, we aim at developing such a strong statistical guarantee for the loss function in eq., which is much more challenging but more practical than the squared loss for the classification problem. This study provides the first performance guarantee for the recovery of one-hidden-layer neural networks using the cross entropy loss function, to the best of our knowledge. More specifically, our contributions are summarized as follows.• For multi-neuron classification problem with sigmoid activations, we show that, if the input is Gaussian, the empirical risk function f n (W) = 1 n n i=1 (W ; x i) based on the cross entropy loss in eq. is uniformly strongly convex in a local neighborhood of the ground truth W of size O(1/K 3/2) as soon as the sample size is O(dK 5 log 2 d), where d is the input dimension and K is the number of neurons.• We further show that, if initialized in this neighborhood, gradient descent converges linearly to a critical point W n (which we show to exist), with a sample complexity of O(dK 5 log 2 d), which is near-optimal up to a polynomial factor in K and log d. Due to the nature of quantized labels here, the recover of W is only up to certain statistical accuracy, and W n converges to W at a rate of O(dK 9/2 log n/n) in the Frobenius norm. Furthermore, such a convergence guarantee does not require a fresh set of samples at each iteration due to the uniform strong convexity in the local neighborhood. To obtain -accuracy, it requires a computational complexity of O(ndK 2 log(1/)).• We adopt the tensor method proposed in BID38, and show it provably provides an initialization in the neighborhood of the ground truth. In particular, our proof replaces the homogeneous assumption on activation functions in BID38 ) by a mild condition on the curvature of activation functions around W, which holds for a larger class of activation functions including sigmoid and tanh. In order to analyze the challenging cross-entropy loss function, our proof develops various new machineries in order to exploit the statistical information of the geometric curvatures, including the gradient and Hessian of the empirical risk, and to develop covering arguments to guarantee uniform concentrations. Our technique also yields similar performance guarantees for the classification problem using the squared loss in eq., which we omit due to space limitations, as it is easier to analyze than cross entropy. Due to page limitations we focus on the most relevant literature on theoretical and algorithmic aspects of learning shallow neural networks via nonconvex optimization. The parameter recovery viewpoint is relevant to the success of non-convex learning in signal processing problems such as matrix completion, phase retrieval, blind deconvolution, dictionary learning and tensor decomposition BID31 BID6 BID13 BID30 BID2, to name a few. The statistical model for data generation effectively removes worst-case instances and allows us to focus on average-case performance, which often possess much benign geometric properties that enable global convergence of simple local search algorithms. The studies of one-hidden-layer network model can be further categorized into two classes, landscape analysis and model recovery. In the landscape analysis, it is known that if the network size is large enough compared to the data input, then there are no spurious local minima in the optimization landscape, and all local minima are global BID3 BID25 BID23. For the case with multiple neurons (2 ≤ K ≤ d) in the under-parameterized setting, the work of Tian BID33 studied the landscape of the population squared loss surface with ReLU activations. In particular, there exist spurious bad local minima in the optimization landscape BID26 even at the population level. Zhong et. al. BID38 provided several important characterizations for the local Hessian for the regression problem for a variety of activation functions for the squared loss. In the model recovery problem, the number of neurons is smaller than the dimension of inputs. In the case with a single neuron (K = 1), under Gaussian input, BID28 showed that gradient descent converges linearly when the activation function is ReLU, i.e. φ(z) = max{z, 0}, with a zero initialization, as long as the sample complexity is O(d) for the regression problem. On the other end, BID21 showed that when φ(·) has bounded first, second and third derivatives, there is no other critical points than the unique global minimum (within a constrained region of interest), and (projected) gradient descent converges linearly with an arbitrary initialization, as long as the sample complexity is O(d log 2 d) with sub-Gaussian inputs for the classification problem using the squared loss. Moreover, BID38 shows that the ground truth From a technical perspective, our study differs from all the aforementioned work in that the cross entropy loss function we analyze has a very different form. Furthermore, we study the model recovery classification problem under the multi-neuron case, which has not been studied before. Finally, we note that several papers study one-hidden-layer or two-layer neural networks with different structures under Gaussian input. For example, BID9 b; BID37 ) studied the non-overlapping convolutional neural network, BID19 ) studied a two-layer feedforward networks with ReLU activations and identity mapping, and BID11 introduced the Porcupine Neural Network. These are not directly comparable to ours since both the networks and the loss functions are different. The rest of the paper is organized as follows. Section 2 describes the problem formulation. Section 3 presents the main on local geometry and local linear convergence of gradient descent. Section 4 discusses the initialization method. Numerical examples are demonstrated in Section 5, and finally, are drawn in Section 6.Throughout this paper, we use boldface letters to denote vectors and matrices, e.g. w and W. The transpose of W is denoted by W, and W, W F denote the spectral norm and the Frobenius norm. For a positive semidefinite (PSD) matrix A, we write A 0. The identity matrix is denoted by I. The gradient and the Hessian of a function f (W) is denoted by ∇f (W) and ∇ 2 f (W), respectively. Let σ i (W) denote the i-th singular value of W. Denote · ψ1 as the sub-exponential norm of a random variable. We use c, C, C 1,... to denote constants whose values may vary from line to line. For nonnegative functions f (x) and g(x), f (x) = O (g(x)) means there exist positive constants c and a such that f (x) ≤ cg(x) for all x ≥ a; f (x) = Ω (g(x)) means there exist positive constants c and a such that f (x) ≥ cg(x) for all x ≥ a. We first describe the generative model for training data, and then describe the gradient descent algorithm for learning the network weights. Suppose we are given n training samples {(x i, y i)} n i=1 ∼ (x, y) that are drawn i.i.d., where x ∼ N (0, I). Assume the activation function is sigmoid, i.e. φ (z) = 1/(1 + e −z) for all z. Conditioned on x ∈ R d, we consider the classification setting, where y is mapped to a discrete label using the one-hidden layer neural network model as follows: DISPLAYFORM0 and P(y = 0|x) = 1 − P(y = 1|x), where K is the number of neurons. Our goal is to estimate W = [w 1, · · ·, w K], via minimizing the following empirical risk function: DISPLAYFORM1 where (W ; x):= (W ; x, y) is the cross entropy loss, i.e., the negative log-likelihood function, i.e., DISPLAYFORM2 With slight abuse of notation, we denote the gradient and Hessian of (W ; x) with respect to the vector w. To estimate W, since is a highly nonconvex function, vanilla gradient descent with an arbitrary initialization may get stuck at local minima. Therefore, we implement the gradient descent algorithm with a well-designed initialization scheme that is described in detail in Section 4. The update rule is given as DISPLAYFORM0, where η is the step size. The algorithm is summarized in Algorithm 1. DISPLAYFORM1 We note that throughout the execution of the algorithm, the same set of training samples is used which is the standard implementation of gradient descent. This is in sharp contrast to existing work such as BID38 that employs the impractical scheme of resampling, where a fresh set of training samples is used at every iteration of gradient descent. Before stating our main , we first introduce an important quantity regarding φ(z) that captures the geometric properties of the loss function, distilled in BID38. Figure 1: ρ (σ) for sigmoid activation. DISPLAYFORM0 Note that the definition here is different from that in (b, Property 3 .2) but consistent with (b, Lemma D.4) which removes the third term in (b, Property 3.2). For the activation function considered in this paper, the first two terms suffice. We depict ρ(σ) as a function of σ in a certain range for the sigmoid activation in Fig. 1. It is easy to observe that ρ(σ) > 0 for all σ > 0. We first characterize the local strong convexity of f n (W) in a neighborhood of the ground truth W. Let B (W, r) denote a Euclidean ball centered at W ∈ R d×K with a radius r, i.e. DISPLAYFORM0 Let σ i:= σ i (W) denote the i-th singular value of W. Let the condition number be κ = σ 1 /σ K, and λ = DISPLAYFORM1 The following theorem guarantees the Hessian of the empirical risk function f n (W) in the local neighborhood of W is positive definite with high probability. Theorem 1. For the classification model with sigmoid activation function, assume W F ≤ 1, then there exists some constant C, such that if DISPLAYFORM2 then with probability at least 1 − d −10, for all W ∈ B(W, r), DISPLAYFORM3 hold, where r:= min DISPLAYFORM4 We note that all column permutations of W are equivalent global minima of the loss function, and Theorem 1 applies to all such permutation matrices of W. The proof of Theorem 1 is outlined in Appendix A. Theorem 1 guarantees that the Hessian of the empirical cross-entropy loss function f n (W) is positive definite (PD) in a neighborhood of the ground truth W, as long as ρ(σ K) > 0 (i.e. W is full-column rank), when the sample size n is sufficiently large for the sigmoid activation. The bounds in Theorem 1 depend on the dimension parameters of the network (n and K), as well as the activation function and the ground truth (ρ(σ K), λ). As a special case, suppose W is composed of orthonormal columns with ρ(σ K) = O, κ = 1, λ = 1. Then, Theorem 1 guarantees DISPLAYFORM5, as soon as the sample complexity n = Ω(dK 5 log 2 d). The sample complexity is order-wise near-optimal in d up to polynomial factors of K and log d, since the number of unknown parameters is dK. For the classification problem, due to the nature of quantized labels, W is no longer a critical point of f n (W). By the strong convexity of the empirical risk function f n (W) in the local neighborhood of W, there can exist at most one critical point in B(W, r), which is the unique local minimizer in B (W, r) if it exists. The following theorem shows that there indeed exists such a critical point W n, which is provably close to the ground truth W, and gradient descent converges linearly to W n. Theorem 2. For the classification model with sigmoid activation function, and assume W F ≤ 1, there exist some constants C, C 1 > 0 such that if the sample size n ≥ C · dK DISPLAYFORM0 then with probability at least 1 − d −10, there exists a unique critical point W n in B(W, r) with DISPLAYFORM1 Moreover, if the initial point W 0 ∈ B (W, r), then gradient descent converges linearly to W n, i.e. DISPLAYFORM2 where DISPLAYFORM3, as long as the step size DISPLAYFORM4 Similarly to Theorem 1, Theorem 2 also holds for all column permutations of W. The proof can be found in Appendix B. Theorem 2 guarantees that there exists a critical point W n in B(W, r) which converges to W at the rate of O(K 9/4 d log n/n), and therefore W can be recovered consistently as n goes to infinity. Moreover, gradient descent converges linearly to W n at a linear rate, as long as it is initialized in the basin of attraction. To achieve -accuracy, i.e. DISPLAYFORM5 requires a computational complexity of O ndK 2 log (1/), which is linear in n, d and log(1/). Our initialization adopts the tensor method proposed in BID38. In this section, we first briefly describe this method, and then present the performance guarantee of the initialization with remarks on the differences from that in BID38. This subsection briefly introduces the tensor method proposed in BID38, to which a reader can refer for more details. We first define a product ⊗ as follows. If v ∈ R d is a vector and I is the identity matrix, then DISPLAYFORM0 Definition 3. Let α ∈ R d denote a randomly picked vector. We define P 2 and P 3 as follows: DISPLAYFORM1 1 where j 2 = min{j ≥ 2|M j = 0}, and P 3 = M j3 (I, I, I, α, · · ·, α), where j 3 = min{j ≥ 3|M j = 0}.We further denote w = w/ w. The initialization algorithm based on the tensor method is summarized in Algorithm 2, which includes two major steps. Step 1 first estimates the direction of each column of W by decomposing P 2 to approximate the subspace spanned by {w 1, w 2, · · ·, w K} (denoted by V), then reduces the third-order tensor P 3 to a lower-dimension tensor R 3 = P 3 (V, V, V) ∈ R K×K×K, and applys non-orthogonal tensor decomposition on R 3 to output the estimate s i V w i, where s i ∈ {1, −1} is a random sign. Step 2 approximates the magnitude of w i and the sign s i by solving a linear system of equations. For the classification problem, we make the following technical assumptions, similarly in (b, Assumption 5. 3) for the regression problem. Assumption 1. The activation function φ(z) satisfies the following conditions: DISPLAYFORM0 2. At least one of M 3 and M 4 is non-zero. Furthermore, we do not require the homogeneous assumption ((i.e., φ(az) = a p z for an integer p)) required in BID38, which can be restrictive. Instead, we assume the following condition on the curvature of the activation function around the ground truth, which holds for a larger class of activation functions such as sigmoid and tanh. Assumption 2. Let l 1 be the index of the first nonzero M i where i = 1,..., 4. For the activation function φ (·), there exists a positive constant δ such that m l1,i (·) is strictly monotone over the interval (w i − δ, w i + δ), and the derivative of m l1,i (·) is lower bounded by some constant for all i. We next present the performance guarantee for the initialization algorithm in the following theorem. Theorem 3. For the classification model, under Assumptions 1 and 2, if the sample size n ≥ dpoly (K, κ, t, log d, 1/), then the output W 0 ∈ R d×K of Algorithm 2 satisfies DISPLAYFORM1 with probability at least 1 − d−Ω(t).The proof of Theorem 3 consists of (a) showing the estimation of the direction of W is sufficiently accurate and (b) showing the approximation of the norm of W is accurate enough. Our proof of part (a) is the same as that in BID38 ), but our argument in part (b) is different, where we relax the homogeneous assumption on activation functions. More details can be found in the supplementary materials in Appendix C. In this section, we first implement gradient descent to verify that the empirical risk function is strongly convex in the local region around W. If we initialize multiple times in such a local region, it is expected that gradient descent converges to the same critical point W n, with the same set of training samples. Given a set of training samples, we randomly initialize multiple times, and then calculate the variance of the output of gradient descent. Denote the output of the th run as w DISPLAYFORM0, where L = 20 is the total number of random initializations. Adopted in BID21, it quantifies the standard deviation of the estimator W n under different initializations with the same set of training samples. We say an experiment is successful, if SD n ≤ 10 −2.Figure 2 (a) shows the successful rate of gradient descent by averaging over 50 sets of training samples for each pair of n and d, where K = 3 and d = 15, 20, 25 respectively. The maximum iterations for gradient descent is set as iter max = 3500. It can be seen that as long as the sample complexity is large enough, gradient descent converges to the same local minima with high probability. We next show that the statistical accuracy of the local minimizer for gradient descent if it is initialized close enough to the ground truth. Suppose we initialize around the ground truth such that W 0 − W F ≤ 0.1 · W F. We calculate the average estimation error as DISPLAYFORM0 Monte Carlo simulations with random initializations. FIG1 shows the average estimation error with respect to the sample complexity when K = 3 and d = 20, 35, 50 respectively. It can be seen that the estimation error decreases gracefully as we increase the sample size and matches with the theoretical prediction of error rates reasonably well. We further compare the performance of gradient descent algorithm applied to both the cross entropy loss and the squared loss, respectively. As shown in FIG1, when K = 3, d = 20, cross entropy loss with gradient descent achieves a much lower error than the squared loss. Clearly, the cross entropy loss is favored in the classification problem over the squared loss. In this paper, we have studied the model recovery of a one-hidden-layer neural network using the cross entropy loss in a multi-neuron classification problem. In particular, we have characterized the sample complexity to guarantee local strong convexity in a neighborhood (whose size we have characterized as well) of the ground truth when the training data are generated from a classification model. This guarantees that with high probability, gradient descent converges linearly to the ground truth if initialized properly. In the future, it will be interesting to extend the analysis in this paper to more general class of activation functions, particularly ReLU-like activations; and more general network structures, such as convolutional neural networks BID10 BID37. To begin, denote the population loss function as DISPLAYFORM0 where the expectation is taken with respect to the distribution of the training sample (x; y).The proof of Theorem 1 follows the following steps:1. We first show that the Hessian ∇ 2 f (W) of the population loss function is smooth with respect to ∇ 2 f (W) (Lemma 1);2. We then show that ∇ 2 f (W) satisfies local strong convexity and smoothness in a neighborhood of W, B(W, r) with appropriately chosen radius by leveraging similar properties of ∇ 2 f (W) (Lemma 2); 3. Next, we show that the Hessian of the empirical loss function ∇ 2 f n (W) is close to its popular counterpart ∇ 2 f (W) uniformly in B(W, r) with high probability (Lemma 3).4. Finally, putting all the arguments together, we establish ∇ 2 f n (W) satisfies local strong convexity and smoothness in B(W, r).We will first show that the Hessian of the population risk is smooth enough around W in the following lemma. Lemma 1. For sigmoid activations, assume W F ≤ 1, we have DISPLAYFORM1 holds for some large enough constant C, when W − W F ≤ 0.7.The proof is given in Appendix D.2. Lemma 1 together with the fact that ∇ 2 f (W) be lower and upper bounded, will allow us to bound ∇ 2 f (W) in a neighborhood around ground truth, given below. Lemma 2 (Local Strong Convexity and Smoothness of Population Loss). For sigmoid activations, there exists some constant C, such that DISPLAYFORM2 holds for all W ∈ B(W, r) with r:= min DISPLAYFORM3 The proof is given in Appendix D.3. The next step is to show the Hessian of the empirical loss function is close to the Hessian of the population loss function in a uniform sense, which can be summarized as following. Lemma 3. For sigmoid activations, there exists constant C such that as long as n ≥ C · dK log dK, with probability at least 1 − d −10, the following holds DISPLAYFORM4 where r:= min DISPLAYFORM5 The proof can be found in Appendix D.4.The final step is to combine Lemma 3 and Lemma 1 to obtain Theorem 1 as follows,Proof of Theorem 1. By Lemma 3 and Lemma 2, we have with probability at least DISPLAYFORM6 As long as the sample size n is set such that DISPLAYFORM7 holds for all W ∈ B (W, r). Similarly, we have DISPLAYFORM8 holds for all W ∈ B (W, r). We have established that f n (W) is strongly convex in B(W, r) in Theorem 1, thus there exists at most one critical point in B(W, r). The proof of Theorem 2 follows the steps below:1. We first show that the gradient ∇f n (W) concentrates around ∇f (W) in B(W, r) (Lemma 4), and then invoke BID21, Theorem 2) to guarantee there indeed exists a critical point W n in B(W, r);2. We next show W n is close to W and gradient descent converges linearly to W n with a properly chosen step size. The following lemma establishes that ∇f n (W) uniformly concentrates around ∇f (W). Lemma 4. For sigmoid activation function, assume W F ≤ 1, there exists constant C such that as long as n ≥ CdK log(dK), with probability at least 1 − d −10, the following holds DISPLAYFORM0 where r:= min DISPLAYFORM1 Notice that for the population risk function, f (W), W is the unique critical point in B(W, r) due to local strong convexity. With Lemma 3 and Lemma 4, we can invoke (, Theorem 2), which guarantees the following. Corollary 1. There exists one and only one critical point W n ∈ B (W *, r) that satisfies DISPLAYFORM2 We first show that W n is close to W. By the intermediate value theorem, ∃W ∈ B (W, r) such that DISPLAYFORM3 where the last inequality follows from the optimality of W n. By Theorem 1, we have DISPLAYFORM4 On the other hand, by the Cauchy-Schwarz inequality, we have DISPLAYFORM5 where the last line follows from Lemma 4. Plugging FORMULA0 and FORMULA0 into FORMULA0, we have DISPLAYFORM6 Now we have established there indeed exists a critical point in B(W, r). We can establish local linear convergence of gradient descent as below. Let W t be the estimate at the t-th iteration. According to the update rule, we have DISPLAYFORM7 Moreover, by the fundamental theorem of calculus BID17, ∇f n (W t) can be written as DISPLAYFORM8. By Theorem 1, we have DISPLAYFORM9 where DISPLAYFORM10 and H max = C. Therefore, we have DISPLAYFORM11 Hence, DISPLAYFORM12 as long as we set η < DISPLAYFORM13. In summary, gradient descent converges linearly to the local minimizer W n. The proof contains two parts. Part (a) proves that the estimation of the direction of W is sufficiently accurate, which follows the arguments similar to those in BID38 and is only briefly summarized below. Part (b) is different, where we do not require the homogeneous condition for the activation function, and instead, our proof is based on a mild condition in Assumption 2. We detail our proof in part (b).We first define a tensor operation as follows. For a tensor T ∈ R n1×n2×n3 and three matrices A ∈ R n1×d1, B ∈ R n2×d2, C ∈ R n3×d3, the (i, j, k)-th entry of the tensor T (A, B, C) is given by DISPLAYFORM0 (a) In order to estimate the direction of each w i for i = 1,..., K, BID38 shows that for the regression problem, if the sample size n ≥ dpoly (K, κ, t, log d), then DISPLAYFORM1 holds with high probability. Such a also holds for the classification problem with only slight difference in the proof as we describe as follows. The main idea of the proof is to bound the estimation error of P 2 and R 3 via Bernstein inequality. For the regression problem, Bernstein inequality was applied to terms associated with each neuron individually, and the bounds were then put together via triangle inequality in BID38, whereas for the classification problem here, we apply Bernstein inequality to terms associated with all neurons all together. Another difference is that the label y i of the classification model is bounded by nature, whereas the output y i in the regression model needs to be upper bounded via homogeneously bounded conditions of the activation function. A reader can refer to BID38 for the details of the proof for this part.(b) In order to estimate w i for i = 1,..., K, we provide a different proof from BID38, which does not require the homogeneous condition on the activation function, but assumes a more relaxed condition in Assumption 2.We define a quantity Q 1 as follows: DISPLAYFORM2 where l 1 is the first non-zero index such that M l1 = 0. For example, if l 1 = 3, then Q 1 takes the following form DISPLAYFORM3 where w = w/ w and by definition m 3,i (DISPLAYFORM4 Clearly, Q 1 has information of w i, which can be estimated by solving the following optimization problem: DISPLAYFORM5 where each entry of the solution takes the form DISPLAYFORM6 In the initialization, we substitute Q 1 (estimated from training data) for Q 1, V u i (estimated in part (a)) for s i w i into FORMULA55, and obtain an estimate β of β. We then substitute β for β and V u i for s i w i into to obtain an estimate a i of w i via the following equation DISPLAYFORM7 Furthermore, since m l1,i (x) has fixed sign for x > 0 and for l 1 ≥ 1, s i can be estimated correctly from the sign of β i for i = 1,..., K.For notational simplicity, let β 1,i:= DISPLAYFORM8 2, and then FORMULA0 and FORMULA57 DISPLAYFORM9 By Assumption 2 and, there exists a constant δ > 0 such that the inverse function g(·) of m 3,1 (·) has upper-bounded derivative in the interval (β 1,i − δ, β 1,i + δ), i.e., |g (x)| < Γ for a constant Γ. By employing the in BID38, if the sample size n ≥ dpoly (K, κ, t, log d), then Q 1 and Q 1, V u i and s i w i can be arbitrarily close so that |β DISPLAYFORM10 Thus, by and mean value theorem, we obtain DISPLAYFORM11 where ξ is between β 1,i and β 1,i, and hence |g (ξ)| < Γ. Therefore, DISPLAYFORM12, which is the desired . We introduce some useful definitions and that will be used in the proofs. The first one is the definition of norms of random variable, i.e. Definition 4 (Sub-gaussian and Sub-exponential norm). The sub-gaussian norm of a random variable X, denotes as X ψ2, is defined as DISPLAYFORM0 and the sub-exponential norm of X, denoted as X ψ1, is defined as DISPLAYFORM1 The definition is summarized from (, Def 5.7,Def 5.13), and if X ψ2 is upper bounded, then X is a sub-gaussian random variable and it satisfies DISPLAYFORM2 Next we provide the calculations of the gradient and Hessian of E [(W ; DISPLAYFORM3 where if j = l, DISPLAYFORM4 and if j = l, DISPLAYFORM5 Next we will evaluate ∆ j,l . From FORMULA28 we can write the hessian block more concisely as DISPLAYFORM6 where g j,l (W) = ξ j,l (W) (p(W)(1−p(W))) 2 ∈ R, and then by the mean value theorem, we can write g j,l (W) as DISPLAYFORM7 where W = η · W + (1 − η) W for some η ∈. Thus we can calculate ∆ j,l as DISPLAYFORM8 and plug it back to we can obtain DISPLAYFORM9 for the third equality we have used the fact that DISPLAYFORM10 since the variable of g j,l W is in the form of w i x. and for the last two inequalities, we have used Cauchy-Schwarz inequality. Our next goal is to upper bound E T 2 j,l,k. Further since DISPLAYFORM11 which aligns with x and the scalar coefficient is upper bounded by DISPLAYFORM12 are all upper bounded, thus we leave only the denominator. And then DISPLAYFORM13 holds for some constant C, where the second inequality follows from Lemma 5.Lemma 5. Let x ∼ N (0, I), t = max {w 1 2, · · · w K 2} and z ∈ Z such that z ≥ 1, for the sigmoid activation function φ (x) = 1 1+e −x, the following DISPLAYFORM14 holds for a large enough constant C which depends on the constant z. Plugging FORMULA1 into FORMULA1, we can obtain DISPLAYFORM15 Further since e DISPLAYFORM16, where we have used the assumption that W F ≤ 1 thus we can conclude that if DISPLAYFORM17 holds for some constant C. Proof. We will first present upper and lower bounds of the Hessian of the population risk at ground truth, i.e. ∇ 2 f (W), and then apply Lemma 1 to obtain a uniform bound in the neighborhood of W. As a reminder, DISPLAYFORM0 and let a = [a 1, · · ·, a K] ∈ R dK, we can write DISPLAYFORM1 the second inequality holds due to the fact that DISPLAYFORM2, and the last inequality follows from (b, Lemmas D.4 and D.6).Further more, we can uppder bound ∇ 2 f (W) as DISPLAYFORM3 where for the third and fourth inequality we have used the fact that φ w i x 1 − φ w i x ≤ 1 4and DISPLAYFORM4 Thus together with the lower bound we can conclude that DISPLAYFORM5 From Lemma 1, we have DISPLAYFORM6 therefore, when W − W F ≤ 0.7 and DISPLAYFORM7 i.e., when W − W F ≤ min DISPLAYFORM8 κ 2 λ, 0.7 for some constant C, we have DISPLAYFORM9 Moreover, within the same neighborhood, by the triangle inequality we have DISPLAYFORM10 D.4 PROOF OF LEMMA 3Proof. We adapt the analysis in BID21 to our setting. Let N be the -covering number of the Euclidean ball B (W, r). It is known that log N ≤ dK log (3r/) BID35. Let W = {W 1, · · ·, W N} be the -cover set with N elements. For any W ∈ B (W, r), let j (W) = argmin j∈[N] W − W j(W) F ≤ for all W ∈ B (W, r).For any W ∈ B (W, r), we have DISPLAYFORM11 Hence, we have DISPLAYFORM12 where the events A t, B t and C t are defined as DISPLAYFORM13 DISPLAYFORM14 DISPLAYFORM15 In the sequel, we will bound the terms P (A t), P (B t), and P (C t), separately.1. Upper bound P (B t). Before continuing, let us state a simple technical lemma that is useful for our proof, whose proof can be found in BID21. Let G i = v, ∇ 2 (W ; x i) − E ∇ 2 (W ; x) v where E[G i] = 0. Let a = a 1, · · ·, a K ∈ R dK. Then we can show that G i ψ1 is upper bounded, which we summariz as follows. Lemma 7. There exists some constant C such that DISPLAYFORM16' == 'We provide the first theoretical analysis of guaranteed recovery of one-hidden-layer neural networks under cross entropy loss for classification problems.' != 'We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.'

Question: Relational reasoning, the ability to model interactions and relations between objects, is valuable for robust multi-object tracking and pivotal for trajectory prediction. In this paper, we propose MOHART, a class-agnostic, end-to-end multi-object tracking and trajectory prediction algorithm, which explicitly accounts for permutation invariance in its relational reasoning. We explore a number of permutation invariant architectures and show that multi-headed self-attention outperforms the provided baselines and better accounts for complex physical interactions in a challenging toy experiment. We show on three real-world tracking datasets that adding relational reasoning capabilities in this way increases the tracking and trajectory prediction performance, particularly in the presence of ego-motion, occlusions, crowded scenes, and faulty sensor inputs. To the best of our knowledge, MOHART is the first fully end-to-end multi-object tracking from vision approach applied to real-world data reported in the literature. Real-world environments can be rich and contain countless types of interacting objects. Intelligent autonomous agents need to understand both the objects and interactions between them if they are to operate in those environments. This motivates the need for class-agnostic algorithms for tracking multiple objects-a capability that is not supported by the popular tracking-by-detection paradigm. In tracking-by-detection, objects are detected in each frame independently, e. g., by a pre-trained deep convolutional neural network (CNN) such as YOLO , and then linked across frames. Algorithms from this family can achieve high accuracy, provided sufficient labelled data to train the object detector, and given that all encountered objects can be associated with known classes, but fail when faced with objects from previously unseen categories. Hierarchical attentive recurrent tracking (HART) is a recently-proposed, alternative method for single-object tracking (SOT), which can track arbitrary objects indicated by the user . This is done by providing an initial bounding-box, which may be placed over any part of the image, regardless of whether it contains an object or what class the object is. HART efficiently processes just the relevant part of an image using spatial attention; it also integrates object detection, feature extraction, and motion modelling into one network, which is trained fully end-to-end. Contrary to tracking-by-detection, where only one video frame is typically processed at any given time to generate bounding box proposals, end-to-end learning in HART allows for discovering complex visual and spatio-temporal patterns in videos, which is conducive to inferring what an object is and how it moves. In the original formulation, HART is limited to the single-object modality-as are other existing end-to-end trackers (; ;). In this work, we present MOHART, a class-agnostic tracker with complex relational reasoning capabilities provided by a multi-headed self-attention module . MOHART infers the latent state of every tracked object in parallel, and uses self-attention to inform per-object states about other tracked objects. This helps to avoid performance loss under self-occlusions of tracked objects or strong camera motion. Moreover, since the model is trained end-to-end, it is able to learn how to manage faulty or missing sensor inputs. See fig. 1 for a high-level illustration of MOHART. In order to track objects, MOHART estimates their states, which can be naturally used to predict future trajectories over short temporal horizons, which is especially useful for planning in the context of autonomous agents. MOHART can be trained simultaneously for object tracking and trajectory prediction at the same time, thereby increasing statistical efficiency of learning. In contrast to prior art, where trajectory prediction and object tracking are usually addressed as separate problems with unrelated solutions, our work show trajectory prediction and object tracking are best addressed jointly. Section 2 describes prior art in tracking-by-detection, end-to-end tracking and predestrian trajectory prediction. In Section 3, we describe our approach, which uses a permutation-invariant self-attention module to enable tracking multiple objects end-to-end with relational reasoning. Section 4 contrasts our approach with multi-object trackers which do not explicitly enforce permutation invariance but have the capacity to learn it, simpler permutation-invariant architectures, as well as multiple single-object trackers running in parallel. We show that multi-headed self-attention significantly outperforms other approaches. Finally, in Section 5, we apply MOHART to real world datasets and show that permutation-invariant relational reasoning leads to consistent performance improvement compared to HART both in tracking and trajectory prediction. Tracking-by-Detection Vision-based tracking approaches typically follow a tracking-by-detection paradigm: objects are first detected in each frame independently, and then a tracking algorithm links the detections from different frames to propose a coherent trajectory (; ; ;). Motion models and appearance are often used to improve the association between detected bounding-boxes in a postprocessing step. Tracking-by-detection algorithms currently provide the state-of-the-art in multi-object tracking on common benchmark suites, and we fully acknowledge that MOHART is not competitive at this stage in scenarios where high-quality detections are available for each frame. MOHART can in principle be equipped with the ability to use bounding boxes provided by an object detector, but this is beyond the scope of this project. End-to-End Tracking A newly established and much less explored stream of work approaches tracking in an end-to-end fashion. A key difficulty here is that extracting an image crop (according to bounding-boxes provided by a detector), is non-differentiable and in high-variance gradient estimators. propose an end-to-end tracker with soft spatial-attention using a 2D grid of Gaussians instead of a hard bounding-box. HART draws inspiration from this idea, employs an additional attention mechanism, and shows promising performance on the real-world KITTI dataset . HART forms the foundation of this work. It has also been extended to incorporate depth information from RGBD cameras . propose an approach in which the crop corresponds to the scaled up previous bounding-box. This simplifies the approach, but does not allow the model to learn where to look-i. e., no gradient is backpropagated through crop coordinates. To the best of our knowledge, there are no successful implementations of any such end-to-end approaches for multi-object tracking beyond SQAIR , which works only on datasets with static s. On real-world data, the only end-to-end approaches correspond to applying multiple single-object trackers in parallel-a method which does not leverage the potential of scene context or inter-object interactions. Pedestrian trajectory prediction Predicting pedestrian trajectories has a long history in computer vision and robotics. Initial research modelled social forces using hand-crafted features (; ; ;) or MDP-based motion transition models , while more recent approaches learn from context information, e. g., positions of other pedestrians or landmarks in the environment. Social-LSTM ) employs a long short-term memory (LSTM) to predict pedestrian trajectories and uses max-pooling to model global social context. Attention mechanisms have been employed to query the most relevant information, such as neighbouring pedestrians, in a learnable fashion (; ;). Apart from relational learning, context , periodical time information , and constant motion priors (Schöller et al. ) have proven effective in predicting long-term trajectories. Our work stands apart from this prior art by not relying on ground truth tracklets. It addresses the more challenging task of working directly with visual input, performing tracking, modelling interactions, and, depending on the application scenario, simultaneously predicting future motions. As such, it can also be compared to Visual Interaction Networks (VIN) , which use a CNN to encode three consecutive frames into state vectors-one per object-and feed these into a recurrent neural network (RNN), which has an Interaction Network at its core. More recently, Relational Neural Expectation Maximization (R-NEM) has been proposed as an unsupervised approach which combines scene segmentation and relational reasoning (van). Both VINs and R-NEM make accurate predictions in physical scenarios, but, to the best of our knowledge, have not been applied to real world data. This section describes the model architecture in fig. 1. We start by describing the hierarchical attentive recurrent tracking (HART) algorithm , and then follow with an extension of HART to tracking multiple objects, where multiple instances of HART communicate with each other using multiheaded attention to facilitate relational reasoning. We also explain how this method can be extended to trajectory prediction instead of just tracking. HART is an attention-based recurrent algorithm, which can efficiently track single objects in a video. It uses a spatial attention mechanism to extract a glimpse g t, which corresponds to a small crop of the image x t at time-step t, containing the object of interest. This allows it to dispense with the processing of the whole image and can significantly decrease the amount of computation required. HART uses a CNN to convert the FIGURE 2: The relational reasoning module in MOHART based on multi-headed self-attention. Here, we show the computation of the interaction of the red object with all other objects. Object representations f t,m are computed using visual features, positional encoding and the hidden state from the recurrent module. These are linearly projected onto keys (k), queries (q), and values (v) to compute a weighted sum of interactions between objects, yielding an interaction vector o t,m. Subscripts t, m are dropped from all variables for clarity of presentation, so is the splitting into multiple heads. glimpse g t into features f t, which then update the hidden state h t of a LSTM core. The hidden state is used to estimate the current bounding-box b t, spatial attention parameters for the next time-step a t+1, as well as object appearance. Importantly, the recurrent core can learn to predict complicated motion conditioned on the past history of the tracked object, which leads to relatively small attention glimpses-contrary to CNN-based approaches , HART does not need to analyse large regions-of-interest to search for tracked objects. In the original paper, HART processes the glimpse with an additional ventral and dorsal stream on top of the feature extractor. Early experiments have shown that this does not improve performance on the MOTChallenge dataset, presumably due to the oftentimes small objects and overall small amount of training data. Further details are provided in Appendix B. The algorithm is initialised with a bounding-box 1 b 1 for the first time-step, and operates on a sequence of raw images x 1:T. For time-steps t ≥ 2, it recursively outputs bounding-box estimates for the current time-step and predicted attention parameters for the next time-step. The performance is measured as intersection-overunion (IoU) averaged over all time steps in which an object is present, excluding the first time step. Although HART can track arbitrary objects, it is limited to tracking one object at a time. While it can be deployed on several objects in parallel, different HART instances have no means of communication. This in performance loss, as it is more difficult to identify occlusions, ego-motion and object interactions. Below, we propose an extension of HART which remedies these shortcomings. Multi-object support in HART requires the following modifications. Firstly, in order to handle a dynamically changing number of objects, we apply HART to multiple objects in parallel, where all parameters between HART instances are shared. We refer to each HART instance as a tracker. Secondly, we introduce a presence variable p t,m for object m. It is used to mark whether an object should interact with other objects, as well as to mask the loss function (described in ) for the given object when it is not present. In this setup, parallel trackers cannot exchange information and are conceptually still single-object trackers, which we use as a baseline, referred to as HART (despite it being an extension of the original algorithm). Finally, to enable communication between trackers, we augment HART with an additional step between feature extraction and the LSTM. For each object, a glimpse is extracted and processed by a CNN (see fig. 1). Furthermore, spatial attention parameters are linearly projected on a vector of the same size and added to this representation, acting as a positional encoding. This is then concatenated with the hidden state of the recurrent module of the respective object (see fig. 2). Let f t,m denote the ing feature vector corresponding to the m th object, and let f t,1:M be the set of such features for all objects. Since different objects can interact with each other, it is necessary to use a method that can inform each object about the effects of their interactions with other objects. Moreover, since features extracted from different objects comprise a set, this method should be permutation-equivariant, i. e., the should not depend on the order in which object features are processed. Therefore, we use the multi-head self-attention block , which is able to account for higher-order interactions between set elements when computing their representations. Intuitively, in our case, SAB allows any of the trackers to query other trackers about attributes of their respective objects, e. g., distance between objects, their direction of movement, or their relation to the camera. This is implemented as follows, where o m is the output of the relational reasoning module for object m. Time-step subscripts are dropped to decrease clutter. In Eq. 1, each of the extracted features f t,m is linearly projected into a triplet of key k t,m, query q t,m and value v t,m vectors. Together, they comprise K, Q and V matrices with M rows and d q, d k, d k columns, respectively. K, Q and V are then split up into multiple heads H ∈ N +, which allows to query different attributes by comparing and aggregating different projection of features. Multiplying in Eq. 2 allows to compare every query vector q t,m,i to all key vectors k t,1:M,i, where the value of the corresponding dot-products represents the degree of similarity. Similarities are then normalised via a softmax operation and used to aggregate values V. Finally, outputs of different attention heads are concatenated in Eq. 3. SAB produces M output vectors, one for each input, which are then concatenated with corresponding inputs and fed into separate LSTMs for further processing, as in HART-see fig. 1. MOHART is trained fully end-to-end, contrary to other tracking approaches. It maintains a hidden state, which can contain information about the object's motion. One benefit is that in order to predict future trajectories, one can simply feed black frames into the model. Our experiments show that the model learns to fall back on the motion model captured by the LSTM in this case. Multilayer perceptron (MLP) In this version, the representations of all objects are concatenated and fed into a fully connected layer followed by ELU activations. The output is then again concatenated to the unaltered feature vector of each object. This concatenated version is then fed to the recurrent module of HART. This way of exchanging information allows for universal function approximation (in the limit of infinite layer sizes) but does not impose permutation invariance. DeepSets Here, the learned representations of the different objects are summed up instead of concatenated and then divided by total number of objects. This is closely related to DeepSets and allows for universal function approximation of all permutation invariant functions . Max-Pooling Similar to DeepSets, but using max-pooling as the permutation invariant operation. This way of exchanging information is used, e.g., by Crucially, each circle is randomly assigned its identity in each time step. Hence, the algorithm can not infer the forces exerted on one object without knowledge of the state of the other objects in the current time step. The forces in this scenario scale with 1/ √ r and the algorithm was trained to predict one time step into the future. HART (top) is indeed unable to predict the future location of the objects accurately. The achieved average IoU is 47%, which is only slightly higher than predicting the objects to have the same position in the next time step as in the current one (34%). Using the relational reasoning module, MOHART (bottom) is able to make meaningful predictions (76% IoU). The numbers in the bottom row indicate the self-attention weights from the perspective of the top left tracker (yellow number box). Interestingly, the attention scores have a strong correlation with the interaction strength (which scales with distance) without receiving supervision. and momentum conservation) between objects and with walls. To investigate how the model understands motion patterns and interactions between objects, we train it to predict future object locations in contrast to traditional tracking. In the first experiment, each circle exerts repulsive forces on each other, where the force scales with 1/r, r being the distance between them. Predicting the future location just using the previous motion of one object (i.e. without relational reasoning) accurately is therefore challenging. We show that HART as an end-toend single-object tracker is able to capture complex motion patterns and leverage these to make accurate predictions (see Appendix C). This indicates that HART is able to draw about the (deterministic, but not static) force field. In the second experiment, we introduce randomness, rendering the scenario not solvable for a single object tracker as it requires knowledge about the state of the other objects and relational reasoning (see fig. 3). In each time step, we assign a colour-coded identity to the objects. Objects of the same identity repel each other, object of different identities attract each other (the objects can be thought of as electrons and protons). The qualitative in fig. 3 show that MOHART, using self-attention for relational reasoning, is able to capture these interactions with high accuracy. Figure 4 (left) shows a quantitative comparison of augmenting HART with different relational reasoning modules when identities are re-assigned in every timestep (randomness = 1.0). Exchanging information between trackers of different objects in the latent space with an MLP leads to slightly worse performance than the HART baseline, while simple max-pooling performs significantly better (∆IoU ∼ 17%). This can be explained through the permutation invariance of the problem: the list of latent representation of the different objects has no meaningful order and the output of the model should therefore be invariant to the ordering of the objects. The MLP is in itself not permutation fig. 3 (randomness = 1.0). Right: performance depending on how often agents are re-assigned identities randomly (sequence length 15). The higher the randomness, the less static the force field is and the more vital relational reasoning is. For randomness = 0.0, identities still have to be reassigned in some cases in order to prevent deadlocks, this leads to a performance loss for all models, which explains lower performance of self-attention for randomness = 0.0. invariant and therefore prone to overfit to the (meaningless) order of the objects in the training data. Maxpooling, however, is permutation invariant and can in theory, despite its simplicity, be used to approximate any permutation invariant function given a sufficiently large latent space . Maxpooling is often used to exchange information between different tracklets, e.g., in the trajectory prediction domain; ). However, self-attention, allowing for learned querying and encoding of information, solves the relational reasoning task significantly more accurately. In fig. 4 (right), the frequency with which object identities are reassigned randomly is varied. The show that, in a deterministic environment, tracking does not necessarily profit from relational reasoning -even in the presence of long-range interactions. The less random, the more static the force field is and a static force field can be inferred from a small number of observations (see fig. 6). This does not mean that all stochastic environments profit from relational reasoning. What these experiments indicate is that tracking can not be expected to profit from relational reasoning by default in any environment, but instead in environments which feature (potentially non-deterministic) dynamics and predictable interactions. Having established that MOHART is capable of performing complex relational reasoning, we now test the algorithm on three real world datasets and analyse the effects of relational reasoning on performance depending on dataset and task. We find consistent improvements of MOHART compared to HART throughout. Relational reasoning yields particularly high gains for scenes with ego-motion, crowded scenes, and simulated faulty sensor inputs. We investigate three qualitatively different datasets: the MOTChallenge dataset , the UA-DETRAC dataset , and the Stanford Drone dataset ). To increase scene dynamics and make the tracking/prediction problems more challenging, we sub-sample some of the high framerate scenes with a stride of two, ing in scenes with 7-15 frames per second. Training and architecture details are given in Appendices A and B. We conduct experiments in three different modes: FIGURE 5: Camera blackout experiment on a street scene from the MOTChallenge dataset with strong egomotion. Solid boxes are MOHART predictions (for t ≥ 2), faded bounding boxes indicate object locations in the first frame. As the model is trained end-to-end, MOHART learns to fall back onto its internal motion model if no new observations are available (black frames). As soon as new observations come in, the model'snaps' back onto the tracked objects. Tracking. The model is initialised with the ground truth bounding boxes for a set of objects in the first frame. It then consecutively sees the following frames and predicts the bounding boxes. The sequence length is 30 time steps and the performance is measured as intersection over union (IoU) averaged over the entire sequence excluding the first frame. This algorithm is either applied to the entire dataset or subsets of it to study the influence of certain properties of the data. Camera Blackout. This simulates unsteady or faulty sensor inputs. The setup is the same as in Tracking, but sub-sequences of the input are replaced with black images. The algorithm is expected to recognise that no new information is available and that it should resort to its internal motion model. Prediction. Testing MOHART's ability to capture motion patterns, only the first two frames are shown to the model followed by three black frames. IoU is measured seperately for each time step. On the MOTChallenge dataset, HART achieves 66.6% intersection over union (see Table 1), which in itself is impressive given the small amount of training data of only 5225 training frames and no pre-training. MOHART achieves 68.5% (both numbers are averaged over 5 runs, independent samples t-test ed in p < 0.0001). The performance gain increases when only considering ego-motion data. This is readily explained: movements of objects in the image space due to ego-motion are correlated and can therefore be better understood when combining information from movements of multiple objects, i.e. performing relational reasoning. In another ablation, we filtered for only crowded scenes by requesting five objects to be present for, on average, 90% of the frames in a sub-sequence. For the MOT-Challenge dataset, this only leads to a minor increase of the performance gain of MOHART indicating that the dataset exhibits a sufficient density of objects to learn interactions. The biggest benefit from relational reasoning can be observed in the camera blackout experiments (setup explained in Section 5.1). Both HART and MOHART learn to rely on their internal motion models when confronted with black frames and propagate the bounding boxes according to the previous movement of the objects. It is unsurprising that this scenario profits particularly from relational reasoning. Qualitative tracking and camera blackout are shown in fig. 5 and Appendix E. Tracking performance on the UA-DETRAC dataset only profits from relational reasoning when filtering for crowded scenes (see Table 2). The fact that the performance of MOHART is slightly worse on the vanilla dataset (∆ = −0.3%) can be explained with more overfitting. As there is no exchange between trackers for each object, each object constitutes an independent training sample. The Stanford drone dataset (see Table 3) is different to the other two-it is filmed from a birds-eye view. The scenes are more crowded and each object covers a small number of pixels, rendering it a difficult problem for tracking. The dataset was designed for trajectory prediction-a setup where an algorithm is typically provided with ground-truth tracklets in coordinate space and potentially an image as context information. The task is then to extrapolate these tracklets into the future. The tracking performance profits from relational reasoning more than on the UA-DETRAC dataset but less than on the MOTChallenge dataset. The performance gain on the camera blackout experiments are particularly strong when only considering cyclists. In the prediction experiments (see Appendix D), MOHART consistently outperforms HART. On both datasets, the model outperforms a baseline which uses momentum to linearly extrapolate the bounding boxes from the first two frames. This shows that even from just two frames, the model learns to capture motion models which are more complex than what could be observed from just the bounding boxes (i.e. momentum), suggesting that it uses visual information (HART & MOHART) as well as relational reasoning (MOHART). With MOHART, we introduce an end-to-end multi-object tracker that is capable of capturing complex interactions and leveraging these for precise predictions as experiments both on toy and real world data show. However, the experiments also show that the benefit of relational reasoning strongly depends on the nature of the data. The toy experiments showed that in an entirely deterministic world relational reasoning was much less important than in a stochastic environment. Amongst the real-world dataset, the highest performance gains from relational reasoning were achieved on the MOTChallenge dataset, which features crowded scenes, ego-motion and occlusions. The MOTChallenge and the UA-DETRAC dataset discussed in this section are intended to be used as a benchmark suite for multi-object-tracking in a tracking-by-detection paradigm. Therefore, ground truth bounding boxes are only available for the training datasets. The user is encouraged to upload their model which performs tracking in a data association paradigm leveraging the provided bounding box proposals from an external object detector. As we are interested in a different analysis (IoU given inital bounding boxes), we divide the training data further into training and test sequences. To make up for the smaller training data, we extend the MOTChallenge 2017 dataset with three sequences from the 2015 dataset (ETHSunnyday, PETS09-S2L1, ETH-Bahnhof). We use the first 70% of the frames of each of the ten sequences for training and the rest for testing. Sequences with high frame rates (30Hz) are sub-sampled with a stride of two. For the UA-DETRAC dataset, we split the 60 available sequences into 44 training sequences and 16 test sequences. For the considerably larger Stanford Drone dataset we took three videos of the scene deathCircle for training and the remaining two videos from the same scene for testing. The videos of the drone dataset were also sub-sampled with a stride of two to increase scene dynamics. The architecture details were chosen to optimise HART performance on the MOTChallenge dataset. They deviate from the original HART implementation as follows: A presence variable predicts whether an object is in the scene and successfully tracked. This is trained with a binary cross entropy loss. The maximum number of objects to be tracked simultaneously was set to 5 for the UA-DETRAC and MOTChallenge dataset. For the more crowded Stanford drone dataset, this number was set to 10. The feature extractor is a three layer convolutional network with a kernel size of 5, a stride of 2 in the first and last layer, 32 channels in the first two layers, 64 channels in the last layer, ELU activations, and skip connections. This converts the initial 32 × 32 × 3 glimpse into a 7 × 7 × 64 feature representation. This is followed by a fully connected layer with a 128 dimensional output and an elu activation. The spatial attention parameters are linearly projected onto 128 dimensions and added to this feature representation serving as a positional encoding. The LSTM has a hidden state size of 128. The self-attention unit in MOHART comprises linear projects the inputs to dimensionality 128 for each keys, queries and values. For the real-world experiments, in addition to the extracted features from the glimpse, the hidden states from the previous LSTM state are also fed as an input by concatinating them with the features. In all cases, the output of the attention module is concatenated to the input features of the respective object. As an optimizer, we used RMSProp with momentum set to 0.9 and learning rate 5 * 10 −6. For the MOTChallenge dataset and the UA-DETRAC dataset, the models were trained for 100,000 iterations of batch size 10 and the reported IoU is exponentially smoothed over iterations to achieve lower variance. For the Stanford Drone dataset, the batch size was increased to 32, reducing time to convergence and hence model training to 50,000 iterations. In our first experiment in the toy domain (Figure 6), four circles each exert repulsive forces on each other, where the force scales with 1/r, r being their distance. HART is applied four times in parallel and is trained to predict the location of each circle three time steps into the future. The different forces from different objects lead to a non-trivial force field at each time step. Predicting the future location just using the previous motion of one object (Figure 6 shows that each spatial attention box covers only the current object) accurately is therefore challenging. Surprisingly, the single object tracker solves this task with an average of 95% IoU over sequences of 15 time steps. This shows the efficacy of end-to-end tracking to capture complex motion FIGURE 7: Peeking into the future. Only the first two frames are shown to the tracking algorithm followed by three black frames. MOHART learns to fall back on its internal motion model when no observation (i.e. only a black frame) is available. The reported IoU scores show the performance for the respective frames 0, 1, 2, and 3 time steps into the future. patterns and use them to predict future locations. This, of course, could also be used to generate robust bounding boxes for a tracking task. In the from the prediction experiments (see Figure 7) MOHART consistently outperforms HART. On both datasets, the model outperforms a baseline which uses momentum to linearly extrapolate the bounding boxes from the first two frames. This shows that even from just two frames, the model learns to capture motion models which are more complex than what could be observed from just the bounding boxes (i.e. momentum), suggesting that it uses visual information (HART & MOHART) as well as relational reasoning (MOHART). The strong performance gain of MOHART compared to HART on the UA-DETRAC dataset, despite the small differences for tracking on this dataset, can be explained as follows: this dataset features little interactions but strong correlations in motion. Hence when only having access to the first two frames, MOHART profits from estimating the velocities of multiple cars simultaneously. In Section 5, we tested MOHART on three different real world data sets and in a number of different setups. Figure 8 shows qualitative both for HART and MOHART on the MOTChallenge dataset. Furthermore, we conducted a set of camera blackout experiments to test MOHART's capability of dealing with faulty sensor inputs. While traditional pipeline methods require careful consideration of different types of corner cases to properly handle erroneous sensor inputs, MOHART is able to capture these automatically, especially when confronted with similar issues in the training scenarios. To simulate this, we replace subsequences of the images with black frames. Figure 9 and Figure 5 show two such examples from the test data together with the model's prediction. MOHART learns not to update its internal model when confronted with black frames and instead uses the LSTM to propagate the bounding boxes. When proper sensor input is available again, the model uses this to make a rapid adjustment to its predicted location and'snap' back onto the object. This works remarkably well in both the presence of occlusion (Figure 9) and ego-motion (Figure 5). Tables 1 to 3 show that the benefit of relational reasoning is particularly high in these scenarios specifically. These experiments can also be seen as a proof of concept of MOHART's capabalities of predicting future trajectories-and how this profits from relational reasoning. FIGURE 9: Camera blackout experiment on a pedestrian street scene from the MOTChallenge dataset without ego-motion. Subsequent frames are displayed going from top left to bottom right. Shown are the inputs to the model (some of them being black frames, i.e. arrays of zeroes) and bounding boxes predicted by MOHART (coloured boxes). This scene is particularly challenging as occlusion and missing sensor input coincide (fourth row).

represent the tldr for finding the corresponding scientific article

MOHART uses a self-attention mechanism to perform relational reasoning in multi-object tracking.

Represent the short summary for finding the corresponding scientific article (long)

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.