{'fraction_non_alphanumeric': 0.038720689964962716, 'fraction_numerical': 0.00844488365825173, 'mean_word_length': 4.609221466364324, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
NEMO: Future Object Localization Using Noisy Ego Priors

Predictive models for forecasting future behavior of road agents should consider the multi-modal nature and be aware of the uncertainty of their predictions. Particularly from the egocentric view where the motion of other agents is captured with respect to the ego-motion, the uncertainty of ego-motion prediction is critical to determine their interactive reactions and behaviors. Along this line, we propose NEMO (Noisy Ego MOtion priors for future object localization) for future forecast of road agents in the egocentric view. A predictive distribution of future forecast is jointly modeled with the uncertainty of predictions. For this, we divide the problem into two tasks: future ego-motion prediction and future object localization. We first model the multi-modal distribution of future ego-motion with uncertainty estimates. The resulting distribution of ego-behavior is used to sample multiple modes of future ego-motion. Then, each modality is used as a prior to understand the interactions between the ego-vehicle and target agent. We predict the multi-modal future locations of the target from individual modes of the ego-vehicle, modeling the uncertainty of target's behavior. To this end, we extensively evaluate the proposed framework using the publicly available benchmark dataset (HEV-I) with an addition of Inertial Measurement Unit (IMU) data to it.


I. INTRODUCTION
Predicting the future motion of agents in dynamic environments is one of the important tasks in vehicle control and vehicle navigation [1][2][3]. For safe operation of cars, these systems should understand the context of the environment and predict the future locations of agents or their trajectories to avoid collisions. Although some approaches [4][5][6] have worked on finding a deterministic solution from the given history of agent's motion, future forecast is inherently multimodal in nature where multiple paths are plausible.
Recent approaches [7][8][9][10][11] have been focusing on modeling a distribution of all possible paths to tackle the multimodality of future forecast. However, their predictive distribution is either (i) naively learned in a data driven manner with no consideration of the uncertainty; or (ii) simply generated to sample different types of motion using deep generative models. To provide a more persuasive solution to the multi-modality of the problem, a distribution of agents' behavior should be jointly learned with uncertainty estimates -uncertainty of data (i.e., aleatoric) as well as that of the prediction model (i.e., epistemic). In this way, the predicted modalities can be quantified how uncertain/noisy they are, thereby the future motions generated from the predictive distribution are more confident within the modality.
There have been research efforts in single-modal future forecast [12], [13] to show that uncertainty embedding  improves the overall performance for predicting agents' future motion. However, [12] restricts their uncertainty to be epistemic and overlooks noise inherent in the dataset, which is infeasible to recover from a small number of the observations. Also, their problem setting (i.e., aerial RGB imagery as input) is not practical to deploy in autonomous driving (AD) and advanced driving assistance systems (ADAS). In [13], both aleatoric and epistemic uncertainty are considered from the ego-car perspective, where ego-motion as a prior affects the future motion of other agents. However, the uncertainty of ego-motion prediction is not taken into account, which seems most critical to forecast the interactive behavior of other agents. In this view, we propose a multi-modal future forecast framework, NEMO, which aims to (i) model both aleatoric and epistemic uncertainty of ego-vehicle as well as other agents; (ii) condition future object localization on multiple modes of ego-motion priors, which results in different types of target agent's behavior; and (iii) apply such a framework to immediate applications of AD and ADAS with an easyavailable frontal-facing RGB camera. Fig. 1 shows the overall idea of the proposed framework. NEMO first models both aleatoric and epistemic uncertainty of future ego-behavior using the past motion history of the ego-vehicle. Then, the multiple modes of future ego-motion are sampled from the probability distribution with uncertainty estimates. Each modality is given to the future object localization stream as a prior to assess interactive responses of the target agent with respect to the different types of future ego-motion. We further consider the uncertainty of target agent's future motion and its multi-modality. An overview of the proposed approach is presented in Fig. 2. In this process, NEMO generates multi-modal future motions of the target over the uncertainty of future ego-motion, which is reflective of real-world egocentric interactions. For more accurate ego-motion prediction, we release new IMU data for HEV-I [5], which can extend its use far beyond the future object localization problem to visual odometry estimate [14] and other 2D image-based control learning tasks [15], [16]. The updated IMU sensor data will be made available at https://usa.honda-ri.com/hevi

A. Uncertainty Modeling
Denker et al. [17] and MacKay et al. [18] have made an effort to study the uncertainty of the model parameters using Bayesian neural networks (BNNs). Recently, Gal et al. [19], [20] have shown that Bayesian inference can be approximated with a traditional network architecture. They models epistemic uncertainty by sampling from the posterior distribution of the learned model using dropout during inference, which is equivalent to approximated Bayesian inference. In addition, Kendall et al. [21] shows that aleatoric uncertainty can be captured using negative log-likelihood loss by outputting the extra parameters for variance from the network output. It enables the network to learn the noise parameters that are originated by noise inherent in the dataset. Following the success of uncertainty modeling in single-modal forecast [12], [13], we embed the uncertainty of future prediction into our multi-modal pipeline.
The works in [4], [5], [13] are directly related to future object localization in the first-person view. Yagi et al. [4] uses human poses as a prior to forecast the future motion of humans, but their model is not applicable to vehicles in the transportation domain. The rest works [5], [13] mainly focus on single-modal localization of road agents. Yao et al. [5] uses object appearance and the ground-truth future egomotion for future localization, but they do not consider the uncertainty of prediction. Unlike this method, Battacharya et al. [13] generates the future ego-motion and employs it as a prior to localize other agents with uncertainty estimates. However, they simply overlook the uncertainly of the future ego-motion, which is critical to determine the interactive reactions of other agents and their future behaviors toward the ego-vehicle. In contrast, we address the uncertainty of future ego-motion prediction and introduce noisy ego-priors for multi-modal future object localization.

C. Future Trajectory Forecast
Besides these works, there have appeared numerous studies on future trajectory forecast in top-down view. Social-LSTM [7] introduces a social pooling module for interaction encoding, and Social-GAN [9] efficiently improves its performance by replacing the pooling with a multi-layer perceptron. Social-Attention [10] introduces a soft attention mechanism to find more useful interactions. Other than these methods, Gated-RN [12] observes spatio-temporal interactions using images and infers relational behavior between agents. Their relational inference is adopted into DROGON [11] that uses intention as a prior for trajectory prediction, focusing on causation between intention and the future motion.
However, these methods do not provide their seamless applicability to egocentric videos captured from driver's perspective because of the following reasons: (i) unlike their static platform in top-down view, the distance between objects should be jointly assumed from the location and scale in frontal view images; (ii) the interactions between agents are relative to the ego-motion, while their models are not able to be aware of it; and (iii) their consideration of the uncertainty is zero or minimal, which might not be a comprehensive solution to multi-modal predictions. Therefore, we present NEMO for future forecast in the egocentric view.

III. BAYESIAN UNCERTAINTY MODELING
In this section, we show how aleatoric and epistemic uncertainty can be jointly modeled using a single framework.

A. Aleatoric Modeling
Aleatoric uncertainty comes from inherent noise in the observations due to the probabilistic variability. To model this type of uncertainty during training, the network incorporates noise parameters (µ t , Σ yt ) at time t, where µ denotes the mean and Σ yt denotes the co-variance matrix for the groundtruth label y t . The co-variance matrix Σ yt is learned using negative log-likelihood loss function as follows: We predict (µ t , Σ yt ) at T observed time-steps from time T obs +1 to T pred . Eq. 1 is used to compute how likely the observations come from the posterior distribution N (µ t , Σ yt ).
For numerical stability, having zeros in denominator is not suggested. Thus, we substitute log(Σ 2 yt ) with s yt , which results in Eq. 2 as follows:

B. Epistemic Modeling
Epistemic uncertainty is caused by the model's weight parameters that are inadequately measured from the observations. Thus, this type of uncertainty is reducible by taking more measurements. Dropout is well-known in deep learning community, which is originally used as a regularization method to avoid overfitting. However, recent study in [20] introduces dropout to learn a distribution of weights to approximate variational inference in Bayesian modeling [19]. Given the dataset X, Y the posterior over weights P (w|X, Y ) is approximated using a dropout distribution q(w) [36]. During inference, we generate N samples from the distribution q(w) of the network's learned weight parameters w using dropout. Then, N number of noisy outputs are used to compute the variance Σ y between the predicted outputs fŵ i (x) and ground-truth labels y t at each time-step t. The details are shown in Eq 3 as follows: log(P (y t |fŵ(x t ))), Note that the computation of the mean and variance is performed during inference using dropout.

C. Joint Modeling of Aleatoric and Epistemic Uncertainty
We update the noise parameters (µ y , Σ y ) by adding aleartoric uncertainty given in Eq. 2 to epistemic uncertainty in Eq. 3. The total variance and mean is computed as shown in Eq. 4.
for randomly sampled weightsŵ from the dropout distribution q(w).
As a result, we output the noise parameters for the data posterior distribution N (µ t , Σ yt ) together with the learned distribution of the model's weights q(w) during inference. In practice, different node connectionsŵ i are sampled for N times using dropout, and corresponding aleatoric and epistemic uncertainty is computed using Eq. 4.

IV. NEMO FRAMEWORK
The proposed NEMO framework is designed to properly model the uncertainty of future ego-motion, which is most important to determine other agents' future motion in the egocentric view. As shown in Fig. 2, we divide the future forecast problem into two tasks: future ego-motion prediction and future object localization. For future ego-motion prediction, we first encode the past motion of the ego-vehicle and generate its future motion through the ego-motion decoder. To model the joint uncertainty of ego-motion, the model weightsŵ E for the ego-motion decoder are drawn from the weight distribution q(w E ). Here, we generate multiple modes of prediction over the uncertainty distribution P (E|x E ,ŵ E ) over the velocity v and yaw rateθ, where E = {v,θ} is the future ego-motion and x E is past ego-motion.
From the other stream, the motion of other agents are encoded using the bounding box encoder and flow encoder, respectively. We then concatenate these information and use the bounding box decoder to learn the weight parameterŝ w B . To properly model their future behavior respective to the noisy ego-motion, we use the output of future ego-motion prediction as a prior. In this way, the bounding box decoder reacts to each modalityÊ of the ego-vehicle while predicting others' future motion. Similar to joint uncertainty modeling of the ego-motion decoder, the weightsŵ B for the bounding box decoder are drawn from the weight distribution q(w B ). We estimate the noise parameters for the center (c x , c y ) and the dimension (w, h) of the bounding box using the weightŝ w B . Finally, we predict B = {c x , c y , w, h} by sampling from the uncertainty distribution P (B|G,ŵ B ,Ê), G = Φ(x F ) Θ(x B ) concatenation of past flow x F and bounding box x B encoding (Φ is flow encoder, Θ is past bounding box encoder, and is concatenation operator).

A. Future Ego-Motion Prediction
Predicting the future ego-motion given the past observation is prone to multiple possibilities. Thus, we model multi-modal predictions for the ego-motion with uncertainty estimates. We take the past observations x E,t from IMU odometry (v t ,θ t ) for T obs time steps and encode the egomotion using Gated Recurrent Units (GRU). Multi Layer Perceptron (MLP) is used to convert the past ego-motion to the embedding of the GRU. The prediction output of a GRUbased decoder is a 5-dimensional vector [µ v , µθ, σ v , σθ, ρ] at each future time step from T obs + 1 to T pred , where µ v is mean and σ v is noise in velocity prediction, µθ is mean and σθ is noise in yaw rate prediction, and ρ is correlation coefficient between those two dimensions. During inference, we sample velocityv t and yaw rateθ t from the uncertainty distribution generated by the noise parameters. The input is x E,t = [v t ,θ t ] t={1:T obs } and output is y E,t = [v t ,θ t ] t={T obs +1:T pred } .

B. Future Object Localization
We use the past bounding box information x B,t and past ROI pooled Flow information x F,t , which are separately processed using the respective GRU encoders. In addition to that, we use the predicted future ego-motion y E,t as a prior to generate future motion of the target y O,t at time t. The output of future object localization is a 10-dimensional vector [µ cx , µ cy , σ cx , σ cy , ρ c , µ w , µ h , σ w , σ h , ρ d ] at each future time step, where (µ w , µ h , σ w , σ h , ρ c ) is a set of mean and covariance parameters for the center and (µ w , µ h , σ w , σ h , ρ d ) is that for the bounding box dimension. By assuming two 2D-Gaussian functions for the uncertainty, we reduced the number of parameters to regress from 20 (4D-Gaussian) to 10 (2 2D-Gaussian). The output center (ĉ x,t ,ĉ y,t ) and dimension (ŵ t ,ĥ t ) of the bounding boxes are sampled from the uncertainty distribution generated by these noise parameters.

V. EXPERIMENTS A. Dataset
The HEV-I dataset [5] is publicly available, which consists of 2477 vehicles in 230 videos collected from urban driving scenarios. The dataset includes the motion of the ego-vehicle obtained by ORB-SLAM2 [37]. However, the estimated translation is a normalized unit vector which does not recover the full 3D motion of the ego-vehicle. Moreover, the dynamic motion of surrounding agents often causes association errors, which severely affects the rotation estimates. Therefore, we provide IMU odometry (v,θ) decoded from the CAN message of the ego-vehicle. We observed that a drift error is less than 0.2 meters for new IMU odometry compared to LIDAR odometry for the HEV-I sequences.

B. Implementation
NEMO is trained with a TITAN Xp GPU using the PyTorch framework. We first train the ego-motion prediction stream from scratch. Then, the learned model is jointly optimized with the future object localization stream.
1) Future Ego Motion Prediction: We use a batch size of 32 and learning rate of 0.001 for negative log-likelihood loss as future ego motion loss with the RMSProp optimizer. For the learning rate, we drop the value by a factor of 2 after every 20 epochs. The network is converged after 100 epochs. For evaluation, we reconstruct the 2D trajectory from the predicted velocity and yaw rate using Eq. 5 with respect to the last observed frame, assuming planar motion.
where R i+1 i ∈ R 2×2 is a 2D rotation matrix and T i+1 i ∈ R 2 is a 2D translation vector. We use a right handed coordinate system for the ego-motion. Note that velocity and yaw rate is converted into translation in meters and degrees between every time step using the time interval of 0.1 sec.
2) Future Object Localization: We use input image of size W = 1920 and H = 1200 pixels. The bounding box centers and dimensions are normalized to a range of [0, 1]. While training the module, we use a batch size of 32 and learning rate of 0.001 that is reduced by a factor of 5 after every 20 epochs. We use a weighting λ e = 0.2 for the pre-trained model for future ego motion prediction loss and λ f = 1 for future object localization loss.

VI. RESULTS
Although prior works [5], [13] report 1 second of prediction is enough for safe operation of the vehicle travelling with a speed up to 25 MPH, we found that it is underestimated for natural driving in urban areas. Note that the cars travel with a speed up to 43 MPH in the HEV-I dataset. Thus, we predict the motion of 2 seconds in future, observing that of past 1 second. We sample k = 10 future predictions from the distribution and report the result with a minimum error as y opt = min k ŷ k − y 2 , whereŷ k is trajectory prediction sample and y is ground truth trajectory. For evaluation, we compute Average Distance Error (ADE) and Final Distance Error (FDE) for motion prediction, and Final Intersection over Union (FIOU) for bounding box prediction. The reported ADE/FDE for ego-motion prediction is in meter, while those for bounding box prediction is in pixel.

A. Future Ego Motion Prediction
We use Const-Vel [38] as one of our baselines where the output for future 20 time steps is the same as the input observed at time t = 10. The RNN baseline has a GRU-based encoder and decoder, which highly improves the performance compared to the Const-Vel baseline. For RNN-E, we model epistemic uncertainty with a dropout in the decoder's MLP layer. RNN-A is with aleatoric uncertainty, where we sample 10 trajectories from the learned likelihood distribution parameters. RNN-AE is combined aleatoric and epistemic uncertainty modeling. We observe that uncertainty modeling apparently improves the performance in all three cases (RNN-E, RNN-A, RNN-AE) compared to their counterparts (Const-Vel, RNN). Overall, RNN-AE performs better than all the baselines as shown in Fig. 3 and Tbl. I, validating the efficacy of uncertainty modeling. The uncertainty estimates are shown in Fig. 4

B. Future Object Localization
We use the Const-Vel baseline for bounding box prediction in pixel coordinates. However, it does not consider the scaling factors in the egocentric videos. For fair comparison, . Also, the predicted trajectory of RNN-AE (Ours) is visualize with the ground-truth . Fig. 6: Qualitative evaluation of NEMO (RNN-AE) with the uncertainty of future object localization. The predicted centers of the bounding box from T obs + 1 to T pred are shown as a trajectory with ground-truth . Also, the bounding box at T pred is sampled from the probability distribution (red indicates high probability).
we linearly scale the bounding box dimensions using the transformation of the last two observations. As shown in Tbl II, linearly scaled bounding box dimensions improves the FIOU performance. RNN-NP does not use any priors of the ego-motion. Interestingly, both ADE and FDE are better but its FIOU is worse compared to the Const-Vel baseline. RNN-P (ORB) uses ORB-SLAM2 [37] based ego-motion as a prior [5], while RNN-P (IMU) uses IMU odometry for ego-motion prediction similar to [13]. We observe that IMUbased ego-motion, RNN-P (IMU), improves the performance when compared with the ORB-SLAM2-based ego-motion, which validates our claim to use IMU odometry for future object localization. For RNN-AP, we use the pre-trained ego-motion prediction module with aleatoric uncertainty and train jointly with future object localization. Similarly, RNN-EP uses the pre-trained ego-motion prediction module with epistemic uncertainty and is trained jointly with future object localization. Use of these uncertainty models to condition other agents' motion forecast significantly improves the overall performance. This comparison validates the rationale of our use of the uncertainty to model more robust interactions of other agents with the ego-vehicle. For RNN-A, both egomotion prediction and future object localization modules are trained with aleatoric uncertainty. Similarly, RNN-E is trained with epistemic uncertainty for both tasks. These baseline models further decrease the error rate compared to RNN-AP and RNN-EP. Finally, RNN-AE (Ours) models aleatoric and epistemic uncertainty throughout the NEMO pipeline, which best predicts future motion of agents as well as others' bounding box locations and scales. Fig. 5

VII. CONCLUSION
We introduced the NEMO framework to condition future object localization on the uncertainty of future ego-motion priors. For this, we jointly modeled aleatoric and epistemic uncertainty of ego-motion prediction to sample multiple modes of future ego-behavior. Then, each modality was used as a prior to capture interactive reactions of other agents with respect to the different types of ego-motion. We also considered the uncertainty of future object localization as well as its multi-modality. To this end, ablative tests were conducted using the public benchmark dataset comparing NEMO with the state-of-the-art methods and self-generated baseline models. We observed that combined epistemic and aleatoric uncertainty modeling in both future ego-motion prediction and future object localization achieved lowest prediction error from both motion and bounding box prediction.