Memory Enhanced Global-Local Aggregation for Video Object Detection
Yihong Chen Yue Cao Han Hu Liwei Wang
cs.CV cs.LG eess.IV







Memory Enhanced Global-Local Aggregation for Video Object Detection
    Yihong Chen1,4This work is done when Yihong Chen is an intern at Microsoft Research Asia.   Yue Cao3   Han Hu3   Liwei Wang1,2

1Center of Data Science, Peking University 

2Key Laboratory of Machine Perception, MOE, School of EECS, Peking University 
 
3Microsoft Research Asia   4Zhejiang Lab

chenyihong@pku.edu.cn   {yuecao,hanhu}@microsoft.com   wanglw@cis.pku.edu.cn











    
	
	  NOMAD Laboratory, Fritz Haber Institute of the Max Planck Society, Berlin, Germany;    Machine Learning Group, Technische Universität Berlin, Germany;    Institute of Mathematics, Technische Universität Berlin, Germany;    Citrine Informatics, Redwood City, CA, USA    Present address: Department of Computer and Information Science, University of Konstanz, Germany.
 

===================================================================================================================================================================================================================================================================================================================================================================================================





   How do humans recognize an object in a piece of video? Due to the deteriorated quality of single frame, it may be hard for people to identify an occluded object in this frame by just utilizing information within one image. We argue that there are two important cues for humans to recognize objects in videos: the global semantic information and the local localization information. Recently, plenty of methods adopt the self-attention mechanisms to enhance the features in key frame with either global semantic information or local localization information. In this paper we introduce memory enhanced global-local aggregation (MEGA) network, which is among the first trials that takes full consideration of both global and local information. Furthermore, empowered by a novel and carefully-designed Long Range Memory (LRM) module, our proposed MEGA could enable the key frame to get access to much more content than any previous methods. Enhanced by these two sources of information, our method achieves state-of-the-art performance on ImageNet VID dataset. Code is available at <https://github.com/Scalsol/mega.pytorch>.






§ INTRODUCTION




What differs detecting objects in videos from detecting them in static images? A quick answer is the information lies in the temporal dimension. When isolated frame may suffer from problems such as motion blur, occlusion or out of focus, it is natural for humans to find clues from the entire video to identify objects.

When people are not certain about the identity of an object, they would seek to find a distinct object from other frames that shares high semantic similarity with current object and assign them together. We refer this clue as global semantic information for every frame in the video could be reference to. But semantic information alone fails in the case if we are not sure about whether an object exists or not, , a black cat walking in the dark. We could not rely on the semantic information to tell us where it is, as the existence of the instance has not been approved in key frame yet. This problem could be alleviated if nearby frames were given. With information such as motion calculated by the difference between nearby frames, we could localize the object in key frame. We refer this source of information as local localization information. To be general, people identify objects mainly with these two sources of information.






Consequently, it is direct to borrow this idea from human to enhance the video object detection methods with the information inside the whole video, just as shown in Figure <ref>. However, due to the huge amount of boxes exist in the whole video, enhancement with the information in the entire video is infeasible. This motivates us to perform approximation while keeping a balance between efficiency and accuracy. Recent methods towards solving video object detection could be viewed as different approaches of approximation and can be classified into two main categories: local aggregation methods and global aggregation methods. Methods like <cit.> consider utilizing both semantic and localization information in a short local range just as shown in Figure <ref>. On the other hand, <cit.> just consider the semantic impact between boxes, as shown in Figure <ref>. Unfortunately, none of these methods takes a joint view of both local and global information. We refer this as the ineffective problem.





Another problem existing in recent works is the size of frames for aggregation, which means the amount of information the key frame could gather from. In previous state-of-the-art methods <cit.>, only 20-30 reference frames are selected, which lasts only 1-2 seconds, for feature aggregation. This is also illustrated in Figure <ref> and <ref>. We argue that the size of aggregation at this scale is an insufficient approximation of either local influence or global influence, to say nothing of Figure <ref>.



In this paper, we present Memory Enhanced Global-Local Aggregation (MEGA) to overcome the aforementioned ineffective and insufficient problem. Specifically, MEGA augments candidate box features of key frame by effectively aggregating global and local information. 

MEGA is instantiated as a multi-stage structure. In the first stage, MEGA aims to solve the ineffective problem by aggregating both global and local information to key frames. However, just as shown in the top half of Figure <ref>, the available content is still quite limited. So in the second stage, we introduce a novel Long Range Memory (LRM) module which enables key frame to get access to much more content than any previous methods. In particular, instead of computing features from scratch for current key frame, we reuse the precomputed features obtained in the detection process of previous frames. These precomputed features are cached in LRM and builds up a recurrent connection between current frame and previous frames. Unlike traditional memory, please note that these cached features are first enhanced by global information, which means current key frame is able to access more information not only locally, but also globally. The aggregation size is shown in the bottom half of Figure <ref>. By introducing LRM, a great leap towards solving the insufficient problem is achieved while remaining simple and fast. 

Thanks to the enormous aggregation size of MEGA empowered by LRM, we achieve 85.4% mAP on ImageNet VID dataset, which is to-date the best reported result.



§ RELATED WORK


Object Detection from Images. Current leading object detectors are build upon deep Convolutional Neural Networks (CNNs) <cit.> and can be classified into two main familys, namely, anchor-based detectors (, R-CNN <cit.>, Fast(er) R-CNN <cit.>, Cascade R-CNN <cit.>) and anchor-free detectors (, CornerNet <cit.>, ExtremeNet <cit.>). Our method is built upon Faster-RCNN with ResNet-101, which is one of the state-of-the-art object detector.

Object Detection in Videos. Due to the complex manner of video variation, , motion blur, occlusion and out of focus, it is not trivial to generalize the success of image detector into the video domain. The main focus of recent methods <cit.> towards solving video object detection is improving the performance of per-frame detection by exploiting information in the temporal dimension. These methods could be categorized into local aggregation methods and global aggregation methods.



Local aggregation methods <cit.> primarily utilize information in local temporal range to aid the detection of current frame. For example, FGFA <cit.>, MANet <cit.> utilize optical flow predicted by FlowNet <cit.> to propagate features across frames. Methods like STSN <cit.>, STMN <cit.> directly learn to align and aggregate features without optical flow. Besides of these pixel-level aggregation methods, RDN <cit.> based on Relation Network <cit.> directly learns the relation among candidate boxes of different frames in a local range to enhance the box-level features.

Global aggregation methods <cit.> seek to enhance the pixel or box features directly with semantic information. Unlike optical flow or position relationship between boxes that depend on locality in temporal range to some extent, semantic similarity is somewhat independent of temporal distance. However, on the one hand getting rid of locality could enable model utilizing rich information beyond a fixed temporal window, on the other hand the lack of locality information would introduce weakness when localizing. 

Unlike these methods that separately view video object detection globally or locally, MEGA seeks to take full merit of both local and global aggregation to enhance the feature representation. Moreover, enabled by memory, information from longer content could be utilized.

Information Aggregation beyond Local Range. Like our method that tries to aggregate information beyond a small local range, <cit.> share a similar spirit and shows superior results in different areas. <cit.> also tries to aggregate information both globally and locally, however their "global" is just a larger local range while our "global" is the whole video. <cit.> keeps a carefully designed memory to aid video segmentation while ours memory is simpler and already efficient. <cit.> shares a similar relation aggregation module with ours. However how this module is instantiated is not the main focus of our work.









§ METHOD


In this section, we will elaborate how we devise MEGA to enable the whole architecture to fully utilize both global and local information. To be specific, MEGA first aggregates selected global features to local features and afterwards, these global-enhanced local features together with a novel Long Range Memory (LRM) module aggregate longer content of global and local information into key frame for better detection. An overview is depicted in Figure <ref>. 





 §.§ Preliminary



The goal of video object detection is to give detection results for every frame of video {I_t}_t=1^T. Suppose the current frame to be detected is I_k and B_t={b_t^i} denotes the candidate boxes generated by RPN at each frame I_t. All candidate boxes in adjacent frames {I_t}_t=k-τ^k+τ are grouped together to form the local pool, that is, 𝐋={B_t}_t=k-τ^k+τ. For global features, we randomly shuffle the ordered index sequence {1,…,T} to obtain a shuffled index sequence S, then we sequentially select T_g frames and group all boxes in them to form the global pool. It could be denoted as 𝐆={B_S_i}_i=k^k+T_g-1. At last, a novel long range memory module 𝐌 stores intermediate features produced in the detection process of previous frame is introduced to allow the key frame
to exploit cached information, leading to an ability of modeling longer-term global and local dependency. Our ultimate goal is to give classification and regression results to all candidate boxes B_k in the key frame with the help of 𝐋,𝐆 and 𝐌.

Furthermore, we represent each box b^i with its semantic feature f_i and localization feature g_i. g_i represents both the spatial information (, height, width, center location) and the temporal information (, the frame number).

Relation Module. The operator that we choose to mine the relationship between boxes is the relation module introduced in <cit.> which is inspired by the multi-head attention <cit.>. Given a set of boxes B={b_i}, object relation module is devised to enhance
each box b_i by computing M relation features as a weighted sum of semantic features from other boxes, where M denotes the number of heads. Technically, the m-th relation features of b_i is computed as

    f_R^m,*(b_i, B)=∑_jω_i j^m,*·(W_V^m· f_j),   m=1, ⋯, M,

where W_V^m is a linear transformation matrix. The relation weight ω_i j^m, * indicates the impact between b_i and b_j measured with semantic features f and probably localization features g. Here * ∈{L, N} denotes whether the localization features g is incorporated in ω, where L means incorporated and N means not. As localization features between two distant boxes in the temporal dimension are redundant and may harm the overall performance, we design the location-free version urging the relation module to focus only on semantic features. Notice that unlike <cit.>, temporal information is also contained in our localization features to distinguish the impact of boxes from different frames. This temporal information is incorporated in ω in a relative manner as in <cit.>.

Finally, by concatenating all M relation features and its original feature, we obtain the output augmented feature:

    f_r m^*(b_i, B)=f_i+concat[{f_R^m,*(b_i, B)}_m=1^M],

where * ∈{L, N} and the meaning is the same as before.

After the augmented feature is produced, we additionally append a non-linear transformation function h(·) implemented as a fully-connected layer and ReLU after it.

Further, We could extend the relation module to modeling the relationship between two sets of boxes. For convenience, we use notation f_rm^*(B,P) to represent the collection of all augmented proposal features, , {f_r m^*(b_i, P)}, which means all the bounding boxes in 𝐁 are augmented via the features of bounding boxes in 𝐏.



 §.§ Memory Enhanced Global-Local Aggregation
 

Global-Local Aggregation for ineffective problem. First we will elaborate how we design the network by aggregating global and local features together to address the ineffective problem, which means separately considering global or local information. We denote this architecture as base model and it is depicted in Figure <ref>. 

Specifically, the global features from 𝐆 are aggregated into 𝐋 at first. The update function could be formulated as:

    𝐋^g=𝒩_g(𝐋,𝐆),

where 𝒩_g(·) is a function composed of stacked location-free relation modules and 𝐋^g denotes the final global-enhanced version of 𝐋 after this function. As our aim is to fully exploit the potential of global features to enhance local features, we iterate the relation reasoning in a stacked
manner equipped with N_g relation modules to better characterize the relations between 𝐆 and 𝐋. To be specific, the calculation process in the k-th relation module is given as:

    𝐋^g,k=f_rm^N(𝐋^g,k-1, 𝐆),      k=1,...,N_g,

where f_rm^N(·) denotes location-free relation module, defined in Eq (<ref>) and 𝐋^g,0=𝐋 denotes the input of the first relation module. Latter relation module takes the output from previous relation module as input. Finally, the output of the N_g-th relation module is taken as 𝐋^g.

After the global features are aggregated into local features, we then seek to utilize semantic and localization information underlies these local features to further enhance themselves. To achieve this, a stack of N_l location-based relation modules is adopted. Technically, the overall function could be summarized as:

    𝐋^l=𝒩_l(𝐋^g),

where 𝐋^l represents the final enhanced version of local pool. We decompose the whole procedure of 𝒩_l(·) in below. The computation pipeline in the k-th relation module is similar with its counterpart in 𝒩_g(·):

    𝐋^l,k=f_rm^L(𝐋^l,k-1, 𝐋^l,k-1),      k=1,...,N_l,

where f_rm^L(·) denotes location-based relation module and we adopt 𝐋^g, the global-enhanced version of 𝐋, as the input of the first location-based relation module. 𝐋^l,N_l is taken as the output enhanced pool 𝐋^l. After the final update is done, all box features in 𝐋^l that belong to the key frame will be extracted and be propagated through traditional RCNN head to give classification and regression result. These extracted features is denoted as 𝐂.

Long Range Memory for insufficient problem. With base model, a single frame is able to aggregate totally T_g frames of global features and T_l frames of local features as shown in Figure <ref>, which is a big step towards solving the ineffective problem. But the insufficient problem, which stands for the size of frames for key frame to aggregate is too small, is still leaving untouched until now. This problem can be naively solved by increasing T_g and T_l to approach the length of the video if infinite memory and computation is available. However, this is infeasible because the resource is limited in practice.

So how could we solve the insufficient problem while keeping the computation cost affordable? Inspired by the recurrence mechanism introduced in <cit.>, we design a novel module named Long Range Memory (LRM) to fulfill this target. To summarize, LRM empowers base model to capture content from much longer content, both globally and locally, by taking full use of precomputed features. We name this memory enhanced version as MEGA. We refer readers to Figure <ref> for an overview of how MEGA works.

To see the defect of base model, suppose I_k-1 and I_k are two consecutive frames. 
When we shift the detection process from I_k-1 to I_k, we discard all intermediate features produced by the detection on I_k-1. So the detection process of I_k could not take any merit of the detection process of I_k-1, though they are consecutive in the temporal dimension. Every time we move to a new frame, we need to recompute everything from the very start. This motivates us to memorize precomputed features to allow current frame to exploit more information in the history. In practice, besides utilizing information from adjacent frames {I_t}_t=k-τ^k+τ, long range memory 𝐌 of size T_m would additionally provide features of T_m frames of information ahead of adjacent frames, namely {I_t}_t=k-τ-T_m^k-τ-1, to help detecting on I_k.

To be specific, after detection process for I_k-1 is done, Unlike in base model where we discard all features computed in the detection process, at this time intermediate features of I_k-τ-1, the first frame of adjacent frames of I_k-1, would be cached in long range memory 𝐌. That means, as our local aggregation function 𝒩_l(·) defined in Eq (<ref>) is composed of a stack of N_l relation modules and 𝐋^l, i, i ∈{0, N_l} is the features enhanced after the i-th relation module (i=0 denotes the input), we would extract and store all level of features of I_k-τ-1, namely 𝐋_k-τ-1^l, i, i ∈{0, N_l} in 𝐌. Concretely, 𝐌 is of N_l+1 levels where 𝐌^i caches 𝐋_k-τ-1^l, i. Every time the detection process is done for a new frame, features corresponding to the first frame of adjacent frames of this new frame would be added into 𝐌. 



By introducing 𝐌 and reusing these precomputed features cached in it, we could enhance our local aggregation stage to incorporate the information from 𝐌 for detection at I_k and latter frames. The enhanced version of local aggregation could be summarized as

    𝐋^l=𝒩_El(𝐋^g, 𝐌)


Like 𝒩_l(·), 𝒩_El(·) is also build upon a stack of N_l location-based relation module while taking the impact of 𝐌 into consideration:

    𝐋^l,k=f_rm^L(𝐋^l,k-1, [𝐋^l,k-1,𝐌^k-1]), k=1,...,N_l,

where [·,·] indicates the concatenation of two information pools. Compared to the standard update function, the critical difference lies in the formation of the reference pool. Like in base model, after the final update is done, 𝐂 will be extracted and be propagated through traditional RCNN head to give classification and regression result of current key frame. The detailed inference process of MEGA is given in Algorithm <ref>. 



To what extent does LRM address the ineffective and insufficient approximation problem? By appending long range memory 𝐌 of size T_m, a direct increase of T_m in feature number is obvious. But the increase of sight is far beyond this number. Please note that our enhanced local stage is of N_l stacks, thanks to the recurrent connection introduced by 𝐌, the key frame could gather information from additional T_m frames every time we iterate the relation reasoning just as shown in Figure <ref>. The last and most importantly, as the cached features of every frame are first enhanced by different set of global features, long range memory does not only increase the aggregation size locally, but also globally. To summarize, model with N_l-level enhanced local aggregation stage could gather information from totally N_l × T_m + T_l local reference frames and N_l × T_m + T_g global reference frames, where T_l, T_g, T_m denotes the size of local pool, global pool and memory, respectively. It is a great leap towards T_l and T_g in the base model. With this hugely enlarged aggregation size, we argue the ineffective and insufficient problem is better addressed by our model while not increasing the running time too much. This is further justified by the superior experimental results shown in Table <ref> in Section <ref>. 





§ EXPERIMENTS
 



 §.§ Dataset and Evaluation Setup


We evaluate our proposed methods on ImageNet VID dataset <cit.>. The ImageNet VID dataset is a large scale benchmark for video object detection task consisting of 3,862 videos in the training set and 555 videos in the validation set. 30 object categories are contained in this dataset. Following protocols widely adopted in <cit.>, we evaluate our method on the validation set and use mean average precision (mAP) as the evaluation metric.



 §.§ Network Architecture


Feature Extractor. We mainly use ResNet-101<cit.> and ResNeXt-101 <cit.> as the feature extractor. As a common practice in <cit.>, we enlarge the resolution of feature maps by modifying the stride of the first convolution block in last stage of convolution, namely conv5, from 2 to 1. To retain the receptive field size, the dilation of these convolutional layers is set as 2.

Detection Network. We use Faster R-CNN <cit.> as our detection module. The RPN head is added on the top of conv4 stage. In RPN, the anchors are of 3 aspect ratios {1:2, 1:1, 2:1} and 4 scales {64^2, 128^2, 256^2, 512^2}, resulting in 12 anchors for each spatial location. During training and inference, N=300 candidate boxes are generated for each frame at an NMS threshold of 0.7 IoU. After boxes are generated, we apply RoI-Align <cit.> and a 1024-D fully-connected layer after conv5 stage to extract RoI feature for each box.

MEGA. At training and inference stages, the local temporal window size is set to T_l=25 (τ=12). Note that the temporal span could be different on two sides of key frame in practice. For the sake of efficiency, we do not keep all candidate boxes produced by RPN for each local reference frame, instead 80 candidate boxes with highest objectness scores are selected. And in the latter stack of local aggregation stage, the number of boxes are further reduced to 20. As for the global reference frame, we totally select T_g=10 frames and 80 proposals with highest objectness scores each frame. The size T_m of Long Range Memory is set to 25.

As for the global and local aggregation stage, the number of relation modules is set to N_g=1 and N_l=3, respectively. For each relation module, the hyperparameter setting is the same as <cit.>.



 §.§ Implementation Details

Following common protocols in <cit.>, we train our model on a combination of ImageNet VID and DET datasets. For the DET dataset, we select images that are of the same 30 classes as in the VID dataset. We implement MEGA mainly on maskrcnn-benchmark <cit.>. The iuput images are resized to have their shorter side to be 600 pixels. The whole architecture is trained on 4 RTX 2080ti GPUs with SGD. Each GPU holds one mini-batch and each mini-batch contains one set of images or frames. We train our network for totally 120K iteration, with learning rate 10^-3 and 10^-4 in the first 80K and in the last 40K iterations, respectively. At inference, an NMS of 0.5 IoU threshold is adopted to suppress reduplicate detection boxes.

Training. As the number of boxes prohibits the training process to be the same as the inference process, we adopt the strategy of temporal dropout<cit.> to train our model. Given key frame I_k, we randomly sample two frames from {I_t}_t=k-τ^k+τ and two frames from the whole video to approximately form 𝐋,𝐆. Additional two frames from {I_t}_t=k-τ-T_m^k-τ-1 are selected to construct 𝐌. For convenience, we name them as 𝐋̂,𝐆̂ and 𝐋̂_M. We first apply base model on 𝐆̂ and 𝐋̂_M and store all intermediate features produced as 𝐌̂. After that, 𝐋̂,𝐆̂,𝐌̂ are propagated through full MEGA to generate 𝐂. Finally, the whole model are trained with classification and regression losses
over 𝐂. One thing needed to be pointed out is that we stop the gradient flow in the construction of 𝐌̂. This behavior is similar to <cit.> but with different motivation: we would like the model to pay more attention on most adjacent frames.



 §.§ Main Results






Table <ref> shows the result comparison between state-of-the-art end-to-end models without any post-processing. Among all methods, MEGA achieves the best performance and is the only method that takes full use of both global and local information. With ResNet-101 backbone, MEGA can achieve 82.9% mAP, 1.1% absolute improvement over the strongest competitor RDN. By replacing the backbone feature extractor from ResNet-101 to a stronger backbone ResNeXt-101, our method achieves better performance of 84.1% mAP just as expected. Among all competitors, RDN is the most representative method in local aggregation scheme while SELSA is the one in global aggregation scheme. RDN only models the relationship within a short local temporal range, while SELSA only builds sparse global connection. As discussed in Section <ref>, these methods may suffer from the ineffective and insufficient approximation problems which lead to inferior results than ours.

Like many previous methods that could get further improvement by post processing, it could also benefit ours. Here the post processing technique we adopt is BLR <cit.>, which is done by finding optimal paths in the whole video and then re-score detection boxes in each path. Table <ref> summarizes the results of state-of-the-art methods with different post-processing techniques. With no doubt, our method still performs the best, obtaining 84.5% and 85.4% mAP with backbone ResNet-101 and ResNeXt-101, respectively.



 §.§ Ablation Study


To examine the impact of the key components in our MEGA, we conduct extensive experiments to study how they contribute to our final performance.

Long Range Memory. We would first explore the effect of LRM as it plays the key role in our full model. We show the performance comparison results between base model and MEGA in Table <ref>. As shown in the table, a gap of 1.5% mAP exists between these two models, which is a significant improvement. Gap of 1% mAP still exists after increasing base model's local span τ to τ+T_m/2, while running at a much slower speed. We argue the superior performance of MEGA is brought by the novel LRM module which enables one frame could gather information efficiently from much longer content, both globally and locally.





Table <ref> shows the results of removing global or local information from MEGA. In the default setting, the number of relation modules in the global aggregation stage and local aggregation stage is N_g=1 and N_l=3, respectively. To study the effect of global features, We simply set N_g to 0 to remove the influence from it. As shown in the table, MEGA experiences a sheer performance loss of 1.6% mAP by removing the global features. We also conduct an experiment by setting N_l to 4 which means to keep the number of parameters as the same as MEGA though it is not fully comparable (the local range is larger). And this experiment gives 81.6% mAP which is still lower. The above results show the importance of global feature aggregation. 

To see the importance of local information, we conduct an experiment by setting N_g to 4 and N_l to 0, also the local temporal window size T_l and the number of global reference frames T_g is changed to 1 and 25, respectively. Under this setting, one frame could only enhanced by global information while keeping the number of parameters the same as MEGA. The result is given in the last row of Table <ref>. As can be seen, this result is 1.1% lower than our full model, indicating the necessity of local features.







From Table <ref> and <ref>, an interesting conclusion could be drawn. It is the combination of local features, global features and memory that empowers MEGA to obtain superior result, and three components alone is not sufficient. With memory, one frame could get access to more global and local features and in turn, this enhanced frame could offer us a more compact memory. This manner justifies our intuition that a better approximation of Figure <ref> is a promising direction to boost the performance on video object detection.



Figure <ref> showcases one hard example in video object detection. As the lizard is presented in a rare pose for quite a while, only exploiting local information is not able to tell us what it is. The top two rows shows the result of single frame and local aggregation. They are not able to recognize the object with just local information. By taking global features into consideration, the model overcomes this difficult case by aggregating features from distinct object from global frames. The last two rows show the result of base model and MEGA. The result gets better when more information is incorporated just as expected.

Aggregation Scale. The aggregation scale here means the content one frame could gather from, globally or locally. Totally four hyperparameters, global reference frame number T_g, local reference frame number T_l, number of relation modules in local aggregation stage N_l and memory size T_m would influence the aggregation scale. The result is given in Table <ref>. (a) The result of different number of T_g. As can be seen, there is only minor difference between them. As discussed in Section <ref>, the number of global reference frames one frame could see is N_l × T_m + T_g, which indicates the influence from T_g is minor compared to N_l and T_m. (b) As for T_l, the result is similar with T_g. (c)(d) The result for N_l and T_m. These two parameters jointly influence the global and local range of MEGA. The performance of MEGA gets worse when N_l or T_m is small, which implies larger aggregation scale matters. The improvement saturates when N_l or T_m gets bigger, which may suggest that a proper approximation is sufficient enough.



Types of Relation Modules. As we has discussed earlier, we differentiate two types of relation modules by whether incorporating the location information into the relation weights. By incorporating location information in global aggregation stage, we obtain inferior result of 82.5% mAP, which verifies that incorporating the location information to the global aggregation stage would harm the overall performance.

Online Setting. How would our model behave if current frame could only access information from previous frames? We refer this setting as online setting, which is an often confronted situation in practice. In experiment, we still keep the number of local reference frames T_l to be 25 while modifying the adjacent frames of key frame I_k to be {I_t}_t=k-T_l+1^k. As for global information, the sampling range is limited to all previous frames of current frame while keeping global sampling number T_g the same as before. With this limited sight, the modified MEGA could also obtain 81.9% mAP. To the best of our knowledge, the previous state-of-the-art performance under the same setting is OGEMN's 80.0% mAP, which uses a variant of memory network to boost detection. As memory network could be treated as a way to build sparse connection among different frames, it could be classified into global aggregation scheme. This method also suffers from the ineffective and insufficient problem, which results in inferior performance than ours.

Running Time. The running time of our model is given in Table <ref>. Compared to the base model, MEGA boosts the overall performance by a large margin while only introducing little computation overhead. This result suggests the efficiency of our method.


















§ CONCLUSION


In this work, we present Memory Enhance Global-Local Aggregation Network (MEGA), which takes a joint view of both global and local information aggregation to solve video object detection. Particularly, we first thoroughly analyze the ineffecitve and insufficient problems existed in recent methods. Then we propose to solve these problems in a two-stage manner, in the first stage we aggregate global features into local features towards solving the ineffective problem. Afterwards, a novel Long Range Memory is introduced to solve the insufficient problem. Experiments conducted on ImageNet VID dataset validate the effectiveness of our method. 



§ ACKNOWLEDGEMENT


This work is supported by National Key R&D Program of China (2018YFB1402600), BJNSF (L172037), Beijing Acedemy of Artificial Intelligence and Zhejiang Lab.

ieee_fullname



