ReMotENet: Efficient Relevant Motion Event Detection
for Large-scale Home Surveillance Videos
Ruichi Yu1,2

arXiv:1801.02031v1 [cs.CV] 6 Jan 2018

1

Hongcheng Wang2

University of Maryland, College Park
{richyu, lsd}@umiacs.umd.edu

2

Larry S. Davis1

Comcast Applied AI Research, D.C.

Hongcheng_Wang@comcast.com

Abstract
This paper addresses the problem of detecting relevant
motion caused by objects of interest (e.g., person and vehicles) in large scale home surveillance videos. The traditional method usually consists of two separate steps, i.e.,
detecting moving objects with background subtraction running on the camera, and filtering out nuisance motion
events with deep learning based object detection and tracking running on cloud. The method is extremely slow, and
does not fully leverage the spatial-temporal redundancies
with a pre-trained off-the-shelf object detector. To dramatically speedup relevant motion event detection and improve its performance, we propose a novel network for relevant motion event detection, ReMotENet, which is a unified, end-to-end data-driven method using spatial-temporal
attention-based 3D ConvNets to jointly model the appearance and motion of objects-of-interest in a video. ReMotENet parses an entire video clip in one forward pass
of a neural network to achieve significant speedup, which
exploits the properties of home surveillance videos, and
enhances 3D ConvNets with a spatial-temporal attention
model and frame differencing to encourage the network to
focus on the relevant moving objects. Experiments demonstrate that our method can achieve comparable or event better performance than the object detection based method but
with three to four orders of magnitude speedup (up to 20k×)
on GPU devices. Our network is efficient, compact and
light-weight. It can detect relevant motion on a 15s surveillance video clip within 4-8 milliseconds on a GPU and a
fraction of second (0.17-0.39s) on a CPU with a model size
of less than 1MB.

1. Introduction
With the development of home security and surveillance
system, more and more home surveillance cameras have

Figure 1. Comparison between the traditional object detection
based method and ReMotENet: Traditional method conducts
background subtraction, object detection and tracking on each
frame; ReMotENet parses a 4D representation of the entire video
clip in one forward pass of 3D ConvNets to efficiently detect relevant motion involving objects-of-interest.

been installed to monitor customers’ home 24/7 for security
and safety purpose. Most existing commercial solutions run
motion detection on the edge (camera), and show the detected motion events (usually in short clips of, e.g., 15s) for
end users’ review on the web or mobile.
Motion detection has been a challenging problem in
spite of many years’ development in academia and industry [36, 30]. Nuisance alarm sources, such as tree motion,
shadows, reflections, rain/snow, flags, result in many irrelevant motion events for customers.
Relevant motion detection is responsive to customers’
needs. It involves pre-specified relevant objects, e.g., people, vehicles and pets, and these objects should have human
recognizable location changes in the video. It not only helps
to remove nuisance events, but also supports applications
such as semantic video search and video summarization.
As shown in Figure 1, one natural method is to apply state-of-the-art object detectors based on deep convolutional neural networks (CNNs) [8, 7, 27, 19, 25, 26] to
identify objects of interest. Given a video clip, background
subtraction is applied to each frame to filter out stationary

frames. Object detection is then applied to frames that have
motion to identify the moving objects. Finally, the system
generates track from the detection results to filter out temporally inconsistent false detections or stationary ones.
There are at least two problems with this object detection based method. First, it is computationally expensive,
especially the object detection module. The state-of-theart object detectors [8, 7, 27, 19, 25, 26] need to be run
on expensive GPUs devices and achieve at most 40-60 FPS
[14]. Scaling to tens of thousands of motion events coming
from millions of cameras becomes cost ineffective. Second,
the method usually consists of several separate pre-trained
methods or hand-crafted rules, and does not fully utilize the
spatial-temporal information of an entire video clip. For example, moving object categories are detected mainly by object detection, which ignores motion patterns that can also
be utilized to classify the categories of moving objects.
To address these problems, we propose a network for relevant motion event detection, ReMotENet, which is a unified, end-to-end data-driven method using Spatial-temporal
Attention-based 3D ConvNets to jointly model the appearance and motion of objects-of-interest in a video event. As
shown in Figure 1, ReMotENet parses an entire video clip
in one forward pass of a neural network to achieve significant speedup (up to 20k×) on a single GPU. This makes the
system easily scalable to millions of motion events and reduces latency. Meanwhile, it exploits the properties of home
surveillance videos, e.g., relevant motion is sparse both spatially and temporally, and enhances 3D ConvNets with a
spatial-temporal attention model and frame differencing to
encourage the network to focus on relevant moving objects.
To train and evaluate our model, we collected a large
dataset of 38,360 real home surveillance video clips of 15s
from 78 cameras covering various scenes, including indoor/outdoor, day/night, different lighting conditions and
weather. To avoid the cost of training annotations, our
method is weakly supervised by the results of the object
detection based method. For evaluation, we manually annotated 9,628 video clips with binary labels of relevant motion caused by different objects. Experiments demonstrate
that ReMotENet achieves comparable or even better performance, but is three to four orders of magnitude faster than
the object detection based method. Our network is efficient,
compact and light-weight. It can precisely detect relevant
motion in a 15s video in 4-8 milliseconds on a GPU and a
fraction of second on a CPU with model size of less than
1MB.

2. Related Work
Our application is to detect relevant motion events in
surveillance videos. As far as we know, there is no existing
work addressing the relevant motion detection application
by unifying motion detection and objects of interest detec-

tion. Most relevant work usually addresses this problem
with a two-step detection based approach consisting of detecting moving objects with background subtraction and filtering out nuisance motion events with object detection and
tracking.
Background subtraction has been widely used and studied [36, 30, 37, 31, 43] to detect moving objects from
videos. It extracts motion blobs but does not recognize the
semantic categories of the moving objects, and many nuisance alarm sources result in irrelevant motion events.
Semantic object detection is then applied to further filter
out these nuisance alarms. One of the state-of-the-art object
detection frameworks is R-CNN and its variants [8, 7, 27].
Recently, more efficient detectors, e.g., YOLO [25, 26]
and SSD [19] have been proposed to speedup the detection pipeline with some performance degradation compared
to R-CNN. To identify moving objects in a video, tracking
techniques (traditional [2, 3, 39] and deep network based
[12, 1, 21]) are usually used to further reduce false detections. The detection based approach is computationally expensive. Significant GPU resources are needed when
we consider millions of surveillance cameras. Meanwhile,
some hand-crafted and ad-hoc hyper-parameters or thresholds (e.g., minimum motion object size, the detection confidence threshold and length of valid tracker threshold) are
needed to model relevant motion. In addition, one cannot
fully leverage the spatial-temporal redundancies with just
a pre-trained off-the-shelf object detector and a frame-byframe detection.
Our work is also related to video activity recognition,
which is to detect and categorize activities (usually human
activities) in videos. To model motion and temporal information in a video, two-stream networks [29], long-term recurrent neural networks [4] and 3D convolution networks
(3D ConvNets) based methods [34] have been proposed.
Our task is different from the video activity recognition task
in two aspects: first, we only care about the categories of
moving objects, rather than fine-grained categories of the
activities of one specific object class; second, due to the
large volume of videos, small computational cost has higher
priority in our application.
Recently, Kang et al. [15] proposed NoScope, a preprocessing method to reduce the number of frames needed
to be parsed in an object-detection based video query system. They utilize frame difference and specialized networks to filter out frames without moving relevant objects.
There are several major difference between NoScope and
ReMotENet: First, ReMotENet is a unified end-to-end solution without explicit object detection, but NoScope is a preprocessing step for object detection which combines several
pre-trained models. Although our method is weakly supervised by object detection, it learns the general motion and
appearance patterns of different objects from the noisy la-

Conv5&

Conv1&

Conv6&
Conv9&

…&
Frame&

3D&ConvNets&
Temporal2wise&&
Max&Pooling&

Spa0al2Temporal&
A6en0on&Module&

Relevant&Mo0on&2&&

…&

…&

Diﬀerencing&

Relevant&Mo0on&1&

Pooling&
nvs&
1&Co
1x1x

3D&ConvNets&
Spa0al2wise&&
Max&Pooling&

Global2Avg&

Relevant&Mo0on&k&

Figure 2. ReMotENet for Relevant Motion Event Detection. The
input is a 4D representation of an entire video clip and the outputs are several binary predictions of relevant motion involving
different objects-of-interest. The low-level 3D ConvNets in blue
only abstract spatial features with spatial-wise max pooling. The
high-level 3D ConvNets in orange abstract temporal features using temporal-wise max pooling. We predict a spatial-temporal
mask and multiply it with the extracted features from Conv5 (with
Pool5) before it is fed as the input to Conv6.

bels and recovers from some mistakes made by the detection pipeline. However, as a pre-processing step, NoScope
highly relies on the performance of pre-trained object detectors, which can be unreliable, especially on home surveillance videos with low video quality, lighting changes and
various weather conditions. Second, ReMotENet jointly
models frames in a video clip, but NoScope conducts detection independently in a frame-by-frame fashion. Since
NoScope relies on object detectors, the run-time speedup
is insignificant. From the results shown in [15], when the
run-time speedup is > 100×, the performance of NoScope
drops quickly. However, our method can achieve more than
19k× speedup while achieving comparable or better performance than the object detection baseline. We do not quantitatively compare to NoScope mainly because the two methods are designed for different tasks: NoScope focuses on
frame-level object detection while ReMotENet is designed
for video-clip-level relevant motion detection.

3. Our Approach
To dramatically speedup relevant motion event detection
and improve its performance, instead of directly compressing deep network [41, 18], we propose a novel network
for relevant motion event detection, ReMotENet, which
is a unified, end-to-end data-driven method using spatialtemporal attention-based 3D ConvNets to jointly model the
appearance and motion of objects-of-interest in a video.

3.1. Analyzing an Entire Video Clip at Once
In contrast to an object detection based method based on
frame-by-frame processing, we propose a unified, end-toend data-driven framework that takes an entire video clip as
input to detect relevant motion using 3D ConvNets [34].
One crucial advantage of using 3D ConvNets is that they

can parse an entire video clip in one forward pass of a
deep network, which is extremely efficient. Meanwhile,
unlike the traditional pipeline that conducts object detection and tracking independently, a 3D ConvNet is an end-toend model that jointly models the appearance of objects and
their motion patterns. To fit an entire video in memory, we
down-sample the video frames spatially and temporally. We
sample one FPS to uniformly sample 15 frames from a 15s
video clip, and reduce the original resolution (1280×720)
by a factor of 8. The input tensor of our 3D ConvNets is
15×90×160×3.

3.2. ReMotENet using Spatial-temporal Attentionbased C3D
3.2.1

Frame Differencing

Global or local context of both background or foreground
objects has proven to be useful for activity recognition[10,
35] (e.g., some sports only happen on playgrounds; some
collective activities have certain spatial arrangements of the
objects that participate). However, since home surveillance
cameras capture different scenes at different times with various weather and lighting conditions, the same relevant motion could involve different background and foreground arrangements.
So, we conduct a simple and efficient frame differencing on the 4D input tensor to suppress the influence of the
background and foreground variance: for each frame subsampled from a video clip, we select its previous frame as
a “reference-frame" and difference them. The image quality of surveillance cameras is usually low, and the moving
objects are relatively small due to large field of view. As a
result, it requires high resolution videos for object detector
to capture fine-grained features, such as texture, to detect
relevant objects. However, for ReMotENet with frame differencing, since we skip explicit object detection, it is sufficient for the network to learn coarse appearance features of
objects, e.g., shape, contour and aspect-ratio with frames of
low resolution, which leads to significant speedup.
3.2.2

Spatial-temporal Attention-based 3D ConvNets

Most video clips captured by a home surveillance camera
may only contain the stationary scene with irrelevant motion such as shadow, rain and parked vehicles. Meanwhile,
our network should be capable of differentiating appearance
and motion pattern of different objects-of-interest.
Besides using frame differencing to suppress the background, we propose a Spatial-Temporal Attention-based
(STA) model as shown in Figure 2 to guide the network to
only pay attention to the moving objects of interest and filter
out foreground motion caused by irrelevant objects. Unlike
most of the attention models that are trained end-to-end with
the groundtruth of final predictions, we use the detection

confidence scores and bounding boxes of the moving relevant objects obtained from a general detector (e.g., Faster
R-CNN) as pseudo-groundtruth to train the STA model.
The supervision from the detection results suppresses the
influence of features from irrelevant regions of each frame,
and the binary labels of motion adjusts the STA layer to
pay attention to the spatial-temporal locations that are most
informative for predicting the relevant motion. The spatialtemporal attention model can be viewed as a combination of
multi-task learning and a traditional attention model. In the
weakly supervised framework, it helps our network to learn
from noisy labels and recover from some of the mistakes
caused by the object detection based method. For instance,
some relevant motion is missed because of bad thresholds
of the tracking module, but the object detector has correctly
detected the relevant objects. The STA model trained with
those detection results allows our network to recover from
the binary prediction mistake.
In contrast to the original C3D model that conducts max
pooling both spatially and temporally [34], we separate
the spatial and temporal max pooling as shown in Figure
2. This allows us to generate an attention mask on each
input frame to capture fine-grained temporal information,
and makes the network deeper to learn better representations. We first apply five layers of 3D convolutions (Conv1Conv5) with spatial max pooling on the 4D input tensor after frame differencing to extract the appearance based features. Then, we apply another 3D convolution layer (STA
layer) on the output of Pool5 to obtain a binary prediction
of whether whether to attend each spatial-temporal location.
We conduct a softmax operation on the binary predictions
to compute a soft probability of attention for each spatialtemporal location. We scale the features from Pool5 by applying an element-wise multiplication between the attention
mask and the extracted features. Another four layers of 3D
ConvNets are used with temporal max pooling to abstract
temporal features.
The network in [34] utilized several fully connected layers after the last convolution layer, which leads to a huge
number of parameters and computations. Inspired by [32],
we apply a spatial global average pooling (GAP) to aggregate spatial features after Pool9 and use several 1×1×1
convolution layers with two filters (denoted as “Binary" layers) to predict the final binary results. The use of GAP and
1×1×1 convolutions significantly reduces the number of
parameters and model size of our method. The final outputs of our network are several binary predictions indicating
whether there is any relevant motion of a certain object or a
group of objects. The detailed network structure is shown in
Table 1. For each Conv layer, we use ReLU as its activation.

Table 1. Network Structure of the ReMotENet using Spatialtemporal Attention-based 3D ConvNets
Layer

Input Size

Kernel Size

Stride

Num of Filters

Conv1
Pool1
Conv2
Pool2
Conv3
Pool3
Conv4
Pool4
Conv5
Pool5

15×90×160×3
15×90×160×3
15×45×80×16
15×45×80×16
15×23×40×16
15×23×40×16
15×12×20×16
15×12×20×16
15×6×10×16
15×6×10×16

3×3×3
1×2×2
3×3×3
1×2×2
3×3×3
1×2×2
3×3×3
1×2×2
3×3×3
1×2×2

1×1×1
1×2×2
1×1×1
1×2×2
1×1×1
1×2×2
1×1×1
1×2×2
1×1×1
1×2×2

16
16
16
16
16
-

STA

15×3×5×16

3×3×3

1×1×1

2

Conv6
Pool6
Conv7
Pool7
Conv8
Pool8
Conv9
Pool9
GAP
Binary

15×3×5×16
15×3×5×16
8×3×5×16
8×3×5×16
4×3×5×16
4×3×5×16
2×3×5×16
2×3×5×16
1×3×5×16
1×1×1×16

3×3×3
2×1×1
3×3×3
2×1×1
3×3×3
2×1×1
3×3×3
2×1×1
1×3×5
1×1×1

1×1×1
2×1×1
1×1×1
2×1×1
1×1×1
2×1×1
1×1×1
2×1×1
1×1×1
1×1×1

16
16
16
16
2

3.3. Network Training
Considering the large volume of home surveillance
videos, it is time-consuming to annotate each training video
with binary labels. So, we adopt a weakly-supervised learning framework that utilizes the pseudo-groundtruth generated from the object detection based method (details are
discussed in section 4.1). Besides binary labels, we also utilize the pseudo-groundtruth of detection confidence scores
and bounding boxes of moving objects-of-interest obtained
from the object detection based method to train the STA
layer. The loss function of the STA layer is:
 XX
1
loss =
C1
wn,i CE(gn,i , yn,i )
(1)
N
n
i

XX
C2
+
CE(stan,w,h,t , Gstan,w,h,t )
W ·H ·T n
w,h,t

The first part is the cross-entropy loss (CE) for each relevant motion category; the second part is the CE loss between the predicted attention of each spatial-temporal location produced by “STA" layer and the pseudo-groundtruth
obtained from the object detector. W, H, T are spatial and
temporal sizes of the responses of layer “STA"; yn,i and gn,i
are the predicted and groundtruth motion labels of the nth
sample; stan,w,h,t and Gstan,w,h,t are the predicted and
groundtruth attention probabilities; wn,i is the loss weight
of the nth sample, which is used to balance the biased number of positive and negative training samples for the ith motion category; N is the batch size; C1 and C2 are used to balance binary and STA loss. We choose C1 = 1 and C2 = 0.5
in this paper and train our network using Adam optimizer
[16] with 0.001 initial learning rate. The training process
converges fast with batch size 40 (<5,000 iterations).

4. Experiments
4.1. Dataset
We collect 38,360 video clips from 78 home surveillance
cameras. Each video is 15s and captured with FPS 10 and
1280×720 resolutions. The videos cover different times
and various scenes, e.g., front door, backyard, street and indoor living room. The longest period a camera recorded is
around 3 days. Those videos mostly capture only stationary
background or irrelevant motion caused by shadows, lighting changes or snow/rain. Some of the videos contain relevant motion caused by people and vehicles (car, bus and
truck).
The “relevant motion" in our system is defined with a
list of relevant objects. We consider three kinds of relevant motion: “People motion", caused by object “people";
“Vehicle motion", caused by at least one object from {car,
bus, truck}; “P+V Motion" (P+V), caused by at least one
object from {people, car, bus, truck}. The detection performance of “P+V Motion" evaluates the ability of our method
to detect general motion, and the detection performance of
“People/Vehicle motion" evaluates the ability of differentiating motion caused by different kinds of objects.
Our network is trained using weak supervisions obtained
from the object detection based method. We run Faster RCNN based object detection with FPS 10 on the original
1280×720 video frames, and apply a state-of-the-art realtime online tracker from [2] to capture temporal consistency to obtain labels of relevant motion involving different
objects-of-interest. We first run the above method on the
entire dataset to detect videos with relevant motion of P+V.
Then, we randomly split the dataset into training and testing
set with a 3:1 ratio. This leads to a training set with 28,732
video clips and a testing set with 9,628 video clips.
We have three annotators watching the testing videos
and annotating binary labels for P+V, people and vehicle
motion. If there is any human recognizable motion of the
relevant objects in a video, we annotate “1" for the corresponding category. All three annotators must agree with
each other, otherwise we mark the video as ambiguous and
remove it from the testing set. We have 9,606 testing videos
left after several rounds of annotation and cross-checking.
Among the testing set, there are 973 videos with relevant
motion events contain either people or vehicles (P+V), 429
videos have people motion only, 594 videos have vehicle
motion only and 50 videos have both motion.

4.2. Baseline: Object Detection based Method
For the object detection based method, if one tracklet has
at least two frames overlapped with the detected bounding
boxes of the relevant objects with Intersection over Union
(IOU) > 0.9, the average detection confidence score > 0.8,
and the maximum relative location change ratio over the

Table 2. F-score of relevant motion detection using different object detectors with different FPS and resolution settings. We conclude that Faster R-CNN based method significantly outperforms
the SSD based ones. Meanwhile, high resolution and large FPS
are needed for the object detection based method.
Detector
P+V

People

Vehicle

F R-CNN
SSD-Incep
SSD-Mobile
F R-CNN
SSD-Incep
SSD-Mobile
F R-CNN
SSD-Incep
SSD-Mobile
Detector

P+V
People
Vehicle

F R-CNN
F R-CNN
F R-CNN

FPS 1
1280×720
0.075
0.103
0.048
0.116
0.156
0.054
0.020
0.062
0.029
FPS 10
320×180
0.667
0.670
0.610

FPS 2
1280×720
0.405
0.191
0105
0.519
0.276
0.139
0.244
0.103
0.062
FPS 10
160×90
0.255
0.251
0.226

FPS 5
1280×720
0.745
0.403
0.277
0.766
0.467
0.252
0.627
0.267
0.197
FPS 5
320×180
0.565
0.574
0.517

FPS 10
1280×720
0.785
0.258
0.258
0.795
0.549
0.258
0.665
0.499
0.252
FPS 5
160×90
0.255
0.256
0.122

entire tracklet along width or height of the frame is large
enough (> 0.2), we consider it as a valid tracklet. If there
is at least one valid tracklet of an object in a video clip, we
consider that video has valid motion of the specific object.
For background subtraction, we utilize the method proposed in [43]. We employ the start-of-the-art real-time online tracking method from [2]. The above two methods can
be efficiently run on CPU. For object detection, we utilized
the Tensorflow Object Detection API [14]. Following the
discussions in [14], we choose three state-of-the-art detectors: Faster R-CNN [27] with ResNet 101 [11], SSD [19]
with inception V2 [33] and SSD with MobileNet [13]. According to [14], Faster R-CNN is the best detector considering detection accuracy and robustness, and is widely used in
many applications [40, 17, 42, 20, 5, 6]. However, the SSD
based detector is much faster while suffering some accuracy
degradation. To further reduce the computational cost of
detection, MobileNet [13] is utilized as the base network in
SSD framework. SSD-MobileNet is one of the most compact and fastest detectors. We do not compare to YOLO
v2 due to its similar performance with SSD [26]. We do
not compare to other detectors such as tinyYOLO [25] and
SqueezeDet [38] due to their significant degradation of performance (e.g., >20% reduction in mAP from YOLO v2 to
Tiny YOLO). We use F-score to jointly evaluate the detection precision and recall. The results are shown in Table
2.
From the tables we conclude that Faster R-CNN based
detector achieves much better performance than the other
two efficient detection methods. Meanwhile, image resolution and frame sample rate (FPS) have significant influence.
If resolution or FPS is small, the performance of the detector and tracker drops significantly. So, to achieve reasonable detection results, we need to employ a robust object
detection framework (Faster R-CNN) with large FPS and
resolutions, which is inefficient and heavy (with model size

Table 3. The path from traditional 3D ConvNets to ReMotENet using Spatial-temporal Attention Model. “FD" denotes frame
differencing; “D" denotes separating spatial and temporal max pooling and making the network deeper; “MT" denotes we use the detection
pseudo-groundtruth to train the STA layer to localize moving objects using multi-task learning [9, 28], but do not use the attention mask
to scale the features extracted by Pool5; “T" and “NT" denote using the attention mask predicted by STA layer to scale the features,
with/without using the detection pseudo-groundtruth to train the STA layer; “H" means using higher input resolution; “32" denotes using
more filters per layer. There are two significant performance improvements along the path. The first is from C3D to FD-C3D by using
frame differencing; The second is from FD-D to FD-D-STA-T with the spatial-temporal attention. Other design choices, e.g., larger input
resolution (FD-D-STA-T-H: from 160×90 to 320×180) and more filters per layer (FD-D-STA-T-32: from 16 to 32) lead to comparable
performance. train STA layer with detection pseudo-groundtruth
Network structures
of ReMotENet

C3D

FD-C3D

FD-D

FD-D-MT

3D ConvNets?
frame differencing?
deeper network?
train STA layer with detection pseudo-GT?
ST Attention?
high resolution?
more filters?

X

X
X

X
X
X

X
X
X
X

AP: P+V
AP: People
AP: Vehicle

77.79
62.25
66.13

82.29
72.21
73.03

83.98
73.69
73.71

> 400MB).
We also conduct experiments without tracking. In general, tracking improves precision but leads to lower recall.
Taking the detection of P+V motion using Faster R-CNN
with FPS 10 and 1280×720 resolution as an example, without tracking, the method will have high recall (e.g., 0.8536)
but very low precision (e.g., 0.2631); with tracking, it has
lower recall (e.g., 0.7321) but much higher precision (e.g.,
0.8467), and achieves much higher F-score. The detailed
results of different settings of the object detection based
method are included in the supplementary materials.

4.3. ReMotENet Performance
The outputs of ReMotENet are three binary predictions.
After applying softmax on each binary prediction, we obtain probabilities of having P+V motion, people motion and
vehicle motion in a video clip. We adopt Average Precision, which is a widely used evaluation metric for object
detection and other detection tasks from [14] to evaluate
our method. We evaluate different architectures and design choices of our methods, and report the average precision of detecting P+V motion, people motion and vehicle motion in Table 3. To show the improvement of our
proposed method, we build a basic 3D ConvNets following
[34]. We design a 3D ConvNets with 5 Conv layers followed by spatial-temporal max pooling. Similar to the C3D
in [34], we conduct 3×3×3 3D convolution with 1×1×1
stride for Conv1-Conv5, and 2×2×2 spatial-temporal max
pooling with 2×2×2 stride on Pool2-Pool5. For Pool1, we
conduct 1×2×2 spatial max pooling with 1×2×2 stride.
Different from C3D in [34], we only have one layer of convolution in Conv1-Conv5. Meanwhile, instead of several
fully connected layers, we apply a global average pooling
followed by several 1×1×1 convolution layers after Conv5.

84.25
74.41
74.25

FD-D
-STA-NT

FD-D
-STA-T

FD-DSTA-T-H

FD-DSTA-T-32

FD-D-STA
-T-H-32

X
X
X
X

X
X
X
X
X

X
X
X
X
X
X

X
X
X
X
X
X

X
X
X
X
X
X
X

84.91
75.82
75.47

86.71
78.95
77.84

85.67
79.78
76.85

87.07
77.92
76.81

86.09
77.54
76.92

Figure 3. Predicted Attention Mask of “FD-D-STA-NT" Method.
Without pseudo-groundtruth bounding boxes of the semantic moving relevant objects obtained from the object detection based
method, the attention model will focus on some “irrelevant" motion caused by the objects outside the pre-specified relevant object
list, e.g., pets, trees and flags. The overlaid red rectangles are the
predicted motion masks (has probability > 0.9).

The above basic architecture is called “C3D" in Table 3.
4.3.1

Evaluation of ReMotENet

First, we evaluate the effect of frame differencing. Column
2-3 in Table 3 show that by using frame differencing (FD),
our 3D ConvNets achieve much higher average precision
for all three categories of motion, especially on people and
vehicle motion detection tasks.
To evaluate the spatial-temporal attention model, we
modify the basic C3D network architecture to separate
spatial and temporal max pooling as shown in Table 1.
We call the network with nine 3D ConvNets (without the
STA layer) “FD-D". “FD-D-MT" denotes using multi-task
learning: we use the STA layer to predict the ST attention mask, which is trained by pseudo-groundtruth obtained
from the object detection based method, but we do not mul-

(a) PR Curve: P+V

(b) PR Curve: People

(c) PR Curve: Vehicle

(d) Run-time per 15s video on GPU (e) Run-time per 15s video on CPU
(second)
(second)

(f) Model Size (KB)

Figure 4. Comparing with baseline. (a)-(c) show the precision-recall (PR) curves of our methods and the precision-recall of object detection
based methods. For all three motion categories, the PR curve of our “FD-D-STA-T" method is always higher than all the PR points of
object detection based methods, which means by properly setting the probabilistic threshold, our method can achieve comparable or better
detection performance than the baselines. The notation of object detection based methods is constructed as “detector_FPS_resolution
reduction factor_tracking" (d)-(f) show the run-time and model size comparison between our methods and the baselines.

tiply the attention mask with the extracted features after
Pool5. Another model is “FD-D-STA-NT": we multiply
the attention mask with the extracted features after Pool5
layer. However, the STA layer is trained with only binary labels of motion categories but without the detection
pseudo-groundtruth. We observe that incorporating multitask learning or end-to-end attention model only leads to a
small improvement, but once we combine both methods,
our “FD-D-STA-T" model achieves significant improvement. Meanwhile, due to the sparsity of moving relevant
objects in the videos, the number of positive and negative spatial-temporal location from the detection pseudogroundtruth is extremely biased. This leads to overfitting of
the model to predict the attention of all the spatial-temporal
locations as 0. Meanwhile, adding the attention model without multi-task learning also leads to small improvement.
We observe that without the weak supervision of specific
objects and their locations, the attention mask predicted by
“FD-D-STA-NT" may focus on motion caused by irrelevant
objects, such as pets, trees and flags shown in Figure 3.
To encourage our network to pay attention to the relevant
objects (in this paper, people and vehicles), we propose our
full model “FD-D-STA-T", which can be viewed as a combination of multi-task learning and attention model. We use
the detected bounding boxes of relevant moving objects to
train the STA layer, and multiply the predicted attention
mask with the extracted features from Pool5 layer. “FDD-STA-T" achieves much higher average precision than the
previous models in all three motion categories. The results

of the above models are listed in column 5-8 of Table 3.
We also conduct experiments with other network design
choices of ReMotENet. For instance, we add more filters
in each convolution layer, or enlarge the input resolution
from 160×90 to 320×180. As shown in Table 3, those design choices lead to insignificant improvements. The experiments demonstrate that ReMotENet can precisely detect
relevant motion with small input FPS and resolution.
4.3.2

Comparing with the Object Detection based
Method

Although ReMotENet is trained with the pseudogroundtruth obtained from the object detection based
method, it outperforms the baseline in several ways:
Detection Performance. Although the training labels
are noisy, ReMotENet can learn patterns of relevant motion and generalize well. We show the PR curve of our
methods and the performance of the object detection based
method in Figure 4(a), 4(b) and 4(c). From the PR curve
of three models: “C3D", “FD-C3D" and “FD-D-STA-T",
we demonstrate the effectiveness of frame differencing and
the STA model. The PR curves of our full model “FD-DSTA-T" are higher than all PR points of the object detection
based method, which shows that our method can achieve
similar or better performance than the baselines by properly
choosing the detection threshold. Another advantage of our
method is that it is a probabilistic model, and one can tune
the threshold based on the priority of precision or recall.

4.4. Visualization
(a) Failure Cases: Object Detection based method

(b) Spatial-temporal Attention Prediction: ReMotENet (FD-D-STA-T)

(c) Spatial-temporal Attention: Pseudo-groundtruth

(d) Failure Cases: ReMotENet (FD-D-STA-T)

Figure 5. Visualization of results from the detection based method
and ReMotENet. The details about how to generate pseudogroundtruth in (c) can be found in the supplementary materials.

Run-time and Model Size. From Table 2 we observe
that with a fixed detector, the object detection based method
needs large FPS and resolutions to achieve good detection
performance, especially for SSD based detectors. So, we
show the time and model size benchmark results of the baselines with FPS 10 and 1280×720 resolution, which achieves
the best detection performance, in Figure 4(d), 4(e) and 4(f).
For baselines, the run-time consists of the time for background subtraction, object detection and tracking; for ReMotENet, the run-time consists of frame differencing and
the forward pass of our 3D ConvNets. We omit the time for
decoding and sampling frames from the video clips for both
methods. The model size is the size of Tensorflow model
data file. We omit the size of meta and index file. Our
method can achieve a 9,514×-19,515× speedup on GPU
(GeForce GTX 1080) and a 3,062×-7,000× speedup on a
CPU (Intel Xeon E5-2650 @2.00GHz). Meanwhile, since
our 3D ConvNet is fully-covolutional, and very compact
(16 filters per Conv layer), we can achieve a 449×-1,304×
reduction on model size. Our “FD-C3D" model can analyze more than 256 15s video clips per second on a GPU
and around 6 videos on a CPU with a 300KB model; “FDD-STA-T" can parse 125 or 2.5 videos of 15s on GPU or
CPU devices respectively with a model smaller than 1MB.
Our method not only efficiently analyze home surveillance
videos on cloud with GPUs, but can also be potentially run
on the edge.

We visualize the results of the baseline and our method
in Figure 5(a) to 5(d). Figure 5(a) shows miss/false detection of people and vehicle motion using the object detection based method. Due to the low quality of video, shadows, small object size and occlusions, object detector may
fail. However, because our method is a data-driven framework trained specifically on the home surveillance video
with the above properties, and it jointly models both object appearance and motion, we can handle the above cases.
Figure 5(b) and 5(c) show the predicted spatial-temporal attention mask (red rectangles) of our “FD-D-STA-T" model
and the pseudo-groundtruth obtained from the object detection based method. Although our “FD-D-STA-T" model
can only predict a coarse attention mask, it captures the motion pattern of the people in the video. Meanwhile, although
trained by the pseudo-groundtruth, our method can recover
from mistakes of the object detection based method (see the
first figures of 5(b) and 5(c)). Figure 5(d) shows several failure cases of our method. For the miss detection (the second
and forth figures), although our 3D ConvNet predicts there
is no relevant motion, it still correctly predicts the coarse attention mask that covers the car and people. For the falsely
detected vehicle motion (the first figure), we find that due
to reflection of light, the input after frame differencing is
very noisy; for the false people motion detection (the third
figure), our method falsely detects bird motion as people.

5. Conclusion
We propose an end-to-end data-driven framework to detect relevant motion from large-scale home surveillance
videos. Instead of parsing a video in a frame-by-frame
fashion using the object detection based method, we proposed to use 3D ConvNets to parse an entire video clip at
once to dramatically speedup the process. We extended the
3D ConvNets by incorporating a spatial-temporal attention
model to encourage the network to pay more attention to the
moving objects. Evaluations demonstrate that ReMotENet
achieves comparable or better performance than the object
detection based method while achieving three to four orders of magnitude speedup on GPU and CPU devices. ReMotENet is very efficient and compact, and therefore naturally implementable on the edge (camera).

Acknowledgement
The research was done at the Comcast Applied AI Research.
Partial support from the Office of Naval Research under Grant
N000141612713 (Visual Common Sense Reasoning for Multiagent Activity Prediction and Recognition) is acknowledged. The
fruitful discussion with MD Mahmudul Hasan, Upal Mahbub and
Jan Neumann is highly appreciated. Special thanks go to all Comcast TPS volunteers who donated their home videos.

6. Supplementary Materials
We show more evaluation results in this material including performance evaluation of the object detection based method with different settings; different reference-frames for frame differencing;
detection performance v.s. training iterations showing good converging property of training ReMotENet; the benchmark of ReMotENet with the object detection based method (with vs. without tracking); and the details of how we obtained the pseudogroundtruth from the object detection based method.

6.1. Precision and Recall for Object Detection based
Method with Different Settings
In contrast to traditional sequential prediction methods [4, 23,
22, 24], our method takes an entire video clip as input. Figure 6(a)6(c) shows the precision and recall comparison w.r.t. the object detection based method with and without object tracking. We can see
that for the object detection based method, without object tracking, a high recall can be achieved, but the precision is very low
due to a large number of temporally inconsistent false detections;
with object tracking, the method achieved a much higher precision with slightly lower recall; the better overall performance (FScore) was achieved with tracking. Meanwhile, the object detection based method requires robust and powerful but less efficient
detection framework (such as Faster R-CNN), high resolution and
large FPS to achieve good performance. Efficient detection framework (SSD) and compact network (MobileNet) suffer performance
degradation. As shown in Figure 6(a)-6(c), ReMotENet achieves
comparable or better performance when compared to the object
detection based method with or without tracking.

6.2. Experiments: ReMotENet
6.2.1

Visualization of Frame Differencing

Figure 7(a) shows some frames after frame differencing for a video
clip contain relevant motion (person). We observe that the frame
differencing successfully suppresses the background and highlights the moving foreground of person. Figure 7(b) shows the
frames from a video clip without relevant motion. However, due
to shadow and wind, the input is not all-zero, but the frame differencing still suppresses the stationary background to reduce the
learning complexity of our method.

6.2.2

Evaluation of Frame Differencing

First, we evaluate the effect of different frame differencing
schemes in our framework. We tested two choices of reference frame: global reference-frame (RefG), which is the first
sub-sampled frame of a video clip; local reference-frame (RefL),
which is the previous sub-sampled frame of the current frame. We
show examples of frames subtracted from RefG and RefL in Figure 8. Our intuition is that if there are relevant objects in the first
frame, and if we choose it as the global reference-frame, there will
always be holes of those objects in the subsequent frames, which
may be misleading for the network. Our experiments also validated that RefL leads to better performance than RefG, especially
on the people and vehicle motion detection task (as shown in Fig.
4). That is why we used RefL in all the benchmark work in the
paper.

Table 4. Evaluation of Frame Differencing (Average Precision):
C3D denotes using raw frames; RefG-C3D denotes using the first
frame as the reference frame; RefL-C3D denotes using the previous frame as the reference frame. Using frame differencing, especially RefL-C3D leads to significant improvement of the relevant
motion detection when compared to using raw frames as input.
C3D
RefG-C3D
RefL-C3D

6.2.3

P+V

P Motion

V Motion

77.8
81.8
82.3

62.3
70.7
72.2

66.1
69.2
73.0

Testing Performance v.s. Training Iterations

We show the testing average precision v.s. number of training iterations in Figure 9. ReMotENet takes a small number of training iterations to achieve good detection performance. This shows
good converging properties of training ReMotENets.

6.2.4

Pseudo-groundtruth Obtained from the Object
Detection based Method

Besides the final binary labels, we also use the object detection based pseudo-groundtruth to train the STA layer and achieve
significant performance improvement. To obtain the pseudogroundtruth, we first conduct the pipeline consisting of background subtraction, object detection using Faster R-CNN with
1280×720 resolution and 10 FPS and tracking, to obtain valid
tracklets of the objects-of-interest in each video clip (the hyperparameters are the same as those in section 4.2 in the main paper).
Then, for each detected bounding box of a relevant object, we consider it a true positive if the detection confidence score is > 0.8.
For each frame, we construct a binary mask based on all detected
true positive bounding boxes of the relevant objects.
To map the mask with original resolution (1280×720) to the
pseudo-groundtruth of the STA layer, which has a much smaller
spatial size due to spatial-wise max-pooling layers, we divide the
original frame into 5×3 grids (if choosing 160×90 as the input
resolution), which is the spatial size of the output responses from
the STA layer in our network. The size of each grid is 256×240.
For each grid, if more than half of its pixels are labeled as “1" in
the binary mask of the original resolution, we label this grid as
“1". Finally, we obtain a 5×3 binary mask for each frame as the
pseudo-groundtruth to train the STA layer. We show some samples
of the obtained pseudo-groundtruth in Figure 10.

References
[1] L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and
P. H. S. Torr. Fully-convolutional siamese networks for object tracking. CoRR, abs/1606.09549, 2016.
[2] A. Bewley, Z. Ge, L. Ott, F. Ramos, and B. Upcroft. Simple
online and realtime tracking. In 2016 IEEE International
Conference on Image Processing (ICIP), pages 3464–3468,
2016.
[3] D. Comaniciu, V. Ramesh, and P. Meer. Kernel-based object tracking. IEEE Trans. Pattern Anal. Mach. Intell.,
25(5):564–575, May 2003.

Precision & Recall Curve: Motion

Precision & Recall Curve: Person

Precision & Recall Curve: Vehicle

1.0

1.0

0.8

0.8

0.6

0.6

0.4
0.2
0.0
0.0

FD-D-STA-T
FD-C3D
C3D
F-RCNN_10_1_TR
F-RCNN_5_1_TR
F-RCNN_2_1_TR
F-RCNN_1_1_TR
F-RCNN_10_8_TR
F-RCNN_10_4_TR
SSD-Mobile_10_1_TR

0.2

SSD-IncepV2_10_1_TR
F-RCNN_10_1_noTR
F-RCNN_5_1_noTR
F-RCNN_2_1_noTR
F-RCNN_1_1_noTR
F-RCNN_10_8_noTR
F-RCNN_10_4_noTR
SSD-Mobile_10_1_noTR
SSD-IncepV2_10_1_noTR

0.4

Recall

0.6

0.4
0.2

0.8

0.0
0.0

1.0

(a) P+V

FD-D-STA-T
FD-C3D
C3D
F-RCNN_10_1_TR
F-RCNN_5_1_TR
F-RCNN_2_1_TR
F-RCNN_1_1_TR
F-RCNN_10_8_TR
F-RCNN_10_4_TR
SSD-Mobile_10_1_TR

0.2

Precision

0.8
0.6

Precision

Precision

1.0

SSD-IncepV2_10_1_TR
F-RCNN_10_1_noTR
F-RCNN_5_1_noTR
F-RCNN_2_1_noTR
F-RCNN_1_1_noTR
F-RCNN_10_8_noTR
F-RCNN_10_4_noTR
SSD-Mobile_10_1_noTR
SSD-IncepV2_10_1_noTR

0.4

Recall

0.6

(b) Person

0.4
0.2

0.8

1.0

0.0
0.0

FD-D-STA-T
FD-C3D
C3D
F-RCNN_10_1_TR
F-RCNN_5_1_TR
F-RCNN_2_1_TR
F-RCNN_1_1_TR
F-RCNN_10_8_TR
F-RCNN_10_4_TR
SSD-Mobile_10_1_TR

0.2

SSD-IncepV2_10_1_TR
F-RCNN_10_1_noTR
F-RCNN_5_1_noTR
F-RCNN_2_1_noTR
F-RCNN_1_1_noTR
F-RCNN_10_8_noTR
F-RCNN_10_4_noTR
SSD-Mobile_10_1_noTR
SSD-IncepV2_10_1_noTR

0.4

Recall

0.6

0.8

1.0

(c) Vehicle

Figure 6. Precision and Recall for Object Detection based Method with Different Settings. The notation of object detection based methods
is constructed as “detector_FPS_resolution reduction factor_with(out) tracking". The circles denote the object detection based methods
with tracking; the rectangles denote without tracking.

(a) Person Motion

(b) No Motion

Figure 7. Frame Differencing: (a) shows the frames after frame
differencing from a video clip contains person motion; (b) shows
the frames from a video clip without relevant motion. For better
visualization, we conduct dilation on the frames.

Figure 8. Comparison between different reference frames: The
first row shows the raw video frames; the second row shows frames
after subtracting local reference-frame; third row shows frames after subtracting global reference-frame.

Average	
  Precision	
  

0.9	
  
0.85	
  
0.8	
  
0.75	
  

P+V	
  
People	
  

0.7	
  

Vehicle	
  

0.65	
  

50
0
10 	
  
00
15 	
  
00
20 	
  
00
25 	
  
00
30 	
  
00
35 	
  
00
40 	
  
00
45 	
  
00
50 	
  
00
	
  

0.6	
  
Training	
  Itera<ons	
  

Figure 9. Testing Average Precision v.s. Number of training Iterations: RefL-D-STA-T model of ReMotENet.

Figure 10. Pseudo-groundtruth Obtained from the Object Detection based Method.
[4] J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach,
S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. CoRR, abs/1411.4389, 2014.
[5] M. Gao, A. Li, R. Yu, V. I. Morariu, and L. S. Davis. CWSL: count-guided weakly supervised localization. CoRR,
abs/1711.05282, 2017.
[6] M. Gao, R. Yu, A. Li, V. I. Morariu, and L. S. Davis. Dynamic zoom-in network for fast object detection in large images. CoRR, abs/1711.05187, 2017.
[7] R. B. Girshick. Fast R-CNN. CoRR, abs/1504.08083, 2015.
[8] R. B. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich
feature hierarchies for accurate object detection and semantic
segmentation. CoRR, abs/1311.2524, 2013.
[9] X. Han, Z. Wu, Z. Wu, R. Yu, and L. S. Davis. Viton: An
image-based virtual try-on network. CoRR, abs/1711.08447,
2017.
[10] M. Hasan and A. K. Roy-Chowdhury. Context aware active
learning of activity recognition models. In Proceedings of the
IEEE International Conference on Computer Vision, pages
4543–4551, 2015.
[11] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning
for image recognition. CoRR, abs/1512.03385, 2015.
[12] D. Held, S. Thrun, and S. Savarese. Learning to track at 100
FPS with deep regression networks. CoRR, abs/1604.01802,
2016.
[13] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang,
T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Effi-

[14]

[15]

[16]
[17]

[18]

[19]

[20]

[21]

[22]

[23]

[24]
[25]

[26]
[27]

[28]
[29]

[30]

[31]

cient convolutional neural networks for mobile vision applications. CoRR, abs/1704.04861, 2017.
J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara,
A. Fathi, I. Fischer, Z. Wojna, Y. Song, S. Guadarrama, and
K. Murphy. Speed/accuracy trade-offs for modern convolutional object detectors. CoRR, abs/1611.10012, 2016.
D. Kang, J. Emmons, F. Abuzaid, P. Bailis, and M. Zaharia.
Optimizing deep cnn-based queries over video streams at
scale. CoRR, abs/1703.02529, 2017.
D. P. Kingma and J. Ba. Adam: A method for stochastic
optimization. CoRR, abs/1412.6980, 2014.
A. Li, J. Sun, J. Y. Ng, R. Yu, V. I. Morariu, and L. S. Davis.
Generating holistic 3d scene abstractions for text-based image retrieval. CoRR, abs/1611.09392, 2016.
H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf.
Pruning filters for efficient convnets. CoRR, abs/1608.08710,
2016.
W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. E. Reed,
C. Fu, and A. C. Berg. SSD: single shot multibox detector.
CoRR, abs/1512.02325, 2015.
C. Lu, R. Krishna, M. S. Bernstein, and F. Li. Visual relationship detection with language priors. CoRR, abs/1608.00187,
2016.
P. Ondruska and I. Posner. Deep tracking: Seeing beyond seeing using recurrent neural networks.
CoRR,
abs/1602.00991, 2016.
J. Rao, H. He, H. Zhang, F. Ture, R. Sequiera, S. Mohammed, and J. Lin. Integrating lexical and temporal signals
in neural ranking models for social media search. In SIGIR
Workshop on Neural Information Retrieval (Neu-IR), 2017.
J. Rao, F. Ture, H. He, O. Jojic, and J. Lin. Talking to your
tv: Context-aware voice search with hierarchical recurrent
neural networks. CIKM, 2017.
J. Rao, F. Ture, X. Niu, and J. Lin. Mining the temporal
statistics of ery terms for searching social media posts, 2017.
J. Redmon, S. K. Divvala, R. B. Girshick, and A. Farhadi.
You only look once: Unified, real-time object detection.
CoRR, abs/1506.02640, 2015.
J. Redmon and A. Farhadi. YOLO9000: better, faster,
stronger. CoRR, abs/1612.08242, 2016.
S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems 28, pages 91–99. Curran Associates, Inc., 2015.
S. Ruder. An overview of multi-task learning in deep neural
networks. CoRR, abs/1706.05098, 2017.
K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. CoRR,
abs/1406.2199, 2014.
A. Sobral and T. Bouwmans. Bgs library: A library framework for algorithm’s evaluation in foreground/background
segmentation. In Background Modeling and Foreground Detection for Video Surveillance. CRC Press, Taylor and Francis Group., 2014.
C. Stauffer and W. E. L. Grimson. Adaptive background mixture models for real-time tracking. In CVPR, pages 2246–
2252. IEEE Computer Society, 1999.

[32] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. E. Reed,
D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich.
Going deeper with convolutions. CoRR, abs/1409.4842,
2014.
[33] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna.
Rethinking the inception architecture for computer vision.
CoRR, abs/1512.00567, 2015.
[34] D. Tran, L. D. Bourdev, R. Fergus, L. Torresani, and
M. Paluri. C3D: generic features for video analysis. CoRR,
abs/1412.0767, 2014.
[35] M. Wang, B. Ni, and X. Yang. Recurrent modeling of interaction context for collective activity recognition. In The
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[36] Y. Wang, P.-M. Jodoin, F. Porikli, J. Konrad, Y. Benezeth,
and P. Ishwar. Cdnet 2014: An expanded change detection
benchmark dataset. In Proc. IEEE Workshop on Change Detection (CDW-2014) at CVPR-2014, pages 387–394, 2014.
[37] C. Wren, A. Azarbayejani, T. Darrell, and A. Pentland.
Pfinder: Real-time tracking of the human body. IEEE
Transactions on Pattern Analysis and Machine Intelligence,
19:780–785, 1997.
[38] B. Wu, F. Iandola, P. H. Jin, and K. Keutzer. Squeezedet:
Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving.
In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR) Workshops, July 2017.
[39] R. Y. D. Xu, J. G. Allen, and J. S. Jin. Robust real-time tracking of non-rigid objects. In Proceedings of the Pan-Sydney
Area Workshop on Visual Information Processing, VIP ’05,
pages 95–98, Darlinghurst, Australia, Australia, 2004. Australian Computer Society, Inc.
[40] R. Yu, X. Chen, V. I. Morariu, and L. S. Davis. The role
of context selection in object detection. In British Machine
Vision Conference (BMVC), 2016.
[41] R. Yu, A. Li, C.-F. Chen, J.-H. Lai, V. I. Morariu, X. Han,
M. Gao, C.-Y. Lin, and L. S. Davis. Nisp: Pruning networks using neuron importance score propagation. CoRR,
abs/1711.05908, 2017.
[42] R. Yu, A. Li, V. I. Morariu, and L. S. Davis. Visual relationship detection with internal and external linguistic knowledge distillation. CoRR, abs/1707.09423, 2017.
[43] Z. Zivkovic. Improved adaptive gaussian mixture model for
background subtraction. ICPR ’04, pages 28–31, Washington, DC, USA, 2004. IEEE Computer Society.

