Sequential Event Detection Using Multimodal Data
in Nonstationary Environments
Taposh Banerjee∗ , Gene Whipps† , Prudhvi Gurram‡† , Vahid Tarokh§

arXiv:1803.08947v1 [stat.AP] 23 Mar 2018

∗ Harvard

University, School of Engineering and Applied Sciences, Cambridge, MA 02138
† U.S. Army Research Laboratory, Adelphi, MD 20783
‡ Booz Allen Hamilton, McLean, VA 22102
§ Duke University, Department of ECE, Durham, NC 27708

Abstract—The problem of sequential detection of anomalies
in multimodal data is considered. The objective is to observe
physical sensor data from CCTV cameras, and social media data
from Twitter and Instagram to detect anomalous behaviors or
events. Data from each modality is transformed to discrete time
count data by using an artificial neural network to obtain counts
of objects in CCTV images and by counting the number of tweets
or Instagram posts in a geographical area. The anomaly detection
problem is then formulated as a problem of quickest detection
of changes in count statistics. The quickest detection problem
is then solved using the framework of partially observable
Markov decision processes (POMDP), and structural results on
the optimal policy are obtained. The resulting optimal policy is
then applied to real multimodal data collected from New York
City around a 5K race to detect the race. The count data both
before and after the change is found to be nonstationary in
nature. The proposed mathematical approach to this problem
provides a framework for event detection in such nonstationary
environments and across multiple data modalities.

I. I NTRODUCTION
Event detection has many real-world applications such
as surveillance [1], [2], border security using unattended
ground sensors (UGS) [3], crime hot-spot detection for law
enforcement [4], cyber-infrastructure monitoring [5], real-time
traffic monitoring [6], and environmental and natural disaster
monitoring [7], [8]. We address the problem of real-time event
detection for gathering tactical intelligence, which is critical
for military and law-enforcement missions. For instance, in
tactical scenarios, like cordon and search, there is a need
for gathering real-time intelligence that can help Soldiers
at the squad level gain situational understanding of a scene
and to quickly make mission-oriented decisions. To help gain
such actionable information, Soldiers may deploy a variety
of sensors such as cameras for imagery and video. The squad
may also have access to auxiliary information such as SIGINT,
SPOT reports, Blue Force Tracking data, and local social
network feeds. Currently, relevant information is processed
and analyzed in a far-off rear position, but there can be
significant delays in receiving important decisions at forward
positions. Some of the automated decisions that could help the
squad carry out the mission successfully include the detections
and locations of enemy entities such as personnel and vehicles.
The goal is to provide real-time threat indicators from all
available information sources at the point of need.

We tackle two fundamental questions that arise in such
scenarios. First, we have to process and fuse the information
from traditional physics-based sensing systems, such as video
sensors, and non-traditional sensing systems, such as social
networks, to provide indications and warnings. Second, we
have to push the processing to the operational environment
where the information is needed most, and hence there is a
need for real-time event detection. Motivated by these two
questions, in this paper, we consider the problem of real-time
event or anomaly detection using multimodal data.
To solve the complex multimodal event detection problem,
we need to develop useful mathematical models as well as
efficient algorithms and validate them on real-world data
collected during tactical scenarios. However, access to such
data for research is severely restricted. To overcome this, we
instead use publicly available data sources as surrogates for
tactical data sources. In the case of imagery, we use New
York City (NYC) traffic CCTV cameras as surrogates for low
altitude UAVs with video sensors. The video sensors onboard
tactical UAVs typically are low-resolution and have a wide
range of ground sample distances. Simiar image qualities are
present in the publicly available CCTV traffic camera imagery.
Instagram has medium to high-resolution imagery, which can
be viewed as surrogates for imagery collected by Soldierworn cameras. In scenarios where social media posts are not
available, the social media posts in this data collection could
be viewed as surrogates for SIGINT data (e.g., counts of
communications packets through local nodes).
In this work, we are interested in the subtle information
available in the dynamics of sequences of sub-events, e.g.,
changes in the counts of persons and vehicles in a spatial
region and changes in the corresponding social network posts
in the same region. As a result, we utilize the images from
the CCTV cameras to extract counts of persons and vehicles
in a spatial region. We also utilize the social media posts to
generate count sequences of Twitter and Instagram posts in
the constrained region. We develop a theoretical framework
and a novel algorithm for sequential detection of changes
in count statistics. The developed algorithm is then applied
to data collected from the NYC CCTV cameras and social
media feeds to detect a 5K race. The proposed mathematical
framework, and the developed algorithm can also be adapted
to other event detection problems. For example, in cyber-

infrastructure monitoring, the types and counts of intrusion
attempts can indicate the onset of a coordinated attack [9].
Towards developing a mathematical model for the problem,
we first study the statistical behavior of the count data on
the day of the event (a 5K race in NYC) and also on the
non-event days (see Section II). We observe that the count
sequences have nonstationary rates, i.e., the average counts
of persons, vehicles, or social media posts, change over time,
on each day. Thus, the event detection problem of interest in
this paper is a problem of detecting changes in the levels of
nonstationarity of rates. We use the framework of POMDP to
model the rate level change detection problem as detection
of time to absorption in a hidden Markov model (HMM)
[10], [11] (see Section III). Our POMDP problem is more
general than the one studied in [12] as we detect both increases
and decreases in rates. As a result, it is not apparent if our
POMDP solution has the threshold structure that was found
in the problem in [12]. However, in this paper, we show
that under certain assumptions on the transition structure of
the HMM, the solution to our POMDP problem also has a
simple threshold structure (see Section IV). We then apply
the resulting belief sum algorithm to detect the event (see
Section V).
II. DATA C OLLECTION AND M ODELING
We collected imagery from CCTV cameras and social
networks around the Tunnel to Towers 5K run that occurred
on September 24th, 2017, in NYC. Data was also collected on
two weekends before the run, on September 10th and 17th, and
a weekend after the run, on October 1st. CCTV imagery and
social media posts were collected over a geographic region
from the Red Hook village in Brooklyn on the south end
to the Tribeca village on the north end of the collection
area. Data were collected between 8:30 am and 2 pm local
on each of the 4 days. On average, the frame rate from 7
CCTV cameras was roughly 0.5 frames per second. While
the average post rates from Twitter and Instagram for the
geographic region and collection period were 1.4 and 0.7 posts
per second, respectively. Note that for this initial modeling and
analysis work, no other filtering of social posts was applied
(e.g., hashtag clustering or content analysis).
The objective is to detect the event in terms of location
and time of the 5K run from the multimodal data. It is to be
expected that the run would increase the number of persons
on the streets overlapping with the route followed for the run.
The run would also cause a sudden decrease in the number
of cars on the same streets. It is also expected that the event
would cause a surge in the number of tweets or Instagram
posts pertaining to the event. Motivated by these observations,
we approach this problem through the framework of quickest
detection in count data. The multimodal data is used to obtain
counts of objects (persons and vehicles) per frame and counts
of tweets and Instagram posts per second. Fig. 1 illustrates the
block diagram of the event detection system.
To obtain the counts of persons and vehicles, first, we use
a Convolutional Neural Network (CNN) based object detector

Counts of
objects in
images per
second

Number of
Tweets
per second

Event
Detection

Count
Data

Number of
Instagram posts
per second

Fig. 1: Event detection system

to detect persons and vehicles in each frame coming from the
CCTV cameras. Specifically, we use faster R-CNN [13], which
uses VGG16 architecture [14] as the base CNN with region
proposal networks, to perform real-time object detection. The
faster R-CNN is trained on the PASCAL Visual Object Classes
(VOC) dataset [15]. The dataset has labeled training data for
person class as well as vehicle classes that include bus, car, and
motorbike. The counts of persons and vehicles are generated
by simply counting the number of detected objects belonging
to the corresponding class in each frame.
In Fig. 2, we have plotted the total person counts, summed
across the seven CCTV cameras of interest, for each of the
four separate dates. Similar data for the total number of cars
are shown in Fig. 3. In this figure and the figures in the rest
of this paper, the horizontal axis is time in multiples of six
seconds. In Fig. 4, we have plotted the person counts for two
cameras: camera C1 is on the path of the race while camera C2
is off the path. In Fig. 5, we have similarly plotted individual
car counts for camera C1 and C2. We see a clear increase
in the rate of the number of persons and a slight decrease
in the number of cars on the day of the event between the
time slots 500 and 2000. Finally, in Fig. 6, we have plotted
cumulative counts of Instagram posts in geographical vicinity
of camera C1 and C2 for the four days. We see an increase in
the cumulative Instagram counts just around time slot 1500,
just before the person and car counts return to their normal
rates. We hypothesize that the latter is due to the fact that the
participants started posting on social media after completing
the race.
From these figures, one can make an observation that the
rates and counts are nonstationary in nature. Thus, the problem
of event detection here can be posed as a detection problem
in nonstationary environments. Since the event detection has
to be performed in real time, this would translate to sequential
detection of changes in rate from one nonstationary level to
another. In the next section, we formulate this problem in a
POMDP framework and solve it to obtain structural results on
the optimal solution. The resulting optimal algorithm will then

be applied to the collected data to detect the event.

Fig. 5: Car counts from two specific camera C1 and C2 on
different days.
Fig. 2: Total person counts from seven cameras for each day.
Horizontal axis is time in multiple of six seconds.

Fig. 6: Cumulative counts of Instagram posts in geographical
areas near cameras C1 and C2.
Fig. 3: Total person counts from seven cameras for each day.

Fig. 4: Person counts from two specific camera C1 and C2 on
different days. Camera C1 is on the path of the race, while
camera C2 is off the path.

parameter probability distribution, discrete or continuous, with
sums replacing integrals, where appropriate. Also, for simplicity, we develop the theory for event detection in a single
stream of count data here. The resulting algorithm is then
trained and applied to every count sequence generated from
every modality. However, the mathematical model can easily
be extended to a vector stream of observations to detect an
event jointly across modalities.
As observed in Fig. 2–Fig. 6, count sequences are nonstationary in nature, on both the event day and the no-event days.
In order to capture this nonstationarity, we model the count
data as an HMM. In this HMM, there are a finite number of
hidden states, and for each hidden state, the rate or mean of the
observed count or Poisson random variable is different. Thus,
if Y represents the observed count variable, X represent the
hidden state variable, and if λ1 < · · · < λN are N possible
rates for pre-event data, then
Y |X = k ∼ Pois(λk ).

III. P ROBLEM F ORMULATION
We note that the count data generated from any modality
is a sequence of discrete positive integers. For simplicity,
we model the count data as a sequence of Poisson random
variables. The results below are also valid for any single

When the event starts, which we call a change point, the rate
of counts either decreases (as in the case of cars) or increases
(as in the case of persons). In practice, the post-change rates
may not be known or may be hard to learn due to a lack
of enough training data of a rare event. Motivated by this, we

model the pre-change and post-change rates by boundary rates
λ0 (to capture a decrease in rates) and λN +1 (to capture an
increase in rates) such that
λ0 < λ1 < · · · < λN < λN +1 .
In other words, λ0 and λN +1 represent the minimum amount
of change the designer of the system is interested in detecting.
Note that while the number of cars decreases during the event
in this data, one may also observe an increase in the numbers
due to congestion of traffic. Thus, both increase and decrease
of rates are of interest to us, and our model allows for both
these possibilities.
We have N + 2 states, N normal states with corresponding
Poisson rates {λ1 < · · · < λN }, and 2 abnormal states
corresponding to Poisson rates {λ0 , λN +1 }. Our aim is to
observe the Poisson count data Y , and optimally detect the
change of the hidden rate from normal to abnormal rates.
Specifically, we want to detect this change as quickly as
possible subject to a constraint on the false alarm rate. This
leads us to the realm of quickest change detection [16], [17],
[18]. Here, we solve the rate change detection problem by
formulating it as a POMDP [10], [11].
A. POMDP Formulation
•

•

•

States
Let {Xk } be the sequence of states with values {xk }.
The state process is a finite state Markov chain taking
values Xk ∈ {A, 0, 1, 2, · · · , N, N + 1}, ∀k. The state A
is a special absorbing state introduced for mathematical
convenience in a stopping time POMDP [10]. Its role
will be clear when we define the cost structure below.
The transition probability matrix of the Markov chain is
a function of the control, and will also be defined below.
Control
The control sequence Uk taking values {uk } is binary
valued: Uk ∈ {1 (stop), 2 (continue)}. The control Uk =
2 is used to continue the observation process and Uk = 1
is used to stop it. At the time of stopping, an alarm is
raised indicating that a change in the rate of the data has
occurred.
Observations
The observations are Poisson distributed with rate λm , if
the state Xk = m ∈ {0, · · · , N + 1} and if the control is
to continue:
(Yk |Xk = m,Uk−1 = 2) ∼ Pois(λm ),
m ∈ {0, 1, 2, · · · , N + 1}, ∀k.
The distribution of observations, if the state is A or if
the control is to stop, is irrelevant. We use Bxy (u) to
denote the law of Y = y when the state is X = x and
the control is U = u. We also assume that the variable
Yk is independent of the past states and controls, given
the current state and last control. That is
(Yk |Xk , Uk−1 = 2) ∼
(Yk |X1 , · · · , Xk , U1 , · · · , Uk−2 , Uk−1 = 2)

•

Transition Structure
The transition structure depends on the control Uk . Let
Pk+1|k (uk ) = P (uk ) be the transition matrix for the
Markov chain from time k to k + 1, given the control
is Uk = uk . Then,we have
(
P1 , if uk = 1
Pk+1|k (uk ) = P (uk ) =
P2 , if uk = 2.
Here,


1
1

P1 =  .
 ..

0
0
..
.

0
0
..
.

...
...
..
.

1

0

0

...


0
0

.. 
.
0

and

1
0
0
a
1

0
p
10

P2 =  .
..
 ..
.

0
pN 0
0 1 − aN +1


0
1 − a1 

p1N p1(N +1) 

.
..
..

.
.

pN 1 . . . pN N pN (N +1) 
0
...
0
aN +1
(1)
To understand these two transition structures, we first
define the initial distribution π0 for the Markov chain
{Xk } as
0
0
p11
..
.

...
...
...
..
.

0
0

π0 = (π0 (A), π0 (0), π0 (1), · · · , π0 (N ), π0 (N + 1))T ,
which satisfies π0 (A) = π0 (0) = π0 (N + 1) = 0.
Thus, the Markov chain {Xk } starts in one of the states
{1, · · · , N }. As long as the control Uk = 2, which means
to continue, the states evolve according to the transition
probability matrix P2 . The transition probabilities


p10 p11 . . . p1N p1(N +1)


..
..
..
..
(2)
P̄ =  ...

.
.
.
.
pN 0

pN 1

...

pN N

pN (N +1)

that are part of the matrix P2 in (1) control the transition
of the Markov chain within the states {1, · · · , N }, and its
jump to the absorbing states 0 and N +1. We assume that
absorption to the states 0 and N + 1 is inevitable. Once
in these two states, the Markov chain jumps between
these two states with probabilities controlled by a0 and
aN +1 . We are especially interested in the case when
a1 = aN +1 = 1. This is because the states {1, · · · , N }
correspond to the normal states for the counts before the
change. After the change, we expect that either the rate
will increase, corresponding to absorption of the Markov
chain to the state N +1, or it will decrease, corresponding
to absorption to the state 0. Once the rate increases or
decreases, it is unnatural to expect that rate will transition
between too low and too high rates. However, the case
a1 = aN +1 = 0.5, is of mathematical interest, and its
role and importance will be briefly discussed below.

•

Cost
Our objective is to detect a change in the rate of counts
from normal rates (λ1 , · · · , λN ) to abnormal rates λ0
and λN +1 . This is equivalent to detecting the time to
absorption of the Markov chain {Xk } from the states
(1, · · · , N ) to the states (0, N +1). We now define a cost
structure for the POMDP to capture the sequential event
detection framework. Let C(x, u) be the cost associated
with state X = x, and control U = u and defined as
C(x, 1) =
C(x, 2) =

CfT ex
CdT ex

= (0, 0, cf , cf , · · · , cf , 0) ex ,
= (0, cd , 0, 0, · · · , 0, cd ) ex .

Policy
Let Ik = (y1 , · · · , yk , u1 , · · · , uk−1 ) be the information
at time k. Also define a policy Φ = (φ1 , φ2 , · · · ) to be a
sequence of mappings such that uk = φk (Ik ), ∀k.

We want to find a control policy so as to optimize the long
term cost, which is
"∞
#
X
V (π0 ) = min E
C(xk , uk ) ,
Φ

k=1

where uk = φk (Ik ). Let τ = inf{k : xk+1 = A}. Then,
" τ
#
X
V (π0 ) = min E
C(xk , uk ) .
Φ

(3)

k=1

Thus, the cost is finite if E[τ ] < ∞. The role of the extra state
A is now clear. After the stopping control uk = 1 is applied,
the Markov chain’s transition is governed by the transition
matrix P1 . As a result, the Markov chain gets absorbed into
the state A immediately. From here, due to the cost structure,
the cost to go is zero, no matter what control is chosen. In
conclusion, we search over policies Φ for which E[τ ] < ∞.
This is hardly a concern since any open-loop policy, where
we always stop at a fixed time, satisfies this condition. We
are looking for policies better than that, i.e., for closed-loop
control that allows us to stop dynamically after observing the
system.
IV. S TRUCTURE OF THE O PTIMAL P OLICY
Let πk be the belief at time k defined as πk = P(Xk =
xk |Ik ). Note that the belief is a vector of length N + 3.
By standard Bayes arguments, this belief can be computed
recursively as
πk+1 = T (πk , yk+1 , uk ) =

Byk+1 (uk ) P (uk )T πk
.
1T Byk+1 (uk ) P (uk )T πk

By = diag(BAy (u), B0y (u), · · · , B(N +1)y (u))
and 1 = (1, · · · , 1)T is a vector of all 1s of length N + 3.
It is a standard result in the POMDP literature that the cost
to go V (π) in (3) satisfies the Bellman’s equation
(
)
X
T
T
V (π) = min Cf π, Cd π +
V (T (π, y, 2)) σ(π, y, 2) ,
y
T

Here, ex is a unit column vector with value 1 at the xth
position. The constant cf captures the cost of false alarm,
and is incurred when the control is to stop and the state
is in (1, · · · , N ). Similarly, cd captures the cost of delay,
and is incurred when the control is to continue even if the
Markov chain is absorbed in either of the states (0, N +1).
Note that the cost of being in state A is zero independent
of the choice of control.
•

Here, By (u) is a diagonal matrix of emission probabilities

(4)

where σ(π, y, u) = 1 By (u)P (u)T π. It can be shown that
the optimal policy is stationary and is a function of only the
belief state. Furthermore, the value function can be computed
using value iteration [10], [11]. That is, the optimal policy is of
the form Φ∗ = (µ∗ (π), µ∗ (π), · · · ). In addition, the following
result can also be shown.
Theorem 4.1 ( [19], [10]): Let
R1 = {π : µ∗ (π) = 1}
be the region of the belief space on which the control U = 1 is
chosen, or the stopping decision is made. Then R1 is convex.
A standard approach to solving POMDP problems, which
are typically hard to solve due to the high-dimensionality of
the belief space, is to establish additional structural results on
the policy µ∗ . Specifically, it is of interest to show that the
optimal stopping time in a POMDP has a threshold structure,
or the policy µ∗ (π) is, in some sense, monotone in π. The
threshold structure motivates the use of policies that are linear
in the belief state. See [10] for a detailed discussion.
Unfortunately, all the conditions needed to establish the
threshold structure are not satisfied in our problem. For example, the transition structure and the emission probabilities
satisfy the so-called total positivity conditions. But, the cost
structure does not have the required monotonicity and submodularity structure [10]. Even a transformation of the problem,
as suggested in [10] does not help. The main issue is that in
comparison with the results in [12], in this paper, we have two
absorbing states, one for the low rate and another for the high
rate. However, we now establish that under some additional
assumptions, the optimal policy can be shown to be only a
function of the probabilities πk (0) and πk (N + 1).
Theorem 4.2: Let the rows of the matrix P̄ in (2), and hence
the corresponding elements of P2 in (1), be identical. Also,
let a1 = aN +1 = 1 in P2 . Then, the optimal policy depends
only on the two components π(0) and π(N + 1) of the belief
state π.
Proof: Note that even without the addition assumptions
that are made in the theorem statement, the value function
satisfies the condition
(
)
X
T
T
V (π) = min Cf π, Cd π +
V (T (π, y, 2)) σ(π, y, 2)
y

= min {cf (1 − π(0) − π(N + 1)),
)
X
cd (π(0) + π(N + 1)) +
V (T (π, y, 2)) σ(π, y, 2) .
y

This is because of the special structure of the cost function
assumed in the paper. Now, we can show that under the
assumptions of the theorem, the belief recursion (4) can be
computed just based on the values of π(0) and π(N + 1).
Hence, the fixed point equation of the value function is only
a function of these two values. The rest follows by using the
standard value iteration arguments.
The condition that the rows of P̄ be same is easily satisfied
in the following special case:
 1

1
. . . N1+2 N1+2
N +2
N +2

..
..
..  .
..
P̄ =  ...
(5)
.
.
.
. 
1
1
1
1
.
.
.
N +2
N +2
N +2
N +2
Thus, with this choice of P̄ , the Markov chain moves around
the states 1 to N randomly and can get absorbed to 0 and
N + 1, all with equal probability. Numerical evaluation of
the optimal policy for this case, under some parameters of
choice, shows that the optimal policy is a function of π(0) and
π(N + 1) only through π(0) + π(N + 1). In fact, according
to this numerical study, the optimal stopping rule is
τ ∗ = inf{k : πk (0) + πk (N + 1) > A}.

(6)

We note that although the marginal costs are a function
of π(0) and π(N + 1) only through π(0) + π(N + 1), the
belief recursion cannot be computed just using this sum. To
compute (4) we need both π(0) and π(N + 1) individually.
This can be verified by explicitly writing the belief recursion
under the stated assumptions. Thus, it is not clear to us at this
moment if this threshold structure, the optimal policy being
only a function of the sum π(0) + π(N + 1), holds for more
general cases.
However, if we make the assumption that a1 = aN +1 = 0.5,
then in this case, we can show that the optimal policy is only
a function of the sum π(0) + π(N + 1).
Theorem 4.3: Let the rows of the matrix P̄ in (2), and hence
the corresponding elements of P2 in (1), be identical. Also,
let a1 = aN +1 = 0.5 in P2 . Then, the optimal policy depends
only on the two components π(0) and π(N + 1) of the belief
state π through their sum π(0) + π(N + 1).
Proof: Under the stated assumptions, the belief recursion
can be shown to be a function of only the sum π(0)+π(N +1).
The rest of the proof is identical to that of Theorem 4.2.
Note that this means that under the assumptions made in
Theorem 4.3, and in Theorem 4.2, the quickest change detection problem studied here reduces to the classical quickest
change detection problem [20] in some sense. In the classical
problem, there are two hidden states, one before the change
and another after the change. The hidden Markov chain starts
with one state and gets absorbed into the other. The objective
in the classical change point problem is to detect this time to
absorption. The optimal algorithm for the classical problem is
to stop the first time, the belief that the Markov chain is in
the post-change state, is above a threshold.

In the change point problem considered in this paper, we
have two classes of states, one class consisting of pre-change
or normal states {1, · · · , N }, and another class consisting of
post-change or abnormal states {0, N + 1}. And the objective
here is to detect the time at which the Markov chain {Xk }
moves from the pre-change class to the post-change class. The
above theorems suggest that under the stated assumptions, the
optimal stopping rule has a similar structure. That is, it is
optimal to stop the first time the probability that the Markov
chain is in the post-change class of states is above a threshold,
such as that in stopping rule (6).
V. T HE B ELIEF SUM A LGORITHM AND ITS A PPLICATIONS
In the previous section, we established conditions on the
transition structure of the HMM under which the algorithm,
τ ∗ = inf{k : πk (0) + πk (N + 1) > A},

(7)

is optimal. However, note that under more general cost structures, it is not obvious if this is still the optimal algorithm. As
a result, and motivated by the optimality of (7) under some
assumptions, we use a more general class of algorithms that
are linear in the beliefs πk (0) and πk (N + 1). That is, we use
the sum belief algorithm using a convex combination of the
beliefs
τ ∗ = inf{k : α πk (0) + (1 − α) πk (N + 1) > A},

(8)

where α ∈ [0, 1], and optimize over the choice of α. Due to
a paucity of space, a detailed delay and false alarm analysis
of this algorithm will be reported elsewhere. In this section,
however, we apply the algorithm to real data to show its
effectiveness. In the following, we often use α = 0.5. In those
cases, we actually report values of the sum statistic in (7).
A. Global Event Detection
We now apply the algorithm to data collected around the
5K run. The details on the data are provided in Section II. In
practice, the algorithm should be applied to data collected from
each individual source: to outputs of each camera and also to
outputs of social media data in each geographical region. A
high value of the sum statistic would indicate an abnormal
behavior in a stream. This can be used to both detect and
isolate the event. This is done in the next subsection. However,
we may also wish to apply the algorithm to the global sum of
data collected to detect global trends, in case they generate a
collaborative effect.
We applied the algorithm to total count data from all
cameras for global detection of the event. We first used the data
from the first recorded day, Sept. 10th, to learn the Poisson
rates. We then applied the trained algorithm to data from other
days. The parameters used were λ0 = 0.001, λ1 = 5, λ2 = 10,
λ3 = 15, λ4 = 20, λ5 = 25, and λ6 = 65, i.e., N = 5. The
transition matrix used was assumed fixed as in (5) and (1) with
a1 = aN +1 = 1. Note that the rate parameters we learn from
the data are λ1 to λN . The values λ0 and λN +1 are chosen to
be the boundary of the learned rates based on multiple standard
deviations from normal rates.

Fig. 7: Evolution of the sum statistic (7) for data from the
non-event days Sept. 17 and Oct 1.

Fig. 9: The evolution of the sum statistic (7) for data from the
non-event days.

Fig. 8: Evolution of the sum statistic (7) for data from the event
day Sept. 24th. We see high-values of the statistic between the
times 500 and 2000, the times of the event as seen in Fig. 2.

Fig. 10: The evolution of the sum statistic (7) for total car
data from the event day. We see sustained high values for the
statistic around the time of the event.

In Fig. 7 and Fig. 8, we report the results on application
to total person count data. The data for the event day and
a non-event day is shown in Fig. 2. Note that the statistic is
sporadically large on the non-event days, but consistently fires
around the event on the day of the event.
Similar results are reported in Fig. 9 and Fig. 10 for the total
car count data from Fig. 3. The learned values of rates from
Sept. 10th data are λ0 = 0.00001, λ1 = 0.001, and λ2 = 55,
with the rest of the parameters kept the same. As seen in Fig.9,
the statistic fires sometimes even on the non-event days. This
is because we are applying the algorithm to the sum of count
data from all cameras, and this may reduce the quality of the
count sequence.

and from camera C2. As can be seen in the figures, the statistic
corresponding to C1 on the event day fires around the event,
while we see almost no activity in other streams.
In Fig. 13, we report results for the Instagram count data.
We have again used (8) with α = 0. The parameters learned
from the data on Sept. 10th are λ0 = 0.001, λ1 = 0.1,
and λ2 = 2. The sum statistic fires and stays close to one
for counts from area close to camera C1 on the event day,
while there is sporadic activity for data from other streams.
Thus, qualitatively, the sum statistic or sum belief algorithm
successfully detects the 5K event.

B. Event Localization
In this section, we report results on the application of the
algorithm (8) to individual data streams. In Fig. 11, we have
plotted the evolution of the sum statistic for data from the
camera C1. The count data is shown in Fig. 4. The parameters
learned from the data on Sept. 10th are λ0 = 0.001, λ1 = 2,
λ2 = 4, λ3 = 6, λ4 = 8, and λ5 = 55. We have also used
(8) with α = 0. In Fig. 12, we have plotted the sum statistic
corresponding to data from camera C1 on the non-event days,

VI. C ONCLUSIONS
We proposed a theoretical framework for event detection in
nonstationary environments using multimodal data. Motivated
by the statistical behavior of count data extracted from CCTV
images and social network posts, we formulated the event
detection problem as a quickest change detection problem for
detecting changes in count rates from one family of rates to
another. We then obtained structural results for the optimal
policy for the resulting POMDP and motivated a belief sum
algorithm. We then applied the algorithm to real data collected
around a 5K run in NYC to detect the event. For simplicity,

Fig. 11: Evolution of the sum statistic (8) with α = 0 applied
to data from camera C1 on the day of the event.

Fig. 12: Evolution of the sum statistic (8) with α = 0 applied
to data from camera C1 on non-event days and camera C2,
which is off the run path, on event and non-event days.

we developed the framework for a single stream of count data
here. However, the mathematical model can easily be extended
to a vector stream of observations to detect an event jointly
across modalities. The POMDP model studied in this paper
is a Bayesian model. In the future, we will explore detection
in non-Bayesian models. We will also explore more general
parametric and non-parametric models for count data for wider
applicability.
ACKNOWLEDGMENT
The work of Taposh Banerjee and Vahid Tarokh was supported by a grant from the Army Research Office, W911NF15-1-0479.
R EFERENCES
[1] R. Panda and A. K. Roy-Chowdhury, “Multi-view surveillance video
summarization via joint embedding and sparse optimization,” IEEE
Transactions on Multimedia, vol. 19, no. 9, pp. 2010–2021, 2017.
[2] S. C. Lee and R. Nevatia, “Hierarchical abnormal event detection by
real time and semi-real time multi-tasking video surveillance system,”
Machine vision and applications, vol. 25, no. 1, pp. 133–143, 2014.
[3] R. Szechtman, M. Kress, K. Lin, and D. Cfir, “Models of sensor
operations for border surveillance,” Naval Research Logistics (NRL),
vol. 55, no. 1, pp. 27–41, 2008.

Fig. 13: Evolution of the sum statistic (8) with α = 0 applied
to data from Instagram counts near camera C1 and C2 on event
and non-event days.

[4] D. B. Neill and W. L. Gorr, “Detecting and preventing emerging
epidemics of crime,” Advances in Disease Surveillance, vol. 4, no. 13,
2007.
[5] R. Mitchell and I. R. Chen, “Effect of intrusion detection and response
on reliability of cyber physical systems,” IEEE Transactions on Reliability, vol. 62, pp. 199–210, March 2013.
[6] E. D’Andrea, P. Ducange, B. Lazzerini, and F. Marcelloni, “Real-time
detection of traffic from Twitter stream analysis,” IEEE Transactions on
Intelligent Transportation Systems, vol. 16, pp. 2269–2283, Aug 2015.
[7] E. W. Dereszynski and T. G. Dietterich, “Probabilistic models for
anomaly detection in remote sensor data streams,” arXiv preprint
arXiv:1206.5250, 2012.
[8] T. Sakaki, M. Okazaki, and Y. Matsuo, “Earthquake shakes Twitter users:
Real-time event detection by social sensors,” in Proceedings of the 19th
Int. Conf. on World Wide Web, pp. 851–860, ACM, 2010.
[9] R. Harang and A. Kott, “Burstiness of intrusion detection process:
Empirical evidence and a modeling approach,” IEEE Transactions on
Information Forensics and Security, vol. 12, pp. 2348–2359, Oct 2017.
[10] V. Krishnamurthy, Partially Observed Markov Decision Processes. Cambridge University Press, 2016.
[11] D. P. Bertsekas and S. Shreve, Stochastic optimal control: the discretetime case. Academic Press, 1978.
[12] V. Krishnamurthy, “Bayesian sequential detection with phase-distributed
change time and nonlinear penaltya pomdp lattice programming approach,” IEEE Transactions on Information Theory, vol. 57, no. 10,
pp. 7096–7124, 2011.
[13] S. Ren, K. He, R. B. Girshick, and J. Sun, “Faster R-CNN: towards real-time object detection with region proposal networks,” CoRR,
vol. abs/1506.01497, 2015.
[14] K. Simonyan and A. Zisserman, “Very deep convolutional networks for
large-scale image recognition,” CoRR, vol. abs/1409.1556, 2014.
[15] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” International
Journal of Computer Vision, vol. 88, pp. 303–338, June 2010.
[16] V. V. Veeravalli and T. Banerjee, Quickest Change Detection. Academic
Press Library in Signal Processing: Volume 3 – Array and Statistical
Signal Processing, 2014. http://arxiv.org/abs/1210.5552.
[17] H. V. Poor and O. Hadjiliadis, Quickest detection. Cambridge University
Press, 2009.
[18] A. G. Tartakovsky, I. V. Nikiforov, and M. Basseville, Sequential
Analysis: Hypothesis Testing and Change-Point Detection. Statistics,
CRC Press, 2014.
[19] W. S. Lovejoy, “On the convexity of policy regions in partially observed
systems,” Operations Research, vol. 35, no. 4, pp. 619–621, 1987.
[20] A. N. Shiryayev, Optimal Stopping Rules. New York: Springer-Verlag,
1978.

