Detection and Description of Change in Visual Streams
Davis Gilton Ruotian Luo Rebecca Willett Greg Shakhnarovich
cs.CV


Event-Triggered Quantized Average Consensus via Mass Summation
    Apostolos I. Rikos, Member, IEEE
	Apostolos I. Rikos is with Division of Decision and Control Systems, KTH Royal Institute of Technology, SE-100 44 Stockholm, Sweden. E-mail: rikos@kth.se., 
Christoforos N. Hadjicostis, Fellow, IEEE 
	Christoforos N. Hadjicostis is with the Department of Electrical and Computer Engineering at the University of Cyprus, Nicosia, Cyprus, and also with the Department of Electrical and Computer Engineering at the University of Illinois, Urbana-Champaign, IL, USA. E-mail: chadjic@ucy.ac.cy.
	Parts of the results for quantized average consensus via event-triggered mass summation appear in <cit.>. The present version of the paper includes complete proofs for convergence as well as an enhanced version of the algorithm that allows nodes to determine when to seize transmissions, once quantized average consensus is reached (not addressed in <cit.>).

    
========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================




	This paper presents a framework for the analysis of changes in
	visual streams: ordered sequences of images, possibly separated by
	significant time gaps. We propose
	a new approach to incorporating unlabeled data into training
	to generate natural language descriptions of change. We also develop a framework for estimating the time of
	change in visual stream. We use learned representations for
	change evidence and consistency of perceived change, and combine these
	in a regularized graph cut based change detector. Experimental
	evaluation on visual stream datasets, which we release as part of
	our contribution, shows that representation learning driven by natural
	language descriptions significantly improves change detection
	accuracy, compared to methods that do not rely on language. 



§ INTRODUCTION

Visual streams depict changes in a scene over time, and this paper describes a learning-based approach to describing  those changes in a manner that (a) improves the detection changes beyond what is possible based solely on image analysis without change descriptions, (b) incorporates unlabled data to yield strong performance even with a limited number of labeled samples, and (c) is robust to nuisance changes such as lighting or perspective shifts. 

In general, visual streams (illustrated in Figure <ref>) consist of time series of images of a scene, such as time lapse imagery or a sequence of satellite images. Given such visual streams, our goal is to detect and describe “relevant” changes while ignoring nuisance changes such as lighting or perspective changes or seasonal change. This change detection and description task is relevant to intelligence, insurance, urban planning, and natural disaster relief. 




Our contributions to visual stream change detection and captioning
are:

	
  * We formulate the joint change detection and description
	problem and develop two new datasets, one synthetic and one
	real, of visual streams with known changepoints and change
	descriptions in natural language. 
	
  * We propose a semi-supervised framework for describing
	changes between images that leverages unlabeled image pairs
	to improve performance, an important feature given
	the cost of human labeling of visual streams.
	
  * We propose and analyze an approach to change
	detection within a visual stream. It is based on a regularized 
	graph cut cost that also incorporates the
	consistency of perceived change for image pairs straddling the change point. The
	representation used to assess such consistency emerges from
	learning to describe change with language. We show that this approach is
	superior to baselines that do not incorporate language in learning.


The efficacy of including our language model in the change
detection process shows that access to language change descriptions at training time may provide a richer form of supervision
than training using the times of changes alone. This is a new, previously unexplored avenue for leveraging the interaction between vision and language.



§ RELATED WORK


While automatic image captioning has received growing attention in the
computer vision community in the past several years
<cit.>,
such tools are inadequate for  detection of changes in a 
visual stream. Describing changes in a scene is fundamentally
different from recognizing objects in an image. 
Unlike in classical automatic image captioning, a visual stream may contain no relevant changes and it may contain multiple objects that do not change. 
Furthermore, video captioning methods <cit.> are inapplicable because in many visual streams there are relatively large periods of time between subsequent images; for instance, a satellite may pass over a location only once per day. Finally, the images of
interest may be corrupted by blur, shifts, and varying environmental
conditions that make identifying changes challenging. 



 §.§ Automatic description of the visual world

Visual descriptions
The task of describing the content of an image or video in natural
language, often referred to as image/video captioning, has emerged as
one of the central tasks in computer vision in recent
years <cit.>.
Typically,
captioning is approached as a translation task,
mapping an input image to a sequence of words.
In most existing work, learning is fully supervised,
using existing data sets <cit.> of “natural images/videos” with associated
descriptions. However, when faced with drastically different
visual content, remote sensing or surveillance videos, and a different type of descriptions,
in our case describing change, models trained on natural data lack both the visual
“understanding” of the input and an appropriate vocabulary for the
output. We address this limitation by
leveraging novel semi-supervised learning methods, requiring only a small amount of labeled
data.

Automatic description of
	change. Change is implicitly
handled in methods for captioning video <cit.>.
However, when describing change in the videos, these
methods tend to focus on describing the “updated scene” rather than the
nature of the change between the scenes.

The most related work to ours is <cit.>, who also 
attempt to generate captions for non-trivial changes in images.
<cit.> proposes the Spot-the-Diff dataset, where pairs of images are 
extracted from surveillance videos. The viewpoints are fixed and there is always some change.
<cit.> proposes CLEVR-Change, a dataset where pairs of synthesized images are provided with
their synthesized captions. Lighting and viewpoint can vary between image pairs.
We extend <cit.> in two ways. First, we consider a series of images instead of two,
and a change can either happen at any timestep, or not at all. Secondly, we consider a
semi-supervised setting where only a subset of image series has associated captions. Moreover,
we report results not only on CLEVR-change and Spot-the-Diff, but also a newly collected street-view dataset.

Semi/unsupervised captioning
The goal of unsupervised/unpaired captioning is to learn to caption from a set of images
and a corpus, but with no mapping between the two. Usually, some pivoting is needed:
<cit.> tackle unpaired captioning
with a third modality,
chinese captions and scene graph respectively. Similarly, <cit.>
use visual concept detectors
trained on external data to bridge the unpaired images and captions.


In the semi-supervised setting, extra unlabeled data supplement paired image caption data.
<cit.> uses a self-retrieval module to mine negative samples from unlabeled
images to improve captioning performance.
<cit.> uses a GAN to generate pseudo labels, making extra unpaired images and captions paired.
<cit.> exploit additional unpaired text and image for novel object captioning.

Our work is most similiar to <cit.> but with several important differences. First, we use a real-fake discriminator instead of a retrieval-based discriminator.
Second, the datasets/tasks are different. Our datasets are more under-annotated and out-of-domain than
COCO<cit.>, a large natural image dataset that can benefit
easiliy from pretrained vision networks.
Note that, despite the similar technique, our focus in this paper is not
to propose a new semi-supervised technique, but a more general setting of change descriptions in an annotation-restrained situation.




 §.§ Change point detection and localization

Change point detection and localization is a classical problem in time
series analysis. Typical approaches aim to determine whether and at what time(s)
the underlying generative model associated with the signal being
recorded has changed <cit.>.
Such techniques are also used in video indexing, to detect a scene cut, a shot boundary, etc.
<cit.>.

In this work, we focus on finding a point that a nontrivial change in content happens while 
many nuisance changes
such as misalignments or shifts between images and illumination
changes, seasonal variations, etc., may dominate the visual changes.
Much current work on high-dimensional change point detection (<cit.>) assumes that the time series has a mean that is
piecewise-constant over time; this is clearly violated by real world
data, such as shown in Fig <ref>.
We show that learning to describe changes can help locate changepoints.



§ SEMISUPERVISED LEARNING OF CHANGE DESCRIPTION

Suppose we have L image pairs labeled with
change descriptions, (^ℓ,^ℓ,w^ℓ),
ℓ=1,…,L. Of these, some may correspond to no change, with
w^ℓ= like “no change” or “nothing changed”.
An additional set of U unlabeled pairs
(^u,^u), u=1,…,U is presumed to only
contain pairs where a change does occur between ^u and
^u, although no description of this change is available.




Learning to generate descriptions proceeds in three phases:

	
  * Phase 1 (Sec. <ref>): We  use labeled data to
	pretrain a description generator G, Fig. <ref> (left). A conditional generative
	model, G allows us to calculate the likelihood p(w|I_t, I_t') of a change description w
	given an image pair, as well as sample (estimate) a caption ŵ_G for a given
	image pair. The former capacity is used only during training. 
	As part of its operation, G(,) computes a
	hidden state h(,). 
	
  * Phase 2 (Sec. <ref>): We use both labeled and
	unlabeled data to train a discriminator D, Fig. <ref> (center).
	For two images I_t and I_t' and caption w, this
	discriminator computes a “compatibility score”  D(w,I_t,I_t') indicating whether
	w is a valid description of the change between I_t and I_t'.
	
  * Phase 3 (Sec. <ref>): We fine-tune the
	description generator G using both labeled and unlabeled data, leveraging the discriminator D, Fig. <ref> (right). 

Before describing these training phases in detail, we outline the
architecture choices for G and D. G uses the DUDA architecture introduced in <cit.>, which can be trained
to be robust to lighting and pose changes. Briefly, it extracts visual
features from I_t and I_t' using a ResNet <cit.>, and
distills them via spatial attention maps A_t and A_t' into a
triplet of visual feature vectors, associated with I_t, I_t' and the
difference between the two. Finally, these and the caption
w=(w_1,…,w_K) are fed into an LSTM-based
recurrent network with temporal attention states a_1,…,a_K used
to combine the visual features at each time step.

The discriminator D shares the components of DUDA with G. However,
instead of using the LSTM model to estimate proability of the word
sequence w=(w_1,…,w_K) given (I_t,I_t'), here the LSTM model just consumes
the words in w, updating the hidden state. The final state h_K is
fed to a multi-layer fully connected network, and finally to a linear binary
classifier. Its output D(w,I_t,I_t') is an estimate of the probability that the w is
a valid description of change from I_t to I_t'.




 §.§ Phase 1: Training description generator G



As in <cit.>, training of the description generator G is done with the regularized cross-entropy loss,

    L_G :=  -∑_ℓ=1^L log p(w^ℓ | I_t^ℓ, I_t'^ℓ) + λA_t^ℓ_1 + λA_t'^ℓ_1 - μ∑_k=1^K_ℓ H(a_k^ℓ)

where w^ℓ is the ground truth change description for
the labeled pair (^l,^l), A_t^ℓ and A_t'^ℓ are
the spatial maps computed inside DUDA for that pair, and
{a_k^ℓ} is the sequence of temporal attention states computed
inside DUDA; H is the entropy. For further details, refer to <cit.>.





 §.§ Phase 2: Learning to recognize valid change captions



We train a discriminator using both labeled and unlabeled
image pairs to minimize the loss

    L_D :=    -∑_ℓ=1^L log D(w^ℓ, I_t^ℓ, I_t'^ℓ)   [positives from labeled pairs]
        - ∑_ℓ≠ℓ'log (1-D(w^ℓ, I_t^ℓ',
    I_t'^ℓ'))   [negatives from labeled pairs]
        - ∑_ℓ, ulog(1-D(w^ℓ,I_t^u,I_t'^u))
       [negatives across labeled/unlabeled]

The sum in (<ref>) is over labeled pairs with known
change annotations; these are the positive examples for D. Note that
these may includes pairs with no change, annotated by captions like
w=“no change”. The second sum in (<ref>) is over
examples obtained by matching one pair with the annotation
for another pair. Here we assume that with high probabilty such a
randomly matched annotation will not in fact be valid for the given
pair, and so these are negative examples for D.

Finally, we turn to the unlabeled pairs. The third sum
in (<ref>)
matches annotations from labeled data to unlabeled
pairs, which we assume will not create valid
descriptions. This procedure creates additional negative examples for D.

We include negative examples from labeled images for two reasons. The first
is to ensure the network is trained using negative examples even in the fully-supervised case.
Secondly, we wish to prevent overfitting, where D learns simply to output 
0 for all pairs in the unlabeled set, and 1 for all of the labeled set.



 §.§ Phase 3: Updating  generator G using discriminator D

   
With D in hand, we can refine the generator G using both labeled
and unlabeled data. We  train G to generate change captions that the discriminator
labels as “valid,” while still staying close to the ground truth data. To ensure that G generates captions that resemble true labels, we simultaneously maximize the likelihood of the ground truth and the outputs of the discriminator for captions generated by G. We use the following loss:

    L_3 :=   ∑_ℓ=1^L L_G(w^ℓ,I_t^ℓ, I_t'^ℓ) - λ∑_ℓ=1^L log D(G(I_t^ℓ,I_t'^ℓ),I_t^ℓ,I_t'^ℓ) 
            - λ∑_u=1^U log D(G(I_t^u,I_t'^u),I_t^u,I_t'^u)


Since sampling captions from G to pass to D is not differentiable, we treat training G using D as a reinforcement learning problem, which we solve using REINFORCE <cit.>. This approach has been used elsewhere for learning in sequence generation <cit.>. 

In this case, the policy is given by G, parametrized by the network parameters θ_G, which predicts a distribution p_G(w|I_t, I_t') over captions. We wish to maximize some reward, R(w, I_t, I_t'), which should be high for valid change captions and their associated images. The gradient of the expected reward for G is approximated by:

    ∇_G 𝔼_w[ R(ŵ_G,I_t, I_t') ] ≈ R(ŵ_G,I_t, I_t') ∇_θ_Glog p_G(ŵ_G|I_t, I_t'),

where ŵ_G is the result of sampling from p_G(w|I_t, I_t'), and p_G(ŵ_G|I_t, I_t') is defined by the output of G. In our setting, R(ŵ_G, I_t, I_t') is given by log D(ŵ_G,I_t,I_t'). This permits the gradient of L_3 to be approximated at training points, and for unlabeled data to be used while training G, to ensure that as much data is incorporated into the training procedure as possible.




While GAN training traditionally uses feedback from a learned discriminator as a training signal for a generator, typically GAN training includes optimization over a minimax objective, where the discriminator and generator compete over the course of training. We do not update the discriminator in an adversarial fashion, instead freezing it during the final phase of training.



 §.§ Training with visual streams

The above relies on pairs of images. More generally, we may have
access to a training set of visual streams, which
consist of ordered image sequences {I_t}_t=0^T,
associated times τ∈{1,…,T} at which a change occurs,
and (possibly) natural language descriptions of the changes. Such
sequences may be used to generate training data as follows. Consider
an annotated stream (I_1,…,I_T, τ, w). For every
t≤τ,t'> τ it yields an annotated pair (I_t,I_t',w), while for
every t,t'≤τ, we get (I_t,I_t',“no change”), and
similarly for every t,t'>τ. See Figure <ref>.

If the stream (I_1,…,I_T, τ) is not annotated with change
description, we still can get the pairs annotated with “no change”
for the frame indices that do not straddle τ. However, for
t≤τ,t'>τ we can only get unlabeled pairs (I_t,I_t').



§ CHANGE DETECTION


The system described in Section <ref> can be used not only to caption 
changes in visual streams, but also to detect and localize the occurence of changes.
In this section, we describe how to use the learned modules from Section <ref>, which operate in a pairwise manner, to identify changepoints in image sequences. 

More formally, assume we are given a visual stream, represented as an ordered sequence of images {I_t}_t=0^T. Nuisance changes, such as viewpoint, lighting, or weather variation, are observed between each sequential pair of frames of the stream. There is some unknown τ∈{1,…,T} such that between t=τ-1 and t'=τ a relevant (non-nuisance) change occurs, after which only nuisance changes occur between frames. Our goal in change detection is to estimate τ. 

We begin by defining an abstract pairwise relation P(I_t, I_t'): P(I_t, I_t') is a relation that, given two images (I_t, I_t'), outputs a statistic which is high for pairs (I_t, I_t') where a change has occurred between t and t', and is low otherwise.

We can now use these pairwise statistics to define different score functions for various candidate values of τ in a visual stream; in general, we will compute an estimate τ̂ by maximizing these score functions. For any method which generates a score S_X, τ_X = _τ S_X.





  
Step-wise scores.
We define the step-wise score as

    S_ step(τ) := P(I_τ-1,I_τ)

and the associated change point estimate as τ_ step = _τ S_ step(τ).
This approach finds the consecutive pair of frames in the stream which is most likely to contain a change between that pair. Unfortunately, if P(I_t, I_t') is noisy or imperfect, the step-wise  approach may be prone to errors and spurious detections. 



  
Graph cut scores. To address the noise sensitivity for the step-wise approach, we propose a strategy based on graph cuts. Specifically, we form a graph with each node corresponding to a t ∈{1,…,T}. We assign an edge weight to each pair of nodes t and t' (specifics below) and define a score function with the edge weights.
Selecting a candidate change point τ is then equivalent to selecting a graph cut, where the edges that are cut connect some t ≤τ and t' > τ; 
we denote the set of these edges as E_τ. A visualization is provided in Figure <ref>, where a candidate τ results in an edge partitioning into E_τ, in black, and E^c_τ, in green.

The graph-cut score we consider is, for E^c_τ the complement of E_τ,

    S_GC = 1/|E_τ|∑_(t, t') ∈ E_τ P(I_t, I_t')  - 1/|E_τ^c|∑_(t, t') ∈ E^c_τ P(I_t, I_t').




  
Computing Change Statistics. In order to compute the pairwise statistic P(I_t, I_t'), we explore ways to use the networks D and G introduced in Section <ref>, as well as a standalone, “image-only” network trained specially for the task.
First, we note that by fixing the test caption to the “no change” token, we can define:

    P_D(I_t, I_t') := - D(“no change”, I_t, I_t')

which uses the discriminator's label as a measure of how poorly the “no change” description fits the pair of images I_t and I_t'.

We also define D̃(I_t, I_t'), an “image-only" change detector. D̃(I_t, I_t') learns to produce visual features using the same encoder architecture E_t,t' as the generator in Section <ref>. The visual features are passed through several fully-connected layers, and the output of D̃(I_t, I_t') is a single label, rather than a caption. D̃(I_t, I_t') is trained to classify whether pairs contain a change or not using a binary classification loss, where a label of 1 indicates no change. 
Now we can define:

    P̃(I_t, I_t') := -D̃(I_t, I_t'),

so P̃(I_t, I_t') is an estimate of the probability that a change has occurred between t and t' that has been trained without using language information. We define an “image-only” stepwise score S_ step-IO where P̃ is used for P. Similarly, we also define an “image-only” variant of the graph cut score of S_ GC-IO using P̃.



  
Consistency-regularized graph cut scores.

The above step-wise and graph cut change detection scores depend only on the pairwise relation P(I_t, I_t'). However, we may have access to learned hidden representations h(I_t, I_t') that contain more information relevant to change detection, which may help regularize the above graph cut scores. Specifically, we regularize our score function to help ensure that all the pairs (I_t,I_t') with t ≤τ and t' > τ have similar representations (since they should represent the same change). We measure the similarity between representations of two pairs, h(I_t,I_t') and h(I_s,I_s') using the cosine similarity.

This leads to a regularized “representation consistency” score  of 

    S_RC =   _τλ/|E_τ|∑_(t, t') ∈ E_τ P(I_t, I_t') -λ/|E^c_τ|∑_(t, t') ∈ E^c_τ P(I_t, I_t')
             + ∑_(t, t') ∈ E_τ, (s, s') ∈ E_τcos[h(I_t, I_t'), h(I_s, I_s') ]


for a user-specified tuning parameter λ≥ 0; then τ̂_ RC := _τ S_ RC. The consistency measure in the final term of (<ref>) sums over all pairs of black edges in Figure <ref>, finding the average similarity of the associated hidden representations.

To make the above concrete, we may choose P to be P_D from Equation <ref>, and let the representation of a pair (I_t,I_t') be the final hidden state of the caption generator G(I_t, I_t'). In this formulation, a natural interpretation of the estimator arises: we would like to determine a changepoint τ̂ where for all pairs (I_t,I_t'), where t ≤τ but t' > τ, D(“no change”, I_t, I_t') is low. In addition, the representation consistency pushes us to choose a changepoint where the generator G will output similar descriptions of the change that has occurred across τ for all possible pairs. In this formulation, the language model becomes an integral part of the changepoint detection method.

Absent a language model, we can also define an “image-only representation consistency”. To do this, in Equation <ref> we utilize P̃, and for internal representation h̃(I_t, I_t') we use the penultimate layer of the image pair used by D̃(I_t, I_t'), a configuration that computes S_ RC-IO.



§ DATA


The experiments in Section <ref> consider both (a) an assessment of our proposed system of change captioning and the role of unlabeled data in the performance, and (b) an evaluation of our approach when both annotated and unannotated visual streams are available for change annotation and detection. The datasets used for these experiments are described below.



 §.§ Semi-supervised Change Captioning


CLEVR-Change is a simulated dataset, with a variety of distractor changes and a curated set of “relevant” changes, first presented in <cit.>. Each image consists of colored geometric primitives on a flat, infinite ground plane. The images are generated with the CLEVR engine <cit.>, a standard tool in a variety of visual reasoning tasks.  includes, for each initial frame, a frame in which only nuisance changes have occurred, as well as a frame in which both nuisance and relevant
changes have occurred. 
There were initially roughly 39000 change/distractor pairs in the CLEVR-change dataset, which we have augmented with 10000 additional unlabeled pairs of images by mimicking their generation process. The additional unlabeled pairs always contain a change.

Spot-the-Diff <cit.> is a video surveillance change description dataset which consists of pairs of surveillance images and associated human-generated captions for the visual changes in each pair. Each pair is aligned and assumed to contain a change, with similar lighting conditions between the “before” and “after” frames, so this dataset does not contain the nuisance changes present in the  dataset. Spot-the-Diff contains 13192 image pairs, which we split into 10000 training pairs and 3192 testing pairs.



 §.§ Visual Streams


Due to the lack of standard datasets for our task, we propose two datasets for evaluation of captioning and changepoint detection on visual streams.

The first dataset we introduce is a modification of the  synthetic dataset. By modifying the machinery in <cit.>, we generate a visual stream instead of a pair of images. Over the course of each sequence, the camera follows a random walk from its initial location, while the light intensity also follows a random walk, to ensure that the proposed methodology must be robust to nuisance changes. At some time in the sequence, one of the non-nuisance changes used in the original  dataset occurs. Each sequence contains 10 images, with the changepoints  uniformly distributed in t∈{1,…,8} (assuming zero indexing).  Further details can be found in the supplementary materials. Figure <ref> shows a subsequence drawn from one CLEVR-Sequence stream.

We also propose the “Street Change” dataset, which consists of 151 image sequences captured in an urban environment during different seasons and times of day, extracted from the VL-CMU dataset <cit.>. Each sequence contains multiple views of a particular scene that are captured at an initial date, and then a second subsequence that is captured at a later date. The original change detection dataset was curated by 
<cit.>, and we have utilized human labelers on Amazon's Mechanical Turk to provide change annotations for training. The annotation set includes at least three captions for both the original and time-reversed sequences, which were curated after data collection to ensure high-quality labels.

The “Street Change” dataset contains both minor and major nuisance changes across the dataset. First, from an annotation perspective, each frame is captured at a different viewpoint, and there are often lighting changes across the sequence. Moreover, since some sequences span seasons, environmental changes like plant growth, snowfall, and weather variation are present in the dataset, which any successful trained model must learn to ignore. An example sequence is shown in
<ref>, along with the associated human annotations. More examples are included in the supplementary materials. The average length of each stream is 9 images, the mean sentence length is 6.2 words with standard deviation 1.9, and the maximum sentence length is 16 words. There are 420 unique words in the captions.




§ EXPERIMENTS


In this section we empirically demonstrate the performance of the proposed method on both caption generation and changepoint detection. 
We find that the proposed training regime improves on the state of the art for change captioning, as well as an adapted semi-supervised approach from standard image captioning. 

We find that the RC approach, which leverages both D and the generative language model G, outperforms our other proposed changepoint detection methods. While adding the representation consistency term also improves “Image-Only” approaches, we show that incorporating both D and G results in good performance on language metrics for change captioning, as well as superior change detection performance..


 §.§ Implementation Details
 G is trained as in Phase 1 for 40 epochs using Adam <cit.>, and the discriminator D is also trained for 40 epochs using Adam. Phase 3 of training proceeds for 20 epochs. The learning rate begins at 0.001 and decays by a factor of 0.9 each 5 epochs. λ in L_3 is set to be 0.2.
The image-only discriminator D̃ is trained for 40 epochs using Adam with a learning rate of 0.001.

G is identical in architecture to the network used in <cit.> for all three training sets. The three architectures differ only in the dimension of the final output of G, p_G(w|I_t, I_t'), which is 64, 2404, and 420 for CLEVR, Spot-the-diff, and Street Change respectively. At test time, sampling captions is done in a greedy fashion. 


  
Adapting Show, Tell, and Discriminate to change captioning. An alternative approach to recognizing valid descriptions is proposed in <cit.> for single-image captioning. Their approach, called Show, Tell, and Discriminate (ST&D),  embeds the descriptions w and images I in a common space, and then measures the similarity of the embeddings by measuring the cosine similarity s(w, I) between them. The embedding module is trained with a hard triplet loss on labeled data, of the form: L_ST&D = ∑_i=1^L max_i≠ j(α - s(w^i, I^i) + s(w^i, I^j) )_+
for margin parameter α. 
Training in stage three proceeds similarly to our approach, with L_ST&D providing a form of discriminative reward to G.

We have modified the approach to perform change captioning, so the embedding module operates on pairs of images rather than single images. In Section <ref> we compare the performance of the modified to our approach. The learned joint embedding space is dimension 2048, and the margin used for the triplet loss is 0.1. We follow the authors' lead and set the weighting of the self-retrieval module in stage three to be 1.








  
Evaluation For language tasks, we report BLEU-4 <cit.>, CIDEr <cit.>, ROUGE-L <cit.>, and METEOR <cit.>. To evaluate the quality of change detection methods, we generate precision-recall curves; we report full precision-recall curves in Figure <ref>, as well as summarizing with Average Precision in Table <ref>.



  
Semisupervised Change Captioning We begin by comparing our semisupervised approach to Show, Tell, and Discriminate (ST&D), a method for semisupervised single-image captioning, which does not utilize unlabeled data to train the network that provides feedback to G. Table <ref> explores a range of training set sizes, varying both L and U, the number of labeled and image pairs respectively. Except in the case of extremely scarce labeled data (L=1k), we see consistent improvements from adding unlabeled data; our approach appears to use additional unlabeled data more effectively, overtaking the adapted approach.

Table <ref> compares captioning performance with respect to different labeled and unlabeled training set sizes for both Spot-the-Diff (left) and Street Change (right). Note the training set sizes for Street Change are in terms of sequences. 


  
Change Detection. In Figures <ref> a) and b), we illustrate the performance of all changepoint methods introduced on both CLEVR Sequence and Street Change. For all methods with the “Image Only” prefix “-IO,” we utilize P̃ to calculate scores, and for hidden representations h̃(I_t, I_t') we utilize the penultimate layer of D̃(I_t, I_t'). For methods without the “-IO” prefix, we use P_D, which utilizes the learned change caption discriminator, and the hidden state of the caption generator G as h(I_t, I_t').





To choose λ in S_ RC, a held-out set of 20 sequences from CLEVR Sequence were used. We use λ=5/4 in both CLEVR Sequence and Street Change tests. We also explore setting λ=0 to isolate the performance of the representation consistency loss as a standalone method.

Figure <ref> illustrates the effect of adding both labeled and unlabeled data to the training set of G and D. 

We observe a consistent improvement from image-only methods to their counterparts that use a language-informed D. Graph-cut and representation consistency terms both improve detection accuracy compared to stepwise approaches; combining the two leads to further improvements.

In Table <ref>, we explore  the effect of loosening the requirements for a “correct” change detection. In these cases, we vary a window size parameter Δ, so that a detection is called correct if the absolute error |τ̂ - τ^*| ≤Δ is smaller than Δ. Δ=0 is the default for all other tables and figures. 






§ CONCLUSIONS


In this work, we explore natural language captioning and detection of change in visual streams, illustrating that learning to generate captions to describe change also enhances our ability to accurately detect visual changes over streams. While natural language labels have a strong positive impact on change detection performance, we recognize that labeled training data is often difficult and expensive to obtain. With this challenge in mind, we develop a semi-supervised training regimen that fully leverages unlabeled data. A broad array of experiments on newly developed datasets consisting of visual streams with natural language labels help quantify the performance gains associated with different amounts of unlabeled data on both captioning and detection.


splncs04




In the following sections we provide further information regarding the “Street Change” and CLEVR-Sequence datasets. We provide several example sequences, explain the annotation gathering procedure for Street Change, and provide further information that was not contained in the main document.



§ CLEVR-SEQUENCE


CLEVR-Sequence is an extension of the CLEVR-Change dataset of <cit.>. The CLEVR-Change dataset consists of triplets of images along with annotations: one “original” image, a “change” image, and a “distractor” image. 

CLEVR-Sequence, by contrast, consists of sequences of ten images. The changes between all but one pair in the sequence are “distractor” changes, in which the camera moves and the lighting changes slightly. For one pair, there is, in addition to the distractor change, a non-distractor change, in which one of the possible changes from <cit.> occurs with uniform probability over the change types. For clarity, these are: Color (an object changes color), Texture (the material of a single object changes), Add (a new object appears), Drop (a previously-present object disappears), and Move (a single object moves in the scene).

We generate 400 sequences for each candidate changepoint in the range {1,2,3,4,5,6,7,8 }, assuming 0 indexing. We also generate 400 “distractor” sequences in which no change occurs at any point.

We attempt to imitate the characteristics of distractor changes as outlined by the authors of <cit.> as closely as possible. Annotations are produced automatically for each sequence using the same methodology as the CLEVR-Change dataset.






§ STREET CHANGE


The images in the Street Change dataset were curated by <cit.>, and are grouped into 151 unique image sequences of variable lengths. The minimum sequence length is 2 (this length-2 sequence was omitted from training and testing), and the maximum is 42. As mentioned in the main body, the average sequence length is approximately 9.

In Street Change, the change point is always the center frame of the sequence. For methods which learn by leveraging the entire sequence simultaneously, this may be a problematic bias, but since we learn and infer using only pairwise comparisons, this bias is less concerning.

Captions were gathered using Amazon's Mechanical Turk. Annotators were presented with a pair of images and asked to describe in a single brief sentence the major changes between them, ignoring nuisance changes. Annotators were given guidance on what to consider nuisance changes, like weather, lighting, time of day, and camera angle. Annotations for both the original sequences and time-reversed sequences were gathered, as a form of data augmentation.

The gathered annotations were curated by hand to remove descriptions of nuisance changes, ensure consistent spelling, ensure each annotation was a single sentence, and confirm accuracy of the annotations. We collected six annotations for each sequence (and six more for the time-reversed sequence), to ensure that there would be at least three acceptable annotations for each sequence. 





 §.§ Example Street Change Sequences


Here we illustrate some sample full sequences from the Street Change Dataset, along with the associated ground-truth annotations, and the annotations produced by our method with 100 labeled training sequences and 30 unlabeled training sequences. The displayed images are drawn from the test sequences.  To generate multiple captions, different pairs in the sequence were sampled; the language model is sampled greedily for all pairs.

All images best viewed digitally.







