Weakly-Supervised Action Localization by Generative Attention Modeling
Baifeng Shi Qi Dai Yadong Mu Jingdong Wang
cs.CV






Weakly-Supervised Action Localization
by Generative Attention Modeling
    Baifeng Shi^1Work was done during internship at Microsoft.     Qi Dai^2     Yadong Mu^1    Jingdong Wang^2

  ^1Peking University     
  ^2Microsoft Research Asia

  {bfshi,myd}@pku.edu.cn, {qid,jingdw}@microsoft.com

    March 2020
=============================================================================================================================================================================================================================




Weakly-supervised temporal action localization is a problem of learning an action localization model with only video-level action labeling available.
The general framework largely relies on the classification activation, which employs an attention model to identify the action-related frames and then categorizes them into different classes.
Such method results in the action-context confusion issue: context frames near action clips tend to be recognized as action frames themselves, since they are closely related to the specific classes.
To solve the problem, in this paper we propose to model the class-agnostic frame-wise probability conditioned on the frame attention using conditional Variational Auto-Encoder (VAE).
With the observation that the context exhibits notable difference from the action at representation level, a probabilistic model, i.e., conditional VAE, is learned to model the likelihood of each frame given the attention.
By maximizing the conditional probability with respect to the attention, the action and non-action frames are well separated.
Experiments on THUMOS14 and ActivityNet1.2 demonstrate advantage of our method and effectiveness in handling action-context confusion problem. Code is now available on GitHub[https://github.com/bfshi/DGAM-Weakly-Supervised-Action-Localization].










§ INTRODUCTION


Action localization is one of the most challenging tasks in video analytics and understanding <cit.>.
The goal is to predict accurate start and end time stamps of different human actions.
Owing to its wide application (, surveillance <cit.>, video summarization <cit.>, highlight detection <cit.>), action localization has drawn lots of attention in the community.
Thanks to the powerful convolutional neural network (CNN) <cit.>, performance achieved on this task has gone through a phenomenal surge in the past few years <cit.>.
Nevertheless, these fully-supervised methods require temporal annotations of action intervals during training, which is extremely expensive and time-consuming.
Therefore, the task of weakly-supervised action localization (WSAL) has been put forward, where only video-level category labels are available.

To date in the literature, there are two main categories of approaches in WSAL.
The first type <cit.> generally builds a top-down pipeline, which learns a video-level classifier and then obtains frame attention by checking the produced temporal class activation map (TCAM) <cit.>.
Note that a frame indicates a small snippet from which appearance or motion feature could be extracted.
On the other hand, the second paradigm works in a bottom-up way, , temporal attention is directly predicted from raw data <cit.>. Then attention is optimized in the task of video classification with video-level supervision.
Frames with high attention are thus treated as action part, otherwise the background part.




Both kinds of methods largely rely on the video-level classification model, which would lead to the intractable action-context confusion <cit.> issue in the absence of frame-wise labels.
Take the long jump in Figure <ref> as an example, the action has three stages, , approaching, jumping, and landing.
In addition, the frames before and after the action, , preparing and finishing, contain the content that is closely related to long jump, but are not parts of the action.
We refer to such frames as context, which is a special kind of background.
In this example, the context parts include the track field and sandpit, which could in fact significantly encourage the recognition of the action.
Without frame-wise annotations, the classifier is normally learned by aggregating the features of all related frames, where context and action are roughly mixed up.
The context frames thus tend to be easily recognized as action frames themselves.
The action-context confusion problem has not been fully studied though it is common in WSAL.
One recent exploration <cit.> attempts to solve the problem by assuming a strong prior that context clips should be stationary, , no motions in them.
However, such assumption has massive limitations and ignores the inherent difference between context and action.






To separate context and action, the model should be able to capture the underlying discrepancy between them.
Intuitively, context frame indeed exhibits obvious difference from action frame at the appearance or motion level.
For example, among the five stages in Figure <ref>, the action stages (approaching, jumping, and landing) clearly demonstrate more intense body postures than the context stages (preparing and finishing).
In other words, the extracted feature representations for context and action are also different.
Such difference exists regardless of the action category.


Inspired by this observation, we propose a novel generative attention mechanism to model the frame representation conditioned on frame attention. 
In addition to the above intuition, we build a graphical model to theoretically demonstrate that the localization problem is associated with both the conventional classification and the proposed representation modeling.
Our framework thus consists of two parts: the Discriminative and Generative Attention Modeling (DGAM).
On one hand, the discriminative attention modeling trains a classification model on temporally pooled features weighted by the frame attention.
On the other hand, a generative model, , conditional Variational Auto-Encoder (VAE), is learned to model the class-agnostic frame-wise distribution of representation conditioned on attention values.
By maximizing likelihood of the representation, the frame-wise attention is optimized accordingly, leading to well separation of action and context frames.
Extensive experiments are conducted on THUMOS14 <cit.> and ActivityNet1.2 <cit.> to show that DGAM outperforms the state-of-the-arts by a significant margin.
Comprehensive analysis further validates its effectiveness on separating action and context.

The main contribution of this work is the proposed DGAM framework for addressing the issue of action-context confusion in WSAL by modeling the frame representation conditioned on different attentions.
The solution has led to elegant views of how localization is associated with the representation distribution
and how to learn better attentions by modeling the representation,
which have not been discussed in the literature.












§ RELATED WORKS


Video action recognition is a fundamental problem in video analytics.
Most video-related tasks leverage the off-the-shelf action recognition models to extract features for further analysis.
Early methods normally devise hand-crafted features <cit.> for recognition.
Recently, thanks to the development of deep learning techniques, lots of approaches focus on automatic feature extraction with end-to-end learning, , two-stream network <cit.>, temporal segment network (TSN) <cit.>, 3D ConvNet (C3D) <cit.>, Pseudo 3D (P3D) <cit.>, Inflated 3D (I3D) <cit.>.
In our experiments, I3D is utilized for feature extraction.


Fully-supervised action localization has been extensively studied recently.
Many works follow the paradigms that are widely applied in object detection area <cit.> due to their commonalities in problem setting.
To be more specific, there are mainly two directions, namely two-stage method and one-stage method.
Two-stage methods <cit.> first generate action proposals and then classify them with further refinement on temporal boundaries.
One-stage methods <cit.> instead predict action category and location directly from raw data.
In fully-supervised setting, the action-context confusion could be alleviated with frame-wise annotations.

Weakly-supervised action localization is drawing increasing attention due to the time-consuming manual labeling in fully-supervised setting.
As introduced in Section <ref>, WSAL methods can be grouped into two categories, namely top-down and bottom-up methods.
In top-down pipeline (UntrimmedNet <cit.>), video-level classification model is learned first, and then frames with high classification activation are selected as action locations.
W-TALC <cit.> and 3C-Net <cit.> also force foreground features from the same class to be similar, otherwise dissimilar.
Unlike top-down scheme, the bottom-up methods directly produce the attention for each frame from data, and train a classification model with the features weighted by attention.
Based on this paradigm, STPN <cit.> further adds a regularization term to encourage the sparsity of action.
AutoLoc <cit.> proposes the Outer-Inner-Contrastive (OIC) loss by assuming that a complete action clip should look different from its neighbours.
MAAN <cit.> proposes to suppress dominance of the most salient action frames and retrieve less salient ones.
Nguyen  <cit.> propose to penalize the discriminative capacity of background, which is also utilized in our classification module.
Besides, a video-level clustering loss is applied in <cit.> to separate foreground and background.
Nevertheless, all of the aforementioned methods ignore the challenging action-context confusion issue caused by the absence of frame-wise label.
Though Liu  <cit.> try to separate action and context using hard negative mining, their method is based on the strong assumption that context clips should be stationary, which has many limitations and may hence cause negative influence on the prediction.

Generative model has also experienced a fast development in recent years <cit.>.
GAN <cit.> employs a generator to approximate real data distribution by the adversarial training between generator and discriminator.
However, the learned approximating distribution is implicitly determined by generator and thus cannot be analytically expressed.
VAE <cit.> approximates the real distribution by optimizing the variational lower bound on the marginal likelihood of data.
Given a latent code, the conditional distribution is explicitly modeled as a Gaussian distribution, hence data distribution can be analytically expressed by sampling latent vectors and calculating the Gaussian. 
Flow-based model <cit.> uses invertible layers as the generative mapping, where data distribution can be calculated given the Jacobian of each layer.
However, all layers must have the same dimensions, which is much less flexible.
In our work, we exploit Conditional VAE (CVAE) <cit.> to model the frame feature distribution conditioned on attention value.




§ METHOD




Suppose we have a set of training videos and the corresponding video-level labels.
For each video, we sample T frames (snippets) to extract the RGB or optical flow features 𝐗 = (𝐱_t)_t = 1^T with a pre-trained model, where 𝐱_t ∈ℝ^d is the feature of frame t, and d is feature dimension.
The video-level label is denoted as y ∈{0, 1, ⋯, C}, where C is the number of classes and 0 corresponds to background.
For brevity, we assume that each video only belongs to one class, though the following discussion can also apply to multi-label videos.

Our method follows the bottom-up pipeline for WSAL, which learns the attention λ = (λ_t)_t = 1^T directly from data, where λ_t ∈ [0, 1] is the attention of frame t.
Before discussing the details of our method, we examine the action localization problem from the beginning.





 §.§ Attention-based Framework


In attention-based action localization problem, the target is to predict the frame attention λ, which is equivalent to solving the maximum a posteriori (MAP) problem:

    max_λ_t ∈ [0, 1]log p(λ | 𝐗, y),

where p(λ | 𝐗, y) is the unknown probability distribution of λ given 𝐗 and y.
In the absence of frame-level labels (ground truth of λ), it is difficult to approximate and optimize p(λ|𝐗, y) directly.
Therefore, we transform the optimization target using Bayes' theorem,

    log p(λ | 𝐗, y)    = log p(𝐗, y | λ) + log p(λ) - log p(𝐗, y) 
       = log p(y | 𝐗, λ) + log p(𝐗 | λ) + log p(λ) 
          - log p(𝐗, y) 
       ∝log p(y | 𝐗, λ) + log p(𝐗 | λ),

where in the last step, we discard the constant term log p(𝐗, y) and assume a uniform prior of λ, , p(λ) = const. Our optimization problem thus becomes

    max_λ∈ [0, 1]log p(y | 𝐗, λ) + log p(𝐗 | λ).

This formulation indicates two different aspects for optimizing λ.
The first term log p(y | 𝐗, λ) prefers λ with high discriminative capacity for action classification, which is the main optimization target in previous works.
In contrast, the second term log p(𝐗 | λ) forces the representation of frames to be accurately predicted from the attention λ.
Given the feature difference between foreground and background, this objective encourages the model to impose different attentions on different features.
In specific, we exploit a generative model to approximate p(𝐗 | λ), and force the feature 𝐗 to be accurately reconstructed by the model.

Figure <ref> shows the graphical model of the above problem. The model parameters (θ,ψ,ϕ) and the latent variables in generative model (𝐳_t) will be discussed later.
Based on (<ref>), the framework of our method consists of two components, , the discriminative attention modeling and the generative attention modeling, as illustrated in Figure <ref>.








 §.§ Discriminative Attention Modeling


The discriminative attention module learns the frame attention by optimizing the video-level recognition task.
In specific, we utilize attention λ as weight to perform temporal average pooling over all frames in the video and produce a video-level foreground feature 𝐱_fg∈ℝ^d given by

    𝐱_fg = ∑_t = 1^Tλ_t 𝐱_t/∑_t = 1^Tλ_t.

Similarly, we can also utilize 1 - λ as the weight to calculate a background feature 𝐱_bg:

    𝐱_bg = ∑_t = 1^T (1 - λ_t) 𝐱_t/∑_t = 1^T (1 - λ_t).

To optimize λ, we encourage high discriminative capability of the foreground feature 𝐱_fg and simultaneously punish any discriminative capability of the background feature 𝐱_bg <cit.>.
This is equivalent to minimizing the following discriminative loss (softmax loss):

    ℒ_d = ℒ_fg + α·ℒ_bg =-log p_θ (y | 𝐱_fg) - α·log p_θ (0 | 𝐱_bg),

where α is a hyper-parameter, and p_θ is our classification module modeled by a fully-connected layer with weight 𝐰_c ∈ℝ^d for each class c and a following softmax layer. During training, attention module and classification module are jointly optimized.
The graphical model of this part is illustrated in Figure <ref> with dash-dot lines.




 §.§ Generative Attention Modeling


The discriminative attention optimization generally has difficulty in separating context and foreground when frame-wise annotations are unavailable.
Based on the observation that context differs from foreground in terms of feature representation, we utilize a Conditional Variational Auto-Encoder (CVAE) to model the representation distribution of different frames.
Before explaining the details, we briefly review the Variational Auto-Encoder (VAE).

Given the observed variable 𝐱, VAE <cit.> introduces a latent variable 𝐳, and aims to generate 𝐱 from 𝐳, i.e.,

    p_ψ(𝐱)=𝔼_p_ψ(𝐳) [p_ψ(𝐱 | 𝐳)],

where ψ denotes the parameters of generative model, p_ψ(𝐳) is the prior (a standard Gaussian),
and p_ψ(𝐱 | 𝐳) is the conditional distribution indicating the generation procedure, which is typically estimated with a neural network f_ψ(·) that is referred to as decoder.
The key idea behind is to sample values of 𝐳 that are likely to produce 𝐱, which means that we need an approximation q_ϕ(𝐳|𝐱) to the intractable posterior p(𝐳|𝐱).
ϕ denotes the parameters of approximation model, and q_ϕ(𝐳|𝐱) is also estimated via a neural network f_ϕ(·), which is referred to as encoder.
VAE incorporates encoder q_ϕ(𝐳|𝐱) and decoder p_ψ(𝐱 | 𝐳), and learns parameters by maximizing the variational lower bound:

    𝒥_VAE   = -KL(q_ϕ(𝐳|𝐱)||p_ψ(𝐳))+𝔼_q_ϕ(𝐳|𝐱)[log p_ψ(𝐱 | 𝐳)],

where KL(q||p) is the KL divergence of p from q.

In our DGAM model, we expect to generate the observation 𝐗 based on the attention λ, , p(𝐗 | λ), which can be written as p(𝐗 | λ) = Π_t = 1^T p(𝐱_t | λ_t) by assuming independence between frames in a video.
Similarly, we introduce a latent variable 𝐳_t, and attempt to generate each 𝐱_t from 𝐳_t and λ_t, which forms a Conditional VAE problem:

    p_ψ(𝐱_t|λ_t)=𝔼_p_ψ(𝐳_t|λ_t) [p_ψ(𝐱_t |λ_t, 𝐳_t)].

Note that the desired distribution of 𝐱_t is modeled as a Gaussian, , p_ψ(𝐱_t |λ_t, 𝐳_t)=𝒩(𝐱_t | f_ψ(λ_t,𝐳_t),σ^2*𝐈), where f_ψ(·) is the decoder, σ is a hyper-parameter, and 𝐈 is the unit matrix.
Ideally, 𝐳_t is sampled from the prior p_ψ(𝐳_t|λ_t).
In DGAM, we set the prior as a Gaussian, , p_ψ (𝐳_t | λ_t)=𝒩(𝐳_t|r λ_t ·1, 𝐈), where 1 is all-ones vector and r is a hyper-parameter indicating the discrepancy between priors of different attention value λ_t.
When r = 0, prior p_ψ (𝐳_t | λ_t) is independent of λ_t.


During training of CVAE, we also approximate the intractable posterior p(𝐳_t | 𝐱_t, λ_t) by a Gaussian q_ϕ(𝐳_t | 𝐱_t, λ_t)=𝒩(𝐳_t|μ_ϕ,Σ_ϕ), where μ_ϕ and Σ_ϕ are the outputs of the encoder f_ϕ(𝐱_t,λ_t).
We then minimize the variational loss ℒ_CVAE:

    ℒ_CVAE   = -𝔼_q_ϕ(𝐳_t | 𝐱_t, λ_t)log p_ψ (𝐱_t | λ_t, 𝐳_t) 
          +  β· KL(q_ϕ (𝐳_t | 𝐱_t, λ_t) || p_ψ (𝐳_t | λ_t))
       ≃ -1/L∑_l = 1^Llog p_ψ (𝐱_t | λ_t, 𝐳_t^(l)) 
          +  β· KL(q_ϕ (𝐳_t | 𝐱_t, λ_t) || p_ψ (𝐳_t | λ_t)),

where 𝐳_t^(l) is l-th sample from q_ϕ(𝐳_t | 𝐱_t, λ_t).
Note that the Monte Carlo estimation of the expectation 𝔼_q_ϕ(𝐳_t | 𝐱_t, λ_t)(·) is employed
with L samples.
β is a hyper-parameter for trade-off between reconstruction quality and sampling accuracy.


For the generative attention modeling of λ, we fix CVAE and minimize the reconstruction loss L_re given by

    ℒ_re
           = -∑_t = 1^Tlog{𝔼_p_ψ(𝐳_t | λ_t) [p_ψ (𝐱_t | λ_t, 𝐳_t)] }
       ≃ -∑_t = 1^Tlog{1/L∑_l = 1^L p_ψ (𝐱_t | λ_t, 𝐳_t^(l)) },

where 𝐳_t^(l) is sampled from the prior p_ψ(𝐳_t|λ_t).
In our experiments, L is set to 1, and (<ref>) can be written as

    ℒ_re = -∑_t = 1^Tlog p_ψ (𝐱_t | λ_t, 𝐳_t) ∝∑_t = 1^T ||𝐱_t-f_ψ(λ_t,𝐳_t)||^2.

The graphical model of generative attention modeling is illustrated in Figure <ref> with solid and dashed lines.


In our framework, the CVAE cannot be directly and solely optimized due to the unavailability of ground truth λ_t.
Therefore, we propose to train attention module and CVAE in an alternating way, , we first update CVAE with “pseudo label” of λ_t given by the attention module, and then train attention module with fixed CVAE.
The two stages are repeated for several iterations.
Since there exist other loss terms for attention modeling (ℒ_d), the pseudo label can be high-quality and hence a good convergence can be reached.
Experimental results empirically validate it.




 §.§ Optimization


In addition to the above objectives, we exploit a self-guided regularization <cit.> to further refine the attention.
The temporal class activation maps (TCAM) <cit.> are utilized to produce the top-down, class-aware attention maps.
In specific, given a video with label y, the TCAM are computed by

    λ̂_t^fg   = G(σ_s) ∗exp^𝐰_y^T𝐱_t/∑_c = 0^Cexp^𝐰_c^T𝐱_t, 
    λ̂_t^bg   = G(σ_s) ∗∑_c = 1^Cexp^𝐰_c^T𝐱_t/∑_c = 0^Cexp^𝐰_c^T𝐱_t,

where 𝐰_c indicates the parameters of the classification module for class c. λ̂_t^fg and λ̂_t^bg are foreground and background TCAM, respectively.
G(σ_s) is a Gaussian smooth filter with standard deviation σ_s, and ∗ represents convolution.
The generated λ̂_t^fg and λ̂_t^bg are expected to be consistent with the bottom-up, class-agnostic attention λ, hence the loss ℒ_guide can be formulated as

    ℒ_guide = 1/T∑_t = 1^T |λ_t - λ̂_t^fg| + |λ_t - λ̂_t^bg|.



To sum up, we optimize the whole framework by alternately executing the following two steps:

    
  * Update attention and classification modules with loss
    
    ℒ = ℒ_d + γ_1 ℒ_re + γ_2 ℒ_guide,

    where γ_1,γ_2 denote the hyper-parameters.
    
  * Update CVAE with loss ℒ_CVAE.

The whole architecture is illustrated in Figure <ref>.






 §.§ Action Prediction


To generate action proposals for a video during inference, we feed the video to DGAM and obtain the attention λ = (λ_t)_t = 1^T.
By filtering out frames with attention lower than a threshold t_att, we extract consecutive segments with high attention values as the predicted locations.
For each segment [t_s, t_e], we temporally pool the features with attention, and get the classification score s(t_s, t_e, c) for class c, which is the output of classification module before softmax.
We further follow <cit.> to refine s(t_s, t_e, c) by subtracting the score of its surroundings. The final score s^∗(t_s, t_e, c) is calculated by

    s^∗ (t_s, t_e, c)    = s(t_s, t_e, c) - η· s(t_s - t_e - t_s/4, t_s, c) 
          - η· s(t_e, t_e + t_e - t_s/4, c),

where η is the subtraction parameter.






§ EXPERIMENTS




 §.§ Datasets and Evaluation Metrics


For evaluation, we conduct experiments on two benchmarks, THUMOS14 <cit.> and ActivityNet1.2 <cit.>. During training, only video-level category labels are available.

THUMOS14 contains videos from 20 classes for action localization task.
We follow the convention to train on validation set with 200 videos and evaluate on test set with 212 videos.
Note that we exclude the wrongly annotated video#270 from test set, following <cit.>.
This dataset is challenging for its finely annotated action instances.
Each video contains 15.5 action clips on average.
Length of action instance varies widely, from a few seconds to minutes.
Video length also ranges from a few seconds to 26 minutes, with an average of around 3 minutes.
Compared to other large-scale datasets, , ActivityNet1.2, THUMOS14 has less training data which indicates higher requirement of model's generalization ability and robustness.

ActivtyNet1.2 contains 100 classes of videos with both video-level labels and temporal annotations.
Each video contains 1.5 action instances on average. Following <cit.>, we train our model on training set with 4819 videos and evaluate on validation set with 2383 videos.

Evaluation Metrics. We follow the standard evaluation protocol and report mean Average Precision (mAP) at different intersection over union (IoU) thresholds.
The results are calculated using the benchmark code provided by ActivityNet official codebase[https://github.com/activitynet/ActivityNet/tree/master/Evaluation]. For fair comparison, all results on THUMOS14 are averaged over five runs.





 §.§ Implementation Details


We utilize I3D <cit.> network pre-trained on Kinetics <cit.> as the feature extractor[https://github.com/deepmind/kinetics-i3d].
In specific, we first extract optical flow from RGB data using TV-L1 algorithm <cit.>.
Then we divide both streams into non-overlapping 16-frame snippets and send them into the pre-trained I3D network to obtain two 1024-dimension feature frames for each snippet.
We train separate DGAMs for RGB and flow streams. The proposals from them are combined with Non-Maximum Suppression (NMS) during inference.
Following <cit.>, we set T to 400 for all videos during training.
During evaluation, we feed all frames of each video to our network if the frame number is less than T_max, otherwise we sample T_max frames uniformly.
T_max is 400 for THUMOS14, and 200 for ActivityNet1.2.




We set α = 0.03 in Eq. (<ref>) and β = 0.1 in Eq. (<ref>). In Eq. (<ref>), we set γ_1 to 0.5 for RGB stream, and 0.3 for flow stream. γ_2 is set as 0.1.
The whole architecture is implemented with PyTorch <cit.> and trained on single NVIDIA Tesla M40 GPU using Adam optimizer <cit.> with learning rate of 10^-3.
To stabilize the training of DGAM, we leverage a warm-up strategy in the first 300 epochs when updating ℒ_CVAE and ℒ_re.












 §.§ Statistical Evaluation on Attention


We first evaluate the learned attention of DGAM and its effectiveness on handling action-context confusion.
For comparison, an “old” model is trained by removing the generative attention modeling (GAM) from DGAM, and our DGAM is denoted as the “new” model.
Note that only Attention and Classification modules are involved during inference.
When evaluating, we assemble specific models by alternately choosing the two modules from “old” or “new” models.
Table <ref> details the mAP results on THUMOS14.
It can be found that the new attention module largely improves the performance, while there is little or no improvement with the new classification module.
This observation indicates that DGAM indeed learns better attention values. Even with “old” classifier, the “new” attention can boost the localization significantly.

We further collect several statistics to show the improvement intuitively in Table <ref>.
Experiments are conducted on both “old" (w/o GAM) and “new" (w/ GAM) models.
In particular, att (cls) indicates the set of frames with attention values (classification scores) larger than a threshold t^*=0.5, and gt is the set of ground truth frames.
|·| represents size of a set. `a-b', `a ∩ b' and `a' indicate set exclusion, intersection and complement, separately. Though such simple thresholding is not exactly the predicted locations, it somewhat reflects the quality of localization.

In Table <ref>, |att-gt|/|gt| or |gt-att|/|gt| indicates the percentage of frames falsely captured or omitted by attention.
It shows that both false activation and omission can be reduced with GAM.
Moreover, an improvement in |(cls-gt) ∩att| / |gt| demonstrates that GAM can better filter out the false positives (context frames) made by classifier.
|(att ∩ gt) - cls|/|gt| measures how attention can capture the false negatives, , action frames neglected by classifier.
Since GAM is devised for excluding the false positives produced by classifier, it is not surprising that GAM contributes little to it.











 §.§ Ablation Studies


Next we study how each component in DGAM influences the overall performance.
We start with the basic model that directly optimizes the attention based foreground classification loss ℒ_fg. The background classification loss ℒ_bg, the self-guided regularization loss ℒ_guide, and the feature reconstruction loss ℒ_re are further included step by step. Note that adding ℒ_re indicates involving the generative attention modeling, where ℒ_CVAE is also optimized.

Table <ref> summarizes the performance by considering one more factor at each stage on THUMOS14.
Background classification is a general approach for both video recognition and localization. In our case, it is part of our discriminative attention modeling, which brings a performance gain of 3.3%.
Self-guided regularization is the additional optimization of our system, which leads to 1.9% mAP improvement.
Our generative attention modeling further contributes a significant increase of 2.1% and the performance of DGAM finally reaches 28.8%.













 §.§ Evaluation on Parameters


To further understand the proposed model, we conduct evaluations to analyze the impact of different parameter settings in DGAM. mAP@0.5 on THUMOS14 is reported.





Discrepancy between latent prior of different λ_t.
In generative attention modeling, different attentions λ_t correspond to different feature distributions p_ψ (𝐱_t | λ_t).
The discrepancy between these distributions can be implicitly modeled by the
discrepancy between latent codes 𝐳_t sampled from different priors, which are modeled as different Gaussian distributions p_ψ (𝐳_t | λ_t)=𝒩(𝐳_t|r λ_t ·1, 𝐈).
Here r controls the discrepancy.
We evaluate r every 0.25 from 0 to 1.5, and the results are shown in Figure <ref>.
In general, the performance is relatively stable with small fluctuation, demonstrating the robustness of DGAM.





Dimension of latent space.
The dimension of latent space in CVAE is crucial for quality of reconstruction and complexity of modeled distribution.
High dimension can facilitate the approximation of feature distribution, hence leading to more accurate attention learning.
However, more training data is also required.
We evaluate different dimensions of 2^n, n = 4, 5, ⋯, 9. As shown in Table <ref>, mAP improves rapidly with increasing dimension, which indicates better generative attention modeling.
The result reaches the peak at dimension 2^7 = 128. After that, the performance starts dropping, partially because of the sparsity of limited data in high-dimensional latent space.


Reconstruction-sampling trade-off in CVAE. The hyper-parameter β in Eq. (<ref>) balances reconstruction quality (the first term) and sampling accuracy (the second term).
With larger β, we expect the approximated posterior to be closer to the prior, which improves the precision when sampling latent vectors from prior, while the reconstruction quality (the quality of learned distribution) will decrease.
We test different β from 0 to 1. As shown in Table <ref>, the performance fluctuates in a small range from 28% to 28.8%, indicating that our method is insensitive to β.




 §.§ Comparisons with State-of-the-Art


Table <ref> compares our DGAM with existing approaches in both weakly-supervised and fully-supervised action localization on THUMOS14.
Our method outperforms other weakly-supervised methods, especially at high IoU threshold, which means DGAM could produce finer and more precise predictions.
Compared with state of the art, DGAM improves mAP at IoU=0.5 by 2%.
Note that Nguyen  <cit.> achieves better performance at IoU=0.1 and 0.2 than our model, partially because our generative attention modeling may discard out-of-distribution hard candidates (outliers), which become common when IoU is low.
Furthermore, our results are comparable with several fully-supervised methods, indicating the effectiveness of the proposed DGAM.



On ActivityNet1.2, we summarize the performance comparisons in Table <ref>.
Our method significantly outperforms the state-of-the-arts.
Particularly, DGAM surpasses the best competitor by 2% on mAP@AVG.
Our method also demonstrates comparable results to fully-supervised methods.








§ CONCLUSION


We have presented a novel Discriminative and Generative Attention Modeling (DGAM) method to solve the action-context confusion issue in weakly-supervised action localization.
Particularly, we study the problem of modeling frame-wise attention based on the distribution of frame features.
With the observation that context feature obviously differs from action feature, we devise a conditional variation auto-encoder (CVAE) to construct different feature distributions conditioned on different attentions.
The learned CVAE in turn refines the desired frame-wise attention according to their features.
Experiments conducted on two benchmarks, , THUMOS14 and ActivityNet1.2, validate our method and analysis. More remarkably, we achieve the new state-of-the-art results on both datasets.

Acknowledgements  This work is supported by Beijing Municipal Commission of Science and Technology under Grant Z181100008918005, National Natural Science Foundation of China (NSFC) under Grant 61772037. Baifeng Shi thanks Prof. Tingting Jiang and Daochang Liu for enlightening discussions.





ieee_fullname



