title: A Self-Reasoning Framework for Anomaly Detection Using Video-Level Labels
authors: Muhammad Zaigham Zaheer Arif Mahmood Hochul Shin Seung-Ik Lee
text: 
  
  
  
  
  
   
  
  
  
  
  
  
  
  [
    \begin@twocolumnfalse
          This is a draft version of the article accepted by the IEEE Signal Processing Letters.
          2020 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. 
   A Self-Reasoning Framework for Anomaly Detection Using Video-Level Labels
      Muhammad Zaigham Zaheer, Arif Mahmood, Hochul Shin and Seung-Ik Lee
  This work is supported by the ICT R&D program of MSIP/IITP. [2017-0-00306, Dvlp. of Multimodal Sensor-based Intll. Sys. for Outdoor Surveillance Robots]. We thank Jin-ha Lee & Marcella Astrid for their dedicated help.
  M. Zaigham Zaheer, Hochul Shin and Seung-Ik Lee are with the Electronics and Telecommunications Research Institute(ETRI) and the University of Science and Technology (UST), Daejeon, Korea. mzz@ust.ac.kr, {creatrix, the_silee}@etri.re.kr.
  Arif Mahmood is with the Information Technology University, Lahore, Pakistan. rfmahmood@gmail.com.
  
  
      Aug. 2020
  ====================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================
  
  
  
   \end@twocolumnfalse
   ]
  
  
   2020 IEEE. IEEE SIGNAL PROCESSING LETTERS, Vol. X, No. Y, July 2020
  Shell et al.: Bare Demo of IEEEtran.cls for IEEE Journals
  
  Anomalous event detection in surveillance videos is a challenging and practical research problem among image and video processing community.
  Compared to the frame-level annotations of anomalous events, obtaining video-level annotations is quite fast and cheap though such high-level labels may contain significant noise. More specifically, an anomalous labeled video may actually contain anomaly only in a short duration while the rest of the video frames may be normal.
  In the current work, we propose a weakly supervised anomaly detection framework based on deep neural networks which is trained in a self-reasoning fashion using only video-level labels. To carry out the self-reasoning based training, we generate pseudo labels by using binary clustering of spatio-temporal video features which helps in mitigating the noise present in the labels of anomalous videos. Our proposed formulation encourages both the main network and the clustering to complement each other in achieving the goal of more accurate anomaly detection. The proposed framework has been evaluated on publicly available real-world anomaly detection datasets including UCF-crime, ShanghaiTech and UCSD Ped2. The experiments demonstrate superiority of our proposed framework over the current state-of-the-art methods.
   
  
  
  Anomalous events detection, self reasoning framework, video understanding, weakly supervised learning
  
  
  
  
  
  
  
  
  
  
  
  § INTRODUCTION
  
  
  Anomalous event detection is a challenging problem in signal, image and video processing areas because of its applications in real-world surveillance systems <cit.>. 
  Infrequent occurrences make the anomalous events to usually appear as outliers from the normal behavior <cit.>. Therefore, anomaly detection has often been carried out using one-class classifiers which learn the frequently occurring events as normal <cit.>. Anomalies are then detected based on their deviations from the learned representations of the normal class. However, it is not usually feasible to collect a complete set of all possible normal scenarios for training, therefore new occurrences of the normal class may also substantially differ from the learned representations and may be misdetected as anomalous <cit.>.
  
  Another widely used anomaly detection approach is to utilize the weakly supervised learning paradigm to train a binary classifier using both the normal and the anomalous data instances <cit.>. In such setting, all events in a normal labeled  video are marked as normal. Whereas in an abnormal labeled video, though some of the events may actually be normal, all events are marked as anomalous resulting in noisy labels. It reduces the efforts required in obtaining detailed manual annotations of the dataset, however the training using such type of labels is quite challenging. 
  In the current work, we propose an approach for anomalous event detection using such video-level labels.
  
  
  
  
  Recently, anomaly detection problem in weakly labeled videos has been formulated as Multiple Instance Learning (MIL) task <cit.>. Each video is divided into a number of fragments such that  each fragment consists of several consecutive frames.
  A bag of fragments is created using a complete set of fragments from a single video. Training of the network is then carried out by defining a ranking loss between two top-scoring fragments, one from the anomalous and the other from the normal bag. However, this approach necessitates to convert each video of the dataset into the same number of fragments which may not always be an appropriate choice. 
  
  Since the real-world datasets contain significantly varying length of videos, fixed number of fragments may not be able to represent events happening over a short span of time in lengthy videos. This problem has been addressed by Zhong <cit.> who proposed an anomaly detection approach using weakly labeled videos. In their approach, training is performed using noisy labels, where the noise refers to normal fragments within anomalous videos. 
  They take advantage of an action classification model <cit.> to train a graph convolution network <cit.>, which then helps in cleaning noisy labels from the anomalous videos. The essence of our proposed approach may be considered similar to theirs because we also attempt to mitigate noisy labels. However, our formulation of the problem is entirely different. In our architecture, instead of relying on action classifiers or graph convolution network, we propose to employ a binary clustering based self-reasoning approach which not only attempts to remove noisy labels but also contributes in enhancing the performance of our base framework through a clustering distance loss. 
  The main contributions of the current work are summarized below:
  
    
    * Our framework trains in a weakly supervised manner using only video-level annotations to localize anomalous events.
   
    * We propose a clustering based self-reasoning approach to clean noise from the labels of anomalous videos. Our approach encourages the FC network to improve clusters, which in return enhances the capability of the network to discriminate anomalous portions of a video. Thus, enabling both the network and the clustering  to complement each other during training.
  
    * Our method outperforms existing SOTA by yielding frame-level AUC of 79.54% on UCF-crime, 84.16% on ShanghaiTech, and 94.47% on UCSD Ped2 datasets.
  
  
  
  
  
  
  
  
  § PROPOSED ARCHITECTURE
  
  The overall proposed framework is visualized in Fig. <ref>, and all of its components are discussed below: 
  
  
  Group of Video Fragments:
  All frames from a video V_i are divided into a group of fragments in such a way that each fragment F_(i,j) contains k non-overlapping frames, where i∈[1,n] is the video index in the dataset of n videos and j∈[1,m_i] is the index of m_i fragments in V_i. For each video, only one binary label {normal = 0, anomalous = 1} is provided.
  
  
  Feature Extractor:
  Spatio-temporal features of each fragment F_(i,j) are computed by employing a pre-trained feature extractor model such as Convolution 3D (C3D) introduced by Tran et al. <cit.>. Our proposed framework is generic and may employ any spatio-temporal feature extractor.
  
  
  Fully Connected Base Network:
  Our base network consists of two fully connected (FC) layers, each followed by a ReLU activation function and a dropout layer, as shown in Fig. <ref>(d). The input layer receives a spatio-temporal feature vector and the output layer produces an anomaly regression score in the range of [0, 1] through a Sigmoid activation function, where 0 represents a normal fragment and 1 represents an anomalous fragment.
  
  
  Clustering Based Self-Reasoning:
  Given that anomaly detection is a binary problem, clustering algorithms such as k-means <cit.> can be employed to distribute all fragments into two clusters. These clusters are created using the feature representations of each fragment taken from the output of FC-1 layer.
  
  
  Clustering here serves two purposes: 1) It helps in generating fragment-level pseudo annotations from video-level labels. 2) It encourages the network to push both clusters away in the case of an anomalous video, and brings both clusters closer in the case of a normal video.
  
  
  
  
  
  
   §.§ Training
  
  As explained previously, our architecture is trained using only video-level labels, therefore we propose a self-reasoning-enabled approach to create pseudo annotations. Moreover, our configuration also encourages FC network and clustering to complement each other towards improving the results throughout training iterations.
  
  Creating Pseudo Annotations:
  For the normal labeled videos, as no anomaly is present, each fragment of these videos can simply be annotated as normal. However, in the case of anomalous videos, several normal fragments may also be present. To handle this, pseudo annotations y^p_(i,j) are generated for the anomalous videos. Overall, for a given video V_i, the fragment-level labels y_(i,j)∈𝐲_(i)∈{0, 1}^m_i for training are given as: 
  
      y_(i,j)=
          
          0,    if V_i is normal
      
          y^p_(i,j),    if V_i is anomalous.
  
  To obtain y^p_(i,j), all fragments from an anomalous video are divided into two clusters assuming one cluster would contain normal while the other would contain anomalous fragments. At this point, one of the two cases may occur: Case 1) cluster A contains most of the normal fragments while cluster B contains most of the anomalous fragments. Case 2) cluster B contains most of the normal fragments while cluster A contains most of the anomalous fragments. It is important to determine the exact case so that the appropriate pseudo-labels may be assigned. In order to do so, a similarity score s_1 is computed between the labels predicted by the FC network, 𝐲̂_(i)∈ [0,1]^m_i, and the labels generated by clustering, 𝐲^c_(i)∈{0,1}^m_i for all fragments of V_i,
  
  
  
  
  
      s_1 =  𝐲̂_(i)·𝐲^c_(i)/ ||𝐲̂_(i)||_2 ||𝐲^c_(i)||_2.
  
  To resolve the ambiguity, another similarity score s_2 is also computed between 𝐲̂_(i) and the inverted clusters' labels 𝐲^c_(i), where  is the logical negation:
  
  
  
  
  
      s_2 = 𝐲̂_(i)·𝐲^c_(i)/||𝐲̂_(i)||_2||𝐲^c_(i)||_2.
  
  
  Finally, given j^th fragment F_(i,j) in V_i, the pseudo-label is given as:
  
      y^p_(i,j)=
          
          y^c_(i,j),    if  s_1 ≥ s_2
       y^c_(i,j),    otherwise,
  
  
  
  Thus, we force the FC network to learn normal fragments in normal videos which are noise-free. For the case of anomalous videos, we find a cluster label configuration which is in accordance with the labels predicted by the FC network. Though, the final fragment labels in an abnormal video are decided by the cluster labels. The pseudo labels thus found are then used for the computation of network loss function.
  
  
  
  
  
   §.§ Training Losses
  
  Overall, our network is trained to minimize the loss: 
  
      L= L_r + λ L_c,
  
  where L_r is given as:
  
      L_r = 1/m_i∑_j=1^m_i ((y_(i,j) - ŷ_(i,j)))^2
  
  and λ is a trade-off hyperparameter.
  Moreover, clustering distance loss, L_c, is defined as:
  
      L_c=
          min(α, d_i),    if V_i is normal
      1/d_i,    if V_i is anomalous,
  
  where d_i = ||C_1 - C_2||_2 is the distance between the cluster centers C_1 and C_2  obtained using k-means algorithm over the fragments of V_i, and α is an upper bound on the distance loss.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  § EXPERIMENTS
  
  
  
   §.§ Datasets
  
  Proposed framework is evaluated on three publicly available anomalous events detection datasets including UCF-crime, ShanghaiTech and UCSD Ped2. UCF-crime <cit.> is a weakly labeled anomalous event detection dataset obtained from real-world surveillance videos. For training, it contains 810 videos of anomalous and 800  of normal classes. For testing, it contains 140 anomalous and 150 normal videos.
  ShanghaiTech <cit.> is also an anomalous event dataset recorded in a university campus. We follow a recent split introduced by Zhong <cit.> containing 63 anomalous and 175 normal videos for training and 44 anomalous and 155 normal videos for testing.  
  UCSD Ped2 <cit.> dataset comprises of 16 normal and 12 anomolous videos. Pedestrians dominate most of the frames whereas anomalies include skateboards, vehicles, bicycles, etc. Following the protocol proposed by He <cit.>, we randomly selected 10 videos for training (4 normal and 6 anomolous) and 18 videos for testing (12 normal and 6 anomolous). To remove the bias in random selection, experiment is repeated five times and the average performance is reported.
  
  
  Following existing SOTA <cit.>, Area Under the Curve (AUC) of the  ROC curve for frame-level performance is used as an evaluation metric. For the training of backbone network, Adam optimizer <cit.> is used with a learning rate of 5 × 10^-5. In Eqs. (<ref>) and (<ref>), λ and α are set to 0.05  and 1 respectively. Our proposed framework is generic and may use several types of feature extractors, however in accordance with the existing techniques, we employ C3D architecture <cit.>. In C3D, the number of frames per fragment, k, is set to 16.
  
  
  
  
  
   §.§ Comparisons with Existing State-of-the-Art Approaches
  
  
  Table <ref> summarizes a comparison of our approach with the existing state-of-the-art methods on all three datasets. For the case of UCF-crime, our approach demonstrates superior performance compared to the methods including Sultani <cit.>, Hasan <cit.>, Lu <cit.>, Zhang <cit.> with a significant margin while comparable results to Zhong <cit.>. For ShanghaiTech dataset, using the same splits and C3D features, our framework outperforms Zhong <cit.> by a significant margin of 7.72% in AUC. For UCSD Ped2 dataset, again our proposed model outperforms most compared methods by a significant margin.
  
  On the average taken over all three reported datasets, our proposed framework obtains 86.06% AUC, while Zhong <cit.> obtains an average performance of 83.57% which demonstrates the overall superiority of our proposed approach. 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
   §.§ Ablation Study
  
  A detailed ablation study on all three datasets including UCF-crime, ShanghaiTech and UCSD Ped2,  is provided in Table <ref>. We follow a top-down approach in which different components of the proposed framework are systematically removed to observe the loss of performance compared to the full system. In the case of ShanghaiTech dataset, removal of clustering distance loss (L_c) resulted in a drop of 0.79% whereas removal of clustering based pseudo annotations, 𝐲^p for anomalous videos, resulted in a drop of  2.51%. Note that when we remove 𝐲^p, the labels of all fragments of an anomalous video are set to 1 in Eq. (<ref>).
  Experiments on UCF-crime and UCSD Ped2 datasets also demonstrated similar trends (Table <ref>).
  
  
  
  
  
   §.§ Qualitative Results
  
  Anomaly score plots of several normal and anomalous test videos from the UCF-crime dataset are visualized in Fig. <ref>. Overall, our network produces distinctive scores for anomalous portions of the videos. For the case of Arrest event (Fig. <ref> (a)), a significant increase in the anomaly score shows the start of the anomalous event. At the end of this event, the anomaly score still retains higher value. It is because even after the arrest has been made, the cars parked on roadside still represent some extent of abnormality. A similar behavior can also be observed in the robbery event (Fig. <ref> (c)) where the annotated part of the event is smaller than the actual event. For the case of burglary (Fig. <ref> (d)), our framework detects quite accurately the start and the end of the anomalous event. For the two normal cases (Figs. <ref> (e) & (f)), our proposed framework outputs significantly low scores, making it easy to discriminate anomalous events from the normal events.
  
  Due to the self-reasoning, our proposed system evolves over iterations on the training videos to produce higher anomaly scores in the abnormal regions despite using only video-level labels. Fig. <ref> shows such evolution of the system on a test video from UCF-crime dataset. As the number of iterations increases, the difference between the anomaly score over normal and anomalous regions also increases. 
  Fig. <ref> shows the reduction in clustering distances loss over consecutive training iterations. It demonstrates that the backbone network is learning to produce better internal representations, causing reduction in inter-cluster distance for the case of normal and increase in this distance for the case of anomalous videos. 
  
  
  
  
  
  
  § CONCLUSION
  
  A weakly-supervised approach is proposed to learn anomalous events using video-level labels.
  
   Compared to using frame-level annotations, the video-level annotations contain significant noise in case of anomalous videos.
  
  
  To this end, binary clustering is employed which enables self-reasoning to mitigate these noisy labels. The proposed framework enables both the FC network and the clustering algorithm to complement each other in improving the quality of results. Our method demonstrates state-of-the-art results by yielding 79.54%,  84.16% and 94.47% frame-level AUC on  UCF-crime, ShanghaiTech and UCSD Ped2 datasets.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
    
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  IEEEtran 
  
   
  