title: Cloze Test Helps: Effective Video Anomaly Detection via Learning to Complete Video Events
authors: Guang Yu Siqi Wang Zhiping Cai En Zhu Chuanfu Xu Jianping Yin Marius Kloft
text: 
  	
  	
  	
  
  
  
  
  
  
  
  
  Authors contributed equally to this work.
  
  [1]
  
  	National University of Defense Technology
  
  yuguangnudt, wangsiqi10c@gmail.com
  
  
  
  	National University of Defense Technology
  
  zpcai@nudt.edu.cn
  
  
  
  	National University of Defense Technology
  
  enzhu@nudt.edu.cn
  
  
  
  	National University of Defense Technology
  
  xuchuanfu@nudt.edu.cn
  
  
  
  	Dongguan University of Technology
  
  jpyin@dgut.edu.cn
  
  
  
  	TU Kaiserslautern
  
  kloft@cs.uni-kl.de
  
  
  
  
  
  
  
  
  
  
  
  
  
  	As a vital topic in media content interpretation, video anomaly detection (VAD) has made fruitful progress via deep neural network (DNN). However, existing methods usually follow a reconstruction or frame prediction routine. They suffer from two gaps: (1) They cannot localize video activities in a both precise and comprehensive manner. (2) They lack sufficient abilities to utilize high-level semantics and temporal context information. Inspired by frequently-used cloze test in language study, we propose a brand-new VAD solution named Video Event Completion (VEC) to bridge gaps above: First, we propose a novel pipeline to achieve both precise and comprehensive enclosure of video activities. Appearance and motion are exploited as mutually complimentary cues to localize regions of interest (RoIs). A normalized spatio-temporal cube (STC) is built from each RoI as a video event, which lays the foundation of VEC and serves as a basic processing unit. Second, we encourage DNN to capture high-level semantics by solving a visual cloze test. To build such a visual cloze test, a certain patch of STC is erased to yield an incomplete event (IE). The DNN learns to restore the original video event from the IE by inferring the missing patch. Third, to incorporate richer motion dynamics, another DNN is trained to infer erased patches' optical flow. Finally, two ensemble strategies using different types of IE and modalities are proposed to boost VAD performance, so as to fully exploit the temporal context and modality information for VAD. VEC can consistently outperform state-of-the-art methods by a notable margin (typically 1.5%–5% AUROC) on commonly-used VAD benchmarks. Our codes and results can be verified at <github.com/yuguangnudt/VEC_VAD>.
  
  
  
  
  
  
  
  	<ccs2012>
  	<concept>
  	<concept_id>10010147.10010178.10010224.10010225.10011295</concept_id>
  	<concept_desc>Computing methodologies Scene anomaly detection</concept_desc>
  	<concept_significance>500</concept_significance>
  	</concept>
  	<concept>
  	<concept_id>10010147.10010257.10010258.10010260</concept_id>
  	<concept_desc>Computing methodologies Unsupervised learning</concept_desc>
  	<concept_significance>500</concept_significance>
  	</concept>
  	</ccs2012>
  
  
  [500]Computing methodologies Scene anomaly detection
  [500]Computing methodologies Unsupervised learning
  
  
  
  
  
  
  
  
  
  
  Cloze Test Helps: Effective Video Anomaly Detection via Learning to Complete Video Events
      Marius Kloft
      
  =========================================================================================
  
  
  
  
  
  § INTRODUCTION
  
  
  
  
  
  Videos play a key role in multimedia. Video anomaly detection (VAD), which performs anomaly detection by interpreting the video content automatically, has been appealing to both academia and industry, since it is valuable to various safety-critical scenarios like municipal and traffic management. Formally, VAD refers to detecting video activities that divert significantly from the observed normal routine. Despite many efforts, VAD remains challenging for two features of anomaly <cit.>: (1) Scarcity. As anomalies are usually rare, collecting real anomalies for training is often hard or even impossible. (2) Ambiguity. The anomaly does not possess fixed semantics and may refer to different activities based on different context, so it can be highly variable and unpredictable. Such features render modeling anomalies directly unrealistic. Thus, VAD usually adopts an one-class classification setup <cit.>. This setup collects training videos with only normal activities, which are much more accessible than anomalies, to build a normality model. Activities that do not conform to this model are then viewed as anomalies. As all training data are normal, discriminative supervised learning is usually not applicable. Instead, the unsupervised/self-supervised learning has been the commonly-used scheme in VAD. Following such a scheme, existing VAD solutions fall into two categories: (1) Classic VAD, which requires domain knowledge to design hand-crafted descriptors to depict high-level features (e.g. trajectory, speed) or low-level features (e.g. gradient, texture) of video activities. Extracted features are fed into classic anomaly detection methods like one-class support vector machine (OCSVM) to spot anomalies. Feature engineering of classic VAD can be labor-intensive and sub-optimal <cit.>, and designed descriptors are often hard to transfer among different scenes. (2) Deep neural network (DNN) based VAD, which is inspired by DNN's success in traditional vision tasks <cit.>. Due to DNNs' strong capabilities in feature learning and activity modeling <cit.>, DNN based VAD achieves superior performance and enjoys surging popularity when compared with classic VAD.
  
  Despite the fruitful progress, DNN based VAD still suffers from two gaps. Gap #1: Existing DNN based VAD methods cannot localize video activities in a both precise and comprehensive manner. A standard practice in VAD is to use a sliding window with motion filtering <cit.>, but such localization is obviously imprecise. Recent DNN based VAD methods like <cit.> simply ignore this issue by learning on the whole frame, but this suffers from scale variations incurred by image depth and foreground-background imbalance problem <cit.>. Few studies <cit.> improve localization precision by a pre-trained object detector, but it causes a “closed world” problem–The detector only recognize objects in training data and tends to omit video novelties, which leads to non-comprehensive localization results. Such a gap degrades later video activitiy modeling; Gap #2: Existing DNN based methods lack sufficient abilities to exploit high-level semantics and temporal context in video activities. As illustrated by Fig. <ref>, two paradigms (reconstruction and frame prediction) dominate DNN based VAD in the literature: Reconstruction based methods learn to reconstruct inputs and detect poorly reconstructed data as anomalies. However, in this case DNNs tend to memorize low-level details rather than learning high-level semantics <cit.>, and they even reconstruct anomalies well due to overly strong modeling power <cit.>; Frame prediction based methods learn to predict a normal video frame from previous video frames, and detect poorly predicted frames as anomalies. Prediction makes it hard to simply memorize details for reducing training loss, but it scores anomalies by the prediction error of a single frame, which overlooks the temporal context. Thus, neither reconstruction nor frame prediction provides a perfect solution. Unlike recent research that focuses on exploring better network architectures to improve reconstruction or frame prediction, we are inspired by cloze test in language study and mitigate gaps above by proposing Video Event Completion (VEC) as a new DNN based VAD solution (see Fig. <ref>). Our contributions are summarized below:
  
  
  	
    * VEC for the first time combines both appearance and motion cues to localize video activities and extract video events. It overcomes the “closed world” problem and enables both precise and comprehensive video activity enclosure, and it lays a firm foundation for video event modeling in VEC.
  	
  	
    * VEC for the first time designs visual cloze tests as a new learning paradigm, which trains DNNs to complete the erased patches of incomplete video events, to substitute frequently-used reconstruction or frame prediction based methods.
  	
  	
    * VEC also learns to complete the erased patches' optical flow, so as to integrate richer information of motion dynamics.
  	
  	
    * VEC utilizes two ensemble strategies to fuse detection results yielded by different types of incomplete events and data modalities, which can further boost VAD performance.
  
  
  
  
  § RELATED WORK
  
  
    Classic VAD.
  Classic VAD usually consists of two stages: Feature extraction by hand-crafted descriptors and anomaly detection by classic machine learning methods. As to feature extraction, early VAD methods usually adopt tracking <cit.> to extract high-level features like motion trajectory <cit.> and destination <cit.>. However, they are hardly applicable to crowded scenes <cit.>. To this end, low-level features are extensively studied for VAD, such as dynamic texture <cit.>, histogram of optical flow <cit.>, spatio-temporal gradients <cit.>, 3D SIFT <cit.>, etc. Afterwards, various classic machine learning methods are explored to perform anomaly detection, such as probabilistic models <cit.>, sparse coding and its variants <cit.>, one-class classifier <cit.>, sociology inspired models <cit.>. However, feature extraction is the major bottleneck for classic VAD: Manual feature engineering is complicated and labor-intensive, while the designed descriptors often suffer from limited discriminative power and poor transferability among different scenes.
  
  DNN Based VAD.
  DNN based VAD differs from classic VAD by learning features automatically from raw inputs with DNNs. The learned features are fed into a classic model or embedded into DNNs for end-to-end VAD. With only normal videos for training, existing DNN based VAD basically falls into a reconstruction or frame prediction routine. They are reviewed respectively below: (1) Reconstruction based methods learn to reconstruct inputs from normal training videos, and assume that a large reconstruction error signifies the anomaly. Autoencoder (AE) and its variants are the most popular DNNs to perform reconstruction. For example, <cit.> pioneers DNN based VAD by introducing stacked denoising AE (SDAE) and propose its improvement <cit.>; <cit.> adopts convolutional AE (CAE) that are more suitable for modeling videos, while recent works explore numerous CAE variants such as Winner-take-all CAE (WTA-CAE) <cit.> and Long Short Term Memory based CAE (ConvLSTM-AE) <cit.>; <cit.> integrates variational AE into a two-stream recurrent framework (R-VAE) to realize VAD; <cit.> equips AE with a parametric density estimator (PDE-AE) for anomaly detection; <cit.> propose a memory-augmented AE (Mem-AE) to make AE's reconstruction error more discriminative. In addition to AE, other types of DNNs are also used for the reconstruction purpose, e.g. sparse coding based recurrent neural network (SRNN) <cit.> and generative adversarial network (GAN) <cit.>. Cross-modality reconstruction is also shown to produce good VAD performance by learning the appearance-motion correspondence (AM-CORR) <cit.>. (2) Frame prediction based methods learn to predict current frames by several previous frames, while a poor prediction is viewed as abnormal. <cit.> for the first time formulates frame prediction as an independent VAD method, and imposes appearance and motion constraints for prediction quality. <cit.> improves frame prediction by using a convolutional variational RNN (Conv-VRNN). However, prediction by a per-frame basis leads to a bias to background <cit.>, and <cit.> proposes attention mechanism to ease the issue. Another natural idea is to combine prediction with reconstruction as a hybrid VAD solution: <cit.> design a spatio-temporal CAE (ST-CAE), in which an encoder is followed by two decoders for reconstruction and prediction purpose respectively; <cit.> reconstructs and predicts human skeletons by a message-passing encoder-decoder RNN (MPED-RNN); <cit.> integrates reconstruction into prediction by a predictive coding network based framework (AnoPCN); <cit.> conducts prediction and reconstruction in a sequential manner.
  
  
  
  § VIDEO EVENT COMPLETION (VEC)
  
  
  
  
   §.§ Video Event Extraction
  
  
  In this paper, video event extraction aims to enclose video activities by numerous normalized spatio-temporal cubes (STCs). A STC is then viewed as a video event, which serves as the basic processing unit in VEC. A both precise and comprehensive localization of video activities is the key to video event extraction. Ideally, precise localization expects the subject of a video activity is intactly extracted with minimal irrelevant background, while comprehensive localization requires all subjects associated with video activities are extracted. As explained by Gap #1 (Sec. <ref>) and the intuitive comparison in Fig. <ref>, existing DNN based VAD methods fail to realize precise and comprehensive localization simultaneously, which undermines the quality of video activity modeling and VAD performance. Hence, we exploit appearance and motion as mutually complementary cues to achieve both precise and comprehensive localization (Fig. <ref> (d)). This new pipeline is detailed as follows:
  
  
  
  Motivation. As video activities are behaviors conducted by certain subjects in videos, we consider both appearance cues from those subjects and motion cues from their behaviors to localize regions of interest (RoIs) that enclose those video activities. To utilize appearance cues, a natural solution is modern object detection <cit.>. With pre-training on large-scale public datasets like Microsoft COCO, pre-trained detectors can precisely localize RoIs with frequently-seen objects. Therefore, we use a pre-trained object detector to realize precise localization of most video activities by detecting their associated subjects, e.g. humans. However, pre-trained detectors only detect objects in the “closed world” formed by known object classes in the training dataset. This leads to a fatal problem: Anomalies, which are often novel classes outside the “closed world”, will be omitted. To this end, motion cues like temporal gradients are proposed as complimentary information to accomplish more comprehensive RoI localization. More importantly, we argue that appearance and motion cues should not be isolated: RoIs already localized by appearance should be filtered when exploiting motion cues, which reduces computation and makes motion based RoI localization more precise (see Fig. <ref> (d)). As illustrated by the overview in Fig. <ref> and Algorithm <ref>, we elaborate each component of the new video event extraction pipeline below.
  
  
  
  
  
  Appearance Based RoI Extraction. Given a video frame I and a pre-trained object detector model M, our goal is to obtain a RoI set B_a based on appearance cues from subjects of video activities, where B_a⊆ℝ^4 and each entry of B_a refers to a RoI enclosed by a bounding box. The bounding box is denoted by the coordinates of its top-left and bottom-right vertex, which is a 4-dimensional vector. As shown by the green module in Fig. <ref>, we first feed I into M, and obtain a preliminary RoI set B_ap with confidence scores above the threshold T_s (class labels are discarded). Then, we introduce two efficient heuristic rules to filter unreasonable RoIs: (1) RoI area threshold T_a that filters out overly small RoIs. (2) Overlapping ratio T_o that removes RoIs that are nested or significantly overlapped with larger RoIs in B_ap. In this way, we ensure that extracted RoIs can precisely enclose subjects of most everyday video activities.
  
  Motion Based RoI Extraction. To enclose those activities outside the “closed world”, motion based RoI extraction aims to yield a complementary bounding box set B_m based on motion cues. We leverage the temporal gradients of frames as motion cues and complementary information. As shown by the red module of Fig. <ref>, we first binarize the absolute values of temporal gradients by a threshold T_g, so as to yield a binary map that indicates regions with intense motion. Instead of using this map directly, we propose to subtract appearance based RoIs B_a from the map, which benefits motion based RoI extraction in two ways: First, the subtraction of appearance based RoIs enables us to better localize objects that are not detected by appearance cues, otherwise the gradient map of multiple objects may be overlapped and jointly produce large and imprecise RoIs (see Fig. <ref> (b)). Second, the subtraction reduces the computation. Finally, we propose to perform contour detection to yield the contour and its corresponding bounding box b_m, while simple heuristics (RoI area threshold T_a and maximum aspect-ratio threshold T_ar) are used to obtain final RoI set B_m. Based on two complementary RoI sets, the final RoI set B=B_a∪ B_m. The whole RoI extraction process is formally presented in Algorithm <ref>.
  
  Spatio-temporal Cube Construction. Finally, we use each RoI in B to build a spatio-temporal cube (STC) as the video event, which represents the fundamental unit to enclose video activities. As shown by yellow module in Fig. <ref>, we not only extract the patch p_1 in the RoI from current frame, but also extract corresponding patches p_2,⋯, p_D by this RoI from previous (D-1) frames. In this way, we incorporate the temporal context into the extracted video event. To normalize video activities with different scales, we resize patches from the same RoI into H× W new patches p'_1, ⋯, p'_D, which are then stacked into a H× W× D STC: C=[p'_1; ⋯; p'_D].
  
  
  
   §.§ Visual Cloze Tests
  
  As explained by Gap #2 in Sec. <ref>, previous methods typically rely on a reconstruction or frame prediction paradigm. They cannot fully exploit high-level semantics and temporal context information. As a remedy, we propose to build visual cloze tests as a new learning paradigm, so as to model normal video activities represented by STCs above. We present it in terms of the following aspects:
  
  
  Motivation.
  We are inspired by cloze test, an extensively-used exercise in language learning and instruction. It requires completing a text with its certain words erased. Cloze test aims to test students' ability to understand the vocabulary semantics and language context <cit.>. Recently, learning to solve cloze tests is also shown to be a fairly effective pretraining method in natural language processing (NLP), which enables DNN to capture richer high-level semantics from text <cit.>. This naturally inspires us to compare the patch sequence of a STC to the word sequence in classic cloze test. Similarly, we can erase a certain patch p'_i in video event (STC) to build a visual cloze test, which is solved by completing the resultant incomplete event (IE) with the DNN's inferred p'_i. Such a learning paradigm benefits DNN in two aspects: (1) In order to complete such a visual cloze test, DNN is encouraged to capture high-level semantics in STC. For example, suppose a video event contains a walking person. DNN must attend to those key moving parts (e.g. the forwarding leg and swinging arm) in the patch to achieve a good completion. This makes visual cloze test more meaningful than frequently-used reconstruction, which tends to memorize every low-level detail of inputs to minimize training loss. (2) Since any patch in a STC can be erased to generate an IE, we can build multiple cloze tests by erasing the patch at different temporal positions. This enables us to fully exploit the temporal context by enumerating all possible IEs for cloze tests. By contrast, prediction based VAD methods only consider prediction errors of a single frame to detect anomalies, which involves poor temporal context information in video activities. As shown in Fig. <ref>, we detail VEC's components below.
  
  Appearance Completion.
  Given j-th video event represented by the STC C_j=[p'_j,1; ⋯; p'_j,D], we first erase the i-th patch p'_j,i of C_j to build an IE C_j^(i)=[p'_j,1;⋯ p'_j,i-1; p'_j,i+1;⋯ p'_j,D] as a cloze test, i∈{1,⋯ D} (blue module in Fig. <ref>). All IEs built by erasing the i-th patch of the STC are collected as the type-i IE set 𝒞^(i)={C_1^(i), ⋯ C_N^(i)}, where N is the number of extracted video events (STCs). Afterwards, as shown by red module in Fig. <ref>, a type-i IE C_j^(i) in 𝒞^(i) and its corresponding erased patch p'_j,i are used as the input and learning target respectively to train a generative DNN f^(i)_a (e.g. autoencoder, generative adversarial networks, U-Net, etc.), which aims to generate a patch p'_j,i=f^(i)_a(C_j^(i)) to complete the IE C_j^(i) into the original event C_j. To train f^(i)_a, here we can simply minimize the pixel-wise appearance loss ℒ^(i)_a for type-i IE set 𝒞^(i):
  
      ℒ^(i)_a = 1/N∑^N_j=1‖p'_j,i-p'_j,i‖^p_p
  
  where ‖·‖_p denotes p-norm. In our experiments, we found that choosing p=2 already works well. To further improve the fidelity of generated patch, other values of p or techniques like adversarial training can also be explored to design ℒ^(i)_a. For inference, any error measure 𝒮^(i)_a(p'_j,i, p'_j,i) can be used to yield the appearance anomaly score of patch p'_j,i, such as mean square error (MSE) or Peak Signal to Noise Ratio (PSNR) <cit.>. Empirically, choosing 𝒮^(i)_a(p'_j,i, p'_j,i) to be simple MSE has been effective enough to score anomalies, and poorly completed STCs with higher MSE are more likely to be anomalies. Since appearance completion is actually a challenging learning task for DNN, an independent DNN f^(i)_a is trained to handle one IE type 𝒞^(i). Otherwise, using one DNN for all IE types will degrade the performance of appearance completion.
  
  Motion Completion.
  Motion is another type of valuable prior in videos. Optical flow, which estimates the pixel-wise motion velocity and direction between two consecutive frames, is a popular low-level motion representation in videos. We feed two consecutive frames into a pre-trained FlowNet model <cit.>, and a forward pass can yield the optical flow efficiently. For each STC C_j, we extract optical flow patches o_j,1, ⋯ o_j,D that correspond to video patches p_j,1, ⋯ p_j,D, and also resize them into H× W patches o'_j,1, ⋯ o'_j,D. Motion completion requires a DNN f^(i)_m to infer the optical flow patch o'_j,i of the erased patch p'_j,i by the type-i IE C_j^(i), i.e. o'_j,i=f^(i)_m(C_j^(i)). f^(i)_m can be trained by the motion loss ℒ^(i)_m:
  
  
      ℒ^(i)_m = 1/N∑^N_j=1‖o'_j,i-o'_j,i‖^p_p
  
  Likewise, we also adopt p=2 for ℒ^(i)_m and simple MSE to compute the motion anomaly score 𝒮^(i)_m(o'_j,i, o'_j,i) during inference. With motion completion, we encourage DNN to infer the motion statistics from the temporal context provided by IEs, which enables VEC to consider richer motion dynamics. The process of both appearance and motion completion for a type-i IE are shown in Fig. <ref>.
  
  Ensemble Strategies.
  Ensemble is a powerful tool that combines multiple models into a stronger one <cit.>. During inference, we propose two ensemble strategies to improve the VEC performance: (1) IE type ensemble. Erasing a different patch in STC produces a different IE, which contains a different patch combination as temporal context. To fully exploit all possible temporal context for VAD, we compute the final appearance anomaly score by an ensemble of scores, which are yielded by multiple DNNs for different IE types:
  
  
      𝒮_a(C_j) = 1/D∑^D_i=1𝒮^(i)_a(p'_j,i, p'_j,i)
  
  IE type ensemble is also applicable to the final motion score 𝒮_m(C_j):
  
      𝒮_m(C_j) = 1/D∑^D_i=1𝒮^(i)_m(o'_j,i, o'_j,i)
  
  (2) Modality ensemble. Since two different modalities, raw pixels and optical flow, are considered to perform completion in VEC, we must fuse their results to yield the overall anomaly score. For simplicity, we use a weighted sum of 𝒮_a(C_j) and 𝒮_m(C_j) to compute the overall anomaly score S(C_j) for a video event C_j:
  
  
      𝒮(C_j)=w_a·𝒮_a(C_j)-𝒮_a/σ_a + w_m·𝒮_m(C_j)-𝒮_m/σ_m
  
  where 𝒮_a, σ_a, 𝒮_m, σ_m denote the means and standard deviations of appearance and motion scores for all normal events in training, which are used to normalize appearance and motion scores into the same scale. In addition to this straightforward weighting strategy, other more sophisticated strategies like late fusion <cit.> are also applicable to achieve better modality ensemble performance.
  
  
  
  § EVALUATION
  
  
  
  
  
   §.§ Experimental Settings
  
  Evaluation is performed on three most commonly-used benchmark datasets for VAD: UCSDped2 <cit.>, Avenue <cit.> and ShanghaiTech <cit.>. For video event extraction, cascade R-CNN <cit.> pre-trained on COCO dataset is used as object detector as it achieves a good trade-off between performance and speed. Other parameters are set as follows for UCSDped2, Avenue and ShanghaiTech respectively: Confidence score threshold T_s: (0.5, 0.25, 0.5); RoI area threshold T_a: (10×10, 40×40, 8×8); Overlapping ratio T_o: (0.6, 0.6, 0.65); Gradient binarization threshold T_g: (18, 18, 15); Maximum aspect-ratio threshold T_ar=10. For cube construction, we set H=W=32 and D=5. As to VEC, we adopt U-Net <cit.> as the basic network architecture of generative DNNs (see Fig. <ref>), which are optimized by the default Adam optimizer in PyTorch <cit.>. Considering the dataset scale, DNNs are trained by 5, 20, 30 epochs with a batch size 128 on UCSDped2, Avenue and ShanghaiTech respectively. For anomaly scoring, we set (w_a, w_m) to be (0.5, 1), (1, 1) and (1, 0.5) for UCSDped2, Avenue and ShanghaiTech respectively. For quantitative evaluation, we adopt the most frequently-used metric: Area Under the Receiver Operating Characteristic curves (AUROC) that are computed with frame-level detection criteria <cit.>. Frame-level Equal Error Rate (EER) <cit.> is also reported in the supplementary material. We run experiments on a PC with 64 GiB RAM, Nvidia Titan Xp GPUs and a 3.6GHz Intel i7-9700k CPU.
  
  
  
  
  
   §.§ Comparison with State-of-the-Art methods
  
  
  Within our best knowledge, we extensively compare VEC's performance with 18 state-of-the-art DNN based VAD methods reviewed in Sec. <ref>. Note that we exclude <cit.> from comparison as it actually uses a different evaluation metric from commonly-used frame-level AUROC, which leads to an unfair comparison. As discussed in Sec. <ref>, existing methods can be categorized into reconstruction based, frame prediction based and hybrid methods in Table <ref>. As to VEC, we design two configurations for IE type ensemble: (1) VEC-A: IE type ensemble is applied to appearance completion, while it is not applied to motion completion (i.e. only type-D IEs are used to train the DNN for motion completion). (2) VEC-AM: IE type ensemble is applied to both appearance completion and motion completion. Besides, modality ensemble is applied to both VEC-A and VEC-AM. The results are reported in Table <ref>, and we visualize the yielded frame-level ROC curves in Fig. <ref>. We draw the following observations: First, both VEC-A and VEC-AM consistently outperform existing state-of-the-art DNN based VAD methods on three benchmarks. In particular, we note that VEC achieves notable performance gain on recent challenging benchmarks (Avenue and ShanghaiTech) with constantly ≥ 3% and ≥ 1% AUROC improvement respectively against all state-of-the-art methods. Meanwhile, we note that VEC-A even achieves over 90% frame-level AUROC on Avenue, which is the best performance ever achieved on Avenue dataset to our knowledge. In terms of the comparison between VEC-A and VEC-AM, two configurations yield fairly close performance, despite of slight differences on different datasets. As a consequence, in addition to those thoroughly-studied reconstruction or prediction based methods, the proposed VEC provides a highly promising alternative for DNN based VAD with state-of-the-art performance.
  
  
  
  
  
  
  
   §.§ Detailed Analysis
  
  
  
  
  Ablation Studies. To show the role of the proposed video event extraction and ensemble strategies, we perform corresponding ablation studies and display the results in Table <ref>: (1) As to video event extraction, we compare four practices for localizing video activities: Frame (FR, i.e. no localization at all), multi-scale sliding windows with motion filtering (SDW), appearance based RoI extraction only (APR) and the proposed appearance and motion based RoI extraction (APR+MT). Note that we did not report SDW's results on ShanghaiTech, since it produces excessive STCs that are beyond the limit of our hardware, which is actually an important downside of SDW. There are several observations: First, with the same ensemble strategies, the proposed APR+MT constantly outperforms other methods by a sensible margin. Specifically, APR+MT has an obvious advantage (2.7%-4.6% AUROC gain) over FR and SDW, which are commonly-used strategies of recent VAD methods. Interestingly, we note that SDW performs worse than FR on UCSDped2 and Avenue. This indicates that an imprecise localization of video activities even degrades VAD performance, and it also justifies the importance of a precise localization. Meanwhile, when compared with APR, the proposed APR+MT brings evident improvement by 1.8%, 2.5% and 1.2% AUROC gain on UCSDped2, Avenue and ShanghaiTech respectively. Such observations demonstrate that a more comprehensive localization of video activities will contribute to VAD performance. (2) As to ensemble strategies, we compare three cases: Not using IE type ensemble (for both appearance and motion completion), not using modality ensemble (w_m=0), and both IE type and modality ensemble are used (VEC-AM). We yield the following observations: First, IE type ensemble contributes to VAD performance by 1.3%, 2.1% and 0.4% AUROC on UCSDped2, Avenue and ShanghaiTech respectively, which justifies the importance to fully exploit temporal context. Second, modality ensemble enables a remarkable 8% AUROC gain on UCSDped2. This is because UCSDped2 contains low-resolution gray-scale frames, and motion clues are more important for detecting anomalies. For Avenue and ShanghaiTech with high-resolution colored frames, modality ensemble also enables over 1% AUROC improvement, although using the modality of raw pixel only already leads to satisfactory performance.
  
  Visualization. To show how visual cloze tests in VEC helps discriminating anomalies in a more intuitive way, we visualize generated patches and optical flow of representative normal/abnormal video events in Fig. <ref>. Heat maps are used for a better visualization of the pixel-wise completion errors. By Fig. <ref>, it is worth noting several phenomena: First of all, VEC can effectively complete normal events and their optical flow. For normal events, minor completion errors are observed to be distributed around foreground contour in a relatively uniform manner, and their optical flow can also be soundly recovered. By contrast, abnormal events produces prominent completion errors in terms of both raw pixel and optical flow completion. Next, it is noted that the distribution of anomalies' completion errors is highly non-uniform. As shown by heat maps, large completion errors are often observed at those regions that have clear high-level semantics, e.g. the bicycle that the man was riding with (UCSDped2), falling paper with its shape and position wrongly inferred (Avenue), the backpack that was thrown (ShanghaiTech). By contrast, other regions are endowed with relatively smaller errors. Such observations imply that VEC indeed attends to those parts with high-level semantics in abnormal events. 
  
  
  
  Other Remarks. (1) VEC adopts a similar U-Net architecture to previous works, but it achieves significantly better performance, which exactly verifies visual cloze tests' effectiveness as a new learning paradigm. Thus, better network architecture can be explored, while techniques like adversarial training and attention mechanism are also applicable. In fact, VEC with only type-D IEs can be viewed as predicting the last patch of STCs (as shown in Table <ref>, it is also better than frame prediction <cit.>). Besides, when visual cloze tests in VEC are replaced by plain reconstruction, experiments report a 3% to 7% AUROC loss on benchmarks, which demonstrates that our video event extraction and visual cloze tests are both indispensable. (2) Details of VEC's computation cost and parameter sensitivity are also discussed in the supplementary material.
  
  
  
  § CONCLUSION
  
  
  In this paper, we propose VEC as a new solution to DNN based VAD. VEC first extracts STCs by exploiting both appearance and motion cues, which enables both precise and comprehensive video event extraction. Subsequently, motivated by the widely-used cloze test, VEC learns to solve visual cloze tests, i.e. training DNNs to infer deliberately erased patches from incomplete video events/STCs, so as to learn better high-level semantics. Motion modality is also involved by using DNNs to infer the erased patches' optical flow. Two ensemble strategies are further adopted to fully exploit temporal context and motion dynamics, so as to enhance VAD performance.
  
  	
  The work is supported by National Natural Science Foundation of China under Grant No. 61702539, Hunan Provincial Natural Science Foundation of China under Grant No. 2018JJ3611, No. 2020JJ5673, NUDT Research Project under Grant No. ZK-18-03-47, ZK20-10, and The National Key Research and Development Program of China (2018YFB0204301, 2018YFB1800202, SQ2019ZD090149). Siqi Wang, Zhiping Cai and Jianping Yin are corresponding authors.
  
  
  
  
  ACM-Reference-Format
  
  
  