Question: Represent the article for finding a single-sentence TLDR

Question: Long-term video prediction is highly challenging since it entails simultaneously capturing spatial and temporal information across a long range of image frames. Standard recurrent models are ineffective since they are prone to error propagation and cannot effectively capture higher-order correlations. A potential solution is to extend to higher-order spatio-temporal recurrent models. However, such a model requires a large number of parameters and operations, making it intractable to learn in practice and is prone to overfitting. In this work, we propose convolutional tensor-train LSTM (Conv-TT-LSTM), which learns higher-orderConvolutional LSTM (ConvLSTM) efficiently using convolutional tensor-train decomposition (CTTD). Our proposed model naturally incorporates higher-order spatio-temporal information at a small cost of memory and computation by using efficient low-rank tensor representations. We evaluate our model on Moving-MNIST and KTH datasets and show improvements over standard ConvLSTM and better/comparable to other ConvLSTM-based approaches, but with much fewer parameters. Understanding dynamics of videos and performing long-term predictions of the future is a highly challenging problem. It entails learning complex representation of real-world environment without external supervision. This arises in a wide range of applications, including autonomous driving, robot control, or other visual perception tasks like action recognition or object tracking . However, long-term video prediction remains an open problem due to high complexity of the video contents. Therefore, prior works mostly focus on next or first few frames prediction (; ;). Many recent video models use Convolutional LSTM (ConvLSTM) as a basic block , where spatio-temporal information is encoded as a tensor explicitly in each cell. In ConvL-STM networks, each cell is a first-order recurrent model, where the hidden state is updated based on its immediate previous step. Therefore, they cannot easily capture higher-order temporal correlations needed for long-term prediction. Moreover, they are highly prone to error propagation. Various approaches have been proposed to augment ConvLSTM, either by modifying networks to explicitly modeling motion , or by integrating spatio-temporal interaction in ConvLSTM cells (; 2018a). These approaches are often incapable of capturing longterm dependencies and produce blurry prediction. Another direction to augment ConvLSTM is to incorporate a higher-order RNNs inside each LSTM cell, where its hidden state is updated using multiple past steps. However, a higher-order model for high-dimensional data (e.g. video) requires a huge number of model parameters, and the computation grows exponentially with the order of the RNNs. A principled approach to address the curse of dimensionality is tensor decomposition, where a higher-order tensor is compressed into smaller core tensors . Tensor representations are powerful since they retain rich expressivity even with a small number of parameters. In this work, we propose a novel convolutional tensor decomposition, which allows for compact higher-order ConvLSTM. Contributions. We propose Convolutional Tensor-Train LSTM (Conv-TT-LSTM), a modification of ConvLSTM, to build a higher-order spatio-temporal model. We introduce Convolutional Tensor-Train Decomposition (CTTD) that factorizes a large convolutional kernel into a chain of Figure 1: Illustration of (a) convolutional tensor-train (Eqs. and) and the difference between convolutional tensor-train LSTM (b) Fixed window version (Eqs. (11a) and) and (c) Sliding window version (Eqs. (11b) and and 1c ), and we found that the SW version performs better than the FW one. We found that training higher-order tensor models is not straightforward due to gradient instability. We present several approaches to overcome this such as good learning schedules and gradient clipping. In the experiments, we show our proposed Conv-TT-LSTM consistently produces sharp prediction over a long period of time for both Moving-MNIST-2 and KTH action datasets. Conv-TT-LSTM outperforms the state-of-the-art PredRNN++ (a) in LPIPS by 0.050 on the Moving-MNIST-2 and 0.071 on the KTH action dataset, with 5.6 times fewer parameters. Thus, we obtain best of both worlds: better long-term prediction and model compression. Tensor Decomposition In machine learning, tensor decompositions, including CP decomposition , Tucker decomposition , and tensor-train decomposition , are widely used for dimensionality reduction and learning probabilistic models . In deep learning, prior works focused on their application in model compression, where the parameters tensors are factorized into smaller tensors. This technique has been used in compressing convolutional networks (; ; ; ; ; ;, recurrent networks and transformers . demonstrates that the accuracy of video classification increases if the parameters in recurrent networks are compressed by tensor-train decomposition . Yu et al. (2017 used tensor-train decomposition to constrain the complexity of higher-order LSTM, where each next step is computed based on the outer product of previous steps. While this work only considers vector input at each step, we extend their approach to higher-order ConvLSTM, where each step also encodes spatial information. Video Prediction Prior works on video prediction have focused on several directions: predicting short-term video , decomposing motion and contents (; ; ;), improving the objective function , and handling diversity of the future (; ;). Many of these works use Convolutional LSTM (ConvL-STM) as a base module, which deploys 2D convolutional operations in LSTM to efficiently exploit spatio-temporal information. used ConvLSTM to model pixel motion. Some works modified the standard ConvLSTM to better capture spatio-temporal correlations (; 2018a). Wang et al. (2018b) integrated 3D convolutions into ConvLSTM. In addition, current cell states are combined with its historical records using self-attention to efficiently recall the history information. applied ConvLSTM in all possible directions to capture full contexts in video and also demonstrated strong performance using a deep ConvLSTM network as a baseline. This baseline is adapted to obtain the base architecture in the present paper. The goal of tensor decomposition is to represent a higher-order tensor as a set of smaller and lowerorder core tensors, with fewer parameters while preserve essential information. , tensor-train decomposition is used to reduce both parameters and computations in higher-order recurrent models, which we review in the first part of this section. However, the approach in only considers recurrent models with vector inputs and cannot cope with image inputs directly. In the second part, we extend the standard tensor-train decomposition to convolutional tensor-train decomposition (CTTD). With CTTD, a large convolutional kernel is factorized into a chain of smaller kernels. We show that such decomposition can reduce both parameters and operations of higher-order spatio-temporal recurrent models. Standard Tensor-train decomposition Given an m-order tensor T ∈ R I1×···×Im, where I l is the dimension of its l-th order, a standard tensor-train decomposition (TTD) factorizes the tensor T into a set of m core tensors where tensor-train ranks {R l} m l=0 (with R 0 = R m = 1) control the number of parameters in the tensor-train format Eq.. With TTD, the original tensor T of size (entries, which grows linearly with the order m (assuming R l 's are constant). Therefore, TTD is commonly used to approximate higher-order tensors with fewer parameters. The sequential structure in tensor-train decomposition makes it particularly suitable for sequence modeling . Consider a higher-order recurrent model that predicts a scalar output v ∈ R based on the outer product of a sequence of input vectors {u according to: This model is intractable in practice since the number of parameters in T ∈ R I1×···Im (and therefore computational complexity of Eq.) grows exponentially with the order m. Now suppose T takes a tensor-train format as in Eq., we prove in Appendix A that can be efficiently computed as where the vectors {v are the intermediate steps, with v ∈ R initialized as v = 1, and final output v = v (m). Notice that the higher-order tensor T is never reconstructed in the sequential process in Eq., therefore both space and computational complexities grow linearly (not exponentially compared to Eq.)with the order m assuming all tensor-train ranks are constants. Convolutional Tensor-Train Decomposition A convolutional layer in neural network is typically parameterized by a 4-th order tensor T ∈ R K×K×Rm×R0, where K is the kernel size, R m and R 0 are the number of input and output channels respectively. Suppose the kernel size K takes the form K = m(k − 1) + 1 (e.g. K = 7 and m = 3, k = 3), a convolutional tensor-train decomposition (CTTD) factorizes T into a set of m core tensors:,:,r1,r0 * T where * denotes convolution between 2D-filters, and {R l} m l=1 are the convolutional tensor-train ranks that control the complexity of the convolutional tensor-train format in Eq.. With CTTD, the number of parameters in the decomposed format reduces from Similar to standard TTD, its convolutional counterpart can also be used to compress higher-order spatio-temporal recurrent models with convolutional operations. Consider a model that predicts a 3-rd order feature V ∈ R H×W ×R0 based on a sequence of 3-rd features (where H, W are height/width of the features and R l is the number of channels in U (l) ) such that where is the corresponding weights tensor for U (l). Suppose each W (l) takes a convolutional tensor-train format in Eq., we prove in Appendix A that the model in Eq. can be computed sequentially similarly without reconstructing the original W (l)'s: where are intermediate of the sequential process, where V (m) ∈ R H×W ×Rm is initialized as all zeros and final prediction V = V. The operations in Eq. is illustrated in Figure 1a. In this paper, we denote the Eq. ). Convolutional LSTM is a basic block for most recent video forecasting models , where the spatial information is encoded explicitly as tensors in the LSTM cells. In a ConvLSTM network, each cell is a first-order Markov model, i.e. the hidden state is updated based on its previous step. In this section, we propose convolutional tensor-train LSTM, where convolutional tensor-train is incorporated to model multi-steps spatio-temporal correlation explicitly. Notations. In this section, the symbol * is overloaded to denote convolution between higher-order tensors. For instance, given a 4-th order weights tensor W ∈ R K×K×S×C and a 3-rd order input tensor X ∈ R H×W ×S, Y = W * X computes a 3-rd output tensor Y ∈ R H×W ×T as Y:,:,c = s=1 W:,:,s,c * X:,:,s. The symbol • is used to denote element-wise product between two tensors, and σ represents a function that performs element-wise (nonlinear) transformation on a tensor. extended fully-connected LSTM (FC-LSTM) to Convolutional LSTM (ConvLSTM) to model spatio-temporal structures within each recurrent unit, where all features are encoded as 3-rd order tensors with dimensions (height × width × channels) and matrix multiplications are replaced by convolutions between tensors. In a ConvLSTM cell, the parameters are characterized by two 4-th order tensors W ∈ R K×K×S×4C and T ∈ R K×K×C×4C, where K is the kernel size of all convolutions and S and C are the numbers of channels of the input X (t) ∈ R H×W ×S and hidden states H (t) ∈ R H×W ×C respectively. At each time step t, a ConvLSTM cell updates its hidden states H (t) ∈ R H×W ×C based on the previous step H (t−1) and the current input X (t), where H and W are the height/width that are the same for X (t) and H (t). where σ(·) applies sigmoid on the input gate I (t), forget gate F (t), output gate O (t), and hyperbolic tangent on memory cellC (t). Note that all tensors Convolutional Tensor-Train LSTM In Conv-TT-LSTM, we introduce a higher-order recurrent unit to capture multi-steps spatio-temporal correlations in LSTM, where the hidden state H (t) is updated based on its n previous steps with an m-order convolutional tensor-train (CTT) as in Eq.. Concretely, suppose the parameters in CTT are characterized by m tensors of 4-th order, Conv-TT-LSTM replaces Eq. in ConvLSTM by two equations:, ·) takes a sequence of m tensors as inputs, the first step in Eq. maps the n inputs, ·) and compute the gates according to Eq.. We propose two realizations of Eq., where the first realization uses a fixed window of to compute eachH (t,o), while the second one adopts a sliding window strategy. At each step, the Conv-TT-LSTM model computes H (t) by replacing Eq. by either Eq. (11a) or (11b). Conv-TT-LSTM-SW: In the fixed window version, the previous steps {H (l) } n l=1 are concatenated into a 3-rd order tensor H (t,o) ∈ R H×W ×nC, which is then mapped to a tensorH (t,o) ∈ R H×W ×R by 2D-convolution with a kernel K (l) ∈ R k×k×nC×R. And in the sliding window version, {H (l) } n l=1 are concatenated into a 4-th order tensorĤ (t,o) ∈ R H×W ×D×C (with D = n − m + 1), which is mapped toH (t,o) ∈ R H×W ×R by 3D-convolution with a kernel K (l) ∈ R k×k×D×R. For later reference, we name the model with Eqs.(11a) and We first evaluate our approach extensively on the synthetic Moving-MNIST-2 dataset . In addition, we use KTH human action dataset to test the performance of our models in more realistic scenario. Model Architecture All experiments use a stack of 12-layers of ConvLSTM or Conv-TT-LSTM with 32 channels for the first and last 3 layers, and 48 channels for the 6 layers in the middle. A convolutional layer is applied on top of all LSTM layers to compute the predicted frames. , two skip connections performing concatenation over channels are added between and layers. Illustration of the network architecture is included in the appendix. All parameters are initialized by Xavier's normalized initializer and initial states in ConvLSTM or Conv-TT-LSTM are initialized as zeros. Evaluation Metrics We use two traditional metrics MSE (or PSNR) and SSIM , and a recently proposed deep-learning based metric LPIPS , which measures the similarity between deep features. Since MSE (or PSNR) is based on pixel-wise difference, it favors vague and blurry predictions, which is not a proper measurement of perceptual similarity. While SSIM was originally proposed to address the problem, shows that their proposed LPIPS metric aligns better to human perception. Learning Strategy All models are trained with ADAM optimizer with L 1 + L 2 loss. Learning rate decay and scheduled sampling are used to ease training. Scheduled sampling is started once the model does not improve in 20 epochs (in term of validation loss), and the sampling ratio is decreased linearly from 1 until it reaches zero (by 2 × 10 −4 each epoch for Moving-MNIST-2 and 5 × 10 −4 for KTH). Learning rate decay is further activated if the loss does not drop in 20 epochs, and the rate is decreased exponentially by 0.98 every 5 epochs. We perform a wide range of hyper-parameters search for Conv-TT-LSTM to identify the best model, and of 10 −3 is found for the models of kernel size 3 and 10 −4 for the models of kernel size 5. We found that Conv-TT-LSTM models suffer from exploding gradients when learning rate is high (e.g. 10 −3 in our experiments), therefore we also explore various gradient clipping values and select 1 for all Conv-TT-LSTM models. All hyper-parameters are selected using the best validation performance. The Moving-MNIST-2 dataset is generated by moving two digits of size 28 × 28 in MNIST dataset within a 64 × 64 black canvas. These digits are placed at a random initial location, and move with constant velocity in the canvas and bounce when they reach the boundary. Following Wang et al. (2018a), we generate 10,000 videos for training, 3,000 for validation, and 5,000 for test with default parameters in the generator 1. All our models are trained to predict 10 frames given 10 input frames. All our models use kernel size 5: Conv-TT-LSTM-FW has hyperparameters as (order 1, steps 3, ranks 8), and Conv-TT-LSTM-SW has hyperparameters as (order 3, steps 3, ranks 8). Figure 2: Frame-wise comparison in MSE, SSIM and PIPS on Moving-MNIST-2. For MSE and LPIPS, lower curves denote higher quality; while for SSIM, higher curves imply better quality. Table 2 reports the average statistics for 10 and 30 frames prediction, and Figure 2 shows comparison of per-frame statistics for PredRNN++ model, ConvLSTM baseline and our proposed Conv-TT-LSTM models. Our Conv-TT-LSTM models consistently outperform the 1 https://github.com/jthsieh/DDPAE-video-prediction/blob/master/data/moving_mnist.py 2 The are cited from the original paper, where the miscalculation of MSE is corrected in the table. 3 The are reproduced from https://github.com/Yunbo426/predrnn-pp with the same datasets in this paper. The original implementation crops each frame into patches as the input to the model. We find out such pre-processing is unnecessary and the performance is better than the original paper. 12-layer ConvLSTM baseline for both 10 and 30 frames prediction with fewer parameters; The Conv-TT-LSTMs outperform previous approaches in terms of SSIM and LPIPS (especially on 30 frames prediction), with less than one fifth of the model parameters. We reproduce the PredRNN++ model (a) from their source code 2, and we find that The PredRNN++ model tends to output vague and blurry in long-term prediction (especially after 20 steps). and our Conv-TT-LSTMs are able to produce sharp and realistic digits over all steps. An example of comparison for different models is shown in Figure 3. The visualization is consistent with the in Ablation Study To understand whether our proposed Conv-TT-LSTM universally improves upon ConvLSTM (i.e. not tied to specific architecture, loss function and learning schedule), we perform three ablation studies: Reduce the number of layers from 12 layers to 4 layers (same as and Wang et al. (2018a) ); Change the loss function from L 1 + L 2 to L 1 only; Disable the scheduled sampling and use teacher forcing during training process. We evaluate the ConvLSTM baseline and our proposed Conv-TT-LSTM in these three settings, and summarize their comparisons in Table 3. The show that our proposed Conv-TT-LSTM outperforms ConvLSTM consistently for all settings, i.e. the Conv-TT-LSTM model improves upon ConvLSTM in a board range of setups, which is not limited to the certain setting used in our paper. These ablation studies further show that our setup is optimal for predictive learning in Moving-MNIST-2. KTH action dataset contains videos of 25 individuals performing 6 types of actions on a simple . Our experimental setup follows Wang et al. (2018a), which uses persons 1-16 for training and 17-25 for testing, and each frame is resized to 128 × 128 pixels. All our models are trained to predict 10 frames given 10 input frames. During training, we randomly select 20 contiguous frames from the training videos as a sample and group every 10,000 samples into one epoch to apply the learning strategy as explained at the beginning of this section. 25.95 0.804 -----E3D-LSTM (b) 29 Results In Table 4, we report the evaluation on both 20 and 40 frames prediction. Our models are consistently better than the ConvLSTM baseline for both 20 and 40 frames prediction. While our proposed Conv-TT-LSTMs achieve lower SSIM value compared to the state-of-the-art models in 20 frames prediction, they outperform all previous models in LPIPS for both 20 and 40 frames prediction. An example of the predictions by the baseline and Conv-TT-LSTMs is shown in Figure 3. In this paper, we proposed convolutional tensor-train decomposition to factorize a large convolutional kernel into a set of smaller core tensors. We applied this technique to efficiently construct convolutional tensor-train LSTM (Conv-TT-LSTM), a high-order spatio-temporal recurrent model whose parameters are represented in tensor-train format. We empirically demonstrated that our proposed Conv-TT-LSTM outperforms standard ConvLSTM and produce better/comparable compared to other state-of-the-art models with fewer parameters. Utilizing the proposed model for high-resolution videos is still challenging due to gradient vanishing or explosion. Future direction will include investigating other training strategies or a model design to ease the training process. In this section, we prove the sequential algorithms in Eq. for tensor-train decomposition and Eq. for convolutional tensor-train decomposition both by induction. Proof of Eq. For simplicity, we denote the standard tensor-train decomposition in Eq. as ), then Eq. can be rewritten as Eq. since R 0 = 1 and v i1,r0,r1 · · · T where R 0 = 1, v

we propose convolutional tensor-train LSTM,  which learns higher-order Convolutional LSTM efficiently using convolutional tensor-train decomposition.

Represent the short summary for finding the corresponding scientific article (long)

We compare graph RNNs and graph ConvNets, and we consider the most generic class of graph ConvNets with residuality.


Question: Represent the scientific article to find a short TLDR

Question: While deep neural networks have shown outstanding in a wide range of applications, learning from a very limited number of examples is still a challenging task. Despite the difficulties of the few-shot learning, metric-learning techniques showed the potential of the neural networks for this task. While these methods perform well, they don’t provide satisfactory . In this work, the idea of metric-learning is extended with Support Vector Machines (SVM) working mechanism, which is well known for generalization capabilities on a small dataset. Furthermore, this paper presents an end-to-end learning framework for training adaptive kernel SVMs, which eliminates the problem of choosing a correct kernel and good features for SVMs. Next, the one-shot learning problem is redefined for audio signals. Then the model was tested on vision task (using Omniglot dataset) and speech task (using TIMIT dataset) as well. Actually, the algorithm using Omniglot dataset improved accuracy from 98.1% to 98.5% on the one-shot classification task and from 98.9% to 99.3% on the few-shot classification task. Deep learning has shown the ability to achieve outstanding for real-world problems in various areas such as image, audio and natural language processing BID18. However these networks require large datasets, so the model fitting demands significant computational resources. On the other hand, there are techniques for learning on small datasets, such as data augmentation and special regularization methods, but these methods' accuracy is far from desirable on a very limited dataset. As well as slowness of the training process is caused by the many weight update iterations, which is required due to the parametric aspect of the model. Humans are capable of learning the concept from only a few or even from one example. This learning characteristic differs much from the deep neural networks' learning curve. This discovery leads us to one-shot learning task BID6, which consists of learning each class from only one example. Nevertheless, one single example is not always enough for humans to understand new concepts. In view of the previous fact, the generalization of one-shot learning task exists as well, it is called few-shot learning or k-shot learning, where the algorithm learns from exactly k samples per class. Deep learning approaches data-poor problems by doing transfer learning BID2: the parameters are optimized on a closely related data-rich problem and then the model is fine-tuned on the given data. In contrast, one-shot learning problem is extremely data-poor, but it requires similar approach as transfer learning: in order to learn good representation, the model is trained on similar data, where the classes are distinct from the one-shot dataset. In the next step, standard machine learning tools are used on the learned features to classify the one-shot samples. As a matter of fact, BID26 claimed that parameterless models perform the best, but they concentrated on only k-nearest neighbors algorithm. Considering this observation this work applies Support Vector Machine BID0, which can be regarded as a parameterless model. This paper presents the k-shot related former work in the following section. Then the proposed model, which is called Siamese kernel SVM, is introduced with a brief summary of the used wellknown methods. In Section 4 the experimental setup is described for both a vision and an auditory task, where minor refinement of the problem is required. The most obvious solution for the one-shot learning task is the k-nearest neighbors algorithm (k-NN). However, there is one problem with this algorithm, it requires complex feature engineering to work efficiently. When the number of available training data points is limited, Support Vector Machines are often used, as they generalize well using only a handful of examples, which makes them suitable for few-shot learning. The problem with SVMs is the same as with the k-nearest neighbors method: one must find set of descriptive features for a given task. One of the neural network solutions for the one-shot learning problem is called Siamese network BID1, which relies on calculating pairwise similarities between data points. This architecture uses two instances of the same feedforward network to calculate representation before the similarity of the two observed samples are determined. Historically this architecture is created for verification problems, but it turned out that the model's learned representations can be used for classification tasks as well BID3. The first versions of Siamese networks used energy based, contrastive loss function BID3 An improved version of the architecture is the Convolutional Siamese Net BID16, which uses binary cross-entropy as loss function and convolutional network to learn features. Our work uses exactly the same convolutional architecture for vision task with a different loss function, which can learn better features for SVM classification. A different improvement of the Siamese architecture is the Triplet network BID10, which approaches the problem as a comparison of the data to a negative and a positive sample at the same time. This model uses three instances of the same feedforward networks: one for positive examples, one for negative examples and one for the investigated samples, which is put to the more similar class. One of the latest state-of-the-art models is Matching Network BID26, which can be considered as an end-to-end k-nearest neighbors algorithm. This extension of the Siamese network contains N + 1 instances of the same network, where N is the number of classes. The algorithm compares the sample to every classes' data points and chooses the class, which has the data points most similar to the investigated sample. So far the distance metric learning approaches have been discussed, but there are different successful methods to solve the problem, such as Memory-Augmented Neural Network BID22, which is a Neural Turing Machine BID8. It uses external memory to achieve good in one-shot learning by memorizing the most descriptive samples. Another approach to the problem is meta-learning. BID21 use an LSTM BID9 based meta-learner that is trained to optimize the model's parameters for few-shot learning. Linear Support Vector Machines (Linear SVM) are created for binary classification BID0. Seeing that, the given data is labeled with +1 and -1: {(x i, y i)|x i ∈ R D, y i ∈ {+1; −1}). Training of Linear SVMs are done by calculating the following constrained optimization, which is minimized with respect to w ∈ R D as it is described in Equation 1. DISPLAYFORM0 In Equation 1, w T w provides the maximal margin between different classes, which can be considered as a regularization technique. ξ i -s are slack variables to create soft margin, they penalize data points inside the margin. Therefore, the C coefficient controls the amount of the regularization. As SVMs are kernel machines, features of the data points are not required, only a positive-definite kernel is needed for training. Fortunately learned similarity metric is positive-definite. The SVM optimization problem's dual form makes it possible to optimize in kernel space, which may in creating a nonlinear decision boundary. This means that during training only the kernel function is required, which can be a precomputed Gram matrix. The dual form of the optimization problem has other useful properties: the training will find a sparse solution while the computational cost is lower if the number of training points is less than the number of features. SVMs are binary classifiers, but they can be extended to multiclass classification with one-vs-rest method BID4. Although this paper investigates only the one-vs-rest approach, other methods are known for multiclass classification BID12 as well. The one-vs-rest approach can be interpreted as training N different SVMs (where N is the number of classes), each of which is used for deciding between given class and another. Equation 2 shows the prediction, where w k is k-th model's weight vector. DISPLAYFORM1 3.2 SIAMESE NETWORKS Siamese network was first created for solving verification problem, where the data is given as (x 1, x 2, y 1,2), two samples and one label. Thus, the task is to predict, whether the x 1 example comes from the same class as the x 2 data point. The idea of Siamese network is to create a feedforward network in two instances with weight sharing, then construct a function to calculate the similarity or distance metric between the two instances BID1. The network's structure can be seen in Figure 1. The feedforward network does representation learning. Eventually, the similarity calculation can be a predefined function BID3 or it can be learned during the training BID16 as well. The main requirements of the Siamese networks are:• Siamese networks are symmetric. If two inputs are given in different order ((x 1, x 2) or (x 2, x 1)), the must be the same. This is provided via the similarity function.• Siamese networks are consistent as well. Two very similar inputs are not projected to very different areas of the vector space. This is the consequence of the weight sharing. Application of Siamese networks can be considered as a method for learning a similarity matrix (called Gram matrix) for all the possible pairs of samples. Siamese networks can be used for classification too. Similarity can be transformed to distance, which is suitable for a k-NN classifier. This is the most popular solution for one-shot learning. Similarity matrix can be used by SVMs as we will see in Section3.3. Otherwise, representation of each instance can be used by any machine learning algorithm for classification. Figure 1: Verification Model: The network is fed with data pairs. g Θ is a feature extractor function. Two instances of the g Θ exist, the Θ parameter set is shared between instances. SVM layer separates same class pairs from different class pairs. In the previous subsections, the two principal components of the model have been introduced. As Section 3.2 mentioned that Siamese networks were first trained on verification task. The verification architecture can be seen in Figure 1. The data preprocessing for the model is the same as Siamese network's process. Notably, the number of positive and negative samples are recommended to be equal. This is provided as all positive pairs are generated and the same amount of negative pairs are generated by choosing samples from different classes randomly. The negative sample generation is done on the fly, this can be considered as a mild augmentation. In the verification architecture the SVM layer and its loss function have two parts:• Equation 3 shows the feature difference calculation. Siamese network's symmetric attribution is provided via this function. Element-wise p-norm is perfect choice, this paper uses L 1 norm. In equation 3 n-th and m-th samples are compared, where a i refers to the vector's i-th element. This Φ i n,m is used by the SVM as input.∀i: DISPLAYFORM0 • This paper uses a popular version of linear SVMs, called L2-SVM, which minimizes squared hinge loss. Neural networks with different SVM loss functions are investigated in , L2-SVM loss variant is considered to be the best by the author of the paper. This loss function can be seen in Equation 4, where y n,m ∈ {+1; −1} is the label of the pair. The loss function's minimal solution is equivalent to the optimal solution of Equation 1. DISPLAYFORM1 The used kernel is linear, so the data points' vectors in the SVM's feature space can be represented with finite dimension. Linear SVMs perform the best when data points in the SVM's feature space are separable by a hyperplane, which can be reached through high dimensional feature space. For this reason, a large number of neurons in g Θ's last layer may increase performance when the number of classes is large. Another solution for increasing the feature space dimension is using a nonlinear kernel in the SVM Layer and in the loss function. For example, Radial Basis Function (RBF) kernel in infinite dimension in the SVM's feature space. The main drawback of a nonlinear kernel is that the loss function must use the dual form of Support Vector Machine optimization BID0, which can be computationally expensive in case of this architecture. Typically, the number of training samples is large for a deep model and the complexity of the gradient calculation for the last layer's weights in dual form is rather enormous. This computational complexity is O(m 2) indeed, where m is the number of samples and all examples can be considered as a potential support vector. These gradients can be determined via dual coordinate descent method, which is analyzed in details in the BID11 article. In , the complexity of calculating the loss values for one epoch is O(n 4) due to the Siamese architecture's sample pair generation (considering batch size is independent of n), where n is the number of samples. This makes the model hard to train on a large dataset. Yet, another problem of the dual form is that the number of parameters in the SVM Layer is equal to the number of samples, which makes the model tied to the dataset. Therefore, this paper investigates only linear SVM solutions due to this problem. K-shot learning has two learning phases, the first is described as the verification learning phase, which is used for representation learning on an isolated dataset (see Figure 1). In the second phase, which is referred to as few-shot learning in this paper, the new classes are learned by a linear multiclass SVM (see FIG0). The classifier model uses the representation of the data, which is provided by g Θ. The mentioned SVM has the same C parameter as the squared hinge loss function, which is why the optimal representation for the Support Vector Machine is learned by g Θ. Therefore, this learning characteristic makes the neural network of an adaptive kernel for the SVM.In a former paragraph, the possibility of a nonlinear kernel is investigated in the representation learning phase. This idea can be used in the second learning phase as well. However, the g Θ function's output can not be used as an input data point of the nonlinear SVM because it is in its feature space. It is desired to use kernel space optimization instead. The verification network's output can be transformed to a valid kernel function. Hence, the Gram matrix can be generated by calculating the verification network's for each pair. The SVMs can learn from a Gram matrix in kernel space, without features. This method can be used for linear kernel too, but the computational cost of this approach is larger because calculating the Gram matrix requires O(n 2) forward step in the neural network as it calculates all possible pairs while determining the pure features needs only O(n) forward step. The described model is an end-to-end neural SVM, which has an adaptive kernel. In the next section, the model is used in several experiments on different datasets, then compared to the end-to-end K-NN model described in BID26. 4.1 OMNIGLOT The Omniglot BID17 dataset is a set of handwritten characters from different alphabets. Each character is written 20 times by different people. Furthermore, the total number of characters is 1623 and the characters come from 50 different alphabets. FIG1 shows example images from the dataset. The dataset is collected via Amazon's Mechanical Turk. The evaluation method is the same as described in BID26 paper, the models' accuracies on this dataset are shown in TAB0. For the experiment characters were mixed independently from alphabets: the first 1150 characters were used for training the kernel in order to learn representation, the next 50 characters were used for validation to select the best model. The remaining items were used for testing, where n classes are chosen, and k samples were used for training the SVM. It is called n-way k-shot learning. Each test was run 10 times using different classes to get robust . During the training, no explicit data augmentation was used. The used model's g Θ 1 is identical to Convolutional Siamese Network BID16, which can be seen in FIG2. The only difference is the regularization, the original model used L2 weight decay, while this model uses dropout layers BID24 with 0.1 rates after every max pooling layer and before its last layer. The SVM's C parameter for regularization is 0.2. This model is trained for maximum 200 epochs with Adam optimizer BID15. Early stopping is used, which uses accuracy on "same or different class" task as stopping criteria. This slight modification in the training method in big performance improvement as seen in TAB0.The representation can be fine-tuned if the k-shot learning's training data is used for further fitting. This may in massive overfitting and it can't be prevented with cross-validation in case of one-shot learning. During fine-tuning, the model is trained for 10 epochs. The data for fine-tuning is generated as described in Section 3.3. This can not be applied for one-shot learning, where the same class pairs don't exist. For this purpose, the pair is created from the original image and its augmented version. The task of one-shot learning is poorly defined on audio data because one-shot can be 1 second or even 5 seconds as well, therefore it is required to redefine the task. In this paper k-sec learning is defined so that the length of the training data is k seconds regardless of the sample rate. Eventually, w len can be considered as a hyperparameter of the model, so optimal value of w len depends on the task as we will see. Furthermore, the length of each data point is exactly w len seconds, where w len ≤ k is satisfied. In addition, these data points can partially overlap, but k seconds length training data points mustn't overlap with evaluation points. Few seconds classification is an important task in real-world applications because it is exhausting to collect a large amount of data from speakers for robust classification. In this section, two scenarios are investigated: the first is a real-time application for speaker recognition, where k is 1 second, this can be considered as the upper limit of the online recognition. The second case is where k is 5 seconds, it is considered as an offline scenario. TIMIT BID7 ) is one of the most widely used English speech corpus. The dataset was originally designed for speech-to-text tasks. However, this dataset is perfect for speaker identification task too. The used dataset, which is projected from TIMIT contains audio files and their labels are the speakers. It contains 630 native speakers and the total number of sentences is 6300. Each speaker speaks for about 30 seconds. The official training set contains 462 people's voice. As a matter of fact, the training set is distinct from evaluation set regarding speakers, so neither of the training set speakers appears in the test set due to TIMIT is a speech-to-text task oriented dataset. This partitioning of the data makes the dataset unsuitable for a classical classification task, but it makes the TIMIT dataset perfect for k-sec learning task. There is no baseline known for k-sec learning problem on this dataset, so two different baseline models are introduced. In this experiment, the official training set is used for training the models to learn representation and the chosen subsets of the evaluation set are used to train the model for the k-sec learning problem. The evaluation is the same as in the previous section, it is done on 10 different subsets. For the neural models, the audio data is converted to a spectrogram, which can be handled as an image, see FIG3. Baseline models:1. The first classifier uses handcrafted features with ensembles of SVMs. The used features are aggregated MFCC BID13 and LPCC BID27. This classifier has two versions for different length of training data. The first version is optimized for 1-sec learning, which uses 0.3 sec long audio with 0.1 sec offset sliding window. The second version is optimized for longer training data, which used 3 sec long slices and the sliding window steps by 0.1 sec.2. The second model uses a neural network, which consists of convolutional layers (the architecture can be seen in FIG4) and a fully connected layer on the top of the network. It is pretrained on the training set and the fully connected layer is changed to fit the problem then the model is fine-tuned for the chosen classes with transfer learning. The idea of using different window length for different tasks can be used here too. On the other hand, the re- The used neural SVM model's feature extractor can be seen in FIG4. In the network, Batch Normalization layers BID14 are used after every convolution layer in order to promote faster convergence and dropout layers BID24 are also used with 0.1 rates after every max pooling layer to regularize the model. This model is trained for maximum 200 epochs with Adam optimizer BID15 ) and the best model has been selected with respect the same/different class accuracy for evaluation. The value of C is set to 15. As experiments with baseline models proved, optimizing sliding window length (w len) to the task may significantly improve accuracy. During the spectrogram generation, 64x64 pixel resolution images are created, which represents w len sec length audio, the exact value of w len in the experiments and the accuracies of the experiments can be seen in TAB1. Furthermore, a sliding window is used with 0.05 sec step size on the evaluation set, but 0.4 sec step size is used on training set due to computational complexity considerations. The (see TAB1) prove that the proposed task is complex enough for classical machine learning algorithms to not achieve satisfying accuracy and pure transfer learning not enough for suitable . However, the proposed method's accuracy is far better than baselines. There is no major surprise, it is designed to perform well on a few data. In this work, Siamese kernel SVM was introduced, which is capable of state-of-the-art performance on multiple domains on few-shot learning subject to accuracy. The key point of this model is combining Support Vector Machines' generalizing capabilities with Siamese networks one-shot learning abilities, which can improve the combined model's on the k-shot learning task. The main observation of this work is that learning representation for another model is much easier when the feature extractor is taught as an end-to-end version of the other model. In addition, parameterless models achieve the best on the previously defined problem, which makes SVMs an adequate choice for the task. This paper also introduced the concept of k-sec learning, which can be used for audio and video recognition tasks, and it gave a baseline for this task on the TIMIT dataset. The author hopes defining k-sec learning task encourage others to measure one-shot learning models' accuracy on various domains.

Represent the short summary for finding the corresponding scientific article (long)

The proposed method is an end-to-end neural SVM, which is optimized for few-shot learning.

represent the tldr for finding the corresponding scientific article

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.


Question: Represent the scientific article to find a short TLDR

Question: Most existing 3D CNN structures for video representation learning are clip-based methods, and do not consider video-level temporal evolution of spatio-temporal features. In this paper, we propose Video-level 4D Convolutional Neural Networks, namely V4D, to model the evolution of long-range spatio-temporal representation with 4D convolutions, as well as preserving 3D spatio-temporal representations with residual connections. We further introduce the training and inference methods for the proposed V4D. Extensive experiments are conducted on three video recognition benchmarks, where V4D achieves excellent , surpassing recent 3D CNNs by a large margin. 3D convolutional neural networks (3D CNNs) and their variants (; ; ; ; b) provide a simple extension from 2D counterparts for video representation learning. However, due to practical issues such as memory consumption and computational cost, these models are mainly used for clip-level feature learning instead of training from the whole video. In this sense, during training, the clip-based methods randomly sample a short clip (e.g., 32 frames) from the video for representation learning. During testing, they uniformly sample several clips from the whole video in a sliding window manner and calculate the prediction scores for each clip independently. Finally the prediction scores of all clips are simply averaged to yield the video-level prediction. Although achieving very competitive accuracy, these clip-based models ignore the video-level structure and long-range spatio-temporal dependency during training, as they only sample a small portion of the entire video. In fact, sometimes it could be very hard to recognize action class only with partial observation. Meanwhile, simply averaging the prediction scores of all clips could be also sub-optimal during testing. To overcome this issue, Temporal Segment Network (TSN) uniformly samples multiple clips from the entire video and uses their average score to guide back-propagation during training. Thus TSN is a video-level representation learning framework. However, the inter-clip interaction and video-level fusion in TSN is only performed at very late stage, which fails to capture finer temporal structures. In this paper, we propose a general and flexible framework for video-level representation learning, called V4D. As shown in Figure 1, to model long-range dependency in a more efficient and principled way, V4D is composed of two critical design: holistic sampling strategy and 4D convolutional interaction. We first introduce a video-level sampling strategy by uniformly sampling a sequence of short-term units covering the holistic video. Then we model long-range spatio-temporal dependency by designing a unique 4D residual block. Specifically, we present a 4D convolutional operation to capture inter-clip interaction, which could enhance the representation power of the original cliplevel 3D CNNs. The 4D residual blocks could be easily integrated into the existing 3D CNNs to perform long-range modeling more earlier and hierarchically than TSN. We also design a specific video-level inference algorithm for V4D. Specifically, we verify the effectiveness of V4D on three video action recognition benchmarks, Mini-Kinetics , Kinetics-400 and Something-Something-V1 . V4D structures achieve very competitive performance on these benchmarks and obtain evident performance improvement over their 3D counterparts. The architectures for video recognition can be roughly categorized into three groups: Two-stream CNNs, 3D CNNs, and long-term modeling framework. Two-stream architecture was first proposed by , where one stream is used for learning from RGB images, and the other one is applied for modeling optical flow. The produced by the two streams are then fused at later stages, yielding the final prediction. Two-stream CNNs have achieved impressive on various video recognition tasks. However, the main drawback is that the computation of optical flow often takes rather long time with expensive resource explored. Recent effort has been devoted to reducing the computational cost on modeling optical flow, such as (; ; . Two-stream input and fusion is a general method to boost the accuracy of various CNN structures, which is orthogonal with our proposed V4D. 3D CNNs have recently been proposed (; ; a; b;). By considering a video as a stack of frames, it is natural to utilize 3D convolutions directly on video data. However, 3D CNNs often have a larger number of model parameters, which require more training data to achieve high performance. Recent experimental on large scale benchmark of Kinetics-400 , as reported in (b;), show that 3D CNNs can surpass their 2D counterparts in most cases, even on par with or better than the two-stream 2D CNNs. It is noteworthy that most of 3D CNNs are clip-based methods, which means that they only explore a certain part of the holistic video. Long-term modeling frameworks have been developed for capture more complex temporal structure for video-level represenation learning. Back to the past, video compositional models were proposed to jointly model local video events in , where the temporal pyramid matching was combined with the bag-of-visual-words framework to capture long-term temporal structure. However, the rigid composition only works under some conditions, e.g. prefixed duration and anchor points in time. A mainstream method operated on a continuous video frame sequence with recurrent neural networks; with 2D CNNs for frame-level feature extraction. Temporal Segment Network (TSN) has been proposed to model video-level temporal information with a sparse sampling and aggregation strategy. TSN sparsely sampled frames from the whole video and these frames are modelled by the same CNN backbone. These scores are averaged to generate video-level prediction. Although originally designed for 2D CNNs, TSN can also be applied to 3D CNNs, which is set as one of the baselines in this paper. One obvious drawback of TSN is that due to the simple average aggregation, it can not model finer temporal structure. Temporal Relational Reasoning Network (TRN) ) models temporal segment relation by encoding individual representation of each segment with relation networks. TRN is able to model video-level temporal order but lacks capacity of capturing finer temporal structures. Our proposed V4D, however, significantly surpass these previous video-level learning methods on both appearance-dominated video recognition benchmark (e.g., Kinetics) and motion-dominated video recognition benchmark (e.g., Something-Something). V4D framework is able to model both short-term and long-term temporal structures with a unique design of 4D residual block. In this section, we introduce novel Video-level 4D Convolution Neural Networks, namely V4D, for video action recognition. This is the first attempt to design 4D convolutions for RGB-based video recognition. Previous methods, such as , utilize 4D CNNs to process videos of point cloud so that their input is 4D data. Instead, our V4D processes videos of RGB frames so that our input is 3D data. This basically makes the methods and tasks quite different. Existing 3D CNNs take a short-term snippet as input, without considering the evolution of 3D spatiotemporal features for video-level representation. Wang et al. (2018b);; proposed self-attention mechanisms to model non-local spatio-temporal features, but these methods are originally designed for clip-based 3D CNNs. It remains unclear how to incorporate such operations on holistic video representation, and whether such operations are useful for video-level learning. Our goal is to model 3D spatio-temporal features globally, which can be implemented in a higher dimension. In this work, we introduce new Residual 4D Blocks, which allow us to cast 3D CNNs into 4D CNNs for learning long-range interactions of the 3D features, ing in a "time of time" video-level representation. To model meaningful video-level representation for action recognition, the input to the networks has to cover the holistic duration of a given video, and at the same time preserve short-term action details. A straightforward approach is to implement per-frame training of the networks yet this is not practical by considering the limit of computation resource. In this work, we uniformly divide the whole video into U sections, and select a snippet from each section to represent a short-term action pattern, called "action unit". Then we have U action units to represent the holistic action in a video. Formally, we denote the video-level input V = {A 1, A 2, ..., A U}, where A i ∈ R C×T ×H×W. During training, each action unit A i is randomly selected from each U section. During testing, the center of each A i locates exactly at the center of each section. 3D Convolutional kernels have been proposed for years, and are powerful to model short-term spatiotemporal features. However, the receptive fields of 3D kernels are often limited due to the small sizes of kernels, and pooling operations are often applied to enlarge the receptive fields, ing in a significant cost of information loss. This inspired us to develop new operations which are able to model both short-and long-term spatio-temporal representations simultaneously, with easy implementations and fast training. From this prospective, we propose 4D convolutions for better modeling long-range spatio-temporal interactions. Formally, we denote the input to 4D convolutions as a tensor V of size (C, U, T, H, W), where C is number of channel, U is the number of action units (the fourth dimension in this paper), T, H, W are temporal length, height and width of the action units, respectively. We omit the batch dimension for simplicity. Following the annotations from , a pixel at position (u, t, h, w) of the jth channel in the output is denoted as o uthw j, a 4D convolution operation can be formulated as: where b j is the bias term, c is one of the C in input channels of the feature maps from input V, S × P × Q × R is the shape of 4D convolutional kernel, W spqr jc is the weight at the position (s, p, q, r) of the kernel, corresponding to the c-th channel of the input feature maps and j-th channel of the output feature maps. We visualize the implementation of 4D kernels which is compared to that of 3D kernels. U denotes the number of action units, each of which has a shape of T, H, W. Channel and batch dimensions are omitted for clarity. The kernels are colored in Blue, with the center of each kernel colored in Green. Convolution operation are linear, and the sequence of sum operations in E.q. 1 are exchangeable. Thus we can generate E.q. 2, where the expression in the parentheses can be implemented by 3D convolutions. This is how we implement 4D convolutions with 3D convolutions while most deep learning libraries do not provide 4D convolutional operations. With the 4D convolutional kernel, the short-term 3D features of an individual action unit and longterm temporal evolution of multiple action units can be modeled simultaneously in the 4D space. Compared to 3D convolutions, the proposed 4D convolutions are able to model videos in a more meaningful 4D feature space that enables it to learn more complicated interactions of long-range 3D spatio-temporal representations. However, 4D convolutions inevitably introduce more parameters and computation cost. In practice, for example, a 4D convolutional kernel of In this section, we aim to incorporate 4D convolutions into existing CNN architecture for action recognition. To fully utilize current state-of-the-art 3D CNNs, we propose a new Residual 4D Convolution Block, by designing a 4D convolution in a residual structure . This allows it to aggregate both short-term 3D features and long-term evolution of the spatio-temporal representations for video-level action recognition. Specifically, we define a permutation function The Residual 4D Convolution Block can be formulated as: where, and U is merged into batch dimension so that X 3D, Y 3D can be directly processed by standard 3D CNNs. Note that we employ ϕ to permute the dimensions of X 3D from U ×C ×T ×H ×W to C ×U ×T ×H ×W so that it can be processed by 4D convolutions. Then the output of 4D convolution is permuted back to 3D form so that the output dimensions are consistent with X 3D. Batch Normalization and ReLU activation are then applied. The detailed structure is shown in Figure 1. Theoretically, any 3D CNN structure can be cast to 4D CNNs by integrating our 4D Convolutional Blocks. As shown in previous works (; ; b;), better performance can be obtained by applying 2D convolutions at lower layers and 3D convolutions at higher layers of the 3D networks. In our framework, we utilize the "Slowpath" from as our backbone, denoted as I3D-S. Although the original "Slowpath" is designed for ResNet50, we can extend it to I3D-S ResNet18 for further experiments. The detailed structures of our 3D backbones are shown in Table 1: We use I3D-Slowpath from as our backbone. The output size of an example is shown in the right column, where the input has a size of 4×224×224. No temporal degenerating is performed in this structure. Training. As shown in Figure 1, the convolutional part of the network is composed of 3D convolution layers and the proposed Residual 4D Blocks. Each action unit is trained individually and in parallel in the 3D convolution layers, which share the same parameters. These individual 3D features computed from each action units are then fed to the Residual 4D Block for modelling the long-term temporal evolution of the consecutive action units. Finally, global average pooling is applied on the sequence of all action units to form a video-level representation. Inference. Given U action units {A 1, A 2, ..., A U} of a video, we denote U train as the number of action units for training and U inf er as the number of action units for inference. U train and U inf er are usually different because computation resource is limited in training, but high accuracy is encouraged in inference. We develop a new video-level inference method, which is shown in Algorithm 1. The 3D convolutional layers are denote as N 3D, followed by the proposed 4D Blocks, N 4D. } are processed by N 4D to form a prediction score set {P 1, P 2, ..., P U combined}; 4 {P 1, P 2, ..., P U combined} are averaged to give the final video-level prediction. In this section, we will show that the proposed V4D can be considered as a 4D generalization of a number of recent widely-applied methods, which may partially explain why V4D works practically well on learning meaningful video-level representation. Temporal Segment Network. Our V4D is closely related to Temporal Segment Network (TSN). Although originally designed for 2D CNN, TSN can be directly applied to 3D CNN to model video-level representation. It also employs a video-level sampling strategy with each action unit named "segment". During training, each segment is calculated individually and the prediction scores after the fully-connected layer are then averaged. Since the fully-connected layer is a linear classifier, it is mathematically identical to calculating the average before the fully-connected layer (similar to our global average pooling) or after the fully-connected layer (similar to TSN). Thus our V4D can be considered as 3D CNN + TSN if all parameters in 4D Blocks are assigned zero. Dilated Temporal Convolution. One special form of 4D convolution kernel, k × 1 × 1 × 1, is closely related to Temporal Dilated Convolution . The input tensor V can be considered as a (C, U × T, H, W) tensor when all action units are concatenated along the temporal dimension. In this case, the k × 1 × 1 × 1 4D convolution can be considered as a dilated 3D convolution kernel of k × 1 × 1 with a dilation of T frames. Note that the k × 1 × 1 × 1 kernel is just the simplest form of our 4D convolutions, while our V4D architectures utilize more complex kernels and thus can be more meaningful for learning stronger video representation. Furthermore, our 4D Blocks utilize residual connections, ensuring that both long-term and short-term representation can be learned jointly. Simply applying the dilated convolution might discard the short-term fine-grained features. We conduct experiments on three standard benchmarks: Mini-Kinetics , Kinetics-400 , and Something-Something-v1 . Mini-kinetics dataset covers 200 action classes, and is a subset of Kinetics-400. Since some videos are no longer available for Kinetics dataset, our version of Kinetics-400 contains 240,436 and 19,796 videos in the training and validation subsets, respectively. Our version of Mini-kinetics contains 78,422 videos for training, and 4,994 videos for validation. Each video has around 300 frames. Something-Somethingv1 contains 108,499 videos totally, with 86,017 for training, 11,522 for validation, and 10,960 for testing. Each video has 36 to 72 frames. We use pre-trained weights from ImageNet to initialize the model. For training, we adapt the holistic sampling strategy mentioned in section 3.1. We uniformly divide the whole video into U sections, and randomly select a clip of 32 frames from each section. For each clip, by following the sampling strategy in , we uniformly sample 4 frames with a fixed stride of 8 to form an action unit. We will study the impact of U in the following experiments. We first resize every frame to 320 × 256, and then randomly cropping is applied as Wang et al. (2018b). Then the cropped region is further resized to 224 × 224. We utilize SGD optimizer with an initial learning rate of 0.01, weight decay is set to 10 −5 with a momentum of 0.9. The learning rate drops by 10 at epoch 35, 60, 80 and the model is trained for 100 epochs in total. To make a fair comparison, we use spatial fully convolutional testing by following Wang et al. (2018b);;. We sample 10 action units evenly from a full-length video, and crop 256 × 256 regions to spatially cover the whole frame for each action unit. Then we apply the proposed V4D inference. Note that, for the original TSN, 25 clips and 10-crop testing are used during inference. To make a fair comparison between I3D and our V4D, we instead apply this 10 clips and 3-crop inference strategy for TSN. Results and Effectiveness. To verify the effectiveness of V4D, we compare it with the clip-based method I3D-S, and video-based method TSN+3D CNN. To compensate the extra parameters introduced by 4D blocks, we add a 3 × 3 × 3 residual block at res4 for I3D-S for a fair comparison, denoted as I3D-S ResNet18++. As shown in Table 2a, even V4D uses 4 times less frames than I3D-S during inference and with less parameters than I3D-S ResNet18++, V4D still obtain a 2.0% higher top-1 accuracy than I3D-S. Comparing with current state-of-the-art video-level method TSN+3D CNN, V4D significantly outperforms it by 2.6% top-1 accuracy, by using the same protocol for training and inference. Different Forms of 4D Convolution Kernels. As mentioned, our 4D convolution kernels can use 3 typical forms: simplicity, and apply a single 4D block at the end of res4 in I3D-S ResNet18. As shown in Table 2c, V4D with 3 × 3 × 3 × 3 kernel can achieve the highest performance. However, by considering the trade-off between model parameters and performance, we use the 3 × 3 × 1 × 1 kernel in the following experiments. Position and Number of 4D Blocks. We evaluate the impact of position and number of 4D Blocks for our V4D. We investigate the performance of V4D by using one 3 × 3 × 1 × 1 4D block at res3, res4 or res5. As shown in Table 2d, a higher accuracy can be obtained by applying the 4D block at res3 or res4, indicating that the merged long-short term features of the 4D block need to be further refined by 3D convolutions to generate more meaningful representation. Furthermore, inserting one 4D block at res3 and one at res4 can achieve a higher accuracy. Number of Action Units U. We further evaluate our V4D by using different numbers of action units for training, with different values of hyperparameter U. In this experiment, one 3 × 3 × 1 × 1 Residual 4D block is inserted at the end of res4 of ResNet18. As shown in Table 2e, U does not have a significant impact on the performance, which suggests that: V4D is a video-level feature learning model, which is robust against the number of short-term units; an action generally does not contain many stages, and thus increasing U is not helpful. Also, the number of action units increasing means that the fourth dimension is increasing, which needs a larger 4D kernel to cover the long-range evolution of spatio-temporal representation. Comparison with State-Of-The-Art. We compare our V4D with previous state-of-the-art methods on Mini-Kinetics. 4D Residual Blocks are added into every other 3D residual blocks in res3 and res4. With much fewer frames utilized during training and inference, our V4D ResNet50 achieves a higher accuracy than all reported on this benchmark, which is even higher than 3D ResNet101 with 5 Compact Generalized Non-local Blocks. Note that our V4D ResNet18 can achieve a higher accuracy than 3D ResNet50, which further verify the effectiveness of our V4D structure. We further conduct experiments on large-scale video recognition benchmark, Kinetics-400, to evaluate the capability of our V4D. To make a fair comparison, we utilize ResNet50 as backbone for V4D. The training and inference sampling strategy is identical to previous section, except that each action unit now contains 8 frames instead of 4. We set U = 4 so that there are 8 × 4 frames in total for training. Due to the limit of computation resource, we choose to train the model in multiple stages. We first train the 3D ResNet50 backbone with 8-frame inputs. Then we load the 3D ResNet50 weights to Model Backbone Ttrain × Utrain T inf er × U inf er × #crop top-1 top5 S3D S3D Inception 64 × 1 N/A 78.9 -I3D 3D ResNet50 32 × 1 32 × 10 × 3 75.5 92.2 I3D 3D ResNet101 Table 3: Comparison with state-of-the-art on Mini-Kinetics. T indicates temporal length of each action unit. U represents the number of action units. V4D ResNet50, with all 4D Blocks fixed to zero. The V4D ResNet50 is then fine-tuned with 8 × 4 input frames. Finally, we optimize all 4D Blocks and train the V4D with 8 × 4 frames. As shown in Table 4, our V4D achieves competitive on Kinetics-400 benchmark. Model Backbone top-1 top-5 ARTNet with TSN (a) ARTNet ResNet18 70.7 89.3 ECO BN-Inception+3D ResNet18 70.0 89.4 S3D-G S3D Inception 74.7 93.4 Nonlocal Network (a) 3D ResNet50 76.5 92.6 SlowFast SlowFast ResNet50 77.0 92.6 I3D I3D Inception 72.1 90.3 Two-stream I3D I3D Inception 75.7 92.0 I3D-S Slow pathway ResNet50 74.9 91.5 V4D(Ours) V4D ResNet50 77.4 93.1 Table 4: Comparison with state-of-the-art on Kinetics. Something-Something is a rather different dataset compared to Mini-Kinetics and Kinetics. Instead of enhancing high-level action concepts, Something-Something focuses on modeling temporal information and motion. The is much cleaner than Kinetics but the motions of action categories are much more complicated. Each video in Something-Something contains one single and continuous action with clear start and end on temporal dimension. Comparison with Prior Works. As shown in Table 4.4, our V4D achieves competitive on the Something-Something-v1. We use V4D ResNet50 pre-trained on Kinetics for experiments. Model Backbone top-1 MultiScale TRN BN-Inception 34.4 ECO BN-Inception+3D ResNet18 46.4 S3D-G S3D Inception 45.8 Nonlocal Network+GCN 3D ResNet50 46.1 TrajectoryNet S3D ResNet18 47.8 V4D(Ours) V4D ResNet50 50.4 Table 5: Comparison with state-of-the-art on Something-Something-v1. Temporal Order As shown in , the performance can drop considerably by reversing the temporal order of short-term 3D features, which demonstrates that the strong temporal order information has been learned by 3D CNNs. For our V4D, there are two levels of temporal order, a short-term order and a long-term order. As shown in Table 6, either by reversing the frames inside each action unit or by reversing the sequence of action units, the top-1 accuracy drops significantly, which indicates that our V4D is able to capture both long-term and short-term temporal order. Table 6: V4D is able to capture the arrow of time. We have introduced new Video-level 4D Convolutional Neural Networks, namely V4D, to learn strong temporal evolution of long-range spatio-temporal representation, as well as retaining 3D features with residual connections. In addition, we further introduce the training and inference methods for our V4D. Experiments were conducted on three video recognition benchmarks, where our V4D achieved the state-of-the-art . A APPENDIX In order to check the generalization ability of our proposed V4D, we also conduct experiments for untrimmed video classification. To be specific, we choose ActivityNet v1.3 , which is a large-scale untrimmed video dataset, containing videos of 5 to 10 minutes and typically large time lapses of the videos are not related with any activity of interest. We adopt V4D ResNet50 to compare with previous works. During inference, Multi-scale Temporal Window Integration is applied following. The evaluation metric is mean average precision (mAP) for action recognition. Note that only RGB modality is used as input. Model Backbone mAP BN-Inception 79.7 Inception V3 83.3 TSN-Top3 Inception V3 84.5 V4D(Ours) V4D ResNet50 88.9 Table 7: Comparison with state-of-the-art on ActivityNet v1.3. We implement 3D CAM based on , which was originally implemented for 2D cases.

Represent the short summary for finding the corresponding scientific article (long).

A novel 4D CNN structure for video-level representation learning, surpassing  recent 3D CNNs.

We present a new deep architecture, VarPSOM, and its extension to time series data, VarTPSOM,  which achieve superior clustering performance compared to current deep clustering methods on static and temporal data.