A Generative Approach to Zero-Shot and Few-Shot Action Recognition
Ashish Mishra Vinay Kumar Verma M Shiva Krishna Reddy Arulkumar S Piyush Rai Anurag Mittal
cs.CV











 
A Generative Approach to Zero-Shot and Few-Shot Action Recognition
     Ashish Mishra^*,1 , Vinay Kumar Verma^†,1, M Shiva Krishna Reddy^*, Arulkumar S^*
   Piyush Rai^† and Anurag Mittal^*

 ^*Indian Institute of Technology Madras     ^†Indian Institute of Technology Kanpur

{vkverma,piyush}@cse.iitk.ac.in, {mishra,shiva,aruls,amittal}@cse.iitm.ac.in

    December 30, 2023
===============================================================================================================================================================================================================================================================================================

empty



   We present a generative framework for zero-shot action recognition where some of the possible action classes do not occur in the training data. Our approach is based on modeling each action class using a probability distribution whose parameters are functions of the attribute vector representing that action class. In particular, we assume that the distribution parameters for any action class in the visual space can be expressed as a linear combination of a set of basis vectors where the combination weights are given by the attributes of the action class. These basis vectors can be learned solely using labeled data from the known (i.e., previously seen) action classes, and can then be used to predict the parameters of the probability distributions of unseen action classes. We consider two settings: (1) Inductive setting, where we use only the labeled examples of the seen action classes to predict the unseen action class parameters; and (2) Transductive setting which further leverages unlabeled data from the unseen action classes. Our framework also naturally extends to few-shot action recognition where a few labelled examples from unseen classes are available. Our experiments on benchmark datasets (UCF101, HMDB51 and Olympic) show significant performance improvements as compared to various baselines, in both standard zero-shot (disjoint seen and unseen classes) and generalized zero-shot learning settings. 
   







§ INTRODUCTION

[1]Both authors contributed equally.
Action Recognition is an important problem in Computer Vision in which knowledge about a sequence of actions is learned from a large collection of video clips.  It is a challenging task due to the inherent variability in actions, non-deterministic occlusion patterns, abrupt changes in illumination, cluttered dynamic background, and noisy videos. Knowledge about an action is inferred usually by learning from the labelled data in a supervised manner.  Even as more complex models are being built, it is a common observation that the number of categories of actions is progressively increasing (for example, one of the earliest benchmark datasets KTH has 6 categories while Olympic, HMDB and UCF datasets have 16, 51, and 101 categories, respectively).  Consequently, annotating videos of this growing number of categories can be a very cumbersome task and consequently restricts the scalability of a fully supervised action recognition for a large number of categories.


   


To circumvent this problem, Zero-Shot Learning (ZSL) of actions has been actively pursued <cit.>.  In the conventional Action Recognition framework, only the classes present in the training data can be recognized by the model during the test phase. In Zero-Shot Learning, however, the model is expected to recognize and categorize action classes that did not appear in the training phase at all. The information about the unseen classes is provided via other modalities such as language in the form of textual descriptions, word2vec <cit.> or human annotated attributes.  Essentially, the model has to learn to recognize the unseen action classes based on the knowledge acquired from the data instances of the seen action classes. Zero-shot learning is typically defined in two settings: (1) the conventional setting, in which set of classes for the training and test instances are disjoint (Y_tr∩ T_te=∅); and (2) the generalized zero-shot (GZSL) setting, in which the set of classes for the training and test instances may have an overlap <cit.>. The generalized zero-shot setting is considered much harder than standard setting (disjoint setting) since the learned models tend to be biased towards predicting seen classes at training time (as they are learned solely from the unseen class training data). While much of the prior work in ZSL has focused on the conventional setting, the focus has recently shifted to the more realistic GZSL setting. 



In this work, we present a simple generative approach for zero shot action recognition, which works in both standard as well as generalized ZSL setting. Our approach models each action class as a probability distribution in the visual space where the parameters of this distribution are assumed to be a linear combination of a set of “basis” parameters, where the combination weights are given by the (known) attribute vector of that class. This is akin to assuming that each action class can be represented as a combination of a set of “prototype” action classes. The complete architecture of our model is shown in Fig. <ref>.





Once the basis vectors are learned using the training instances from seen action classes, the parameters of an unseen action class distribution can be easily computed via a weighted combination of the learned basis vectors, with weights being the attributes of the respective unseen class. The loss function (More details in Methodology section) is formulated in such a way that, for each seen action class, the weighted combination of the basis vectors is close to the maximum likelihood estimate (MLE) of the class distribution's parameters. The MLE estimate can also be replaced by maximum-a-posteriori (MAP) estimate. Our approach is akin to the one proposed recently in <cit.> for zero-shot learning, though our focus an application is specifically the zero-shot action recognition problem. In addition, we add an additional “reverse direction” regularizer to encourage reconstruction ability of the class attribute vectors from the parameters of the seen class distributions so as to minimize the information loss. Note that this is akin to an autoencoder (cf, Fig. <ref>). One of the appealing aspects of our model is that it admits a simple closed-form solution.

Our main contributions can be summarized as follows

 
  * We provide a probabilistic generative approach for zero-shot learning (ZSL) where each action class is represented by a Gaussian distribution (although the Gaussian can be replaced by other distributions without changing the rest of our model).
 
  * We show that our approach, although simple, generalizes well to the unseen classes in the inductive setting and improves over the state-of-the-art.
 
  * We show that our approach can be easily generalized to the transductive setting where unlabeled data from unseen classes are available at training time.
 
  * Our model also naturally extends to be “few-shot learning” setting where a few examples of each unseen class are available as well. In particular, the parameters of the class distribution can be updated easily given a few additional labeled examples from that class. Through extensive experimentation on three benchmark datasets, we show that our simple approach gives significant performance gains in all three settings over the state-of-the-art methods. 
 
  * Finally, since our approach is generative, we can also synthesize novel examples for any unseen class by sampling from the respective class distribution. Since we can now have labeled data from seen as well as unseen classes, it is possible to train a classifier in the generalized zero-shot setting which is much harder than the standard (disjoint) setting.





§ METHODOLOGY


For the zero-shot action recognition setting, we denote the total number of seen action classes by S and the total number of unseen action classes by U. We take a generative classification based approach to the action recognition problem where we assume that the data instances of each action class (seen/unseen) c are generated by a distribution p( 𝐱|θ_c). Without loss of generality, and for simplicity of exposition, we will assume these distributions to be Gaussians (note that our approach can be used with other distributions as well). In the Gaussian case, the parameters θ_c consist of the mean vector μ_c∈ℝ^D and a diagonal covariance  matrix Σ_c = diag(σ_c^2), where σ_c^2∈ℝ_+^D. We assume a diagonal covariance matrix to reduce the total number of parameter estimated and prevent overfitting especially when the number of examples from each class is small. However, other forms for the covariance matrix can also be used. 

Given labeled data from the seen classes, it is straightforward to estimate the parameters μ_c,σ_c using Maximum Likelihood Estimation (MLE) or Maximum-a-Posteriori (MAP) estimation. For example, using MLE, the mean is estimated as μ_c=1/N_c∑_i=1^N_c𝐱_𝐢 and σ_c^2=diag(1/N_c∑_i=1^N_c(𝐱_𝐢-μ_c)(𝐱_𝐢-μ_c)^⊤) where N_c denotes the number of labeled examples from class c.

However, this approach cannot be used to estimate the parameters θ_c (c=S+1,...,S+U) of unseen classes due to unavailability of labeled data corresponding to unseen classes. To resolve this problem, we model the parameters θ_c = (μ_c,σ_c^2) of each seen/unseen clas  as a function of the respective class attribute vector 𝐚_𝐜, i.e., θ_c = f(𝐚_𝐜). In the zero-shot learning setting, the class attribute vector 𝐚_𝐜∈ℝ^K is either provided by a human expert or as the word2vec embedding of the name of the action.

The function f can be linear or nonlinear and can be learned using the labeled data instances of seen classes in visual feature space. Once learned, the function f can be used to predict θ_c for all the unseen class actions c=S+1,…,S+U using their respective class attributes. 

A simple choice of f is a linear model that maps the class attributes 𝐚_𝐜 to the class parameters θ_c. In the Gaussian class distribution case, for the mean μ_c, such a linear function f can be defined as


    μ_c = f_μ(𝐚_𝐜) = 𝐖_μ𝐚_𝐜

Note that the above linear model represents the mean μ_c∈ℝ^D as a weighted linear combination of K basis vectors 𝐖_μ =[𝐰_μ_1,𝐰_μ_2,..,𝐰_μ_𝐊]∈ℝ^D× K is a set of learned basis vectors in the visual space. 

The basis vectors 𝐖_μ can be learned using the seen class training data. In particular, given the empirical estimates μ̂_c, c=1,…,S of means of the seen class distributions, we can use (𝐚_𝐜,μ̂_c) as “training data” to learn the regression model 𝐖_μ that maps 𝐚_𝐜 to μ̂_c.

While the above model can be seen as mapping the class attribute vector 𝐚_𝐜 to the class mean μ_c, we further impose the condition that the class means can also be used to reconstruct the class attribute vector via a “reverse map” akin to an autoencoder, i.e., 𝐚_𝐜 = 𝐖_μ^𝐓 μ_𝐜, which leads to

    μ_c = 𝐖_μ 𝐚_𝐜 = 𝐖_μ 𝐖_μ^⊤μ_𝐜


A similar procedure can be employed for learning the mapping from the class attributes 𝐚_𝐜 to the variance parameters σ_c^2 of the distribution of class c via another set of basis vectors 𝐖_σ^2. Sections <ref> and <ref> provide more details.
























 Once the basis vectors 𝐖_μ,𝐖_σ^2 (which define the functions f_μ and f_σ) are learned, we can use them to estimate the parameters (e.g., μ_c,σ_c^2) of the distribution of each unseen class. For example, given the class attribute vector 𝐚_𝐜 of an unseen class c=S+1,…,S+U, we can estimate μ_c simply as μ_c = 𝐖_μ𝐚_𝐜.
 
 The mapping f (which is essentially a regression model) from the class attribute vector to the parameters of the class distribution can be linear or nonlinear. We describe both these cases in the next two sections.
 


 §.§ Linear Regression
 

Given the labeled data from seen classes c=1,...,S, we can estimate their class distribution parameters using MLE. We can then learn the functions f_μ and f_σ^2 using training data of the form (𝐚_𝐜,μ_𝐜)_c=1^S and (𝐚_𝐜,σ^2_𝐜)_c=1^S. In the linear regression approach μ_c = f_μ(𝐚_𝐜) and σ_c^2 = f_σ(𝐚_𝐜), we assume the functions f_μ and f_σ to be linear projections with weight matrices, 𝐖_μ and 𝐖_σ, making this problem equivalent to the following regression problem:

    μ_c=𝐖_μ𝐚_𝐜   s.t.   𝐚_𝐜=𝐖_μ^𝐓μ_𝐜


    ρ_c=logσ^2_c=𝐖_σ^2𝐚_𝐜   s.t.   𝐚_𝐜=𝐖_σ^2^𝐓σ^2_𝐜


The projection matrices 𝐖_μ and 𝐖_σ^2 can be easily learned using a multi-output ridge regression problem with training data (𝐚_𝐜,μ_𝐜)_c=1^S and (𝐚_𝐜,σ^2_𝐜)_c=1^S. These problems have simple closed form solution and we omit the equations here for brevity. We give details equations for the nonlinear case, as shown below.



 §.§ Nonlinear Regression


For the non-linear regression, we first map the attributes {𝐚_𝐜}_c=1^S to the kernel space using the kernel function k which is defined as a nonlinear mapping ϕ. Using the Representer theorem <cit.>, we can re-formulate the regression problem in kernel space as given in Eq. <ref>. Note that instead of computing the ϕ( 𝐚_𝐜 ) explicitly, we have to compute only the dot product ϕ(𝐚_𝐜)^Tϕ(𝐚_𝐜')=k(𝐚_𝐜,𝐚_𝐜') for the non-linear mapping of the two class c and c'. 
Let 𝐊 be the kernel matrix of size S× S containing pairwise similarities of the attributes of the seen classes, 𝐌 be the D× S matrix containing the means of the distributions of all the seen classes, then the attribute to mean nonlinear mapping can be learned by solving the following problem

    min_𝐖_μ||𝐌 - 𝐖_μ𝐊|| ^2_F +λ_μ||𝐖_μ||^2_2
    s.t.   𝐊=𝐖_μ^*𝐌


Eq <ref> shows our main objective function. Here the first term can be interpreted as learning an optimal weight matrix that projects the attribute space to the visual space using the kernel regression. The second term ensures that we can reconstruct the attribute vector from the visual space and acts as a regularization term. Akin to an autoencoder <cit.>, we assume the two mappings to be reverse of each other


    𝐖_μ^*=𝐖_μ^T

Therefore the complete objective can be written as:

    𝐖^*_μ=_𝐖_μ||𝐌 - 𝐖_μ𝐊||^2_F+λ_μ||𝐖μ||^2_2
    
    +λ_1||𝐊-𝐖_μ^𝐓𝐌||_F^2


The next section provides details of the optimization procedure used for solving Eq. <ref>


  §.§.§ Optimization

Noting Tr(𝐊)=Tr(𝐊^T) and Tr(𝐖_μ^T 𝐌)=Tr( 𝐌^T 𝐖_μ), Eq. <ref> can be written as:


    𝐖^*_μ= _𝐖_μ||𝐌 - 𝐖_μ𝐊||^2_F+λ_μ||𝐖μ||^2_2
    
    +λ_1||𝐊^T-𝐌^T𝐖_μ||_F^2

Taking the derivative of Eq. <ref> and equating to zero we have.

    𝐌𝐌^T𝐖_μ+ 𝐖_μλ_1 𝐊𝐊^T+λ_μ 𝐖_μ=(1+λ_1) 𝐌𝐊^T


    𝐌𝐌^T𝐖_μ+𝐖_μ (λ_1𝐊𝐊^T+λ_μ) =(1+λ_1)𝐌𝐊^T

The above equation has the form


    𝐀𝐖+𝐖𝐁=𝐂


This is a well-known Sylvester equation which can be solved using the Bartels-Stewart algorithm <cit.> efficiently, and several off-the-shelf solvers exist (we used a MATLAB implementation for the same). The various quantities in the above equation are defined as

    𝐀=𝐌𝐌^T


    𝐁=λ_1𝐊𝐊^T+λ_μ


    𝐂=(1+λ_1)𝐌𝐊^T

Likewise, the nonlinear model f_σ^2_i can be learned by solving:



    𝐖^*_σ^2=_ 𝐖 _σ^2||𝐑 - 𝐖_σ^2𝐊||^2_F+λ_σ^2|| 𝐖_σ^2||^2_2
     +λ_2|| 𝐊-𝐖_σ^2^T 𝐑||_F^2


Again, taking derivatives and setting to zero gives

    𝐑𝐑^T𝐖_σ^2+𝐖_σ^2 (λ_2𝐊𝐊^T+λ_σ^2) =(1+λ_2)𝐑𝐊^T

The above equation is also in the form of 𝐀𝐖+𝐖𝐁=𝐂

    𝐀=𝐑𝐑^T


    𝐁=λ_2 𝐊𝐊^T+λ_σ^2


    𝐂=(1+λ_2) 𝐑𝐊^T

Given the learned parameters 𝐖_μ_c and 𝐖_σ^2_c, the parameters of data distribution for unseen classes c=S+1,…,S+U are estimated as:

    μ_c = 𝐖_μ𝐤_𝐜,   &  σ^2_c =exp(ρ_c)=exp( 𝐖_σ^2𝐤_𝐜)

Where 𝐤_𝐜=[𝐤(𝐚_𝐜,𝐚_1),...,𝐤(𝐚_𝐜,𝐚_𝐒)] denotes an S× 1 vector of kernel-based similarities of the class attribute vectors of the unseen class c with the class attribute vectors of all the seen classes.

In the aforementioned procedure for estimation of the unseen class distribution parameters uses only seen class labelled data. In this setting, the unseen classes unlabeled data have not been used. This setting is called as an inductive setting. If we have access to the unseen classes test instances at the training time, we can use these to improve the estimation of distribution parameters of unseen classes. This is the transductive setting which we describe next.



 §.§ Transductive setting

One of the unique advantages of the proposed generative approach is that unlabeled data from unseen classes can be leveraged to improve the parameter estimates (μ_c and σ_c). In zero-shot learning, training and test data could possibly come from different domains. Therefore, it is very likely that parameters learned in the training, will not work well for the test data. This phenomenon is called domain shift. An illustrative view of the domain shift can be seen in Fig. <ref>. One way to overcome this issue is to use unlabeled data to further fine-tune the parameters learned by the inductive approach which only uses the labeled data from the seen action classes. In the transductive setting <cit.>, we assume that the test data is also available at the training time. This data can help mitigate the bias towards the seen classes. In this work, we handle the domain shift problem by initializing the parameters μ_c,σ_c using the learned basis vectors from the inductive learning phase, which are then fine-tuned using the unlabeled test data from the unseen classes using the an Expectation-Maximization (EM) algorithm.



Since each class distribution is assumed to be a Gaussian, this EM based procedure is equivalent to a Gaussian mixture model (GMM) on the unlabeled test data ( 𝐱_𝐧)_n=1^N_u from unseen classes. This GMM has U mixture components, with each corresponding to an unseen class and is initialized by the estimated parameters of unseen classes (μ_c,σ^2_c)_c=S+1^S+U  in the inductive setting. 
The procedure for transductive setting can be briefly summarized as follows


  * Initialize: Let the initial estimate of the unseen class parameters be Θ=(μ_c,σ^2_c)_c=S+1^S+U where μ_c =  𝐖_μ 𝐚_𝐜, σ^2_c = exp(𝐖_σ^2 𝐚_𝐜). Here 𝐖_μ and 𝐖_σ^2 are estimated from seen class data using equations <ref>, <ref> (assuming we have used the nonlinear regression model in the inductive phase). 

  * Expectation Step: Infer the probabilities for each example 𝐱_𝐧 belonging to each of the unseen classes c=S+1,...,S+U as


    p(y_n=c| 𝐱_𝐧,θ)∝𝒩(𝐱_𝐧|μ_c,σ^2_c)

where the class priors p(c) are assumed to be uniform.

  * Maximization Step: Use the inferred class labels to re-estimate Θ=(μ_c,σ^2_c)_c=S+1^S+U. These updates have closed form solution as in the standard GMM.

  * Go to step 2 if not converged.





 §.§ Few-shot Action Recognition

In few-shot action recognition, we have a small number of labeled examples for each of the unseen classes. Since our method assumes a Gaussian distribution for each class, we can easily extend our zero-shot action recognition method to few-shot action recognition. To this end, we treat the initial estimate obtained using the previous approach as the prior. Due to the conjugate nature of the Gaussian, we can update the estimates (μ_c,σ^2_c)_c=S+1^S+U obtained from zero-shot action recognition method in a straightforward manner when such labeled data for unseen classes is provided. In particular, given a small number of labeled data ( 𝐱_𝐧)_n=1^N_c for unseen class c the parameters of this class can be directly updated as:

    μ^FS_c = μ+∑_n=1^N_c 𝐱_𝐧/1+N_c


    σ^2(FS)_c=(1/σ^2_c+N_c/σ^2*)^-1

where σ^2*=1/N_c∑_n=1^N_c(𝐱_𝐧-μ_c)^2 denotes empirical variance of N_c observations from the unseen class c.



 §.§ Extension to Other Distributions


Finally, we would like to emphasize that although we have consider Gaussians to model each class, our approach applies to any parameteric distribution p(𝐱|θ_c) as it essentially boils down to learning a mapping from the class attribute vectors 𝐚_c to the distribution parameters θ_c. The choice of a Gaussian or an exponential family distribution <cit.>, due to conjugacy, makes our estimation procedure particularly simple in the transductive and few-shot settings, but our framework is not restricted to these. Other density estimation methods such as deep generative models can also be used <cit.>.



§ RELATED WORK

ZSL can be viewed as an interplay of three subproblems: a visual representation of data instances (feature representation), semantic representation of all classes such as word2vec representation <cit.>, and learning a function which establishes the relationship between visual representations and semantic representations of each class<cit.>.

For visual (or feature) representation of class instance, popular hand-crafted features such as HOG <cit.>, HOF <cit.>, ITF <cit.> were designed.  However, the proven utility of deep features for many tasks such as Object Recognition <cit.>, Object Detection <cit.>, etc., has  made features from well performing CNNs such as <cit.>, Two-Stream CNN<cit.>, 3DCNN <cit.> ubiquitous for Action Recognition tasks including the zero shot setting. By using 3DCNN features in ZSL, a significant boost in accuracy has been observed <cit.>.
Semantic representation of a class provides additional, complementary information to the visual features of the classes. Typically, two types of semantic representations have been widely used in the ZSL literature: attribute representations <cit.> and word vector representations <cit.>. Attribute representations are manually annotated vectors for each class based on the gesture and motion appearance of the objects in the video.  Word Vector representations are automatically learned from a large amount of textual data (Wikipedia Corpus). Word2vec models have been used successfully for extracting semantic word vectors from class names <cit.>. The core step in ZSL is to find a function or projection matrix which can establish a relationship between visual space and semantic space in such a way that visual features of classes map close to their semantic features and vice versa.  For example, we would like to have visual features of `running' map close to semantic features of `running' and far away from an unrelated action such as `eating'.

Note that our framework is similar in spirit to such methods with a key difference: Instead of learning a mapping between the semantic feature and visual features, we learn a mapping from the semantic features and the parameters of the distributions representing the classes.

Most methods for zero-shot learning are evaluated on image classification whereas only a few methods have been proposed for zero-shot action recognition in the literature <cit.>. Such methods typically assume the inductive or the transductive setting. The most popular approach to ZSL is learning a linear compatibility between the visual and semantic space  <cit.>. <cit.> provide novel regularizations while learning a linear compatibility function.  ESZSL <cit.> models the relationship between features and attributes as a linear compatibility function while explicitly regularizing the objective. UDA <cit.> uses a domain adaptation technique by using unlabeled data of unseen classes for better estimation of the parameters.

Our model is inspired by the recently proposed model <cit.>, which is a simple generative approach for zero-shot learning. However, their model does not have the reconstruction regularizer (autoencoder-style reverse mapping) from visual to attribute space and their focus is on image classification whereas here we have focused on action recognition. In another recent work, <cit.> proposed a semantic auto-encoder for zero-shot learning which introduced the reconstructability regularizer. This paper works only in the inductive setting and their approach is not generative. Our generative approach can be seen as a combination of the generative approach of <cit.> with auto-encoder style regularizer proposed by <cit.>. 

Among prior works on zero-shot action recognition in transductive setting, <cit.> proposed a transductive framework for zero-shot action recognition, which uses unlabeled unseen class data for training the model. In their work, they introduced a manifold-regularized regression and a data augmentation strategy to enhance the performance. They have also introduced a multi-task visual-semantic mapping for zero-shot action recognition.  In addition, they used prioritized auxiliary data augmentation for domain adaptation and improved the mapping between visual and semantic spaces. 

Because of the generative nature of our proposed approach, we can synthesize the data from unseen class based on attribute and train the classifier. This approach helps to reduce the baisness in the case of Generalize Zero-Shot Learning. The efficacy of the proposed approach for the GZSL as well as ZSL can be seen from the experiment on three standard datasets. 


§ EXPERIMENTS 


Datasets and Settings: 
We evaluate our proposed method in three of the most challenging video action recognition datasets, UCF101 <cit.>, HMDB51 <cit.> and Olympic <cit.>, widely used as benchmark datasets. We report mean accuracy along with standard deviation on 30 independent test runs with random train/test class splits. 


  * UCF101: <cit.> is human action recognition data set with 101 different classes of actions and total of 13320 video clips. In our experiments, we split the classes into 51 seen and 50 unseen class respectively.
`
  * HMDB51: <cit.> is the one of the most challenging human action recognition dataset with 51 different classes of human actions and total number of 6766 video clips. Each class has more than 100 video clips. For the evaluation of our model, we perform a 26/25 split for seen and unseen classes respectively.

  * Olympic: <cit.> This dataset has 783 videos from 16 different classes with seen/unseen class split being 8/8. 







Visual features:
The quality of visual features directly affect the efficacy of the model. We use deep features as they have been shown to be successful in many  computer vision tasks.  In our experiments, we use the latest  convolutional 3D(C3D) visual features provided by <cit.>. This model was pre-trained on the sports-1M dataset. We extract the outputs of fc6 layer for all segments similar to <cit.> and then averaged over the segments to form a 4096-dimensional video representation which is used as the input visual features.





Class attributes:
Two types of class attribute vectors (semantic representation of the classes) are widely used in ZSL: human labeled attributes <cit.> and automatically learned distributed semantic representations such as word vectors <cit.>. Word vector representation is learned automatically by a skip-gram model  trained on the google news text corpus provided by Google. Each word is represented by a 300 dimensional vector. We experiment on both attribute and word2vec representations. For HMDB51 dataset, to the best of our knowledge, there is no publicly available attribute representations of the classes. Hence only word2vec is used for HMDB51. However, for UCF101 and Olympic datasets, 115 and 40 dimensional attribute vectors are available respectively <cit.>.



Hyper-parameters:
Our model consists of four hyper-parameters: λ_μ, λ_1 (Eq. <ref>) and λ_σ^2, λ_2  (Eq.  <ref>) for estimating the projection matrix for mean  and variance. The optimal values of hyper-parameters are chosen via cross validation on the seen classes. For cross validation, we randomly fix 1/4th of the seen classes as validation classes and conduct five trials on 30 random splits (same as <cit.>). For generalized ZSL setting,  the number of synthesize examples for unseen classes is also hyper-parameter which we find using cross-validation and observe best model performance for 200 synthesized examples. 



 §.§ Inductive and Transductive ZSL

In our first set of experiments, we evaluate our model for zero-shot action recognition with inductive and transductive setting and compare with a number of state-of-the-art methods.


Evaluation Metric:
We evaluate our model using 30 different splits into seen and unseen classes provided by <cit.> for UCF101 (51/50), HMDB51 (26/25) datasets. For Olympic dataset, we generate  30 random splits for seen and unseen classes (8/8). We use the average accuracy for all 30 splits as the evaluation metric. For fair comparison, we run five such trials for 30 random splits and present the final accuracy with average and standard deviation.

For generalized zero-shot setting we have evaluated for 30 different splits as above and calculated the average accuracy for seen and unseen classes. The final evaluation metric of our model is on the harmonic mean of the average accuracy of seen and unseen classes, as used in <cit.>.

     
Inductive setting: In this setting, it is assumed that only the labeled data from the seen classes is available during training. Table <ref> shows the experimental results in the inductive setting of the zero-shot action recognition problem. We assume that the train and test classes are disjoint. Note that this assumption is made for all the evaluation settings in this work. In this setting, we obtain an improvement of 3% over the state-of-the-art on the Olympic dataset. On UCF-101, which is the most used dataset for zero shot action recognition, the proposed model outperforms state-of-the-art on attribute-based semantic representations. For HMDB dataset, the attribute vectors are not available. Hence, we present results only on word2vec embeddings.  Our model outperforms the state-of-the-art for this dataset as well. We believe the improvements can be attributed to its inherent nature of sharing information across classes (by modeling each as a basis combination of prototype classes) and its simple estimation procedure. 




Transductive setting: In the transductive setting, it is assumed that the unlabeled data of the unseen classes is also available at train time. Table <ref> shows the performance of our model in the transductive setting. The unlabeled data from unseen classes helps us mitigate the bias towards the seen classes. In this setting, our model outperforms the state-of-the-art in the Olympic and HMDB datasets. The performance on the UCF-101 dataset is slightly worse, where <cit.> has the best performance. However, note that we outperform <cit.> in the inductive setting. 
 








 §.§ Generalized ZSL

In this setting, the test data may come from both seen and unseen classes. In this setting, from the seen classes, we separate 20% of the data for testing and remaining 80% data is used as training data for calculating 𝐖_μ and 𝐖_σ^2 which is used to predict the mean (μ_c) and variance (σ_c^2) for the unseen classes. One way to handle this setting is to assign each test data-point to the class whose estimated distribution gives the highest score. However, we notice that such an approach is biased towards seen classes since the model has not seen any unseen class examples.  In our approach, we propose the following solution to this issue: we synthesize class instances of unseen classes using the μ_c and σ_c^2 which are obtained from the transductive setting approach; these class instances are called pseudo class instances for unseen classes. Here we generate 200 instances for each unseen classes. Since we now have labelled data for seen classes and pseudo labelled data for unseen classes, we train SVM classifier for labelled seen classes data and pseudo labelled data for unseen classes. We then pass the test data (unseen class data plus 20% seen class data) to the trained SVM classifier for classification. 
Table <ref> presents the performance of our model in the generalized setting for zero-shot action classification which clearly shows that it significantly outperforms state-of-art on all the datasets. 







 §.§ Few-shot action recognition

Finally, we experiment with the few shot action recognition setting and present the results. Here only a small number of examples for each of the unseen classes are available during training. Our generative model provides a simple way to update the parameters of the class distribution using equation <ref>, <ref> . It is clear from the Table <ref> that availability of the few data points of the unseen classes significantly improves the performance which is now comparable to that of supervised learning.
Note that we do not assume any unlabeled data from the unseen classes in this setting. We test our model with varying number of examples of each unseen classes. The plot of accuracy with respect to the number of samples per class is shown in Figure <ref>. 










§ CONCLUSION

We have presented a simple, probabilistic, generative model based framework for zero-shot action recognition. The proposed approach performs well in both the inductive and transductive setting for the standard (disjoint) and generalized zero-shot learning. The generative aspect of our model unables synthesizing unseen class examples and can effectively work in the generalized ZSL setting. In addition, the ability of leverage unlabeled data (transductive setting) helps address the domain shift problem between seen and unseen classes. A particularly appealing aspect of our model is that it yields a closed form solution for the parameters to make it fast and easy to implement. Experimental results are shown to achieve state-of-the-art performance. The proposed method also generalizes to few-shot action recognition setting, achieving comparable results to fully supervised learning using only a few synthesized examples from each unseen class.

Acknowledgments: Thanks to Vismay who always gives his support in discussion and experiment. Special thanks to Prof. Hema A Murthy for her valuable feedback. Vinay Verma acknowledges support from Visvesvaraya Ph.D. fellowship. Piyush Rai acknowledges support from Visvesvaraya Faculty Fellowship and a grant from Tower Research.

ieee.bst



