Noname manuscript No.
(will be inserted by the editor)

Multi-Label Zero-Shot Human Action Recognition via Joint Latent
Embedding

arXiv:1709.05107v1 [cs.CV] 15 Sep 2017

Qian Wang · Ke Chen

Received: date / Accepted: date

Abstract Human action recognition refers to automatic recognizing human actions from a video clip, which is one
of the most challenging tasks in computer vision. Due to
the fact that annotating video data is laborious and timeconsuming, most of the existing works in human action
recognition are limited to a number of small scale benchmark datasets where there are a small number of video clips
associated with only a few human actions and a video clip
often contains only a single action. In reality, however, there
often exist multiple human actions in a video stream. Such
a video stream is often weakly-annotated with a set of relevant human action labels at a global level rather than assigning each label to a specific video episode corresponding to a
single action, which leads to a multi-label learning problem.
Furthermore, there are a great number of meaningful human
actions in reality but it would be extremely difficult, if not
impossible, to collect/annotate video clips regarding all of
various human actions, which leads to a zero-shot learning
scenario. To the best of our knowledge, there is no work that
has addressed all the above issues together in human action
recognition. In this paper, we formulate a real-world human
action recognition task as a multi-label zero-shot learning
problem and propose a framework to tackle this problem.
Our framework simultaneously tackles the issue of unknown
temporal boundaries between different actions for multilabel learning and exploits the side information regarding
the semantic relationship between different human actions
for zero-shot learning. As a result, our framework leads to
a joint latent embedding representation for multi-label zeroQian Wang
The University of Manchester, UK
E-mail: Qian.Wang@manchester.ac.uk
Ke Chen (corresponding author)
The University of Manchester, UK
E-mail: Ke.Chen@manchester.ac.uk

shot human action recognition. The joint latent embedding
is learned with two component models by exploring temporal coherence underlying video data and the intrinsic relationship between visual and semantic domain. We evaluate
our framework with different settings, including a novel data
split scheme designed especially for evaluating multi-label
zero-shot learning, on two weakly annotated multi-label human action datasets: Breakfast and Charades. The experimental results demonstrate the effectiveness of our framework in multi-label zero-shot human action recognition.
Keywords Human action recognition · Multi-label
learning · Zero-shot learning · Joint latent embedding ·
Weakly supervised learning
1 Introduction
As one of the most challenging tasks in computer vision, human action recognition refers to automatic recognizing human actions conveyed in a video clip. In last two decades,
human action recognition has been extensively studied. As
there are many different human actions in reality, this task
is generally formulated as a multi-class classification problem. To train a multi-class classifier for human action recognition, a great number of examples for each single action
are required in the current setting. To collect such training examples, one needs to manually trim a video steam
to ensure that there is only one human action appearing
in a trimmed video episode. This annotation process is laborious and time-consuming and there is hence no largescale dataset with “fine-grained” annotation for human action recognition. In contrast to ImageNet (Deng et al. 2009)
for object recognition, where it consists of a total of 3.2 million cleanly labelled images spread over 5,247 categories,
there are much fewer annotated video clips involving only
a small number of human actions. For instance, HMDB51

2

and UCF101 are among the most commonly used benchmark datasets in human action recognition, where there are
6,676 and 13,320 instances of only 51 and 101 different human actions, respectively. The limitation of human action
datasets in such a scale has become an obstacle in developing a large-scale human action recognition system.
In a real scenario, a video clip often conveys multiple
human actions corresponding to different concepts. Hence,
a set of multiple action labels have to be used to characterize
its complete semantics underlying human actions conveyed
in this video clip. For example, video clips on YouTube are
usually uploaded by users along with some descriptive terms
that can be used to infer the human actions conveyed in those
video clips, where descriptive terms may be viewed as a set
of coherent labels that collectively characterize the semantics at a global level. Recently, a very large multi-label video
dataset YouTube-8M (Abu-El-Haija et al. 2016) has been
collected by Google Research. Although the dataset is not
restricted to human action video clips, it paves a new way
for various video analyses including human action recognition. One of essential video analysis problems on such a data
set may be formulated as multi-label learning that predicts a
set of labels associated to a given instance or a set of confidence scores corresponding to the candidate labels related to
this instance. In multi-label learning, a training example often consists of an input instance and a set of labels associated
with this instance at a global level (no need of explicitly associating each of those labels to a relevant object within this
instance). While multi-label data are common in many domains and multi-label classification has been studied under
different applications (Zhang and Zhou 2014), e.g., semantic image tagging, text categorization and gene functionality
prediction, only few works focusing on multi-label human
action recognition can be found in literature due to a lack of
human action datasets annotated with multiple class labels.
To fill in this gap, a dataset dubbed with Charades (Sigurdsson et al. 2016) was collected especially for multi-label human action recognition and made publicly available very recently. In addition, other datasets were also considered to be
used in multi-label human action recognition although they
were originally collected for different tasks. Thus, such data
sets provide a proper test bed for multi-label human action
recognition studies.
Multi-label human action recognition works on weakly
labelled video data, i.e., the training data are annotated at
the video level without exhaustively trimming and annotating multiple action episodes. While it is easier to collect
such video clips associated with a set of labels at a global
level than those with “fine-grained” annotation, it would be
still very challenging to collect all the training examples due
to the existence of many different human actions. Zero-shot
learning (ZSL) provides an alternative solution to alleviate
this problem. ZSL aims to recognize the instances belong-

Qian Wang, Ke Chen

ing to novel classes which are not seen during training. It has
been formally shown that a ZSL system trained on a dataset
of finite classes could be used to predict infinite number of
classes unseen during the learning (Zhang et al. 2016a). Under the ZSL framework, we merely need to collect and annotate training examples for a moderate set of training classes
and expect that a large number of novel classes can be recognized via exploiting the semantic relationship between different human actions. To this end, a ZSL algorithm needs to
transfer the knowledge regarding the relations between visual features and class label semantics learned from known
or training classes to unseen or test classes. The knowledge
transfer is enabled by modelling the semantic representations of different classes, which can be easily obtained from
side information, e.g., descriptive texts, with a much less effort. Nevertheless, most of the existing ZSL methods were
proposed to tackle single-label ZSL problems. Although
some of those methods may be extended to multi-label scenarios, their effectiveness of different ZSL algorithms have
not been extensively investigated in the multi-label learning
scenarios. To the best of our knowledge, there exists no work
in multi-label zero-shot human action recognition.
In this paper, we address the multi-label ZSL issue in
the context of human action recognition. The training video
data are weakly annotated so that the exact temporal-spatial
locations of multiple human actions in a video clip are unknown. In addition, training examples are only available for
training/known labels, a subset of the action label collection
considered in the recognition stage. To address the multilabel ZSL problem in human action recognition, we propose
a novel joint latent embedding (JLE) framework. The framework aims to learn a joint latent space from visual and semantic representations. In the learned joint space, visual and
semantic representations of human actions are mapped into
visual and semantic embedding, respectively. Then, the visual and semantic domains are related via the joint embedding space where any human actions in a considered action
label collection can be recognized regardless of known or
unseen classes during learning. In our framework, a visual
model learns mapping visual representations of a video clip
into latent visual embedding, and a semantic model learns
mapping the semantic representations of human action labels into latent semantic embedding in the same joint embedding space. In the visual model, a long short-term memory (LSTM) network is employed to explore saliency of the
multiple human actions conveyed in the weakly annotated
multi-label video data, which plays a role in implicitly finding out a specific episode containing one of multiple human
actions conveyed in a video clip. In the semantic model, a
three-layer fully-connected neural network is employed to
learn the latent semantic embedding so that the semantic
gap between visual and semantic space can be bridged and
the semantic relations between all different labels in a con-

Multi-Label Zero-Shot Human Action Recognition via Joint Latent Embedding

sidered action collection can be well explored and modeled
properly. During learning, both visual and semantic models
are trained with our proposed alternate learning algorithm.
For recognition, the trained visual and semantic models are
used together to recognize a test video clip by measuring
the distance between its visual embedding and the semantic
embedding of all the labels.
Our main contributions in this paper are summarized as
follows:
– By considering real scenarios, we formulate generic human action recognition as a multi-label zero-shot learning problem. To the best of our knowledge, our work
presented in this paper is the first attempt to study human action recognition from a multi-label zero-shot perspective.
– To address the multi-label zero-shot issue arising from
weakly annotated data for human action recognition, we
propose a novel joint latent embedding framework consisting of visual and semantic embedding models. To
train two embedding models effectively, we come up
with a learning algorithm that alternately optimizes the
parameters in two embedding models during learning.
– To test the performance of our proposed framework, we
conduct a thorough evaluation via a comparative study
on two benchmark multi-label human action datasets,
Breakfast and Charades, with various evaluation metrics
and different settings including a novel data split protocol simulating a real scenario of zero-shot human action
recognition.

3

world, e.g., web videos. In a human action video clip, multiple actions could happen simultaneously, e.g., sitting, eating and listening. In this scenario, no episode in such a
video clip can be characterized by a single action label and
a set of labels hence have to be collectively used to describe
this video clip. Even though a video clip can be divided
into several episodes corresponding to different human actions, the segmentation and annotation process could be difficult, tedious, laborious and time-consuming. In particular,
semantic image segmentation and human action detection
in video streams remain unsolved. As a result, multi-label
learning is often formulated as a weakly supervised learning task that predicts a set of labels associated with an instance but does not address the issue in assigning each of
labels in the set to a specific object within this instance.
To tackle a weakly supervised multi-label learning problem, two different representation methods are used to characterize input data: instance-level and object-level representation. Instance-level representation is a holistic representation of an instance without considering objects containing in
this instance, while object-level representation is an objectbased representation that extracts potential objects from instances to form bag-of-object representations. Depending on
the representation of input data, multi-label learning methods can be divided into two categories.

2.1 Multi-label Learning

In multi-label learning, most of the existing methods
(Guillaumin et al. 2009; Nam et al. 2014; Wang et al. 2016;
Zhang et al. 2016b) work on an instance-based representation, a single feature vector holistically representing multiple objects within an instance. Recently, Fast0Tag (Zhang
et al. 2016b) was proposed for multi-label image tagging
by learning a mapping from visual space to a label space.
An image containing multiple objects is represented by
one aggregated visual representation. Alternatively, TagProp
(Guillaumin et al. 2009) uses an adapted nearest neighbour
model for multi-label learning in the visual space where
each image of multiple objects is also represented by one
feature vector. Wang et al. (2016) use a convolutional neural
network (CNN) directly working on raw images of multiple objects to learn image-level deep visual representations
for multi-label classification. Nam et al. (2014) use a deep
neural network with a ranking loss in learning for largescale multi-label text classification where an input document
is represented with a feature vector. Although representing
one instance with a single feature vector is straightforward
and convenient, it might neglect the intrinsic relationship between multiple objects within an instance. Thus, a holistic
instance-level representation might result in a catastrophic
information loss, especially for long-term dependent and
complex video data.

In a real scenario, data that need to be characterized with
multiple labels are often complicated but common in real

To overcome the weakness in neglecting the information
regarding the intrinsic relationship between objects within
an instance, researchers attempt to exploit such information

The rest of this paper is organized as follows. Section
2 reviews related works. Section 3 presents our framework
for multi-label zero-shot human action recognition. Section
4 describes our experimental settings, and Section 5 reports
the experimental results. The last section draws conclusions.

2 Related Work
In this section, we review the existing works relating to
multi-label human action recognition, especially for those
applicable to multi-label zero-shot learning scenarios. we
also address the limitations of the existing works and make
a connection between our work and those related works. As
a result, we first overview existing multi-label classification
methods and then focus on the existing works in multi-label
ZSL learning despite the fact that none of such multi-label
ZSL methods has been applied to human action recognition.

4

in their multi-label learning methods. Although it is difficult,
the segmentation of multiple objects within an instance turns
out to be beneficial to multi-label learning. One framework
named multi-instance multi-label learning (MIML) (Zhou
and Zhang 2007) demonstrates that multi-label learning can
be fulfilled effectively if multiple objects within an instance
have been explicitly separated or segmented even if no label
is explicitly assigned to each of multiple objects within an
instance during learning. In real applications, however, automatic semantic segmentation of objects in an instance is also
challenging, and a manual segmentation process is laborious and time-consuming. Recently, some efforts have been
made to explore object-based representations without using
any explicit semantic object segmentation techniques, which
seeks a synergy between the MIML and object-level representations. Gu et al. (2016) address this weakly supervised
issue in multi-label human action detection with a two-stage
solution. First, a set of potential objects or spatial-temporal
volumes are generated and selected from a video instance
with a set of handcrafted rules. Then the problem is transformed into a MIML problem which can be solved by those
traditional multi-label learning algorithms under the MIML
framework. A similar idea was also explored by Wei et al.
(2016) and Tang et al. for multi-label image classification.
However, the extraction of true positive objects from the
original example is a very challenging yet non-trivial task,
which critically determines the multi-label learning performance. To extract all the meaningful objects within an instance, a lot of candidate proposals have to be considered so
that it might suffer from a high computational burden. Instead of using the MIML, Cabral et al. (2015) attempt to explore the information regarding multiple objects in instances
via a matrix completion method. Their method works on the
assumption that an instance may be represented by a linear combination of those object representations in this instance. Experimental results reported by Cabral et al. (2015)
demonstrate the effectiveness of this method via an objectlevel bag-of-words image representation. However, it remains unknown whether this method is also applicable to
other kinds of representations, such as those popular yet
powerful deep representations.

2.2 Multi-label Zero-shot Learning
Zero-shot learning (ZSL) has attracted much attention in recent years and provides a promising technique for recognizing a large number of classes without the need of the training
data concerning all the classes. Very recently, Zhang et al.
(2016a) have formally shown that it is feasible to predict a
collection of infinite unseen labels with a classifier learned
on training data concerning only a number of labels in this
collection or a subset of this collection, where multi-label

Qian Wang, Ke Chen

ZSL is a special case in this so-called “infinite-label learning” paradigm. According to a taxonomy of ZSL (Wang and
Chen 2017b), existing ZSL approaches are divided into three
categories, namely, direct mapping (Akata et al. 2015; Fu
and Huang 2010; Lampert et al. 2014; Xian et al. 2016),
model parameter transfer (Changpinyo et al. 2016; Mensink
et al. 2014) and joint space learning (Wang and Chen 2017b;
Zhang and Saligrama 2015). Although most existing works
focus on single-label ZSL, efforts have been made to extend ZSL to multi-label learning scenarios (Fu et al. 2014;
Mensink et al. 2014; Nam et al. 2015; Ren et al. 2015; Sandouk and Chen 2016b; Zhang et al. 2016b).
For direct mapping, it needs to learn a direct mapping
from visual to semantic space for zero-shot recognition,
which poses a challenge to multi-label ZSL. In single-label
ZSL, a training example provides a visual-semantic representation pair used to learn a one-to-one direct mapping.
In multi-label ZSL, however, one instance has to be associated with a set of multiple labels and the number of labels
associated with different instances are various. As a label
is represented with a semantic feature vector, e.g., a vector
of attributes or a word vector, in a semantic space, it is no
longer straightforward to learn a direct mapping from visual to semantic space in the context of multi-label ZSL.
How to model complex semantics underlying a set of labels associated with an instance becomes a central issue in
multi-label ZSL. To tackle this issue, most of existing works
(Fu et al. 2014; Sandouk and Chen 2016b) make use of the
composition properties of semantic representations such as
word vectors by using the average of semantic representations of multiple labels to a collective semantic representation for a set of labels associated with the instance. Thus,
a training example is formed with a pair of an instancelevel visual representation and its corresponding collective
semantic representation, which enables one to learning a direct mapping for multi-label ZSL. Apparently, such a collective representation cannot avoid information loss even
though a contextualized semantic representation (Sandouk
and Chen 2016b) was used. To alleviate the information loss
problem in generating a collective semantic representation,
Fast0Tag (Zhang et al. 2016b) introduces an alternative solution to collective semantic representations. In this method,
each visual instance is mapped into a “principal direction”
in the semantic space based on an assumption that there is
always a direction for an multi-labelled instance in a semantic space, e.g., word vector space and all the labels relevant
to this instance always have higher “ranking” values over
irrelevant labels. In other words, we can always find a hyperplane to separate the relevant labels from the irrelevant
ones for any multi-labelled instance. While this assumption holds for those datasets as demonstrated in their zeroshot image tagging work (Zhang et al. 2016b), it remains
unclear for other datasets in different domain, e.g., human

Multi-Label Zero-Shot Human Action Recognition via Joint Latent Embedding

5

action recognition, especially due to the existence of various semantic representations that lead to different semantic spaces. From a different perspective, Ren et al. (2015)
suggest using an object-level visual presentation under the
direct mapping framework for multi-label zero-shot object
recognition. Before multi-label learning takes place, an image thus has to be semantically segmented into meaningful
subregions and each subregion can be characterized by one
label. As a result, their solution is actually a special case of
the MIML (Zhou and Zhang 2007) and hence has to rely on
sophisticated semantic segmentation techniques that remain
unavailable up to date.
Like the works in extending direct mapping to multilabel ZSL, the model parameter transfer idea is also taken
into account for multi-label ZSL. COSTA (Mensink et al.
2014) aims to estimate a model for each unseen label
by a linear weighted combination of known-label models,
e.g., support vector machines (SVMs). The combination coefficients are determined by label co-occurrences derived
from either annotations of datasets in hand or external web
sources. In COSTA, the models regarding each of known
labels are trained independently by means of a one-vs-rest
binary SVM classifier without considering the relation and
coherence among labels assigned to a single object. Furthermore, COSTA only uses label co-occurrences to model the
relatedness between a pair of labels but neglects the semantic of an individual label itself.
Among the three kinds of ZSL algorithms, those joint
space learning methods (Frome et al. 2013; Lei Ba et al.
2015; Wang and Chen 2017b; Zhang et al. 2017; Zhang and
Saligrama 2015; 2016) often yield more promising results
in single-label ZSL. Apart from being used in ZSL, the joint
space learning methodology has also been applied to traditional multi-label learning tasks (Gong et al. 2014; Weston
et al. 2010) and led to favorable results. The main idea underlying joint space learning is learning a joint (common) latent representation space for both visual and semantic space
to bridge the semantic gap. To the best of our knowledge,
however, there is no extension of existing joint space learning methods to multi-label ZSL problems, to a great extent,
due to the difficulty in modelling complex semantics underlying a set of labels describing an instance, as elucidated
above regarding the direct-mapping. In this paper, we hence
propose a novel approach to multi-label zero-shot human action recognition under the joint space learning framework by
addressing those critical issues.

tations have been developed from different perspectives including attributes of labels, label embedding, co-occurrence
of labels and concept embedding.
Attributes of labels are a semantic representation where
a label is characterized by a list of attributes common to all
the labels (Lampert et al. 2014). Label embedding refers to
a class of approaches to embedding labels onto a semantic
space where the semantic relatedness of labels are reflected
(Mikolov et al. 2013). Label embedding is often carried out
via learning on external textural resources. For example, the
famous Word2Vec semantic embedding is obtained by training a skip-gram neural network on the large-scale corpora,
e.g., Google News dataset (Mikolov et al. 2013). Such semantic representations are widely used in ZSL, e.g., (Fu
et al. 2014; Ren et al. 2015; Wang and Chen 2017b; Zhang
et al. 2016a;b). Unlike label embedding obtained with external resources , co-occurrence of labels is an approach to
capturing the relatedness between different labels with the
information on co-occurrence of labels or hierarchical relationship between labels in a given dataset used in ZSL, e.g.,
(Nam et al. 2015). Alternatively, the co-occurrence information on different class labels are also extracted from external
resources for ZSL (Mensink et al. 2014). In particular, cooccurrence of labels allows for capturing the relatedness between labels jointly used to describe an instance. The label
co-occurrence information may be incorporated into learning semantic embedding for a given dataset, e.g., (Nam et al.
2015). In concept embedding, the semantic meaning of a label is assumed to be polysemous depending on different labels (together treated its context of this target label) jointly
used to describe an instance. Hence, the semantic meaning
of a label under a specific context frames a concept. As a
result, concept embedding (Sandouk and Chen 2016a) can
be viewed as contextualized label embedding where a label
may have multiple semantic representations in the embedding space if it is used with different labels in describing different instances. The concept embedding leads to a specific
approach to multi-label ZSL (Sandouk and Chen 2016b).
However, contextualized semantic representation learning
suffers a high computational burden and is difficult to deal
with a large-scale label collection. Our proposed framework
for multi-label zero-shot human action action is generic so
that all the semantic representations apart from concept embedding may be used directly in our proposed framework.

2.3 Semantic Representation

In this section, we present a novel framework for multi-label
zero-shot human action recognition. First, we overview the
proposed framework along with our motivation. To make it
self-contained, we then describe the LSTM unit, an important mechanism used in our framework for implicit saliency
detection on video data. Next, we present a method for joint

Regardless of different ZSL scenarios, modelling semantics
underlying a collection of labels and their relatedness plays
a critical role in knowledge transfer required by ZSL. Miscellaneous methods in semantics modelling and represen-

3 Model Description

6

Qian Wang, Ke Chen

Model

Ranking Loss

Ranking Loss

Average Pooling

Average Pooling

Score Layer

…

Vis. Embedding

…

Dense Layer

…

Score Layer

Ev

Es

Sem. Embedding

Recognition
cut_orange peel_fruit
pour_milk
fry_egg
… Squeeze_orange

Semantic
model
Es
Label
Scores

Hidden Layer
LSTM Layer

…

Vis. Rep.

…

Video Clips

ev
Sem. Rep.

…

Action Labels
cut_orange
pour_milk
fry_egg
…

Visual
model

Test Video Clip

Fig. 1 Our multi-label zero-shot human action recognition framework. This framework shown in the left box is composed of two component
models: visual and semantic model highlighted with grey and red colors. After the joint latent embedding learning, the trained visual and semantic
models work together for multi-label zero-shot recognition, as shown in the right box. Action labels marked with brown color are training classes
or known labels during learning, while action labels marked with blue colour are test classes or unseen labels during learning.

latent embedding learning including loss functions used in
the learning and a learning algorithm especially designed for
the proposed architecture. Finally, we specify a procedure
for multi-label zero-shot human action recognition based on
a trained joint latent embedding model.

3.1 Overview
Our proposed framework aims at multi-label zero-shot human action recognition. We formulate this problem as learning a mapping φ : x → R|C| , where x is a visual input,
e.g., a set of segment-level visual feature vectors extracted
from a video clip, and y ∈ R|C| is a list of label-relatedness
scores for x with respect to a action label collection, C =
{1, · · · , |C|}, where C is further divided into two mutually
exclusive label subsets corresponding to known (training)
and unseen actions; i.e., C = CTr ∪CU and C = CTr ∩CU = 0.
During learning the mapping φ , only training examples of
labels in C = CTr are available. However, the learned mapping φ is used to predict any actions in C including those
unseen actions defined in CU from a video clip.
To tackle the problem formulated above, we propose a
joint latent embedding framework. Motivated by the joint
space learning idea used in ZSL (Frome et al. 2013; Lei Ba

et al. 2015; Wang and Chen 2017b; Zhang et al. 2017; Zhang
and Saligrama 2015; 2016), we would tackle the knowledge
transfer issue in a joint latent space where both visual and
semantic representation embedding resides. By embedding
visual and semantic representations into this latent space, we
expect that semantic gap can be narrowed considerably and
the semantic relatedness of known and unseen labels may be
effectively explored and exploited in zero-shot recognition.
Thus, our framework consists of two component models: visual and semantic models used to learn visual and semantic
embedding as well as joint latent embedding for knowledge
transfer, as illustrated in the left box of Figure 1.
For visual embedding, we encounter two major technical issues due to the nature of weakly annotated data: a) for
a visual input, it remains unknown where an episode conveying an action, and b) it remains unclear which of those
action labels describing a video clip is associated with a
specific video episode. Nevertheless, a video clip is an ordered sequence of frames and we could explore the temporal
coherence underlying a video clip to tackle two aforementioned technical issues. Motivated by recent works in video
classification and activity recognition (Donahue et al. 2015;
Ma et al. 2017; Yue-Hei Ng et al. 2015), we employ a long
short-term memory (LSTM) (Hochreiter and Schmidhuber
1997) recurrent neural network layer to capture temporal

Multi-Label Zero-Shot Human Action Recognition via Joint Latent Embedding

coherence underlying an action episode. Thus, the LSTM
layer (c.f. Section 3.2) is first used to process a sequence of
visual representations extracted from video segments. With
the memorising and forgetting mechanism of LSTM units,
we expect that the LSTM layer explores the temporal structure of human actions conveyed in a video sequence; the
LSTM units would memorize the input segments until parsing an episode regarding a human action is completed and
then forget all the previous input segments when an episode
conveying another action starts. Thus, an implicit saliency
detection is carried out where no action episode boundaries
are explicitly specified For visual embedding, we further
employ two fully-connected layers, dense and visual embedding layers in Figure 1, to capture salient features on
the temporal coherence representation yielded by the LSTM
layer. While this specific visual model is used in our experiments, its capacity can be increased by adding more hidden layers if necessary. The score and averaging pool layers
above the visual embedding layer are used for joint latent
embedding learning as presented in Section 3.3. Thus, the
visual model is a deep network of heterogeneous layers.
For semantic embedding, we employ a three-layer fullyconnected neural network to carry out the semantic model.
We expect that this learning model is capable of capturing
the intricate semantic relatedness between different actions
in a label collection of a moderate size, e.g. those datasets
use in our experiments. In general, its capacity can be increased by adding more hidden layers if necessary. As a result, the neural network is fed with a specific semantic representation of action labels, e.g., word vectors and subsequently map them into the semantic embedding layer via a
hidden layer, as shown in Figure 1. Likewise, the score and
averaging pool layers above the semantic embedding layer
are used for joint latent embedding learning. To explore the
semantic relatedness between different labels in bridging the
semantic gap between visual and semantic space, semantic
embedding learning needs to exploit the information carried
in training data, e.g., frequency of label co-occurrence in a
training dataset (c.f. Section 3.3).
During the joint latent embedding learning, the visual
and semantic models are coupled to form a joint latent embedding space. For learning, we propose a learning algorithm working alternately on two models for parameter estimation by promoting positive video-label pairs over negative ones. Thus, the visual model learns the visual embedding of a video example such that its corresponding relevant
labels rank ahead of other irrelevant ones in terms of the
relatedness scores estimated on the semantic latent space,
E s . Alternately, the semantic model learns the semantic embedding of action labels such that the relevant videos rank
higher than the irrelevant ones in terms of relatedness scores
calculated in the visual latent space, E v . Once the learning
is completed, the trained joint latent embedding model can

7

xt

ct-1

xt

it

ct-1
ht-1
xt
ct-1

ht-1

ht-1

ct-1

ot

ct

tanh

tanh

ht
ct

ot
ct-1 h xt
t-1

Fig. 2 LSTM unit and its information flow.

be applied to a test video clip for human action recognition. As a result, the relatedness scores corresponding to all
the known and unseen action labels in a label collection are
achieved by using both visual and semantic models (c.f. Section 3.4), as illustrated in the right box of Figure 1.

3.2 Long Short Term Memory
As described previously, the LSTM layer plays a crucial rule
in tackling two technical issues arising from weakly annotated data for visual embedding in our proposed framework.
Here, we briefly review the mechanism of an LSTM unit to
facilitate understanding our framework.
As shown in Figure 2, an LSTM unit consists of a memory cell reserving the historic information at previous time
steps. The output in time step t is determined by the current input xt and the activation value in this memory cell.
Three gates, input,output and forgetting gates, are used to
control the information flow in the LSTM unit. The information flow in an LSTM unit (Graves 2013; Hochreiter and
Schmidhuber 1997) is formulated as follows:
it = σ (Wxi xt +Whi ht−1 +Wci ct−1 + bi ),
ft = σ (Wx f xt +Wh f ht−1 +Wc f ct−1 + b f ),
ct = ft ct−1 + it tanh(Wxc xt +Whc ht−1 + bc ),
ot = σ (Wxo xt +Who ht−1 +Wco ct + bo ),
and
ht = ot tanh(ct ),
where σ (·) and tanh(·) are the sigmoid and the tangent hyperbolic functions, and i, f , o refer to input, forget and output gates, respectively. c is the cell state, representing the
information stored in the cell, and h is output of the LSTM
unit or input receiving from the output of another LSTM

8

Qian Wang, Ke Chen

Table 1 Nomenclature.

3.3.1 General Description

Notation

Description

D, DT
| · |, || · ||1
C, CTr , CU
Ev, Es
x it

training, test datasets
cardinality of a set, L1 norm of a vector
label collection, training and unseen class label subsets
visual and semantic embedding space
visual representation for the t-th segment of the i-th
example
collection of all the segment-level visual representations of the i-th example
semantic representation for the c-th label
binary target label matrix of training dataset
binary target label set of the i-th example, i.e., the i-th
column of matrix Y
binary indicator vector of the c-th label appearing examples, i.e., the c-th row of matrix Y
visual embedding matrix and the column vector for ith video clip
semantic embedding matrix and the column vector for
the c-th label
dimensions of visual, semantic, latent embedding
space
relatedness scores between the i-th example and all the
candidate labels in visual model
relatedness scores between the c-th label and all the
training examples in semantic model
visual embedding function and parameters
semantic
 embedding function and parameters
φ = φ v ,φφ s , mapping function for multi-label zeroshot recognition
the ground-truth label set of test instance x̂x, the ranking list of all the labels predicted for x̂x in terms of the
relatedness scores
Set of test instances of which ground-truth label sets
include the c-th label, the ranking list of all the test
instances in terms of the relatedness scores for the c-th
label

Given a training set of N annotated video clips, D =
Tr
|D|
{xxi ,yyi }i=1 , where x i is the visual input and y i ∈ {0, 1}|C | is
its binary target label vector in the i-th example: 0/1 element
indicates the presence/absence of a specific action belonging
to CTr in x i .
For a video instance x i in D, we divide it evenly into
T segments1 , segment-level visual representations are extracted, which are collectively denoted by {xxi1 ,xxi2 , · · · ,xxiT }.
At the t-th time step, the segment representation x it is fed
into a hidden LSTM layer and processed by this LSTM layer
and two subsequent fully-connected layers of linear activation functions (c.f. Figure 1). The latent visual embedding
of the t-th segment, e vit , is obtained as follows:

xi
sc
Y
yi
yc
E v , e vi
E s , e sc
dx , ds , de
oi
oc
φ v , Θv
φ s , Θs
φ
C(x̂x), L(x̂x)
Dc , Lc

unit retrospectively). W ’s and b’s are weight matrices and
bias vectors associated to different gates.
When LSTM units are used in a hidden layer as done
in our framework, the dimension of hidden vector h is determined by the number of LSTM units used in this hidden
layer. Hence, it is a hyper-parameter that has to be tuned on
a given dataset. The dimensions of weight matrices W are
thus determined by the dimensions of output vector h and
input x. The gates i, f , o and the cell state c are of the same
dimension as that of h.

3.3 Joint Latent Embedding Learning
Now we present the joint latent embedding learning in our
proposed framework. To facilitate our presentation, we summarize the notations used in this paper in Table 1.

e vit = φ v (xxit ;Θv ).

(1)

Here, φ v is the visual embedding function implemented by
the parametric visual model and Θv is a collective notation
of all the parameters in this model, including weights and
biases involved in this deep network.
Likewise, as depicted in Figure 1, the c-th label in a label
collection is first represented by a specific semantic representation, sc , that is fed to the semantic embedding function,
φ s , implemented by the parametric semantic model of which
all the parameters are denoted by Θs collectively. Thus, the
semantic embedding, e sc , of the c-th label is
e sc = φ s (ssc ;Θs ).

(2)

A score layer is employed in each of the visual and the
semantic models. In the visual model, the score layer takes
the outputs of the visual embedding layer at all the time steps
to yield the relatedness scores regarding all the labels for xi
with a dot product between the visual embedding of each
segment in xi and the semantic embedding of all the labels
in a label collection:
E s >,
o it =< e vit ,E

(3)
Tr

where E s ∈ Rde ×|C | is a collective notation of the semantic
B >= a T B is a vectoembedding of all the labels. Here, <aa,B
rial notation of the dot product between a vector, a , and each
column of a matrix, B . Then the relatedness scores between
this video instance and different labels are achieved by averaging over the scores on all the segments of this video instance:
oi =

1
T

T

∑ o it .

(4)

t=1

Likewise, the relatedness scores between different video instances and the c-th label in the label collection, o c ∈ RN×1 ,
1

A segment refers to a volume of multiple consecutive frames.

Multi-Label Zero-Shot Human Action Recognition via Joint Latent Embedding

is estimated in the same manner used in the visual model
based on the visual embedding of those video instances and
the semantic embedding of the c-th label. Thus, the i-th element of o c , the relatedness score regarding the i-th visual
instance is
oci =

1
T

T

∑ < e vit ,eesc > .

(5)

t=1

For the joint latent embedding learning, we need to optimize the parameters in the visual and the semantic models
with training data and proper loss functions (the technical
details are presented in Section 3.3.2). Assume that lv (·, ·)
and ls (·, ·) are two loss functions with respect to the visual
and the semantic model, the joint latent embedding learning
is boiled down to solving the following optimization problems:
|D|

Θv∗ = argminΘv ∑ lv (ooi ,yyi ),

(6)

i=1

Θs∗ = argminΘs

∑Tr ls (ooc ,yyc ).

(7)

c∈C

Here, the binary indicator vector y c ∈ I1×|D| is a row vector
Tr
in the target label matrix Y ∈ I|C |×|D| of a training dataset,
D, and non-zero elements in yc indicate that the c-th label
appears in the target label sets of the corresponding training
Tr
examples in D. The binary indicator vector y i ∈ I|C |×1 is
a column vector in Y , and non-zero elements in y i refers to
those labels in the target label set associated with the i-th
training example in D.
3.3.2 Loss Function
As described in Section 3.1, multi-label zero-shot learning
needs to establish a mapping that outputs a label-relatedness
score list for a video input where the scores of the relevant
labels are higher than those of irrelevant ones. Motivated by
the previous works (Burges et al. 2005; Zhang et al. 2016b),
we develop regularized ranking-based loss functions for our
joint latent embedding learning.
For the target label set expressed with binary indicators,
yi , in the i-th training example, (xxi ,yyi ), non-zero elements
indicate the relevant labels and zero elements expresses the
irrelevant labels to x i . By converting this binary indicator
representation, we can easily use non-zero elements in y i
and 1 −yyi to express the relevant and the irrelevant labels to
x i , respectively, where 1 is a vector where all the elements
are one and it has the same dimension as that of y i . Likewise,
y c , a binary indicator regarding whether the c-th action appears in training examples in D can be handled in the same
manner. Thus, the relatedness scores of x i to its positive and
negative labels, o i , are achieved with Eqs. (2) and (3), and

9

the relatedness scores of the c-th label to all the training examples, o c , are calculated with Eqs. (1) and (5). Based on
the above quantities, we can define our ranking-based loss
functions, lv (ooi ,yyi ) and ls (ooc ,yyc ): no loss is exerted to an instance if all the positive labels rank ahead of the negative
ones according to their relatedness scores. Otherwise, any
violation of the expected ranking incurs a loss.
Formally, we define the loss function for visual embedding of x i as follows:


lv (ooi ,yyi ) = ωi ∑ ∑ log 1 + exp v(oip , oiq , yip , yiq ) ,
p∈CTr q∈CTr

(8)
where ωi = (||yyi ||1 · ||11 − y i ||1 )−1 normalizes this perinstance RankNet loss as specified in (Burges et al. 2005).
The ranking loss incurred on two specific labels, p and q, is
v(oip , oiq , yip , yiq ) = (1 − yip )oip − yiq oiq .

(9)

Intuitively, the loss function defined by Eqs.(8) and (9) is
used to ensure ranking the relevant labels ahead of irrelevant ones when yip = 0 and yiq = 1. During learning, the
relatedness scores of the irrelevant labels will be decreased
radically when yip = 0 and yiq = 0, but the relatedness scores
of relevant labels are increased considerably when yip = 1
and yiq = 1.
Likewise, we define the loss function for semantic embedding of label c as follows:
|D| |D|

ls (ooc ,yyc ) = ωc

∑ ∑ log



1 + exp s(ocp , ocq , ycp , ycq ) , (10)

p=1 q=1

where ωc = (||yyc ||1 · ||11 − y c ||1 )−1 normalizes this per-class
RankNet loss (Burges et al. 2005) and 1 is a vector where
all the elements are one and it has the same dimension as
that of y c . The ranking loss incurred regarding two specific
visual instances, p and q, is
s(ocp , ocq , ycp , ycq ) = (1 − ycp )ocp − ycq ocq .

(11)

Intuitively, the loss function defined by Eqs.(10) and (11) is
able to ensure that the video instances conveying the action
of the c-th label are ranked above all those without this action when ycp = 0 and ycq = 1. Similarly, the video instances
without action c tend to have small relatedness scores when
ycp = 0 and ycq = 0, while the video instances having this action tend to have large relatedness scores when ycp = 1 and
ycq = 1.
To understand the motivation behind our loss functions
formulated in Eqs. (8)-(11), we would make the following
remarks. Our loss functions have the nature of both crossentropy and ranking-based losses. On the one hand, a crossentropy loss tends to enlarge marching scores of its relevant labels and to diminish those of its irrelevant labels simultaneously for training examples during learning. To this

10

end, all the labels are considered independently and treated
equally so that the less frequently used relevant labels might
be overlooked during learning. On the other hand, a rankingbased loss generally makes use of pairwise constraints to
exploring a relationship between labels associated with an
instance explicitly. However, the relatedness scores in the
ranking-based loss are not bounded and could hence vary
across different examples. Thus, some “difficult” pairs of labels is likely to incur larger cost that predominates the overall loss, which could make the learning biased to those pairs
of labels only. Furthermore, relatedness scores may vary in
a large range for different examples in a training set even
though proper ranking relationships among them are established. In this circumstance, the performance appears poor
in terms of a label-centric evaluation metric, which will be
described in Section 4.5. Thus, we tackle the above issues
with the loss functions that seek the synergy between crossentropy and ranking-based losses. In our loss functions,
losses incurred by the irrelevant labels may predominate at
the beginning of learning as there are much more irrelevant
labels than relevant ones to a training example. However,
minimizing this term allows for lowering relatedness scores
of irrelevant labels considerably so that the loss functions
gradually guide the learning to generate proper relatedness
scores for relevant labels via the ranking-based loss. Thus,
our proposed loss functions actually work on ranking-based
constraints with the regularization via properly confining
the magnitude of relatedness scores for relevant and irrelevant labels. Thus, our loss functions significantly distinguish
from a ranking-based loss without the regularization such as
the one used in Fast0Tag (Zhang et al. 2016b). Hence, we
expect that our regularized ranking-based losses would lead
to better generalization, which will be verified in our experiments.
3.3.3 Learning Algorithm
As formulated in Eqs. (6) and (7), learning is going to find
the optimal parameters, Θv∗ and Θs∗ , by minimizing two loss
functions, lv (ooi ,yyi ) and ls (ooc ,yyc ), defined in Eqs. (8) – (11).
However, the relatedness scores required in lv (ooi ,yyi ) regarding the visual model involve the output of the semantic
model, E s , and vice versa (c.f. Figure 1). Moreover, lv (ooi ,yyi )
requires the relatedness scores between all the candidate labels and each of training examples, while ls (ooc ,yyc ) needs
the relatedness scores between all the training examples and
each of all the action labels in CTr . Thus, our optimization
problems are very complex and unsolvable simultaneously
with commonly used local search methods, e.g., gradientdescent based methods.
Motivated by the works dealing with similar optimization problems, e.g., (Jiang et al. 2017; Kavukcuoglu et al.
2010), we come up with an learning algorithm to train the

Qian Wang, Ke Chen

Algorithm 1 Joint Latent Embedding Learning
Input: Randomly initialize parameters, Θv0 and Θs0 , in the visual and
the semantic models, respectively; extract the visual representations of training example, x i , i = 1, · · · , N, and the semantic representations of all the training labels, sc , ∀c ∈ CTr ; input the target
label matrix of the training set, Y ; pre-set the dimensionality of
joint latent embedding space, de .
Output: Optimal model parameters: Θv∗ and Θs∗ .
1: Generate the initial semantic embedding φ s (ssc ;Θs0 ), ∀c ∈
CTr ; t ← 0.
2: repeat
3: t ← t + 1;
4: Θvt = argminΘv ∑Ni=1 lv (ooi ,yyi ) with the current semantic embedding for one epoch;
5: Generate the visual embedding with the current visual model,
φ v (xxi ;Θvt ), i = 1, · · · , N;
6: Θst = argminΘs ∑c∈CTr ls (ooc ,yyc ) with the current visual embedding for one epoch;
7: Generate the semantic embedding with the current semantic
model φ s (ssc ;Θst ), ∀c ∈ CTr ;
8: until Stopping condition is met.
9: Θv∗ ← Θvt and Θs∗ ← Θst .

visual and the semantic models alternately during learning.
In our alternate learning strategy, our learning algorithm begins with randomly initializing the parameters in the semantic model and then use the initialized parameter to generate
the initial semantic embedding. By using the initial semantic embedding in lv (ooi ,yyi ), the visual model can be trained
with a local search method such as the mini-batch stochastic
gradient decent method. After one epoch, the current parameters in the visual model are frozen and used to generate the
visual embedding for all the examples. In the same manner, By using the current visual embedding in ls (ooc ,yyc ), the
semantic model is trained in the same manner. This alternate learning process carries on until a stopping condition is
satisfied. The details of this alternate learning algorithm is
described in Algorithm 1.
It is worth stating that two loss functions defined for visual and semantic model are related and the optimisation
of one model would naturally promote the other towards
its optimal solution. Hence, our alternative learning algorithms may converge after running finite epochs with the
same properties held for similar methods (Jiang et al. 2017;
Kavukcuoglu et al. 2010).

3.4 Multi-Label Zero-Shot Recognition
Once the joint latent embedding learning
is completed, we

obtain a mapping function: φ (xx,cc) = φ v (xx|Θv∗ ),φφ s (cc|Θs∗ )
where φ v (xx|Θv∗ ) and φ s (cc|Θs∗ )} are the visual and the semantic embedding functions implemented by the trained visual and semantic models, respectively. Then, we can use
this mapping function for multi-label zero-shot human action recognition.

Multi-Label Zero-Shot Human Action Recognition via Joint Latent Embedding

For recognition, we first extract the semantic representations of all the labels, including both known and unseen
labels, in a considered label collection: s c , ∀c ∈ C; C =
CTr ∪ CU and CTr ∩ CU = 0. By using the semantic embedding function, we achieve the sematic embedding of all
the labels: êesc = φ s (ssc |Θs∗ ), ∀c ∈ C. For a test video clip,
we divide it into T segments and extract its segment-level
representations collectively denoted by x̂x = {x̂x1 , x̂x2 , ..., x̂xT }.
By feeding x̂x to the visual embedding function, we achieve
its visual embedding: êev = φ v (x̂x|Θv∗ ). Thus, the relatedness
scores between this test video clip and all the actions in the
considered label collection C, including known and unseen
labels during learning, is achieved by
Score(x̂x, c) =< êev , êesc >, ∀c ∈ C.

(12)

Finally, we achieve a ranking action label list, L(x̂x), for this
test video clip by sorting its relatedness scores measured
against all the labels in C with Eq. (12):
 |C|
L(x̂x) = ci i=1 ,
(13)
where ∀ci , c j ∈ C, Score(x̂x, ci ) ≥ Score(x̂x, c j ) if i < j.

11

Breakfast: In this dataset (Kuehne et al. 2014), there are
1,989 video clips totally, where a video clip conveys several cooking actions. Totally, there are 49 cooking actions
(excluding the “silence” label), such as ‘stirring”, “pouring milk” and “opening the fridge”. Those actions are performed by 52 people in different kitchens. Although this
dataset is not collected especially for multi-label human action recognition, we would use it as a proof-of-concept test
bed.
Charades: This dataset (Sigurdsson et al. 2016) is collected
from hundreds of people recording videos in their own home
especially for video-based human activity analysis in daily
lives. Hence, it is very challenging for multi-label human action recognition. In this dataset, there are 9,848 video clips
involving 157 different human actions totally, acting out casual everyday activities. An average duration of video clips
is around 30 seconds and an average number of actions involved in a video clip is 6.8. Those actions are performed by
267 people from three continents, and more than one person appear in over 15% of all the video clips. The raw video
data (scaled to 480p) are used in our experiments, which are
available from the Charades project page2 .
4.1.2 Dataset Splits

4 Experimental Setting
In this section, we describe our experimental design and
settings, including datasets, visual and semantic representations, model learning, evaluation scenarios and criteria used
in our experiments. Moreover, we design a number of comparative experiments to exhibit the gain resulting from different components in our framework and to demonstrate the
effectiveness of our framework by a comparison to several
state-of-the-art multi-label ZSL methods that could be applied to human action recognition.

4.1 Datasets and Splits
We first describe datasets and their split settings used in our
experiments for simulation of zero-shot scenarios.
4.1.1 Datasets
To evaluate our framework, we employ two publicly available video datasets: Breakfast (Kuehne et al. 2014) and Charades (Sigurdsson et al. 2016), in our experiments. In both
datasets, at least two actions are involved in each video
clip and the duration of each video clip is relatively long,
which implies the temporal coherence information may be
explored and exploited in human action recognition. Hence,
both datasets are suitable to evaluate multi-label human action recognition. Below, we summarize the main aspects of
two video datasets.

To simulate a zero-shot scenario, we need to split a dataset
into training and test sets where a training set contains examples associated with only known classes while a test set
has test instances involving at least one unseen class. Unlike single-label ZSL where a dataset is automatically split
into training and test sets once unseen classes are specified,
the dataset split issue in multi-label ZSL becomes much
more complicated. In our experiments, we make two different split settings named instance-first and label-first split,
respectively, as illustrated in Figure 3.
Instance-First Split
This is a commonly used data split setting in nearly all the
existing multi-label ZSL works, e.g., (Mensink et al. 2014;
Nam et al. 2015; Zhang et al. 2016b). In this setting, instances in a dataset are first split into training, validation
and test subsets. The training and the validation subsets are
used for parameter estimation and hyper-parameter tuning,
the dimension of latent embedding space de and the number
of iterations of Algorithm 1 for our model. The test set that
may or may not involve unseen labels is reserved for performance evaluation. Then, we divide the action label collection into mutually exclusive known and unseen label sets.
Before learning, any unseen labels in the target label set of
an instance in the training and the validation subsets are removed as shown in the left plot of Figure 3. In other words,
only known labels in the target label sets of an instance in
those two datasets are used in learning. It is worth clarifying
2

http://allenai.org/plato/charades/

12

Qian Wang, Ke Chen

Table 2 Information on two different data split settings.
Dataset
Breakfast
Charades

Split Method
Instance-first
Label-first
Instance-first
Label-first

Instance-First Split

# Training Inst.
1196
1019/917/823
6385
4580/4176/3987

Label-First Split
Test
Instances

Test
Instances
Validation
Instances

Validation
Instances

Unseen
Labels

Training
Instances

Training
Instances

Unseen
Labels

Known
Labels

# Validation Inst.
126
200
1600
1000

Known
Labels

Fig. 3 Two different data split settings used in our experiments. In
the instance-first split (left plot), a data set is simply split into three
mutually exclusive subsets: training, validation and test subsets. Any
unseen labels associated with instances in training and validation subsets (shaded area) are removed from their target label sets before being
used in learning. In the label-first split (right plot), a number of labels
are first specified as unseen labels. Instances associated with any of
unseen labels form a test subset. The remaining data of known labels
are further divided into training and validation subsets to be used in
learning.

that unlike single-label ZSL, it is often infeasible to simulate a ZSL scenario by manipulating the validation set due
to insufficient data in two datasets used in our experiments.
Hence, the validation for hyper-parameter tuning in our experiments has to follow the typical protocol used in multilabel learning (Zhang and Zhou 2014).
To split the Breakfast dataset with this setting, we adopt
the pre-split by data collectors (Kuehne et al. 2014), where
the video clips of 13 people are reserved for test. We further
divide the rest video clips for training and validation: video
clips of 32 people for training and the remaining video clips
of seven people for validation. As a result, the numbers of
video clips for training, validation and test are 1,196, 126
and 667, respectively. Then we randomly split the 49 labels
into known and unseen labels: 10 labels reserved as unseen
labels and the rest 39 as known labels.
For the Charades dataset, we also adopt its pre-split
provided by data collectors (Sigurdsson et al. 2016), where
7,985 and 1,863 video clips are used for training and
test, respectively. We further divide training data into two
subsets: 6,385 for training and 1,600 for validation in our
experiments. Then we randomly choose 40 out of 157

# Test Inst.
667
770/872/966
1863
4268/4672/4861

# Known Labels
39
39
117
137

# Unseen Labels
10
10
40
20

human actions as unseen classes and the rest 117 human
actions are hence known actions.
Label-First Split
Although the instance-first data split setting is widely used
in multi-label ZSL, it suffers from a fundamental limitation.
It is well known that multiple labels together could frame
a specific concept and removing any label from this label
cohort may lead to a less accurate semantic meaning and
biases in learning. Furthermore, the instance-first split allows for accessing to visual features of instances involved
in unseen actions. To overcome this limitation, we propose
a novel data split setting for multi-label ZSL named labelfirst split. In this new setting, all the labels in a label collection used in a dataset are first divided into two mutually
exclusive subsets: known and unseen labels. Then, all the
instances having any unseen labels are reserved for test and
the rest instances of known labels only are further divided
into two subsets for training and validation. Due to sparsity of training data, the validation in the label-first split also
adopts the protocol used in multi-label learning (Zhang and
Zhou 2014).
To split the Breakfast dataset with this setting, we randomly choose 10 labels for unseen labels and the rest 39 labels are designated as known labels accordingly. Hence, this
dataset are naturally split into two sets for training and test.
The training data are further divided for training and validation. For validation, we randomly choose 200 instances from
the training data. Likewise, the Charades dataset is split by
using 20 randomly chosen label for unseen labels. Thus, the
remaining 137 labels become known labels. From the instances of known labels, we randomly choose 1,000 instance
used for validation.
For reliability, we repeat our experiments on each dataset
under each split setting for three trials. During a trial, training data given in the pre-split of each dataset are randomly divided into training and validation subsets with the
instance-first split setting, and a known/unseen label split on
each dataset is chosen randomly with the label-first split setting. For clarity, we summarize the data split information3
on two datasets in Table 2.
3 All the data splits and source code used in our experiments
are available on our project website: http://staff.cs.manchester.ac.uk/
∼kechen/MLZSHAR.

Multi-Label Zero-Shot Human Action Recognition via Joint Latent Embedding

4.2 Visual and Semantic Representations
In our experiments, we use visual representations extracted
with the existing C3D deep network (Tran et al. 2015) and
word vectors as semantic representations (Mikolov et al.
2013).
As suggested by Tran et al. (2015), the C3D features are
extracted for a segment of 16 frames with eight frames overlapping between two adjacent segments. Thus, a video clip
is always divided into T segments with the following treatment: we simply pad the sequence of all zeros to short video
clips and down-sample long video clips to ensure that each
video clip can be divided into T segments. Then, T C3D feature vectors collectively form the features of this video clip.
In our comparative study, we also convert T feature vectors
into a holistic visual representation via averaging those T
feature vectors. The use of such a holistic visual representation, we would demonstrate a performance gain benefiting from exploring/exploiting temporal coherence information underlying segments in a video clip. Based on a crossvalidation experiment, we choose T = 300 for Breakfast and
T = 20 for Charades. Although only C3D features are used
in our experiments, it is worth mentioning that other kinds of
visual representations, e.g., IDT features (Wang and Schmid
2013) and deep image features extracted on a frame basis,
can also be used straightforwardly in our framework.
In our experiments, we adopt Word2Vec as our semantic representation. Word2Vec was trained with a skip-gram
neural network on the Google News dataset of 100 billion
words (Mikolov et al. 2013). As a result, one action label is represented by a 300-dimensional word vector. Although only 300-dimensional word vectors are used in our
experiments, word vectors of different dimensionality may
be used, and moreover, other semantic representations, e.g.,
attributes, can be used in our framework without any difficulty if available.

4.3 Model Learning
In our experiments, model learning is implemented on Keras
(Chollet 2015), a high-level neural networks library, running
on top of either TensorFlow or Theano. As we use two neural networks to carry out the visual and the semantic models
(c.f. the left box in Figure 1), we need to decide the specific network architectures and relevant hyper-parameters on
two datasets during the model learning. The optimal hyperparameters are found by a grid-based search via a crossvalidation procedure. The Adam (Kingma and Ba 2014),
a stochastic optimisation method, is used for training our
model with its default configuration.
The visual model takes an segment-level C3D representation of dx = 4, 096 features as input to the LSTM layer

13

where there are N1 LSTM units. To improve the generalization, we also apply the dropout procedure (Srivastava et al.
2014) to the LSTM layer where a dropout rate needs specifying. The output of LSTM units are fed to a fully connected
dense layer of N2 neurons, and the output of this dense layer
are further fed to the visual embedding layer of de neurons.
During learning, there are no hyper-parameters involved in
the score and the average pooling layer in the visual model.
As described in Section 3.1, the semantic model is carried out by a fully-connected three-layer feed-forward neural network. The word vectors of ds = 300 dimensions are
first input to a hidden layer of N1 ReLu units (Nair and Hinton 2010). Subsequently, the output of this hidden layer are
fed to the semantic embedding layer of de linear units. Note
that for joint latent embedding learning, the dimension of
the semantic embedding space is set to the same of the visual embedding space in our experiments. Likewise, there
are no hyper-parameters involved in the score and the average pooling layer in the semantic model during learning.
4.4 Evaluation Scenarios
Multi-label zero-shot recognition is complex given the fact
that a test instance may be associated with a label set including both known and unseen class labels. Thus, there are different evaluation scenarios in previous works (Sandouk and
Chen 2016b; Zhang et al. 2016b); each focuses on a specific
aspect. Following their settings (Sandouk and Chen 2016b;
Zhang et al. 2016b), we evaluate our framework along with
other learning models used in our comparative study described later on in this section in three different scenarios:
Known-action only: In this setting, the performance is evaluated regarding only known (training) actions. This scenario
boils down to the conventional supervised multi-label learning. In this circumstance, we no longer take any unseen action labels into account during test; for a test instance, its
relatedness score ranking list contains only those regarding
known action labels in CTr and any unseen action label in
CU in its ground-truth label set, if there is, will be removed
such that the modified ground-truth set includes only known
action labels in CTr .
Unseen-action only: In this setting, the performance is evaluated regarding unseen (test) actions. This scenario is boils
down to a standard ZSL setting. In this situation, we no
longer consider any known action labels; for a test instance,
its relatedness score ranking list contains only those regarding unseen action labels in CU and any known action label in
CTr in its ground-truth label set, if there is, will be removed
such that the modified ground-truth set includes only unseen
action labels in CU .
Generalized ZSL: In this setting, the performance is evaluated regarding all the actions of which labels appearing in
a label collection C without considering if an action label

14

Qian Wang, Ke Chen

is known or unseen during learning. This scenario has been
named generalized ZSL in the machine learning community.
In this situation, both known and unseen action labels are
treated equally; for a test instance, its relatedness score ranking list contains those regarding all the action labels in C and
the evaluation is made against its ground-truth label set that
could be a mixture of known and unseen labels. It is worth
highlighting that the generalized ZSL setting is required by
multi-label zero-shot human action recognition in a real application.

4.5 Evaluation Metrics
There are a variety of evaluation metrics for multi-label
learning. Depending on the output of a multi-label learning
system, the evaluation metrics are generally divided into two
types: ranking-based and bipartition-based metrics (Nam
et al. 2015; Sorower 2010). Ranking-based metrics work
for the situation that a learning system yields a ranking list
of continuous-valued relatedness scores on all the candidate labels. In contrast, bipartition-based metrics are used
when a learning system produces only a binary indicator
vector for all the candidate labels, where 0/1 element expresses the presence/absence. Since our model yields a ranking list of continuous-valued relatedness scores, we employ
two commonly used ranking-based metrics for performance
evaluation (Li et al. 2016; Mensink et al. 2014; Nam et al.
2015; Sorower 2010; Zhang et al. 2016b), Instance-centric
Mean Average Precision (I-MAP) and Label-centric Mean
Average Precision (L-MAP). In addition, we employ other
metrics, precision, recall and F1 score, which have also been
used in the performance evaluation of multi-label learning
(Gong et al. 2013; Zhang and Zhou 2014).
To facilitate our presentation, we first define the
precision-at-k (Manning et al. 2009) in a generic form:
P@k(A, B) =

1
A ∩ B[1, · · · , k] ,
k

(14)

where A is a ground-truth set, B is a set of all the retrieved
entities ranked in terms of relevance, and B[1, · · · , k] indi |D |
cates top k entities in B. Given a test dataset, DT = x̂xi i=1T ,
a learning model yields a label-based ranking list for a test
instance, x̂xi ∈ DT , in terms of its relatedness scores to all the
 |C|
labels in C (c.f. Eqs. (12) and (13)): L(x̂xi ) = c j j=1 , where
∀c p , cq ∈ C, Score(x̂xi , c p ) ≥ Score(x̂xi , cq ) if p < q. Let C(x̂xi )
denote the ground-truth label set of x̂xi . I-MAP over a test
dataset DT is defined by



|C|
1 |DT | ∑c=1 P@c C(x̂xi ), L(x̂xi ) δ c,C(x̂xi )
,
I−MAP =
∑
|DT | i=1
|C(x̂xi )|
(15)



where δ c,C(x̂xi ) = 1 if c ∈ C(x̂xi ) and δ c,C(x̂xi ) = 0 otherwise.
While I-MAP measures the accuracy in terms of test instances, L-MAP is use to evaluate the performance from a
different perspective in light of candidate labels. Given a
specific label c ∈ C, a model predicts the relatedness scores
against the action specified by the c-th label for all the
test instances in DT . Hence, we can achieve an instance
|DT |
, in
based ranking list for the c-th label, Lc = x̂xi j j=1
terms of their relatedness scores against the c-th label where
∀x̂xi p , x̂xiq ∈ DT , Score(x̂xi p , c) ≥ Score(x̂xiq , c) if p < q. Let Dc
denote the collection of those test instances of which their
ground-truth label sets indeed include the c-th label. Thus,
the L-MAP over a test dataset, DT , is defined by
|C|

L−MAP =

1
∑
|C| c=1




|D |
∑i=1T P@i Dc , Lc δ x̂xi , Dc )
|Dc |

,

(16)

where δ (x̂xi , Dc ) = 1 if x̂xi ∈ Dc and δ (x̂xi , Dc ) = 0 otherwise.
Those widely used evaluation metrics in information retrieval have also been used in evaluating multi-label learning
systems, e.g., (Gong et al. 2013; Zhang and Zhou 2014). In
our experiments, we adopt overall top-k precision, recall and
F1 score measured over a test dataset, DT , which are defined
as follows:


|D |
∑i=1T P@k C(x̂xi ), L(x̂xi )
precision(k) =
,
k ∗ |DT |

recall(k) =

F1 (k) =


|D |
∑i=1T P@k C(x̂xi ), L(x̂xi )
|D |

∑i=1T |C(x̂xi )|

2 ∗ precision(k) ∗ recall(k)
.
precision(k) + recall(k)

,

(17)

(18)

(19)

4.6 Comparative Study
In our experiments, we conduct a comparative study from
two different perspectives: baseline and state-of-the-art
models. As a result, a number of baseline systems are designed to demonstrate roles played by the main components
in our framework while several state-of-the-art multi-label
ZSL algorithms are adapted for human action recognition.
For the comparative study, we evaluate each of different
models on three evaluation scenarios with evaluation metrics described in Sections 4.4 and 4.5 under the exactly same
conditions, including visual and semantic representations.

Multi-Label Zero-Shot Human Action Recognition via Joint Latent Embedding

4.6.1 Baseline Models
To investigate the roles played by different component
mechanisms employed in our framework, we design four
baseline models, random guess of scores, non-recurrent
connection, without semantic embedding and randomized
label representation, by manipulating our framework with
different purposes described as follows:
Random guess of scores (RGS): This is a general baseline
that provides a lowest performance bound used for a
reference to improvement made by a learning model. In our
work, we randomly generate relatedness scores of all the
candidate labels for a test instance. Then the performance
of this baseline model is evaluated based on the random
guess of scores. For reliability, we repeat the RGS process
100 times in our experiments and the statistics of the RGS
performance including mean and standard error of mean are
reported.
Non-recurrent connection (NRC): In our framework,
a LSTM layer of recurrent connections is employed to
capture temporal coherence underlying sequential video
data in the visual embedding learning. To examine the
role played by the LSTM layer, we replace the recurrent
connected layer with a fully connected layer without
recurrent connections and keep all other components in
our framework unchanged. By this setting, our model
is converted into a baseline model named non-recurrent
connection. During learning, obviously, this baseline model
no longer explicitly makes use of the temporal dependency
information underlying sequential segments in a video clip.
Algorithm 1 is used directly for parameter estimation.
Without semantic embedding (WSE): In our framework,
there is a semantic model for semantic embedding with the
motivation that the use of a joint latent embedding space
narrows the semantic gap between visual and semantic
domains and the zero-shot recognition should be done in
the joint latent embedding space. However, some existing
works, e.g., Fast0Tag (Zhang et al. 2016b), do not learn a
semantic embedding and the zero-shot recognition takes
place directly in the semantic space. To examine the
effectiveness of our semantic embedding, we come up with
a baseline model named without semantic embedding by
removing the semantic model from our framework. Thus,
the original semantic representations are used to replace
the semantic embedding representations, E s , required by
score layer of the visual model in our framework, which
is amount to mapping the visual space directly onto the
original semantic space. As this baseline model has only the
visual model, the learning becomes simpler; i.e., solving
the optimal problem formulated in Eq. (6) based on the
original semantic representations. Thus, the The Adam
(Kingma and Ba 2014) is used for parameter estimation.
It is worth clarifying that this baseline model is similar to

15

Fast0Tag (Zhang et al. 2016b) apart from an LSTM-based
visual embedding model used in our framework while a
feed-forward neural network without recurrent connections
is employed by Fast0Tag for visual embedding.
Randomized label representation (RLR): One of the
most important issues in ZSL is exploring/exploiting the
side information conveyed in the semantic domain. As our
framework works for multi-label zero-shot recognition,
we would investigate whether the semantic relatedness
information encoded in the semantic embedding, inherited
from the original semantic representations, is effectively
used in knowledge transfer. To this end, we design another
baseline model named randomized label representation
by replacing the word vector of a label with a vector of
the same dimensionality that is generated randomly and
normalized with the l2 norm to ensure that it has the same
range as that of the word vector. Apparently, the semantic
relatedness information no longer exists in such randomized
label representations. For parameter estimation, Algorithm
1 is used directly via replacing the semantic representations
of labels with the randomized label representations in
training data.

4.6.2 State-of-the-Art Methods
Although, to the best of our knowledge, there exists no work
in multi-label zero-shot human action recognition, we notice that there are a few multi-label ZSL algorithms. In our
comparative study, we adopt and extend those multi-label
ZSL algorithms for human action recognition for a thorough
evaluation of our proposed framework. Below, we briefly
describe those multi-label ZSL algorithms used in our experiments.
Direct Semantic Prediction (DSP): DSP is a well-known
baseline model used in previous works for multi-label ZSL,
e.g., (Sandouk and Chen 2016b). DSP is derived from direct attribute prediction originally proposed for single-label
ZSL (Lampert et al. 2014). The idea behind DSP is learning
a mapping function φ : X → S from visual to semantic space
directly for ZSL. By using the composition property of word
vectors, given a multi-labelled video clip, we use the mean
word vector achieved by averaging those word vectors of
the labels associated with this video clip to be its semantic
representation. Thus, the multi-label ZSL problem is boiled
down to single-label ZSL. In our experiments, we employ
support vector regressor models to learn the mapping function φ (·). Given a test instance x̂ , the learned φ (·) is used to
predict its compositional semantic representation ŝ = φ (x̂x̂),
then the prediction scores regarding different labels are estimated by measuring distances between the predictions and
the word vectors of all the labels in a label collection C:
Score(x̂x̂, c) =< ŝ ,ssc >, ∀c ∈ C, which leads to a label-based

16

Qian Wang, Ke Chen

ranking list, L(x̂x̂), in terms of semantic relatedness.
Convex combination of Semantic Embedding (ConSE):
ConSE is a ZSL algorithm proposed by Norouzi et al.
(2014), which can be naturally applied to multi-label ZSL.
As same as formulated in DSP, ConSE also learns a mapping
to predict a compositional semantic representation from the
visual representation of a given video clip. Instead of learning a direct mapping function as DSP does, however, ConSE
learns the conditional probabilities, P(c|xx) for ∀c ∈ CTr , regarding all the known actions with training data. For recognition, the compositional or collective semantic representation of a test instance x̂ is estimated by a convex combination of the semantic representations of top-5 known actions of the highest conditional probabilities. The combination weights are the l2 normalized conditional probabilities
of those top-5 known actions: ŝ = ∑c∈CTr P(c|x̂x̂)ssc . Thus, the
prediction scores regarding all the labels in an action label
collection C are Score(x̂x̂, c) =< ŝ ,ssc >, ∀c ∈ C, which leads
to a label-based ranking list, L(x̂x̂).
COSTA: COSTA is a method proposed by Mensink et al.
(2014) especially for multi-label zero-shot classification.
In this method, multi-label classification is converted into
a number of binary classification problems via a one-vsrest setting. |CTr | linear binary SVMs are trained based on
the examples regarding |CTr | known actions. Then the parameters of the SVM for an unseen label c ∈ CU is estimated by a weighted combination of the parameters of
|CTr | trained SVMs corresponding to known actions: w c =
|CTr |

∑k=1 αk βckw k . Here, w k is the parameters of the SVM regarding the k-th label in CTr and αk is a combination coefficient regarding the importance of this SVM achieved
|CTr |

via learning. βck = exp(−dck )/∑ j=1 exp(−dc j ), which is
a factor indicating the relatedness between a unseen label,
c ∈ CU , and a known label, k ∈ CTr , and dck is the semantic distance between labels c and k measured via their word
vectors. Thus, the SVM with the parameters w c can be used
to predict the unseen label c for a given test instance. Note
that our experimental results not reported in this paper due
to the limited space suggest that learning αk is not only time
consuming but also yields the poorer performance than that
where all |CTr | SVMs are treated equally; i.e., αk = 1 for
k = 1, · · · , |CTr |. Later on, we only report the best performance under this setting.
Fast0Tag: Fast0Tag is one of the latest state-of-the-art methods proposed for multi-label image tagging and multi-label
ZSL (Zhang et al. 2016b). The main idea behind Fast0Tag
is learning a mapping function φ : X → S from visual to semantic space for multi-label zero-shot tagging and recognition. Unlike DSP, a ranking-based loss function, RankNet,
is used to train a deep network to carry out φ (·) so that for
an video clip, its relevant labels should be ranked ahead of
those irrelevant ones. For recognition, the mapping function
yields the predicted semantic representation, φ (x̂x), for a test

instance, x̂x. Then, we can achieve the relatedness scores to
all the labels in an action label collection and the label-based
ranking list as same as done in DSP and ConSE. In our experiments, we strictly follow the same settings suggested by
Zhang et al. (2016b).
Fast0Tag+: Our work presented in this paper suggests that
the use of learned semantic embedding leads to better performance than the use of the original semantic representations directly. To further investigate this idea, we make an
extension of Fast0Tag by incorporating our semantic model
into the Fast0Tag model and name our extension Fast0Tag+.
As a result, Fast0Tag+ has an architecture resembling ours
(c.f. the left box of Fig.1), where the visual model is carried
out by the original Fast0Tag architecture while the semantic
model is the same as ours presented in Section 3. The original ranking-base loss functions in Fast0Tag are used and
our alternate learning algorithm described in Algorithm 1
is used for parameter estimation. For recognition, the same
procedure presented in Section 3.4 is used for a given test
instance. Here, we argue that this extension would provide
further evidence in examining the effectiveness of semantic
embedding learning.
In our comparative study, the optimal hyper-parameters
involved in baseline and state-of-the-art learning models are
sought during their learning with the same cross-validation
procedure as described in Section 4.3. Moreover, five stateof-the-art methods described above and ours are extensible to multi-label recognition straightforwardly; i.e., all
the actions are known in advance and their training examples are available during learning. Thus, we also report
the multi-label recognition performance, which not only extends our comparative study in a wider scope but also provides a benchmark to see how much the performance of each
method is degraded in a zero-shot circumstance.

5 Experimental Results
In this section, we report the detailed experimental results in
different settings and exemplify some typical test instances
via visual inspection.

5.1 Results on Optimal Hyper-parameters
As described in Section 4.3, we employ a grid-based search
procedure via cross-validation to find out the optimal hyperparameters in terms of the I-MAP performance (c.f. Section
4.5). In our experiments, we seek an optimal value from a set
of candidate hyper-parameters involved in different learning
models as follows:
Network architecture: The optimal architecture of neural
networks in a model used in our experiments is investigated by tuning different number of neurons in each hid-

Multi-Label Zero-Shot Human Action Recognition via Joint Latent Embedding

17

Table 3 Optimal hyper-parameter values of different learning models found by random search. Notation: IFS – Instance-First Split; LFS –
Label-First Split; V – Visual model; S – Semantic model; lr – learning rate; C, ε – soft-margin and percentage of support vectors in SVM/SVR.
N1 → N2 → de indicates a neural network architecture where N1 (dropout rate) is the number of neurons in the first hidden layer and dropout rate
used in learning; N2 is the number of hidden neurons in the second hidden layer; and de is the number of neurons in the latent embedding layer.

Dataset

Data Split

Model
NRC
WSE
RLR

IFS

DSP
ConSE
COSTA
Fast0Tag
Fast0Tag+
Ours

Breakfast
NRC
WSE
RLR
LFS

DSP
ConSE
COSTA
Fast0Tag
Fast0Tag+
Ours
NRC
WSE
RLR

IFS

DSP
ConSE
COSTA
Fast0Tag
Fast0Tag+
Ours

Charades
NRC
WSE
RLR
LFS

DSP
ConSE
COSTA
Fast0Tag
Fast0Tag+
Ours

1
V : lr = 1e−4; 2048(0.5) → 1024 → 500
S : lr = 1e−6; 500 → 500
V : lr = 1e−4; 1024(0) → 1024
V : lr = 1e−4; 256(0.5) → 1024 → 500
S : lr = 1e−6; 500 → 500
C = 1, ε = 0.1
C=1
C=1
V : lr = 1e−4; 8192(0) → 1024
V : lr = 1e−4; 4096(0) → 1024 → 800
S : lr = 1e−6; 500 → 800
V : lr = 1e−4; 256(0.5) → 2048 → 500
S : lr = 1e−6; 700 → 500
V : lr = 1e−4; 1024(0.5) → 1024 → 500
S : lr = 1e−6; 500 → 500
V : lr = 1e−4; 1024(0) → 1024
V : lr = 1e−4; 1024(0) → 2048 → 500
S : lr = 1e−6; 300 → 500
C = 100, ε = 0.1
C = 100
C = 100
V : lr = 1e−4; 4096(0.5) → 2048
V : lr = 1e−4; 8192(0.5) → 1024 → 800
S : lr = 1e−6; 500 → 800
V : lr = 1e−4; 512(0.5) → 1024 → 200
S : lr = 1e−6; 700 → 200
V : lr = 1e−4; 1024(0) → 1024 → 500
S : lr = 1e−6; 500 → 500
V : lr = 1e−4; 1024(0.5) → 1024
V : lr = 1e−4; 512(0.5) → 2048 → 800
S : lr = 1e−6; 700 → 800
C = 1, ε = 0.1
C=1
C=1
V : lr = 1e−4; 4096(0.5) → 2048
V : lr = 1e−4; 8192(0.5) → 2048 → 800
S : lr = 1e−6; 700 → 800
V : lr = 1e−4; 256(0.5) → 2048 → 800
S : lr = 1e−6; 700 → 800
V : lr = 1e−4; 4096(0) → 2048 → 500
S : lr = 1e−6; 300 → 500
V : lr = 1e−4; 1024(0.5) → 2048
V : lr = 1e−4; 256(0.5) → 2048 → 200
S : lr = 1e−6; 300 → 200
C = 1, ε = 0.1
C=1
C=1
V : lr = 1e−4; 4096(0.5) → 1024
V : lr = 1e−4; 8192(0.5) → 2048 → 800
S : lr = 1e−6; 700 → 800
V : lr = 1e−4; 1024(0.5) → 2048 → 800
S : lr = 1e−6; 500 → 800

den layer. In our proposed model, there are totally four
structural hyper-parameters. The number of hidden units in
the LSTM layer is selected from the candidate set, N1 =
256, 512, 1024. In the visual model, the number of neurons in the hidden layer above the LSTM layer is investigated with N2 = 1024, 2048. In the semantic model, the
number of neurons in the first hidden layer in the semantic model is chosen from N1 = 300, 500, 700. As a critical hyper-parameter in our algorithm, the dimension of la-

Split
2
V : lr = 1e−4; 2048(0.5) → 1024 → 500
S : lr = 1e−6; 500 → 500
V : lr = 1e−4; 512(0.5) → 2048
V : lr = 1e−4; 256(0.5) → 1024 → 500
S : lr = 1e−6; 500 → 500
C = 1, ε = 0.1
C=1
C=1
V : lr = 1e−4; 8192(0.5) → 2048
V : lr = 1e−4; 8192(0.5) → 2048 → 800
S : lr = 1e−6; 700 → 800
V : lr = 1e−4; 256(0.5) → 2048 → 200
S : lr = 1e−6; 300 → 200
V : lr = 1e−4; 1024(0.5) → 2048 → 200
S : lr = 1e−6; 300 → 200
V : lr = 1e−4; 512(0.5) → 2048
V : lr = 1e−4; 256(0.5) → 2048 → 200
S : lr = 1e−6; 300 → 200
C = 100, ε = 0.1
C = 100
C = 100
V : lr = 1e−4; 4096(0) → 1024
V : lr = 1e−4; 8192(0.5) → 1024 → 800
S : lr = 1e−6; 500 → 800
V : lr = 1e−4; 256(0.5) → 2048 → 200
S : lr = 1e−6; 300 → 200
V : lr = 1e−4; 1024(0) → 1024 → 500
S : lr = 1e−6; 500 → 500
V : lr = 1e−4; 512(0.5) → 2048
V : lr = 1e−4; 512(0.5) → 2048 → 800
S : lr = 1e−6; 700 → 800
C = 1, ε = 0.1
C=1
C=1
V : lr = 1e−4; 8192(0) → 2048
V : lr = 1e−4; 8192(0.5) → 2048 → 800
S : lr = 1e−6; 700 → 800
V : lr = 1e−4; 1024(0.5) → 2048 → 200
S : lr = 1e−6; 700 → 500
V : lr = 1e−4; 1024(0) → 2048 → 800
S : lr = 1e−6; 700 → 800
V : lr = 1e−4; 512(0.5) → 2048
V : lr = 1e−4; 256(0.5) → 1024 → 500
S : lr = 1e−6; 500 → 500
C = 1, ε = 0.1
C=1
C=1
V : lr = 1e−4; 8192(0.5) → 2048
V : lr = 1e−4; 8192(0) → 1024 → 500
S : lr = 1e−6; 500 → 500
V : lr = 1e−4; 512(0.5) → 1024 → 800
S : lr = 1e−6; 500 → 800

3
V : lr = 1e−4; 1024(0.5) → 1024 → 500
S : lr = 1e−6; 700 → 500
V : lr = 1e−4; 512(0.5) → 2048
V : lr = 1e−4; 1024(0) → 2048 → 500
S : lr = 1e−6; 300 → 500
C = 1, ε = 0.1
C=1
C=1
V : lr = 1e−2; 8192(0.5) → 2048
V : lr = 1e−4; 8192(0) → 1024 → 200
S : lr = 1e−6; 300 → 200
V : lr = 1e−4; 256(0.5) → 1024 → 500
S : lr = 1e−6; 500 → 500
V : lr = 1e−4; 1024(0.5) → 2048 → 200
S : lr = 1e−6; 300 → 200
V : lr = 1e−4; 1024(0) → 2048
V : lr = 1e−4; 512(0.5) → 1024 → 200
S : lr = 1e−6; 700 → 200
C = 100, ε = 0.1
C = 100
C = 100
V : lr = 1e−2; 8192(0.5) → 2048
V : lr = 1e−4; 4096(0) → 1024 → 200
S : lr = 1e−6; 300 → 200
V : lr = 1e−4; 256(0) → 2048 → 600
S : lr = 1e−6; 500 → 500
V : lr = 1e−4; 2048(0) → 1024 → 200
S : lr = 1e−6; 700 → 200
V : lr = 1e−4; 256(0.5) → 2048
V : lr = 1e−4; 256(0.5) → 1024 → 500
S : lr = 1e−6; 500 → 500
C = 1, ε = 0.1
C=1
C=1
V : lr = 1e−4; 4096(0.5) → 2048
V : lr = 1e−4; 8192(0.5) → 1024 → 800
S : lr = 1e−6; 500 → 800
V : lr = 1e−4; 1024(0.5) → 1024 → 800
S : lr = 1e−6; 500 → 800
V : lr = 1e−4; 4096(0) → 2048 → 500
S : lr = 1e−6; 300 → 500
V : lr = 1e−4; 1024(0.5) → 2048
V : lr = 1e−4; 256(0.5) → 1024 → 200
S : lr = 1e−6; 500 → 500
C = 1, ε = 0.1
C=1
C=1
V : lr = 1e−4; 4096(0.5) → 1024
V : lr = 1e−4; 4096(0.5) → 1024 → 500
S : lr = 1e−6; 500 → 500
V : lr = 1e−4; 512(0.5) → 2048 → 800
S : lr = 1e−6; 700 → 800

tent embedding space de , the number of neurons in the
visual/semantic embedding layers, is investigated by setting the candidate values, de = 200, 500, 800. For the nonrecurrent baseline model, the number of neurons in the first
hidden layer replacing the LSTM layer is chosen from N1 =
1024, 2048, 4096. For Fast0Tag and Fast0Tag+, the number
of neurons in the first and the second hidden layers are selected from N1 = 2048, 4096, 8092 and N2 = 1024, 2048, respectively.

18

Qian Wang, Ke Chen

Table 4 Multi-label zero-shot recognition performance (mean±SEM%) of the baseline models and our full model in three evaluation scenarios
under different data split settings. Notation: GZSL – generalized ZSL scenario; KnownA – Known-action only scenario; UnseenA – Unseen-action
only scenario. Other notations are the same as described in Table 3

Data Split

Evaluation Scenario

GZSL

IFS

KnownA

UnseenA

GZSL

LFS

KnownA

UnseenA

Model
RGS
NRC
WSE
RLR
Ours
RGS
NRC
WSE
RLR
Ours
RGS
NRC
WSE
RLR
Ours
RGS
NRC
WSE
RLR
Ours
RGS
NRC
WSE
RLR
Ours
RGS
NRC
WSE
RLR
Ours

Breakfast
L-MAP
I-MAP
P
R
10.9 ± 0.0 15.4 ± 0.0 8.8 ± 0.0 10.2 ± 0.0
27.6 ± 0.9 49.7 ± 1.4 36.3 ± 1.0 41.9 ± 1.1
31.2 ± 0.3 51.3 ± 0.2 37.1 ± 0.5 42.9 ± 0.5
28.7 ± 0.4 50.1 ± 0.8 35.6 ± 0.9 41.0 ± 1.1
30.3 ± 0.2 52.0 ± 1.0 37.3 ± 0.7 43.0 ± 0.8
11.4 ± 0.2 17.4 ± 0.1 9.6 ± 0.2 12.9 ± 0.0
29.8 ± 1.1 54.4 ± 0.2 36.2 ± 0.8 48.7 ± 0.4
33.6 ± 0.6 56.7 ± 0.8 38.0 ± 0.6 51.1 ± 0.2
33.0 ± 0.7 56.8 ± 0.4 37.1 ± 1.1 49.8 ± 0.7
33.0 ± 0.5 57.5 ± 0.7 38.0 ± 1.0 51.0 ± 0.8
8.5 ± 0.6 30.7 ± 0.6 6.1 ± 0.6 50.0 ± 0.0
18.2 ± 0.5 48.7 ± 1.9 9.6 ± 1.1 78.1 ± 2.0
21.1 ± 1.4 47.9 ± 4.2 9.0 ± 2.0 71.4 ± 9.6
10.3 ± 1.4 34.0 ± 5.6 5.8 ± 0.5 47.9 ± 4.7
18.2 ± 1.0 47.0 ± 2.9 9.8 ± 1.6 78.7 ± 6.0
15.2 ± 1.5 17.8 ± 0.4 11.3 ± 0.4 10.2 ± 0.0
23.0 ± 2.5 25.1 ± 0.8 16.5 ± 1.0 15.0 ± 1.1
25.2 ± 1.8 32.1 ± 2.5 23.7 ± 1.7 21.3 ± 1.1
22.7 ± 1.6 30.0 ± 1.7 22.0 ± 0.4 19.8 ± 0.3
24.1 ± 1.8 31.6 ± 1.4 23.5 ± 0.3 21.2 ± 0.7
14.7 ± 1.9 17.8 ± 0.2 10.1 ± 0.2 12.8 ± 0.0
23.1 ± 3.2 27.3 ± 1.2 16.2 ± 1.1 20.7 ± 1.7
25.6 ± 2.2 36.1 ± 3.0 23.7 ± 1.8 30.2 ± 2.6
24.2 ± 2.8 34.1 ± 2.8 22.3 ± 0.6 28.4 ± 0.8
25.0 ± 2.2 35.2 ± 2.0 23.7 ± 0.1 30.2 ± 0.7
16.8 ± 1.6 34.2 ± 1.3 16.2 ± 1.6 50.0 ± 0.0
23.0 ± 0.7 44.6 ± 3.2 20.5 ± 3.0 62.6 ± 4.7
24.4 ± 1.4 42.1 ± 5.0 20.4 ± 3.4 62.3 ± 7.0
19.2 ± 1.7 33.6 ± 0.9 16.0 ± 0.7 50.0 ± 3.2
21.7 ± 2.1 42.3 ± 4.1 20.3 ± 2.1 62.8 ± 4.8

Learning rate: For all the neural networks in the proposed
model, the baseline and the state-of-the-art models, candidate learning rates are to be {1e-2, 1e-4} for the visual model
and {1e-4, 1e-6} for the semantic model, respectively.
Number of epoches: Learning is stopped when the I-MAP
performance on a validation set is no longer improved within
the last 10 epoches. The optimal model chosen is the one that
yields the highest value of I-MAP on the validation set.
Dropout rate: The dropout rate used in the first layer of a
neural model during learning is selected from {0,0.5}.
SVM hyper-parameters: In our comparative study, ConSE
(Norouzi et al. 2014) and COSTA (Mensink et al. 2014)
employ a linear SVM for classification and DSP (Lampert
et al. 2014) uses a linear SVR for regression. In our experiments, an optimal soft-margin value C is sought from
C = 0.01, 1, 100. For SVR, the percentage of support vectors is always set to ε = 0.1 as suggested in literature.
As a result, the resultant optimal hyper-parameter values
in different experimental settings are summarized in Table 3.

5.2 Results on Comparison to Baseline Models
Table 4 summarizes all the results yielded by four baseline models described in Section 4.6.1 and our full model
described in Section 3.3. The experimental results are reported based on two different data split settings described in

F1
9.4 ± 0.0
38.9 ± 1.0
39.8 ± 0.5
38.1 ± 1.0
40.0 ± 0.8
11.0 ± 0.1
41.5 ± 0.7
43.6 ± 0.4
42.5 ± 1.0
43.5 ± 0.9
10.9 ± 0.9
17.1 ± 1.9
15.9 ± 3.4
10.3 ± 0.8
17.4 ± 2.7
10.7 ± 0.2
15.7 ± 1.1
22.4 ± 1.4
20.8 ± 0.1
22.3 ± 0.4
11.3 ± 0.1
18.2 ± 1.3
26.5 ± 2.1
25.0 ± 0.7
26.5 ± 0.3
24.4 ± 1.8
30.7 ± 4.0
30.6 ± 4.7
24.2 ± 0.7
30.5 ± 2.8

L-MAP
5.9 ± 0.0
8.3 ± 0.1
9.6 ± 0.1
9.8 ± 0.1
10.0 ± 0.0
6.1 ± 0.1
9.0 ± 0.2
10.4 ± 0.2
11.2 ± 0.3
10.9 ± 0.2
5.4 ± 0.0
6.3 ± 0.3
7.2 ± 0.3
5.6 ± 0.3
7.4 ± 0.5
5.2 ± 0.1
8.7 ± 0.1
9.1 ± 0.0
9.3 ± 0.2
9.3 ± 0.1
5.0 ± 0.1
8.6 ± 0.1
9.1 ± 0.1
9.6 ± 0.2
9.3 ± 0.1
6.9 ± 0.1
8.8 ± 0.3
9.6 ± 0.1
7.6 ± 0.2
9.3 ± 0.3

I-MAP
8.5 ± 0.0
20.8 ± 0.9
21.1 ± 0.8
20.8 ± 0.8
22.5 ± 0.4
9.4 ± 0.1
25.3 ± 0.7
24.9 ± 0.7
27.1 ± 0.8
26.3 ± 0.3
13.9 ± 0.0
20.2 ± 0.8
22.1 ± 2.1
12.2 ± 0.8
22.3 ± 1.6
7.9 ± 0.1
20.1 ± 0.2
20.3 ± 0.5
19.5 ± 1.1
21.3 ± 0.3
8.0 ± 0.1
22.0 ± 0.2
22.3 ± 0.5
23.8 ± 0.6
23.4 ± 0.6
19.4 ± 0.1
29.7 ± 0.7
29.2 ± 2.0
19.2 ± 0.3
30.0 ± 1.6

Charades
P
5.6 ± 0.0
20.9 ± 1.3
21.3 ± 0.8
21.2 ± 0.7
22.7 ± 0.4
5.8 ± 0.1
23.1 ± 0.9
22.7 ± 0.9
24.3 ± 0.8
23.9 ± 0.4
5.1 ± 0.0
8.5 ± 0.4
9.0 ± 0.7
4.1 ± 0.5
9.6 ± 1.0
5.1 ± 0.1
20.8 ± 0.4
20.9 ± 0.8
20.2 ± 1.3
22.0 ± 0.5
4.8 ± 0.1
20.8 ± 0.3
21.2 ± 0.5
22.3 ± 0.6
22.0 ± 0.7
6.7 ± 0.1
11.4 ± 0.3
10.9 ± 0.5
6.1 ± 0.2
11.5 ± 0.3

R
3.2 ± 0.0
11.9 ± 0.8
12.2 ± 0.5
12.1 ± 0.4
13.0 ± 0.3
4.3 ± 0.0
17.2 ± 0.4
16.9 ± 0.4
18.1 ± 0.4
17.8 ± 0.1
12.5 ± 0.0
21.1 ± 1.5
22.3 ± 2.4
10.3 ± 1.7
23.9 ± 2.4
3.2 ± 0.0
13.1 ± 0.3
13.1 ± 0.5
12.7 ± 0.7
13.8 ± 0.3
3.7 ± 0.0
15.7 ± 0.2
16.0 ± 0.2
16.8 ± 0.2
16.6 ± 0.3
25.0 ± 0.0
42.4 ± 1.4
40.9 ± 2.2
22.6 ± 0.7
42.9 ± 1.5

F1
4.1 ± 0.0
15.2 ± 1.0
15.5 ± 0.6
15.4 ± 0.5
16.5 ± 0.3
4.9 ± 0.0
19.7 ± 0.6
19.4 ± 0.6
20.7 ± 0.5
20.4 ± 0.2
7.2 ± 0.0
12.1 ± 0.6
12.8 ± 1.1
5.9 ± 0.8
13.7 ± 1.4
3.9 ± 0.0
16.0 ± 0.3
16.1 ± 0.6
15.6 ± 0.9
17.0 ± 0.3
4.2 ± 0.0
17.9 ± 0.1
18.2 ± 0.3
19.1 ± 0.3
18.9 ± 0.4
10.5 ± 0.1
17.9 ± 0.4
17.3 ± 0.8
9.6 ± 0.4
18.1 ± 0.6

Section 4.1.2, instance-first split (IFS) and label-first split
(LFS), under three different evaluation scenarios described
in Section 4.4; i.e., generalized ZSL, known-action only and
unseen-action only scenarios. For reliability, we report the
mean and standard error of the mean (SEM) of results (k = 5
used in evaluation metrics) over three randomly generated
known/unseen label splits for each evaluation scenario.
For the IFS setting, it is observed from Table 4 that all
the baseline models and ours perform significantly better
than the RGS, a random guess model, on two datasets regardless of evaluation scenarios apart from the RLR model
under the unseen-action only scenario. Due to a lack of
knowledge transfer in a random label representation, the
zero-shot performance of the RLR is expected. Overall,
our full model outperforms all the baseline models on both
datasets in the generalized ZSL scenario regardless of evaluation metrics and in the unseen-action only scenario in terms
of precision, recall and F1 although the WSE performs better on Breakfast in this scenario in terms of L-MAP and IMAP. In the known-action only scenario, the WSE and ours
yield the best performance on Breakfast in general. This result implies that the necessity of semantic embedding learning depends on the complexity of semantics underlying the
actions involved in a specific dataset. Although there are 49
human actions involved in Breakfast, all the video clips simply record the cooking processes of 10 dishes (Kuehne et al.
2014). Thus, multiple action labels regarding video clips on

Multi-Label Zero-Shot Human Action Recognition via Joint Latent Embedding

Breakfast are already well characterized by relevant word
vectors, the raw semantic representation used in our experiments. Hence, the semantic embedding learning does not
lead to any gain due to the simplicity of semantics underlying those cooking actions. In contrast, the performance of
the WSE and our full model on Charades lends evidence
to support the semantic embedding learning in the presence
of complex semantics underlying actions given the fact that
the semantic embedding learning leads to a gain over the
WSE on this data set in the known-action only scenario. We
also observe that the RLR leads to the best performance on
Charades in the known-action only scenario where the performance of our full model is slightly worse than that of the
RLR. This result suggests that a label representation is not
critically important for known actions in multi-label learning. Nevertheless, the performance in the unseen-action only
and the generalized ZSL scenarios clearly indicate the importance of a action label semantic representation for knowledge transfer required by ZSL. It is also evident from Table 4
that our full model always outperforms the NRC where there
are no recurrent connections. Thus, the comparison to the
baseline models clearly suggest that the performance gain is
brought by the use of an LSTM layer in the visual model
and the semantic embedding learning fulfilled in the semantic model.
For the LFS setting, results shown in Table 4 suggest
that all the baseline models perform significantly better than
a random guess. Overall, our full model outperforms those
baseline models on Charades in the generalized ZSL and
the unseen-action only scenarios. However, our full model
slightly under-performs the WSE in the generalized ZSL
and the known-action only scenarios for the same reason as
stated earlier for the results under the IFS setting. Also, the
RLR the NRC generally performs better on Charades in the
known-action only scenario and on Breakfast in the unseenaction only scenario, respectively. These results reveal that
the recurrent layer used in the visual model and the semantic
embedding learning in the sematic model are not sufficient
in generalizing for capturing temporal coherence underlying
an episode conveying an unseen action and exploring the intrinsic semantic relatedness between known and unseen actions due to insufficient training data in the LFS setting as
described below.
In general, results in Table 4 reveal that all the models
in question behave differently on two different data split settings. The performance of a model on the LFS setting is generally worse than that on the IFS setting. On Breakfast, there
are the same number of unseen labels but the recognition
performance on the LFS is much lower than that on the IFS.
On Charades, the performance in the LFS is also inferior to
that in the IFS despite the fact that the number of unseen actions in the LFS is a half of that in the IFS. This observation
suggests that the LFS setting, which is closer to a real appli-

19

cation, is more challenging than the IFS setting, which has
been widely used in existing methods in literature. With a
closer look at our experimental protocol, we notice that there
are more training examples in the IFS than those in the LFS
(c.f., Table 4.1.2). More critically, training examples contain
the instances associated with unseen labels in the IFS though
the relevant unseen labels to such an instance are not used
in learning. Note that those known and unseen labels are
linked by not only their semantic representations but also
the label co-occurrences in training examples. For example, we know that actions “pour cereal” and “stir cereal” are
closely related in the semantic space and have high probability of co-occurrences. Suppose those two actions fall into
known and unseen action categories. In the IFS, any test instance with confidently high score for “pour cereal” could
lead to a high score for “stir cereal”. Thanks to the label cooccurrences, “stir cereal” could be easily recognized despite
being an unseen action. In contrast, it is unlikely that in the
LFS the label-occurrence property is exploited since a video
clip containing the “pour cereal” action would be excluded
from the training data if it happens to contain the unseen
action “stir cereal”.
In summary, the comparison to the elaborated baseline
models facilitates the understanding of different components
employed in our proposed multi-label zero-shot human action recognition framework. By comparison to four baseline
models, our full model generally leads to favorable results
on two datasets measured with different evaluation metrics
in all three evaluation scenarios, although the experimental
results also reveal the limitation of components used in our
full model to be studied in our future research.

5.3 Results on Comparison to State-of-the-Art Methods
Table 5 summarizes the experimental results of the comparative study described in Section 4.6.2. Multi-label ZSL performance of five different methods (including Fast0Tag+, an
extension of Fast0Tag made by ourselves) with the reference
to a random guess is reported to be compared with our proposed framework. Again, all the experiments are conducted
with two different data split settings and evaluated under
three evaluation scenarios, as described in Section 5.2. For
reliability, we report the mean and the SEM of results (k = 5
used in evaluation metrics) over three randomly generated
known/unseen label splits for each evaluation scenario.
For the IFS setting, it is seen from Table 5 that all the
models perform better than random guess in most of evaluation scenarios. However, DSP and ConSE result in the
poorer performance than random guess in the unseen-action
only scenario on Breakfast in terms of some specific metric,
e.g., I-MAP. Overall, DSP and ConSE under-perform other
methods considerably in terms of all five evaluation metrics

20

Qian Wang, Ke Chen

Table 5 Multi-label zero-shot recognition performance (mean±SEM%) of five state-of-the-art methods and ours with the reference to random
guess in three evaluation scenarios under different data split settings. The notations are the same as used in Tables 3 and 4.

Data Split

Evaluation Scenario

GZSL

IFS

KnownA

UnseenA

GZSL

LFS

KnownA

UnseenA

Model
RGS
DSP(Lampert et al. 2014)
ConSE(Norouzi et al. 2014)
COSTA(Mensink et al. 2014)
Fast0Tag(Zhang et al. 2016b)
Fast0Tag+
Ours
RGS
DSP(Lampert et al. 2014)
ConSE(Norouzi et al. 2014)
COSTA(Mensink et al. 2014)
Fast0Tag(Zhang et al. 2016b)
Fast0Tag+
Ours
RGS
DSP(Lampert et al. 2014)
ConSE(Norouzi et al. 2014)
COSTA(Mensink et al. 2014)
Fast0Tag(Zhang et al. 2016b)
Fast0Tag+
Ours
RGS
DSP(Lampert et al. 2014)
ConSE(Norouzi et al. 2014)
COSTA(Mensink et al. 2014)
Fast0Tag(Zhang et al. 2016b)
Fast0Tag+
Ours
RGS
DSP(Lampert et al. 2014)
ConSE(Norouzi et al. 2014)
COSTA(Mensink et al. 2014)
Fast0Tag(Zhang et al. 2016b)
Fast0Tag+
Ours
RGS
DSP(Lampert et al. 2014)
ConSE(Norouzi et al. 2014)
COSTA(Mensink et al. 2014)
Fast0Tag(Zhang et al. 2016b)
Fast0Tag+
Ours

L-MAP
10.9 ± 0.0
21.3 ± 0.0
16.1 ± 0.2
19.7 ± 0.2
22.3 ± 1.1
23.4 ± 0.3
30.3 ± 0.2
11.4 ± 0.2
22.6 ± 0.5
17.1 ± 0.3
22.1 ± 0.7
23.9 ± 1.3
25.4 ± 0.2
33.0 ± 0.5
8.5 ± 0.6
15.9 ± 1.5
12.2 ± 0.3
9.2 ± 1.2
15.3 ± 0.9
15.1 ± 1.2
18.2 ± 1.0
15.2 ± 1.5
20.7 ± 1.7
18.5 ± 2.1
19.3 ± 2.1
22.5 ± 1.5
21.9 ± 1.1
24.1 ± 1.8
14.7 ± 1.9
14.1 ± 1.2
12.2 ± 0.8
19.8 ± 2.9
23.1 ± 2.0
22.2 ± 1.7
25.0 ± 2.2
16.8 ± 1.6
16.6 ± 0.8
13.7 ± 1.7
18.2 ± 1.1
21.1 ± 1.1
21.2 ± 0.8
21.7 ± 2.1

I-MAP
15.4 ± 0.0
25.4 ± 0.9
29.6 ± 0.3
37.4 ± 0.3
38.6 ± 0.2
40.6 ± 0.7
52.0 ± 1.0
17.4 ± 0.1
30.8 ± 1.9
33.0 ± 1.2
41.6 ± 0.7
44.3 ± 0.5
45.0 ± 0.5
57.5 ± 0.7
30.7 ± 0.6
27.0 ± 4.7
29.9 ± 5.0
37.4 ± 2.8
36.7 ± 4.1
39.4 ± 1.3
47.0 ± 2.9
17.8 ± 0.4
18.6 ± 1.9
20.2 ± 1.8
22.7 ± 2.1
24.3 ± 1.7
23.3 ± 1.1
31.6 ± 1.4
17.8 ± 0.2
18.1 ± 2.7
22.1 ± 0.6
25.1 ± 2.7
26.5 ± 1.9
24.9 ± 1.5
35.2 ± 2.0
34.2 ± 1.3
28.8 ± 1.4
33.0 ± 0.7
35.8 ± 1.2
39.1 ± 4.0
45.5 ± 3.4
42.3 ± 4.1

under all three evaluation scenarios. Such results demonstrate that simply combining semantic representations of cooccurred multiple labels into one collective representation
leads to catastrophic loss of semantic information, which is
mainly responsible for the poor performance of DSP and
ConSE in multi-label recognition. Fast0Tag generally outperforms COSTA on two datasets in terms of most of evaluation metrics. While COSTA learns a classifier for each
label separately without considering a relationship among
co-occurred labels, taking into account such a relationship
in Fast0Tag accounts for its better performance. By incorporating the semantic embedding learning into Fast0Tag,
Fast0Tag+, our extension of Fast0Tag, constantly improves
the performance of its original version in all the circumstances on two datasets in general. Once again, this result
lends us evidence to justify the necessity of semantic embedding learning used in our framework for zero-shot multilabel ZSL. In contrast, our model generally outperforms all
five models significantly in terms of five evaluation metrics under different evaluation scenarios on two datasets,

Breakfast
P
8.8 ± 0.0
16.8 ± 0.4
20.3 ± 0.1
28.8 ± 0.4
27.5 ± 0.1
29.2 ± 0.3
37.3 ± 0.7
9.6 ± 0.2
19.1 ± 0.9
21.0 ± 0.2
28.8 ± 0.4
29.8 ± 0.7
29.9 ± 0.4
38.0 ± 1.0
6.1 ± 0.6
5.8 ± 1.6
6.2 ± 0.9
7.4 ± 0.7
7.0 ± 1.4
7.4 ± 0.9
9.8 ± 1.6
11.3 ± 0.4
11.0 ± 1.5
12.7 ± 1.1
16.8 ± 1.0
16.2 ± 0.6
15.3 ± 0.2
23.5 ± 0.3
10.1 ± 0.2
8.5 ± 1.8
13.1 ± 0.6
16.8 ± 1.0
15.9 ± 0.4
15.1 ± 0.6
23.7 ± 0.1
16.2 ± 1.6
14.7 ± 0.6
14.7 ± 0.5
16.9 ± 1.7
18.5 ± 3.6
19.9 ± 1.8
20.3 ± 2.1

R
10.2 ± 0.0
19.4 ± 0.4
23.4 ± 0.1
33.2 ± 0.5
31.8 ± 0.1
33.7 ± 0.3
43.0 ± 0.8
12.9 ± 0.0
25.7 ± 1.3
28.2 ± 0.7
38.7 ± 0.2
40.0 ± 0.6
40.2 ± 1.1
51.0 ± 0.8
50.0 ± 0.0
47.6 ± 11.5
51.0 ± 6.9
60.1 ± 2.5
55.9 ± 6.7
60.1 ± 3.6
78.7 ± 6.0
10.2 ± 0.0
9.9 ± 1.0
11.4 ± 0.6
15.1 ± 0.7
14.6 ± 0.1
13.8 ± 0.6
21.2 ± 0.7
12.8 ± 0.0
10.3 ± 2.1
14.6 ± 1.2
21.3 ± 0.8
20.2 ± 0.8
19.3 ± 1.1
30.2 ± 0.7
50.0 ± 0.0
42.8 ± 1.3
50.9 ± 4.6
52.2 ± 2.0
56.0 ± 6.5
61.7 ± 2.4
62.8 ± 4.8

F1
9.4 ± 0.0
18.0 ± 0.4
21.7 ± 0.1
30.8 ± 0.5
29.5 ± 0.1
31.3 ± 0.3
40.0 ± 0.8
11.0 ± 0.1
21.9 ± 1.1
24.1 ± 0.4
33.0 ± 0.3
34.1 ± 0.6
34.3 ± 0.6
43.5 ± 0.9
10.9 ± 0.9
10.4 ± 2.7
11.1 ± 1.5
13.1 ± 1.0
12.4 ± 2.3
13.2 ± 1.5
17.4 ± 2.7
10.7 ± 0.2
10.4 ± 1.2
12.0 ± 0.8
15.9 ± 0.8
15.4 ± 0.3
14.5 ± 0.4
22.3 ± 0.4
11.3 ± 0.1
9.4 ± 1.9
13.8 ± 0.9
18.7 ± 0.9
17.8 ± 0.5
17.0 ± 0.8
26.5 ± 0.3
24.4 ± 1.8
21.9 ± 0.7
22.7 ± 0.2
25.4 ± 2.0
27.8 ± 4.9
30.0 ± 2.1
30.5 ± 2.8

L-MAP
5.9 ± 0.0
7.9 ± 0.0
7.3 ± 0.0
8.5 ± 0.1
9.3 ± 0.0
9.7 ± 0.1
10.0 ± 0.0
6.1 ± 0.1
8.5 ± 0.1
7.8 ± 0.2
9.3 ± 0.2
10.0 ± 0.2
10.5 ± 0.3
10.9 ± 0.2
5.4 ± 0.0
6.2 ± 0.4
5.8 ± 0.4
6.0 ± 0.3
7.1 ± 0.4
7.3 ± 0.4
7.4 ± 0.5
5.2 ± 0.1
7.4 ± 0.1
7.0 ± 0.1
8.9 ± 0.1
8.6 ± 0.1
9.0 ± 0.1
9.3 ± 0.1
5.0 ± 0.1
7.3 ± 0.1
6.8 ± 0.1
9.0 ± 0.1
8.4 ± 0.1
9.0 ± 0.1
9.3 ± 0.1
6.9 ± 0.1
8.5 ± 0.2
7.9 ± 0.4
7.6 ± 0.2
9.5 ± 0.1
9.3 ± 0.1
9.3 ± 0.3

I-MAP
8.5 ± 0.0
12.5 ± 0.1
13.9 ± 0.3
16.9 ± 0.7
20.6 ± 1.0
21.7 ± 0.6
22.5 ± 0.4
9.4 ± 0.1
13.9 ± 0.3
15.9 ± 0.4
19.7 ± 0.6
24.6 ± 0.6
25.7 ± 0.7
26.3 ± 0.3
13.9 ± 0.0
17.3 ± 1.4
17.3 ± 0.8
15.5 ± 1.0
20.2 ± 2.4
19.3 ± 0.5
22.3 ± 1.6
7.9 ± 0.1
12.1 ± 0.3
13.8 ± 0.1
17.3 ± 0.1
20.1 ± 0.4
20.9 ± 0.3
21.3 ± 0.3
8.0 ± 0.1
12.9 ± 0.4
15.0 ± 0.1
19.5 ± 0.1
22.4 ± 0.7
22.8 ± 0.5
23.4 ± 0.6
19.4 ± 0.1
22.8 ± 0.9
23.2 ± 0.4
24.9 ± 1.0
28.4 ± 2.5
30.2 ± 2.9
30.0 ± 1.6

Charades
P
5.6 ± 0.0
12.5 ± 0.2
14.6 ± 0.5
17.6 ± 0.9
20.6 ± 1.4
21.9 ± 0.8
22.7 ± 0.4
5.8 ± 0.1
12.4 ± 0.4
14.6 ± 0.7
17.6 ± 0.9
22.4 ± 0.7
23.3 ± 0.9
23.9 ± 0.4
5.1 ± 0.0
7.4 ± 0.8
7.0 ± 0.7
6.3 ± 0.8
8.3 ± 0.8
8.1 ± 0.3
9.6 ± 1.0
5.1 ± 0.1
12.4 ± 0.6
14.8 ± 0.1
18.6 ± 0.1
20.1 ± 0.9
21.4 ± 0.6
22.0 ± 0.5
4.8 ± 0.1
12.3 ± 0.7
14.7 ± 0.2
18.6 ± 0.1
20.8 ± 0.7
21.3 ± 0.5
22.0 ± 0.7
6.7 ± 0.1
8.1 ± 0.4
8.5 ± 0.0
9.2 ± 0.4
11.0 ± 0.5
11.6 ± 0.7
11.5 ± 0.3

R
3.2 ± 0.0
7.2 ± 0.1
8.3 ± 0.3
10.1 ± 0.5
11.8 ± 0.8
12.5 ± 0.5
13.0 ± 0.3
4.3 ± 0.0
9.2 ± 0.3
10.8 ± 0.3
13.1 ± 0.5
16.6 ± 0.3
17.3 ± 0.4
17.8 ± 0.1
12.5 ± 0.0
18.4 ± 1.9
17.2 ± 1.1
15.4 ± 1.5
20.8 ± 2.7
20.0 ± 0.6
23.9 ± 2.4
3.2 ± 0.0
7.7 ± 0.3
9.3 ± 0.1
11.7 ± 0.2
12.6 ± 0.6
13.5 ± 0.4
13.8 ± 0.3
3.7 ± 0.0
9.3 ± 0.3
11.1 ± 0.1
14.1 ± 0.2
15.7 ± 0.2
16.1 ± 0.1
16.6 ± 0.3
25.0 ± 0.0
30.3 ± 1.3
31.9 ± 0.3
34.4 ± 1.8
41.1 ± 2.2
43.2 ± 3.1
42.9 ± 1.5

F1
4.1 ± 0.0
9.1 ± 0.1
10.6 ± 0.4
12.8 ± 0.6
15.0 ± 1.0
15.9 ± 0.6
16.5 ± 0.3
4.9 ± 0.0
10.6 ± 0.3
12.4 ± 0.5
15.0 ± 0.6
19.1 ± 0.4
19.8 ± 0.6
20.4 ± 0.2
7.2 ± 0.0
10.6 ± 1.1
9.9 ± 0.8
8.9 ± 1.0
11.9 ± 1.2
11.5 ± 0.3
13.7 ± 1.4
3.9 ± 0.0
9.5 ± 0.4
11.4 ± 0.1
14.4 ± 0.1
15.5 ± 0.7
16.5 ± 0.5
17.0 ± 0.3
4.2 ± 0.0
10.6 ± 0.5
12.6 ± 0.1
16.0 ± 0.1
17.9 ± 0.4
18.3 ± 0.3
18.9 ± 0.4
10.5 ± 0.1
12.8 ± 0.6
13.5 ± 0.1
14.5 ± 0.7
17.4 ± 0.8
18.2 ± 1.2
18.1 ± 0.6

as highlighted with bold-font in Table 5. By comparing our
model to Fast0Tag+, we see three main differences between
them as follows: visual representations, network architectures for the visual model and loss functions. Regarding visual representations, our model uses the segment-based visual features for an instance while Fast0Tag+ employs an
instance-level holistic visual representation. For network architectures, we employ an LSTM layer with recurrent connections to capture temporal coherence among segments of a
video clip while Fast0Tag+ simply uses a feed-forward network. As described in Section 3.3.2, we propose an alternative loss function to that in Fast0Tag. Thus, those differences
together leverage our performance gain over Fast0Tag+,
which yet again lends us evidence to support our proposed
framework.
For the LFS setting, experimental results suggest that
most of the models in question have similar behavior to
that in the IFS setting, as shown in Table 5. Once again,
DSP and ConSE generally perform worse than other models and even under-perform random guess on Charades in

Multi-Label Zero-Shot Human Action Recognition via Joint Latent Embedding

21

Table 6 Multi-label recognition performance (mean±std%) of five state-of-the-art methods and ours.

Model
DSP(Lampert et al. 2014)
ConSE(Norouzi et al. 2014)
COSTA(Mensink et al. 2014)
Fast0Tag(Zhang et al. 2016b)
Fast0Tag+
Ours

L-MAP
21.6 ± 0.0
16.7 ± 0.0
21.5 ± 0.0
21.6 ± 1.5
23.2 ± 0.3
33.7 ± 0.3

I-MAP
25.2 ± 0.0
30.8 ± 0.0
39.3 ± 0.0
40.7 ± 0.6
42.1 ± 0.5
55.0 ± 0.2

Breakfast
P
16.6 ± 0.0
21.1 ± 0.0
29.5 ± 0.0
28.7 ± 0.6
30.4 ± 0.3
39.6 ± 0.3

R
19.1 ± 0.0
24.4 ± 0.0
34.0 ± 0.0
33.1 ± 0.7
35.1 ± 0.3
45.7 ± 0.3

the unseen-action only scenario. While COSTA yields better performance than DSP and ConSE overall, it generally
under-performs Fast0Tag, Fast0Tag+ and ours in all three
evaluation scenarios. Nevertheless, overall, our model does
not perform better than Fast0Tag+ in the unseen-action only
scenario although it outperforms Fast0Tag in the generalized
ZSL and the known-action scenarios. As described in Section 5.2, the LFS setting is more challenging than the IFS
setting and some salient visual features on test instances corresponding to unseen actions could completely miss in training examples. In this case, the use of a segment-level based
visual representation and an LSTM layer in the visual model
may not be able to generalize well due to a lack of training
examples. Although such a result does not sufficiently favor
the use of a segment-level based visual representation and
an LSTM layer in the visual model in the presence of limited training data, it is no doubt that introducing a semantic model to Fast0Tag leverages the performance gain. Once
again, experimental results here along with those compared
to the baseline models under our LFS setting reveal a training data sparsity issue that has to be addressed in any future
multi-label zero-shot human action recognition study.
Furthermore, Table 6 shows the experimental results in
conventional multi-label human action recognition, i.e., all
the actions are known in learning. In this circumstance, only
the IFS setting is applicable. Hence, we use the same IFS
setting as described in Section 4.1.2 but, unlike what has
been done for simulating a zero-shot scenario, do not reserve
any actions. Also we use the same procedure as done for
zero-shot learning to search for optimal hyper-parameters
for five models and ours and repeat the experiments on the
same data split as the IFS setting for three trials with different parameter initialization. As a result, we report the mean
and the standard deviation (std) of three-trial results yielded
by different methods. It is evident from Table 6 that our
model performs the best in conventional multi-label recognition on both datasets. To see the degraded performance in
a zero-shot scenario, we can compare the performance in the
generalized ZSL evaluation scenario under the IFS setting,
as shown in Table 5, to that reported in Table 6. By such
a comparison, it is seen that the zero-shot performance of
our model drops with a narrow margin (approximately less
than 10% overall in terms of five different evaluation met-

F1
17.7 ± 0.0
22.7 ± 0.0
31.6 ± 0.0
30.7 ± 0.6
32.6 ± 0.3
42.5 ± 0.3

L-MAP
8.0 ± 0.0
7.4 ± 0.0
9.1 ± 0.0
9.5 ± 0.0
10.0 ± 0.0
10.3 ± 0.2

I-MAP
12.6 ± 0.0
14.6 ± 0.0
18.6 ± 0.0
23.4 ± 0.0
24.3 ± 0.2
25.2 ± 0.1

Charades
P
13.0 ± 0.0
15.8 ± 0.0
19.5 ± 0.0
23.7 ± 0.1
24.6 ± 0.3
25.6 ± 0.2

R
F1
7.4 ± 0.0
9.5 ± 0.0
9.0 ± 0.0 11.5 ± 0.0
11.2 ± 0.0 14.2 ± 0.0
13.6 ± 0.1 17.3 ± 0.1
14.1 ± 0.2 17.9 ± 0.2
14.6 ± 0.1 18.6 ± 0.1

rics). Given the fact that 10 out of 49 and 40 out of 157
human actions are reserved as unseen labels on Breakfast
and Charades, respectively, this comparison on experimental results suggests that our proposed framework yields the
promising performance for multi-label zero-shot human action recognition, which is close to the performance in multilabel human action recognition. Experimental results shown
in Table 6 also suggest that other state-of-the-art methods
behave similarly to ours in general. However, we also observe an unusual phenomenon from their performance; i.e.,
by a comparison to the generalized ZSL performance reported in Table 5, DSP always yields slightly better performance in multi-label recognition than that in the generalized
ZSL scenario regardless of evaluation metrics on Breakfast
and so do Fast0Tag and Fast0Tag+ in terms of L-MAP. By a
closer look at the dataset and results in two experiments as
well as our analysis, we find that at least two factors account
for this unusual phenomenon: a) co-occurred labels associated with most of video clips on Breakfast are redundant
in light of semantics, and b) the single collective semantic
representation of co-occurred multiple labels used in DSP
is insensitive to missing of few co-occurred labels due to
the label information redundancy and the information loss
resulting from the averaging operation in forming the single representation. Thus, we reckon that this phenomenon is
rather specific to the nature of this dataset and the ZSL setting where there are only a small number of unseen labels.
In summary, our comparative study suggests that our
proposed framework yields the favorable results and outperforms the existing state-of-the-art methods in general although it does not always perform better than Fast0Tag+, an
extension of Fast0Tag made by ourselves. Also our experimental results demonstrate challenges in multi-label ZSL
via our novel LFS setting especially when training data
are less correlated to test instances associated with unseen
classes in both semantic and visual domains.

5.4 Visual Inspection
In general, visual inspection provides a manner that helps
us understand the behavior of a method intuitively. To gain
an intuitive insight into the multi-label zero-shot human ac-

22

Qian Wang, Ke Chen

DSP

ConSE

COSTA

Fast0Tag

Fast0Tag+

Ours

stir_egg
take_butter
crack_egg
pour_flour
stir_dough

crack_egg
stir_fry_egg
stir_egg
fry_egg
stir_dough

crack_egg
take_plate
butter_pan
stir_dough
pour_milk

crack_egg
fry_pancake
add_salt_pepper
put_egg2plate
pour_milk

crack_egg
butter_pan
pour_milk
stir_dough
take_plate

crack_egg
put_egg2plate
butter_pan
stir_egg
add_salt_pepper

Fig. 4 A test video clip in the IFS setting and the top-5 labels predicted by different methods. Its ground-truth labels are take bowl, crack egg,
put egg2plate, take plate,stir egg, pour egg2pan, stir fry egg, add salt pepper, butter pan.

DSP

ConSE

COSTA

Fast0Tag

Fast0Tag+

Ours

squeeze_orange
cut_orange
cut_fruit
take_squeezer
take_bowl

squeeze_orange
pour_juice
cut_orange
pour_milk
pour_sugar

squeeze_orange
cut_orange
take_glass
pour_juice
pour_milk

pour_milk
pour_juice
take_glass
spoon_powder
squeeze_orange

squeeze_orange
take_glass
cut_orange
pour_juice
take_bowl

put_fruit2bowl
cut_fruit
take_knife
peel_fruit
take_glass

Fig. 5 A test video clip in the IFS setting and the top-5 labels predicted by different methods. Its ground-truth labels are cut orange,
squeeze orange, pour juice, take glass.

DSP

ConSE

COSTA

Fast0Tag

Fast0Tag+

Ours

pour_milk
stir_egg
pour_flour
stir_milk
pour_sugar

crack_egg
pour_egg2pan
fry_egg
stir_fry_egg
put_egg2plate

crack_egg
put_egg2plate
pour_milk
add_salt_pepper
fry_egg

crack_egg
pour_milk
fry_pancake
put_egg2plate
take_plate

pour_milk
stir_milk
spoon_powder
add_teabag
pour_water

pour_milk
spoon_powder
stir_milk
add_teabag
pour_water

Fig. 6 A test video clip in the IFS setting and the top-5 labels predicted by different methods. Its ground-truth labels are cut orange,
squeeze orange, pour juice.

tion recognition, we visualize a number of typical test video
clips on Breakfast and the top-5 labels predicted by different
state-of-the-art methods described in Section 4.6.2 and ours
in terms of semantic relatedness scores. Our visual inspection mainly focuses on understanding of the behavior of our
model and issues arising from our work. As a result, Figures
4-7 illustrate several key frames to human actions in typical test video clips and the top-5 predicted labels by different methods or baseline models, where a correctly predicted

known label is highlighted with bold font and a correctly
predicted unseen label is marked with bold-italic font.
For the IFS setting, Figures 4-6 illustrate three typical results yielded by different methods. Figure 4 exemplifies the
success of our model, where all the top-5 labels predicted by
our model are the ground-truth actions including one unseen
action and no other methods can match the performance of
our model. This exemplified test instance suggests that the
use of an LSTM layer in our visual model facilitates the
recognition of distinctive actions in a video clip. Figure 5 re-

Multi-Label Zero-Shot Human Action Recognition via Joint Latent Embedding

23

Data Split

DSP

ConSE

COSTA

Fast0Tag

Fast0Tag+

Ours

IFS

stir_egg
crack_egg
take_eggs
take_butter
pour_flour

pour_egg2pan
pour_milk
pour_flour
pour_oil
pour_sugar

put_egg2plate
pour_milk
crack_egg
pour_oil
add_salt_pepper

crack_egg
fry_pancake
put_egg2plate
pour_milk
add_salt_pepper

crack_egg
pour_milk
stir_egg
put_egg2plate
pour_oil

crack_egg
take_plate
pour_oil
put_egg2plate
add_salt_pepper

LFS

squeeze_orange
spoon_powder
stir_egg
stir_milk
cut_bun

pour_sugar
pour_flour
pour_milk
pour_juice
pour_dough2pan

pour_sugar
add_salt_pepper
pour_milk
spoon_powder
put_bun_Together

pour_milk
spoon_powder
fry_egg
stir_milk
pour_juice

pour_milk
spoon_powder
stir_milk
cut_orange
pour_juice

squeeze_orange
cut_orange
pour_juice
cut_fruit
peel_fruit

Fig. 7 A test video clip appearing in in the IFS and LFS settings and the top-5 labels predicted by different methods in two data split settings. Its
ground truth labels are take bowl, crack egg, put egg2plate, take plate, stir egg, pour egg2pan, stir fry egg, add salt pepper, butter pan in
the IFS setting, and take bowl, crack egg, put egg2plate, take plate, stir egg, pour egg2pan, stir fry egg, add salt pepper, butter pan in the
LFS setting, respectively.

veals a test instance on which all other methods outperform
ours. By comparing those labels predicted by our model to
the ground truth, we can see that some predicted labels are
semantically correlated to the ground truth, e.g., cut fruit vs.
cut orange and put fruit2bowl vs. pour juice. This test instance demonstrates the limitation of our model in capturing
the fine features in visual domain to be associated to the accurate semantic description in semantic domain when there
exist many similar actions in a targeted application. Figure
6 shows a test instance where all the methods fail to have
any ground-truth labels in their top-5 predicted labels. Our
visual inspection on this test instance reveals that non-trivial
objects pertaining to different actions are concentrated in a
small region located in top-right of frames in this video clip.
Thus, it is extremely difficult to capture the useful information in visual domain, which poses a challenge to existing
human action recognition techniques. Three test instances illustrated in Figures 4-6 also provide some insight as to how
other methods work. For instance, ConSE is more likely to
yield the labels regarding frequently used words in a human
action domain. It is evident that ConSE predicts 4 out of the
top-5 labels pertaining to different actions taken on “egg”
for the instance shown in Figure 4 and all five “pour” actions
commonly taken in kitchen for the instance shown in Figure
6. This limitation is due to the fact that ConSE uses a single
collective semantic representation resulting from averaging
the semantic representations of multiple co-occurred labels,
which favors those frequently used word vectors but diminishes the opportunity of finding out infrequently used word
vectors in prediction.
Experimental results reported in Tables 4 and 5 suggest
that all the models including ours generally perform worse

under the LFS setting than under the IFS setting. On the one
hand, the LFS setting results in a training data sparsity issue in contrast to the IFS setting. To see this issue, let us take
the first split on Breakfast as an example. In this split shown
in Table 2, there are 1,196 and 1,019 training examples
in the IFS and the LFS setting, respectively. However, the
number of training examples pertaining to specific known
actions is significantly different in two split settings due
to different data split protocols described in Section 4.1.2.
For example, there are 330, 217, 254, 109 and 156 training
examples with target labels,“crack egg”, “put egg2plate”,
“take plate”, “stir fry egg” and “add salt pepper”, respectively, in the IFS setting. In contrast, there are only 23, 23,
81, 23 and 20 examples with the same target labels, respectively, in the LFS setting. On the other hand, there is a major
difference between those models and ours; i.e., our visual
model employs a hidden layer of recurrent connection to
capture temporal coherence underlying intrinsic visual features while those state-of-the-art models used in our comparative study do not have such a mechanism. It is well known
that a learning model of a higher complexity or a larger capacity demands more informative training data. To this end,
the training data sparsity issue more severely affects the performance of our model than other models; it is evident that
the performance gain from the use of an LSTM layer in our
visual model disappears due to a lack of sufficient training
data required in training our visual model for capture temporal coherence.
To understand the difference between the IFS and the
LFS settings and the training data sparsity issue intuitively,
we illustrate the results yielded by the state-of-the-art methods and ours on a common instance appearing in test sets

24

in two data split settings, as shown in Figure 7. It is evident that 4 out of the top-5 action labels predicted by our
model are the ground truth and all other models can predict some of ground-truth actions correctly under the IFS
setting. In contrast, however, none of the models correctly
predicts more than one ground-truth action for this exactly
same test instance under the LFS setting. The visual inspection on this test instance clearly demonstrates the distinction
between two data split settings, where visual features associated with unseen actions are available in the IFS setting but
unavailable in the LFS setting, and the training data sparsity
issue in the LFS setting, which poses a big challenge to not
only ours but also all the existing multi-label ZSL methods.
6 Concluding Remarks
In this paper, we have formulated human action recognition
as a multi-label zero-shot learning problem and provide an
effective solution by proposing a novel framework via joint
latent embedding learning. To carry out our framework, we
employ a neural network of the heterogeneous architecture
for visual embedding, where an LSTM layer is used to facilitate capturing temporal coherence information underlying
different actions from weakly annotated video data. Also,
we advocate the use of semantic embedding learning to facilitate bridging the semantic gap and effective knowledge
transfer, which is implemented by a feed-forward neural network. All the above contributions have been thoroughly verified via our comparative study with various well-motivated
settings. Experimental results on two benchmark multi-label
human action datasets suggest that our proposed framework
generally outperforms not only the baseline systems but also
several state-of-the-art multi-label ZSL approaches in all the
different test scenarios.
Although we have demonstrated favorable results on two
benchmark datasets in comparison to state-of-the-art approaches, our observations on the performance of all the
approaches used in our comparative study including ours
suggest that the existing multi-label ZSL techniques are not
ready for a real application; in particular, the performance
becomes even worse under the label-first split setting corresponding to a real scenario that a human action recognition
system is deployed. Nevertheless, our experiment results including visual inspection provide insightful information for
improving our proposed framework. In our ongoing work,
we would address issues arising from our experiments and
observations with proper techniques. To address the training
data sparsity issue revealed in our experiments, we would
develop unsupervised learning algorithms to discover salient
yet intrinsic visual features from unlabeled video clips and
further incorporate proper temporal constraints into our loss
functions to better capture temporal coherence. Moreover,
we would introduce attention mechanisms to our model for

Qian Wang, Ke Chen

improving implicit salient feature extraction of different actions involved in a video clip during the visual embedding
learning. Also, we would employ alternative semantic representations developed by ourselves (Wang and Chen 2017a),
which encode the semantic relatedness between action labels more accurately, in the semantic embedding learning to
facilitate knowledge transfer.
While our framework is proposed especially for multilabel zero-shot human action recognition, we would highlight that it is directly applicable to multi-label human action
recognition without modification as demonstrated in our experiments. Also, our framework is easy to adapt for tackling various multi-label ZSL problems in different domains.
For example, we can apply our framework to miscellaneous
multi-label zero-shot classification tasks on temporal or sequential data, e.g., acoustic event classification, straightforward as well as multi-label zero-shot learning tasks on static
data, e.g., object recognition, by replacing a neural network
of the heterogeneous architecture only with a neural network
of only feed-forward connections in the visual model. Thus,
we are going to explore such extensions and applications in
our future work.

References
Abu-El-Haija, S., Kothari, N., Lee, J., Natsev, P., Toderici, G.,
Varadarajan, B., and Vijayanarasimhan, S. (2016). Youtube-8M:
A large-scale video classification benchmark. arXiv preprint
arXiv:1609.08675, .
Akata, Z., Reed, S., Walter, D., Lee, H., and Schiele, B. (2015). Evaluation of output embeddings for fine-grained image classification.
In IEEE Conference on Computer Vision and Pattern Recognition
(CVPR) (pp. 2927–2936).
Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton,
N., and Hullender, G. (2005). Learning to rank using gradient
descent. In Proceedings of the 22nd international conference on
Machine learning (ICML) (pp. 89–96). ACM.
Cabral, R., De la Torre, F., Costeira, J. P., and Bernardino, A. (2015).
Matrix completion for weakly-supervised multi-label image classification. IEEE transactions on pattern analysis and machine
intelligence, 37, 121–135.
Changpinyo, S., Chao, W.-L., Gong, B., and Sha, F. (2016). Synthesized classifiers for zero-shot learning. In IEEE Conference on
Computer Vision and Pattern Recognition (CVPR).
Chollet, F. (2015). Keras. https://github.com/fchollet/keras.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009).
Imagenet: A large-scale hierarchical image database. In IEEE
Conference on Computer Vision and Pattern Recognition (CVPR)
(pp. 248–255). IEEE.
Donahue, J., Anne Hendricks, L., Guadarrama, S., Rohrbach, M.,
Venugopalan, S., Saenko, K., and Darrell, T. (2015). Long-term
recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition (CVPR) (pp. 2625–2634).
Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Mikolov, T.
et al. (2013). Devise: A deep visual-semantic embedding model.
In Advances in neural information processing systems (NIPS) (pp.
2121–2129).

Multi-Label Zero-Shot Human Action Recognition via Joint Latent Embedding
Fu, Y., and Huang, T. (2010). Manifold and subspace learning for
pattern recognition. Pattern Recognition and Machine Vision, 6,
215.
Fu, Y., Yang, Y., Hospedales, T., Xiang, T., and Gong, S. (2014). Transductive multi-label zero-shot learning. In Proceedings of British
Machine Vision Conference (BMVC).
Gong, Y., Jia, Y., Leung, T., Toshev, A., and Ioffe, S. (2013). Deep
convolutional ranking for multilabel image annotation. arXiv
preprint arXiv:1312.4894, .
Gong, Y., Ke, Q., Isard, M., and Lazebnik, S. (2014). A multi-view
embedding space for modeling internet images, tags, and their
semantics. International journal of computer vision, 106, 210–
233.
Graves, A. (2013). Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, .
Gu, F., Sridhar, M., Cohn, A., Hogg, D., Flórez-Revuelta, F., Monekosso, D., and Remagnino, P. (2016). Weakly supervised activity analysis with spatio-temporal localisation. Neurocomputing,
216, 778–789.
Guillaumin, M., Mensink, T., Verbeek, J., and Schmid, C. (2009). Tagprop: Discriminative metric learning in nearest neighbor models
for image auto-annotation. In IEEE International Conference on
Computer Vision (ICCV) (pp. 309–316). IEEE.
Hochreiter, S., and Schmidhuber, J. (1997). Long short-term memory.
Neural computation, 9, 1735–1780.
Jiang, Y.-G., Wu, Z., Wang, J., Xue, X., and Chang, S.-F. (2017). Exploiting feature and class relationships in video categorization
with regularized deep neural networks. IEEE Transactions on
Pattern Analysis and Machine Intelligence, (p. (in presss)).
Kavukcuoglu, K., Ranzato, M., and LeCun, Y. (2010). Fast inference in
sparse coding algorithms with applications to object recognition.
arXiv preprint arXiv:1010.3467, .
Kingma, D. P., and Ba, J. (2014). Adam: A method for stochastic
optimization. In Proceedings of the 3rd International Conference
on Learning Representations (ICLR).
Kuehne, H., Arslan, A., and Serre, T. (2014). The language of actions:
Recovering the syntax and semantics of goal-directed human activities. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition (CVPR) (pp. 780–787).
Lampert, C. H., Nickisch, H., and Harmeling, S. (2014). Attributebased classification for zero-shot visual object categorization.
IEEE Transactions on Pattern Analysis and Machine Intelligence,
36, 453–465.
Lei Ba, J., Swersky, K., Fidler, S. et al. (2015). Predicting deep zeroshot convolutional neural networks using textual descriptions. In
Proceedings of the IEEE International Conference on Computer
Vision (ICCV) (pp. 4247–4255).
Li, X., Uricchio, T., Ballan, L., Bertini, M., Snoek, C. G., and Bimbo,
A. D. (2016). Socializing the semantic gap: A comparative survey
on image tag assignment, refinement, and retrieval. ACM Computing Surveys, 49, 14.
Ma, C.-Y., Chen, M.-H., Kira, Z., and AlRegib, G. (2017). TS-LSTM
and temporal-inception: Exploiting spatiotemporal dynamics for
activity recognition. arXiv preprint arXiv:1703.10667, .
Manning, C. D., Raghavan, P., and Schtze, H. (2009). An introduction
to inf. retrieval. In An Introduction to Inf. Retrieval. Cambridge
University Press.
Mensink, T., Gavves, E., and Snoek, C. G. (2014). Costa: Cooccurrence statistics for zero-shot classification. In Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2441–2448).
Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J.
(2013). Distributed representations of words and phrases and
their compositionality. In Advances in neural information processing systems (NIPS) (pp. 3111–3119).

25

Nair, V., and Hinton, G. (2010). Rectified linear units improve restricted boltzmann machines. In Internationl Conference on Machine Learning (ICML).
Nam, J., Kim, J., Mencı́a, E. L., Gurevych, I., and Fürnkranz, J. (2014).
Large-scale multi-label text classification – revisiting neural networks. In Joint European Conference on Machine Learning and
Knowledge Discovery in Databases (ECML-PKDD) (pp. 437–
452). Springer.
Nam, J., Mencı́a, E. L., Kim, H. J., and Fürnkranz, J. (2015). Predicting unseen labels using label hierarchies in large-scale multilabel learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML-PKDD) (pp.
102–118). Springer.
Norouzi, M., Mikolov, T., Bengio, S., Singer, Y., Shlens, J., Frome,
A., Corrado, G. S., and Dean, J. (2014). Zero-shot learning by
convex combination of semantic embeddings. In International
Conference on Learning Representations (ICLR).
Ren, Z., Jin, H., Lin, Z., Fang, C., and Yuille, A. (2015). Multi-instance
visual-semantic embedding. arXiv preprint arXiv:1512.06963, .
Sandouk, U., and Chen, K. (2016a). Learning contextualized semantics from co-occurring terms via a siamese architecture. Neural
Networks, 76, 65–96.
Sandouk, U., and Chen, K. (2016b). Multi-label zero-shot learning via
concept embedding. arXiv preprint arXiv:1606.00282, .
Sigurdsson, G. A., Varol, G., Wang, X., Farhadi, A., Laptev, I., and
Gupta, A. (2016). Hollywood in homes: Crowdsourcing data collection for activity understanding. In European Conference on
Computer Vision (ECCV) (pp. 510–526). Springer.
Sorower, M. S. (2010). A literature survey on algorithms for multilabel learning. Technical Report, Oregon State University, Corvallis, U.S.A., .
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov (2014). Dropout: A simple way to prevent neural networks
from overfitting. Journal of Machine Learning Research, 15,
1929–1958.
Tang, P., Wang, X., Huang, Z., Bai, X., and Liu, W. (). Deep patch
learning for weakly supervised object classification and discovery. Pattern Recognition, (p. (in press)).
Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015).
Learning spatiotemporal features with 3d convolutional networks.
In IEEE International Conference on Computer Vision (ICCV)
(pp. 4489–4497). IEEE.
Wang, H., and Schmid, C. (2013). Action recognition with improved
trajectories. In Proceedings of the IEEE international conference
on computer vision (pp. 3551–3558).
Wang, J., Yang, Y., Mao, J., Huang, Z., Huang, C., and Xu, W. (2016).
Cnn-rnn: A unified framework for multi-label image classification. In Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition (CVPR) (pp. 2285–2294).
Wang, Q., and Chen, K. (2017a). Alternative semantic representations
for zero-shot human action recognition. In Proceedings of European Conference on Machine Learning and Principles and Practice of Knowledge Discovery (ECML-PKDD’17).
Wang, Q., and Chen, K. (2017b). Zero-shot visual recognition via bidirectional latent embedding. International Journal of Computer
Vision, 124, 356–383.
Wei, Y., Xia, W., Lin, M., Huang, J., Ni, B., Dong, J., Zhao, Y., and
Yan, S. (2016). HCP: A flexible cnn framework for multi-label
image classification. IEEE transactions on pattern analysis and
machine intelligence, 38, 1901–1907.
Weston, J., Bengio, S., and Usunier, N. (2010). Large scale image
annotation: Learning to rank with joint word-image embeddings.
Machine learning, 81, 21–35.
Xian, Y., Akata, Z., Sharma, G., Nguyen, Q., Hein, M., and Schiele,
B. (2016). Latent embeddings for zero-shot classification. In
IEEE Conference on Computer Vision and Pattern Recognition

26
(CVPR).
Yue-Hei Ng, J., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O.,
Monga, R., and Toderici, G. (2015). Beyond short snippets: Deep
networks for video classification. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR)
(pp. 4694–4702).
Zhang, L., Xiang, T., and Gong, S. (2017). Learning a deep embedding
model for zero-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Zhang, M. L., and Zhou, Z. H. (2014). A review on multi-label learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 26, 1819–1837.
Zhang, Y., Acharyya, R., Liu, J., and Gong, B. (2016a). Infinitelabel learning with semantic output codes. arXiv preprint
arXiv:1608.06608, .
Zhang, Y., Gong, B., and Shah, M. (2016b). Fast zero-shot image tagging. In Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition (CVPR) (pp. 5985–5994). IEEE.
Zhang, Z., and Saligrama, V. (2015). Zero-shot learning via semantic similarity embedding. In IEEE International Conference on
Computer Vision (ICCV) (pp. 4166–4174).
Zhang, Z., and Saligrama, V. (2016). Zero-shot learning via joint latent
similarity embedding. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition (CVPR) (pp. 6034–
6042).
Zhou, Z.-H., and Zhang, M.-L. (2007). Multi-instance multi-label
learning with application to scene classification. Advances in neural information processing systems (NIPS), 19, 1609.

Qian Wang, Ke Chen

