1

Human Activity Recognition Using Robust
Adaptive Privileged Probabilistic Learning

arXiv:1709.06447v1 [] 19 Sep 2017

Michalis Vrigkas, Evangelos Kazakos, Christophoros Nikou, and Ioannis A. Kakadiaris
Abstract—In this work, a novel method based on the learning using privileged information (LUPI) paradigm for recognizing complex
human activities is proposed that handles missing information during testing. We present a supervised probabilistic approach that
integrates LUPI into a hidden conditional random field (HCRF) model. The proposed model is called HCRF+ and may be trained using
both maximum likelihood and maximum margin approaches. It employs a self-training technique for automatic estimation of the
regularization parameters of the objective functions. Moreover, the method provides robustness to outliers (such as noise or missing
data) by modeling the conditional distribution of the privileged information by a Student’s t-density function, which is naturally integrated
into the HCRF+ framework. Different forms of privileged information were investigated. The proposed method was evaluated using four
challenging publicly available datasets and the experimental results demonstrate its effectiveness with respect to the-state-of-the-art in
the LUPI framework using both hand-crafted features and features extracted from a convolutional neural network.
Index Terms—Hidden conditional random fields, learning using privileged information, human activity recognition

✦

1

I NTRODUCTION

Recent advances in computer vision such as video surveillance and human-machine interactions [1], [2] rely on machine learning techniques trained on large scale human
annotated datasets. However, training data may not always
be available during testing and learning using privileged
information (LUPI) [3], [4] has been used to overcome
this problem. The insight of privileged information is that
one may have access to additional information about the
training samples, which is not available during testing.
Consequently, classification models may often suffer
from “structure imbalance” between training and testing
data, which may be represented by the LUPI paradigm.
The LUPI technique simulates a real-life learning condition,
when a student learns from his/her teacher, where the latter
provides the student with additional knowledge, comments,
explanations, or rewards in class. Subsequently, the student
should be able to face any problem related to what he/she
has learned without the help of the teacher. Taking advantage of this learning model, the LUPI framework has also
been used in several machine learning applications such as
boosting [5] and clustering [6].
The problem of human activity understanding using
privileged knowledge is on its own a very challenging
task. Since privileged information is only available during
training, one should combine both regular and privileged
information into a unified classifier to predict the true class
label. However, it is quite difficult to identify the most
useful information to be used as privileged as the lack of
informative data or the presence of misleading information
may influence the performance of the model.
We address these issues by presenting a new probabilistic approach, based on hidden conditional random fields
(HCRFs) [7], called HCRF+. The proposed method is able to
learn human activities by exploiting additional information
about the input data, that may reflect on natural or auxiliary

$YDLODEOHLQIRUPDWLRQ
GXULQJWUDLQLQJ
2ULJLQDOLQIRUPDWLRQ
YLGHR

2ULJLQDOLQIRUPDWLRQ
YLGHR

3ULYLOHJHGLQIRUPDWLRQ
L$XGLR

LL3RVH

$YDLODEOHLQIRUPDWLRQ
GXULQJWHVWLQJ

LLL$WWULEXWHV
&ODSSLQJKDQGV 1R
6KDNHKDQGV
<HV
,QGRRU
<HV
8UEDQVFHQH
1R
'DQFLQJPXVLF 1R
)RUPDOVSHHFK <HV
&DPHUD]RRP 1R

+&5)
&ODVVLILFDWLRQ
$FWLYLW\ODEHO

3XVK
+XJ
.LFN
+DQGVKDNH9
.LVV
$JJUHVVLYH
%LUWKGD\SDUW\

+LJKILYH
1HXWUDO
'HSDUW
$SSURDFK
3XQFK
:HGGLQJGDQFH
*UDGXDWLRQ

Fig. 1: Robust learning using privileged information. Given
a set of training examples and a set of additional information about the training samples (left), our system can successfully recognize the class label of the underlying activity
without having access to the additional information during
testing (right). We explore three different forms of privileged information (e.g., audio signals, human poses, and
attributes) by modeling them with a Student’s t-distribution
and incorporating them into the HCRF+ model.
properties about classes and members of the classes of the
training data (Fig. 1) and it is used for training purposes
only but not for predicting the true classes (where, in
general, this information is missing).
In particular, the proposed HCRF+ method differentiates
from previous approaches [8], which may also use the
LUPI paradigm, by incorporating privileged information
in a supervised probabilistic manner, which facilitates the
training process by learning the conditional probability dis-

2

tribution between human activities and observations. We
also introduce a novel technique for automatic estimation
of the optimal regularization parameters for the learning
process. The method is adaptive as the regularization parameters are computed from the training data through a
self-training procedure. It is worth noting that the proposed
methodology is not limited to the use of a specific form of
privileged information, but it is general and may handle any
form of additional data.
Moreover, our method can efficiently manage dissimilarities in input data, which may correspond to noise, missing
data, or outliers, using a Student’s t-distribution to model
the conditional probability of the privileged information.
Such dissimilarities may harm the classification accuracy
and lead to excessive sensitivity when input data is insufficient or contains large intra-class variations. In particular,
the use of Student’s t-distribution is justified by the property
that it has heavier tails than a standard Gaussian distribution, thus providing robustness to outliers [9].
The main contributions of our work can be summarized in the following points. A human activity recognition
method is proposed, which exploits privileged information
in a probabilistic manner by introducing a novel classification scheme based on HCRFs to deal with missing or
incomplete data during testing. Both maximum likelihood
and maximum margin approaches are incorporated into
the proposed HCRF+ model. Moreover, a novel technique
for adaptive estimation of the regularization term during
the learning process is introduced by incorporating both
privileged and regular data. Finally, contrary to previous
methods, which may be sensitive to outlying data measurements, a robust framework for recognizing human activities
is intergraded by employing a Student’s t-distribution to
attain robustness against outliers.
The remainder of the paper is organized as follows: in
Section 2, a review of the related work is presented. Section
3 presents the proposed HCRF+ approach including the
maximum likelihood and maximum margin approaches for
learning the model’s parameters and the automatic estimation of the regularization terms. In Section 4, experimental
results are reported, and a discussion about the performance
of the proposed approach is offered in Section 5. Finally,
conclusions are drawn in Section 6.

2

R ELATED WORK

A major family of methods relies on learning human activities by building visual models and assigning activity
roles to people associated with an event [10], [11]. In recent
years, there has been an increased focus on the combination
of different kinds of modalities, such as visual and audio
information, for activity classification [12], [13]. A shared
representation of human poses and visual information has
also been explored [14], [15]. However, the effectiveness of
such methods is limited by tracking inaccuracies in human
poses and complex backgrounds. To this end, Cherian et
al. [16] explored several kinematic and part-occlusion constraints for decomposing human poses into separate limbs
to localize the human body. Eweiwi et al. [17] reduced the
required amount of pose data using a fixed length vector of
more informative motion features for each skeletal point.

Special focus has also been given in recognizing human
activities from movies or TV shows by exploiting scene
contexts to localize activities and understand human interactions [18], [19]. Ramanathan et al. [20] improved the
recognition accuracy of such complex videos by relating
textual descriptions and visual context to a unified framework. Guadarrama et al. [21] proposed an alternative to
the previous approach that takes a video clip as input and
generates short textual descriptions, which may correspond
to an activity label that is unseen during training. However,
natural video sequences may contain irrelevant scenes or
scenes with multiple actions. Shao et al. [22] mixed appearance and motion features using multi-task deep learning
for recognizing group activities in crowded scenes collected
from the web. Marı́n-Jiménez et al. [23] used a bag of visualaudio words scheme along with late fusion for recognizing
human interactions in TV shows. Even though their method
performs well in recognizing human interaction, the lack of
an intrinsic audio-visual relationship estimation limits the
recognition problem.
Intermediate semantic features representation for recognizing unseen actions during training has been extensively studied [24], [25], [26]. These intermediate features
are learned during training and enable parameter sharing
between classes by capturing the correlations between frequently occurring low-level features [27].
Recent methods that exploited deep neural networks have demonstrated remarkable results in large-scale
datasets [28]. Donahue et al. [29] proposed a recurrent
convolutional architecture, where long short-term memory
(LSTM) networks [30] are connected to convolutional neural
networks (CNNs) that can be jointly trained to simultaneously learn spatio-temporal dynamics. Wang et al. [31]
presented a new video representation that employs CNNs
to learn multi-scale convolutional feature maps and introduced the strategies of trajectory-constrained sampling and
pooling to encode deep features into informative descriptors. Tran et al. [32] introduced a 3D ConvNet architecture
that learns spatio-temporal features using 3D convolutions.
A novel video representation, that can summarize a video
into a single image by applying rank pooling on the raw
image pixels, was proposed by Bilen et al. [33]. Feichtenhofer
et al. [34] introduced a novel architecture for two stream
ConvNets and studied different ways for spatio-temporal
fusion of the ConvNet towers. Zhu et al. [35] argued that
videos contain one or more key volumes that are discriminative and most volumes are irrelevant to the recognition
process. To this end, they proposed a unified deep learning
framework to simultaneously identify discriminative key
volumes and train classifiers, while they discarded all irrelevant volumes.
The LUPI paradigm was first introduced by Vapnik
and Vashist [3] as a new classification setting to model a
real world learning process (i.e., teacher-student learning
relationship) in a max-margin framework, called SVM+.
Pechyony and Vapnik [36] formulated an algorithm for risk
bound minimization with privileged information. Several
variants of the original SVM+ have been proposed in the
literature including SVM+ with L1 regularization [37] and
multi-task SVM+ [38].
Fouad et al. [39] proposed a combination of privileged

3

3.1 HCRF+ Model Formulation

y

h1

x1

h2

x∗1

x2

hT

···

x∗2

···

xT

x∗T

Fig. 2: Graphical representation of the chain structure model.
The grey nodes are the observed features (xi ), the privileged
information (x∗i ), and the unknown labels (y ), respectively.
The white nodes are the unobserved hidden variables (h).

information and metric learning. The privileged information
was used to change the metric of the input data and thus any
classifier could be used. Wand and Ji [40] also proposed two
different loss functions that exploit privileged information
and can be used with any classifier. The first model encoded
privileged information as an additional feature during training, while the second approach considered that privileged
information can be represented as secondary labels. Wang et
al. [41] incorporated privileged information in a latent maxmargin model, where the additional knowledge was propagated through the latent nodes and the classification was
performed from the regular data. Although this approach
relaxes the strong assumptions of regular and privileged
data relation for classification, it is limited by the slack
variable estimation through SVM optimization. In this work,
we address this problem by replacing the slack variables for
the maximum margin violation and solve the unconstrained
soft-margin SVM optimization problem.
Serra-Toro et al. [42] proved that successfully selecting information that can be treated as privileged is not a
straightforward problem. The choice of different types of
privileged information in the context of an object classification task implemented in a max-margin scheme was also
discussed in [43]. Both regular and privileged features were
considered of equivalent difficulty for recognizing the true
class. Wang et al. [44] proposed a Bayesian network to learn
the joint probability distribution of input features, output
target, and privileged information. A combination of the
LUPI framework and active learning has also been explored
by Vrigkas et al. [45] to model human activities in a semisupervised scheme. Recently, the LUPI paradigm has been
employed with applications on gender classification, facial
expression recognition, and age estimation [46], [47], [48].

3

M ETHODOLOGY

We consider a labeled data set with N video sequences
consisting of triplets D = {(xi,j , x∗i,j , yi )}N
i=1 , where xi,j ∈
RMx ×T is an observation sequence of length T with j =
1 . . . T . For example, xi,j might correspond to the j th frame
of the ith video sequence. Furthermore, yi corresponds to
a class label defined in a finite label set Y . In the context
of robust learning using a privileged information paradigm,
additional information about the observations xi is encoded
in a feature vector x∗i,j ∈ RMx∗ ×T . Such privileged information is provided only at the training step and it is not
available during testing. Note that we do not make any
assumption about the form of the privileged data.
In particular, x∗i,j does not necessarily share the same
characteristics with the regular data, but is rather computed
as a very different kind of information, which may contain
verbal and/or non-verbal multimodal cues such as (i) visual
features, (ii) semantic attributes, (iii) textual descriptions of
the observations, (iv) image/video tags, (v) human poses,
and (vi) audio cues. The goal of LUPI is to use the privileged
information x∗i,j as a medium to construct a better classifier
for solving practical problems than one would learn without
it. In what follows, we omit indices i and j for simplicity.
The HCRF+ model is a member of the exponential family
and the probability of the class label given an observation
sequence is given by:
X
exp (E(y, h|x, x∗ ; w) − A(w)) , (1)
p(y|x, x∗ ; w) =
h

where w = [θ, ω] is a vector of model parameters, and
h = {h1 , h2 , . . . , hT }, with hj ∈ H being a set of latent
variables. In particular, the number of latent variables may
be different from the number of samples, as hj may correspond to a substructure in an observation. Moreover, the
features follow the structure of the graph, in which no
feature may depend on more than two hidden states hj and
hk [7]. This property not only captures the synchronization
points between the different sets of information of the same
state, but also models the compatibility between pairs of
consecutive states. We assume that our model follows the
first-order Markov chain structure (i.e., the current state
affects the next state). Finally, E(y, h|x; w) is a vector of
sufficient statistics and A(w) is the log-partition function
ensuring normalization:
XX
exp (E(y ′ , h|x, x∗ ; w)) .
(2)
A(w) = log
y′

Different sufficient statistics E(y|x, x∗ ; w) in (1) define
different distributions. In the general case, sufficient statistics consist of indicator functions for each possible configuration of unary and pairwise terms:
E(y, h|x, x∗ ; w) =

X

j∈V

Our method uses HCRFs, which are defined by a chained
structured undirected graph G = (V, E) (Fig. 2), as the
probabilistic framework for modeling the behavior of a
subject in a video. During training, a classifier and the
mapping from observations to the label set are learned. In
testing, a probe sequence is classified into its respective state
using loopy belief propagation (LBP) [49].

h

Φ(y, hj , xj , x∗j ; θ) +

X

Ψ(y, hj , hk ; ω) ,

j,k∈E

(3)
where the parameters θ and ω are the unary and the pairwise weights, respectively, that need to be learned. Moreover, the potential functions correspond to the structure of
the graphical model as illustrated in Fig. 2. For example, a
unary potential does not depend on more than two hidden
variables hj and hk , and a pairwise potential may depend

4

on hj and hk , which means that there must be an edge (j, k)
in the graphical model.
The unary potential is expressed by:
X
φ1,ℓ (y, hj ; θ1,ℓ )
Φ(y, hj , xj ,x∗j ; θ) =
(4)
ℓ
+ φ2 (hj , xj ; θ2 ) + φ3 (hj , x∗j ; θ3 ) ,

thus, a global solution is not guaranteed and we can estimate w∗ that are locally optimal. The loss function in Eq. (9)
is optimized using a gradient-descent method such as the
limited-memory BFGS (LBFGS) method [50].

and it can be seen as a state function, which consists of three
different feature functions. The label feature function, which
models the relationship between the label y and the hidden
variables hj , is expressed by:
X X
θ1,ℓ 1(y = λ)1(hj = a) , (5)
φ1,ℓ (y, hj ; θ1,ℓ ) =

We can easily transform the optimization problem of the
loss function defined in Eq. (9) into a max-margin problem
by substituting the log of the summation over the hidden
states and the labels in Eq. (10) with maximization [51]. The
goal is to maximize the margin between the score of the
correct label and the score of the other labels. To learn the
parameters w∗ we need to minimize a loss function of the
form:

λ∈Y a∈H

where 1(·) is the indicator function, which is equal to 1, if
its argument is true and 0 otherwise. The number of the
label feature functions is |Y| × |H|. The observation feature
function, which models the relationship between the hidden
variables hj and the observations xj , is defined by:
X ⊤
θ 2 1(hj = a)xj .
φ2 (hj , xj ; θ2 ) =
(6)

3.3 Maximum Margin Learning

L(w) =

h

y 6=yi ,
h

and ξi ≥ 0,
where parameter λ is a tuning parameter. Although we
add slack variables ξ to max-margin optimization, they
eventually vanish. We do not estimate the slacks, but we
replace them with the Hinge loss error [52] that penalizes
the loss when the constraints in Eq. (11) are violated:

ℓi (w) = max(0, 1 + ( ′max E(y ′ , h|xi , x∗i ; w)
y 6=yi ,h

a∈H

The pairwise potential is a transition function and represents the association between a pair of connected hidden
states hj and hk and the label y . It is expressed by:
X X
ω ℓ 1(y = λ)1(hj = a)1(hk = b) .
Ψ(y, hj , hk ; ω) =

− max E(yi , h|xi , x∗i ; w))) .

3.2 Maximum Likelihood Learning
In the training step the optimal parameters w∗ are estimated
by maximizing the following loss function:
N
X
1
1
L(w) =
log p(yi |xi , x∗i ; w) − 2 kwk2 .
λ
2σ
i=1 i

(9)

The first term is the log-likelihood of the posterior probability p(y|x, x∗ ; w) and quantifies how well the distribution
in Eq. (1) defined by the parameter vector w matches the
labels y , while λ is a tuning parameter. It can be rewritten
as:
X
log p(yi |xi , x∗i ; w) = log
exp(E(y, h|xi , x∗i ; w))
h
(10)
X
′
∗
exp(E(y , h|xi , xi ; w)) .
− log
y ′ 6=y,h

The second term in Eq. (9) is a Gaussian prior with variance
σ 2 and works as a regularizer. The use of hidden variables
makes the optimization of the loss function non-convex,

(12)

h

The optimization problem in (11) is equivalent to the
optimization of the following unconstrained problem:

L(w) =

ℓ
λ∈Y
a,b∈H

(8)
The number of the transition functions is |Y| × |H|2 . HCRF+
keeps a transition matrix for each label.

(11)

s.t. max
E(y ′ , h|xi ,x∗i ; w) − max E(yi , h|xi , x∗i ; w) ≤ ξi − 1,
′

a∈H

The number of the observation feature functions is considered to be |Y| × |Mx |. Finally, the privileged feature
function, which models the relationship between the hidden variables hj and the privileged information x∗j , has
|Y| × |Mx∗ | number of functions and is defined by:
X ⊤
θ 3 1(hj = a)x∗j .
(7)
φ3 (hj , x∗j ; θ3 ) =

N
X
1
1
ξi + 2 kwk2
λ
2σ
i=1 i

N
X
1
1
ℓi (w) + 2 kwk2 .
λ
2σ
i
i=1

(13)

However, the quantity max(0, ·) is not differentiable and
thus, Eq. (11) is hard to solve. To overcome this problem
we adopt the bundle method [53], which uses sub-gradient
descent optimization algorithm.
3.4 Estimation of Regularization Parameters
Both maximum likelihood and max-margin loss functions
introduce regularization parameters that control data fidelity and these regularization parameters in Eq. (9) and Eq.
(13) may be obtained in closed form. Here, we examine the
case of maximum likelihood optimization as the estimation
of the regularization parameters for the max-margin optimization is equivalent. We can rewrite the loss function in
Eq. (9) as the sum of individual smoothing functionals for
each of the training samples N :

L(w) =

N
X

log p(yi |xi , x∗i ; w) − αi (w)kwk2 ,

(14)

i=1

λi
.
where αi (w) ≡
2σ 2
In general, the choice of the regularization parameter for
the optimization of the loss function should be a function

5

of model parameters w. We consider a linear function f (·)
between αi and each term of the loss function:

αi (w) = f log p(yi |xi , x∗i ; w) − αi (w)kwk2

(15)
= γi log p(yi |xi , x∗i ; w) − αi (w)kwk2 ,
where γ is determined by the sufficient conditions for convergence. From Eq. (15), the regularization parameter αi is
computed as:

αi (w) =

log p(yi |xi , x∗i ; w)
,
1
+ kwk2
γi

and therefore:
1
> log p(yi |xi , x∗i ; w) − αi (w)kwk2 .
γi

(16)

(17)

We assume that the privileged information is more informative for classifying human actions than the regular
information. Note that, this is the intuition of using of privileged information as additional features for classification
purposes and it may hold for most of the cases. Thus, the
loss of classifying human actions directly form x should be
greater or equal than classifying from both x and x∗ :

log p(yi |xi ; w) ≥ log p(yi |xi , x∗i ; w) .

(18)

We can then relax the problem and consider that Eq. (17)
is satisfied when γ1i = log p(yi |xi ; w). Thus, the regularization parameter αi for the loss function is given by:

αi (w) =

log p(yi |xi , x∗i ; w)
,
log p(yi |xi ; w) + kwk2

(19)

The regularization parameter αi may act as as the withinclassification balance between data and model parameters.
In each step of the optimization process, we adaptively update the regularization parameter αi providing robustness
to the trade-off between the regularization terms.
Similarly, the regularization parameter αi for the loss
function for the max-margin optimization is given by:

αi (w) =

ℓi (w)
,
ζi (w) + kwk2

(20)

where ζi (w) is the Hinge loss error for classifying directly
from the regular data x:

ζi (w) = max(0, 1 + ( ′max E(y ′ , h|xi ; w)
y 6=yi ,h

− max E(yi , h|xi ; w))) .

(21)

In the general case, the training samples x and x∗ may
be considered to be jointly Gaussian, thus the conditional
distribution p(x∗ |x; w) is also a Gaussian distribution. In
the case of continuous features, the continuous space of
features is quantized to a large number of discrete values
to approximate the true value of the marginalization of Eq.
(22). However, to efficiently cope with outlying measurements about the training data, we consider that the training
samples x and x∗ jointly follow a Student’s t-distribution.
Therefore, the conditional distribution p(x∗ |x; w) is also a
Student’s t-distribution St(x∗ |x; µ∗ , Σ∗ , ν ∗ ), where x∗ forms
T
the first Mx∗ components of (x∗ , x) , x comprises the
∗
remaining M − Mx∗ components, µ is the mean vector,
Σ∗ is the covariance matrix and ν ∗ ∈ [0, ∞) corresponds
to the degrees of freedom of the distribution [54]. Note that
by letting the degrees of freedom ν ∗ go to infinity, we can
recover the Gaussian distribution with the same parameters. If the data contain outliers, the degrees of freedom
parameter ν ∗ is weak and the mean and covariance of the
data are appropriately weighted in order not to take into
account the outliers. More details on how the parameters
of the conditional Student’s t-distribution p(x∗ |x; w) are
estimated can be found in Appendix A.
Although both distributions p(y, h|x, x∗ ; w) and
p(x∗ |x; w) belong to the exponential family, the graph in
Fig. 2 is cyclic, and therefore an exact solution to Eq. (22) is
generally intractable. For this reason, approximate inference
is employed for estimation of the marginal probability by
applying the LBP algorithm [49].

4

E XPERIMENTAL R ESULTS

We evaluated our method on four challenging publicly
available datasets. Three different types of privileged information were used: audio signal, human pose, and semantic
attribute annotation.
We propose four variants of our approach, called Maximum Likelihood LUPI Hidden Conditional Random Field (mlHCRF+), Adaptive Maximum Likelihood LUPI Hidden Conditional Random Field (aml-HCRF+), Maximum Margin LUPI
Hidden Conditional Random Field (mm-HCRF+), and Adaptive Maximum Margin LUPI Hidden Conditional Random Field
(amm-HCRF+), depending on which learning method we apply (i.e., maximum likelihood or max-margin) and whether
we automatically estimate the regularization parameters of
the corresponding loss function or not.

h

4.1 Datasets

3.5 Inference
Having computed the optimal parameters w∗ in the training
step, our goal is to estimate the optimal label configuration
over the testing input, where the optimality is expressed
in terms of a cost function. To this end, we maximize
the posterior probability and marginalize over the latent
variables h and the privileged information x∗ :

y = arg max p(y|x; w)
y
XX
= arg max
p(y, h, x∗ |x; w)
y

= arg max
y

h

x∗

XX
h

x∗

p(y, h|x, x∗ ; w)p(x∗ |x; w) .

(22)

Parliament [55]: This dataset is a collection of 228 video
sequences, depicting political speeches in the Greek parliament, at a resolution of 320 × 240 pixels at 25 fps. The
video sequences were manually labeled with one of three
behavioral labels: friendly, aggressive, or neutral.
TV human interaction (TVHI) [18]: This dataset consists
of 300 video sequences collected from over 20 different TV
shows. The video clips contain four kinds of interactions:
hand shakes, high fives, hugs, and kisses, equally split into
50 video sequences each, while the remaining 100 video
clips do not contain any of the aforementioned interactions.
SBU Kinect Interaction (SBU) [15]: This dataset contains
approximately 300 video sequences depicting two-person

6

interactions captured by a Microsoft Kinect sensor. The
dataset contains eight different classes including approaching, departing, pushing, kicking, punching, exchanging objects, hugging, and shaking hands, which are performed by
seven different persons. It also contains three-dimensional
coordinates of 15 joints for each person at each frame.
Unstructured social activity attribute (USAA) [26]: The
USAA dataset includes eight different semantic class videos
of social occasions such as birthday party, graduation party,
music performance, non-music performance, parade, wedding ceremony, wedding dance, and wedding reception. It
contains around 100 videos per class for training and testing. Each video is annotated with 69 attributes, which can
be divided into five broad classes: actions, objects, scenes,
sounds, and camera movement.
4.2 Implementation Details
Feature selection: For the evaluation of our method, we
used spatio-temporal interest points (STIP) [56] as our base
video representation. First, we extracted local space-time
features at a rate of 25 fps using a 72-dimensional vector
of HoG and 90-dimensional vector of HoF feature descriptors [57] for each STIP, which captures the human motion
between frames. These features were selected because they
can capture salient visual motion patterns in an efficient
and compact way. In addition, for the TVHI dataset, we
also used the provided annotations, which are related to
the locations of the persons in each video clip, including
the head orientations of each subject in the clips, the pair of
subjects who interact with each other, and the corresponding labels. For our experiments on Parliament and TVHI
datasets, we used audio features as privileged information.
More, specifically, we employed the mel-frequency cepstral
coefficients (MFCC) [58] features and their first and second
order derivatives. The audio signal was sampled at 16
KHz and processed over 10 ms using a Hamming window
with 25% overlap. The audio feature vector consisted of a
collection of 13 MFCC coefficients along with the first and
second derivatives forming a 39 dimensional audio feature
vector.
Furthermore, for the SBU dataset, we used the poses provided by the dataset as privileged information. In particular,
along with the positions of the locations of the joints for
each person in each frame, we used six more features such
as joint distance, joint motion, plane, normal plane, velocity,
and normal velocity as described by Yun et al. [15]. As a basic
representation of the video data, we used the STIP features.
Finally, we used the USAA dataset and the provided attribute annotation as privileged information to characterize
each class not with an individual label, but with a feature
vector of semantic attributes. As a representation of the
video data, we used the provided low-level features, which
correspond to SIFT [59], STIP, and MFCC features. Table 1
summarizes all forms of features used either as regular or
privileged for each dataset in our algorithm during training
and testing.
Model selection: The proposed model was trained by
varying the number of hidden states from 3 to 20, with
a maximum of 400 iterations for the termination of the
LBFGS optimization method. The L2 regularization scale

TABLE 1: Types of features used for human activity recognition for each dataset. The numbers in parentheses indicate
the dimension of the features. The checkmark corresponds
to the usage of the specific information as regular or privileged. Privileged features are used only during training.
Dataset

Features (dimension)

Regular

Parliament [55]

STIP (162)
MFCCs (39)

✓

TVHI [18]

STIP (162)
Head orientations (2)
MFCC (39)

✓
✓

SBU [15]

STIP (162)
Pose (15)

✓

USAA [26]

STIP (162)
SIFT (128)
MFCC (39)
Semantic attributes (69)

✓
✓
✓

Privileged
✓

✓
✓

✓

term σ for the non-adaptive methods was set to 10k , with
k ∈ {−3, . . . , 3}. The evaluation of our method was performed using 5-fold cross validation to split the datasets
into training and test sets, and the average results over all
the examined configurations are reported.

4.3 Multimodal Feature Fusion
One drawback of combining features of different modalities
is the different probablility distribution that each modality
may have. Thus, instead of directly combining multimodal
features together one may employ canonical correlation
analysis (CCA) [60] to exploit the correlation between the
different modalities by projecting them onto a common
subspace so that the correlation between the input vectors is
maximized in the projected space. In this paper, we followed
a different approach. Our model is able to learn the relationship between the input data and the privileged features.
To this end, we jointly calibrate the different modalities by
learning a multiple output linear regression model [61]. Let
x ∈ RM×d be the input raw data and x∗ ∈ RM×p be
the set of privileged features. Our goal is to find a set of
weights γ ∈ Rd×p , which relates the privileged features
to the regular features by minimizing a distance function
across the input samples and their attributes:

arg min kxγ − x∗ k2 + ηkγk2 ,

(23)

γ

where kγk2 is a regularization term and η controls the
degree of the regularization, which was chosen to give the
best solution by using cross validation with η ∈ [10−4 , 1].
Following a constrained least squares (CLS) optimization
problem and minimizing kγk2 subject to xγ = x∗ , Eq. (23)
−1 T ∗
has a closed form solution γ = xT x + ηI
x x , where
I is the identity matrix. Note that the minimization of Eq.
(23) is fast since it needs to be solved only once during
training. Finally, we obtain the prediction f of the privileged
features by multiplying the regular features with the learned
weights f = x · γ . The main steps of the proposed method
are summarized in Algorithm 1.

7

Algorithm 1: Robust privileged probabilistic learning

aggressive

∗

1

neutral

0.00

0.00

100.00

2.08

kiss 12.50

0.00

0.00

87.50

ss
ki

g
hu

ve

ke
ha

0.00

0.00

0.00

0.00 100.00 0.00

0.00

0.00

0.00

0.00

music 4.95

0.00

0.00

0.00

non music 3.96

0.00

0.00 66.67 33.33 0.00

0.00

exchange 0.00

0.00

0.00

0.00

0.00

0.00 100.00 0.00

punch 0.00

0.00

0.00

0.00

0.00

0.00

0.00 100.00
c
ex

3.03

4.95

74.26

7.92

0.00

0.00

5.94

1.98

3.96

0.99

60.40

1.98

2.97

14.85 10.89

parade 7.69
ceremony 32.22

10.99

0.00

0.00

70.33

0.00

1.10

9.89

8.70

6.52

4.35

15.22 11.26

9.78

11.96

dance 0.00
reception 11.54

0.00

0.00

5.21

0.00

0.00

87.50

7.29

3.85

1.92

28.46

3.85

7.69

19.23 23.46

ic
us

n
io

m

y
on

3.03

4.04

pt
ce
re

0.00 66.67 33.33 0.00

0.00

2.02

6.06

e
nc
da

0.00

0.00

0.00

3.03

n
no

0.00

hug 0.00

0.00

4.04

ic
us
m
n
tio
ua
ad
gr

shake hands 0.00

8.08

3.03

5.05

graduation 17.17 59.60

y
da
rth
bi

0.00 100.00 0.00

e
ng
ha

0.00

h
nc
pu

push 0.00

3.03

m
re
ce

83.33 0.00 16.67 0.00

birthday 78.79

de
ra
pa

kick 0.00

Fi
gh
hi

S
nd
ha

depart 0.00

(b)

s
nd
ha

Recognition accuracy (%)

Recognition accuracy (%)

72.92

0.00

0.00

g
hu

60
50
40

ml-HCRF+
aml-HCRF+
mm-HCRF+
amm-HCRF+
HCRF (visual+audio)
HCRF (visual)
HCRF (audio)

30
20
10
2

4

6

8

10

12

(d)

Number of hidden states

80

Fig. 4: Confusion matrices of the proposed ml-HCRF+ approach for (a) the Parliament [55], (b) the TVHI [18], (c) the
SBU [15], and (d) the USAA [26] datasets.

70
60
50
40
30

ml-HCRF+
aml-HCRF+
mm-HCRF+
amm-HCRF+
HCRF (visual+audio)
HCRF (visual)
HCRF (audio)

20
10
0
2

14

4

6

8

10

12

Number of hidden states

(a)

14

(b)
60

Recognition accuracy (%)

90

Recognition accuracy (%)

6.25

l
ra
ut
ne

70

80
70
60
50
40

0
6

0.00

hug 18.75

0.00

ck
ki

80

10

0.00

0.00

0.00

rt
pa
de
ch
oa
pr
ap

90

20

0.00

81.25

0.00

0.00

(c)

30

2.08

highFive 18.75

90

100

0

handShake 97.82

(a)
approach 100.00 0.00

e
ak
sh

7

0.00

sh
pu

6

100.00

y

5

0.00

dl
en
fri

3
4

7.14

friendly

ve
si
es
gr
ag

2

Input : Training sets X , and X , training labels Y
Perform feature extraction from both X and X ∗
Employ Eq. (23) and project X and X ∗ onto a
common space
Initialize parameters w randomly
for i ∈ {1, . . . , N } do
/*Maximum likelihood or max-margin learning*/
Estimate the regularization parameter αi using
Eqs. (19) or (20)
w∗ ← Train HCRF+ on triplets (Xi , Xi∗ , Yi )
end
Output: Estimated models’ parameters w∗

0.00

92.86

ml-HCRF+
aml-HCRF+
mm-HCRF+
amm-HCRF+
HCRF (visual+pose)
HCRF (visual)
HCRF (pose)
8

10

12

14

16

18

20

Number of hidden states

(c)

22

50

40

30

20

10

0
6

ml-HCRF+
aml-HCRF+
mm-HCRF+
amm-HCRF+
HCRF (visual+attributes)
HCRF (visual)
HCRF (attributes)
8

10

12

14

16

18

20

Number of hidden states

22

(d)

Fig. 3: Comparison of the recognition accuracy of the four
different variants of the proposed method and standard
HCRF model with respect to the number of hidden states
for (a) the Parliament [55], (b) the TVHI [18], (c) the SBU
[15], and (d) the USAA [26] datasets. The text in parentheses
in the legend of each figure corresponds to the type of
information used both for training and testing.

4.4 Evaluation of Privileged Information
The classification accuracy with respect to the number of
hidden states is depicted in Fig. 3. We may observe that
all four variants have a similar behavior as the number
of hidden states increases. It is clear that when privileged
information is used, in the vast majority of the cases (38
out of 45 cases) all variants of HCRF+ perform better than
the standard HCRF model. In Fig. 3, the HCRF+ variants
and the standard HCRF model suffer from large fluctuations as the number of hidden states increases. This is
because the number of hidden states plays a crucial role
in the recognition process. Many hidden states may lead
to model overfitting, while few hidden states may cause
underfitting. This would be resolved by the estimation of
the optimal number of hidden states during learning, but
this is not straightforward for this model. We may also
observe that the performance of each modality alone is kept
significantly lower for all configurations of hidden states,

which reinforces the fact that privileged information may
help to construct better classification models.
The resulting confusion matrices of the best performing
variant are depicted in Fig. 4. It is worth mentioning that for
both the Parliament and the TVHI datasets the classification
errors between different classes are relatively small. The
SBU dataset has relatively small classification errors, as only
a few classes are confused with each other (e.g., the class
hugging versus the class hand shaking), while four out of
the eight classes were perfectly recognized. It is interesting
to observe that for the USAA dataset the different classes
may be strongly confused. For example, the class wedding
ceremony is confused with the class graduation party and the
class wedding reception is confused with the class non-music
performance. This is because the different classes may share
the same attribute representation as different videos may
have been captured under similar conditions.
The behavior of the proposed adaptive model as a
function of the regularization parameters and the number
of hidden states is depicted in Fig. 5. To be consistent to
the non-adaptive methods, the real-valued regularization
parameters were quantized from the continuous to the
discrete space with α(w) = 10k , k ∈ {−2, . . . , 2} and the
results were averaged. We may observe that the behavior of
the recognition accuracy is smooth for the different values
of α(w) and the number of hidden states, which indicates
that the automatic estimation of α(w) is robust.
4.5 Comparisons using Hand-Crafted Features
In this section, we compare the results of our method with
several state-of-the-art methods. In particular, to show the
benefit of using robust privileged information, we compared
our method both with state-of-the-art methods with and
without incorporating the LUPI paradigm. Also, to demonstrate the efficacy of the robust privileged information to
the problem of human activity recognition, we compared
it with ordinary SVM and HCRF, as if they could access
both the regular and the privileged information at test time.
This means that we do not differentiate between regular and

0.1
1

α(

10
100

Accuracy (%)

w)

3

4

5

7

6

9

8

12

0.01
0.1

ates

n st

dde

# hi

50

0.01
0.1
1

α(

w)

10
100

3

4

5

7

6

9

8

10

11

12

100

4

5

6

7

8

s

12

0
0.01
0.1

ates

n st

dde

# hi

α(

1

w)

10
100

8

9

10

11

12

13

14

15

16

17

18

19

20

40
20
0
0.01
0.1

ates

n st

dde

# hi

60

α(

1

w)

10
100

8

9

10

11

12

13

14

16

15

17

18

19

20

ates

n st

dde

# hi

100

50

0
0.01

tate

en s

dd
# hi

10

w)

100

0

1

α(

9

10

11

50

Accuracy (%)

11

0

100

0.1
1

α(

10

w)

100

(a)

4

5

6

7

8

11

9

10

12

ates

n st

dde
# hi

(b)

100

50

0
0.01
0.1

α(

w)

1

10
100

8

9

10

11

(c)

12

13

14

16

15

18

ates

n st

dde

# hi

17

19

20

Accuracy (%)

0.01

10

50

Accuracy (%)

0

100

Accuracy (%)

50

Accuracy (%)

100

Accuracy (%)

Accuracy (%)

8

60
40
20
0
0.01
0.1

α(

w)

1

10
100

8

9

10

11

12

13

14

16

15

18

19

20

ates

n st

dde

# hi

17

(d)

Fig. 5: Recognition performance of the proposed maximum likelihood (top-row) and max-margin (bottom row) variants as
function of the regularization parameter and the number of hidden states for (a) the Parliament [55], (b) the TVHI [18], (c)
the SBU [15] and (d) the USAA [26] datasets.
TABLE 2: Comparison of the classification accuracies (%)
on the Parliament dataset [55]. Results highlighted with
light purple indicate statistically significant improvement
over the second best method using paired t-test.
Method

Overall

Aggressive

Friendly

Neutral

92.7
50.0
85.7
67.9
100.0
76.9

100.0
66.7
55.6
60.0
60.7
69.8

100.0
84.6
76.9
71.1
95.8
71.1

77.5
77.9
90.7
57.1
92.9
85.7
92.6
85.7

68.9
39.2
100.0
58.1
100.0
80.0
100.0
61.1

88.7
60.5
100.0
57.8
100.0
84.6
97.4
100.0

Methods without privileged information
HCRF (visual+audio) [7]
HCRF (visual) [7]
HCRF (audio) [7]
Wang and Schmid [11]
Vrigkas et al. [55]
SVM [62]

97.6 ± 0.6
67.1 ± 1.4
72.7 ± 1.8
66.6 ± 0.5
85.5 ± 1.7
72.6 ± 0.4

Methods with privileged information
SVM+ [3]
Wang and Ji [40]
Wang et al. [41]
Sharmanska et al. [43]
ml-HCRF+
aml-HCRF+
mm-HCRF+
amm-HCRF+

78.4 ± 0.2
59.2 ± 0.2
96.9 ± 1.1
57.7 ± 0.4
97.6 ± 0.7
83.5 ± 1.3
96.5 ± 0.9
82.3 ± 1.3

privileged information, but use both forms of information as
regular to infer the underlying class label instead. Moreover,
to complete the study, we also trained an HCRF model that
uses only the regular and only the privileged information
for training and testing. To distinguish between the different
types of information that the HCRF model may use, we
specifically report the type of feature in parentheses after
the HCRF caption. Furthermore, for the SVM+ and SVM
we consider a one-versus-one decomposition of multi-class
classification scheme and average the results for every possible configuration. Finally, the optimal parameters for the
SVM and SVM+ were selected using cross validation.
A comparison of the proposed approach with state-ofthe-art methods on the Parliament dataset are depicted in

Table 2. proposed ml-HCRF+ method has highest recognition accuracy (97.6%) among the other variants of the
proposed model, while it achieves the same accuracy with
the standard HCRF model. Although the adaptive HCRF+
approaches may perform worse than the non-adaptive variants, they can still achieve better results than the majority of
the state-of-the-art methods. One reason for this, is that the
estimation of the regularization parameters for the adaptive
variants depends on the input features. Features that belong
to the background may influence the estimation of the
regularization parameters as they may serve as background
noise.
It is also worth mentioning that our method is able
to increase the recognition accuracy by nearly 38% with
respect to the methods of Wang and Ji [40] and the method
of Sharmanska et al. [43], which also incorporate the LUPI
paradigm. This significantly high increase in recognition
accuracy indicates the strength of the proposed method.
Moreover, the performance of the proposed approach is
higher approximately by 19% than the SVM+ model and
25% than the standard SVM approach. The Parliament
dataset contains large intra-class variabilities. For example,
the interaction between an arm lift and the raise in the voice
may not exclusively be combined together as some features
may act as outliers and affect the classification accuracy.
The classification results on the TVHI dataset are demonstrated in Table 3. For this dataset, we significantly managed
to increase the classification accuracy by approximately
10%, with respect to the LUPI-based SVM+ and Wang and Ji
[40] approaches, as our approach achieves very high recognition accuracy (84.9%). The improvement of our method
compared to the method of Sharmanska et al. [43] and the
methods that do not use privileged information was even
higher.
The classification accuracies for the SBU dataset are
presented in Table 4. The ml-HCRF+ approach achieved
the highest accuracy (85.4%), where the improvement over

9

TABLE 3: Comparison of the classification accuracies
(%) on TVHI the dataset [18]. Results highlighted with
light purple indicate statistically significant improvement
using paired t-test.
Method

Overall

Hand Shake

High Five

Hug

Kiss

87.5
56.3
12.5
76.2
74.6

56.3
25.0
12.5
74.6
76.3

87.5
87.5
43.8
74.8
75.8

93.8
75.0
75.0
74.6
76.3

74.6
74.6
93.8
78.3
97.2
93.8
93.8
93.8

76.3
76.3
81.2
54.8
81.3
81.3
81.3
81.3

72.8
72.2
75.1
74.3
72.9
71.8
72.5
68.8

76.2
76.3
87.5
53.5
87.5
87.5
87.5
87.5

Methods without privileged information
HCRF (visual+audio) [7]
HCRF (visual) [7]
HCRF (audio) [7]
Wang and Schmid [11]
SVM [62]

81.3 ± 0.7
60.9 ± 1.3
35.9 ± 1.5
76.1 ± 0.4
75.9 ± 0.6

Methods with privileged information
SVM+ [3]
Wang and Ji [40]
Wang et al. [41]
Sharmanska et al. [43]
ml-HCRF+
aml-HCRF+
mm-HCRF+
amm-HCRF+

75.0 ± 0.2
74.8 ± 0.2
84.4 ± 1.1
65.2 ± 0.1
84.9 ± 0.8
83.6 ± 1.1
83.6 ± 0.6
82.9 ± 0.8

the standard HCRF model is nearly 4%. Comparing our
method to methods that do not use privileged information,
we increased the classification accuracy in all cases. An
interesting observation of the non-privileged HCRF (visual)
and HCRF (pose) methods arises. Despite the fact that for
some classes these methods were able to perfectly recognize
the underlying activity, they completely failed to recognize
some of the classes as the rate of false positives may reach
100%. Considerably high improvements are also reported
when comparing our methods with state-of-the-art methods
that employ privileged information.
The classification results for the USAA dataset are summarized in Table 5. The combination of both raw data and
attribute representation of human activities significantly
outperformed the SVM+ baseline and the method of Wang
and Ji [40] by increasing the classification accuracy by approximately 11% for the amm-HCFR+ model. An improvement of 3% with respect to the methods of Sharmanska et al.
[43] and Wang et al. [41] was also achieved. Furthermore, the
adaptive variants of the proposed method perform better
than their non-adaptive counterparts for this dataset. Automatic estimation of the regularization parameters provides
more flexibility to the model as it allows the model to adjust
its behavior according to the training data.
4.6 Comparisons using Deep Learning Features
In our experiments, we used CNNs for both end-to-end
classification and feature extraction. We employed the pretrained model of Tran et al. [32], which is a 3D ConvNet (Fig.
6). We selected this model because it was trained on a very
large dataset (Sports 1M [63]), which provides good features
for the activity recognition task, especially in our case where
the size of the training data is small, making deep learning
models prone to overfitting.
Because both the Parliament and SBU datasets, are fairly
small datasets, only a few parameters had to be trained
to avoid overfitting. Particularly, we replaced the fullyconnected layers of the pre-trained model with a new fullyconnected layer of size 1, 024 and trained the additional
layer coupled with a softmax layer on top of it. For the TVHI

dataset, we fine-tuned the last group of convolutional layers,
while for the USAA dataset, we fine-tuned the last two
groups. Each group has two convolutional layers, while we
added a new fully-connected layer of size 256 for the TVHI
and 1, 024 for the USAA datasets, respectively. For the optimization process, we used mini-batch stochastic gradient
descent (SGD) with momentum. The size of the mini-batch
was set to 16 and we used a constant momentum of 0.9.
For both the Parliament and SBU datasets, the learning rate
was initialized to 0.01 and it was decayed by a factor of 0.1,
while the total number of training epochs was 1, 000. For the
TVHI and USAA datasets, we used a constant learning rate
of 10−4 and the total number of training epochs was 500 and
250, respectively. For all datasets, we added a dropout layer
after the new fully-connected layer with probability 0.5.
Also, we performed data augmentation on each batch online
and 16 consecutive frames were randomly selected for each
video. These frames were randomly cropped, resulting in
frames of size 112 × 112 and then flipped with probability 0.5. For the classification task, we used the centered
112 × 112 crop on the frames of each video sequence. Then,
for each video, we extracted 10 random clips of 16 frames
and averaged their predictions. Finally, to avoid overfitting,
we used early stopping and extracted CNN features from
the newly added fully-connected layer.
In addition, we compared the proposed HCRF+ method
with the LSTM networks [30], since it has been proven
that they provide good performance in several sequential
classification tasks such as image description and activity
recognition [29]. Although a promising methodology is to
train a CNN stacked with an LSTM layer on top [29] for
end-to-end feature extraction and sequential classification,
our limited size datasets prevented us on training such a
model due to overfitting. To address this issue, we trained
an LSTM layer with a softmax layer on top, on the features
extracted from the pretrained CNN model. Specifically, we
added a dropout layer on the LSTM’s hidden units and an
L2 regularization on the softmax units. For estimating the
hyperparameters, we performed a grid search with 5-fold
cross validation to optimize the learning rate, the number of
hidden units, the dropout rate and the weight decay factor
of the L2 regularizer. We trained the LSTM model for 100
epochs using the Adam optimizer [64] with early stopping.
The comparison of the proposed approach with stateof-the-art methods using the CNN features is summarized
in Table 6. The improvement of accuracy with respect to
the hand-crafted feature classification for all datasets (Tables 2 to 5), indicates that CNNs may efficiently extract
informative features without any need to hand-design them.
It is also worth noting that privileged information works
in favor of the classification task in all cases. The mlHCRF+ variant achieves the highest results among all other
methods for the Parliament, TVHI, and SBU datasets, while
for the USAA dataset, the amm-HCRF+ variant achieves
the highest recognition accuracy (96.4%). Moreover, the improvement in accuracy of the proposed model with respect
to the end-to-end CNN classification for the Parliament,
TVHI, and USAA datasets, was approximately 15%, 33%,
and 29%, respectively. This improvement can be explained
by the fact that the CNN model uses a linear classifier in
the softmax layer, while the proposed approach is a more

10

TABLE 4: Comparison of the classification accuracies (%) on the SBU dataset [15]. Results highlighted with light purple
indicate statistically significant improvement using paired t-test.
Method

Overall

Approach

Depart

Kick

Push

Shake Hands

Hug

Exchange Objects

Punch

100.0
100.0
100.0
76.2
74.9

33.3
100.0
0.0
74.6
67.2

100.0
100.0
100.0
78.6
68.7

66.7
66.7
100.0
78.9
76.9

66.7
100.0
0.0
81.4
100.0

75.0
0.0
0.0
79.2
59.4

100.0
100.0
100.0
84.3
89.4

83.3
0.0
100.0
83.5
100.0

76.4
79.5
100.0
51.6
100.0
100.0
100.0
100.0

72.6
61.4
66.7
79.2
83.3
100.0
75.0
66.7

73.2
59.2
75.0
40.9
100.0
75.0
100.0
83.4

91.5
60.0
66.7
60.0
100.0
77.8
100.0
66.7

70.2
59.7
66.7
74.1
66.7
100.0
66.7
66.7

73.2
60.5
75.5
39.9
33.3
50.0
25.0
75.0

81.4
56.4
100.0
43.6
100.0
66.7
100.0
100.0

100.0
62.6
100.0
61.2
100.0
66.7
100.0
100.0

Methods without privileged information
HCRF (visual+pose) [7]
HCRF (visual) [7]
HCRF (pose) [7]
Wang and Schmid [11]
SVM [62]

81.4 ± 0.8
69.8 ± 1.1
62.5 ± 1.3
79.6 ± 0.4
79.4 ± 0.4

Methods with privileged information
SVM+ [3]
Wang and Ji [40]
Wang et al. [41]
Sharmanska et al. [43]
ml-HCRF+
aml-HCRF+
mm-HCRF+
amm-HCRF+

79.4 ± 0.3
62.4 ± 0.3
83.7 ± 1.6
56.3 ± 0.2
85.4 ± 0.4
79.8 ± 1.3
83.7 ± 0.5
82.8 ± 1.3

TABLE 5: Comparison of the classification accuracies (%) on the USAA dataset [26]. Results highlighted with light purple
indicate statistically significant improvement using paired t-test.
Method

Overall

Birthday

Graduation

Music

Non-music

Parade

Ceremony

Dance

Reception

79.8
74.8
22.2
52.8
47.5

59.6
50.5
41.4
55.3
47.9

48.5
76.4
63.7
57.1
49.4

68.3
50.5
47.5
58.3
45.7

61.5
79.1
35.2
60.2
48.7

4.4
4.3
14.1
49.7
38.2

69.8
80.2
56.3
59.6
36.5

21.2
19.2
0.0
40.1
45.9

52.7
32.9
58.6
56.9
78.8
78.8
79.8
78.8

49.9
44.6
68.7
47.8
59.6
57.6
63.6
61.6

53.3
52.7
58.4
62.0
74.3
78.2
79.2
77.2

50.9
48.9
67.3
62.6
60.4
70.3
59.4
69.3

51.6
52.0
74.7
67.1
70.3
67.0
54.9
69.2

48.7
49.4
17.4
51.8
11.3
3.3
14.6
18.3

41.1
54.7
75.0
57.5
87.5
78.1
85.4
79.2

32.5
53.0
15.4
44.4
23.5
23.1
17.5
21.2

Methods without privileged information
HCRF (visual+attributes) [7]
HCRF (visual) [7]
HCRF (attributes) [7]
Wang and Schmid [11]
SVM [62]

54.0 ± 0.8
55.5 ± 0.9
37.4 ± 1.0
55.6 ± 0.1
47.4 ± 0.1

Methods with privileged information
SVM+ [3]
Wang and Ji [40]
Wang et al. [41]
Sharmanska et al. [43]
ml-HCRF+
aml-HCRF+
mm-HCRF+
amm-HCRF+

48.5 ± 0.1
48.5 ± 0.2
55.3 ± 0.9
56.3 ± 0.2
58.1 ± 1.4
57.5 ± 1.4
56.8 ± 0.6
59.4 ± 0.7

sophisticated model that can efficiently handle sequential
data in a more principled way. Also, the improvement in
performance, brought by the LSTM compared to the endto-end classification with the CNN, validates the ability
of LSTMs to capture long-term dependencies in human
activities as LSTMs have memory of previous activity states
and can better model their complex dynamics. Nonetheless,
the proposed model outperforms the LSTM, for all datasets,
a fact that supports our main hypothesis that the LUPI
paradigm may be beneficial for human activity recognition.

The corresponding confusion matrices of the proposed
method for all datasets, using the CNN-based features, are
depicted in Fig. 7. The combination of privileged information with the feature representation learned from the
CNN model resulted in very small inter- and intra-class
classification errors for all datasets.

5

D ISCUSSION

Our method is able to robustly use privileged information
in a more efficient way than the SVM+ and the other LUPIbased methods, by exploiting the hidden dynamics between
the video clips and the privileged information. We may also
observe that the proposed method outperforms all methods
that do not incorporate privileged information during learning. Since the combination of multimodal data falls natural
to the human perception of understanding complex activities, the incorporation of such information constitutes a
strong attribute for discriminating between different classes,
rather than learning each modality separately.
Statistical significance: In order to provide a statistical
evidence of the recognition accuracy, we computed the pvalues of the obtained results with respect to the compared
methods. Results highlighted with light purple in Tables 2
- 6 indicate statistically significant improvement (p-values

11

fc-487

fc-4096

fc-4096

max-pooling
(2 × 2 × 2)

conv
(512@3 × 3 × 3)

conv
(512@3 × 3 × 3)

max-pooling
(2 × 2 × 2)

conv
(512@3 × 3)

conv
(512@3 × 3 × 3)

max-pooling
(2 × 2 × 2)

conv
(256@3 × 3 ×3)

conv
(256@3 × 3)

max-pooling
(2 × 2 × 2)

conv
(128@3 × 3 × 3)

max-pooling
(1 × 2 × 2)

conv
(64@3 × 3 ×3)

Input video
segment
(16 × 112 × 112)

Fig. 6: Illustration of the 3D ConvNet architecture [32]. The model has 8 convolutional layers, 5 max-pooling layers and
two fully-connected layers followed by the output layer, which is a softmax classifier that classifies videos in 487 categories
[63]. Different colors are used for different types of layers for better understanding. The net takes as input a volume of
16 frames with a height and a width of size 112. Convolutional layers perform 3D convolutions using 3D kernels and
max-pooling layers perform 3D pooling using 3D receptive fields. In convolutional layers, the number of feature maps is
denoted before symbol “@” and follows the size of the kernels. In pooling layers, the size of each receptive field is shown.
In fully-connected layers, it is denoted the number of hidden units.

6.10

friendly

10.00

10.00

0.00

0.00

100.00

0.00

0.00

hug

0.00

0.00

92.86

7.14

kiss

0.00

0.00

0.00

100.00

5.02

88.88
0.00

0.00

neutral

80.00

highFive

handShake

8.88

0.00

91.22

aggressive

100.00

0.00 92.50 7.50

0.00

0.00

0.00

0.00

11.11 0.00 88.89 0.00

0.00

0.00

0.00

0.00

exchange 0.00

0.00

0.00

0.00

0.00

0.00 100.00 0.00

punch 0.00

0.00

0.00

0.00

0.00

0.00

0.00 100.00
ex

0.00

94.44

5.56

0.00

0.00

0.00

0.00

non music 0.00

2.78

2.78

83.33

5.56

0.00

5.56

0.00

parade 2.94

2.94

0.00

0.00

94.12

0.00

0.00

0.00

ceremony 3.13

0.00

0.00

0.00

0.00

96.88

0.00

0.00

dance 0.00

0.00

0.00

0.00

0.00

0.00

97.14

2.86

reception 5.26

0.00

0.00

0.00

0.00

0.00

0.00

94.74

n

io

y

on

pt

e

m

nc

ic

us

n

tio

de

m

ic

y

ua

ge

s

nd

da

ad

ce

2.78

music 0.00

re

0.00

ra

2.86

0.00

n

0.00

0.00

us

0.00

0.00

rth

h

ha

an

nc

e

ch

g

(c)

0.00

0.00

birthday 94.29

bi

0.00 100.00 0.00

pu

0.00

sh

0.00

ak

0.00

ck

Selecting which features can act as privileged information is
not an easy task. The performance of LUPI-based classifiers
relies on the delicate relationship between the regular and
the privileged information. Also, privileged information
is costly or difficult to obtain with respect to producing
additional regular training examples [42]. In general, when
privileged information alone is used as regular it may
not be sufficient for the correct classification of an action
into its respective category, since finding proper privileged
information is not always a straightforward process.
The scope of our approach is not to achieve the best
results possible but to investigate to what extent privileged

hug 0.00

ch

5.1 Why is Privileged Information Important?

0.00

rt

were less than the significance level of 0.05) over the second
best method using paired t-test.
Computational complexity: The proposed method uses
the same sufficient statistics as HCRF and the computational complexity is similar to HCRF. The complexity of our
method is determined by the complexity of the corresponding inference problem and is quadratic to the number of
hidden states.

0.00 87.50 12.50 0.00

oa

92.3 ± 0.3
92.8 ± 0.2
93.2 ± 1.2
93.5 ± 0.2
93.9 ± 0.9
95.9 ± 1.3
95.2 ± 1.0
96.4 ± 1.4

0.00

pa

94.8 ± 0.3
92.7 ± 0.4
91.1 ± 1.3
92.9 ± 0.4
94.9 ± 0.7
92.9 ± 0.4
93.6 ± 1.1
94.6 ± 1.0

0.00

pr

92.5 ± 0.4
88.8 ± 0.2
85.0 ± 1.2
90.0 ± 0.1
93.2 ± 0.6
92.5 ± 1.1
92.5 ± 0.7
90.0 ± 0.8

shake hands 0.00

0.00

97.22

re

graduation 0.00

kick 0.00

ss

0.00

da

0.00

push 0.00

ki

0.00

ce

0.00

pa

0.00

0.00

no

depart 0.00 100.00 0.00

hu

91.6 ± 0.8
91.9 ± 0.3
67.4 ± 0.6
91.3 ± 1.7

sh

91.1 ± 0.4
92.8 ± 0.2
94.2 ± 0.8
94.7 ± 0.7

2.86

m

0.00

gr

0.00

ap

90.0 ± 0.3
83.5 ± 0.4
84.4 ± 0.6
81.8 ± 0.2
93.3 ± 0.7
93.3 ± 0.4
88.9 ± 0.9
86.7 ± 1.2

0.00

pu

89.6 ± 0.5
90.0 ± 0.3
60.5 ± 1.1
88.4 ± 1.5

Methods with privileged information
SVM+ [3]
Wang and Ji [40]
Wang et al. [41]
Sharmanska et al. [43]
ml-HCRF+
aml-HCRF+
mm-HCRF+
amm-HCRF+

0.00

ki

84.4 ± 0.8
89.9 ± 0.5
78.1 ± 0.4
88.3 ± 0.8

(b)

0.00

3.20

de

HCRF [7]
SVM [62]
CNN (end-to-end) [32]
LSTM [30]

(a)
6.80

approach 90.00

g

USAA

hu

SBU

ve

TVHI

e

Parliament

Methods without privileged information

Fi

ak

ve

Method

gh

hi

l

Sh

ra

nd

ut

ha

ne

si

y

es

dl

gr

en

fri

ag

TABLE 6: Comparison of the classification accuracies (%)
on Parliamment [55], TVHI [18], SBU [15], and USAA
[26] datasets using CNN features. Results highlighted with
light purple indicate statistically significant improvement
using paired t-test.

(d)

Fig. 7: Confusion matrices of the proposed ml-HCRF+ approach for (a) the Parliament [55], (b) the TVHI [18], (c) the
SBU [15], and (d) the USAA [26] datasets using the CNN
features.

information can be beneficial under the same evaluation
protocol. The main strength of the proposed method is
that it achieves good classification results, when the LUPI
framework is incorporated with the standard HCRF model.
In the era of deep learning, significant progress has been
made in learning good representations of the data and a
deep learning based technique is the way to go. However,
in cases where datasets are small in size, which is true
in our case, and the distribution of the data is completely
different from the data that the existing pre-trained models
were trained on, then privileged information can be very
helpful. Nonethenless, one may fine-tune the deep neural
model and extract meaningfull feature representations. This
enhances our choice to use deep features with the proposed
HCRF+ model as the experimetal results indicate significant
improvement when these features are used. Thus, the answer to the question “is privileged information necessary?”
is affirmative. For example, in many medical applications,
where annotated data are difficult or expensive to obtain
and pre-trained deep learning models are still not available,
privileged information is the best solution to go.

12

6

C ONCLUSION

The mean µ∗ , the covariance matrix Σ∗ and the degrees of
freedom ν ∗ of the conditional distribution p(x∗ |x; w), are
computed by the respective parts of µ and Σ:

In this paper, we addressed the problem of human activity
categorization in a supervised framework and proposed
a novel probabilistic classification model based on robust
learning using a privileged information paradigm, called
HCRF+. Our model is made robust using Student’s tdistributions to model the conditional distribution of the
privileged information. We proposed two variants for training in the LUPI framework. The first variant uses maximum
likelihood and the second uses maximum margin learning.
Using auxiliary information about the input data, we
were able to produce better classification results than the
standard HCRF [7] approach. We evaluated the performance
of our method on four publicly available datasets and
tested various forms of privileged information. The experimental results indicated that robust privileged information
along with the regular input data for training the model
ameliorates the recognition performance. We demonstrated
improved results with respect to the state-of-the-art LUPI
framework especially when CNN features are employed.
According to our results, the proposed method and its
variants achieved notably higher performance than the majority of the compared classification schemes. We were able
to flexibly understand multimodal human activities with
high accuracy, when not the same amount of information
is available during testing. By automatically estimating the
regularization parameters during learning, we managed
to achieve high recognition accuracy with less effort than
standard cross validation based classification schemes.

The parameters (µ, Σ, ν) of the joint Student’s t-distribution
p(x∗ , x; w), which are defined by the corresponding parT
tition of the vector (x∗ , x) , are estimated using the
expectation-maximization (EM) algorithm [54]. Then, the
parameters of the conditional distribution p(x∗ |x; w) are
computed using Eq. (25)-(27).
It is worth noting that by letting the degrees of freedom
ν ∗ to go to infinity, we can recover the Gaussian distribution
with the same parameters. If the data contain outliers, the
degrees of freedom parameter ν ∗ are weak and the mean
and covariance of the data are appropriately weighted in
order not to take into account the outliers.

A PPENDIX A
C ONDITIONAL D ISTRIBUTION OF THE P RIVILEGED
I NFORMATION

[1]

Recall that x ∈ RMx ×T is an observation sequence of
length T and x∗ ∈ RMx∗ ×T corresponds to the privileged
information of the same length. We partition the original
T
set (x∗ , x) ∈ RM×T into two disjoint subsets, where x∗
T
forms the first Mx∗ components of (x∗ , x) ∈ RM×T and
x comprises the remaining M − Mx components. If the
joint distribution p(x, x∗ ; w) follows a Student’s t-law, with
T
mean vector µ = (µx∗ , µx ) , a real, positive
 definite, and

Σx ∗ x ∗ Σx ∗ x
symmetric M × M covariance matrix Σ =
Σxx
Σxx∗
and ν ∈ [0, ∞) corresponds to the degrees of freedom
of the distribution [54], then the conditional distribution
p(x|x∗ ; w) is also a Student’s t-distribution:

p(x∗ |x; w) = St(x∗ ; µ∗ , Σ∗ , ν ∗ )
=

Γ ((ν ∗ + M ) /2) |Σxx |1/2
M /2

(πν ∗ ) x Γ ((ν ∗ + Mx ) /2) |Σ∗ |1/2
x)

 (ν ∗ +M
2
1 + ν1∗ xT Σ−1
x
xx
×h
.
)
i (ν ∗ +M
2
1
1 + ν ∗ Z T Σ∗ −1 Z

(24)

−1
(x − µx ) ,
µ∗ = µx∗ − Σx∗ x Σxx
T

(x − µx )
νx∗ + (x − µx )
νx ∗ + M x ∗

× Σx∗ x∗ − Σx∗ x Σ−1
xx Σxx∗ ,
ν ∗ = νx ∗ + M x ∗ .

Σ∗ =

(25)

Σ−1
xx

(26)
(27)

ACKNOWLEDGMENTS
This work has been funded in part by the UH Hugh
Roy and Lillie Cranz Cullen Endowment Fund and the
European Commission (H2020-MSCA-IF-2014), under grant
agreement No 656094. All statements of fact, opinion or
conclusions contained herein are those of the authors and
should not be construed as representing the official views
or policies of the sponsors.

R EFERENCES
I. Cohen, F. G. Cozman, N. Sebe, M. C. Cirelo, and T. S. Huang,
“Semisupervised learning of classifiers: theory, algorithms, and
their application to human-computer interaction,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, no. 12,
pp. 1553–1566, December 2004.
[2] A. W. M. Smeulders, D. M. Chu, R. Cucchiara, S. Calderara,
A. Dehghan, and M. Shah, “Visual tracking: an experimental survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence,
vol. 36, no. 7, pp. 1–1, 2014.
[3] V. Vapnik and A. Vashist, “A new learning paradigm: Learning
using privileged information,” Neural Networks, vol. 22, no. 5–6,
pp. 544–557, 2009.
[4] D. Lopez-Paz, L. Bottou, B. Schölkopf, and V. Vapnik, “Unifying
distillation and privileged information,” in Proc. 5th International
Conference on Learning Representations, San Juan, Puerto Rico, May
2-4 2016.
[5] J. Chen, X. Liu, and S. Lyu, “Boosting with side information,” in
Proc. Asian Conference on Computer Vision, ser. Lecture Notes in
Computer Science, vol. 7724, Daejeon, Korea, 2012.
[6] J. Feyereisl and U. Aickelin, “Privileged information for data
clustering,” Information Sciences, vol. 194, no. 0, pp. 4–23, 2012.
[7] A. Quattoni, S. Wang, L. P. Morency, M. Collins, and T. Darrell,
“Hidden conditional random fields,” IEEE Transactions on Pattern
Analysis and Machine Intelligence, vol. 29, no. 10, pp. 1848–1852,
2007.
[8] M. Vrigkas, E. Kazakos, C. Nikou, and I. A. Kakadiaris, “Inferring
human activities using robust privileged probabilistic learning,”
arXiv preprint arXiv:1708.09825, 2017.
[9] D. Peel and G. J. Mclachlan, “Robust mixture modelling using the
t distribution,” Statistics and Computing, vol. 10, pp. 339–348, 2000.
[10] V. Ramanathan, B. Yao, and L. Fei-Fei, “Social role discovery
in human events,” in Proc. IEEE Computer Society Conference on
Computer Vision and Pattern Recognition, Portland, OR, June 2013.

13

[11] H. Wang and C. Schmid, “Action recognition with improved
trajectories,” in Proc. IEEE International Conference on Computer
Vision, Sydney, Australia, December 2013.
[12] Y. Song, L. P. Morency, and R. Davis, “Multi-view latent variable
discriminative models for action recognition,” in Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition,
Providence, Rhode Island, Jun. 2012.
[13] M. Vrigkas, C. Nikou, and I. A. Kakadiaris, “Identifying human
behaviors using synchronized audio-visual cues,” IEEE Transactions on Affective Computing, vol. 8, no. 1, pp. 54–66, Jan. 2017.
[14] V. Ferrari, M. Marin-Jimenez, and A. Zisserman, “Pose search:
Retrieving people using their pose,” in Proc. IEEE Computer Society
Conference on Computer Vision and Pattern Recognition, Miami Beach,
FL, June 2009.
[15] K. Yun, J. Honorio, D. Chattopadhyay, T. L. Berg, and D. Samaras,
“Two-person interaction detection using body-pose features and
multiple instance learning,” in Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Rhode
Island, June 2012.
[16] A. Cherian, J. Mairal, K. Alahari, and C. Schmid, “Mixing bodypart sequences for human pose estimation,” in Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition,
Columbus, OH, June 2014.
[17] A. Eweiwi, M. S. Cheema, C. Bauckhage, and J. Gall, “Efficient
pose-based action recognition,” in Proc. Asian Conference on Computer Vision, Singapore, November 2014.
[18] A. Patron-Perez, M. Marszalek, I. Reid, and A. Zisserman, “Structured learning of human interactions in TV shows,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 12, pp.
2441–2453, Dec. 2012.
[19] M. Hoai and A. Zisserman, “Talking heads: Detecting humans
and recognizing their interactions,” in Proc. IEEE Computer Society
Conference on Computer Vision and Pattern Recognition, Columbus,
OH, June 2014.
[20] V. Ramanathan, P. Liang, and L. Fei-Fei, “Video event understanding using natural language descriptions,” in Proc. IEEE International Conference on Computer Vision, Sydney, Australia, December
2013.
[21] S. Guadarrama, N. Krishnamoorthy, G. Malkarnenkar, S. Venugopalan, R. J. Mooney, T. Darrell, and K. Saenko, “Youtube2text:
Recognizing and describing arbitrary activities using semantic
hierarchies and zero-shot recognition,” in Proc. IEEE International
Conference on Computer Vision, Sydney, Australia, December 2013.
[22] J. Shao, K. Kang, C. C. Loy, and X. Wang, “Deeply learned
attributes for crowded scene understanding,” in Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition,
Boston, MA, June 2015.
[23] M. J. Marı́n-Jiménez, R. M. noz Salinas, E. Yeguas-Bolivar, and
N. P. de la Blanca, “Human interaction categorization by using
audio-visual cues,” Machine Vision and Applications, vol. 25, no. 1,
pp. 71–84, 2014.
[24] Y. Wang and G. Mori, “A discriminative latent model of object
classes and attributes,” in Proc. 11th European Conference on Computer Vision: Part V, Heraklion, Crete, Greece, 2010.
[25] J. Liu, B. Kuipers, and S. Savarese, “Recognizing human actions by
attributes,” in Proc. IEEE Computer Society Conference on Computer
Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
[26] Y. Fu, T. M. Hospedales, T. Xiang, and S. Gong, “Attribute learning
for understanding unstructured social activity,” in Proc. 12th European Conference on Computer Vision, ser. Lecture Notes in Computer
Science, vol. 7575, Florence, Italy, October 2012.
[27] Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid, “Labelembedding for attribute-based classification,” in Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition,
Portland, OR, June 2013.
[28] J. Carreira and A. Zisserman, “Quo vadis, action recognition? A
new model and the kinetics dataset,” in Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Honolulu,
Hawaii, July 2017.
[29] J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach,
S. Venugopalan, K. Saenko, and T. Darrell, “Long-term recurrent
convolutional networks for visual recognition and description,”
in Proc. IEEE Computer Society Conference on Computer Vision and
Pattern Recognition, Boston, MA, June 2015.
[30] S. Hochreiter and J. Schmidhuber, “Long short-term memory,”
Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997.

[31] L. Wang, Y. Qiao, and X. Tang, “Action recognition with trajectorypooled deep-convolutional descriptors,” in Proc. IEEE Computer
Society Conference on Computer Vision and Pattern Recognition,
Boston, MA, June 2015.
[32] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3D convolutional networks,” in
Proc. IEEE International Conference on Computer Vision, Santiago,
Chile, December 2015.
[33] H. Bilen, B. Fernando, E. Gavves, A. Vedaldi, and S. Gould,
“Dynamic image networks for action recognition,” in Proc. IEEE
Computer Society Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, June 2016.
[34] C. Feichtenhofer, A. Pinz, and A. Zisserman, “Convolutional twostream network fusion for video action recognition,” in Proc.
IEEE Computer Society Conference on Computer Vision and Pattern
Recognition, Las Vegas, NV, June 2016.
[35] W. Zhu, J. Hu, G. Sun, X. Cao, and Y. Qiao, “A key volume mining
deep framework for action recognition,” in Proc. IEEE Computer
Society Conference on Computer Vision and Pattern Recognition, Las
Vegas, NV, June 2016.
[36] D. Pechyony and V. Vapnik, “On the theory of learnining with
privileged information,” in Proc. Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada,
December 2010.
[37] L. Niu, Y. Shi, and J. Wu, “Learning using privileged information with L-1 support vector machine,” in Proc. IEEE/WIC/ACM
International Conferences on Web Intelligence and Intelligent Agent
Technology, vol. 3, Macau, China, December 2012.
[38] F. Cai and V. Cherkassky, “Generalized SMO algorithm for SVMbased multitask learning,” IEEE Transactions on Neural Networks
and Learning Systems, vol. 23, no. 6, pp. 997–1003, June 2012.
[39] S. Fouad, P. Tino, S. Raychaudhury, and P. Schneider, “Incorporating privileged information through metric learning,” IEEE
Transactions on Neural Networks and Learning Systems, vol. 24, no. 7,
pp. 1086–1098, July 2013.
[40] Z. Wang and Q. Ji, “Classifier learning with hidden information,”
in Proc. IEEE Computer Society Conference on Computer Vision and
Pattern Recognition, Boston, MA, June 2015.
[41] Z. Wang, T. Gao, and Q. Ji, “Learning with hidden information
using a max-margin latent variable model,” in Proc. International
Conference on Pattern Recognition, Stockholm, Sweden, August
2014.
[42] C. Serra-Toro, V. J. Traver, and F. Pla, “Exploring some practical issues of svm+: Is really privileged information that helps?” Pattern
Recognition Letters, vol. 42, no. 0, pp. 40–46, 2014.
[43] V. Sharmanska, N. Quadrianto, and C. H. Lampert, “Learning
to rank using privileged information,” in Proc. IEEE International
Conference on Computer Vision, Sydney, Australia, December 2013.
[44] S. Wang, M. He, Y. Zhu, S. He, Y. Liu, and Q. Ji, “Learning
with privileged information using Bayesian networks,” Frontiers
of Computer Science, vol. 9, no. 2, pp. 185–199, April 2015.
[45] M. Vrigkas, C. Nikou, and I. A. Kakadiaris, “Active privileged
learning of human activities from weakly labeled samples,”
in Proc. 23rd IEEE International Conference on Image Processing,
Phoenix, AZ, September 2016.
[46] I. Kakadiaris, N. Sarafianos, and C. Nikou, “Show me your body:
Gender classification from still images,” in Proc. IEEE International
Conference on Image Processing, Phoenix, AZ, September 25-28 2016.
[47] M. Vrigkas, C. Nikou, and I. Kakadiaris, “Exploiting privileged
information for facial expression recognition,” in Proc. IEEE International Conference on Biometrics, Halmstad, Sweden, June 13-16
2016.
[48] S. Wang, D. Tao, and J. Yang, “Relative attribute SVM+ learning
for age estimation,” IEEE Transactions on Cybernetics, vol. 46, pp.
827–839, 2015.
[49] N. Komodakis and G. Tziritas, “Image completion using efficient
belief propagation via priority scheduling and dynamic pruning,”
IEEE Transactions on Image Processing, vol. 16, no. 11, pp. 2649–2661,
November 2007.
[50] J. Nocedal and S. J. Wright, Numerical optimization, 2nd ed., ser.
Springer series in operations research and financial engineering.
New York, NY: Springer, 2006.
[51] Y. Wang and G. Mori, “Hidden part models for human action
recognition: probabilistic versus max margin,” IEEE Transactions
on Pattern Analysis and Machine Intelligence, vol. 33, no. 7, pp. 1310–
1323, 2011.

14

[52] T. Hastie, S. Rosset, R. Tibshirani, and J. Zhu, “The entire regularization path for the support vector machine,” Journal of Machine
Learning Research, vol. 5, pp. 1391–1415, December 2004.
[53] C. H. Teo, A. J. Smola, S. V. N. Vishwanathan, and Q. V. Le, “A scalable modular convex solver for regularized risk minimization,” in
Proc. ACM International Conference on Knowledge Discovery and Data
Mining, San Jose, CA, August 2007.
[54] S. Kotz and S. Nadarajah, Multivariate t distributions and their applications. Cambridge, New York, Madrid: Cambridge University
Press, 2004.
[55] M. Vrigkas, C. Nikou, and I. A. Kakadiaris, “Classifying behavioral
attributes using conditional random fields,” in Proc. 8th Hellenic
Conference on Artificial Intelligence, ser. Lecture Notes in Computer
Science, vol. 8445, Ioannina, Greece, May 2014.
[56] I. Laptev, “On space-time interest points,” International Journal of
Computer Vision, vol. 64, no. 2-3, pp. 107–123, September 2005.
[57] A. Kläser, M. Marszałek, and C. Schmid, “A spatio-temporal
descriptor based on 3D-gradients,” in Proc. British Machine Vision
Conference, University of Leeds, Leeds, UK, September 2008.
[58] L. Rabiner and B. H. Juang, Fundamentals of Speech Recognition.
Upper Saddle River, NJ: Prentice-Hall, 1993.
[59] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” International Journal on Computer Vision, vol. 60, no. 2, pp.
91–110, Nov. 2004.
[60] D. R. Hardoon, S. R. Szedmak, and J. R. Shawe-Taylor., “Canonical
correlation analysis: an overview with application to learning
methods,” Neural Computation, vol. 16, no. 12, pp. 2639–2664, Dec.
2004.
[61] M. Palatucci, D. Pomerleau, G. E. Hinton, and T. M. Mitchell,
“Zero-shot learning with semantic output codes,” in Proc. Advances in Neural Information Processing Systems, Vancouver, British
Columbia, Canada, December 2009.
[62] C. M. Bishop, Pattern Recognition and Machine Learning. Springer,
2006.
[63] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and
L. Fei-Fei, “Large-scale video classification with convolutional
neural networks,” in Proc. IEEE Computer Society Conference on
Computer Vision and Pattern Recognition, Columbus, OH, June 2014.
[64] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” CoRR, vol. abs/1412.6980, 2014.

