Simultaneous Learning from Human Pose and Object Cues for Real-Time Activity Recognition
Brian Reily Qingzhao Zhu Christopher Reardon Hao Zhang
cs.CV cs.RO


definitionDefinition
theoremTheorem
lemmaLemma
propositionProposition
propertyProperty
observationObservation
corollaryCorollary


Edge Intelligence: Architectures, Challenges, and Applications
    Dianlei Xu,
        Tong Li,
         Yong Li, Senior Member, IEEE,
         Xiang Su, Member, IEEE,
         Sasu Tarkoma, Senior Member, IEEE,
         Tao Jiang, Fellow, IEEE,
       Jon Crowcroft, Fellow, IEEE,
       and Pan Hui, Fellow, IEEE
         

D. Xu, T. Li, X. Su, and P. Hui are with the Department of Computer Science, University of Helsinki, 00014 Helsinki, Finland.(e-mail: dianlei.xu@helsinki.fi, t.li@connect.ust.hk, xiang.su@helsinki.fi, sasu.tarkoma@helsinki.fi, panhui@cse.ust.hk.) 
D. Xu and Y. Li are with Beijing National Research Center for Information Science and Technology (BNRist), Department of Electronic Engineering, Tsinghua University, Beijing 100084, China.(e-mail: liyong07@tsinghua.edu.cn.) 
T. Li and P. Hui are also with the Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong.
T. Jiang is with the School of Electronics Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China. (e-mail: taojiang@ieee.org)
J. Crowcroft is with the Computer Laboratory, University of Cambridge, William Gates Building, 15 JJ Thomson Avenue, Cambridge CB3 0FD, UK. (e-mail: Jon.Crowcroft@cl.cam.ac.uk.)


    December 30, 2023
============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================

empty
empty



Real-time human activity recognition plays an essential role 
in real-world human-centered robotics applications,
such as assisted living and human-robot collaboration.
Although previous methods based on skeletal data to encode human poses
showed promising results on real-time activity recognition,


they lacked the capability to consider the context provided by objects within the scene and in use by the humans,
which can provide a further discriminant between human activity categories.
In this paper, we propose a novel approach to 
real-time human activity recognition,
through simultaneously learning from observations
of both human poses and objects involved in the human activity.
We formulate human activity recognition as a joint optimization problem under a unified mathematical framework,
which uses a regression-like loss function to integrate human pose and object cues
and defines structured sparsity-inducing norms to identify discriminative body joints and object attributes.


To evaluate our method,
we perform extensive experiments on two benchmark datasets
and a physical robot in a home assistance setting.
Experimental results have shown that
our method outperforms previous methods
and obtains real-time performance for human activity recognition with a processing speed of 10^4 Hz.










§ INTRODUCTION





























Real-time human activity recognition is an essential capability of robots
in human-centered robotics applications,
such as assisted living, service robotics, human-robot teaming, and human-robot interaction
<cit.>.
It allows intelligent robots to understand human behaviors in a timely manner
in order to effectively assist and interact with humans.
Human activity recognition by robots in the real world is a difficult problem,
complicated by both variations in human appearances and poses, 
and by challenges such as illumination changes or occlusions.
Given these challenges, 
it is important for a robot to extract as much relevant information as possible from 
sensory observations.
For example, as illustrated in Figure <ref>,
this information can consist of humans themselves,
such as human poses encoded by the human skeleton representation,

and the context from the objects in the environment
and objects that the human is interacting with,
which provide additional cues to recognize activities.
Moreover,
in most real-world robotics applications,
and especially in time-critical scenarios,
activity recognition must occur in real-time
so that a robot can promptly interact with and assist humans.

Due to the importance of activity recognition,
many methods have been introduced over the past few decades <cit.>.
Especially, techniques based on skeletal data from structured-light cameras <cit.>
have attracted  increasing attention,
due to skeletal data's real-time performance
and invariance to viewing distances and angles.
For example,
the methods can be implemented based on hand-crafted skeletal features
<cit.>,
a concatenation of multiple types of features <cit.>,
and learning skeleton-based representations, e.g.,
by sparse optimization <cit.> or deep learning <cit.>.
These methods generally have the limitation of not learning from the
context of objects that the human is interacting with.
Although several methods used object information <cit.>,
they require explicit knowledge of the objects,
such as object affordances that are typically manually defined to describe how an object can be interacted with. 
Moreover, previous methods cannot estimate the importance of the objects 
in recognizing human activities.



In this paper, we propose a principled method for real-time human activity recognition
based on learning simultaneously from observations of humans and objects.
Our approach formulates activity recognition as a regression-like optimization problem,
and applies structured norms as regularization terms to promote sparsity and identify
discriminative skeletal joints and object attributes.
This formulation is inspired by the fact that many activities rely solely
on a subset of joints (e.g., a waving activity uses only joints in the arm, not in the legs),
or can be recognized
based on context of objects in the scene 
(e.g., reading a book and typing on a laptop
at a table appear similar if only the human pose is considered).
By learning the importance of both to human activities,
the proposed method is able to identify and integrate more relevant information
to improve activity recognition accuracy.
Because classification is integrated in our regression-like convex objective function
(i.e., no separate classifier is needed),
our approach is capable of operating in real-time,
which makes it suitable for robotics applications with real-time needs.

This paper has two major contributions:



  * We propose a novel principled method
that formulates human activity recognition
as simultaneously learning from human and object observations
based on a unified regularized optimization framework.
The method identifies both discriminative skeletal joints
and discriminative object attributes,
and integrates classification with sparse representation learning in order
to enable a high processing speed for real-time recognition.


  * We implement a new iterative optimization algorithm to
solve the formulated regularized optimization problem that has dependent model parameters,
which holds a theoretical guarantee to find the optimal solution.

















§ RELATED WORK








Activity recognition has been shown to be a critical ability for 
robots to work with people in real-world human-centered robotics applications
<cit.>.
This section provides a brief review of existing methods of skeleton-based representations 
and object-assisted activity recognition.

























 §.§ Skeleton-Based Representations for Activity Recognition


Among diverse human representations, skeleton-based representations attracted
extensive attentions since the availability of structured-light or color-depth cameras.
Skeleton-based representations can be based on joint positions, displacement,
orientation, 
and a combination of multiple joint features <cit.>.


Relative spatial displacement between a pair of body joints
is one of the most commonly applied skeleton-based features.
For example, the normalized joint positions were employed to compute pairwise relative distances between joints as features to categorize activities using extreme learning machines <cit.>.
Euclidian distances betweens joint pairs were computed as skeletal features
to recognize activities <cit.>.
Rahmani et al. <cit.> chose a reference joint,
such as the torso center,
and computed relative distances to other body joints.
In addition, 
orientation of a segment between two joints in space or time
is widely studied.
Boubou and Suzuki <cit.> implemented the 
histogram of oriented velocity vector features
that calculate joint velocities between frames and use distributions of joint orientations to classify human activities.
Yang and Tian <cit.> designed 
a descriptor to include joint orientation differences between frames as features.
Joint positions were also directly applied 
as input into long short-term memory networks and recurrent neural networks to recognize activities <cit.>.
Recent methods integrated multiple features.
Luo et al.<cit.> fused sparse-coding skeleton features with a representation named center-symmetric motion local ternary pattern,
which extracts spatial and temporal gradients as features.
A learning-based method is proposed in <cit.>
to estimate the weights of feature modalities for activity recognition.

While most previous methods only consider skeletal information to build representations,
our proposed method focuses on
integrating cues from both humans and objects for activity recognition,
in a unified optimization framework.




 §.§ Object-Assisted Activity Recognition


A few methods were implemented to take into account of objects for activity recognition.
Koppula et al. <cit.> used human and object trajectories as a particle of an anticipatory temporal conditional random field for activity recognition.
Yu et al. <cit.> combined human body joint and object positions
as representations and used a boosting technique to identify human activities.
Wei et al. <cit.> implemented a human-object interaction model
that combines spatiotemporal human body joint displacements with object recognition and localization in sequence of frames.
Hu et al. <cit.> implemented a heterogeneous method for joint features learning that concatenates spatial displacements of the skeleton data over a sequence of frames, as well as color and depth patterns and their gradients around each joint.
Besides extracting joints and objects as independent features,
probabilistic models based on human poses and object interactions were also designed
for activity recognition <cit.>.

Previous object-assisted activity recognition methods
cannot estimate the importance of objects and skeletal features in recognizing
activities.
Several methods <cit.>
also require predefined knowledge about the objects
such as object affordance. 
Our proposed method provides the capability of not only integrating human and object cues,
but also estimate their importance when recognizing human activities.












§ THE PROPOSED JOINT LEARNING APPROACH









Notation.
In this paper, matrices are denoted using boldface uppercase letters, and vectors
using boldface lowercase letters.
For a matrix 𝐌 = { m_ij}∈^p × q, we refer to its i-th
row as 𝐦^i and its j-th column as 𝐦_j, and m_ij as the
element in the i-th row and the j-th column.
The Frobenius norm of a matrix 𝐌 is computed as
𝐌_F = √(∑_i=1^p ∑_j=1^q m_ij^2).
For dimensionality, d_T^j represents the dimensionality of the j-th body joint of
the human and d_O^m indicates the dimensionality of the m-th attribute
modality of an object.



 §.§ Problem Formulation




We assume that the input data instance set, = {, },
consists of paired observations of a human and  objects.
= [ _1, …, _N ] ∈^d_T × N denotes the matrix
of observations of the human, where _i ∈^d_T is the feature
vector describing the human's J joints in the i-th data instance.
Subsections in _i describe individual joints, with
_i^j ∈^d_T^j describing the j-th joint in the i-th data instance.
Each body joint is described by its displacement from the center of the body
(typically, the torso `joint' in many skeletal representations).

Observations of the objects are encoded in the matrix
= [ _1, …, _N ] ∈^d_O × N, where
_i ∈^d_O is the feature vector representing all the objects
in the i-th data instance.
Each object feature vector is sub-divided to encode multiple objects,
each with M attribute modalities,
such that _i^o_m∈^d_O^m represents the features describing the
m-th attribute modality of the o-th object in the i-th data instance.


Activity category labels for each training data instance are denoted in the category indicator matrix
= [ ^1; …; ^N ] ∈^N × C, 
where ^i ∈^C denotes the category indicator vector for the i-th data instance
and C denotes the number of human activity categories.
Specifically, y_ic indicates the probability that the i-th
data instance 𝐱_i = {_i, _i} belongs to the c-th activity category.
In the training phase, these probabilities are either 0 (if the data instance
does not belong to that category) or 1 (if the data instance belongs to that
category).





We formulate human activity recognition based upon both skeletal observations and object observations as a regression-like optimization problem:

    min_,      ^⊤ + ^⊤ - _F^2

where = [ _1, …, _C ] ∈^d_T × C represents a weight matrix indicating
the importance of  to the activity category labels, and
= [ _1, …, _C ] ∈^d_O × C is a weight matrix doing the
same for .
_c ∈^d_T represents weights of joints with respect to c-th category,
with subsections _c^j ∈^d_T^j representing the weights
of the j-th joint to the c-th category.
Similarly, _c ∈^d_O represents weights of object attributes with respect to c-th
category, with subsections
_c^o_m∈^d_O^m representing weights of the m-th attribute of the o-th object
to the c-th category.



 §.§ Learning Discriminative Joints and Object Attributes


When recognizing activities, specific body joints and object attributes are typically more discriminative.
For example, joints in the arm are much more important when a human is retrieving an
object from a shelf than joints in the leg would be.
Similarly, attributes describing the object being retrieved would allow a robot
to understand, for instance, whether the human is about to work on a laptop
or read a book.

In order to identify discriminative joints, 
we introduce the skeletal norm on the weight matrix , defined as: 

    _S = ∑_c=1^C ∑_j=1^J_c^j _2

This skeletal norm enforces the ℓ_2-norm within a joint feature and the ℓ_1-norm between
joints in order to force sparsity and identify discriminative joints (Figure <ref>).

    min_, ^⊤ + ^⊤ - _F^2 + λ_1 _S




    _c =
    [ _c^1;    ⋮; _c^j;    ⋮; _c^J;      ]



Similarly, we also introduce a new attribute norm
to learn the importance of various attribute modalities of the objects.
Attribute modalities can describe the color histograms (e.g., red, green, and blue), 
shape (e.g., gradient features), texture, or relationships of the object to the human (e.g., distances).
We define the attribute norm over the weight matrix  as:

    _A = ∑_c=1^C ∑_o=1^O∑_m=1^M_c^o_m_2

The ℓ_2-norm is employed to enforce similar weights within an attribute modality,
and the ℓ_1-norm is used between these attribute modalities to enforce sparsity 
in order to identify discriminative object attributes (Figure <ref>).






Then, our final formulation formulates activity recognition 
as a regularized optimization problem:

    min_, ^⊤ + ^⊤ - _F^2 + λ_1 _S + λ_2 _A

where λ_1 and λ_2 denote the hyperparameters to balance the importance
of the loss function and regularization norms.




 §.§ Recognizing Human Activities


After we solve the regularized optimization problem in Eq. (<ref>)
using Algorithm <ref>,
we obtain the optimal weight matrices
^* = [ _1^*, …, _C^* ] ∈^d_T × C and
^* = [ _1^*, …, _C^* ] ∈^d_O × C.
Each column _c^* and _c^* denotes the importance of, respectively,
an observation of a human  and objects 
to recognize the c-th activity category.
Given a new observation 𝐱 = {, }, 
the activity category
y ( , ) is classified by:

    y ( , ) = max_c^⊤_c^* + ^⊤_c^*

Since the objective function in our formulation can be used to perform classification,
no separate classifiers is needed. 

By learning the weight matrices for both human and object observations, 
our approach explicitly identifies discriminative human joints and object attributes.
For example,
consider the learned human observation weight matrix ^*:

    ^* =
    [ _1^1    ⋯ _c^1    ⋯ _C^1;    ⋮    ⋱    ⋮         ⋮; _1^j    ⋯ _c^j    ⋯ _C^j;    ⋮         ⋮    ⋱    ⋮; _1^J    ⋯ _c^J    ⋯ _C^J;      ]

where _c^j represents the importance of
the j-th joint to the c-th activity category.




The sum-value of all elements within the sub-matrix _c^j indicates the relative importance
of the j-th human joint when recognizing the c-th activity category.
The sum-value of all elements in the row vector ^j
indicates the importance of the j-th human body joint to recognize all activity categories.
Similarly,  can also be used to analyze and identify which attributes of which objects are important
when recognizing human activities. 



 §.§ Optimization Algorithm





The formulated optimization problem in Eq. (<ref>) is difficult to solve  
because the regularization norms _S and _A are not smooth and because we need
to simultaneously find the optimal solutions for  and , both of which the final
solution depends on.
To solve this, we propose a new iterative optimization solver as presented in Algorithm <ref>.

We calculate the derivative of Eq. (<ref>) with respect to _c
in order to solve :

    ^⊤_c + ^⊤_c - _c + λ_1 𝐃^c_S _c = 0




    _c = ( ^⊤ + λ_1 𝐃^c_S )^-1( _c - ^⊤_c )

where 𝐃^c_S is a block diagonal matrix with
1/2 _c^j _2𝐈_d_T^j as the j-th block.

Similarly, we compute the derivative of Eq. (<ref>) with respect to _c
in order to solve :

    ^⊤_c + ^⊤_c - _c + λ_2 𝐃^c_A _c = 0




    _c = ( ^⊤ + λ_2 𝐃^c_A )^-1( _c - ^⊤_c )

where 𝐃^c_A is a block diagonal matrix having O blocks.
Each of these diagonal blocks is composed of M diagonal blocks, where the m-th
diagonal block is
1/2 _c^o_m_2𝐈_d_O^m.

Because the solution to _c depends on both _c and 𝐃^c_S (which
is dependent on _c), and the solution to _c depends on both _c
and 𝐃^c_A (which is dependent on _c), an iterative optimization
algorithm is necessary to address this problem.
The proposed optimization solver is detailed in Algorithm <ref>,
which alternately solving _c and _c until convergence.
This proposed algorithm holds a theoretical guarantee to converge to the optimal solution:


Algorithm 1 is guaranteed to converge to the optimal solution to the formulated regularized optimization problem in Eq. (<ref>).



See supplementary materials[<hcr.mines.edu/publication/HAR_Supp.pdf>].




The time complexity of Algorithm <ref> is dominated by Steps (5) and (6), because
Steps (3) and (4) are trivial, executing in linear time of
𝒪( C d_T ) and
𝒪( C d_O ), respectively.
Steps (5) and (6) can be solved as a system of linear equations instead
of performing the matrix inverse, and are respectively
𝒪( d_T^2 ) and 𝒪( d_O^2 ).



§ EXPERIMENTAL RESULTS










We assess our approach's performance
on two benchmark activity recognition datasets and using a physical robot as a case study.
In the experiments,
we used one type of skeletal features and three
different object attribute modalities.
The skeletal feature used is the displacement of each body joint
from the central torso joint.

The object attributes used are color, shape, and object-joint distance.
The color attribute is implemented using red, green, and blue histograms.
The shape attribute is implemented using the Histogram of Oriented Gradients (HOG) <cit.> features.
Finally, the object-joint distance attribute is implemented through calculating the
distance in 3D space from the object to each skeletal joint
to model the interaction between objects and the human.
We also evaluate our case study using a multinomial probability
distribution as the object attributes, showing our method's ability to
identify the most important objects with respect to specific human activities.









 §.§ Results on Cornell Activity Dataset


The Cornell Activity Dataset (CAD-60) has been widely used as
a standard benchmark dataset for activity recognition <cit.> in robotics.
The dataset consists of activities performed by 4 humans, with
each activity execution recorded as color images, depth images, and
annotated skeleton joint positions for 15 joints.
Our experiments used six activities that involve
the use of objects, including writing on whiteboard,
cooking (stirring),
cooking (chopping),
working on computer, rinsing mouth with water, and talking on the phone,
as illustrated in Figure <ref>.
All objects existing in a scene were utilized,
even if they did not relate to the human
activity. For example, the whiteboard was still included in our experiments
when recognizing the cooking (stirring) human activity scenes.






The quantitative experimental results are listed in Table <ref>.
It is observed that the proposed approach is able to identify
98.11% of these activity executions correctly.
This table also compares our results with other state-of-the-art methods
for human activity recognition, 
which indicates that our simultaneous
learning from both human observations and object observations
provides superior performance.
This table also shows that limiting our approach to only one of our
proposed sparsity inducing norms degrades our performance.
This drop off is significant when we only deoploy the skeletal norm in the formulation,
showing that object attributes provide useful discriminative
information.








 §.§ Results on MSR Activity 3D Dataset


We further evaluate our approach based on  the MSR Daily Activity 3D Dataset
<cit.>, a commonly used public dataset for benchmarking human
activity recognition approaches.
This dataset consists of daily human activities performed by ten human subjects,
which is recorded with color images, depth images, and annotated joint positions
for 20 skeletal joints.
Examples of color and depth images from activity categories involving the use of objects
are illustrated in Figure <ref>.






The quantitative experimental results are listed in Table <ref>.
It is observed that our approach achieves an activity recognition accuracy of 97.71%.
Table <ref> also compares our results with
existing state-of-the-art approaches that have also been
tested on this dataset, showing again that our simultaneous learning
from both human body joints and object attributes is effective
to recognize human activities.
Moreover, the results indicate the importance of the proposed
attribute norm, as omitting this norm causes
our approach to loss the ability of identifying most important object attributes,
thus decreasing the recognition accuracy.











 §.§ Results in Home Assistance Case Studies






In addition to evaluating our method on the publicly available
benchmark datasets, we also implemented our approach on a physical
robot in order to demonstrate its performance in case study.
We deployed our approach on a Turtlebot robot participating in
a simulated home assistance scenario (Figure <ref>).
The robot has a color-depth sensor onboard
to extract 3D skeleton data and a lightweight netbook for processing.



In this scenario, five activities were defined, with Figure <ref>
showing example color and depth images for each activity.
These activities are drinking wine, storing food,
storing dishes, pouring wine, and eating.
Each activity was performed 20 times.
In order to test our approach in learning simultaneously
from observations of the human and the objects, these
activities were defined to involve similar objects and human poses.
For example, both drinking wine and pouring wine involve a
glass and a bottle; however, drinking wine is
performed while sitting down and pouring wine is performed while standing up.
Similarly, both eating and drinking wine are
activities performed by a sitting human,
but they involve different objects (respectively, a bowl and a spoon
versus a wine glass and a bottle).



The quantitative experimental results are presented in Table <ref>.
We can observe that the proposed approach achieves an overall accuracy of 98.33% in
the case studies.
Comparison with baseline real-time approaches is also listed in Table <ref>,
which shows that our approach is superior to
two standard real-time machine learning methods as baselines.
With only the skeletal norm or the attribute norm as the regularization,
our approach achieves good accuracy but less than with the
complete formulation that uses both norms.







We also tested our method with a different
set of attributes in order to assess its ability to identify
discriminative objects.
In this setup, we defined five attribute modalities, where each modality
is the probability that an object category appeared in the view of the robot.
The 5 object categories used are the wine bottle, glass,
fridge, bowl, and spoon.
The probabilities that an object appeared in a scene were obtained from the YOLO object detection system <cit.>,
which uses a pre-trained neural network to
identify common household objects.
For example, for the activity of drinking wine, the probability of a
bottle or glass appearing would be close to 1, and close
to 0 for the remaining objects.






Using this setup, our approach is able to recognize 96.67% of home
activities correctly.
Additionally, this setup allowed our approach to identify discriminative
objects, as each column of the  matrix contained only five values,
each relating one object to that column's associated human activity.
Figure <ref> displays two columns from the  weight matrix.
In Figure <ref>, we observe that the bottle and
glass are the only objects receiving weights, as these are
very indicative of the drinking wine activity.
Similarly in Figure <ref>, we see that the fridge
receives nearly 90% of the total column weight, identifying
it as being very indicative to recognize the storing food activity.
Similarly, Figure <ref> displays this relationship
for two of the rows of .
Figure <ref> demonstrates that for the bowl object,
the most related activity is eating.
Figure <ref> shows this for the fridge
object, which shows that the most relevant activity is storing food,
the only activity category in which this object appears.








 §.§ Discussion






  §.§.§ Real-Time Performance

One of the major advantages of our approach

is its ability to run at real-time speeds.
For each dataset we evaluated, we analyzed the runtime of our
activity recognition approach, summarized in Table <ref>.
Due to the efficiency of our proposed convex problem formulation
that integrates classification
within the loss function,
our approach obtains recognition processing speeds in excess of 2 × 10^4 Hz,
when executing on an Intel i5 processor with 4Gb memory.
Our approach provides a suitable solution for accurate and real-time recognition
of human activities.





  §.§.§ Hyperparameter Selection


In our problem formulation in Eq. (<ref>),
the hyperparameters
λ_1 and λ_2 control the importance of the
skeletal norm and attribute norm, respectively,
and balance these two norms with the loss function.
As our presented results have demonstrated,
the method's accuracy
decreases with either of these hyperparameters assigned to 0.
As each norm captures different information (i.e., weights of skeletal features
or object attributes),
both are necessary for the proposed approach to achieve its
state-of-the-art accuracy.
Also, we observe that as the values of these hyperparameters
become too large, performance decreases as the loss function
relating observations to activity labels becomes less
important.
Our presented results use the hyperparameter values of
λ_1 = 0.1 and λ_2 = 0.1.




§ CONCLUSION



In this paper, we introduce a principled method for activity recognition
through simultaneous learning from observations of human poses and object attributes.
The proposed approach is capable of identifying discriminative
joints and object attributes, providing an interpretable understanding
of their importance to human activities and the relationships between
joints and objects.
We formulate activity recognition as an optimization problem
that uses a regression-like loss function to integrate teammate and object
cues to perform activity recognition,
and utilizes sparsity-inducing norms to estimate feature importance.

We introduce an iterative algorithm guaranteed to find the optimal solution.
We assess our proposed approach on two benchmark activity
recognition datasets,
and on an actual robot to show a case study.
Experimental results have shown that our approach achieves 
state-of-the-art recognition accuracy and obtains real-time performance.



ieeetr


