Pattern Analysis and Applications manuscript No.
(will be inserted by the editor)

Recognizing Daily Activities from Egocentric Photo-Streams

arXiv:1710.04112v1 [] 11 Oct 2017

Alejandro Cartas · Juan Marı́n · Petia Radeva · Mariella Dimiccoli

Received: date / Accepted: date

Abstract Wearable cameras can gather large amounts
of image data that provide rich visual information about
the daily activities of the wearer. Motivated by the
large number of health applications that could be enabled by the automatic recognition of daily activities,
such as lifestyle characterization for habit improvement,
context-aware personal assistance and tele-rehabilitation
services, we propose a system to classify 21 daily activities from photo-streams acquired by a wearable photocamera. Our approach combines the advantages of a
Late Fusion Ensemble strategy relying on convolutional
neural networks at image level with the ability of recurrent neural networks to account for the temporal
evolution of high level features in photo-streams. The
A.C. was supported by a doctoral fellowship from the Mexican Council of Science and Technology (CONACYT) (grantno. 366596). This work was partially founded by TIN201566951-C2, SGR 1219, CERCA, ICREA Academia’2014 and
20141510 (Marató TV3). The funders had no role in the
study design, data collection, analysis, and preparation of
the manuscript. M.D. is grateful to the NVIDIA donation
program for its support with GPU card.
A. Cartas
University of Barcelona,
Mathematics and Computer Science Department,
08007 Barcelona,
Spain
E-mail: alejandro.cartas@ub.edu
M. Dimiccoli
E-mail: mariella.dimiccoli@ub.edu
P. Radeva · M. Dimiccoli
University of Barcelona,
Mathematics and Computer Science Department,
08007 Barcelona, and
Computer Vision Center,
08193 Cerdanyola del Vallès,
Barcelona,
Spain

proposed method was tested on a set of 44,902 egocentric images and achieved an overall accuracy of 87.07%,
outperforming state of the art end-to-end methodologies.
Keywords Egocentric Vision · Lifelogging · Activity
Recognition · deep learning · recurrent neural networks

1 Introduction
During the last decade there has been a growing interest
in analyzing human activities from sensory data [14].
More recently, the introduction of wearable cameras
opened the unique opportunity to capture richer contextual information than using only traditional sensors
[15]. Lifelogging cameras, as a particular case of wearable cameras, are the only tools that allow to capture
image data during several days thanks to their very low
frame rate (2fpm) without recharging its battery. Consequently, they are best suited to monitor the Activities
of Daily Living (ADLs) of the wearer that include, but
are not limited to, the activities that an independent
person performs on daily basis for living at home or in
a community [13]. The monitoring of these activities has
several applications in health-related research including
active aging monitoring, tele-rehabilitation, frailty prevention and stroke-survivor monitoring among others
[13, 19].
However, activity recognition from first-person (egocentric) photo-streams has received relatively little attention in the literature [4, 3, 16, 2]. One of its major
challenges is that photo-streams are characterized by
a very low frame-rate, and consequently useful important features such as optical flow cannot be reliably
estiimated. With the only exception of [2], all existing

Alejandro Cartas et al.

Reading

TV

Walking outdoor

Biking

2

Fig. 1: Examples of consecutive frames captured while the user was performing four activities: Biking, Walking
outdoor, TV, and Reading.

methods treat photo-streams as a set of unrelated images, neglecting the fact that while the user is performing a given activity such as cooking, the temporal coherence of environment concepts such as food, kitchen,
pan is preserved despite of suddenly image appearance
changes. This can be easily appreciated in Fig. 1, where
examples of consecutive frames are shown while the
user is performing different activities, including very
dynamic activities such as biking and walking outdoor.
Additionally, some images can contain poor information that is difficult to interpret without considering it
within its temporal image context. For instance in the
4th image of the activity walking outdoor in Fig. 1, it
would be impossible to assign the correct label without
looking at neighboring images.
In this paper, we propose a new method for activity recognition from photo-streams that is based on
our previous work [3], but extends it by taking into
account the temporal coherence of high level features.
More specifically, in [3] we applied a Late Fusion Ensemble (LFE) method that merges through a random
decision forest the activity probabilities extracted by
a Convolutional Neural Network (CNN) with contextual information embedded in a fully connected layer
of the CNN. In this work, we integrate a random decision forest within a temporal framework implemented
in terms of a Long Short Term Memory (LSTM) recurrent neural network that processes overlapping batches
of fixed size and learns the temporal evolution of high
level features.
With respect to [3], our contributions in this paper
are as follows:

– we straighten our originally proposed pipeline by
integrating for the first time LFE in a framework
based on the temporal coherence of high-level features in low temporal resolution photo-streams instead of treating images as non-temporally related,
– we extend the annotated dataset from ≈18K images
to about ≈45K and made the annotations publicly
available 1 ,
– we extend significantly the comparisons with state
of the art techniques.
– we provide a more extensive validation of the method
previously proposed by us in [3]
The reminder of the paper is organized as follows: in
section 2, we discuss related work; in section 3, we detail
the proposed approach. In section 4.1, we introduce the
dataset used in the experiments. We detail the methodology we followed for conducting our experiments and
the different combinations of networks and layers we
used in section 4.3, which details the training settings.
The results we obtained are discussed in section 4.4. Finally, in section 5, we present our conclusions and final
remarks.

2 Related work
Recently, activity recognition from egocentric videos
has become an active area of research, specially using
high-temporal frame rate videos. Fathi et al. [7] proposed a probabilistic model that maps activities into
1

The annotations will
http://www.ub.edu/cvub.

be

publicly

available

at

Recognizing Daily Activities from Egocentric Photo-Streams

3

a set of multiple actions, and each action is modeled
per frame as a spatio-temporal relationship between the
hands and the objects involved on it. They further extend their work [8] by proposing another probabilistic
generative model that incorporates the gaze features
and that models an action as a sequence of frames. Pirsiavash and Ramanan [18] introduced a dataset of 18
egocentric actions of daily activities performed by 20
persons in unscripted videos. On this dataset, they presented a temporal pyramid to encode spatio-temporal
features along with detected active objects knowledge.
These temporal pyramids are the input of support vector machines trained for action recognition. More recently, Ma et al. [12] proposed a twin stream CNN architecture for activity recognition from videos. One of
the streams is used for recognizing the appearance of
an object based on a hand segmentation and a region of
interest. The other stream recognizes the action using
an optical flow sequence. In order to recognize activities, both streams are joined and the last layers are
fine-tuned.

same environment, time and global image features such
as color convey useful information for describing the activities in the context of the same wearer. However, the
method can hardly generalize to multiple users, since
the network needs to adapt or be trained again with respect to the contextual information for each new user.
For instance, two distinct persons might have different
daily routines, depending on their job, hobbies, age, etc.
and, consequently, the system needs to deal with an increased intra-class variability. For instance the ”work”
activity for different persons could have a very difference appearance.
To address the problem of generalizing the LFE
method to multiple users, Cartas et al. [3] used the
outputs of different layers of a CNN as contextual information instead of using color and time information
that are too much tied to a single user context. The
authors tested their approach on a dataset acquired by
three different users having different lifestyles. Instead
of focusing on contextual information, Oliveira et al.
[16] used a gradient boosting machine approach to retrieve activities based on their estimated relations with
objects in the scene. Their method was tested on an
extended version of the dataset used by Cartas el al.
[3], showing promising results.
While all the approaches described so far treated
photo-streams as a set of temporally unrelated images,
more recently Cartas et al. [2] proposed an end-to-end
approach that takes into account the temporal coherence of photo-streams. Their proposed architecture consists of Long Short Term Memory units on the top of
a CNN for each frame that is trained by processing the
photo-streams using batches of fixed size. Experimental
results over the same dataset used in [16], have shown a
6% improvement in accuracy with respect to the VGG16 baseline.
Inspired by these results, we propose here a new
approach that exploits the temporal evolution of highorder features to improve the results of the LTE method
proposed in [3].

The method proposed by Singh et al. [21] first extracts hand masks, head motion, and a saliency map
from a sequence of input images. Then, it uses these
features as the input of a neural network architecture
that combines four streams. The first two streams capture temporal egocentric features by processing a video
segment containing binary hand-arm masks. The third
stream processes spatial information by using as input
the full RGB frame. The last stream handles motion
taken from stacked optical flow encoded in a saliency
map.
In contrast to all above mentioned methods that
use egocentric videos, activity classification from lowtemporal frame rate egocentric photo-streams captured
by lifelogging devices has received comparatively little
attention in the literature [1]. This may be due partially
to the lack of available benchmarks, and partially to the
fact that photo-streams provide less contextual action
information with respect to video data since images are
taken at periodic intervals of 20 or 30 seconds. Due to
the lack of temporal coherence, motion features that are
the most largely exploited in egocentric videos, cannot
be reliably estimated. The pioneering work of Castro et.
al. [4] proposed a LFE method that combines, through
a random decision forest, the classification probabilities of a CNN with time and global features, namely
color histogram, from the input image, to classify images into 19 different activity categories. Their method
has been tested on a nonpublic lifelogging dataset made
of 40,103 egocentric images captured by a single person
during a 6 months period. Since the user activities are
performed almost daily at the same time and in the

3 Egocentric activity classification
Our base activity classification method is an ensemble
classifier composed of a CNN and a random forest, as
illustrated on Fig. 2, that acts at image level. Specifically, the random forest takes as input one or more
concatenated output vectors from the final layers of a
CNN. Depending on the CNN architecture, the input
for the random forest can be extracted from the last
convolutional layer, a fully-Connected (FC) layer, or the
softmax layer. For instance, a random forest that takes
as input the output of the softmax and a FC layer is

4

Alejandro Cartas et al.

Softmax
Conv 3
Input image

Conv 1

Conv 4 Conv 5

FC1

FC2

Probability score

Random Forest

Conv 2

Fig. 2: Activity recognition at image level. After fine-tuning a CNN, we combine the softmax probabilities and a
fully connected layer into a single vector. Later on, we train a random forest using this vector as its input.

shown on Fig. 2. The training of the ensemble is a twostep process. In the first step, the CNN is fine-tuned.
In the second step, a random forest is trained over the
output vectors of the CNN. The insight underlying this
method is that a fully connected layer provides global
contextual information which helps to generalize the
features characterizing a given activity among different
users.
With the goal of improving classification performance
while keeping the advantage of the ensemble classifier,
we extended the ensemble architecture to take into account the temporal information from neighbor frames.
Firstly, we considered overlapping batches of fized size
and we used a classifier to make a single prediction for
each batch. From an implementation point of view, this
is equivalent to have a many-to-one predictor since to
all frames of the bach is assigned a single label. More
specifically, we employed a random forest as classifier
that takes as input the concatenation of the outputs
of the first fully-connected layer of the CNN of n consecutive frames as shown in Fig. 3b. This choise was
motivate by the fact that it is less prone to overfitting,
does not expect linear features and has a small number
of hyperparameters. However, since the Random Forest
does not have an internal memory state, it does no take
into account explicitly the temporal evolution of high
level features, but the set of temporal predictions as a
whole.

3b is not end-to-end: the input of the LSTM is the
probability classification score from the ensemble. The
training is performed in two phases. We first train an
ensemble as stated in the previous section. In order to
reduce the computatinal cost in the second phase, we
store the classification scores of each training image.
During the second phase, the LSTM unit is trained using the classification scores of n consecutive frames. As
a way of performing data augmentation, the training
batches are sampled using a sliding window. For example, Fig. 3b shows two consecutive batches of 5 frames
that are sampled.

4 Validation

The main objective of the experiments performed in
this work is to prove that temporal contextual information combined with a CNN LFE improves the state
of the art activity classification accuracy on egocentric
photo-streams. To this goal, we performed two classes
of experiments. Specifically, the goal of the first class of
experiments was to determine the ensemble with the
best combination of layers for performance improvement based on the proposed approach, fistly proposed
in [3], illustrated in Fig. 2. In these experiments, we
used three networks as the base of our ensembles, namely
Secondly, encouraged by the promising results achieved the VGG-16 [20], InceptionV3 [22], and ResNet 50 [10].
The goal of the second class of experiments was to dein [2], showing that it is possible to capture the temtermine the best approach to exploit temporal contexporal evolution of high-level features extracted with a
tual information from consecutive frames. We used the
CNN without knowing event boundaries, and by the
same training setting and CNN as in [2] to compare and
success of LSTM in video classification tasks [23, 6],
evaluate the approaches depicted in Fig. 3 directly to
we propose the architecture showed in Fig. 3b. More
[2], that used the same setting, and other state of the
specifically, we introduce a many-to-many LSTM on
art methods.
the top of the ensemble previously described. Previous
We describe the dataset in section 4.1 and detail the
attempts to employ LSTM for modeling the temporal
ensembles training in section 4.3. We then present the
evolution of features over time have led to end-to-end
experimental results on activity recognition in section
trainable architectures [23, 6, 2]. Similarly to the archi4.4.
tecture proposed in Fig. 3a, the one proposed in Fig.

Recognizing Daily Activities from Egocentric Photo-Streams

5

Input image
Softmax layer
Dense layer
Random Forest
LSTM unit

CNN
architecture

1

2

3
1st batch

4

5

2

3

4
2nd batch

5

6

1

2

3
1st batch

(a)

4

5

2

3

4
2nd batch

5

6

(b)

Fig. 3: Activity recognition using temporal contextual information. With the activity recognition scores obtained
as shown in Fig. 2, in a) we train a random forest by feeding it with overlapping batches of fixed size and a single
label is assigned to each image of the batch; b) we train a many-to-many LSTM, feeding it in the same way than
in (a).
Attending a Seminar

Biking

Driving

Talking

Plane

Shopping

Walking Outdoor

Meeting

Cooking

Cleaning and chores

TV

Drinking together

Eating together

Walking Indoor

Mobile

Resting

Working

Drinking/Eating Alone

Public transport

Reading

Socializing

Fig. 4: Examples of the 21 activity categories from our annotated dataset.

4.1 Dataset
In our experiments, we used a subset of images from the
NTCIR-12 dataset [9]. This dataset consists of 89,593
egocentric pictures belonging to three persons and acquired with an OMG Autograph camera that captured
two pictures per minute. Each user worn this camera in
a period about three weeks, totaling 79 days.
We used a subset of 44,901 images from the NTCIR12 dataset, that we annotated in batches with 21 activity labels, extending the dataset used in [3]. These
pictures correspond to all users at different dates and
times, but having a similar proportion of about 15,000
images per user. The complete list of activity categories and their distribution of the number of images
are shown in Fig. 4 and Fig. 5, respectively. The dataset

splits used for the experiments of both methods are explained below.
Activity recognition at image level. For this
task we used the pictures from the three users for all
categories regardless of their date and time as in [3]
and illustrated in Fig. 6a (a). In order to maintain the
same percentage of samples for each class, we first performed a stratified 10-fold cross-validation over the images. Then a validation split for each fold containing
10% of the data was created by further making a stratified shuffle of its training split.
Activity recognition using temporal information. For this experiment we used the same data split
as in [2], illustrated in Fig. 6b. The purpose of this split
was to separate full days of activities rather than single
frames. This separation made harder the classification
task since similar consecutive frames only appear in one

Alejandro Cartas et al.

434
267
328
1,029

2,702

5,035

5,355
1,000
1,333

/ea

ing

Dr

ink

ink

ing

tog

eth
tin er
Ea g alo
tin
g t ne
og
eth
er
S
At
o
ten
c
din ializi
g a ng
se
mi
na
r
Me
eti
ng
Re
ad
ing
Cl
ea
nin
ga
T
nd V
ch
ore
s
W
ork
ing
Co
ok
in
Sh g
op
pin
g
Ta
l ki
ng
Re
sti
ng
Mo
bil
e
Pla
ne

r

ng

Bik
i

oo

Dr

or

ind

tdo

W
alk

ing

ng

ou

ing

192
176
207
575
594
249
330
1,173
1,660
676
309
2,645
1,245
1,122
2,988

1,154
1,570
2,158
26
816
3,000
688
1,735
1,045
3,468
609
820
304
1,733
0
247
0
247
167
419
831
1,417
510
494
646
1,650
426
405
230
1,061
272
492
1,057
1,821
508
511
0
1,019
1,025
747
0
1,772
879
274
18
1,171
726
2
661
1,389
247
387
192
826
1,456
4,308
1,182

ivi

Dr

W
alk

sp
Tr
an
lic
Pu
b

6,946

Dataset Summary
User 1
User 2
User 3
All

183
233

8000
7000
6000
5000
4000
3000
2000
1000
0

ort

Number of Instances

6

Categories

Fig. 5: Per class and per user label distribution in our dataset.
Training split
Training split

13

2

9

12

8

5

7

6

Test split
3

15

11

4

1

14

10

Test split

(a) Splits used in [3] for activity recognition at image level

(b) Splits used in [2] and this work for activity
recognition at batch level.

Fig. 6: Visual description of the dataset splits for the experiments carried out. Images of the same color belong to
the same day. In (a) the split into training and test does not take into account time metadata, so that images of
the same day can be found in both training and test, whereas in (b) images of the same day, can be found or in
training or in test even of the split does not take into account the temporal adjacency of days so that temporally
adjacent days, say 1 and 2 can be found in different sets (training or test).
split. Additionally, the splits proportionally maintained
the class imbalance of the dataset distribution shown
in Fig. 5. In order to create the splits, all the combinations of training and testing day splits were enumerated. Then, the Bhattacharyya distance between the
normalized distributions of the whole dataset and each
split for all combinations was calculated. Finally, the selected combination was the one with the minimal sum
of Bhattacharyya distances.

each class, they do not take into account of the number of instances available for each class. Moreover, the
results on the activity recognition at the image level
experiments are cross-validated. Considering that the
photo-streams lack of event boundaries, the predictions
done using temporal contextual information were calculated per frame.
4.3 Implementation
4.3.1 Activity recognition at image level

4.2 Evaluation metrics
Since the dataset is highly imbalanced, the classification performance of all methods was assessed by not
only using the accuracy, but also macro metrics for precision, recall, and F1-score. Since macro metrics are an
unweighted average of the metrics taken separately for

We used three different CNN architectures as base models for our ensembles, specifically the VGG-16, InceptionV3, and ResNet 50. All these networks were pretrained on ImageNet using the Keras framework [5].
Moreover, we employed a class weighting scheme for all
CNN models based on [11] to handle class imbalance

Recognizing Daily Activities from Egocentric Photo-Streams

7

during training. The random forests in our ensembles
were trained using the Scikit Learning framework [17].
The training was done on the dataset split described
in the previous subsection. The training details of each
ensemble configuration are detailed below.

work pre-trained on ImageNet. This random forests had
500 trees as estimators.

VGG-16. We fine-tuned a VGG-16 network [20] in
two phases. During the first phase, only the top layers were backpropagated with the objective of initialize
their weights. The optimization method used was the
Stochastic Gradient Descent (SGD) for 10 epochs for
all folds, a learning rate α = 1 × 10−5 , a batch size of
1, a momentum µ = 0.9, and a weight decay equal to
5 × 10−6. In the second phase, the last three convolutional layers were also fine-tuned and the initial weights
were obtained from the best epoch of the first phase.
Moreover, the SGD ran for another 10 epochs for each
fold and set with the same parameters except the learning rate α = 4×10−5 . Additionally, dropout layers were
added after each fully-connected layer as a mechanism
for regularization.

4.3.2 Activity recognition using temporal information

For all the tested ensembles we used the VGG-16 network as the base CNN for two main reasons. First, the
results reported in Table 1 indicated a slightly higher
accuracy. Second, it allows a fairer comparison with previous work [2].
The VGG-16 was trained for 14 epochs using the
SGD algorithm. During the first 10 epochs only the fully
connected layers were optimized using a learning rate
α = 1 × 10−5 , a batch size of 1, a momentum µ = 0.9,
and a weight decay equal to 5 × 10−6. During the last
4 epochs, the last 3 convolutional layers were also finetuned and the learning rate changed to α = 1 × 10−5 .
In comparison with the previous experiment, no class
weighting scheme was used for training.
VGG-16+RF. A set of random forests was trained
VGG-16+RF. Three different combinations of ranusing the distinct layers of the best epoch from the finedom
forests using 500 estimators were trained for this
tuned VGG-16 for each fold. The different combinations
experiment.
The first model used the first convolutional
of layers for the ensembles were: FC1, FC2, FC1+FC2,
layer
(FC1)
as
in the previous experiment. The second
and FC1+softmax. All the random forests had 500 trees
model was trained for comparison purposes following
as estimators as we found experimentally that this was
the description in [4]. In other words, its input was a
a good number.
feature vector obtained by concatenating the softmax
InceptionV3. InceptionV3 was also fine-tuned in
probability scores, the day of the week, the time of the
two phases. During the first phase, the last fully-connected day, and 10-bin size histogram for each color channel.
layer was optimized using SGD for 10 epochs for all
The last trained model used the FC1 layer, the sofmtfolds, a learning rate α = 1 × 10−5 , a batch size of
max probability scores, the day of the week, the hour,
32, a momentum µ = 0.9, and a weight decay equal to
and the time.
5 × 10−6. During the second phase, the last inception
VGG-16+RF+LSTM. For this configuration we
block was added to the optimization process and the
trained a LSTM using the predictions from the ensemnetwork was optimized for another 10 epochs setting
ble of VGG-16 plus the RF on FC1. The LSTM had 32
the learning rate α = 4 × 10−5 and the batch size to 10.
units and its output was connected to a fully-connected
Moreover, dropout layers were added before and after
layer. Dropout layers were added between the input and
the last global average pooling layer.
output of the LSTM layer as a way to perform regularInceptionV3+RF. A random forest was trained
ization. We trained three configurations using batches
using the global average pooling layer from InceptionV3
of consecutive frames, obtained by using a temporal
pre-trained on ImageNet. This random forest had 500
sliding window of size (or timestep) 5, 10, and 15.
trees as estimators.
VGG-16+RF many-to-one. A random forest was
ResNet50. ResNet50 was fine-tuned in two phases.
trained by concatenating the output vector of the FC1
The last fully-connected layer was optimized in the first
layer of the VGG-16 network of a consecutive number
phase using SGD for 10 epochs for all folds, a learning
of frames. This configuration employed the same samrate α = 1 × 10−3 , a batch size of 32, a momentum
pling window strategy for training. For example, Fig. 3a
µ = 0.9, and a weight decay equal to 5 × 10−6. During
shows a 5 timestep configuration training on two conthe last phase, the last residual block was also optimized
secutive batches and predicting the activity from the
using SGD with same learning rate and a batch size of
last frame. Since with this architecture, the same la10 for three additional epochs.
bel is assigned to all frames considered. We considered
configurations with timestep 5 and 10. The number of
ResNet50+RF. A random forest was trained usestimators for both cases was 700.
ing the last activation layer from an InceptionV3 net-

8

Alejandro Cartas et al.

Table 1: Comparison of the ensembles of CNN+Random forest on different combinations of layers. Upper table
shows the recall per class and the lower table shows the performance metrics.

Public Transport
Driving
Walking outdoor
Walking indoor
Biking
Drinking together
Drinking/eating alone
Eating together
Socializing
Attending a seminar
Meeting
Reading
TV
Cleaning and chores
Working
Cooking
Shopping
Talking
Resting
Mobile
Plane

89.23
99.60
90.95
79.17
91.93
80.74
75.94
80.58
89.13
78.70
80.92
81.40
93.74
57.14
95.82
81.98
81.07
72.36
87.97
89.59
87.67

InceptionV3+RF on
Global Avg. Pooling
91.08
99.87
97.20
94.86
97.57
95.77
91.34
96.03
97.75
78.11
93.45
96.67
97.12
92.58
98.69
94.63
95.65
95.80
90.96
97.10
86.39

Accuracy
Macro precision
Macro recall
Macro F1-score

87.07
83.85
84.08
83.77

95.20
94.69
94.22
94.29

Activity

InceptionV3

VGG­16
VGG­16

Public Transport
Driving
WalkingTransport
outdoor
Public
WalkingDriving
indoor
Biking
Walkingtogether
outdoor
Drinking
Walking indoor
Drinking/eating
alone
Biking
Eating
together
Drinking
together
Socializing
Drinking/eating
alone
Attending
seminar
Eatinga together
Meeting
Socializing
Reading
Attending a seminar
TV
Meeting
Cleaning andReading
chores
Working
TV
Cooking
Cleaning and
chores
Shopping
Working
Talking
Cooking
Resting
Shopping
Mobile
Talking
Plane
Resting
Mobile
Plane
Public Transport
Driving
WalkingTransport
outdoor
Public
WalkingDriving
indoor
Biking
Walkingtogether
outdoor
Drinking
Walking
Drinking/eatingindoor
alone
Biking
Eating
together
Drinking
together
Socializing
Drinking/eating
alone
Attending
a
seminar
Eating together
Meeting
Socializing
Attending a Reading
seminar
TV
Meeting
Cleaning andReading
chores
Working
TV
Cooking
Cleaning and
chores
Shopping
Working
Talking
Cooking
Resting
Shopping
Mobile
Talking
Plane
Resting
Mobile
Plane

92.41
99.90
94.61
92.73
99.17
94.93
88.25
96.42
97.47
80.57
96.61
96.83
96.04
92.22
97.34
91.69
93.18
89.64
91.54
95.27
82.61

ResNet50+RF
on Avg. Pooling
91.01
99.93
97.90
94.98
97.15
94.71
89.34
97.17
98.52
78.11
96.39
97.52
96.33
92.83
99.22
95.50
96.59
94.06
91.07
97.66
75.41

94.08
92.60
93.31
92.78

95.17
95.99
93.88
94.73

ResNet50

89.17
99.50
90.89
79.63
92.32
89.42
74.92
88.78
91.60
76.93
90.00
88.23
92.59
61.84
96.27
84.78
82.44
76.25
92.87
93.32
84.94

VGG-16+RF
on FC1
90.37
99.80
97.55
94.69
97.57
95.70
89.28
97.07
98.24
78.31
96.05
96.84
97.05
91.64
98.23
94.61
95.91
93.16
94.90
97.28
76.68

VGG-16+RF
on FC1+FC2
90.82
99.80
97.15
94.75
97.97
95.70
89.59
97.26
98.57
78.11
96.11
96.84
96.68
91.64
97.85
94.44
95.74
93.23
94.23
97.10
78.62

VGG-16+RF on
FC1 + Predictions
91.53
99.83
97.12
94.63
97.97
95.56
89.70
96.98
98.35
78.31
95.99
97.10
96.47
91.89
97.87
94.44
95.74
93.19
94.77
97.48
79.79

VGG-16+RF
on FC2
91.33
99.80
97.15
94.58
97.97
95.56
89.40
97.26
98.57
78.11
96.27
96.84
96.68
91.76
97.88
94.44
95.65
93.08
94.47
97.30
78.14

89.46
87.06
86.51
86.45

95.30
94.69
93.85
94.11

95.19
94.45
93.91
94.02

95.33
94.61
94.03
94.18

95.23
94.46
93.92
94.04

VGG-16

ResNet50
ResNet50

InceptionV3
InceptionV3

1.0

0.8

VGG­16+RF­on­FC1­+­Predictions
VGG­16+RF­on­FC1­+­Predictions

ResNet50+RF­on­Avg ­Pooling
ResNet50+RF­on­Avg ­Pooling

InceptionV3+RF­on­Global­Avg ­Pooling
InceptionV3+RF­on­Global­Avg ­Pooling

1.0
1.0
0.60.8
0.8
0.6
0.4
0.6
0.4
0.20.4
0.2
0.2
0.00.0
0.0

Fig. 7: Normalized confusion matrices of the best combination of layers for each baseline convolutional neural
network. This figure is best seen in color.

4.4 Results and discussion
Activity recognition at image level The results of
adding contextual information from fully connected layers are presented on Table 1. They show that CNN
LFE improves the performance for all baseline CNN on
different ensembles. After using the random forest on
the output of the softmax layer, the accuracy improved
8.13% with Inception V3, 1.11% with ResNet and 5.89%
when using VGG-16. Specifically, the recall of some categories with fewer learning instances improved significantly, such as Cooking, and Cleaning and Choring.

Additionally, high overlapping classes such as Drinking/Eating alone and Eating together also improved
their accuracy. The improvement over the overlapping
classes can also be seen on the confusion matrices shown
in Fig. 7. This means that the random forest improved
the classification of images belonging to categories that
score similar probabilities. Moreover, the only decrease
on accuracy is presented on the class Plane. Since its accuracy on the baseline CNN is very high (87.67%) considering the small number of learning instances (1,026),
we believe this decrease is a consequence of the random forest trying to balance the prediction error among

Recognizing Daily Activities from Egocentric Photo-Streams

75.97
76.64

Macro
Precision
68.50
71.41

Macro
Recall
67.49
66.43

Macro
F1-score
66.80
67.02

77.39

69.66

67.79

66.99

76.91

72.07

66.77

67.27

77.16

75.24

Method

Accuracy

VGG-16
VGG-16+RF on
FC1+Softmax[3]
VGG-16+RF on Softmax +
Date&Time + Color [4]
VGG-16+RF on FC1 +
Softmax + Date&Time
VGG-16+RF
timestep 5
VGG-16+RF
timestep 10
VGG-16+LSTM [2]
timestep 5
VGG-16+LSTM [2]
timestep 10
VGG-16+LSTM [2]
timestep 15
VGG-16+RF+LSTM
timestep 5
VGG-16+RF+LSTM
timestep 10
VGG-16+RF+LSTM
timestep 15

66.96
65.91

67.88

76.42

75.59

67.04

79.68

72.96

71.36

70.87

80.39

75.25

71.86

71.97

81.73

76.68

74.04

74.16

84.46

79.20

81.91

79.21

81.1

77.74

78.57

76.61

87.07

81.05

83.48

81.77

Table 2: Performance summary of the experiments on
activity recognition using temporal contextual information.

classes. Moreover, Table 1 shows that the best ensembles were the IncepvtionV3+RF on Global Avg. Pooling,
VGG-16+RF on FC1 + Predictions, and ResNet+RF
on average pooling. Furthermore, these ensembles improved the baseline accuracy by 7.5%, ≈ 6%, and 1.91%
for InceptionV3, VGG-16, and ResNet, respectively.Some
classification examples are shown in Fig. 8.
Activity recognition at batch level The results
of using temporal contextual information from batch
sequences is shown on Table 2. The fact that accuracy
performance are much higher when using the splits defined as in Fig. 6b (a) for the VGG-16 can be easily
understood when looking at consecutive frames in Fig.
1. Specially in the case of static activities such as reading or watching TV, consecutive frames are very similar
and putting some of them in training and other in test
is almost equivalent at doing the test including training
images, which is unfair.
Comparing the performances of VGG-16+LSTM with
VGG-16+RF+LSTM, our experiments demonstrated
that the activity scores obtained with CNN LFE are
better features for further temporal analysis with respect to features computed by a end-to-end architecture. In both cases, the best performances were achieved
by a larger timestep.
State of the art comparisons In Table 2, we compared the proposed method with the baseline VGG-16
fine-tuned CNN, with our previous method [3], with
the work of Castro et al. [4] and with [2], that exploits
temporal information. Our proposed approach VGG16+RF+LSTM gives the best results when using 15

9

as timestep value, achieving 87.07% accuracy. It is followed by the end-to-end VGG-16+LSTM proposed in
[2] that also takes into account temporal information
and achieves approximatively 6% less in terms of accuracy. In both methods, we can observe that the best
performance is achieved with timestep 15. On the contrary, with VGG-16+RF, being a many-to-one archietecture, the best results are achieved when using a small
timestep. We can also observe that, in general, methods
that take into account temporal information achieve
better performance with respect to methods that act
at image level. Among these latter methods, the VGG16 fine-tuned CNN gives the worst results and it is improved by all LFE methods used for comparison. On
this dataset, the methods [3] and [4] are almost equivalent. To explain this result we performed further experiments to investigate the contribution of time metadata
and we could double check that indeed they help in improving the performances (VGG-16 + RF on FC1 +
softmax + date&time), a result that would unlikely be
generalizable to a dataset with more individuals having a very different lifestyle. The fact that color histogram as contextual feature leads to slightly better
performances might be due to the different lightning
conditions imposed the activity categories.

5 Conclusion
In this paper, we proposed a new pipeline for activity
recognition from egocentric photo-streams that relies on
a two-phases approach. First, a CNN late fusion ensemble classifier, which combines different layers of a CNN
through a random forest is used to predict the activity
probabilities on each image. Specifically, the random
forest takes as input a vector containing the output
of the softmax probability layer and a fully connected
layer encoding global image features. Second, these vector probabilities are used in batches of temporally adjacent images, to train a many-to-many LSTM. Experimental results demonstrated that the proposed approach achieve better performances than a end-to-end
architecture and state of the art methods. The proposed
method has been tested on a subset of the NTCIR-12
egocentric dataset consisting of 44,902 images labeled
with 21 different activity labels and achieved an overall
accuracy of 87.07%.

References
1. Bolaños, M., Dimiccoli, M., Radeva, P.: Toward storytelling from visual lifelogging: An overview. IEEE Transactions on Human-Machine Systems 47(1), 77–90 (2017)

10

Alejandro Cartas et al.
Walking outdoor

VGG-16 Top 5
#
1
3
2
4
5

Activity
Driving
Walking outdoor
Biking
Mobile
Talking

Score
0.9831
0.0060
0.0068
0.0022
0.0008

Activity
Public Transport
Walking indoor
Talking
Walking outdoor
Shopping

Score
0.2618
0.1697
0.1782
0.0923
0.0743

Activity
Public Transport
Talking
Walking indoor
Walking outdoor
Shopping

Score
0.6322
0.1623
0.0374
0.0577
0.0320
0.0183

ResNet Top 5
#
1
2
3
4
5

InceptionV3 Top 5
#
1
2
3
4
5

Activity
Shopping
Walking indoor
Talking
Mobile
Meeting
Drinking/eating alone

#
1
2
3
4
5

Activity
Plane
Reading
Working
Attending a seminar
Public Transport

VGG-16 Top 5
Score
0.5362
0.3613
0.0602
0.0236
0.0167

#
1
2
3
4
5

ResNet Top 5

ResNet Top 5
#
1
3
2
4
5

VGG-16 Top 5

VGG-16 Top 5
#
1
2
4
3
5
6

Public Transport

Plane

Drinking/eating alone

Score
0.2618
0.1782
0.1697
0.0923
0.0743

Activity
Shopping
Drinking/eating alone
Mobile
Eating together
Drinking together

Score
0.5046
0.4725
0.0211
0.0009
0.0004

#
1
2
3
4
5
10

Activity
Shopping
Eating together
Drinking/eating alone
Drinking together
Meeting

Score
0.2656
0.2326
0.2178
0.1957
0.0629

Score
0.9939
0.0048
0.0009
0.0003
0.00003
0.0000

#
1
2
3
4
5

Activity
Plane
Working
Attending a seminar
Mobile
Reading

Score
0.6574
0.1944
0.0433
0.0253
0.0207

Score
0.7789
0.2174
0.0022
0.0005
0.0004

ResNet Top 5
#
1
2
3
4
5

Activity
Shopping
Walking indoor
Walking outdoor
Eating together
Drinking/eating alone

Score
0.9999
0.00006
0.00002
0.00000
0.0000

InceptionV3 Top 5

InceptionV3 Top 5

InceptionV3 Top 5
#
1
2
3
4
5

Activity
Plane
Attending a seminar
Cooking
Mobile
Reading
Working

Activity
Shopping
Walking indoor
Talking
Mobile
Socializing

#
1
2
3
4
5

Activity
Walking indoor
Shopping
Mobile
Walking outdoor
Drinking together

Score
0.6220
0.1909
0.0415
0.0369
0.0157

Fig. 8: Classification activity examples. On top of each image is shown its true activity label and on bottom its
top 5 predictions by VGG-16, ResNet50 and InceptionV3. Additionally, the result of the ensembles VGG-16+RF
on FC1+Softmax, ResNet50+RF on Avg. Pooling, and InceptionV3+RF on Global Avg. Pooling is highlighted
on color in its corresponding table. The green and red colors means true positive and false positive classification,
respectively.

2. Cartas, A., Dimiccoli, M., Radeva, P.: Batch-based activity recognition from egocentric photo-streams. arXiv
preprint arXiv:1708.07889, to appear in the 2nd international workshop on Egocentric Perception, Interaction
and Computing, in conjunction with the International
Conference in Computer Vision (ICCV), Venice, Italy
(2017)
3. Cartas, A., Marı́n, J., Radeva, P., Dimiccoli, M.: Recognizing activities of daily living from egocentric images.
In: Proceedings of the Iberian Conference on Pattern
Recognition and Image Analysis (IbPRIA), pp. 87–95.
Springer, Cham (2017)
4. Castro, D., Hickson, S., Bettadapura, V., Thomaz, E.,
Abowd, G., Christensen, H., Essa, I.: Predicting daily
activities from egocentric images using deep learning. In:
proceedings of the 2015 ACM International symposium
on Wearable Computers, pp. 75–82. ACM (2015)
5. Chollet, F., et al.: Keras.
https://github.com/
fchollet/keras (2015)
6. Donahue, J., Hendricks, L.A., Guadarrama, S.,
Rohrbach, M., Venugopalan, S., Saenko, K., Darrell, T.: Long-term recurrent convolutional networks for
visual recognition and description. In: CVPR (2015)
7. Fathi, A., Farhadi, A., Rehg, J.M.: Understanding egocentric activities. In: 2011 International Conference on
Computer Vision, pp. 407–414. IEEE (2011)
8. Fathi, A., Li, Y., Rehg, J.M.: Learning to recognize daily
actions using gaze. In: European Conference on Computer Vision, pp. 314–327. Springer (2012)
9. Gurrin, C., Joho, H., Hopfgartner, F., Zhou, L., Albatal, R.: Overview of ntcir-12 lifelog task. In: NTCIR12. National Institute of Informatics (NII). URL http:
//eprints.gla.ac.uk/119206/

10. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning
for image recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
11. King, G., Zeng, L.: Logistic regression in rare events data.
Political Analysis 9(2), 137–163 (2001). DOI 10.1093/
oxfordjournals.pan.a004868
12. Ma, M., Fan, H., Kitani, K.M.: Going deeper into firstperson activity recognition. In: The IEEE Conference
on Computer Vision and Pattern Recognition (CVPR)
(2016)
13. Martin-Lesende, I., Vrotsou, K., Vergara, I., Bueno, A.,
Diez, A., et al.: Design and validation of the vida questionnaire, for assessing instrumental activities of daily living in elderly people. J Gerontol Geriat Res 4(214), 2
(2015)
14. Mukhopadhyay, S.C.: Wearable sensors for human activity monitoring: A review. IEEE sensors journal 15(3),
1321–1330 (2015)
15. Nguyen, T.H.C., Nebel, J.C., Florez-Revuelta, F.: Recognition of activities of daily living with egocentric vision:
A review. Sensors (Basel) 16(1), 72 (2016). DOI 10.
3390/s16010072. URL http://www.ncbi.nlm.nih.gov/
pmc/articles/PMC4732105/. Sensors-16-00072[PII]
16. Oliveira-Barra, G., Dimiccoli, M., Radeva, P.: Leveraging Activity Indexing for Egocentric Image Retrieval,
pp. 295–303. Springer International Publishing, Cham
(2017). DOI 10.1007/978-3-319-58838-4 33. URL https:
//doi.org/10.1007/978-3-319-58838-4_33
17. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V.,
Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P.,
Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.:

Recognizing Daily Activities from Egocentric Photo-Streams

18.

19.

20.

21.

22.

23.

Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, 2825–2830 (2011)
Pirsiavash, H., Ramanan, D.: Detecting activities of daily
living in first-person camera views. In: Computer Vision
and Pattern Recognition (CVPR), 2012 IEEE Conference
on, pp. 2847–2854. IEEE (2012)
Schüssler-Fiorenza Rose, S.M., Stineman, M.G., Pan, Q.,
Bogner, H., Kurichi, J.E., Streim, J.E., Xie, D.: Potentially avoidable hospitalizations among people at different activity of daily living limitation stages. Health services research (2016)
Simonyan, K., Zisserman, A.: Very deep convolutional
networks for large-scale image recognition.
CoRR
abs/1409.1556 (2014)
Singh, S., Arora, C., Jawahar, C.V.: First person action
recognition using deep learned descriptors. In: The IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR) (2016)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna,
Z.: Rethinking the inception architecture for computer
vision. In: The IEEE Conference on Computer Vision
and Pattern Recognition (CVPR) (2016)
Yue-Hei Ng, J., Hausknecht, M., Vijayanarasimhan, S.,
Vinyals, O., Monga, R., Toderici, G.: Beyond short snippets: Deep networks for video classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4694–4702 (2015)

11

