arXiv:1802.02534v2 [cs.AI] 8 Feb 2018

FixaTons: A collection of Human Fixations
Datasets and Metrics for Scanpath Similarity
Dario Zanca∗1,2 , Valeria Serchi3 , Pietro Piu3 ,
Francesca Rosini3,4 , and Alessandra Rufa3,4
1 Department of Information Engineering, University of Florence,
2 Department of Information Engineering and Mathematics, University

Florence, Italy
of Siena, Siena, Italy
and Visual Application Lab (EVALab), Department of Medicine, Surgery and Neurosciences,
University of Siena, Siena, Italy
4 Neurological and Neurometabolic Unit, Department of Medicine, Surgery and Neurosciences, University of Siena,
Siena, Italy
3 Eye-tracking

∗

Corresponding Author: dario.zanca@unifi.it

1

Contents
1 Introduction

3

2 Datasets included in the collection
2.1 SIENA12 . . . . . . . . . . . . . . .
2.1.1 Protocol for data collection in
2.2 Other Datasets . . . . . . . . . . . .
2.2.1 MIT1003 . . . . . . . . . . .

3
3
3
5
5

. . . . . .
SIENA12
. . . . . .
. . . . . .

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

3 Online resources

6

4 Structure of the FixaTons collection

7

5 Software included
5.1 List information . .
5.2 Get data (matrices)
5.3 Visualize data . . . .
5.4 Compute metrics . .
5.5 Compute statistics .
5.6 Example of use . . .

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

7
7
8
8
9
10
10

6 Appendix A:
6.1 Euclidean distance . . . . . . . . . . .
6.2 String-edit similarity . . . . . . . . . .
6.3 Time-delay embedding distance . . . .
6.4 Scaled Time-delay embedding distance

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

11
11
11
12
13

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

2

1

Introduction

In the last three decades, human visual attention has been a topic of great interest in various
disciplines. In computer vision, many models have been proposed to predict the distribution
of human ﬁxations on a visual input. Recently, thanks to the creation of large collections of
data, machine learning algorithms have obtained state-of-the-art performance on the task
of saliency map estimation.
On the other hand, computational models of scanpath are much less studied. Works
are often only descriptive [1] or task speciﬁc [2]. Computational models of scanpath with
general purpose are present in the literature, but are then evaluated in tasks of saliency
prediction [3], losing therefore information about the dynamics and the behaviour. This is
due to the fact that the scanpath is harder to model because it must include the description
of a dynamic. In addition to the diﬃculty of the problem itself, two technical reasons have
limited the research. The ﬁrst reason is the lack of robust and uniformly used set of metrics
to compare the similarity between scanpath. The second reason is the lack of suﬃciently
large and varied scanpath datasets.
In this report we want to help in both directions. We present FixaTons, a large collection
of human scanpaths (and saliency maps). It comes along with a software library for easy
data usage, statistics calculation and measures for scanpaths (and saliency maps) similarity.

2

Datasets included in the collection

FixaTons is a collection of publicly available datasets of human ﬁxations. The authors of
this paper have also included SIENA12, which is a further set of grayscale images that aim
at being as more free of semantic content as possible.

2.1

SIENA12

The SIENA12 dataset includes 12 grayscale images1 . It was collected by the author of this
paper in collaboration with EVALAB at Policlinico Alle Scotte in Siena. The images have
been selected so as to minimize the semantic content of the scenes. Images include natural
scenes, human constructions, but also abstract contents.
2.1.1

Protocol for data collection in SIENA12

Visual data are collected through a 240 Hz eye-tracking system (ASL 504, Applied Science
Laboratories, Bedford, MA, USA) allowing for a remote tracking of the point of gaze on a
calibrated surface (LCD screen, 1024 × 768 px, 31 × 51 cm). A chin-rest is used to maintain
constant the relative distances between the eyes of the subject, the eye-tracker optics and
the screen (eye/eye-trackers distance, 68 cm; eye/screen distance, 72 cm). The height of
the chin-rest is set in order to get the line of sight of the subjects at the rest position
1

The authors thank Danilo Pileri for kindly providing images of the dataset.

3

Dataset Name
Number of images
Size
Categories
Number of observers
Age of the observers
Task
Duration
Eye-tracker
Screen
Eye-screen distance
Other information

SIENA12
12
1024x768 px
Outdoor, natural, synthetic
23
From 23 to 52
Free-viewing
5 seconds
ASL 504 (240 Hz)
LCD 1024 × 768 px (31 × 51 cm)
72 cm
Grayscale images

Table 1: Tech. spec. of the dataset SIENA12

perpendicular to the centre of the screen. The stimuli presentation and the data collection
is managed by customized software. All recordings are conducted in complete darkness,
measuring one eye.
A nine-point calibration is performed trough an interactive user interface provided by the
manufacturer. The operator instructs the participant to perform 5 seconds tasks of free
visual exploration. A total of 12 images is presented. Between one image and another,
a central dot (of about 2 seconds) is displayed. The order of the images composing the
sequence of the experimental stimuli is randomly chosen to prevent bias.
Raw data has been processed with the publicly available python library PyGazeAnalyser [4]
in order to extract information about ﬁxations.

4

Figure 1: Images from SIENA12

2.2

Other Datasets

Authors of compatible datasets are encouraged to add them to this collection. Please,
send an email to the corresponding author of this report along with an authorization to
redistribute that data. Below, the list of datasets currently added to the collection.
2.2.1

MIT1003

The dataset MIT1003 [5] is a large database of eye tracking data. The images, eye tracking
data, and accompanying code in Matlab are all available on the web. This dataset can be
used as training data for the MIT300 [6] benchmark since they share the same technical
speciﬁcation. MIT300 is not included in FixaTons collection because its data is kept private
for a fair evaluation of the benchmark. More details about the protocol used for data
collection are given in the referred paper.

5

Dataset Name
Number of images
Size
Categories
Number of observers
Age of the observers
Task
Duration
Eye-tracker
Screen
Eye-screen distance
Other information

MIT1003
1003
Min. dim.: 405 px – Max. dim.: 1024 px
Landscape and portrait images
15
From 18 to 35
Free-viewing
3 seconds
ETL 400 ISCAN (240Hz)
LCD 1024 × 768 px (40.5 × 30 cm)
75 cm
It can be used as training data for MIT300 benchmark

Table 2: Tech. spec. of the dataset MIT1003

3

Online resources
• Webpage of the project: http://sailab.diism.unisi.it/fixatons/
• Data download: https://drive.google.com/open?id=1TQSaq5J0p_oCdkyVZ-IzBltLwJ2cm3UA
• Software library https://github.com/dariozanca/FixaTons.

6

4

Structure of the FixaTons collection
• FixaTons
– DATASET NAME
∗ STIMULI : contains original images. They can have diﬀerent ﬁle format
(jpg, jpeg, png,...)
∗ SCANPATHS : contains one folder for each image
· IMAGE ID : it contains one ﬁle for each scanpath of that image scanpaths are matrices rows of this matrices describe ﬁxations each ﬁxation
is of the form : [x-pixel, y-pixel, initial time, ﬁnal time]. Times are in
seconds.
∗ FIXATION MAPS : contains a ﬁxation map of each original image they are
matrices of zeros (non-ﬁxated pixels) and ones (ﬁxated pixels). They can
have diﬀerent ﬁle format (jpg, jpeg, png,...)
∗ SALIENCY MAPS : contains saliency maps of each original image they are
generated from human data. They can have diﬀerent ﬁle format (jpg, jpeg,
png,...)

5

Software included

Some software tools are provided together with the collection for an easy use and visualization of the data.
Software is written in python. All the functions are included in the ﬁle FixaTons.py.
They make use of the public library OpenCV which should be installed on the machine
before the use of FixaTons.py.
Functions can be divided in ﬁve main categories:
• List information
• Get data (matrices)
• Visualize data
• Compute metrics.
• Compute statistics

5.1

List information

The collection comprehend diﬀerent datasets, each of them with diﬀerent stimuli names,
number of subjects„ subjects id’s, etc. The provided software allows to easily get this kind
of information.

7

• FixaTons.list.dataset(): This functions returns a list with the names of the datasets
included in the collection.
• FixaTons.list.stimuli(DATASET NAME): This functions lists the names of the
stimuli of a speciﬁed dataset.
• FixaTons.list.subjects(DATASET NAME, STIMULUS NAME): This functions lists the id’s of the subjects which have been watching a speciﬁed stimuli of a
dataset.

5.2

Get data (matrices)

Diﬀerent functions allows to get data in form of numpy array.
• FixaTons.get.stimulus(DATASET NAME, STIMULUS NAME): This functions returns the matrix of pixels of a speciﬁed stimulus. Notice that, both
DATASET NAME and STIMULUS NAME need to be speciﬁed. The latter, must
include ﬁle extension. The returned matrix could be 2- or 3-dimensional.
• FixaTons.get.fixation map(DATASET NAME, STIMULUS NAME): This
functions returns the matrix of pixels of the ﬁxation map of a speciﬁed stimulus.
Notice that, both DATASET NAME and STIMULUS NAME need to be speciﬁed.
The latter, must include ﬁle extension. The returned matrix is a 2-dimensional matrix
with 1 on ﬁxated locations and 0 elsewhere.
• FixaTons.get.saliency map(DATASET NAME, STIMULUS NAME): This
functions returns the matrix of pixels of the saliency map of a speciﬁed stimulus.
Saliency map has been obtained by convolving the ﬁxation map with a proper gaussian
ﬁlter (corresponding to one degree of visual angle). Notice that, both
DATASET NAME and STIMULUS NAME need to be speciﬁed. The latter, must
include ﬁle extension. The returned matrix is a 2-dimensional matrix.
• FixaTons.get.scanpath(DATASET NAME, STIMULUS NAME, subject =
0): This functions returns the matrix of ﬁxations of a speciﬁed stimulus. The scanpath
matrix contains a row for each ﬁxation. Each row is of the type [x, y, initial t,
final time]. By default, one random scanpath is chosen between available subjects. For
a speciﬁc subject, it is possible to specify its id on the additional argument subject=id.

5.3

Visualize data

For an easy visualization of the data, some functions have been included in the library.
• FixaTons.show.map(DATASET NAME, STIMULUS NAME, showSalMap
= True, showFixMap = False, plotMaxDim = 0): This functions uses cv2 standard library to visualize a speciﬁed stimulus. By default, stimulus is shown with its
8

saliency map aside. It is possible to deactivate such option by setting the additional
argument showSalMap=False. It is possible to show also (or alternatively) the ﬁxation map by setting the additional argument showFixMap=True. Depending on the
monitor or the image dimensions, it could be convenient to resize the images before to plot them. In such a case, user could indicate in the additional argument
plotMaxDim=500 to set, for example, the maximum dimension to 500. By default,
images are not resized.
• FixaTons.show.scanpath(DATASET NAME, STIMULUS NAME, subject
= 0, animation = False, putNumbers = True, plotMaxDim = 0): This functions uses cv2 standard library to visualize the scanpath of a speciﬁed stimulus. By
default, one random scanpath is chosen between available subjects. For a speciﬁc subject, it is possible to specify its id on the additional argument subject=id. It is possible
to visualize it as an animation by setting the additional argument animation=True.
Depending on the monitor or the image dimensions, it could be convenient to resize
the images before to plot them. In such a case, user could indicate in the additional
argument plotMaxDim=500 to set, for example, the maximum dimension to 500. By
default, images are not resized.

5.4

Compute metrics

An implementation of the most common metrics to compute saliency maps similarity and
scanpaths similarity is included in the software provided with FixaTons. A mathematical
description of the metrics of scanpath similarity is postponed to the Appendix.
• Saliency Map similarities
– FixaTons.metrics.KLdiv(saliencyMap1, saliencyMap2): This function computes the KullbackâĂŞLeibler divergence between two continuous saliency maps.
– FixaTons.metrics.AUC Judd(saliencyMap, fixationMap, jitter = True,
toPlot = False) Given a continuous saliency map (normally the output of a
saliency model) and a ﬁxation map (matrix with 1’s at ﬁxated locations, 0’s
elsewhere), it computes the Area Under the ROC curve, in the implementation
described by Judd in [5].
– FixaTons.metrics.NSS(saliencyMap, fixationMap) Given a continuous saliency
map (normally the output of a saliency model) and a ﬁxation map (matrix with
1’s at ﬁxated locations, 0’s elsewhere), it computes the Normalized Scanpath
Saliency
• Scanpaths similarities
– FixaTons.metrics.euclidean distance(human scanpath, simulated scanpath):
This function computes the euclidean distance between two scanpaths. More details are given on the Appendix.
9

– FixaTons.metrics.string edit distance(stimulus, human scanpath, simulated scanpath, n = 5, substitution cost=1): This function computes the
string-edit distance between two scanpaths. More details are given on the Appendix.
– FixaTons.metrics.time delay embedding distance( human scanpath, simulated scanpath, k = 3, distance mode = ’Mean’): This function computes the time-delay embedding distance between two scanpaths. More details
are given on the Appendix.
– FixaTons.metrics.scaled time delay embedding distance( human scanpath,
simulated scanpath, image, toPlot = False)): This function computes the
scaled time-delay embedding distance between two scanpaths. More details are
given on the Appendix.

5.5

Compute statistics

It is possible to compute statistics for the overall collection, or for a speciﬁc dataset, about
some scanpath properties.
• FixaTons.stats.statistics(DATASET NAME=None): This functions returns a
list with two values: ﬁxations per second and the average of saccades length. If no
dataset is speciﬁed, statistics are calculated on the whole FixaTons collection. To
restrict computation on a speciﬁc dataset, it is suﬃcient to indicate its name on the
additional argument DATASET NAME.

5.6

Example of use

Here we propose a python script which show a complete example of use of some facilities.
1
2

# import t h e l i b r a r y
import FixaTons

3
4
5

#s h u f f l e ( d a t a s e t l i s t )
f o r d a t a s e t i n FixaTons . l i s t . d a t a s e t s ( ) :

6
7
8

# For a l l images i n t h a t d a t a s e t
f o r image i n FixaTons . l i s t . s t i m u l i ( d a t a s e t ) :

9
10
11

# Show t h e image a s i d e i t s s a l i e n c y map ( 5 s e c o n d s dy d e f a u l t ) .
FixaTons . show . map( d a t a s e t , image , plotMaxDim =1500)

12
13
14

# Then , f o r a l l t h e s u b j e c t s t h a t watched t h a t image ,
f o r s u b j e c t i n FixaTons . l i s t . s u b j e c t s ( d a t a s e t , image ) :

15
16
17
18
19

# Show t h e c o r r e s p o n d e n t s c a n p a t h a s an a n i m a t i o n .
# ( Look , time o f e x p l o r a t i o n i n t h e a n i m a t i o n i s t h e
# e x a c t time , from t h e d a t a s e t . )
FixaTons . show . s c a n p a t h ( d a t a s e t , image , s u b j e c t ,

10

a n i m a t i o n=True ,
plotMaxDim =1000 ,
w a i t t i m e =1000)

20
21
22

Listing 1: Example of use. Complete Python script.

6

Appendix A:

In this section we deﬁne metrics to compute similarity between scanpaths. Metrics of course
work for given two sequences of point in the R2 plane, but for the sake of clearness, we
will contextualize to our topic. Let say our goal is to compute a distance between a human
scanpath Hpath and a simulated scanpath Spath
Hpath = Hpath (1), ..., Hpath (n)

(1)

Spath = Spath (1), ..., Spath (m)

(2)

where Hpath (i) ∈ R2 , ∀i ∈ {1, ..., n} and Spath (j) ∈ R2 , ∀j ∈ {1, ..., m}.

6.1

Euclidean distance

We deﬁne the Euclidean distance between two scanpath as the sum of the Euclidean distances between corresponding ﬁxations. To give a mathematical formulation
dEuclidean =

n
X

||Hpath (i) − Spath (i)||2

(3)

i=0

This metric presents some important drawbacks.
i The two scanpaths has to have the same length, i.e. n must be equal to m. But
human scanpaths in the available collection of data rarely have same length, and to
restrict to the minimum length would produce a huge loss of important data.
ii Euclidean metric does not take time-shifts into account. Let us indicate ﬁxation
locations (x, y) ∈ R2 with capital letters: the scanpaths ABCDEF and ZABCDE,
even if very similar, would be very distant.

6.2

String-edit similarity

One of the most common metrics to compute scanpaths similarity is the ”string-edit” or
Levenshtein distance. The algorithm of the dynamic program to compute this metric is
described in [7]. The metric has been used in diﬀerent works for comparing diﬀerent human
scanpaths [9, 8]. It has been shown [10] to be robust to changes in the number of regions
used.
Input stimulus is divided into nxn regions, labeled with characters. Scanpaths are
turned into strings by associating the corresponding character to each ﬁxation. Finally, the
algorithm of string-edit is used to measure the distance between the two generated strings.
11

6.3

Time-delay embedding distance

In [11] the authors describe a metric to compute similarity between scanpaths, which is
inspired and largely used in dynamic systems analysis. Time-delay embedding are used in
order to quantitatively compare the stochastic and dynamic scanpaths of varied lengths.
In general, human scanpath (Hpath ) and simulated scanpath (Spath ) can have diﬀerent
lenghts.
Hpath = Hpath (1), ..., Hpath (n)

(4)

Spath = Spath (1), ..., Spath (m)

(5)

They are ordered list of ﬁxations. Each ﬁxation is deﬁned by a couple of spatial coordinates (that correspond to the ﬁxated location in the image).
For Hpath , we deﬁne
k
(t) = (Hpath (t), ..., Hpath (t + k))
CH

(6)

which is the k-dimensional sub-sequence of ﬁxation extracted from Hpath , starting from
the t − th ﬁxation. We indicate with
k
H (k) = {CH
(t)}t∈(1,...,n−k) ⊂ Rk

(7)

the set of all this possible k-dimensional sub-sequences. Analogously we deﬁne
CSk (τ ) = (Spath (τ ), ..., Spath (τ + k))

(8)

S (k) = {CSk (τ )}τ ∈(1,...,m−k) ⊂ Rk

(9)

and

the set of all possible k-dimensional sub-sequences extracted from Spaht . For each model
generated sub-sequence x ∈ S (k) , we look for the closest human sub-sequence of ﬁxations,
and we consider the distance i.e.
1
min{|x − CSk (τ )|}
(10)
k τ
Once we compute all this distances, we can have a measure of similarity between the
two scanpaths, for example, by considering the mean of all this values, i.e.
dk (x, H (k) ) =

h

i

k
(t), H (k) ) .
T DE(S (k) , H (k) ) = Et dk (CH

(11)

where TDE stands for Time-delay Embedding metric. A further option can be to consider the Hausdorﬀ-distance, i.e.
h

i

k
T DEHaus (S (k) , H (k) ) = max dk (CH
(t), H (k) ) .
t

12

(12)

6.4

Scaled Time-delay embedding distance

The metric described in the previous section is based on time-delay embedding. This allows
capturing some dynamical as well as stochastic of the phenomenon.
The result of that metric is given in pixel, since no re-scaling is applied anywhere. This
is very strictly related on the properties of the dataset and makes results not very clear.
Also consider the case of dataset with diﬀerent size of the images: this would make results
unfair.
For this reason, we deﬁne a scaled version of that metric, that take in account size of
the input. Also, we would like to make it a similarity measure, with values in (0, 1], where
1 expresses perfect overlap of the two scanpaths.
We will try to preserve formalism of the previous section.
Consider a sequence of human ﬁxations over an image I, e.g.
Hpath = Hpath (1), ..., Hpath (n)

(13)

and a model simulated sequence of ﬁxation over the same image
Spath = Spath (1), ..., Spath (m).

(14)

Each of this ﬁxations is a pair of coordinates belonging to the rectangle [0, l1 ] × [0, l2 ],
where l1 and l2 are, respectively, the horizontal and vertical dimensions of I, e.g.
Hpath (1), ..., Hpath (n) = (Hx1 , Hy1 ) , ..., (Hxn , Hyn )

(15)

Spath (1), ..., Spath (n) = (Sx1 , Sy1 ) , ..., (Sxm , Sym )

(16)

First, we re-scale the image I and consequently the coordinates of the ﬁxations, dividing
by the maximum of the dimensions L = max{l1 , l2 }.
Hxn Hyn
Hx1 Hy1
,
,
, ...,
L
L
L
L




Sxm Sym
Sx1 Sy1
,
,
, ...,
=
L
L
L
L

H̄path =
S̄path









(17)
(18)

At this point, we compute the mean TDE for all possible k’s in K = {1, ..., max{n, m}}
h

i

Ek T DE(H̄ (k) , S̄ (k) ) ∈ R+

(19)

This quantity is zero only in the case the seb-sequences of H̄path of any lenght are
contained in S̄path . So we deﬁne an index of similarity of the form
ST DE(Hpath , Spath ) = e−Ek [T DE(H̄

]

(k) ,S̄ (k) )

(20)

where now, STDE stands for ”Scaled Time-Delay Embedding metric”. Please notice
that
13

ST DE(Hpath , Spath ) ∈ (0, 1]

(21)

where now 1 indicates ”perfect similarity”, and the more close to zero the more diﬀerent
scanpaths are.

References
[1] Lee, T. S., Stella, X. Y.: An information-theoretic framework for understanding saccadic
eye movements. Advances in neural information processing systems, pp 834–840. (2000)
[2] Renninger, L. W., Coughlan, J. M., Verghese, P., Malik, J.: An information maximization model of eye movements. Advances in neural information processing systems, pp
1121–1128. (2005)
[3] Zanca, D., Gori, M.: Variational Laws of Visual Attention for Dynamic Scenes. Advances
in Neural Information Processing Systems, pp 3826-âĂŞ3835 (2017)
[4] Dalmaijer, E.S., MathÃťt, S., Van der Stigchel, S.: PyGaze: an open-source, crossplatform toolbox for minimal-eﬀort programming of eye tracking experiments. Behavior
Research Methods, vol 46, pp 913–921 (2014)
[5] Judd, T., Ehinger, K., Durand, F., Torralba, A.: Learning to Predict Where Humans
Look. IEEE International Conference on Computer Vision (2009)
[6] Judd, T., Durand, F., Torralba, A.: A Benchmark of Computational Models of Saliency
to Predict Human Fixations. MIT Technical Report. (2012)
[7] Jurafsky, D., Martin, J. H.: Speech and Language Processing. London: Pearson. (2014)
[8] Brandt, S. A., Stark, L. W.: Spontaneous eye movements during visual imagery reﬂect
the content of the visual scene. Journal of cognitive neuroscience, vol 9, n 1, pp 27–38.
(1997)
[9] Foulsham, T., Underwood, G.: What can saliency models predict about eye movements?
Spatial and sequential aspects of ﬁxations during encoding and recognition. Journal of
Vision, vol = 8, n = 2, pp 6. (2008)
[10] Choi, Y. S., Mosley, A. D., Stark, L. W.: String editing analysis of human visual search.
Optometry and Vision Science, vol 72, pp 439–451. (1995)
[11] Wang, W., Chen, C., Wang, Y., Jiang, T., Fang, F., Yao, Y.: Simulating human saccadic scanpaths on natural images. Computer Vision and Pattern Recognition (CVPR),
pp 441–448. (2011)

14

