arXiv:1803.05796v1 [stat.ML] 15 Mar 2018

Deep architectures for learning
context-dependent ranking functions
Karlson Pfannschmidt

Pritha Gupta

Eyke Hüllermeier

Department of Computer Science
Warburger Str. 100
Paderborn University, Germany
kiudee@mail.upb.de

Department of Computer Science
Warburger Str. 100
Paderborn University, Germany
prithag@mail.upb.de

Department of Computer Science
Warburger Str. 100
Paderborn University, Germany
eyke@upb.de

ABSTRACT
Object ranking is an important problem in the realm of preference learning. On the basis of training data in the form of a set
of rankings of objects, which are typically represented as feature
vectors, the goal is to learn a ranking function that predicts a linear
order of any new set of objects. Current approaches commonly
focus on ranking by scoring, i.e., on learning an underlying latent
utility function that seeks to capture the inherent utility of each
object. These approaches, however, are not able to take possible effects of context-dependence into account, where context-dependence
means that the utility or usefulness of an object may also depend
on what other objects are available as alternatives. In this paper, we
formalize the problem of context-dependent ranking and present
two general approaches based on two natural representations of
context-dependent ranking functions. Both approaches are instantiated by means of appropriate neural network architectures. We
demonstrate empirically that our methods outperform traditional
approaches on benchmark tasks, for which context-dependence is
playing a relevant role.

CCS CONCEPTS
•Computing methodologies −→ Ranking; Learning to rank;
Neural networks; Batch learning;

KEYWORDS
Learning to rank, object ranking, context-dependence, preference
learning, neural networks

1

INTRODUCTION

Preference learning has received increasing attention in machine
learning in recent years [15]. Roughly speaking, the goal in preference learning is to induce preference models from observed data
that reveals information about the preferences of an individual or a
group of individuals in a direct or indirect way; these models are
then used to predict the preferences in a new situation.
In general, a preference learning system is provided with a set
of items (e.g., products) for which preferences are known, and
the task is to learn a function that predicts preferences for a new
set of items (e.g., new products not seen so far), or for the same
set of items in a different situation (e.g., the same products but
for a different user). Frequently, the predicted preference relation
is required to form a total order, in which case we also speak of
a ranking problem. In fact, among the problems in the realm of
preference learning, the task of “learning to rank” has probably
received the most attention in the literature so far, and a number

of different ranking problems have already been introduced. Based
on the type of training data provided and the type of predictions
required, Fürnkranz and Hüllermeier [15] distinguish between the
problems of object ranking [9, 26], label ranking [8, 18, 43], and
instance ranking [16].
The focus of this paper is on object ranking. Given training data
in the form of a set of exemplary rankings of subsets of objects, the
goal in object ranking is to learn a ranking function that is able to
predict the ranking of any new set of objects. As a typical example,
consider an eCommerce scenario, in which a customer is ranking
a set of products, each characterized by different properties and
attributes, according to her preferences.
Classical choice theory assumes that, for a given user, each alternative has an inherent utility, and that choices and decisions are
made on the basis of these utilities. Yet, many studies have shown
that these idealized assumptions are often violated in practice. For
example, choices are also influenced by the decision context, i.e., by
the availability of other alternatives [12, 20, 38, 42]; we will discuss
these effects in more detail later on.
Motivated by observations of that kind, the focus of this paper
is on the problem of context-dependent ranking. In this regard, our
contributions are as follows. First, we formalize the problem of
context-dependent ranking and present two general approaches
based on two natural representations of context-dependent ranking functions: First Evaluate Then Aggregate (FETA) and First
Aggregate Then Evaluate (FATE). Second, both approaches are instantiated by means of appropriate neural network architectures,
called FETA-Net and FATE-Net, respectively. These architectures
can be trained in an end-to-end manner. Third, we conduct an
experimental evaluation of our methods, using both synthetic and
real-world data, for which context-dependence is playing a relevant
role. Empirically, we are able to show that our methods outperform
traditional approaches on these tasks.

2

OBJECT RANKING

We assume a reference set of objects denoted by X, where each
object x ∈ X is described by a feature vector; thus, an object is
a vector x = (x 1 , . . . , xd ) ∈ Rd , and X ⊆ Rd . A ranking task is
specified by a finite subset Q = {x 1 , . . . , x n } ⊆ X of objects, for
some n ∈ N, and the task itself consists of predicting a preferential
ordering of these objects, that is, a ranking. The latter is encoded
in terms of a permutation π ∈ Sn , where Sn denotes the set of all
permutations of length n, i.e., all mappings [n] −→ [n] (symmetric

group of order n). A permutation π represents the total order

C

A

B

ality

(a) Compromise

Price

Price

where π (k) is the position of the kth object x k , and π −1 (k) the index
of the object on position k (π is often called a ranking and π −1 an
ordering). Formally, a ranking function can thus be understood as a
mapping
ρ : Q −→ R ,

C

Price

B

x π −1 (1)  x π −1 (2)  . . .  x π −1 (n) ,

C

B

A
A

ality

(b) Attraction

ality

(c) Similarity

Figure 1: Context effects identified in the literature [35].
Ð
where Q = 2 X \ ∅ is the ranking task space and R = n ∈[N ] Sn
the ranking space.
Methods for object ranking seek to induce a ranking function
N in the form of exemplary ranking
from training data {(Q i , πi )}i=1
tasks Q i together with observed rankings πi . Typically, this is done
by learning a latent utility function
U : X −→ R ,

The compromise effect states that the relative utility of an object
increases by adding an extreme option that makes it a compromise
in the set of alternatives [35]. For instance, consider the set of
objects {A, B} in Figure 1a. The ordering of these objects depends
on how much the consumer is weighing the quality and the price
of the product. If price is the constraint, then the preference order
will be A  B. But as soon as there is another extreme option C
available, the object B becomes a compromise option between the
three alternatives. The preference relation between A and B gets
inverted and turns into B  A.
Figure 1b illustrates the attraction effect. Here, if we add another
object C to the set of objects {A, B}, where C is slightly dominated
by B, the relative utility share for object B increases with respect to
A. The major psychological reason is that consumers have a strong
preference for dominating products [21]. Thus, the preference
relation between A and B may again be influenced.
The similarity or substitution effect is another phenomenon,
according to which the presence of similar objects tends to reduce
the overall probability of an object to be chosen, as it will divide
the loyalty of potential consumers [21]. In Figure 1c, B and C are
two similar objects. Consumers who prefer high quality will be
divided amongst the two objects, resulting in a decrease of the
relative utility share of object B. Again, this may lead to turning a
preference B  A into A  B, at least on an aggregate (population)
level, if preferences are defined on the basis of choice probabilities.
To capture effects of context-dependence, our goal is to learn a
generalized latent utility function

(1)

which assigns a real-valued score to each object x ∈ X. Given a
task Q, a ranking is then simply constructed by sorting the objects
x ∈ Q according to their scores.
Representing a ranking function ρ in terms of a latent utility
function (2) implies important properties of the induced preferences,
i.e., the set of rankings {ρ(Q) | Q ∈ Q} produced by ρ on the ranking
task space. In particular, preferences have to be transitive and,
moreover, context-independent.
Context-independence means that the preference between two
items x and y does not depend on the set of other items Q \ {x, y} in
the query (which we consider as defining the context in which x and
y are ranked). More formally, consider any pair of items x, y ∈ X
and any subsets Q, Q 0 ⊆ X such that x, y ∈ Q ∩ Q 0 . Moreover,
let  be the ranking induced by ρ on Q and  0 the corresponding
ranking on Q 0 . Then, context-independence implies that x  y
if an only if x  0 y. Obviously, context-independence is closely
connected to Luce’s axiom of choice [28].
Let us note that the notion of “context” is also used with a different meaning in the learning-to-rank literature (and in machine
learning in general), namely as a kind of extra dimension. For instance, Agrawal et al. [1] illustrate their notion of “context-sensitive
ranking” with an example in which objects are actors and the extra dimension is the film genre: “Contextual preferences take the
form that item i 1 is preferred to item i 2 in the context of X . For
example, a preference might state the choice for Nicole Kidman
over Penelope Cruz in drama movies, whereas another preference
might choose Penelope Cruz over Nicole Kidman in the context
of Spanish dramas.” Obviously, this differs from our definition of
“context”, which is derived from its use in the economics literature
[42].

3

U : X × 2 X −→ R ,

(2)

which can be used in the same way to assign a score to each object of
the ranking task. Since the utility function has a second argument,
namely a context, it allows for representing context-dependent
ranking functions


ρ {x 1 , . . . , x n } = arg sort U x i , Ci ,
i ∈[n]

where, for each object x i in a task Q = {x 1 , . . . , x n }, we denote by
Ci = C(x i ) = Q \ {x i } = {x 1 , . . . , x i−1 , x i+1 , . . . , x n } its context
in this task.
In this section, we present two general approaches based on two
natural representations of context-dependent ranking functions.
These representations are based on two rather natural ways to decompose the problem of assigning a context-dependent score to an
object: “First evaluate then aggregate” (FETA) first evaluates the object in each “sub-context” of a fixed size, and then aggregates these
evaluations, whereas “first aggregate then evaluate” (FATE) first

CONTEXT-DEPENDENT RANKING

In practice, the assumption of context-independence of preferences
is often violated. This has especially been shown in economics and
marketing studies [11], which found that preferences of individuals
are indeed often influenced by the context in which decisions are
made [3]. Three major context effects have been identified in the
literature: the compromise effect [37], the attraction effect [21], and
the similarity effect [41].
2

aggregates the entire set of alternatives into a single representative,
and then evaluates the object in the context of that representative.
Interestingly, the former approach has already been used in the
literature [44], at least implicitly, while the latter is novel to the
best of our knowledge. Before explaining these approaches in more
detail, we make a few more general remarks on the representation
of context-dependent ranking functions.

3.1

different ranking functions can be produced, because the entire
function is determined by the order on the maximal ranking task
Q = X. Naturally, the number of possible ranking functions should
increase with increasing K. For the extreme case K = N − 1, we
can indeed show that all ranking functions can be generated.
Proposition 3.2. Let X be a set of N objects and Q = 2 X \ ∅ be
the corresponding ranking task space. Let ρ : Q −→ R be a ranking
Ð
function mapping from queries to rankings with R = n ∈[N ] Sn .
There always exist preference functions

Modeling Context-Dependent Utility

The representation of a context-dependent utility function (3) comes
with (at least) two important challenges, which are both connected
to the fact that the second argument of such a function is a set of
variable size. First, the arity of the function is therefore not fixed,
because different ranking tasks, and hence different contexts, can
have different size. Second, the function should be permutationinvariant (symmetric) with regard to the elements in the second
argument, the context, because the order in which the alternative
objects are presented does not play any role.

Uk : X × X k −→ R
such that the corresponding ranking rule of order N − 1
ρ FETA (C) = arg sort U (x i , C)
i ∈[N ]

= ρ(C)
for all ranking tasks C ∈ Q.
Proof. First, notice that ρ FETA (C) = ρ(C) = π if and only if
U (π (1), C) > U (π (2), C) > · · · > U (π (n), C) for a given ranking
task C of size n. Thus ρ FETA (C) = ρ(C) for all C ∈ Q if and only if
all resulting inequalities defined on the scores are satisfied.
The result can be shown by induction over the size K ∈ N of
the maximum ranking task size (i.e. K = maxC ∈Q |C |) for which
ρ is defined. In addition denote with Q (k ) the ranking task space
of ranking tasks with |C | ≤ k for C ∈ Q (k ) . For the base case of
K = 1 the corresponding rankings are all of size 1, which is why
ρ FETA (C) = ρ(C) trivially holds for all C ∈ Q.
For K = 2 we have ranking functions defined on pairs of objects.
Here it suffices to use a preference function U1 with contexts of size
2. Note, that for one fixed ranking task {x i , x j } each preference
score U2 (x i , {x i , x j }) only appears in one inequality (i.e. the one
in which U2 (x i , {x i , x j }) and U2 (x j , {x i , x j }) are compared). It
follows that we can set U2 (x i , {x i , x j }) = Jx i ρ x j K for all pairs
i, j ∈ [N ] and U1 (x i , ·) = 0 for all i ∈ [N ].
For the inductive step assume that for all k ≤ K the equality
ρ FETA (C) = ρ(C) holds for all C ∈ Q (k) . Now we consider the
step K −→ K + 1. We know that preference scores U (x i , C) with
|C | = K + 1 only occur for inequalities defined for rankings of size
K + 1 and further appear only in one inequality since we evalutate
it only once for each object x i .
This time it is not possible to independently set these preference
scores as we did before, since we have to take into account all the
summands in (3.1). We already know by induction hypothesis that
there exists a preference function U for any ρ defined on Q (K )
such that ρ FETA (X ) = ρ(X ) for all X ∈ Q K . For any ranking task
X ∈ Q (K ) with |X | = K let

Definition 3.1. A function f : X k −→ R is permutation-invariant
if and only if


f (x 1 , x 2 , . . . , x k ) = f x π (1) , x π (2) , . . . , x π (k )
for all permutations π of the indices [k] = {1, . . . , k}. A function
with this property is also called symmetric [39].
As for the problem of rating objects in contexts of variable size,
one possibility is to decompose a context into sub-contexts of a
fixed size k. More specifically, the idea is to learn context-dependent
utility functions of the form
Uk : X × X k −→ R ,
and to represent the original function (3) as an aggregation
U (x, C) =
=

K
Õ
k=1
K
Õ
k=1

Ūk (x, C)
1
|C | 
k

Õ

Uk (x, C 0 ) .

(3)

C 0 ⊆C, |C 0 |=k

Note that, provided permutation-invariance holds for Uk as well
as the aggregation, U itself will also be symmetric. Taking the
arithmetic average as an aggregation function, the second condition
is obviously satisfied. Thus, the problem that essentially remains is
to guarantee the symmetry of Uk .
Roughly speaking, the idea of the above decomposition is that
dependencies and interaction effects between objects only occur
up to a certain order K, or at least can be limited to this order
without loosing too much information. This is an assumption that
is commonly made in the literature on aggregation functions [17]
and also in other types of applications. The special cases k = 0
and k = 1 correspond to independence and pairwise interaction,
respectively.
An interesting question concerns the expressivity of a Kth order approximation (3.1), where, for example, expressivity could be
measured in terms of the number of different ranking functions
that can be defined on X. To study this question, suppose that X
is finite and consists of N objects. Obviously, for K = 0, only N !

δ max =

max

x i ,x j ∈X

U (x i , X ) − U (x j , X )

be the maximum score difference using only the existing preference
scores. We will use δ max now as a step size to define the preference
scores for rankings of size K + 1.
Then set the preference scores UK +1 (·, X ) for |X | = K + 1 as
follows:
UK +1 (x i , X ) = (K + 1 − π −1 (i)) · (δ max + ε)
3

where π −1 = ρ −1 (X ∪ {x i }) and ε > 0. In other words, we simply
set the scores inversely proportional to the position of the object in
their respective ranking. To guarantee, that the preference scores
which were defined on |X | < K + 1 do not have an effect, we
additionally scale by the step size δ max + ε.
It follows that for any x i , x j and any C with |C | = K + 1:

Roughly speaking, U1 (x, y) is a measure of how favorable it is
for x that y is part of its context C. In other words, a large value
U1 (x, y) suggests that, whenever x and y are part of the objects to
be ranked, x tends to occupy a high position.

3.3

U (x i , C) − U (x j , C)

To deal with the problem of contexts of variable size, our previous
approach was to decompose the context into sub-contexts of a
fixed size, evaluate an object x in each of the sub-contexts, and
then aggregate these evaluations into an overall assessment. An
alternative to this “first evaluate then aggregate” strategy, and in
a sense contrariwise approach, consists of first aggregating the
context into a representation of fixed size, and then evaluating the
object x in this “super-context”.
More specifically, consider a ranking task Q. To evaluate an
object x in the context C(x) = Q \ {x }, the “first aggregate then
evaluate” (FATE) strategy first computes a representative for the
context:
Õ
1
µC(x ) =
ϕ(y) ,
(4)
|C(x)|

≤ UK +1 (x i , C) − UK +1 (x j , C) + δ max
= (δ max + ε)(π −1 (j) − π −1 (i)) + δ max
= δ max (π −1 (j) − π −1 (i) + 1) + ε(π −1 (j) − π −1 (i))
Since δ max > 0 this equation is > 0 if and only if π −1 (j) > π −1 (i)
and therefore if x i  x j and < 0 otherwise.
We can conclude that we can obtain all possible rankings of size
K + 1 using this construction. Thus, the statement follows.


3.2

First Evaluate Then Aggregate

Our first approach realizes (3.1) for the special case K = 1, which can
be seen as a first-order approximation of a fully context-dependent
ranking function. Thus, we propose the representation of a ranking
function ρ which, in addition to a utility function U0 : X −→ [0, 1],
is based on a pairwise predicate U1 : X × X −→ [0, 1]. Given a
ranking task Q = {x 1 , . . . , x n } ⊆ X, a ranking is obtained as
follows:

y ∈C(x )

where ϕ : X −→ Z maps each object y to an m-dimensional embedding space Z ⊆ Rm . The evaluation itself is then realized by a
context-dependent utility function U : X × Z −→ R, so that we
eventually obtain a ranking

ρ FATE (Q) = arg sort U x, µC(x ) .

ρ FETA (Q) = arg sort U (x i , Ci )

x ∈Q

i ∈[n]


= arg sort U0 (x i ) +
i ∈[n]

1
n−1

Õ

U1 (x i , x j )

A computationally more efficient variant of this approach is obtained by including an object x in its own context, i.e., by setting
C(x) = Q for all x ∈ Q. In this case, the aggregation (3.3) only
needs to be computed once.
Note, that this approach bears resemblance to the recent work
by Zaheer et al. [46] on dealing with set-valued inputs and the
general approach proposed by Ravanbakhsh et al. [33] on encoding
equivariance with respect to group operations.



j ∈[n]\{i }

We refer to this approach as First Evaluate Then Aggregate (FETA).
The observation that FETA is able to capture context-dependence
is quite obvious. As a simple illustration, suppose that U0 ≡ 0 and
U1 is given on X = {a, b, c, d} as follows:
−
©
­ 0.2
U1 = ­
­ 0.5
« 0.7

0.7
−
0.2
0.1

First Aggregate Then Evaluate

0.5
0.8
−
0.5

0.1
0.9
0.4
−

ª
®
®
®
¬

4

NEURAL ARCHITECTURES

In this section, we propose realizations of the FETA and FATE approaches in terms of neural network architectures FETA-Net and
FATE-Net, respectively. Our design goals for both neural networks
are twofold: First, they should be end-to-end trainable on any differentiable listwise ranking loss function. Second, the architectures
should be able to generalize beyond the ranking task sizes encountered in the training data, since in practice it is unreasonable to
expect all rankings to be of similar size. Our focus in this paper will
be on optimizing the 0/1-ranking loss, for which we will introduce
a suitable surrogate loss, though a substitution with any other loss
function is in principle possible.

For the queries Q 1 = {a, b, c} and Q 2 = {a, b, d} we obtain rankings
ρ FETA (Q 1 ) = a  b  c and ρ FETA (Q 2 ) = b  a  d. That is, the
preference between a and b changes depending on whether the
third item to be ranked is c or d.
It is important to note that our interpretation of U1 is not the
standard interpretation in terms of a pairwise preference relation.
Specific properties such as asymmetry (U1 (x, y) = 1 − U1 (y, x)) are
therefore not necessarily required, although they could be incorporated for the purpose of regularization. Instead, U1 (x, y) should be
interpreted more generally as a measure of support given by y to x.
This interpretation is somewhat motivated by Ragain and Ugander
[32], who model distributions on rankings using Markov chains.
Here, individual preferences are defined in terms of probabilities
(of the stationary distribution), and binary relations U1 (x, y) define
transition probabilities. Thus, U1 (x, y) is the probability of moving
from y to x, and the larger the probability of being in x, the higher
the preference for this item.

4.1

FETA-Net Architecture

FETA as outlined above requires the binary predicate U1 to be given.
In FETA-Net, learning this predicate is accomplished by means of
a deep neural network architecture. More specifically, we make
use of the CmpNN architecture proposed by Rigutini et al. [34]. In
our FETA-Net architecture (shown in Figure 2), we evaluate the
CmpNN network on all pairs (x i , x j ) of objects in the ranking task Q
4


The training complexity of FATE-Net is O Ndq 2 , where N
denotes the number of rankings, d is the number of features per
object, and q is an upper bound on the number of objects in each
ranking. For a new query Q (note that we can predict the ranking
for any query size) the prediction can be done in O(d |Q |) time (i.e.,
linear in the number of objects). This is because, if objects are part
of their context, the representative µ Q has to be computed only
once for the forward pass. This makes the FATE-Net architecture
more efficient to use than FETA-Net.

0-order Network
s1
s2

⋮
xi

⋮

Σ

CmpNN

xj

+

si

⋮
⋮

sn

⋮
Objects

⋮

R(·,X )
(1)

R(·,∅)

Scores

Figure 2: The FETA-Net architecture implementing the
FETA decomposition.

4.3

Loss Functions

A key advantage of the above architectures is that they are fully
differentiable, allowing us to use any differentiable loss function `.
In our case, a loss is supposed to compare a ground-truth ranking π
for a task Q = {x 1 , . . . , x n } with a vector s = (s 1 , . . . , sn ) of scores
predicted for the objects in Q. Thus, the loss is of the form `(π , s).
Unfortunately, many interesting ranking losses, such as the (normalized) 0/1-ranking loss

Õ
2
1
dRL (π , s) =
Jsi < s j K + Jsi = s j K
n(n − 1)
2
(i, j):π (i)<π (j)

or the popular nDCG, are not differentiable. Yet, just like in the
binary classification setting, we can define a surrogate loss function
that upper bounds the true binary ranking loss, is differentiable
almost everywhere, and ideally even convex. We propose to use
the hinge ranking loss:
Õ

2
` HL (π , s) =
max 1 − si + s j , 0
n(n − 1)

Figure 3: The FATE-Net architecture implementing the
FATE decomposition. Here we show the score head for object x i .
and build up a pairwise relation R = (r i, j ), where r i, j = U1 (x i , x j ).
Using the notation of Rigutini et al. [34], this relation is defined as
follows:
(
N + ([x i , x j ]) if i < j
r i, j =
N − ([x i , x j ]) otherwise

(i, j):
π (i)<π (j)

It is convex and has a constant subgradient with respect to the
individual scores. Another choice for a differentiable loss function
is the Plackett-Luce (PL) loss:
Õ

n−1
n


Õ
`PL (π , s) =
log
s π −1 (j) − log s π −1 (i) ,

This step is highlighted in blue in Figure 2. Then, each row of the
relation R is summed up to obtain a score Ū1 (x, Q) for each object
x i ∈ Q. Each x i is also passed through a 0th-order network that
directly outputs latent utilities U0 (x i , Q). Here, we use a densely
connected, deep neural network with one output unit. The final
score for object x i is then given by U (x i , Q) = U0 (x i , Q) + Ū1 (x, Q).
The training complexity of FETA-Net is O Ndq 2 , where N
denotes the number of rankings, d is the number of features per
object, and q is an upper bound on the number of objects in each
ranking. For a new ranking task Q (note that we can predict
 the
ranking for any task size) the prediction time is in O d |Q | 2 .

4.2

i=1

j=i

which corresponds to the negative logarithm of the PL-probability
to observe π given parameters s. The networks can then be trained
by gradient descent and backpropagating the loss through the network.

5

EMPIRICAL EVALUATION

In order to empirically evaluate our FATE-Net and FETA-Net
architectures, we make use of synthetic and real-world data. We
mainly address the following questions:
• Are the architectures suitable for learning context-dependent
ranking functions, and how do the approaches FETA and
FATE compare with each other?
• Can the representation learned on one query size be generalized to arbitrary sizes of ranking tasks?
For typical real-world datasets like OHSUMED and LETOR [31], it
is difficult to ascertain how much context-dependence is present in
the data. For the evaluation of context-sensitive ranking models, we
therefore propose two new challenging benchmark problems: the
medoid and the hypervolume problem. These synthetic problems
are inspired by real-world problems and require context-sensitive

FATE-Net Architecture

The FATE-Net architecture is depicted in Figure 3. Inputs are the
n objects of the ranking task Q = {x 1 , . . . , x n } (shown in green).
Each object is independently passed through a deep, densely connected embedding layer (shown in blue). The embedding layer
approximates the function ϕ in (3.3), where, for reasons of computational efficiency, we assume objects to be part of their context
(i.e., C(x) = Q for all x ∈ Q). Note that we employ weight sharing,
i.e., the same embedding is used for each object. Then, the representative µC(x ) = µ Q for the context is computed by averaging the
representations of each object. To calculate the score U (x, Q) for
an object x i , the feature vector is concatenated with µ Q to form
the input to the joint hidden layer (here depicted in orange).
5

5.2

models for successful learning. Besides, we also analyze a realworld dataset related to the problem of depth estimation in images.
As baselines to compare with, we selected representative algorithms for three important classes of ranking methods: Expected
rank regression (ERR) [25] as a representative for so-called pointwise ranking algorithms, RankSVM [24] as a state-of-the-art pairwise ranking model, and a deep version of RankNet [5, 6] that
represents the family of deep latent utility models.
All experiments are implemented in Python, and the code is publicly available1 . The hyperparameters of each algorithm were tuned
with the library scikit-optimize [19] using nested cross-validation.
For all neural network models, we make use of the following techniques:
• We use either ReLU non-linearities + batch normalization
[23] or SELU non-linearities [27] for each hidden layer.
• Regularization: Both L 1 and L 2 penalties are applied.
• Optimizer: Stochastic gradient descent with Nesterov momentum [30].
We evaluate the algorithms in terms of their Spearman rank
correlation, the 0/1-ranking accuracy, and the 0/1-accuracy:
Í
6 ni=0 (π (i) − τs (i))2
,
dSpear (π , τs ) = 1 −
n(n2 − 1)
dRA (π , s) = 1 − dRL (π , s) ,

(1) Generate points for the ranking tasks uniformly on the
negative surface of the unit sphere Q i = {x i,1 , . . . , x i,n } ⊆
{x ∈ Rd | kx k = 1, ∀1 ≤ i ≤ d : x i ≤ 0}.
(2) Construct the corresponding rankings π1 , . . . , π N as follows: For all 1 ≤ i ≤ N ,
(a) compute the contributions ∆ j of each object x i, j to
the hypervolume2 of the ranking task Q i , i.e.,

dACC (π , τs ) = Jπ = τs K ,

where τs is the ranking induced by the predicted score vector s.

5.1

The Hypervolume Problem

In multi-objective optimization, the goal is to find the set of objects
that are non-dominated by any other object in terms of their fitness
(solution quality). The set of all non-dominated objects is called
the Pareto-set. Multi-objective evolutionary algorithms (MOEAs)
approximate the Pareto-set by iteratively improving a population
of objects. During optimization, it is not only important to improve
the population’s fitness, but also to preserve its diversity. This
allows the population to cover the complete Pareto-front.
The hypervolume is a set measure, which computes the volume
dominated by a given set of objects. This very naturally encodes
both dominance as well as diversity, which makes it a popular fitness criterion [2]. Usually, we are also interested in the contribution
of each object/point on the Pareto-front to the hypervolume. Bringmann and Friedrich [4] proved that computing exact hypervolume
contributions is #P-hard and NP-hard to approximate.
Our idea is to convert this problem into a challenging, contextdependent ranking problem. The input for the learners are the
sets of points on the Pareto-front, and the target is a ranking of
these points based on their contribution to the hypervolume. Data
generations is done as follows:

The Medoid Problem

∆i, j = hyp(Q i ) − hyp(Q i \ {x i,j }) ,

Our first synthetic problem is called the medoid problem. The goal
of the algorithms is to sort a set of randomly generated points based
on their distance to the medoid. This problem is inspired by the
setting of similarity learning, where the goal is to learn a similarity
function from triplets of objects [45]. Specifically, we generate
D = {(Q 1 , π1 ), (Q 2 , π2 ), . . . , (Q N , π N )} as follows:
(1) Generate data points Q i = {x i,1 , . . . , x i,n } ⊆ [0, 1]d for
the ranking tasks uniformly at random.
(2) Construct the corresponding rankings π1 , . . . , π N as follows: For all 1 ≤ i ≤ N ,
(a) compute the medoid
n
1Õ
x i∗ = arg min
kx − x i, j k ,
x ∈Q i n j=1

(b) and then the ranking
πi = arg sort {−∆i, j | x i, j ∈ Q i } .
j ∈[n]

It is apparent that a learner given only a set of data points as input,
needs to take all of the points into account to establish an accurate
ranking.
Similar to the Medoid dataset, we generate 300 000 sets of 5 random points and determine the rankings as described. The instances
are split into 1/3 training (100 000) and 2/3 test data (200 000). We
repeat this 5 times to get an estimate of the variation.

5.3

(b) compute the ranking

Image Region Depth Estimation

As a suitable real-world case study, we tackle the problem of relative
depth estimation of regions in monocular images. The dataset
Make3D by Saxena et al. [36] consists of 534 photos with an original
image resolution of 1704×2272. Ewerth et al. [13] use it to construct
an object ranking dataset: The images are segmented into 3355 (i.e.,
61 × 55) super pixels and different feature sets are extracted for each
super pixel. We use the feature set Basic, where basic monocular
depth clues are available for each super pixel: linear perspective,
atmospheric perspective, texture gradients, occlusion of objects,
usual size of objects, relative height, relative size, distribution of
light. These depth clues suggest that context-dependence could be
a relevant issue in depth estimation. The ground truth rankings are

πi = arg sort −kx i∗ − x i, j k .
j ∈[n]

The rankings produced by this procedure take the distance to each
point in the ranking task into account. Thus the medoid and subsequently the resulting ranking are sensitive to changes of the points
in the ranking task. This makes the problem suitable for evaluating
the different approaches.
For the experiment, we generate 20 000 sets of 5 random points
and determine the rankings as described above. The instances are
split into 50 % training and 50 % test data. This is repeated 10 times
to get an estimate of the variation across datasets.
1 https://github.com/kiudee/cs-ranking

2 We

6

use the PyGMO library to compute exact contributions.

Since the representative µ Q in (3.3) is computed as the average of
the object embeddings, this behavior is to be expected when the
ranking function behaves similarly for different ranking task sizes.
The results for the relative depth estimation problem are shown
in Table 3. We additionally report the results obtained by Ewerth
et al. [13] using Rankboost on the same dataset (using the same
split into training and test). Both our architectures achieve comparably high Spearman correlation and ranking accuracy, and slightly
outperform the competitors.

constructed by ordering the super pixels based on their absolute
depth.
Since the size of the rankings is too large for most of the approaches, we sample subrankings of size 172 for training. FETANet in addition also samples several subrankings of size 5 during
the training process. Predictions are always the complete rankings
of size 3355, which are compared with the ground truth rankings
using 0/1-ranking accuracy. Super pixels with a distance of more
than 80 m are treated as tied, because this exceeds the range of the
sensor.

6
5.4

RELATED WORK

Results and Discussion
In this section, we provide a brief overview of other approaches to
object ranking based on neural networks.
Comparison training refers to a framework introduced by Tesauro
[40] for learning from pairwise preferences with a neural network.
The network architecture consists of two subnetworks, which are
connected to a single output node that indicates which of the two
inputs is preferred over the other. The weights associated with
the last hidden layer of one subnetwork is the mirrored version
of the other subnetwork’s weights. Two principal problems are
addressed with this setup, namely efficiency and consistency. In
the evaluation phase, only a single subnetwork is used to evaluate
n alternatives, which are then sorted according to their scores. The
network essentially implements a (latent) utility function (thereby
enforcing transitivity of the predicted preferences).
In the domain of information retrieval, RankNet by Burges et al.
[6] uses pairwise inputs to learn a utility function with the cross
entropy loss. To this end, the training data consists of sample pairs
together with target probabilities. These quantify the probability
to rank the first sample higher than the second. With the introduction of target probabilities, this approach enables the interesting
possibility of modeling ties between samples.
A similar approach called SortNet is proposed by Rigutini et al.
[34]. They introduce a three-layered network architecture called
CmpNN, which takes two input object vectors and outputs two real
values. The architecture establishes a preference function, for which
the properties of reflexivity and symmetry (but not transitivity)
are ensured by a weight-sharing technique. To improve the performance of the network on the complete ranking, the authors employ
an active learning technique, adapting the training and validation
set incrementally. Huybrechts [22] uses the CmpNN architecture
(with 3 hidden layers) in the context of document retrieval.
In contrast to the above approaches, which are pairwise in the
sense of minimizing losses on pairwise preferences, listwise approaches seek to minimize loss functions on complete rankings.
Cao et al. propose the ListNet method, where the assumption is
that the rankings are generated by a probability distribution over
permutations [7, 29]. Volkovs and Zemel [44] introduce the algorithm BoltzRank, which learns a combination of pairwise and
individual scoring functions, thus falling under our category of
FETA approaches.

ZeroOneRankAccuracy

0.95

0.90

0.85

0.80

5

10

15
Number of Objects

20

Figure 4: 0/1-Ranking accuracy of FATE-Net trained on
rankings of size 5, then predicting on ranking tasks of a different size.

The results on the Medoid and the Hypervolume dataset are
shown in Table 1 and Table 2, respectively. ERR and RankSVM
completely fail on both tasks, both having a correlation of 0 with
the target rankings. This can be explained by the fact that both
approaches ultimately learn a linear model, while the problems
are highly non-linear. Being a non-linear latent-utility approach,
RankNet is able to improve upon random guessing and achieves
a 0/1-ranking accuracy of around 68 %. This result is surprising,
considering that RankNet establishes the final ranking by scoring
each point independently, i.e., not taking the other points of the
ranking task into account.
Our architectures FETA-Net and FATE-Net are both able to
make use of the given context provided by the ranking task and
beat the context-insensitive approaches by a wide margin. With a
0/1-ranking accuracy of more than 90 %, FATE-Net even performs
significantly better than FETA-Net. This suggests that the pairwise
decomposition (first-order approximation) is not able to completely
capture the higher order interactions between the objects.
Given that FATE-Net outperforms the other approaches, we
are now wondering whether it is also able to generalize to unseen
ranking task sizes. To this end, we train the network on the Medoid
dataset with 5 objects as ranking task size, and then test it on
ranking task sizes between 3 and 24. The results are shown in
Figure 4. Increasing the size of the ranking task apparently leads to
a lower ranking accuracy, but the deterioration eventually stops.

7

CONCLUSIONS AND FUTURE WORK

In this paper, we addressed the novel problem of learning contextdependent ranking functions in the setting of object ranking and,
moreover, proposed two general solutions to this problem. These
7

Table 1: Mean and standard deviation of the losses on the medoid dataset. Measured across the 10 outer cross validation folds.
The best entry for each loss is marked in bold.
Ranker

Spearman correlation

0/1-Ranking accuracy

0/1-Accuracy

ERR
RankSVM
RankNet

−0.001 ± 0.002
0.000 ± 0.002
0.417 ± 0.001

0.500 ± 0.001
0.500 ± 0.001
0.682 ± 0.001

0.008 ± 0.001
0.008 ± 0.001
0.088 ± 0.001

FETA-Net
FATE-Net

0.594 ± 0.032
0.861 ± 0.007

0.759 ± 0.014
0.901 ± 0.004

0.133 ± 0.014
0.443 ± 0.016

Table 2: Mean and standard deviation of the losses on the hypervolume dataset. Measured across the 5 outer cross validation
folds. The best entry for each loss is marked in bold.
Ranker

Spearman correlation

0/1-Ranking accuracy

0/1-Accuracy

ERR
RankSVM
RankNet

0.001 ± 0.002
−0.001 ± 0.002
0.419 ± 0.001

0.500 ± 0.001
0.500 ± 0.001
0.683 ± 0.001

0.008 ± 0.001
0.008 ± 0.000
0.089 ± 0.001

FETA-Net
FATE-Net

0.682 ± 0.005
0.894 ± 0.004

0.802 ± 0.002
0.920 ± 0.003

0.192 ± 0.004
0.508 ± 0.013

Table 3: Results on the test set for the relative depth dataset with Basic feature set.
Ranker

Spearman correlation

0/1-Ranking accuracy

ERR
RankSVM
RankNet
RankBoost [13, 14]

0.659
0.656
0.633

0.792
0.778
0.770
0.806

FETA-Net
FATE-Net

0.729
0.721

0.813
0.814

solutions are based on two principled ways for representing contextdependent ranking functions that accept ranking problems of any
size as input and guarantee symmetry. FETA (first evaluate then
aggregate) is a first-order approximation to a more general latentutility decomposition, which we proved to be flexible enough to
learn any ranking function in the limit. FATE (first aggregate then
evaluate) first transforms each object into an embedding space and
computes a representative of the context by averaging. Objects are
then scored with this representative as a fixed-size context.
To enable end-to-end optimization of differentiable ranking losses
using these decompositions, we further contribute two new neural network architectures called FETA-Net and FATE-Net. We
demonstrate empirically that both architectures are able to learn
context-dependent ranking functions on both synthetic and realworld data.
We already show in Figure 4 that our FATE-Net architecture
is able to generalize across different ranking task sizes. However,
training on different sizes of ranking tasks is not as straightforward.
We either have to resort to subsampling the rankings to a common
size or to alternate training on different sizes as a kind of meta
gradient descent. Even then it is not clear, which ranking functions
a model trained in this fashion will eventually be able to learn.

This is a future direction of theoretical research that we consider
important and very fruitful.

ACKNOWLEDGMENTS
This work is part of the Collaborative Research Center “On-the-Fly
Computing” at Paderborn University, which is supported by the
German Research Foundation (DFG). Calculations leading to the
results presented here were performed on resources provided by
the Paderborn Center for Parallel Computing.

REFERENCES
[1] Rakesh Agrawal, Ralf Rantzau, and Evimaria Terzi. 2006. Context-sensitive ranking. In Proc. SIGMOD’06, ACM SIGMOD International Conference on Management
of Data. Chicago, USA, 383–394.
[2] Johannes M. Bader. 2010. Hypervolume-Based Search for Multiobjective Optimization: Theory and Methods. CreateSpace, Paramount, CA.
[3] James R Bettman, Mary Frances Luce, and John W Payne. 1998. Constructive
consumer choice processes. Journal of consumer research 25, 3 (1998), 187–217.
[4] Karl Bringmann and Tobias Friedrich. 2012. Approximating the least hypervolume contributor: NP-hard in general, but fast in practice. Theor. Comput. Sci. 425
(2012), 104–116. https://doi.org/10.1016/j.tcs.2010.09.026
[5] Christopher J. C. Burges. 2010. From RankNet to LambdaRank to LambdaMART:
An Overview. (2010).
[6] Christopher J. C. Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds,
Nicole Hamilton, and Gregory N. Hullender. 2005. Learning to rank using gradient descent. In Machine Learning, Proceedings of the Twenty-Second International
8

[7]

[8]

[9]
[10]

[11]
[12]

[13]

[14]

[15]
[16]

[17]
[18]

[19]

[20]

[21]

[22]
[23]

[24]

[25]

Conference (ICML 2005), Bonn, Germany, August 7-11, 2005 (ACM International
Conference Proceeding Series), Luc De Raedt and Stefan Wrobel (Eds.), Vol. 119.
ACM, 89–96. https://doi.org/10.1145/1102351.1102363
Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007. Learning
to rank: from pairwise approach to listwise approach. In Machine Learning,
Proceedings of the Twenty-Fourth International Conference (ICML 2007), Corvallis,
Oregon, USA, June 20-24, 2007 (ACM International Conference Proceeding Series),
Zoubin Ghahramani (Ed.), Vol. 227. ACM, 129–136. https://doi.org/10.1145/
1273496.1273513
Weiwei Cheng, Jens C. Huhn, and Eyke Hüllermeier. 2009. Decision tree and
instance-based learning for label ranking, See [10], 161–168. https://doi.org/10.
1145/1553374.1553395
W.W. Cohen, R.E. Schapire, and Y. Singer. 1999. Learning to Order Things. Journal
of Artificial Intelligence Research 10, 1 (1999), 243–270.
Andrea Pohoreckyj Danyluk, Léon Bottou, and Michael L. Littman (Eds.). 2009.
Proceedings of the 26th Annual International Conference on Machine Learning,
ICML 2009, Montreal, Quebec, Canada, June 14-18, 2009. ACM International
Conference Proceeding Series, Vol. 382. ACM.
Gerard Debreu. 1960. Review of Individual Choice Behavior by R. Luce. American
Economic Review 50 (1960), 186–8.
Ravi Dhar, Stephen M Nowlis, and Steven J Sherman. 2000. Trying hard or hardly
trying: An analysis of context effects in choice. Journal of Consumer Psychology
9, 4 (2000), 189–200.
Ralph Ewerth, Matthias Springstein, Eric Müller, Alexander Balz, Jan Gehlhaar,
Tolga Naziyok, Krzysztof Dembczyński, and Eyke Hüllermeier. 2017. Estimating
Relative Depth in Single Images Via Rankboost. In IEEE International Conference
on Multimedia and Expo, ICME 2017, Hong Kong, July 10-14, 2017.
Yoav Freund, Raj D. Iyer, Robert E. Schapire, and Yoram Singer. 2003. An Efficient
Boosting Algorithm for Combining Preferences. Journal of Machine Learning
Research 4 (2003), 933–969. http://www.jmlr.org/papers/v4/freund03a.html
Johannes Fürnkranz and Eyke Hüllermeier (Eds.). 2010. Preference Learning.
Springer. https://doi.org/10.1007/978-3-642-14125-6
Johannes Fürnkranz, Eyke Hüllermeier, and Stijn Vanderlooy. 2009. Binary
Decomposition Methods for Multipartite Ranking. In Machine Learning and
Knowledge Discovery in Databases, European Conference, ECML PKDD 2009,
Bled, Slovenia, September 7-11, 2009, Proceedings, Part I (Lecture Notes in Computer Science), Wray L. Buntine, Marko Grobelnik, Dunja Mladenic, and John
Shawe-Taylor (Eds.), Vol. 5781. Springer, 359–374. https://doi.org/10.1007/
978-3-642-04180-8 41
M. Grabisch, J.L. Marichal, R. Mesiar, and E. Pap. 2009. Aggregation Functions.
Cambridge University Press.
Sariel Har-Peled, Dan Roth, and Dav Zimak. 2002. Constraint Classification:
A New Approach to Multiclass Classification. In Algorithmic Learning Theory,
13th International Conference, ALT 2002, Lübeck, Germany, November 24-26, 2002,
Proceedings (Lecture Notes in Computer Science), Nicolò Cesa-Bianchi, Masayuki
Numao, and Rüdiger Reischuk (Eds.), Vol. 2533. Springer, 365–379. https://doi.
org/10.1007/3-540-36169-3 29
Tim Head, MechCoder, Gilles Louppe, Iaroslav Shcherbatyi, fcharras, Zé Vinı́cius,
cmmalone, Christopher Schröder, nel215, Nuno Campos, Todd Young, Stefano
Cereda, Thomas Fan, Justus Schwabedal, Hvass-Labs, Mikhail Pak, SoManyUsernamesTaken, Fred Callaway, Loı̈c Estève, Lilian Besson, Peter M. Landwehr,
Pavel Komarov, Mehdi Cherti, Kejia (KJ) Shi, Karlson Pfannschmidt, Fabian
Linzberger, Christophe Cauet, Anna Gut, Andreas Mueller, and Alexander
Fabisch. 2018. scikit-optimize/scikit-optimize: High five - v0.5. (Feb. 2018).
https://doi.org/10.5281/zenodo.1165540
Joel Huber. 1982. Adding Asymmetrically Dominated Alternatives: Violations of
Regularity and the Similarity Hypothesis. The Journal of Consumer Research 9, 1
(1982), 90–98.
Joel Huber and Christopher Puto. 1983. Market boundaries and product choice:
Illustrating attraction and substitution effects. Journal of Consumer Research 10,
1 (1983), 31–44.
Georic Huybrechts. 2016. Learning to Rank with Deep Neural Networks. Master’s
thesis. Université catholique de Louvain.
Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating
Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of
the 32nd International Conference on Machine Learning, ICML 2015, Lille, France,
6-11 July 2015 (JMLR Workshop and Conference Proceedings), Francis R. Bach and
David M. Blei (Eds.), Vol. 37. JMLR.org, 448–456. http://jmlr.org/proceedings/
papers/v37/ioffe15.html
Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In
Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining, July 23-26, 2002, Edmonton, Alberta, Canada. ACM,
133–142. https://doi.org/10.1145/775047.775067
Toshihiro Kamishima, Hideto Kazawa, and Shotaro Akaho. 2005. Supervised
Ordering - An Empirical Survey. In Proceedings of the 5th IEEE International
Conference on Data Mining (ICDM 2005), 27-30 November 2005, Houston, Texas,
USA. IEEE Computer Society, 673–676. https://doi.org/10.1109/ICDM.2005.138

[26] T. Kamishima, H. Kazawa, and S. Akaho. 2010. A Survey and Empirical Comparison of Object Ranking Methods. In Preference Learning, J. Fürnkranz and
E. Hüllermeier (Eds.). Springer-Verlag, Berlin, Heidelberg, 181–202.
[27] Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter.
2017. Self-Normalizing Neural Networks. In Advances in Neural Information
Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus,
S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 972–981. http:
//papers.nips.cc/paper/6698-self-normalizing-neural-networks.pdf
[28] R.. Ducan Luce. 1959. Individual Choice Behavior a Theoretical Analysis. John
Wiley and sons.
[29] Tianyi Luo, Dong Wang, Rong Liu, and Yiqiao Pan. 2015. Stochastic Top-k
ListNet. In Proceedings of the 2015 Conference on Empirical Methods in Natural
Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, Lluı́s
Màrquez, Chris Callison-Burch, Jian Su, Daniele Pighin, and Yuval Marton (Eds.).
The Association for Computational Linguistics, 676–684. http://aclweb.org/
anthology/D/D15/D15-1079.pdf
[30] Yurii Nesterov. 1983. A method of solving a convex programming problem with
convergence rate O (1/k2). In Soviet Mathematics Doklady, Vol. 27. 372–376.
[31] Tao Qin and Tie-Yan Liu. 2013. Introducing LETOR 4.0 Datasets. CoRR
abs/1306.2597 (2013). arXiv:1306.2597 http://arxiv.org/abs/1306.2597
[32] Stephen Ragain and Johan Ugander. 2016. Pairwise Choice Markov Chains.
In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016,
Barcelona, Spain, Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (Eds.). 3198–3206. http://papers.nips.cc/
paper/6287-pairwise-choice-markov-chains
[33] Siamak Ravanbakhsh, Jeff Schneider, and Barnabás Póczos. 2017. Equivariance
Through Parameter-Sharing. In Proceedings of the 34th International Conference on
Machine Learning (Proceedings of Machine Learning Research), Doina Precup and
Yee Whye Teh (Eds.), Vol. 70. PMLR, International Convention Centre, Sydney,
Australia, 2892–2901. http://proceedings.mlr.press/v70/ravanbakhsh17a.html
[34] Leonardo Rigutini, Tiziano Papini, Marco Maggini, and Franco Scarselli. 2011.
SortNet: Learning to Rank by a Neural Preference Function. IEEE Trans. Neural
Networks 22, 9 (2011), 1368–1380. https://doi.org/10.1109/TNN.2011.2160875
[35] Robert P Rooderkerk, Harald J Van Heerde, and Tammo HA Bijmolt. 2011. Incorporating context effects into a choice model. Journal of Marketing Research 48, 4
(2011), 767–780.
[36] Ashutosh Saxena, Min Sun, and Andrew Y. Ng. 2009. Make3D: Learning 3D
Scene Structure from a Single Still Image. IEEE Trans. Pattern Anal. Mach. Intell.
31, 5 (2009), 824–840. https://doi.org/10.1109/TPAMI.2008.132
[37] Itamar Simonson. 1989. Choice based on reasons: The case of attraction and
compromise effects. Journal of consumer research 16, 2 (1989), 158–174.
[38] Itamar Simonson and Amos Tversky. 1992. Choice in Context: Tradeoff Contrast
and Extremeness Aversion. Journal of Marketing Research 29, 3 (1992), 281–295.
http://www.jstor.org/stable/3172740
[39] Richard P. Stanley. 2011. Enumerative Combinatorics: Volume 1 (2nd ed.). Cambridge University Press, New York, NY, USA.
[40] Gerald Tesauro. 1989. Connectionist Learning of Expert Preferences by
Comparison Training. In Advances in Neural Information Processing Systems 1,
D. S. Touretzky (Ed.). Morgan-Kaufmann, 99–106. http://papers.nips.cc/paper/
142-connectionist-learning-of-expert-preferences-by-comparison-training.
pdf
[41] Amos Tversky. 1972. Elimination by aspects: A theory of choice. Psychological
review 79, 4 (1972), 281.
[42] Amos Tversky and Itamar Simonson. 1993. Context-dependent preferences.
Management science 39, 10 (1993), 1179–1189.
[43] Shankar Vembu and Thomas Gärtner. 2010. Label Ranking Algorithms: A Survey.
See [15], 45–64. https://doi.org/10.1007/978-3-642-14125-6 3
[44] Maksims Volkovs and Richard S. Zemel. 2009. BoltzRank: learning to maximize
expected ranking gain, See [10], 1089–1096. https://doi.org/10.1145/1553374.
1553513
[45] Jiang Wang, Yang Song, Thomas Leung, Chuck Rosenberg, Jingbin Wang, James
Philbin, Bo Chen, and Ying Wu. 2014. Learning Fine-Grained Image Similarity
with Deep Ranking. In CVPR. IEEE Computer Society, 1386–1393.
[46] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R
Salakhutdinov, and Alexander J Smola. 2017. Deep Sets. In Advances in Neural
Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach,
R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 3393–
3403. http://papers.nips.cc/paper/6931-deep-sets.pdf

9

