Reinforcement Learning for Fair Dynamic Pricing
Roberto Maestre Juan Duque Alberto Rubio Juan Arévalo





















































Reinforcement Learning for Fair Dynamic Pricing
    Roberto Maestre, Juan Duque, Alberto Rubio, Juan Arevalo
BBVA Data and Analytics

Av. de Burgos, 16D, 28036 
 Madrid, Spain 

{roberto.maestre, juanramon.duque, alberto.rubio.munoz, juanmaria.arevalo}@bbvadata.com
    December 30, 2023
=========================================================================================================================================================================================================================






Unfair pricing policies have been shown to be one of the most negative  perceptions customers can have concerning pricing, and may result in long-term losses for a company. Despite the fact that dynamic pricing models help companies maximize revenue,  fairness and equality should be taken into account in order to avoid unfair price differences between groups of customers. This paper shows how to solve dynamic pricing by using  Reinforcement Learning (RL) techniques so that prices are maximized while keeping a balance between revenue and fairness. We demonstrate that RL provides two main features to support fairness in dynamic pricing: on the one hand, RL is able to learn from recent experience, adapting the pricing policy to complex market environments; on the other hand, it provides a trade-off between short and long-term objectives, hence integrating fairness into the model's core. Considering these two features, we propose the application of RL for revenue optimization, with the additional integration of fairness as part of the learning procedure by using Jain's index as a metric. Results in a simulated environment show a significant improvement in fairness while at the same time maintaining optimisation of revenue.












Reinforcement Learning, Dynamic Pricing, Fairness, Jain's index

















§ INTRODUCTION


Determining the right price of a product or service for a particular customer is a necessary, yet complex endeavour; it requires knowledge of the customer's willingness to pay, estimation of future demands, ability to adjust strategies to competition pricing <cit.>, etc. Dynamic pricing <cit.> represents a promising solution for this challenge due to its intrinsic adjustment to  customer expectations. Indeed, with the advent and establishment of digital channels, unique opportunities for the application of dynamic pricing are arising, thus enhancing research in the field <cit.>. Equality is a critical aspect of dynamic pricing as it influences customers'  perceptions of fairness <cit.>. As long as perceptions of fairness are used by people as a heuristic for trust, inequality may lead to a destruction of that trust. For instance, large price differences between groups of customers, or a disagreement between the distribution of cost and profit are known to affect the relationship between customers and sellers <cit.>. Should this relationship be damaged, it could eventually generate substantial financial losses for a company in the medium- and long-term <cit.>. Although the  definition of fairness depends on the domain context and, consequently, has  a diffuse definition <cit.>, we propose clear design principles for fairness in dynamic pricing with a  focus on elements such as group or individual schemes (in which policy decisions are applied), and equity or definitions of equality. This clear design allows us to specify a well-defined metric and to include it in the overall model in order to ensure fairness.


Different approaches have been proposed to address the problem of maximizing revenue <cit.>. Among these is a promising technique consisting of optimizing pricing policies with Reinforcement Learning (RL) applied to different market scenarios (uni or multi-agent) <cit.>. Despite the extensive application of RL models to dynamic pricing, concepts such as equality and fairness are rarely incorporated into the learned policies. Moreover, the black–box nature of Machine Learning models <cit.> means that it is of utmost importance to ensure a good trade-off between revenue (goal for companies) and equality (principle of fairness and a goal for customers) <cit.>. There are also ethical issues in artificial intelligence such as algorithm bias, in which if the input data (or market environment itself) reflects unfair biases of the broader society  <cit.>, the output model can potentially  capture such biases perpetuating unfair policies. These biases can have a significant  impact on dynamic pricing outcomes, as the efficiency of policies (revenue) has to be balanced against the need to achieve equality treatment of costumers <cit.>. 

Many metrics have been proposed in order to measure fairness from the resource allocation distribution point of view <cit.>. Such a perspective  has been widely studied <cit.> in order to avoid unfair resource allocation for nodes in wireless networks. In the present paper we apply concepts from resource allocation distribution to dynamic pricing so as to  interpret fairness as similarly distributed prices among groups of customers (equality between groups of customers). Identification of different groups of customers is required for an effective price discrimination. In each group, certain assumptions about price sensitivity have to be made; groups could be defined, for instance, by customer's income level, gender, location, communication channel, etc. Our learning procedure provides homogeneous prices among such groups, but at the same time, takes into account price sensitives within each group to maximize revenue. As we will show, maintaining a balance of price distribution among groups of customers will boost fairness perception and thus trust, as the price for different offers is chosen by considering equality.




The present study is organized as follows. Section <ref> provides some useful definitions from the fairness design principles, the resource allocation distribution literature, and reviews the main concepts of RL. The experimental methodology is then presented in section <ref>, where synthetic groups of customers are introduced, and RL is applied to the dynamic pricing problem. Next, we show our experimental results in section <ref>. Finally, we draw some conclusions in section <ref>.



§ BACKGROUND

In this section we will start by first reviewing main concepts of Reinforcement Learning (RL), more specifically Q-Learning as variation of RL. We also define the approximation of Q-values by means of Neural Networks (NN) minimizing a loss function. Thereafter we will spell out the design principles for fairness. Finally, we will  introduce the main Jain's index properties, which will allow us to measure fairness in dynamic pricing policies.



 §.§ Reinforcement Learning concepts


Reinforcement Learning is an area of machine learning that studies how an agent takes actions in an environment to achieve a given goal <cit.>. RL differs from supervised learning as the agent learns by trial and error while interacting with the environment, as opposed to learning with labeled data. As such, RL is best suited for learning within sequential decision problems.


Q-learning is a variant of RL introduced by Watkins <cit.>, whereby the heuristics of the model are directly related to the rewards provided by the environment in each iteration. Q-Learning can be applied to dynamic pricing as each action will modify the state of the environment (related to fairness), thus providing a reward (the bid itself). Equation (<ref>) represents the generic way in which Q-learning is expressed:

    Q(s, a) ← (1-α)Q(s, a) + α( r' + γ_a^* Q(s',a^*) | s,a ).

Here, α is the learning rate, γ the discount factor and r' is the reward obtained after performing action a in state s. Thus, the goal of RL is to find a policy π_*: s → a that maximizes the expected discounted utility with some exploration methods. We use ϵ-greedy strategy that follows a random action with ϵ probability and action with 1-ϵ probability.


Q-Learning in its simplest form uses a table (Q-table) to represent state-action values, i.e. Q-Values. The problem, however, becomes intractable  as the number of states and actions increases <cit.>. In general terms, the Q-value function in equation (<ref>) can be estimated by function approximation, i.e. Q(s,a) ≈ Q(s,a;θ) <cit.>. 

Since NN act as universal function approximators <cit.>, they provide an excellent framework for Q-value estimation.
Indeed, recent advances in NN applied to Q-learning have shown them to greatly outperform traditional RL methods <cit.>. 

The process of learning Q-value estimations is as follows: at each iteration step i, approximated Q-values Q(s,a;θ_i) are trained by Stochastic Gradient Descent, by minimizing the loss L(θ_i):

    L(θ_i) =[ ( y - Q(s,a; θ_i) )^2 ],

where y = [  r' + γa^* Q(s',a^*; θ_i-1) | s,a ].


Reinforcement learning algorithms constitute a suitable method for learning  pricing policies, whenever the expected revenue for taking a pricing action is unknown in the absence of complete information of the environment <cit.>.



 §.§ Fairness Design Principles



We say that a policy in dynamic pricing is fair when the policy provides similarly distributed prices among groups of customers. From this definition, we can establish the next principles: A) equality, which drives homogeneous prices, and B) groups as atomic entities for applying prices, rather than individuals. Taken these two principles into account, we further propose to use Jain's Index <cit.> as fairness metric because: I) it works on multiple groups, II) it provides an indicator of how homogeneous is the policy as a clear indicator of how much the policy is fair in percentage terms (from 0 to 100% fair). Other metrics for measuring equality and fairness suited to be applied to dynamic pricing,  such as entropy, min-max ratio, etc.  are out of the scope of this paper. A complete and comprehensive study on several fairness metrics and their properties can be found in the reference list <cit.>.




 §.§ Metrics for Fairness: Jain's Index


Let 𝒞 be a portfolio of customers. Each customer c_i belongs to one group g_α∈𝒢, where 𝒢 is a collection of groups covering 𝒞. We use g̅_α to represent the average price allocated in group g_α. 

The Jain's index is defined as

    J = (∑_g∈𝒢g̅)^2/|𝒢|∑_g∈𝒢g̅^2.

It provides a measurement of how fair the average price allocation is in 𝒢 (here, |𝒢| is the number of groups in 𝒢). Jain's index poses two convenient characteristics when used to define the heuristics in a RL model: I) values are continuous between [0,1] and II) the index applies to any number of groups. First feature I) fits perfectly with our definitions of states in Section <ref> while the second II) is valuable when several groups of customers are defined. 



As we mentioned before, Jain's index offers an excellent way to know how fair our price allocation is among groups in 𝒢. For instance, if there are only two groups and either g̅_1 ≪g̅_2 or g̅_2 ≪g̅_1, Jain's index is 0.5, reflecting an unfair situation. Therefore, the model should learn to increase prices when possible in group g_1 and/or decrease the prices in g_2. This two-group case is represented in figure <ref>. As observed, values on the diagonal are 1.0, because both groups have the same average price, i.e.: g̅_1=g̅_2. Thus, the policy is fair with an homogeneous price between groups. However, when g̅_1=50, g̅_2=0 or g̅_1=0, g̅_2=50 the policy is fair only with the 50 % of the groups (heterogeneous case). Thus, by means of this metric, we can constrain the differences between groups and provide fairer prices. 





§ METHODOLOGY



We define four synthetic groups 𝒢 = {g_1,g_2,g_3,g_4} to experiment with fairness and revenue. We simulate customers c ∈𝒞 belonging to any group in 𝒢. Each customer responds to each price bid with the following logistic function ϕ∈ [0,1]:

    ϕ(a,g | a ∈𝒜, g ∈𝒢) = [1+e^-(b+w· a
    )]^-1
where a is the action chosen by the agent (the bid), and b_g and w_g are parameters defining the sensitivity of each group, see Table <ref>.




This simple environment provides a scenario in which four different  behaviours are simulated. Customers belonging to g_1 will accept much higher prices than customers in g_2. On the other hand, customers from g_3 exhibit an inverse behaviour: the larger the price, the higher the acceptance probability. This group can be seen as customers whose perception of the product quality increases with price. Finally, g_4 customers are not sensible to any change in the price. Figure <ref> represents the probability of acceptance of a given price for each group (agent action or bid value).



In order to achieve both the maximum revenue and fairness, the model should learn to increase the price within each group (maximizing the revenue), while reducing the differences between groups when possible, so that fairness is also maximized. Please note that maximizing revenue entails finding the maximum price for each group in which the probability of acceptance is the highest <cit.>.

The statistics in the experiments are computed by running 350 epochs. In each epoch 1000 bids are simulated with a 100 customers randomly selected from 𝒞. In order to achieve a good balance between exploration and exploitation, for every epoch t we decrease ϵ according to the next ratio ϵ = 1/e^t/20 (starting by ϵ = 1).

The following subsections define the main elements of RL proposed in the present paper for a fair dynamic pricing. To this end, we define a state that allows us to identify both the customer group, g ∈𝒢, and the average allocated price, g̅; this way, both revenue and fairness can be maximized. As will be shown, revenue maximization is included in the reward through the value of each bid (see equation (<ref>)). 




 §.§ States


We define the state as 𝒮 = { s_c, s_f}, where s_c represents the customer group, and s_f represents the global fairness. We propose to code s_c and s_f as independent one-hot vectors. For the latter, s_f, we define a partition ℐ of mesh p in the interval [0,1], ℐ≡ 0< p<2p< ... < 1 - p< 1, so that there is a k such that ℐ_k≤ s_f≤ℐ_k+1 ∀ s_f. Given this partition, we define the one-hot encoding of s_f as a vector with a 1 at position k, and zeros otherwise. Finally, 𝒮 is a vector resulting from the concatenation of s_c and s_f one-hot encoding, and has the final dimension n=|𝒢|+1/p+1. As an example, if we define a partition with mesh p=1/3, one state with customer c_α∈ g_2 and a global Jain's index with value 0.89, the vector  𝒮 will be represented as follows:

    𝒮_α =
        cccccccc4c𝒢   3cJ
    4c3.0cm0pt   3c3.0cm0pt
    
          g_1     g_2     g_3     g_4     [0,1/3)     [1/3,2/3)     [2/3, 1]   
    ↓   ↓   ↓   ↓   ↓   ↓   ↓
    [cccccccc]
          0   1   0   0   0   0   1





 §.§ Actions

We define the action space, 𝒜, as a partition of mesh q in the interval [min, max], i.e. 

    𝒜≡min < min+q< min+2q< ...< max-q< max.

The action space has dimension m=1/q+1. Please note that as long as 𝒜 is constrained between {min, max} prices, the model will never bid excessively unfair prices –i.e., a non-sense price. 

Extensive experimentation show that best values of p,q for discretizing the state and action vectors lie at around 0.01. With this selection we provide a good balance between expansion and shrinkage over the state/action space. On the one hand, a fine grid mesh (large p and q)  make certain over–the–space defined functions (such as customer sensitivity in equation (<ref>)) more expressive. On the other hand, small discretizations negatively affect learning convergences. Thus, the selection of parameters p and q is critical.



 §.§ Q-value approximation

In accordance with our definitions of state and action, we propose a simple lineal approximation of the policy:

    T_0(w_0,b_0):    ℝ^n →ℝ^m 
        s → a =  w_0 · X+b_0,

where w_0∈ℝ^m× n and b_0∈ℝ^m are learnable parameters. 
Thus, we are approximating Q-values with a linear, one-hidden layer network. Depending on the problem complexity, the NN can be expanded with more layers and different activation functions. With this compact method only two vectors (weights and bias for every layer) are needed to represent all Q-values.



 §.§ Reward

Reward definition in dynamic pricing modeling with RL is generally challenging, as it represents the heuristics of the model itself. In the present paper, a multi-objective heuristic is required to account for both revenue and fairness optimization. Thus, hyper-parameters are needed to balance and be established as specific goals for each objective. For instance, one goal might be learning a policy with a fairness value equal to 0.85 while maximizing the bidding price, or viceversa. Bearing this in mind, we define the reward as:

    r = β_p exp(- ( p -  p_t )^2σ_p) +β_f exp(- ( f -  f_t )^2 σ_f).

The first term in equation (<ref>) stands for price optimization, while the second intends to balance it with fairness. 
Here β_p, σ_p, p_t and  β_f, σ_f, f_t are hyper-parameters that balance and establish a specific goal for p and f, respectively. Variable p ∈ℝ [0,1] relates to the price given in the i^th bidding, with p_t the target price. Similarly, f ∈ℝ [0,1] is the fairness index, and f_t the desired (target) fairness. In order to model the reward, we have chosen a normal distribution of prices and fairness as it exhibits  two convenient properties <cit.>: it points in the direction of the target price p_t and target fairness f_t; it also provides a uniform decay for states distant from the main goal. The following figure <ref> shows several examples of hyper-parameter choices. NB: The reward is scaled between [0,1].




In a real environment, customers will accept or reject each bid, i.e, the bid will be the value itself or 0. Thus, we define p as follows:

    p =
    {[           ν(a) ℬ(ϕ (a,g)) = 0;              a  ℬ(ϕ(a,g)) = 1 ].

Here, ℬ is the binomial distribution, and the function ϕ(a,g | a ∈𝒜, g ∈𝒢) is given by equation (<ref>). The ν(a) function represents the penalization when a bid is rejected. It could be modeled with a zero value, a constant negative value to penalize the rejection, or even a value proportional to the bid itself. However, in dynamic pricing it is important to keep the rejection rate low because the expected revenue of a policy is calculated as the ratio between the average bid value and the rejection ratio. Hence, it is convenient to take ν=constant. From an industrial perspective, we use a large penalization (ν = -0.5) in order to avoid customer rejection.

Next, we define the variable f, that accounts for fairness optimization in equation (<ref>). From a resource allocation point of view, given a fixed number of groups of customers (|𝒢| fixed), a large and homogeneous value of the resources (g̅ large) will increment Jain's index, and thus fairness. However, in the dynamic pricing context, increasing the average bid price will certainly not be perceived as fair. Therefore, we propose to distribute low and homogeneous prices among groups (g̅ low), in contrast to increasing prices where possible to maximize revenue. These ideas can be expressed mathematically as follows:

    f = ( ∑_g ∈ Gmax{𝒜} - g̅)^2 |𝒢| ∑_g ∈ G( max{𝒜} - g̅)^2∈ [0,1],

which can be interpreted as a rotated Jain's index after comparing figures <ref> and <ref>. Nevertheless, our transformation keeps the main properties of Jain's index invariant (i.e. population size, scale and metric independence, boundedness or continuity). Despite the simplicity of this approach, this rotation effectively allows for the balance between revenue and fairness.



Finally, we should point out that the selection of reward's hyper-parameters in equation (<ref>), along with the discretization of the state/action space, must be carred out with care as it dramatically impacts the learning process.






 §.§ Reinforcement Learning with NN



The following pseudo-code defines a standard way for training a NN to approximate Q-values. As illustrated in algorithm <ref>, the agent takes each bid a to evaluate the fairness using averages prices in 𝒢. The fairness value is calculated with the latest information given by 𝒢. With both the bid (p) and fairness information (f), the reward r is obtained. Finally, a gradient descent step is executed to reduce the loss produced between the current and the target Q-value, see equation (<ref>). In the experiments, we use the ADAM gradient descend algorithm <cit.> with a learning rate equal to 0.01.








§ RESULTS AND DISCUSSION


The evaluation of the agent's learning procedure is challenging, as the binomial distribution in equation (<ref>)  introduces statistical fluctuations that lead to changes in the states/actions distribution. In order to alleviate these instabilities, we use the average cumulative bid (F_p) and average fairness (f). Although the bid p can take negative values in equation (<ref>), we constrain p ∈ [0, +∞) for a better representation of the actual revenue. Regarding the reward r in equation (<ref>), we scale the cumulative reward for each experiment between [0,1]. This normalization is applied for a better comparison, because in each experiment different weights are used (β).


We conduct a number of experiments in order to understand the effect of the different competing objectives in equation (<ref>) (i.e., revenue and fairness).
Experiments 1 and 2 are designed to test both objectives separately (revenue and fairness). Experiments 3 and 4 to find a policy with a desired target fairness. Experiment 5 is proposed as a null model which has no learning mechanism, thus all actions are randomly selected (ϵ = 1.0) with zero reward value. The following table provides the main results by comparing the average cumulative bid, the averaged fairness, the reject ratio and the expected cumulative revenue for the previous defined experiments. The expected revenue r is calculated as result of the product between F_p and rejects.






 For every experiment, table <ref> shows the average bid allocated in each group. This table provides valuable information related to the fairness achieved for each policy. Related to the fairness principles design defined in section <ref>, we can observe that the more target fairness given by f_t the more homogeneous prices among groups are. Thus, the policy incorporate the proposed fairness principles definition.








	




Figures <ref>, <ref> and <ref> show the evolution of the average cumulative bid, fairness and reward, respectively.
In experiment 1 ­ the agent learns a policy that minimizes fairness average and maximizes bid average in contrast to experiment 2 in which fairness is maximized obtaining the minimum bid average. Note that the profit in experiment 1 is the highest because the algorithm learns to increase the bid value in groups g_3, g_4 and decrease the bid value in groups g_1 and g_2 (see table <ref> of g̅_α values). In this policy, the bid values are adjusted to minimize the rejection ratio, defined as the price acceptance probability,  see equation (<ref>) and figure <ref>).

Experiments 3 and 4 are proposed with the aim of achieving a target fairness f_t. Indeed, the averaged values map to the selected targets given by f_t (f_t=0.90, f=0.88 and f_t=0.75, f=0.76 for experiments 3 and 4, respectively). In experiment 3 the policy keeps the average price low on g_3 to reach a reasonable fairness (f=88%). Conversely in experiment 4 the calculated policy slightly increases the bid value in g_3 obtaining a less fair policy (f=76%) with more revenue. As a heavy penalization for bid rejections is used in the experiments, given by ν = -0.5, the bid rejected ratio remains low. 

An interesting result arises from the comparison between fairness in experiments 2 and 5 (null model). The average prices in every group are ∼ 5.5 in the null model because they are randomly and uniformly selected from 𝒜. However, due to our definition of the rotated Jain's index in equation  (<ref>), the average fairness in 2 is slightly higher than that in  5.



Related to the learning convergence, a descent in the average learning rate is a good indicator of learning convergence <cit.>. Let p a pair with the form (s,a |  s∈𝒮, a ∈𝒜) and 𝒫^n the set of all pairs p visited by the agent. Then, the learning rate is defined as: 

    l = 𝔼[ { C(x) |∀ x∈𝒫}],  where  C(x) = ∑_p ∈𝒫δ_x p.


Figure <ref> shows the convergence rate of learning rates for the experiments proposed. As observed, the learning rate exhibits a descent to 0, guaranteeing the convergence of the algorithm.







§ CONCLUSION


Fairness, as an ethical definition, is sometimes seen as negative by constraining developments <cit.>. However the present paper represents an example in which both customers and companies can benefit from transparency. Even if there is a disagreement regarding what is a fair policy in dynamic pricing, the fairness design principles proposed  in section <ref> provide an initial definition based on equality, that constitutes a promising way to open a rich dialog. In this paper we have demonstrated that an unfair scenario can be unbiased within a certain degree (the fairness targets given by f_t), by integrating fairness metrics as part of the model optimization. We cover the integration of fairness from a resource allocation point of view, including it as a design principle. 

As future work, there are possible extensions to this study. Firstly, there are many parameter (λ, β_p, β_f, ν, ...) to tuned in the learning process, considering the time required to evaluate each combination of parameters is a challenging task. Secondly, the comparison with other state-of-the-art models, like evolutionary algorithms <cit.>, is a valuable asset to an effective contrast to the Q-learning method proposed. Finally, more complex environment can be defined including more variables as time or external factors.

We demonstrate empirically that an effective balance between revenue and fairness maximization can be achieved in real time models like RL, in which the model learns by performing actions in an environment. We proposed a fairness metrics based on a rotated Jain's index. We have developed a synthetic environment to experiment with four different customer sensitivities. We have also tested several parametrized experiments, demonstrating that a given balance between revenue and fairness can be chosen. 






































IEEEtran

10

url@samestyle








deksnyteDP
I. Deksnyte and P. Zigmas Lydeka, “Dynamic pricing and its forming factors,”
  International Journal of Business and Social Science, vol. 3, no. 23,
  2012.

narahari2005dynamic
Y. Narahari, C. V. L. Raju, K. Ravikumar, and S. Shah, “Dynamic pricing models
  for electronic business,” Sadhana, vol. 30, no. 2, pp. 231–256,
  2005.

Boer:DP
A. V. den Boer, “Dynamic pricing and learning: Historical origins, current
  research, and new directions,” Surveys in Operations Research and
  Management Science, vol. 20, no. 1, pp. 1 – 18, 2015.

JanetAdamy
J. Adamy. (1999) E-tailer price tailoring may be wave of future. [Online].
  Available:
  <http://articles.chicagotribune.com/2000-09-25/business/0009250017_1_prices-amazon-spokesman-bill-curry-don-harter>


RePEc:sym
W. Reinartz, “Customizing prices in online markets,” Symphonya.
  Emerging Issues in Management, no. 1 Market-Space Management, p. 5, 2002.

garbarino2003dynamic
E. Garbarino and O. F. Lee, “Dynamic pricing in internet retail: Effects on
  consumer trust,” Psychology and Marketing, vol. 20, no. 6, pp.
  495–513, 2003.

lee2011perceived
S. Lee, A. Illia, and A. Lawson­Body, “Perceived price fairness of dynamic
  pricing,” Industrial Management and Data Systems, vol. 111, no. 4,
  pp. 531–550, 2011.

xia2004price
L. Xia, K. B. Monroe, and J. L. Cox, “The price is unfair! a conceptual
  framework of price fairness perceptions,” Journal of Marketing,
  vol. 68, no. 4, pp. 1–15, 2004.

weisstein2013effects
F. L. Weisstein, K. B. Monroe, and M. Kukar-Kinney, “Effects of price framing
  on consumers' perceptions of online dynamic pricing practices,”
  Journal of the Academy of Marketing Science, vol. 41, no. 5, pp.
  501–514, 2013.

Dynamic_pricing_fairness
K. L. Haws, W. O. Bearden, D. I. served as editor, and E. A. served as
  associate editor for this article., “Dynamic pricing and consumer fairness
  perceptions,” Journal of Consumer Research, vol. 33, no. 3, pp.
  304–311, 2006.

Odlyzko:2003
A. Odlyzko, “Privacy, economics, and price discrimination on the internet,”
  in Proceedings of the 5th International Conference on Electronic
  Commerce, ser. ICEC '03.1em plus 0.5em minus 0.4emACM, 2003,
  pp. 355–366.

kimesretrospective
S. E. Kimes, “A retrospective commentary on “discounting in the hotel
  industry: a new approach”,” The Cornell Hotel and Restaurant
  Administration Quarterly, vol. 43, no. 4, pp. 92–93, 2002.

kahneman1986fairness
D. Kahneman, J. Knetsch, and R. Thaler, “Fairness as a constraint on profit
  seeking: Entitlements in the market,” American Economic Review,
  vol. 76, no. 4, pp. 728–41, 1986.

finkel2001notfair
N. J. Finkel, Not Fair! The Typology of Commonsense Unfairness,
  1st ed.1em plus 0.5em minus 0.4emAmerican Psychological
  Association (APA), 2001.

KUTSCHINSKI20032207
E. Kutschinski, T. Uthmann, and D. Polani, “Learning competitive pricing
  strategies by multi-agent reinforcement learning,” Journal of Economic
  Dynamics and Control, vol. 27, no. 11, pp. 2207–2218, 2003.

kononen2006dynamic
V. Könönen, “Dynamic pricing based on asymmetric multiagent reinforcement
  learning,” International Journal of Intelligent Systems, vol. 21,
  no. 1, pp. 73–98, 2006.

GuptaRL1168003
M. Gupta, K. Ravikumar, and M. Kumar, “Adaptive strategies for price markdown
  in a multi-unit descending price auction: a comparative study,” in
  IEEE International Conference on Systems, Man and Cybernetics, vol. 1,
  2002, pp. 373–378.

MenonRLDimamic6873894
R. B. Menon, S. B. Menon, D. Srinivasan, and L. Jain, “Online reinforcement
  learning in multi-agent systems for distributed energy systems,” in
  2014 IEEE Innovative Smart Grid Technologies - Asia (ISGT ASIA), 2014,
  pp. 791–796.

fairMLauth
M. Skirpan and M. Gorelick, “The authority of "fair" in machine learning,”
  CoRR, vol. abs/1706.09976, 2017.

MLOpacity
J. Burrell, “How the machine ‘thinks’: Understanding opacity in machine
  learning algorithms,” Big Data and Society, vol. 3, no. 1, 2016.

bostromSuperintell
N. Bostrom, Superintelligence: Paths, Dangers, Strategies, 1st ed.1em plus 0.5em minus 0.4emOxford University Press, 2014.

Cerquitelli:2017:TDM:3110841
T. Cerquitelli, D. Quercia, and F. Pasquale, Transparent Data Mining for
  Big and Small Data, 1st ed.1em plus 0.5em minus 0.4emSpringer
  Publishing Company, Incorporated, 2017.

Mikians
J. Mikians, L. Gyarmati, V. Erramilli, and N. Laoutaris, “Detecting price and
  search discrimination on the internet,” in Proceedings of the 11th ACM
  Workshop on Hot Topics in Networks, ser. HotNets-XI.1em plus 0.5em
  minus 0.4emACM, 2012, pp. 79–84.

tolga-biasGender
T. Bolukbasi, K.-W. Chang, J. Y. Zou, V. Saligrama, and A. T. Kalai, “Man is
  to computer programmer as woman is to homemaker? debiasing word embeddings,”
  in Advances in Neural Information Processing Systems 29.1em
  plus 0.5em minus 0.4emCurran Associates, Inc., 2016, pp. 4349–4357.

bath55288
A. Caliskan, J. J. Bryson, and A. Narayanan, “Semantics derived automatically
  from language corpora contain human-like biases,” Science, vol. 356,
  no. 6334, pp. 183–186, 2017.

hardtPS16
M. Hardt, E. Price, and N. Srebro, “Equality of opportunity in supervised
  learning,” CoRR, vol. abs/1610.02413, 2016.

heeBiasFaces
H. J. Ryu, M. Mitchell, and H. Adam, “Improving Smiling Detection with
  Race and Gender Diversity,” ArXiv e-prints, 2017.

weiss2001online
R. Weiss and A. Mehrotra, “Online Dynamic Pricing: Efficiency, Equity and the
  Future of E-commerce,” Virginia Journal Of Law And Technology,
  vol. 6, no. 2, 2001.

TianLan-Resources
T. Lan, D. T. H. Kao, M. Chiang, and A. Sabharwal, “An axiomatic theory of
  fairness,” CoRR, vol. abs/0906.0557, 2009.

arianpoo2016network
N. Arianpoo and V. C. Leung, “How network monitoring and reinforcement
  learning can improve tcp fairness in wireless multi-hop networks,”
  EURASIP Journal on Wireless Communications and Networking, vol. 2016,
  no. 1, p. 278, Dec 2016.

francis2012techniques
M. Sirajuddin, C. Rupa, and A. Prasad, “Techniques for enhancing the
  performance of tcp in wireless networks,” in Artificial Intelligence
  and Evolutionary Algorithms in Engineering Systems, L. P. Suresh, S. S.
  Dash, and B. K. Panigrahi, Eds., 2015, pp. 159–167.

zhang2010tcp
X. M. Zhang, W. B. Zhu, N. N. Li, and D. K. Sung, “Tcp congestion window
  adaptation through contention detection in ad hoc networks,” IEEE
  Transactions on Vehicular Technology, vol. 59, no. 9, pp. 4578–4588, 2010.

Sutton:1998
R. S. Sutton and A. G. Barto, Introduction to Reinforcement Learning,
  1st ed.1em plus 0.5em minus 0.4emMIT Press, 1998.

Watkins1992
C. J. Watkins and P. Dayan, “Technical note: Q-learning,” Machine
  Learning, vol. 8, no. 3, pp. 279–292, 1992.

Dini12combiningq-learning
S. Dini and M. Serrano, “Combining q-learning with artificial neural networks
  in an adaptive light seeking robot,” 2012.

van2007reinforcement
H. van Hasselt and M. A. Wiering, “Reinforcement learning in continuous action
  spaces,” in 2007 IEEE International Symposium on Approximate Dynamic
  Programming and Reinforcement Learning, 2007, pp. 272–279.

Hornik:1989
K. Hornik, M. Stinchcombe, and H. White, “Multilayer feedforward networks are
  universal approximators,” Neural Networks, vol. 2, no. 5, pp. 359 –
  366, 1989.

Riedmiller-approximation
M. Riedmiller, “Neural fitted q iteration – first experiences with a data
  efficient neural reinforcement learning method,” in Machine Learning:
  ECML 2005, J. Gama, R. Camacho, P. B. Brazdil, A. M. Jorge, and L. Torgo,
  Eds.1em plus 0.5em minus 0.4emSpringer Berlin Heidelberg, 2005,
  pp. 317–328.

Lin-RL-NN
L.-J. Lin, “Reinforcement learning for robots using neural networks,” Ph.D.
  dissertation, School of Computer Science, 1992, uMI Order No. GAX93-22750.

VolodymyrAtari
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and
  M. Riedmiller, “Playing atari with deep reinforcement learning,” 2013, cite
  arxiv:1312.5602Comment: NIPS Deep Learning Workshop 2013.

RANA2015426
R. Rana and F. S. Oliveira, “Dynamic pricing policies for interdependent
  perishable products or services using reinforcement learning,” Expert
  Systems with Applications, vol. 42, no. 1, pp. 426 – 436, 2015.

jain1999throughput
D. A. Jain, R. and G. Babic, “Throughput fairness index: An explanation.” The
  Ohio State University, Tech. Rep., 02 2010.

phillips2005pricing
R. Phillips, Pricing and Revenue Optimization, ser. Stanford Business
  Books.1em plus 0.5em minus 0.4emStanford University Press,
  2005.

Matignon-ImprovingRL
L. Matignon, G. J. Laurent, and N. L. Fort-piat, “Improving reinforcement
  learning speed for robot control,” in 2006 IEEE/RSJ International
  Conference on Intelligent Robots and Systems, 2006, pp. 3172–3177.

KingmaB14
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization.”
  CoRR, vol. abs/1412.6980, 2014.

bowieEthics
N. E. Bowie, Organizational Integrity and Moral Climates.1em plus
  0.5em minus 0.4emSpringer, 2013, pp. 183–205.

evolutionaryAndRL
T. Salimans, J. Ho, X. Chen, and I. Sutskever, “Evolution strategies as a
  scalable alternative to reinforcement learning,” CoRR, vol.
  abs/1703.03864, 2017.





