Deep Reinforcement Learning for Sponsored Search Real-time Bidding
  Bidding optimization is one of the most critical problems in online
advertising. Sponsored search (SS) auction, due to the randomness of user query
behavior and platform nature, usually adopts keyword-level bidding strategies.
In contrast, the display advertising (DA), as a relatively simpler scenario for
auction, has taken advantage of real-time bidding (RTB) to boost the
performance for advertisers. In this paper, we consider the RTB problem in
sponsored search auction, named SS-RTB. SS-RTB has a much more complex dynamic
environment, due to stochastic user query behavior and more complex bidding
policies based on multiple keywords of an ad. Most previous methods for DA
cannot be applied. We propose a reinforcement learning (RL) solution for
handling the complex dynamic environment. Although some RL methods have been
proposed for online advertising, they all fail to address the "environment
changing" problem: the state transition probabilities vary between two days.
Motivated by the observation that auction sequences of two days share similar
transition patterns at a proper aggregation level, we formulate a robust MDP
model at hour-aggregation level of the auction data and propose a
control-by-model framework for SS-RTB. Rather than generating bid prices
directly, we decide a bidding model for impressions of each hour and perform
real-time bidding accordingly. We also extend the method to handle the
multi-agent problem. We deployed the SS-RTB system in the e-commerce search
auction platform of Alibaba. Empirical experiments of offline evaluation and
online A/B test demonstrate the effectiveness of our method.
