Learning to Collaborate: Multi-Scenario Ranking via Multi-Agent
  Reinforcement Learning
  Ranking is a fundamental and widely studied problem in scenarios such as
search, advertising, and recommendation. However, joint optimization for
multi-scenario ranking, which aims to improve the overall performance of
several ranking strategies in different scenarios, is rather untouched.
Separately optimizing each individual strategy has two limitations. The first
one is lack of collaboration between scenarios meaning that each strategy
maximizes its own objective but ignores the goals of other strategies, leading
to a sub-optimal overall performance. The second limitation is the inability of
modeling the correlation between scenarios meaning that independent
optimization in one scenario only uses its own user data but ignores the
context in other scenarios.
  In this paper, we formulate multi-scenario ranking as a fully cooperative,
partially observable, multi-agent sequential decision problem. We propose a
novel model named Multi-Agent Recurrent Deterministic Policy Gradient (MA-RDPG)
which has a communication component for passing messages, several private
actors (agents) for making actions for ranking, and a centralized critic for
evaluating the overall performance of the co-working actors. Each scenario is
treated as an agent (actor). Agents collaborate with each other by sharing a
global action-value function (the critic) and passing messages that encodes
historical information across scenarios. The model is evaluated with online
settings on a large E-commerce platform. Results show that the proposed model
exhibits significant improvements against baselines in terms of the overall
performance.
