Artificially intelligent agents equipped with strategic skills that can negotiate during their interactions with other natural or artificial agents are still underdeveloped. This paper describes a successful application of Deep Reinforcement Learning (DRL) for training intelligent agents with strategic conversational skills, in a situated dialogue setting. Previous studies have modelled the behaviour of strategic agents using supervised learning and traditional reinforcement learning techniques, the latter using tabular representations or learning with linear function approximation. In this study, we apply DRL with a high-dimensional state space to the strategic board game of Settlers of Catan---where players can offer resources in exchange for others and they can also reply to offers made by other players. Our experimental results report that the DRL-based learnt policies significantly outperformed several baselines including random, rule-based, and supervised-based behaviours. The DRL-based policy has a 53 win rate versus 3 automated players ( bots'), whereas a supervised player trained on a dialogue corpus in this setting achieved only 27 , versus the same 3 bots. This result supports the claim that DRL is a promising framework for training dialogue systems, and strategic agents with negotiation abilities.
@cite_29: We propose a hybrid model for automatically acquiring a policy for a complex game, which combines online learning with mining knowledge from a corpus of human game play. Our hypothesis is that a player that learns its policies by combining (online) exploration with biases towards human behaviour that's attested in a corpus of humans playing the game will outperform any agent that uses only one of the knowledge sources. During game play, the agent extracts similar moves made by players in the corpus in similar situations, and approximates their utility alongside other possible options by performing simulations from its current state. We implement and assess our model in an agent playing the complex win-lose board game Settlers of Catan, which lacks an implementation that would challenge a human expert. The results from the preliminary set of experiments illustrate the potential of such a joint model.
@cite_20: We present an empirical framework for testing game strategies in The Settlers of Catan, a complex win-lose game that lacks any analytic solution. This framework provides the means to change different components of an autonomous agent's strategy, and to test them in suitably controlled ways via performance metrics in game simulations and via comparisons of the agent's behaviours with those exhibited in a corpus of humans playing the game. We provide changes to the game strategy that not only improve the agent's strength, but corpus analysis shows that they also bring the agent closer to a model of human players.
@cite_8: 
This paper extends the recent work above on training strategic agents using reinforcement learning, which have either used small state-action spaces or focused on navigation commands rather than negotiation dialogue. The learning agents described in this paper use a high dimensional state representation (160 non-binary features) and a fairly large action space (73 actions) for learning strategic non-cooperative dialogue behaviour. To our knowledge, our results report the highest winning rates reported to date in the game of Settlers of Catan, see @cite_20 @cite_8 @cite_29 . The comprehensive evaluation reported in the previous section is evidence to argue that deep reinforcement learning is a promising framework for training strategic interactive agents.