Artificially intelligent agents equipped with strategic skills that can negotiate during their interactions with other natural or artificial agents are still underdeveloped. This paper describes a successful application of Deep Reinforcement Learning (DRL) for training intelligent agents with strategic conversational skills, in a situated dialogue setting. Previous studies have modelled the behaviour of strategic agents using supervised learning and traditional reinforcement learning techniques, the latter using tabular representations or learning with linear function approximation. In this study, we apply DRL with a high-dimensional state space to the strategic board game of Settlers of Catan---where players can offer resources in exchange for others and they can also reply to offers made by other players. Our experimental results report that the DRL-based learnt policies significantly outperformed several baselines including random, rule-based, and supervised-based behaviours. The DRL-based policy has a 53 win rate versus 3 automated players ( bots'), whereas a supervised player trained on a dialogue corpus in this setting achieved only 27 , versus the same 3 bots. This result supports the claim that DRL is a promising framework for training dialogue systems, and strategic agents with negotiation abilities.
@cite_30: In this paper, we consider the task of learning control policies for text-based games. In these games, all interactions in the virtual world are through text and the underlying state is not observed. The resulting language barrier makes such environments challenging for automatic game players. We employ a deep reinforcement learning framework to jointly learn state representations and action policies using game rewards as feedback. This framework enables us to map text descriptions into vector representations that capture the semantics of the game states. We evaluate our approach on two game worlds, comparing against baselines using bag-ofwords and bag-of-bigrams for state representations. Our algorithm outperforms the baselines on both worlds demonstrating the importance of learning expressive representations. 1
@cite_18: Abstract: The game of Go is more challenging than other board games, due to the difficulty of constructing a position or move evaluation function. In this paper we investigate whether deep convolutional networks can be used to directly represent and learn this knowledge. We train a large 12-layer convolutional neural network by supervised learning from a database of human professional games. The network correctly predicts the expert move in 55 of positions, equalling the accuracy of a 6 dan human player. When the trained convolutional network was used directly to play games of Go, without any search, it beat the traditional search program GnuGo in 97 of games, and matched the performance of a state-of-the-art Monte-Carlo tree search that simulates a million positions per move.
@cite_17: We present the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning. The model is a convolutional neural network, trained with a variant of Q-learning, whose input is raw pixels and whose output is a value function estimating future rewards. We apply our method to seven Atari 2600 games from the Arcade Learning Environment, with no adjustment of the architecture or learning algorithm. We find that it outperforms all previous approaches on six of the games and surpasses a human expert on three of them.
Recent work on deep learning applied to games include the following. @cite_18 train a deep convolutional network for the game of Go, but it is trained in a supervised fashion rather than trained to maximise a long-term reward as in this work. A closely related work to ours is a DRL agent for text-based games @cite_30 . Their states are based on words, their policies are induced using game-based rewards, and their actions are based on directions such as go east west south north'. Another closely related work to ours is DRL agents trained to play ATARI games @cite_17 . Their states are based on pixels from down-sampled images, their policies make use of game-based rewards, and their actions are based on joystick movements. In contrast to these previous works which are based on navigation commands, our agents are use trading dialogue moves (e.g. I will give you ore and sheep for clay', or I accept decline your offer'), which are essential behaviours for strategic interaction.