Reinforcement learning without self-play Self-play seems to be very important for reinforcement learning algorithms. Can we train reinforcement learning algorithms without self-play (or learn mostly from past experiences)?

For example, let's say that I want to build a TRPG AI, which involves natural language and physical interaction between heterogeneous players. I can't do self-play this game so many times. But I happen to have a lot of match log of past human games. 

Can I achieve higher score than humans using reinforcement learning using these logs (maybe like Experience Replay)?
If you have a sane reward function and sufficient examples you can train a neural network on predicting your next state/value given a state/action and then use this neural network as a model to generate many more training examples on which to train a policy by selecting new action combinations.  This may result in unexpected behaviour if you do not have sufficient examples which is quite likely to occur unless you have a very large training set.

Reading up on Actor-Critic reinforcement learning may help you here.