How to train AI Game Bots with screen pixels as input
How to train AI Game Bots with screen pixels as input

Overview of the paper “CURL: Contrastive Unsupervised Representations for Reinforcement Learning” by Laskin et al.

A big step towards building human-like game-playing AI is to remove the unfair advantage that game AIs have with the access to the internal game state information, which is not available to us directly on the screen while playing the game. This can be done by forcing the AI bot to only use the image that we see on the screen and ask it to make decisions based on that information alone.

My GTX 1070 GPU crapping out while trying to train a football RL agent from screen pixels.

From my own experiments in the past building a football AI bot, I have seen promising results with this approach, but have always faced one big problem. Training RL agents on raw pixel information requires lot of compute power and thousands of training samples, because the agent has to not only learn the game itself, but also has to learn to analyze these images and extract meaningful representation from them that are needed for good decision-making.

Learning better pixel represenations

Which is why in this episode I want to share a paper that aims to solve this problem and make it easier to train RL agents on pixel information. It is titled “CURL: Contrastive Unsupervised Representations for Reinforcement Learning” and was introduced earlier this year by the research group at UC Berkeley.

Here’s the core idea of the CURL framework. Consider an RL training process on images of FIFA. Normally, we feed the pixels of the game image directly to RL process and use a CNN to process this information. Uptil now, it hasn’t been clear what the best way is to train this CNN so that it makes the reinforcement learning process easier.

In this paper, the authors present a framework which adds a Contrastive Learning module to train this CNN encoder. It adds a separate contrastive loss to optimize the encoder’s output which can be used either before or along with the RL training process. This takes the image understanding workload off of RL learning process, thereby enabling us to provide fewer training samples and improving the sample efficiecy.

Let’s try to udnerstand at a high level what contrastive learning tries to do. Consider this screenshot of the game fifa as a base or an anchor in our unsupervised learning process.
['Chintan Trivedi']
['Deep Learning', 'Artificial Intelligence', 'Reinforcement Learning', 'Computer Vision', 'Game Futurology']