We propose a reinforcement learning solution to the soccer dribbling task, a scenario in which a soccer agent has to go from the beginning to the end of a region keeping possession of the ball, as an adversary attempts to gain possession. While the adversary uses a stationary policy, the dribbler learns the best action to take at each decision point. After defining meaningful variables to represent the state space, and high-level macro-actions to incorporate domain knowledge, we describe our application of the reinforcement learning algorithm Sarsa with CMAC for function approximation. Our experiments show that, after the training period, the dribbler is able to accomplish its task against a strong adversary around 58 of the time.
@cite_15: This paper describes the structure of the RoboCup team, Andhill, which won the second prize in the RoboCup97 tournament, and the results of the reinforcement learning in which an agent receives a reward when it kicks the ball. In multi-agent reinforcement learning, the trade-off between exploration and exploitation is a serious problem. This research uses observational reinforcement to ease the exploration problem.
@cite_14: In this paper, we propose a reinforcement learning method called a fuzzy Q-learning where an agent determines its action based on the inference result by a fuzzy rule-based system. We apply the proposed method to a soccer agent that intercepts a passed ball by another agent. In the proposed method, the state space is represented by internal information the learning agent maintains such as the relative velocity and the relative position of the ball to the learning agent. We divide the state space into several fuzzy subspaces. A fuzzy if-then rule in the proposed method represents a fuzzy subspace in the state space. The consequent part of the fuzzy if-then rules is a motion vector that suggests the moving direction and velocity of the learning agent. A reward is given to the learning agent if the distance between the ball and the agent becomes smaller or if the agent catches up with the ball. It is expected that the learning agent finally obtains the efficient positioning skill.
@cite_10: Our long-term goal is to build teams of agents where the decision making is based completely on Reinforcement Learning (RL) methods. It requires an appropriate modelling of the learning task and the paper describes how robotic soccer can be seen as a multi-agent Markov Decision Process (MMDP). It discusses how optimality of behaviours of agents can be defined and what difficulties one encounters in developing concrete algorithms which are supposed to reach such optimal agent team policies. We also give an overview of already incorporated algorithms in our 'Karlsruhe Brainstormers' simulator league team and report some results on learning of offensive team behaviour.
Reinforcement learning has long been applied to the robot soccer domain. For example, Andou @cite_15 uses observational reinforcement learning" to refine a function that is used by the soccer agents for deciding their positions on the field. Riedmiller @cite_10 use reinforcement learning to learn low-level soccer skills, such as kicking and ball-interception. Nakashima @cite_14 propose a reinforcement learning meth-od called fuzzy Q-learning", where an agent determines its action based on the inference result of a fuzzy rule-based system. The authors apply the proposed method to the sce-nario where a soccer agent learns to intercept a passed ball.