Card game for Gym: Reward shaping I am working on a card game for openai gym and currently I ask myself how to shape the reward function for it. One round of the game consists of each player picking a card from its hand, whereas not every card can be played depending on the card which has been played by one of the players before. For every set of played cards, there is a total order such that the player with the highest card wins the round.
In the situation in which cards are rejected I want to give the agent some reward.

In case of an invalid card, it is hard to say if that card is any nearer to one of the valid cards than any other. Also the agent should learn that this card is not playable at this point. 

For completeness, the agent gets a discrete observation of everything it can remember of the game (its own cards, cards played in current round, cards played in past rounds, game mode (defines the total order of cards)). Then it should play a discrete action which either is a game mode in the beginning or a card during the round. Then it either gets a reward because its card got rejected or it gets a reward based on whether it wins the round or not. The game accounts a certain amount of points for a won round depending on the constellation of played cards in that round.

My question is how to shape the rewards for card rejection and for winning a round. Any ideas? Positive or negative?

In case any more details are required, just ask for them.
My question is how to shape the rewards for card rejection and for winning a round. Any ideas? Positive or negative?


In reinforcement learning, you must set rewards so that they are maximised when the agent achieves the goals of the problem. You should avoid trying to "help" the agent by setting interim rewards for things that might help it achieve those goals.

For card rejection, if that is part of the game (i.e. it is valid to play a "wrong" card, and you lose your turn), then either no reward, or a negative one might suffice. Probably you should go with no reward, because the punishment would be in not winning that round anyway.

If an invalid card cannot actually be played according to the rules of the game, and there is no "pass" move or equivalent, then you should not allow the agent to select it. Simply remove the action from consideration when making action selection. It is OK for the agent/environment to enforce this in a hard-coded fashion: A common way to do that, if your agent outputs a discrete set of action probabilities or preferences, is to filter that set by the environment's set of allowed actions, and renormalise.

What if you want the agent to learn about correct card selection? Once you have decided that, then it becomes a learning objective and you can use a reward scheme. The action stops being "play a card" and becomes "propose a card to play". If the proposal is valid, then the state change and reward for the round's play are processed as normal. If the proposal is not valid, then the state will not change, and the agent should receive some negative reward. Two things to note about this approach:


Turns in the game and time steps for the agent are now separate. That's not a problem, just be aware of the difference. 
This will probably not encourage the agent to play better (in fact for same number of time steps, it will probably have learned less well how to win, because it is busy learning how to filter cards based on the observed features), but it will enable it to learn to propose correct cards without that being forced on it in a hard-coded fashion.


For winning a round, then you  might want to reward the agent according to the game score it accumulates. Assuming that the winner of the overall game is the player with the highest score, this should be OK. 

However, there is a caveat to that: If by making certain plays the agent opens up other players to score even higher, then simply counting how many points the agent gets is not enough to make it competitive. Instead, you want very simple sparse rewards: e.g. +1 for winning the game, 0 for drawing, -1 for losing. The main advantage of using RL approach in the first place is that the algorithms can and should be able to figure out how to use this sparse information and turn it into an optimal strategy. This is entirely how AlphaGo Zero works for instance - it has absolutely no help evaluating interim positions, it is rewarded only for winning or losing.

If you go with +1 win, -1 lose rewards, then you could maybe make players' current scores part of the state observation. That may help in decision making if there is an element of risk/gambling where a player behind in the scores might be willing to risk everything on the last turns just for a small chance to win overall.