How should I handle invalid actions (when using REINFORCE)? I want to create an AI which can play five-in-a-row/gomoku. I want to use reinforcement learning for this.
I use policy gradient method, namely REINFORCE, with baseline. For the value and policy function approximation, I use a neural network. It has convolutional and fully connected layers. All of the layers, except for the output, are shared. The policy's output layer has $8 \times 8=64$ (the size of the board) output unit and softmax on them. So it is stochastic.
But what if the network produces a very high probability for an invalid action? An invalid action is when the agent wants to check a square that has one X or O in it. I think it can be stuck in that game state.
How should I handle this situation?
My guess is to use the actor-critic method. For an invalid action, we should give a negative reward and pass the turn to the opponent.
Just ignore the invalid moves.

For exploration it is likely that you won't just execute the move with the highest probability, but instead choose moves randomly based on the outputted probability. If you only punish illegal moves they will still retain some probability (however small) and therefore will be executed from time to time (however seldom). So you will always retain an agent which occasionally makes illegal moves.

To me it makes more sense to just set the probabilities of all illegal moves to zero and renormalise the output vector before you choose your move.