How to represent players in a multi agent environment so each model can distinguish its own player
When you trained your agents separately they never saw squares with opposing values (1/-1). So agents don't really know what to expect from visiting that square. 
I'd try adding (1/-1) to the condition on which you base your squares availability. Also try increasing the reward. It's hard to give suggestions without looking at the code.