Would a convolution network make sense for policy based tic-tac-toe approach?
For anyone interested, I'm posting my results of trying out both CNNs and MLPs -

A 3x3 kernel 4 feature maps CNN consistently performed better than a 9 neuron MLP. I trained both on completely random tic-tac-toe training data size of 10kx20 and the CNN learned to to achieve ~80% win/draw against a random opponent while the MLP achieved around ~60% for the same data. A 2x2 CNN performed much worse than 3x3.

Training data - Inputs and outputs are same as mentioned in the question with the exception that the AI is trained only if it loses or draws the game and the dataset is chosen from randomized player for drawn games.

Note: It is possible to achieve much better accuracy with less data but in this case my goal was to just use this game to learn machine learning programming so the code was not particularly optimized for anything. The comparison of MLPs and CNNs is also incidental and needs to be done with more rigor for any serious purposes.