Is it expected that during self-play reinforcement learning that player 1 or player 2 wins the majority of games? I'm testing various learning rates and neural network configurations. I'm testing over 10000 games, with the first 2000 having random starting moves and general randomness throughout of about 20%, i.e. 20% of moves are random.

In all configurations, I initiate the weights to random values.

What I've found is that in all configurations, Player 1 will win the majority of games, or Player 2 will. There's no 50/50 split.

Is this expected or normal? I am training on the Connect Four game.
From wikipedia article on  Connect Four:


  Connect Four is a solved game. The first player can always win by
  playing the right moves.


It's pretty reasonable that even without training 1st player win more often by small margin.
With some training  1st player should mostly win.