Is it expected that during self-play reinforcement learning that player 1 or player 2 wins the majority of games?
From wikipedia article on  Connect Four:


  Connect Four is a solved game. The first player can always win by
  playing the right moves.


It's pretty reasonable that even without training 1st player win more often by small margin.
With some training  1st player should mostly win.