Should I be trying to create a generic or specific (to particular game) reinforcement learning agent?
If what you mean by a generic reinforcement-learning application is an application that can learn any game (or some games), then you can't do it. Why? Because the goal of each game isn't the same, so you have to adapt the rewards depending on the game. If you just want to make an AI for Connect Four, I suggest you to make a specific RL application for that game.

I want to mention another thing: you shouldn't give a reward based on token's position, because it's hard to know what token's position is the best. Instead, just assign a big reward to the winner. That way, you're generalizing your algorithm, and you're avoiding your AI to focus on the wrong goal.

You have to be careful, you don't know for sure what token's position is the best. Try to give a reward ONLY when a player wins the game, because you can know for sure that winning is the best thing to do. It may sound idiot said like that, but that way, you're ensuring you AI to learn by itself what token's position is the best.