How should I define the reward function in the case of Connect Four?
The classic working reward scheme for two player zero sum games (i.e. if I win, you lose and vice versa) is simply:


+1 for a win
0 for a draw
-1 for a loss


These rewards should be associated with the last move made by the player before the game is resolved.


  I thought about giving a negativ reward for the move played before the winning move. 


That is what is normally done.


  Thinking about this I came to the conclusion that this is a bad idea. There'll be always a looser (except for ties), therefor there always be a last move from the loosing player. This one hasn't to be a bad one. Mistakes could have been made much earlier.


It's OK, what the agent is learning is the value of a combination of state and action. It will, eventually, correctly associate the negative reward with a chain of board states that decrease in value and that it will then try to avoid.

This scenario of delayed rewards, not necessarily related to the choice of the current action, is handled correctly by reinforcement learning algorithms. It is still a tough problem though - a more delayed and difficult to predict reward makes for a tougher RL problem. However, even basic learners, such as Q-learning, can eventually solve this.

To get better performance, you might want to look into adding planning algorithms to the RL part.