Cant get a grasp on how a full bridge on IR2104 works
If you operate the H bridge as you describe then yes, your motor will be reversing. The sensible approach is to use one half of the H bridge for PWM and the other half with logic levels that switch either the top or bottom MOSFET on for the duration of rotation in the direction you wish to go.

In my innocence on this subject of PWM switching I raised this question (a few years ago) that you may find useful to read. If you still want to have full PWM on both halves, you have to consider that the 50% PWM scenario is the "motor-stopped" situation.