Effect of drinking coffee before riding?
A recent documentary on the BBC in UK covered some aspects of ingesting caffeine prior to exercise. A written summary is here:

http://www.bbc.co.uk/programmes/articles/23hqLh1TYQqZdql7GkqQZ1L/which-types-of-exercise-benefit-from-caffeine

Without reviewing the science in detail, it basically suggests that for endurance type activities, caffeine blocks receptors in the body, leading the brain to "perceive" less pain and fatigue, allowing someone to exercise longer. The article states an optimum amount of caffeine to be 3mg per kilo of bodyweight (somewhere between 1 and 2 large cups of filter coffee for average man / woman). To experience the benefit you should drink the coffee approximately one hour before starting the activity, and the effect will last for "a few hours". If you regularly drink a lot of coffee (or take other sources of caffeine), it is likely the impact will be reduced.

Your experience may just be the effect of the coffee / caffeine wearing off, plus any lessened impact due to the amount of caffeine you ingest on a daily basis.

Of course, other factors may be causing your fatigue, but your question specifically asked about effects of coffee / caffeine.