Which component upgrades for longer rides
If your goal is comfort over speed, I would emphasize in three upgrades which don't break your bank:


Find a comfortable saddle. Go to your LBS and ask their opinions. A lot of people suggest leather saddles, like Brooks or Selle Anatomica. A good saddle is expensive, but it is a worth upgrade and you will be surprised how big the difference is between a cheap saddle and a good one. If you are on a tight budget, a padded short is a cheaper option.
Do you use a bike rack and panniers? for a long ride especially on hot weather, a set of pannier is better than a backpack.
I think your current tires are already pretty good but it might be worth a try to have wider and fatter tires for more comfortable rides. I assume your current tires are 32mm wide, so upgrade to 35mm tires.