How to charge a battery + check voltage?
Yes it is safe to use a power supply rated for more current than the device to be powered wants.  The power supply current rating says what it can put out if the load demands it.  It has to be rated for at least what the load needs, but may be more.  The voltage has to match though.  Too low voltage and the device won't work.  Too high and it could be damaged.

Valid USB power voltage is a little above 5V, but I don't remember the details.  You can assume the GPS was designed to work with that, but without explicit specs that say it will work at 6V you should assume it won't.  It probably will, at least for a while, but it could also get damaged in subtle ways.  I don't recommend playing that game.

It sounds like the real problem is that you have the wrong GPS.  There are devices meant to be light and portable and run from little power.  I expect there even are GPSs specifically designed for bicycles, but I haven't looked.  I know there are backcountry hiking GPSs.  These are obviously meant to be light weight, are probably conservative with power, and should take rechargeable batteries.  The simplest answer might be to carry a extra rechargeable battery or two intended for your GPS.  That's what I do when going on a long hike with my camera that has a GPS attached.  It actually last a number of hours on a single charged battery pack, but a spare or two is small and light weight.