Low voltage drop regulators and switches for battery powered device
For this situation, you want a SEPIC regulator. This type of regulator can take input voltages higher or lower than the output voltage, which is perfect for your situation where you want to regulate voltages from 6.9V (fully charged 6V lead-acid) down to a little less than 6V (discharged) to a flat 6V. They are sold on Digikey, Mouser, TI, etc..

Unfortunately, they are a little pricey. Possibly a better solution is to get a 12V lead-acid battery and any old 6V regulator. If you are going to regulate the voltage down, any battery voltage will do...