Is there an instrument that can measure the capacity of lead acid batteries
Yes, there are lead acid battery testers that will tell you the condition of each battery. They are often used by UPS service technicians during preventative maintenance checks to check on the health of each battery in a large series string.

Here are a couple examples:

http://www.cadex.com/technology/our-technologies/spectro
http://www.bkprecision.com/products/electrical-battery-testers/battery-capacity-analyzers/600-12v-sla-battery-capacity-analyzer.html

The good ones probably cost more than a set of replacement batteries for your device, but if you can find a UPS dealer or service tech, they may be able to test your batteries for a small fee.