Is it safe to use a battery charger to test the capacity of a non-rechargeable battery?
That charger device clearly states that it is for NiCd and NiMH rechargeable batteries only. It would be a risky proposition to put other batteries in that unit whether they be lithium or alkaline types. I think it would be just too easy to put the unit into the wrong mode.

What you want to do instead is get yourself a digital multimeter that can measure voltage of any battery that you may have laying around. These days you can pickup a suitable device that would fulfill your needs for a surprising low cost. Check out offerings on Amazon for example. The good thing about this approach is that you will find other uses for a digital multimeter once you have it and learn how to use it.