How can I measure the voltage of 40A transformer?
You're not measuring the amperage, you are measuring the voltage. 

Set it to measure DC Volts on the 20 scale, read. The maximum voltage for the meter is printed right on it, 600.

The device you have is not a transformer, it is a DC power supply.

When you connect your lights, measure the voltage again with the lights connected. Don't try to measure the current with this meter.