Should a weak iPhone battery output less than 3.7V?
The answer is yes it could be week.
What you measuring is the Voltage without LOAD.

In that state even an almost completely empty battery will output the Voltage it was designed for.

You need to bridge it with a resistor (load) and then measure voltage.

Your multimeter does that (LOAD) but only very small kind, like micro amps by bridging the outputs with a resistor of 1 Meg or more so it draws very small current

Try using a LED or if you have resistors of lets say 100 Ohm, use that as load (bridge).