Measure lipo voltage while powering arduino
The reference voltage for analogRead should be from the regulated 5V line, not from the unregulated input to the Arduino.

The real problem is that the voltage out of the battery will drop with current load. You're supposed to measure either the unloaded battery, or under a known, controlled load.

You ask "Should I use a step up module to 5V?" The answer is yes, but that's unrelated to your voltage measurement question. An Arduino requires a source of regulated 5V. The internal voltage regulator is a linear regulator that requires somewhat more than 5V as input (7.5V is about right.) You can't power the internal voltage regulator with a single 3.7V lipo battery. You'd need a step-up regulator, and with that, you could bypass the internal regulator on the Arduino (assuming your step-up regulator puts out 5V.)