I'd like to run a 5v fan on 2 3.7v batteries
It seems obvious that I couldn't run the fan on one 3.7v battery

No it does not, many motors which are designed for 5 V will also run on 3.7 V. The same is true for fans. So I would just try and see if it works.

Also, a 3.7 V Li-Ion cell will be have a voltage around 4.2 V when fully charged. That 3.7 V is the voltage when the battery is almost empty.

If one cell doesn't work then I would use a simple switched mode converter module which are cheap, efficient and should "just work" even for people with little experience with electronics.