Using 18650 batteries with a 3.7v to 5v boost converter
is it safe to wire the boost converter directly to the output terminals of the battery/batteries and take the 5V off the +5v pin using a common ground?


Yes. That is the normal way of using them.


  Also, I'm not 100% sure how to go about charging this setup. One option would be to put the batteries in cases and use an external charger with replaceable cells.


That is the simplest option, yes. You shouldn't charge Li-Poly / Li-Ion batteries whilst using them - the load messes with the charge termination detection and can cause your batteries to over-charge and explode.


  Would it make more sense to get one of these? This should be able to handle both charging AND stepping up the voltage, am I right?


That gets around the problem of charging while running, yes. A proper charger like that should bypass the battery when charging so the load is running from the incoming charge power rather than the battery.

One option you haven't considered though is to not use step-up.  Step-up is harder to manage with heavy loads. As you can see that first step-up module you link to has an upper limit of only 1.2A.  If you have 60 LEDs per meter, and 1 meter of LED strip, that's a maximum of 3.6A (60 * 0.02 * 3) which is 3x what that module can provide.

A better solution would be to use two batteries in series, not parallel, to give you about 7.2V. Then you can use a step-down switching regulator to give you your 5V and have far higher current limits.  Looking for a "UBEC" or "SBEC" in a model RC supplier shop is usually a good bet. For example:


Turnigy 5A (8-26v) SBEC for Lipo

HobbyKing™ HKU5 5V/5A UBEC


These are all designed for use with two or more LiPoly batteries in series.