Can someone clarify how to charge a battery without an off-the-shelf battery charger?
You're on the right track. Lithium batteries take what is called a "Constant Current, Constant Voltage" or CC-CV charge profile.

Constant Current:
When the battery is discharged, you begin charging it by applying a fixed current (at the battery's voltage). Typically this current is somewhere in the neighborhood of 1C (the current required to completely discharge the battery from full over the course of 1 hour - i.e., for a 1000mAh capacity battery, 1C would be 1 Amp). Check the specifications for your battery, however, as the best charge current may be more or less than 1C. Because this phase of charging is performed at a single current, it is called "Constant Current".

Constant Voltage:
As the battery charges, its voltage will rise to the maximum battery voltage. Once the battery reaches this voltage, it is not completely charged, but continuing to charge it at the same constant current will cause its voltage to exceed the maximum voltage. Therefore, the charge current must now be tapered off to maintain a fixed voltage on the battery. After some amount of time, the charge current will become very small, and this indicates that the battery is completely charged. Because the objective of this phase of charging is to maintain a constant voltage on the battery, it is called "Constant Voltage".

The charge algorithm can be summarized as follows:


Apply a constant charge current to the battery (Constant Current
mode)
When the voltage hits the max battery voltage, transition
to Constant Voltage mode, applying a continuously decreasing current
to maintain the constant voltage on the terminals.
At some small current, turn off the charger altogether and the battery is charged.


This photo has a nice visualization of the voltage and current throughout the charge cycle:



Your idea to use a power supply set at the max voltage and with a current limit set to the max charge rate of the battery would work just fine. You would have to manually turn the supply off at the end of charging.