Charging li-ion cell using constant-voltage only
(the numbers are given for LiCoO2 chemistry, for LiFePO4 they should be lower)
Short answer: yes, this is OK. I charge lithium cells with a bench supply set to 4.2 V and whatever C/10 figure is for the cell I'm charging. If you are not in a hurry, slow charging is better, even for cells claimed to be capable of withstanding higher charging currents.

However (quoting you):


  charging at a constant voltage (say 4.2V) so long as the maximum
  current is limited to a reasonable value for the cell


means you will have constant current charger till your cell is at ~95%. Up to this point the voltage across the battery will be less than 4.2V if you measure it. Only when your charger starts outputting 4.2V it will become constant voltage.

What you are about to build is CC/CV charger and this is the right thing to do. "Constant voltage only" charger will be set to 4.2 V with no current limiting and it will charge the lithium cell very slowly. You can check it youself, just construct var.voltage circuit and measure the current into (discharged) cell at 3.5, 3.7, 4.0, 4.2, and 4.5 V. Cheap Chinese chargers are constructed like that, they restrict the voltage to 4.2 V so the cell won't ignite after being charged but the consumer would have to wait longer. I once bought a portable emergency charger which could be emptied in a couple of hours then took 3 days to recharge its internal cells.

There are other precautions to observe while charging lithium, you can learn them from any modern charge controller IC datasheet (my favorite is Linear Tech., their literature is very high quality). If you don't implement these precautions in your design never leave it unattended while charging, otherwise it may ruin your morning one day.