Ni-MH battery charging issue - custom built transformers, LM317 gets warm
EDIT: Your description is slightly unclear, but I think you're saying that your LM317 regulator gets too hot when charging Ni-Cd and Ni-MH cells compared to chargin Li-Ion cells, and that you are using 5V as an input to LM317. Correct?
 The heat comes from the larger difference between the 5V input and the average 1.2V of the cells, compared to average 3.8V for Li-Ion.
You have 3.8V vs. 1.2V, which is 3 times the difference in voltage, and therefore 3 times the heat at the same current.  

Below is the answer I wrote earlier, before I have re-read your question (or you may have edited it):
 You need to limit both the maximum voltage and the current.
The Ni-MH cells have higher capacity so they will draw a charging current for a longer time, but they also have higher internal resistance which causes them to heat up more than Ni-Cd. You have not specified the sizes nor the capacities of the cells in your question.
 The charging voltage of the Ni-MH cells should never go above 1.6V.
The maximum charging current is 1C, C being the capacity of the cell, so if the cell has a capacity of 2000mAh (2Ah), the maximum charging current should be 2A.
However, it is better to charge it at a lower current, usually no more than 0.3C to 0.5C. Many chargers use the 0.1C (0.2A for a 2000mAh cell).
Once the cell is charged up, it can have a small "trickle charge" current flowing into it without damaging it, and it value is no more than 0.02C or 1/20 of the capacity (0.04A or 40mA for a 2000mAh cell).
To make it simple and safe, you could set your voltage regulator to 1.6V (or 1.55V) and place a 1-3.3Ω, 1W resistor at its output to limit the current. It could take anywhere from 5 to 20 hours to charge it fully, depending on its charge level at the beginning).
Otherwise, you could use a CC/CV DC-DC converter, set its current to 0.3C and its voltage to 1.55V. It would be faster and more efficient.