LM317-based adj power supply output takes a while to stablize
Given the information provided by your data logger's diagrams, I am fairly sure you are looking at thermal drift. The time constant of your drift is in the range of some 10 seconds to some minutes, which is quite common for thermal settling of small-ish parts. You say that in your first tests, your load takes something like 0.1&nbsp;A, and the difference between input and output is approximately 20&nbsp;V&nbsp;-&nbsp;5&nbsp;V&nbsp;=&nbsp;15&nbsp;V. This means your LM317 is dissipating apporximately 1.5&nbsp;W - enough to give it a noticeable temperature rise, depending on the heatsink. When you set the output to a higher voltage, and assuming the current stays the same, you reduce the power dissipation (heating action), which would explain why the drift takes longer with a higher output voltage. (Edit: Slightly different numbers would apply for your experiments with a 1k load resistor, of course.)

If the LM317's internal reference drifts to a slightly lower voltage with rising temperature, the output will drift as well. If you start at 5.27&nbsp;V at room temperature and get 0.1&nbsp;V less with a hot LM317, this is something like a drift of 2&nbsp;%. Not exactly good, but also not unheard of for a standard integrated voltage regulator...

You can test this theory by using cooling spray (or just by blowing cold air along the part) or a hot air gun (or a soldering iron) and monitoring the regulated output voltage with your precision DMM. Your particular regulator seems to have a negative temperature coefficient, so cooling it should make the output voltage higher and heating it should make it lower.

TI's LM317 data sheet looks like their version of the regulator IC even has a negative tempco, so it should do what your IC does. Other manufacturers' parts may have a positive tempco, however. Cf. top left of pages 4 and 5.