Keeping the output of a heated LM317 constant
This answer is a bit speculative, but its my best guess what's going on.

To get 5 V output, you have your pot turned to get about 720 ohms for "R2". It's got (5-1.25 =) 3.75 V across it, so it's burning about 20 mW. Even though you don't feel it to the touch, it is heating up inside. A pot is probably a relatively large part, which helps explain the very large time constant you're seeing on this voltage drift. Check the datasheet to see if the pot has a postive tempco, or hit it with a hot air gun and watch the output as a first step to confirm or reject this as the source of error.

Another slightly less likely effect, also involving the potentiometer, would be if thermal effects are causing the wiper resistance to drift, or if a chemical change (like oxidation) is happening at the contact of the wiper to the resistive element over time. Since the wiper is part of the "R2" controlling your output voltage, any effect on the wiper resistance will also cause your output to drift. 

Depending what kind of pot you're using, and if you really need adjustability, you could try changing to a higher quality pot with lower tempco and known wiper resistance stability, or switching to a fixed resistor, and see if the drift is reduced or eliminated.