Current decreasing while measuring the current through a constant current source resistor
All semiconductors (as well as some conductors) change forward voltage drop depending on temperature. 

See which parts are getting warmer, using an infrared thermometer, and put a heat-sink on them (or just heat-sink likely culprits). Don't forget your LED string.

As to why you don't see a difference in brightness, visual response to light is approximately logarithmic. The stellar magnitude scale is based on that response. Simply put, you would barely notice a difference in brightness by a factor of ~1.4 of two lights, unless they were adjacent. A difference of 20 mA compared to 16 mA, ~20%, would not be apparent.