Pixel time series trend: What are the causes?
You are very likely to observe trends in either your camera or your light source. 

Sources from the camera can include:


thermal effects inside the sensor (basically, the more you use it, the more current will traverse it in various parts and it will get warmer)
drift in the ADC of your sensor (probably linked to the aforementioned).


Also, light sources can also drift with time: thermal effects will influence light bulbs.

You can try to mitigate these effects by:


changing your light source (I would assume LED lamps are more stable, but this needs to be checked)
waiting long enough before starting the measurements in order to avoid transitory phenomenon
carefully playing with the settings of your camera. Typically, I would fix manually all the exposure-related settings (exposure time, gain, framerate...) so that the autocontrols do not make vary the exposure values when an object enters the scene. However, this will prevent the camera from tracking natural scene trends.
simply fitting a linear regression to your signal in order to remove the trend while processing.