Analog input dependent on samples quantity used in averaging
Here is what the analog to digital sample circuit on that chip looks like...



When you first start off, the pin happens will likely have some residual charge on it, which causes a voltage to appear. Each time you sample, a little bit of that charge is bled off to charge the internal capacitor. As you bleed of more charge with each subsequent sample, the closer the voltage asymptotically gets to Vcc/2.

On the opposing side, that input circuit is has very high input impedance so it only takes a little charge to develop a voltage on the pin. If you connect a dangling wire to the pin, the wire will become an antenna and the voltage that you read on the pin will depend on the EMF in the area. In most places with AC power, the dominant EMF frequency will be 50/60Hz (depending on where you live) coming from the power lines. 

There is another possibility- without seeing the rest of your code, it is possible that buffer is overflowing. If, for example, buffer is only an unsigned int, then eventually it will grow to be bigger than 65536 (the largest value an unsigned in can hold) and will wrap around to a lower value. Make sure buffer can hold at least 1024 * 5000.