Results for GP2Y1010AU0F Dust Sensor
Its because the resolution of the analog read is 10 bits or 1024 discrete values.

In your code the density is basically calculated as:

density = (((5000 / 1024.0) * analog_value * 11) - 400) * 0.2


You can calculate the resolution as:

(5000 / 1024.0) * 11 * 0.2


and you get 10.7421875, which matches your observation.

If your filtering (calculating the average over last 10 values), would calculate and return a float, you would get smoother values.