Why do the center tapped capacitors in Colpitts oscillator give an inversion?
In Colpitts oscillator, we have two capacitors in series, with the
  middle grounded.


Both are grounded and the pretty little way people draw Colpitts oscillators kind of implies something special about that ground node. There isn't anything special - it's ground and those two capacitors connected to ground and are no more in series than R2 and the NPN transistor's emitter.

So, C1 provides some initial phase shift by interacting with the high compliance output of the collector. In effect, the collector can be regarded as a voltage source in series with several kohms of resistance (call it Rc). So, Rc and C1 generate maybe 30 degrees of phase shift. 

Then L and C2 form another phase shifting network that can shift up to a maximum of 180 degrees so you have the basic principle of a colpitts - it's a type of semi-resonant phase shift oscillator and settles on a frequency where the total phase shift is exactly 180 degrees. The fact that the extra phase shift introduced by L and C2 is more than 90 degrees (true resonance) i.e. about 100 to 150 degrees means you have semi-resonance.

See also this answer and this answer that shows a more practical picture of the two capacitors.