Why would I use an inverting amplifier?
Inverting amplifiers have less input impedance. This is because of the feedback resistors. A non-inverting amplifier has very high input impedance, because the signal is applied directly to the positive terminal.

For a low noise amplifier you want the input impedance to be low. This is because the noise current can generate large voltage drops otherwise. See this TI app note for the op-amp noise model.

Of course you can always put a resistor to ground on the positive terminal in order to decrease the input impedance and/or match for bias current. 

For a low noise amplification I would use a transistor amplifier anyway. You can get transistors with tiny noise figures. The only reason not to use them is if you need a DC coupled amplifier. You also benefit from Friis' noise equation.