Downsampling impact on complex phase
Decimating a signal (selecting every Dth sample and discarding the rest) does not distort the signal within the passband in any way other than to cause aliases from higher frequencies to fold into the signal bandwidth.  Depending on how we model the system the phase may be effected since $z^{-n}$ is replaced with $z^{-n/D}$, but the phase will still be linear, so the signal itself will not be distorted. (In multi-rate control loop modelling applications for example I had to pay attention to this change in phase slope). For further information on this side detail see the last paragraph. 



Therefore an "anti-alias" filter is required, no different than what is done in and A/D application, just in the digital domain (decimation is a digital-to-digital converter).

The ideal decimation filter would pass the passband with no distortion while completely rejecting all the energy in the image bands. If that could feasibly be done then the decimation operation itself would be perfect (no effect on the signal in the passband). Actual performance depends on how well we design the decimation filter.  So in your case, of a signal in the second Nyquist zone (that once sampled will be in the first Nyquist zone with spectrum inverted as you describe) will not go through any further distortion with a subsequent decimation (if linear phase filter is used the result will be linear phase).



Since the folding areas are in distinct bands, a multi-band filter is a good choice for decimation filter design (firls and firpm in Matlab support the design of multiband filters).

As far as the impact on noise due to decimation itself, see this post that further details how the variance of the noise is not effected by the decimation: Properties of Up- and Downsampling

However the devil is in the details, as with any digital design you must pay attention to quantization effects, filter realization and passband distortion etc but this is not due to decimation specifically.

To further explain what I touched on in the first paragraph regarding the change in phase slope, consider a unit delay prior to a decimate by 2. A unit delay (with transfer function $z^{-1}$) has a constant magnitude over all frequencies, and a phase shift that varies from 0 to $2\pi$ as the frequency goes from DC to the sampling rate. After the decimate by two, the transfer function of the unit delay becomes $z^{1/2}$ which has a constant magnitude over all frequencies with a phase shift that varies from 0 to $\pi$ as the frequency goes from DC to the new sampling rate. So in the digital domain where we work with normalized frequencies, the slope of the phase has changed, but with regards to the related analog frequency in Hz, the slope is the same: At our new sampling frequency in Hz we see the same phase shift we got at that particular frequency prior to decimation.