Op-amp circuit with feedback: does the polarity matter?
It is not correct to say a opamps are designed to work like comparators. They are two different components and though fundamentally similar, they are not the same. 

Positive feedback tends to allow op-amp behave like a comparator with hysteresis or Schmitt triggers. That said, an op-amp has so much open loop gain that it will behave like a comparator if there is no feedback of any kind, just one that is slow because it is not optimized to work in saturation.

The polarity of the inputs always matters regardless of circuit. When the voltage at the non-inverting input is higher than the voltage at the inverting input, the op-amp output rises. When the inverting input is higher than the non-inverting input, the output falls.

This means that if you connect output to noninverting input, you get positive feedback. The output reinforced the input and the input reinforces the output causing it to climb without limit and saturate unless it is held in check by negative feedback elsewhere. This is a comparator-like operation.

This also means that if you connect output to inverting input, you have negative feedback. The output opposes the input and the op-amp finds an equilibrium between the two. This is how an op-amp is optimized to work (in the linear region, and not in saturation).

The ENTIRE reason the "virtual ground" exists is precisely because negative feedback. The negative feedback causes the output to drive the inverting terminal to be equal to the non-inverting terminal (which is tied to ground). If you reverse the input polarity then you can no longer assume that the the inverting and noninverting terminals are equal because there is no longer any negative feedback to produce a virtual ground.