What does it mean that an operations amplifier is 'high gain'?
By high we mean a value that's adequate for a special purpose, typically in the range of 10,000 to 100,000. Op amps are most often used for feedback systems, therefore their open-loop gain, the gain obtained without any feedback connections, must be chosen in such a way that they meet the degree of precision that is needed for the closed-loop circuit. 

Let me elaborate more on various parameters involving opamp circuitry. The following diagram shows a feedback amplifier. The block G is called the basic amplifier, and the block H is called the feedback system. 


What the feedback system does it sense the output voltage, Vo, and output a voltage fVo, where f is the less than one. This output feedback voltage, namely fVo, is then subtracted from the input voltage, Vi, and the difference is fed back to the basic amplifier. The ratio of the output voltage to the input voltage is called the closed-loop gain, which can be obtained by simply dividing Vo by Vi:

$$V_o=A(V_i-fV_0)$$ 
and therefore,

$$\frac{V_o}{V_i}=\frac{A}{1+Af}.$$
For this equation Af must be large compared to 1. It means that it depends on applications. Sometimes even a value of A=100 might give us the desired accuracy we want. 

In the above equation as A approaches infinity the closed-loop gain tends to 1/f. This suggests that the closed-loop gain is now independent of open-loop gain variations, which is why opamps are widely used as active elements in circuits.