What is the gain margin of the system, when the phase crossover frequency does not exist?
What is the gain margin of the system, when the phase crossover
  frequency does not exist?


You use the word "exist" and in any circuit that "exists", due to the speed of light being finite, there will always be a frequency where there is an inversion of signal aka 180 degrees phase shift.


  I have a system in which the system will never reach -180 degrees for
  any frequency.


No you don't - not in this universe.


A signal with a frequency of 300 MHz has a 1 metre wavelength (or less) hence: -
A 50 mm long wire at 3 GHz produces an end-to-end phase shift of 180 degrees. 
At 30 GHz, 180 degrees phase shift is obtained with a 5 mm long wire.