Does the RMS value for a non-periodic signal exist?
Does the concept of RMS VALUE applicz (sic) to a non periodic continuous
  time signal? If yes how to calculate it?


RMS is rigorously defined for an infinite (not necessarily periodic) time-signal as
$$\lim_{T\to\infty}\sqrt{\frac{1}{2T}\int_{-T}^T (f(t))^2dt}$$

or for signals starting at \$t=0\$ (\$f(t)=0\$ for \$t&lt;0\$) 
 as$$\lim_{T\to\infty}\sqrt{\frac{1}{T}\int_{0}^T (f(t))^2dt}$$

so RMS is quite applicable to nonperiodic signals. When the signal is periodic though, the calculation can be done over a single period (since it's time-averaged) and the above equation reduces to 
$$\sqrt{\frac{1}{T_2-T_1}\int_{T_1}^{T_2}(f(t))^2dt}$$
and even for specific signals we can make further simplifications, such as that for sinusoidal signals the RMS is just the peak voltage divided by \$\sqrt{2}\$.


  What value does an ac voltmeter display for an input of non periodic
  square pulse of say 5V peak?


Your voltmeter won't be doing anything as analytic as the first equation though, since it's only sampling a signal and thus can't know it's behavior into infinity. Instead they are designed to do the time-averaged approach under the assumption that you will only ever want RMS for a periodic signal. If the signal you feed is nonperiodic, the voltmeter readout will be next to meaningless in most cases as it's probably attempting to guess the interval (\$T_2-T_1\$) over which to average. This is why you see the value fluctuate on the readout.