why GMSK use a gaussian filter instead of raised cosine filter?
(Assuming the context is FSK signals, where the pulse shape is applied to the phase, instead of the amplitude as in linear modulation).
The main reason is that the sideband and out-of-band emissions of GMSK are lower than those produced with a raised cosine filter. See for example this plot from Wikipedia: https://en.wikipedia.org/wiki/File:GMSK_PSD.png
This property is especially important for mobile communications. Since spectrum is so limited, it has to be used very efficiently. Users may use channels that are adjacent to each other, with little or no guard band between them. Any power emitted outside the band assigned to a channel resuls in degraded SINR for other users in the cell.
GMSK requires a more complex receiver, because it reduces bandwdith at the cost of introducing ISI. However, in many scenarios this is an acceptable tradeoff, since bandwidth is more valuable than receiver simplicity.