Which filter is good for GSM transmeter design Butterworth, Chebyshev or Bessel?
Which part of the transmitter? There will be many, many filters in use in various ways and locations (some of them digital). 

Loop filters in the reference timebase have very different requirements to anti alias filters after the I/Q dacs, which have different requirements to the RF BPFs ahead of the mixers and after the PAs, which have different requirements to the filters in a slow drain modulator, what have different.... Well you get the idea.

I would suggest that asking this probably means you are not ready to design a GSM transmitter, start with something very much easier.

To answer the question asked, a GSM transmitter will probably use all of those and more, they are all best in different parts of the design.