About designing digital filters
Filtering can be done in the frequency domain which is actually a very efficient technique (and it can very well be, that Matlab does this internally). However, for very long signals it's not as straight-forward as "taking the FFT and applying a frequency response". You can read up on overlap-add filtering which is a possibility to filter in frequency domain. However, this is mainly done because it can be faster.

However, this question is independent from filter design which is a completely different thing. Every FIR filter has a corresponding frequency response and it does not matter whether you convolve in time domain or apply the frequency response in frequency domain. So your approach becomes a question of filter design and the consequences it has.

Just setting unwanted frequencies to zero might completely eliminate those frequencies but it usually comes at the price of a significant ringing in the time domain which is usually unwanted (Wikipedia on ringing). So in fact, what you are proposing is just one way to design a filter and frankly not a good one in most cases.

By allowing a transition band in which the frequency response can gradually go from passband to stopband, degrees of freedom are gained that can be used to improve other properties of the resulting filter (for example eliminate the ringing or obtain a shorter impulse response). That's why Matlab implements so many different filter types, they all have different properties and selecting the most suited one is actually part of designing your signal processing system.

This topic is actually quite complicated and I suggest to read up on filter design. Whole books can be filled with this.