Why are convolutions still used in some Transformer networks for speech enhancement?
We find some justifications in the Conformer paper:

Convolutions are better than Transformers at detecting fine-grained patterns:

While Transformers are good at modeling long-range global context, they are less capable to extract fine-grained local feature patterns. Convolution neural networks (CNNs), on the other hand, exploit local information and are
used as the de-facto computational block in vision.


Together, they Transformers and convolutions work better than separately:

Recent works have shown that combining convolution and self-attention improves over using them individually [14]. Together, they are able to learn both position-wise local features, and use content-based global interactions.