Filtering vs. decimation
Let us define the terms first:


Downsampling means reducing the sampling rate of a signal. If there is energy outside the new Nyquist frequency, there will be aliasing.
Filtering means eliminating all energy in some frequency bands.
Decimation means anti-alias filtering a signal and then downsampling it.


In a typical SDR application, your captured bandwidth is larger than the actual band of interest. There are two things you want to do:


Eliminate all signals outside the band of interest (by filtering).
Downsample to the minimum required rate, in order to reduce the computational complexity of the baseband processing algorithms.


One common way to accomplish this is the way you describe: downconvert the band of interest to DC, or to a low intermediate frequency if DC offset is a concern, and then decimate.

There could be a few reasons why you're seeing that filtering is slower than decimation. One is that maybe your filter order is larger than necessary. Another is that there are algorithms that perform low-pass filtering and downsampling simultaneously and with very low complexity. See for example half-band filters.