Filtering an acceleration signal
I believe the best strategy is to filter prior to calculating the magnitude. To see this easily, consider the low pass filtering as an averaging process and consider the noise as a zero mean Gaussian white noise. The distribution of the noise prior to the magnitude computation as zero-mean Gaussian will average to zero if the noise is white. The distribution of the magnitude of a Gaussian process is Rayleigh which has a maximum at the variance $\sigma$ of the noise distribution with an average of $\sigma \sqrt{\pi/2}$.  With a signal present the distribution is Ricean with similar considerations.

Filtering after will help in removing the variability of your result to the extent the system is stationary but will not reduce the error due to the noise; simple case in point is as above with noise only (and stationary), the result converges to zero; while after the magnitude the result will converge to a constant proportional to the noise level.

I suspect for both filters you will be limited by the dynamics involved in how much you can filter, which is ultimately the trade you will make with desired SNR of your result.

For more details on this specific to filtering before or after computing the magnitude (or equivalently before or after decision) see this great app note by Agilent:

http://my.ece.ucsb.edu/York/Yorklab/Useful%20Stuff/Tutorials/SpecAnalyzerMeas-Noise.pdf

There are applications where it is theoretically better, such as when the phase of the information is not preserved.

Here is a decent reference on both cases:

http://www.acfr.usyd.edu.au/pdfs/training/sensorSystems/10%20Detection%20of%20Signals%20in%20Noise.pdf

If you don't want to read or understand the reference materials in detail, or have little time- I suspect filtering prior to taking the magnitude will be better but you can easily test this by establishing SNR metrics on your result; just pay attention to rate of change as well with regard to how fast your system will be changing and how fast you want to be able to track that versus averaging out those changes.