Are there any mathematical advantages for higher sampling rates?
I always use double sampling rates if possible, for two important reasons.

First reason: to get rid of the characteristics of the anti-imaging filter when working with analog sound sources. What is an anti-imaging filter?

Let's say I am recording on 44100 Hz.
If I would record a sine wave of less then 10 KHz, you could clearly see the sinewave when you plot the sample values in a graph.
If I sample a sine wave of 0dB FS with a 22,5KHz frequency, the samples read 1 and -1 alternately.

Now, here's the problem. If I record a sine wave of 0dB FS with a 30 KHz frequency, and plot the samples, each sample is taking more than half a sine period, and - if you would play back the samples - it would return an 11KHz sinewave. (If you don't believe me, just make a simple drawing.) This behaviour is called the 'imaging effect'.

This means that before sampling the signal, we have to be sure that there are NO frequencies present what so ever above the so-called "nyquist frequency" (which is half the sampling rate). When using digital sound sources that provide their sounds already sampled, this is not really that big a deal, since they can sometimes just be programmed to never generate a signal above half the sampling rate, or they can filter everything out using a linear-phase brickwall filter that has no effect on the rest. 

But, if you are sampling signal from an analog source, this filtering is done before the signal is sampled. The only way to filter analog sound is by use of an electronical circuit. And since the filter is supposed to have a very steep curve, it will affect the frequencies within the audible range, even though the filter was not designed for it. Now there are quite some good filters inside A/D converters, so the problem is minimal, but it gets relatively irritating to listen to when you are working several days on 44.1 KHz audio, compared to using 96KHz. The filter that is going to be applied when you downsample 96 back to 44.1 is of course a digital filter, and is probably of a much better quality. And, it is only applied when you are completely done with all the work, so it won't bother you.

Second reason: to get rid of the characteristics of the dithering signal. 

When you are recording in 24 bits resolution and you plan to have your master at 16 bits, you will need a dithering signal to mask away the rounding errors. Now noise is not a pretty thing to have in your recording and while broadband noise is best for masking rounding errors, noise shaping can be a big improvement applied to the dithering signal in order to make it less disturbing. Now if the recording was made using 96KHz, you can noise shape most of the dithering signal to frequencies higher than 24KHz, so nobody will hear them. The dithering noise is at the end of the recording finally filtered out, at the moment you downsample your project back to 44.1 KHz.

So, bottom lines:
Is it useful when recording analog stuff:


Yes, definitely. You have less disturbance from the anti-imaging filter and less disturbance from the dithering signal when used with proper noise shaping.


Is it useful when working with digital stuff that came right of my softsynth?


Yes, still useful if you plan to work with 24 bits, and mastering it down to 16 bits. You can gain a great deal with noise shaping the dithering signal.