Cost of a partial barrel shifter
While @Janka's comment 


  In a 32-bit barrel shifter, each of these 32 input bits can be mapped to 32 different outputs. This means 32x32=1024 data paths. So we have a weight for a data path: about 5 transistors. Ignoring scaling effects, that would mean a partial barrel shifter where each of the 32 bits may only be mapped to 4 different outputs, such a shifter would wave 128 data path and cosiderung the weight of 5, it needed about 500 transistors.


demonstrates the difference in transistor counts, this doesn't say very much about the difference in power consumption or cost. 


How much does a low-power transistor on your die cost?
Does it make a difference at all, if you already have a couple million transistors?
What's the power consumption of an unused   transistor in your technology? 
How many times will you actually change the data word or the shift?
how many times will you use the individual shifts in relation to each other? Will a 24 bit shift be as common as a 1 bit shift (my guess: no!)?
Will having the full barrel shifter save energy somewhere else (e.g. a multiplier)?


So, up- and downside of any semiconductor design decision will depend on the tech you use!

What we, indeed, can say is that if we stay with the 4-of-32-possible-shifters in parallel architecture, the gate delay will be identical, aside from the shift width decode logic (which you didn't specify).