Most efficient way to calculate fee from getrawtransaction?
To calculate fees.
Take sum of all vouts.For every vin there will be prev txid and outputindex,
Get this txid with getrawtransaction or some other way,go to vouts of this new tx at index, here you will get amount,
Repeat this for every vin.
Take sum of this vins. 
Take difference between vin and vout you will get fees.

To avoid calling ever time you need to store txids somewhere or user something like insight.