bitcoin-cli very slow to respond to RPC calls (Raspberry Pi)
I have the same set up and getinfo for me takes about 10-20s initially but then takes ~5s there after. (I need to restart the daemon periodically as it crashes for some reason). I'm running 0.11.1 and haven't made any special tweaks. gettxoutsetinfo takes around 12 minutes. CPU, RAM and Swap comparable with what you have as well. 

Hardware is probably the most obvious limitation. You could overclock the Pi CPU but I doubt it will drastically reduce the response time though.