When downloading the blockchain my application is become unusable, is there an issue with my code or am I using the BitcoinJ library incorrectly?
The downloadBlockChain() method looks like it downloads the whole blockchain. It looks like the PeerGroup.setFastCatchupTime method might be helpful to prevent it from downloading full blocks before that time. I imagine you'll use a method or change a boolean value to only download block headers for your SPV wallet, but I don't know where that would be.

https://bitcoinj.github.io/speeding-up-chain-sync may be a useful reference.