Removing Bitcoin from my phone to secure storage I am using the following Bitcoin wallet for Android:

https://github.com/bitcoin-wallet/bitcoin-wallet

My device is encrypted and I have encrypted backup files on Google Drive.

However, I'd like to get my coins off my phone altogether, just in case its stolen and the UI is still accessible. I'd like to keep a small amount on my phone, and top up that balance every now and then.

What's the best way to do this?

I think I need a separate "long term" wallet, but that's just moving the same problem to somewhere else.

Is a hardware wallet the only way to go here?
You are overthinking this.  The easiest way to move some of your bitcoin to a different wallet is simply to send it to an address belonging to the second wallet.  This works, regardless of what type of wallet the second wallet is.

For example, if you use Bitcoin Core on a desktop in your house, simply request a payment, which will show a QR code on your screen.  Scan it with your phone, and send whatever amount of bitcoin you want to your desktop wallet.  Yes, there will be a transaction fee, but it will be a lot less value than the time/risk associated with importing private keys into Bitcoin Core and then purging them from your phone wallet.

As for the most secure wallet for long term storage...there's some debate on that.  Here's an earlier answer I gave on the subject:  What options does one have to manage Bitcoins?