Do i delete the Bitcoin folder & data directory in Appdata after i copy it on an encrypted drive?
Make sure to delete wallet.dat - that's the only file with sensitive information. (You might want to run a utility such as unix shred on it to make sure it's really deleted).

Just a warning - before deleting, rename the file, and try to copy it from your backup and see if everything works.