What happens if I use unencrypted time machine backup from a mac that has FileVault turned on?
In your current setup your backups are not encrypted whatsoever. That means if someone steals your external disk your data is fully accessible to an attacker. That challenges the idea of using full disk encryption / Filevault2 in the first place.

I would recommend to either


Create an encrypted sparse bundle image on your external disk and use that as your backup drive. The process is somewhat tricky as it requires tinkering with the sparsebundle image. The process is explained here and here.
Encrypt the external disk entirely. The disadvantage is that you will need the password to open that disk on other Macs. The advantage is that the entire content will be encrypted. To do this simply right-click the disk and select Encrypt...


Either way make sure you note the encryption passwords for Filevault and the backup disk somewhere else than on your computer or the backup disk, as you risk locking yourself out of all your data and backups.

As mentioned in the comment there is the downside that encryption slows down disk performance. That is also true for Filevault2 disk in your Mac. But Intel CPUs are pretty optimized for encryption operations these days and performance degradation should be low.