Downsides of encrypting disk with FileVault
The implementation of Corestorage / Filevault2 is pretty great. That means you will hardly notice that your system is encrypted. I see the following points you may want to consider:


As you say: If you loose the passcode it is next to impossible to access your data. 
There is a very slight performance hit because the CPU needs to constantly encrypt and decrypt your data. But moderns CPUs are optimised to do so and therefore the impact is negligible. 
If your disk is corrupted or faulty in some way it may be more difficult to fix a corestorage volume than a standard HFS volume. Personally, this has never been a problem for me. 
Passwordless boot or waking from standby is not possible anymore.
The most obvious difference is pre-boot authentication. That means you need to enter your password before  the Mac boots up. Without encryption the system would boot and only require your password to log into an account. 
Depending on your level of paranoia Filevault2 alone may not be enough. You should consider to also encrypt your backups (even higher risk for total data loss) and activate the option to destroy the password from RAM while the Mac sleeps.