If you put a Mac in Lost Mode while FileVault is enabled, does it prevent OS X reinstalls?
Filevault prevents the theft of data not the device. If they boot to recovery partition or jump drive they can wipe and reinstall and your data is still safe. if you want to prevent that from happening you need to add a firmware password. So this would be the best combo 


filevault
firmware password
lost mode


however if they know what they are doing they can still open the machine and take it from there. 

There is no 100% fool proof way other keeping your mac with you