Disable iOS messages in-app sounds/vibration
In the Settings App -> Notifications -> Facebook you can turn "sounds" off.

This will also turn off vibrations.

It will not distinguish between whether the app is currently running or not, it'll just always be off. If you want that, then you will have to send an email to facebook requesting they change how their app works.