Interruptions are either broken or some apps disregard settings? So, I updated to Android 5 and I discovered this new feature, interruptions and priority notifications.

I set up the phone to interrupt me only for calls and messages and I went to sleep.

I was woken up by Facebook Messenger notification sound. Pissed off, I decided to check again if it was by chance whitelisted.

Is this behavior normal? Are Whatsapp, Telegram, and those kind of instant messaging app supposed to bypass this setting?
The problem stems from the fact that Android has multiple audio channels, for different kinds of output. The Priority Notifications feature will silence any apps that try to output over the "notification" channel unless they are whitelisted, preventing their notification sounds from being played.

However, not all apps actually use the proper channels. In the case of WhatsApp (and apparently Facebook and Viber), it appears that they may be using the "media" channel, which is intended for music, games, etc. These sounds do not get muted when you set your device to Priority mode because the system does not perceive them as "notifications".

The alternative (silencing all channels in Priority mode) would mean that you could not use any apps that output audio when your phone was silenced. This would effectively be like the (now nonexistent) "silent" mode. Google probably felt that was an unacceptable implementation for what they were trying to do with Priority mode, and has marked a bug related to this as working as intended. The advice given on the bug tracker was to contact the developer(s) of the offending app, because they are not actually using Android's notification API correctly.

In the meantime, you might be able to create a profile with something like Tasker that would help. Something that turned down the media volume to zero overnight might suffice, though I have not actually tested this myself yet. You could also try triggering the profile off of the notification itself, to see if you could just swallow it completely.