How can I set custom notifications?
Typically the apps themselves will need to provide a setting wherein you can pick a notification sound to use. I am not aware of any global settings that will allow you to set per-app notification sounds.

As an example, in Gmail you can go into the settings (use the overflow menu - the 3 vertical dots - and select "Settings") and select an account from the list. Then within the account's settings tap on the notification setting right under the "Email notifications" checkbox. This will allow you to pick a different ringtone for emails on that account.

You can then repeat this process for any other apps you have installed.