How can I make it so my cyanogenmod phone is silent for everything but calls and alarms? I like to keep my phone on silent at night so I can actually sleep without disturbance. The problem with that is that everything becomes silent. No one can reach me in case of emergency or anything. 

Is there a way to put my phone in a "calls only mode" or something where everything is silent except phone calls and alarms?

I have the One Plus One with the latest android cyanogenmod available for that.
What you are looking for is Cyanogen User Profiles which can be found in Settings->Profiles under the Personal section header. Pick a profile or create a new one and select what parts of your phone are active.