Designing a secure IM protocol
I think the very first question before diving into designing the protocol would be to ask yourself what kind of secure communication do you want? Namely, there are two options to consider when it comes to communication between the IM users:


For the client-server secure communication, there are two directions to consider: 


From “sender to server”, you would have to encrypt the message with the public key of the server (assuming using public key encryption). 
From “sender to recipients”, the server would encrypt each message with the respective recipients' public key (assuming using public key encryption). However, if you are consider using symmetric encryption, a good reference for protocol would be the SSL/TLS protocol that is used widely. 

For the end-to-end secure communication, it would be difficult to implement for group message but simpler for individual message. It means that the sender will encrypt the message with a predefined encryption key, and only people who know the encryption key is able to decrypt the message. Sharing the predefined encryption with a group might compromise on the overall security. 


Once decided on the framework type, you can add additional information to the payload (i.e. message) to ensure there is authentication, non-repudiation, data integrity, etc. A good reference would be the SSL/TLS protocol.