Constructing a custom protocol using well-vetted algorithms?
You might want to start by reading Lessons learned and misconceptions regarding encryption and cryptology.  I spotted 3 potential issues in your proposal that are explained there.

Instead of designing your own cryptosystem, you should try hard to use an existing well-vetted scheme, like TLS or DTLS for interactive communication, or GnuPG for storage or store-and-forward messaging.  The fact that communications are routed through a proxy server doesn't change this recommendation; you can still do end-to-end cryptography.

(If you are building your own protocol, you to use authenticated encryption, and deriving cryptographic keys from a passphrase tends to provide poor security.)

Since you haven't given us any specifics why you can't use existing schemes, it's going to be tough to provide you with more detailed advice.