Which encryption is recommended for own messenger
TLS, without a doubt. Note that TLS does not imply HTTP, so you can use it with whatever TCP protocol you have.

For UDP you can use DTLS, which I believe forces the use of CTR mode to combat packet loss.