Secure instant messaging protocol and implementation for an anonymous end-to-end chat? all is said in the title but I will go further.
First question :
I searched here and I found Tox to be the most powerful protocol. It uses Curve25519, XSalsa20 and Poly1305. They seem to be good algorithm but I think I will replace XSalsa20 for a ChaCha20-Poly1305 combination.
What are you think about that ?
Second question :
For the implementation this is my first thougth :
A ask for a secure communication to the Server which register the channel on his DataBase and cipher it with A's symmetric key. When B ask to connect on the channel, the server will send the request to A with his public key, A will cipher symmetric key of the channel (unknown by the server) and send to B back. The server will finally add B to the DataBase of the channel. If B want to send a message he need to cipher it with one of the symmetrical key and send the coded text to the server wich will decipher the DataBase with the key and send to every ip addresses. A will decode with the other symmetric key.
I know it's unclear, if you have any questions feel free to ask them, I want to know if you got better idea. The problem of my system is that the server will keep ip addresses of the channel's owner and the user (while he connect to the server) until all the users are connected. But I don't think it's possible to do it without that.
I searched here and I found Tox to be the most powerful protocol.


The reason, why I'd advise against Tox and for the Signal Protocol is not a one of features.

Tox has seen not much review, especially not from cryptographers. The core library is also still in alpha. In fact, an audit is still an open issue for them on GitHub, so I don't have too much trust in their security for now.

The Signal Protocol on the other hand, has very wide-spread deployment (in apps like Signal, WhatsApp, Google Allo and Facebook Messenger) and thus has seen a lot of scrutiny. The creators also have conducted security audits of their software and academics have given it a lot of attention.


  What are you think about that ?


From a purely cryptographical stand point this is totally fine, except maybe that ChaCha hasn't yet seen as much scrutiny as Salsa20 (and XSalsa20) have yet. But ChaCha is considered to be just as secure.
However, doing this will make you incompatible with all other existing implementations of Tox which will make it "harder" to integrate support for your product into other, potentially harming adoption rates.
Secondly, I assume you'd use the Tox-core library and changing stuff in those libraries is always dangerous and because of the change you may introduce a new vulnerability.


  A ask for a secure communication to the Server [...]


You don't need a centralized server, that's one of the points going for Tox. Everything can be done peer-to-peer (mostly).

As for your actual protocol there are a few issues:


From what I can tell, it's trivial for the server to serve bad public keys and MitM attack connections (this also depends on whether the UI will allow to verify public keys though)
IP adresses can change over time, not ideal for a chat application (depending on how long they need to stay valid)
Depending on how connection establishment works, users sitting behind drop-all firewalls and NAT routers may also have problems
The server learns the social structures of the users (eg who groups up with whom)
You lack a threat model (e.g. what do you want to achieve? Who is considered a threat?)


Most of these issues can be mitigated server-side using software (like don't store IPs, don't store grouping, keep the incoming connections alive) only the mitm attack mitigation would need client-side action.

And of course this quick check by me doesn't replace a full review by a professional cryptographer.