How could AGI build its own communication protocol?
This is a purely theoretical question, currently in the realm of philosophy and speculative fiction.  Nevertheless, it is an interesting question, and may be instructive. 

If we use the standard definition of Artificial General Intelligence as automata with human level intelligence, then they could certainly devise their own communications protocols, just as humans have.

1) These automata are AGI so they are creative and resilient, just like humans, and 'where there is a will there is [sometimes] a way." Absent robotic capacity they wouldn't be able to build anything physical and would have to rely on existing communications infrastructures. If they had access to 3D printing and versatile robots, they could probably build something new, but this would be infeasible for anything that requires extensive capital outlay, unless the automata first acquired a major communications infrastructure firm or two. 

2) It's always interesting (and usually quite entertaining) to observe chatbots conversing with each other, but if they were AGIs, NLP is just one of many functions, and I doubt they would bother conversing with each other in human language, since all data is ultimately reduced to a string. If they were smart enough to be deemed AGI, they would certainly communicate in-species with the most efficient protocol available, and probably optimize it further, if possible, or create a unique protocol for purposes of exclusivity. 

If they are truly AGIs, you wouldn't ostensibly have to induce them to communicate, because they'd be smart enough to understand the benefits of communication and cooperation, and would likely seek to form coalitions as a natural survival function. (Game Theory provides a mathematical basis for this.) 

Multi-agents systems can self organize, even where the intelligence of the given agents is low, and in your scenario, the automata are smart.