Could modulation/coding schemes be generated from an AI?
As per my knowledge, one of the recent papers on this topic can be found here, where the authors used machine learning algorithms to generate the optimal constellation shaping. Surprisingly, the results were in line with an old work: F. R. Kschischang and S. Pasupathy, "Optimal nonuniform signaling for Gaussian channels," in IEEE Transactions on Information Theory, vol. 39, no. 3, pp. 913-929, May 1993.

Moreover, autoencoders, neural networks, reinforcement learning and deep learning are being applied for end-to-end learning (especially in the case of wireless communications), where the mathematical model of the wireless communication channel is not very accurate. One such article can be found here. 

Also, for the case of polar codes, deep-learning-based decoders are very popular nowadays at least in academia.