How to group chat messages by topic?
This problem is related to the following standard problems:


topic classification/modeling, which ranges from simple supervised document classification to unsupervised assignment of topics distributions to every document (the advanced option, with Latent Dirichlet Analysis and variants) 
sequence labeling, a supervised task which predicts classes for every instance in a sequence, taking into account the order of the instances (e.g. it can leverage the fact that the class of instance $n$ is influenced by the class of instance $n-1$). 
text segmentation, more precisely topic segmentation in this case.


I think that one can find a lot of good implementations for the first two problems, which are very common. However adapting these to your case and/or combining them will probably be more complex.

If you have a small set of topics and you have (or can have) a reasonable sample of  messages annotated with these topics, I would suggest starting with sequence labeling with Conditional Random Fields. I think this could give good results given the sequential nature of chat conversations.