How good is AI in math?
Nim was actually one of the first games ever played by an electronic machine. It was called the Nimatron and was displayed at the 1940 New York World's Fair.
It is also well known that neural networks can model the Xor-function, if they have enough layers. Despite that, Marvin Minsky is supposed to have killed neural networks in the sixties, by asserting that networks with a single hidden layer, cannot model XOR.
I think there were also some large scale projects to classify finite groups using computers.
Additionally, there have long been automated theorem-proving programs, that had some success in mathematical logic. I recently saw a paper that used deep learning to improve one of these theorem provers.
That being said, doing real mathematics is probably as difficult as real language understanding. There doesn't seem to have been a real breakthrough yet.