What are the learning limitations of neural networks trained with backpropagation?
Multilayer Perceptron (MLP) can theoretically approximate any bounded, continuous function. There's no guarantee for a discontinuous function. There are plenty of important discontinuous functions, like, say, the prime counting function.
The prime counting function $\pi(n)$ is simply equal to the number of primes less than or equal to $n$. It has a discontinuity about each prime $p$, so good luck trying to approximate this with a neural network!
However, this function is extensively studied and extremely important in number theory. See the Riemann hypothesis.