What can't be done with a neural network
This paper introduces a finite size neural network and proves that it can simulate a Turing Machine, meaning that this NN is Turing complete. Thus anything that may be computed using TMs can be computed by this type of NNs. You may want to read this too.