Theoretically, can everyday computing tasks be broken down into ones solvable by a neural network?
Neural networks can arbitrarily approximate any computable function; see
http://en.wikipedia.org/wiki/Artificial_neural_network
or
http://en.wikipedia.org/wiki/Cybenko_theorem

Constructing a NN for a given function -- or worse yet, learning it from examples -- is a hard problem.