Does a multilayer perceptron taught by simple backprop learn the shape of a character or the exact image of a character?
I would be very surprised if a backpropagation-trained multilayer FFN spontaneously learned concepts of shape. It's not that the notion of "shape" isn't representable by a FFN (given enough layers, anything is). It may even be the case that the globally optimal configuration does capture something about shape. But there must be a strong local minimum about the exact-image replicator, meaning that typical backpropagation (i.e., gradient descent) algorithms will get stuch in such local minima. Unless you've specifically primed your FFN to learn shape, I would be very surprised if it did.