Neural Networks: what's the point of learning features that don't linearly separate?
Indeed, one would like to construct features that make classes linearly separable. However, what patterns exactly should be linearly separable?

It is easy to make patterns in training set linearly separable by greatly increasing feature space dimensionality. However, this will not necessarily imply good generalization (correct classification of new patterns).  Quite opposite, it can lead to overfitting. Thus, it is problematic to use the criterion of linear separability directly to select good nonlinear features for classification. 

Deep learning networks adopt different approach. They try to select features with good generalization of patterns irrelative to the classification task. Thus, “useful” nonlinear features are selected without danger of overfitting. Indeed, it doesn’t necessarily ensure linear separability of classes, but it makes sense from induction point of view. If one extract statistically independent factors from initial features, in which these factors are mixed, it will be easier to solve any further classification task.