Duplicated features for gradient descent
In 'Efficient Backprop' by Lecun and others (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), they explain why correlated variables are bad (§ 4.3 normalizing the inputs). 

Duplicated data is a special case of linear dependence, which is a special case of correlation. Say you have duplicated variables $X1 = X2$, so the network output is constant over the weight lines $w_2 = c - w_1$, where c is a constant. It means that the gradient of the error is 0 along those lines : moving along those lines has no effect on the learning. In that sense the solution won't be unique. 

It's bad because you could be solving a problem of lesser dimension. Removing one of the input will leave you with a network of lesser dimension. 

Other than the dimension problem, for simple learner I don't think it will affect the learning process that much. For more complex learning processes (learning rate depending on time / on variables), it might get more complex.