Will my neural network get lazy if I give it an "easy" feature? Let's say I start with a standard conv net architecture capable of 99% accuracy on MNIST such as this one, but let's say I merge in an "easy" feature to the fully-connecter layer such as a vector of length 10 that encodes the correct output digit 95% of the time and a random digit otherwise.


Will my network ever reach 99% accuracy
Will it take longer to get there?


I think the answer the question 1 is yes, because we will should always be able to find a downhill in the error before the conv net path has reached its full potential. In fact I think it may surpass the accuracy of the original architecture since we are leaking information about the correct output labels.

However I am not so sure about question 2. Does this make the shape of error function unfavorable in any way? I can't decide whether reaching the same accuracy of the original architecture will be faster, slower, or exactly the same.

Hopefully there is an easy answer, otherwise I will run an experiment and report back!
Interesting theoretical question. While I cannot answer this with 100% certainty, my own intuition and experience say:

1) I would be very surprised if this is not the case. If we look at different problems, where we have one very important feature that holds most of the information but some other features with some signal in there the model generally still improves by adding these lower value features. That is exactly what you are doing here, you have the MNIST pixels that individually hold little value but combined tell you quite a bit. The value of these pixels goes down quite a bit by adding such a strong feature, but the value is absolutely not reduced to 0. I think in theory this should lead to a strictly better model.

2) I think it will take around the same time to converge on average. Fitting to the one-hot encoded feature will be very fast which means most of the time will be spent on fitting to the features, which is a similar problem as without the feature. However, instead of going from 10% to 99% accuracy, we are now kind of going from 95% accuracy to 99.7% accuracy.

I'm less certain about the second question but I think this is what you would find if you do the (interesting) experiment. Let us know if you do implement it!