Why does adding random pixels stop my model learning in cnn?
I think the problem here is that you are trying to relate what you know about the input to what the model is trying to predict. I suggest trying to interpret why your model is giving certain outputs for images.

I came across SHAP a while back when reading about model interpretation. The image plot of SHAP should be able to narrow in on why your model is interpreting a label for a specific Image.



This should lead you to figure out why your model is making the prediction.