What does it mean if a deeper conv layer converges first?
I am training a 3-layer convnet to classify images - a very standard problem, I know. I first tried 3 convolutional layers with ReLU, and got this: 

weights from layer 1 with ReLU - looks like edge detection

weights from layer 3 with ReLU - looks like feature detection

The first layer (16 filters) is learning edges, as expected, and the third layer (64 filters) is learning features, as expected. Then, I just wanted to try a different non-linear term so I tried sELU instead.

Oddly, the third layer seems to be now learning features, and the first layer seems to not converge at all? What does it mean for a third layer to learn edges, does it mean I need more layers? I don't see why the first layer would fail to learn edges.

weights from layer 1 with SELU - looks unconverged?

weights from layer 3 with SELU - looks like edge detection?

I don't think the architecture is super important, but I have a 180x180 black-and-white image, and the filters are all 10 x 10 with stride 2 (16 filters for layer 1, 32 for layer 2, 64 for layer 3).

First off, you're confusing terminology.


The notion of convergence applies to an optimization algorithm and whether it arrives at some fixed location in the parameter space or not. If not, it may keep going forever, either improving at infinitesimally slow rate and never arriving at an optimum, oscillating around it or straight up diverging due to numerical precision/gradient explosion issues. In other words, you can speak of your network optimization having converged but not particular filters. You do that by inspecting the training loss plot, not your kernels.
A feature, in deep learning parlance, is a general notion for, well, features - that is any pattern of interest in the data. So edges would definitely be considered features as well. Perhaps you meant texture when mentioning features?


With that covered, you're unfortunately too optimistic about the state of theory regarding neural networks. Interpretation of convolution kernels is very difficult and a great research problem. Nobody can responsibly make a general statement about which course of action you should take given the kernels you observe - there are way too many variables, from the dataset, through network architecture, to hyperparameters like learning rate. From my own experience, networks with all their kernels looking like this "noise" above can achieve very good results, at least on segmentation tasks with which I am working.

If you are beginning with deep learning (which it looks like you are), I would suggest you also look at feature maps, that is inspect the tensors of intermediate values during the forward propagation of your network - you will see how they react to different parts of your picture and may lead you to more insight. You need to remember that aside from the first layer, the further kernels look at already transformed representations of the image so inspecting them without relation to the input featuremaps will not tell you much. A more advanced technique for understanding your kernels is deep visualization. That being said, I encourage you keep doing this kind of experiments and visualizations, since they will help you develop experience and an intuitive sense for what kernels may look like, how they interact and what is to be expected and what is not.
