Can we change the structure of the feature-extraction layers of deep networks architectures?
You can change whatever you like!

The benefits will depend on your data and what exactly you are comparing to. As you didn't say exactly what sequence you have as your starting point, I can't compare anything exactly, but provide a simple outline.

Generally, you follow the following recipe:


Input layer: perform any reshaping or normalisation of your data
Multiplication layer: either fully-connected layer, a convolutional layer or something else
Non-linearity: relu, tanh - gives the network its power to fit non-linear functions
regularisation layer: batchnorm, dropout - help models converge and prevent overfitting


This scheme is then repeated as many times as you feel is necessary.



Doing something like applying several ReLU layers, one after another, definitely doesn't make sense, as it would actually change anything! A ReLU simply filters out negative values, setting them to zero. Applying it a second time would therefore have no effect.