How should dropout change with network depth? I know I've seen this somewhere before, but can't find it now.  Say we have a neural network with a handful of layers, and we're applying dropout to each layer.  As we move closer to the output, should dropout decrease, increase, or stay the same?
We can think of the dropout as of averaging over many networks. Not as a way to mute or denoise your data. Therefore the rule of thumb is the dropout should be applied to higher layers in your network. It is usually applied to fully connected layers (if at all!). It is a common practice to keep 0.5-0.75. neurons active. Personally I prefer L2 much more.