Dropout causes too much noise for network to train I am using dropout of different values to train my network. The problem is, dropout is contributing almost nothing to training, either causing so much noise the error never changes, or seemingly having no effect on the error at all:

The following runs were seeded.

key: dropout = 0.3, means 30% chance of dropout

graph x axis: iteration

y axis: error

dropout=0


dropout = 0.001

dropout = 0.1

dropout = 0.5


I don't quite understand why dropout of 0.5 effectively kills the networks ability to train. This specific network here is rather small, a CNN of architecture:

3x3x3                    Input image
3x3x3                    Convolutional layer: 3x3x3, stride = 1, padding = 1
20x1x1                   Flatten layer: 27 -&gt; 20
20x1x1                   Fully connected layer: 20
10x1x1                   Fully connected layer: 10
2x1x1                    Fully connected layer: 2


But I have tested a CNN with architecture:

10x10x3                  Input image
9x9x12                   Convolutional layer: 4x4x12, stride = 1, padding = 1
8x8x12                   Max pooling layer: 2x2, stride = 1
6x6x24                   Convolutional layer: 3x3x24, stride = 1, padding = 0
5x5x24                   Max pooling layer: 2x2, stride = 1
300x1x1                  Flatten layer: 600 -&gt; 300
300x1x1                  Fully connected layer: 300
100x1x1                  Fully connected layer: 100
2x1x1                    Fully connected layer: 2


overnight with dropout = 0.2 and it completely failed to learn anything, having an accuracy of just below 50%, whereas without dropout, its accuracy is ~85%. I would just like to know if there's a specific reason as to why this might be happening. My implementation of dropout is as follows:

activation = relu(val)*(random.random() &gt; self.dropout)

then at test time:

activation = relu(val)*(1-self.dropout)
why dropout of 0.5 effectively kills the networks ability to train.


because that is too mutch normal values are like 0.15-0.05. Imagine, the 50% of input image is randomly set to 0, THEN it happens on next layer, means in average 25%  remains, etc... also if you have small dataset with too different images for each class, this + drouput wil confuse the network. 
Also your CNN setup is not realy rational. Too much fc layers, replace one or two fc to convo layers.  And i d say the reason of using 4x4 is only with stride 2, else use 3x3. And you sould use batch normalisation and augmentation like small noice would be probably petter then dropout in your case.