Tensorflow : What is actually tf.nn.dropout output_keep_prob?
I am trying to understand concept of output_keep_prob:

So if my example is simple RNN :

    with tf.variable_scope('encoder') as scope:
        cells = rnn.LSTMCell(num_units=500)
        cell = rnn.DropoutWrapper(cell=cells, output_keep_prob=0.5)

        model = tf.nn.bidirectional_dynamic_rnn(cell, cell, inputs=embedding_lookup, sequence_length=sequence_le,
                                                dtype=tf.float32)


My confusion is if I am giving output_keep_prob=0.5 what actually it means? I know that it makes less prone to overfitting (called regularizing) by adding a dropout. It  randomly turns off activations of neurons during training, ok I got this point but I am confused when I give

output_keep_prob=0.5 and my no_of_nodes = 500 then 0.5 means it will randomly turn of 50% nodes at each iteration or it means it will keep only those connections which have more or equal probability of 0.5   

keep_layers whose probability =&gt;0.5  


or

turn off 50% randomly nodes unit at each iteration ??


I tried to understand the concept by this stackoverflow answer but there is also same confusion that what actually 0.5 means? it should drop 50% nodes at each iteration or keep only those nodes which have probability more or equal to 0.5 

if the answer is second keep only those nodes which have probability more or equal to 0.5 :

then it means suppose I have given 500 nodes units and only 30 nodes have 0.5 probability so it will turn of rest 470 nodes and will use only 30 nodes for incoming and outgoing connections?

Because  this answer says :


  Suppose you have 10 units in the layer and set the keep_prob to 0.1,
  Then the activation of 9 randomly chosen units out of 10 will be set
  to 0, and the remaining one will be scaled by a factor of 10. I think
  a more precise description is that you only keep the activation of 10
  percent of the nodes.


While other side this answer by @mrry says :


  it means that each connection between layers (in this case between the
  last densely connected layer and the readout layer) will only be used
  with probability 0.5 when training.


can anyone give a clear explanation which one is correct and what actually this value represent in keep_prob?

Keep_prop means the probability of any given neuron's output to be preserved (as opposed to dropped, that is zeroed out.) In other words, keep_prob = 1 - drop_prob.

The tf.nn.dropout() description states that


  By default, each element is kept or dropped independently.


So if you think about it, if you have a large amount of neurons, like 10,000 in a layer, and the keep_prob is let's say, 0.3, then 3,000 is the expected value of the number of neurons kept. So it's more or less the same thing to say that a keep_prob of 0.3 means to keep the value of 3,000 randomly chosen ones of the 10,000 neurons. But not exactly, because the actual number might vary a bit from 3,000.

Scaling comes into the picture because if you drop a certain number of neurons, then the expected sum of the layer will be reduced. So the remaining ones are multiplied to feed forward the same magnitude of values as they would otherwise. This is especially important if you load a pretrained network and want to continue training but with a different keep_prob value now.

(Please note, you can decide to introduce non-independence into the drop probabilities with the noise_shape argument, please see the tf.nn.drouput() description, but that is outside of the scope of this question.)

The random decision to drop a neuron or not is recalculated for each invocation of the network, so you will have a different set of neurons dropped on every iteration. The idea behind dropout is that subsequent layers cannot overfit and learn to watch for arbitrary constellations of certain activations. You ruin the "secret plan of lazy neurons to overfit" by always changing which previous activations are available.
