overfitting and underfitting adding dropout dropout is one of the most effective and most commonly used regularization techniques for neural networksdeveloped by geoff hinton and his students at the university of toronto dropoutapplied to layerconsists of randomly dropping out (setting to zeroa number of output features of the layer during training let' say given layer would normally return vector [ for given input sample during training after applying dropoutthis vector will have few zero entries distributed at randomfor example[ the dropout rate is the fraction of the features that are zeroed outit' usually set between and at test timeno units are dropped outinsteadthe layer' output values are scaled down by factor equal to the dropout rateto balance for the fact that more units are active than at training time consider numpy matrix containing the output of layerlayer_outputof shape (batch_sizefeaturesat training timewe zero out at random fraction of the values in the matrixlayer_output *np random randint( high= size=layer_output shapeat training timedrops out of the units in the output at test timewe scale down the output by the dropout rate herewe scale by (because we previously dropped half the units)layer_output * at test time note that this process can be implemented by doing both operations at training time and leaving the output unchanged at test timewhich is often the way it' implemented in practice (see figure )layer_output *np random randint( high= size=layer_output shapelayer_output / note that we're scaling up rather scaling down in this case dropout * at training time figure dropout applied to an act ivat ion mat rix at raining timewith rescaling happening during raining at est timet he activat ion matrix is unchanged this technique may seem strange and arbitrary why would this help reduce overfittinghinton says he was inspired byamong other thingsa fraud-prevention mechanism used by banks in his own words" went to my bank the tellers kept changing and asked one of them why he said he didn' know but they got moved around lot licensed to