training deep leaning models given that we have to train multiple models and make predictions on multiple models (and then combine them via voting or averagingthis computational expense is particularly high with deep learning models with multiple layers dropout provides cheap alternative the key idea of dropout is to drop units and their connections randomly while training the network with probability and then to multiply the learned weights with at prediction time (see figure - let us make this idea precise in the form of mathematical expressions standard neural network layer can be expressed as ( )where is the outputx is the inputf is the activation functionand and are the weight vector and bias termsrespectively dropout layer at training time can be expressed as ( ( )where ~bernoulli(pand the symbol denotes pointwise multiplication of two vectors (if [ and [ , , ]then [ , , at prediction timethe dropout layer can be represented as (( xbit is easy to see that the dropout layerwhile trainingactually trains multiple networksas for every distinct rwe have different network it is also easy to see that at prediction timewe are averaging over the multiple networksas (( xbwhile training with dropout with batch stochastic gradienta single value of is used over the entire batch in relevant literaturethe recommended values for are for input units and for hidden units norm regularization found useful with dropout is max-norm regularizationwhere is constrained as || || cwhere is userdefined parameter