feed-forward neural networks types of activation functions we will now look at number of activation functions commonly used for neural networks let' start by enumerating few properties of interest for activation functions in theorywhen an activation function is non-lineara two-layer neural network can approximate any function (given sufficient number of units in the hidden layerthereforewe would always use nonlinear activation functions for solving problems within the realm of deep learning function that is continuously differentiable allows for gradients to be computed and gradient-based methods (optimizersto be used for finding the parameters that minimize our loss function over the data if function is not continuously differentiablegradient-based methods would make no progress in the training of network with gradient-based methodswe can achieve stable performance from function the range of which is finite (as opposed to infinitesmooth functions are preferred (empirical evidenceand monolithic functions for single layer lead to convex error surfaces (this is typically not consideration regarding deep learning alsowe prefer activation functions that are mostly expected to be symmetric around the origin and behave like identity functions near the origin (with thatlet' take brief look at notable options within activation functions