Activation Functions
A combination of multiple linear functions can help one model complex decision regions. Two or more linear functions can be combined to form a piece-wise approximation of a non linear function. Computation of non-linear activation function on a very deep network can be expensive (because of the exponential term involved). ReLu activation function which is a linear activation is used almost by every CNN model and is proved to give better results.
$$ f(x) = max(0,x-1) + max(0,x+1)$$
Consider the function above which is combination of two reLu's.  This approximates the sigmoid function(try plotting the same) and is computationally cheap.