What makes multi-layer neural networks able to perform nonlinear operations?
Nonlinear relations between input and output can be achieved by using a nonlinear activation function on the value of each neuron, before it's passed on to the neurons in the next layer.