Can we learn f(x)=1/x using a neural network exactly? Is there a way to train a neural network as $f(x) = {1 \over x}$ precisely?
Since the activation function itself can be an inverse function, the answer is yes.