Classic Neural Network Layout for regression: is the score function derivable?
It depends on what activation function you use.  With all standard activation functions, neural networks are continuous.  With ReLU activations (the most typical choice of activation function), they are piecewise differentiable.  The partial derivatives (the gradient) can be computed efficiently (using so-called "backpropagation"), and standard neural network frameworks will automate this process for you, as computing the derivative is part of how neural networks are trained.  Consequently, it is possible to use gradient descent methods to find $x$ that (approximately) minimizes $f(x)$, when $f(x)$ is instantiated as a neural network.  $f(x)$ is not convex so you are not guaranteed to find the global minimum but for many problems this may be adequate; in others, you can adjust gradient descent (e.g., using random restarts) to mitigate those problems.