I am working through example 3.10 (page 71) of a [book on variational inference](https://people.eecs.berkeley.edu/%7Ewainwrig/Papers/WaiJor08_FTML.pdf) in statistics and the authors discuss taking the gradient of the following function with respect to $\mu$:
$$
A^\*(\mu) = \sup\_{\theta \in \mathbb{R}} \{\, \theta \cdot \mu - \log(1 + \exp(\theta))\,\}.
$$

They state that the stationary condition $\nabla\_\mu A^\*(\mu) = 0$ is attained at the location $\mu = \exp(\theta)/(1+\exp(\theta))$.

I'm wondering how to derive this since $\nabla A^\*(\mu)$ seems to require pushing the gradient operator through the "sup" operator which has no closed form and which may not even be differentiable!

Any guidance or trick for how this stationary condition is derived is very much appreciated