Understanding Matlab's Automatic Gain Control function
From the diagram in the Algorithms section of the documentation you can see how the different quantities are computed:


Note that $z$ in the diagram is an estimate of the output power.$^1$ The error signal $e$ is computed by comparing the reference value $A$ to $\ln(z)$. So if you choose $$A=\ln(P)\tag{1}$$ then the average output power will be adjusted to the specified value $P$.

Concerning the maximum gain, note that the input is multiplied by $\tilde{g}=e^g$. If $G$ is the maximum gain in dB, you have the following relationship:

$$G=20\log_{10}(\tilde{g})\tag{2}$$

i.e.,

$$\tilde{g}=10^{G/20}\tag{3}$$

and, consequently,

$$g=\ln(\tilde{g})=G\cdot \frac{\ln(10)}{20}\approx G\cdot  0.11513\tag{4}$$


1. There is an error in the diagram: the output of the detector must be multiplied by the square of the gain (because it's an estimate of the power). This is correctly represented by the equations below the diagram in the documentation.