How does the pH of solutions of salts in which the cation is acidic and anion basic, depend only on the relation of their constants?
Disclaimer: as we will see later, the formula

$$[\ce{H+}] = \sqrt{\frac{K_\mathrm{w}K_\mathrm{a}}{K_\mathrm{b}}} \label{1}\tag{1}$$

is only applicable to highly soluble salts with equimolar rests of monovalent weak acid and weak base such as $\ce{NH4OAc}$, $\ce{NH4HCO3}$, $\ce{NH4HSO3}$, $\ce{NH4NO2}$ and cannot be used for polyvalent electrolytes, poorly soluble salts and when hydrolysis is irreversible (e.g. $\ce{Al2S3}$).

Let's derive formula \eqref{1} in for the salt of the general $\ce{BHA}$ formula. As we agreed, the salt is highly soluble (also true for your textbook example, ammonium acetate), so the main approximation here is that we assume this salt dissociates completely in solution:

$$\ce{BHA → BH+ + A-} \label{R1}\tag{R1}$$

Each ion participates in hydrolytic equilibrium (\eqref{R2}, \eqref{R3}), plus there is autoprotolysis of water, \eqref{R4}. Subtracting \eqref{R4} from \eqref{R2} and \eqref{R3} we get a brutto equation for hydrolysis \eqref{R5}:

$$
\begin{align}
&amp;+\quad&amp;\ce{BH+ + H2O &amp;&lt;=&gt; B + H3O+} \label{R2}\tag{R2}\\
&amp;+\quad&amp;\ce{A- + H2O &amp;&lt;=&gt; HA + HO-} \label{R3}\tag{R3}\\
&amp;-\quad&amp;\ce{2 H2O &amp;&lt;=&gt;[$K_\mathrm{w}$] H3O+ + HO-} \label{R4}\tag{R4}\\
\hline
&amp;&amp;\ce{BH+ + A- &amp;&lt;=&gt;[$K_\mathrm{h}$] B + HA}, \label{R5}\tag{R5}\\
\end{align}
$$

where $K_\mathrm{h}$ is hydrolysis constant. According to the law of mass action, it can be written for \eqref{R5} as such:

$$K_\mathrm{h} = \frac{[\ce{B}][\ce{HA}]}{[\ce{BH+}][\ce{A-}]} \label{2}\tag{2}$$

Now let's multiply both divider and denominator in \eqref{2} by $[\ce{H3O+}][\ce{HO-}]$ so that

$$
\begin{align}
K_\mathrm{h} &amp;= \left.\frac{[\ce{B}][\ce{HA}]}{[\ce{BH+}][\ce{A-}]}\qquad \right|\cdot \frac{[\ce{H3O+}][\ce{HO-}]}{[\ce{H3O+}][\ce{HO-}]}\\
 &amp;= \frac{\color{blue}{[\ce{B}]}\color{red}{[\ce{HA}]}[\ce{H3O+}][\ce{HO-}]}{\color{blue}{[\ce{BH+}]}\color{red}{[\ce{A-}][\ce{H3O+}]}\color{blue}{[\ce{HO-}]}}\\
 &amp;= \frac{[\ce{H3O+}][\ce{HO-}]}{\color{red}{K_\mathrm{a}}\color{blue}{K_\mathrm{b}}}\\
 &amp;= \frac{K_\mathrm{w}}{K_\mathrm{a}K_\mathrm{b}}\label{3}\tag{3}
\end{align}
$$

where $K_\mathrm{a}$ and $K_\mathrm{b}$ are dissociation constants of the conjugated acid and conjugated base, respectively:

$$
\begin{align}
\ce{HA + H2O &amp;&lt;=&gt;[$K_\mathrm{a}$] A- + H3O+} \label{R6}\tag{R6}\\
\ce{B + H2O &amp;&lt;=&gt;[$K_\mathrm{b}$] BH+ + HO-} \label{R7}\tag{R7}\\
\end{align}
$$

Since the dissociation of the salt is complete (see \eqref{R1}), its initial concentration $c_0$ equals to the concentrations of $\ce{BH+}$ and $\ce{A-}$ in solution:

$$c(\ce{BH+}) = c(\ce{A-}) = c_0 \label{4}\tag{4}$$

and the corresponding equilibrium concentrations of the ions participating in hydrolysis are 

$$[\ce{BH+}] = [\ce{A-}] = (1-α)c_0, \label{5}\tag{5}$$

where $α$ is the degree of hydrolysis showing the ratio between the amount of hydrolyzed salt $n_\mathrm{h}$ to the total amount of salt $n_0$ ($α = n_\mathrm{h}/n_0$; $α\in [0;1]$). The equilibrium concentrations of the weak acid and base formed during hydrolysis are

$$[\ce{B}] = [\ce{HA}] = αc_0 \label{6}\tag{6}$$

and eq. \eqref{2} for $K_\mathrm{h}$ can be rewritten using degree of hydrolysis:

$$K_\mathrm{h} = \frac{[\ce{B}][\ce{HA}]}{[\ce{BH+}][\ce{A-}]} = \frac{α^2c_0^2}{(1-α)^2c_0^2} = \frac{α^2}{(1-α)^2} \label{7}\tag{7}$$

Note that hydrolysis constant doesn't depend on salt's concentration. Since we are interested in $\ce{pH}$, it is convenient to express $[\ce{H3O+}]$ from $K_\mathrm{a}$:

$$K_\mathrm{a} = \frac{[\ce{H3O+}][\ce{A-}]}{[\ce{HA}]};$$

$$[\ce{H3O+}] = K_\mathrm{a}\frac{[\ce{HA}]}{[\ce{A-}]} = K_\mathrm{a}\frac{αc_0}{(1-α)c_0} = K_\mathrm{a}\frac{α}{1-α} \label{8}\tag{8}$$

From \eqref{7} $α/(1-α) = \sqrt{K_\mathrm{h}}$ so that \eqref{8} finally becomes the first equation you asked about:

$$[\ce{H3O+}] = K_\mathrm{a}\sqrt{K_\mathrm{h}} = K_\mathrm{a}\sqrt{\frac{K_\mathrm{w}}{K_\mathrm{a}K_\mathrm{b}}} = \sqrt{\frac{K_\mathrm{w}K_\mathrm{a}}{K_\mathrm{b}}} \label{9}\tag{9}$$

An equation for $\mathrm{pH}$ is obtained trivially by taking negative decimal logarithm on both parts of \eqref{9}.