Subgame Perfect Equilibrium in Baye, Shin (1999)
True to the saying that a picture is worth a thousand words, let's plot the profit function under the Stackelberg game and see what we're dealing with.

You correctly observed (or copied from the Baye, Shin paper) that the reaction function of player 2 is:

$$
R(x_1)=\frac{2\mathit{x_1}-\sqrt{2}\,\sqrt{2\mathit{x_1}-{{\mathit{x_1}}^{2}}}}{2\mathit{x_1}-2}\
$$

Inserting this reaction function in the profit function of player 1 gives us a complicated expression:

$$
\pi(x_1)=-\frac{{{2}^{\frac{3}{2}}}\,\sqrt{2-\mathit{x_1}}\,{{\mathit{x_1}}^{2}}-2{{\mathit{x_1}}^{\frac{3}{2}}}-3\sqrt{2}\,\sqrt{2-\mathit{x_1}}\,\mathit{x_1}+4\sqrt{\mathit{x_1}}}{{{2}^{\frac{3}{2}}}\,\sqrt{2-\mathit{x_1}}\,\left( \mathit{x_1}-1\right) }
$$

(Perhaps more advanced symbolic math programs can further simplify this, mine won't). In any case this is  a function of one variable $\mathit{x_1}$, and can be plotted. For completeness I have inserted the original equilibrium as well.



As you can see the original NE is no longer a local maximum, and player 1 can increase its profits by increasing their effort.