Magnetic moment of trivalent lanthanide cations
Your question, Why can't we use normal formula to find effective magnetic moment of tripositive rare earth elements (REs)? is answered excellently by  porphyrin' comment (vide supra). Thus, I won't attempt to answer it again except for to mention the calculated values from the equation, $\mu_\mathrm{eff} = g_J \sqrt{J(J+1)}\mu_\mathrm{B}$ (where $\mu_\mathrm{B}$ is Bohr magneton and $g_J$ is Landé factor), have good agreement with experimental values.

Table 1: Theoretical magnetic moment of tripositive rare-earth ions
$$
\begin{array}{c|ccc}
\text{Ion} &amp; 4f^n &amp; S^c &amp; L^c &amp; J^c &amp; g^c &amp; \mu_\mathrm{eff} = g_J \sqrt{J(J+1)}\mu_\mathrm{B} &amp; \text{Observed }\mu^a \\
\hline
\ce{La} &amp; 0  &amp; 0 &amp; 0 &amp; 0 &amp; - &amp; 0.00 &amp; -\\
\ce{Ce} &amp; 1 &amp; \frac{1}{2} &amp; 3 &amp; \frac{5}{2} &amp; \frac{6}{7} &amp; 2.54 &amp; 2.4\\
\ce{Pr} &amp; 2 &amp; 1 &amp; 5 &amp; 4 &amp; \frac{4}{5} &amp; 3.58 &amp; 3.5\\
\ce{Nd} &amp; 3 &amp; \frac{3}{2} &amp; 6 &amp; \frac{9}{2} &amp; \frac{8}{11} &amp; 3.62 &amp; 3.5\\
\ce{Pm} &amp; 4 &amp; 2 &amp; 6 &amp; 4 &amp; \frac{3}{5} &amp; 2.68 &amp; -\\
\ce{Sm} &amp; 5 &amp; \frac{5}{2} &amp; 5 &amp; \frac{5}{2} &amp; \frac{2}{7} &amp; 0.85 &amp; 1.5\\
\ce{Eu} &amp; 6 &amp; 3 &amp; 3 &amp; 0 &amp; - &amp; 0.00 &amp; 3.4\\
\ce{Gd} &amp; 7 &amp; \frac{7}{2} &amp; 0 &amp; \frac{7}{2} &amp; 2 &amp; 7.94 &amp; 8.0 \ (7.23^b)\\
\ce{Tb} &amp; 8 &amp; 3 &amp; 3 &amp;6 &amp; \frac{3}{2} &amp; 9.72 &amp; 9.5 \ (9.58^b)\\
\ce{Dy} &amp; 9 &amp; \frac{5}{2} &amp; 5 &amp; \frac{15}{2} &amp; \frac{4}{3} &amp; 10.65 &amp; 10.6\\
\ce{Ho} &amp; 10 &amp; 2 &amp; 6 &amp; 8 &amp; \frac{5}{4} &amp; 10.61 &amp; 10.4\\
\ce{Er} &amp; 11 &amp; \frac{3}{2} &amp; 6 &amp; \frac{15}{2} &amp; \frac{6}{5} &amp; 9.58 &amp; 9.5\\
\ce{Tm} &amp; 12 &amp; 1 &amp; 5 &amp; 6 &amp; \frac{7}{6} &amp; 7.56 &amp; 7.3\\
\ce{Yb} &amp; 13 &amp; \frac{1}{2} &amp; 3 &amp; \frac{7}{2} &amp; \frac{8}{7} &amp; 4.54 &amp; 4.5\\
\ce{Lu} &amp; 14 &amp; 0 &amp; 0 &amp; 0 &amp; - &amp; 0.00 &amp; -\\
\end{array}
$$
$^a$ Rare-earth and metals really show "Spin"?; $^b$ Reference 1; and $^c$ Reference 3.

The two experimental values from Ref.1 are in $\pu{emu mol−1 Oe−1}$ and have been converted appropriately to be able to compare with the values in hand. The reference also point out that crystals of mixed salt have approximately average of the pure monometal salts. The abstract of Ref.1 states that:


  Magnetic moment and susceptibility measurements of single crystals of pure and mixed rare earth fumarates of gadolinium and terbium were carried out at room temperature. The experimental values of molar susceptibilities for $\ce{Gd2(C4H2O4)3.7H2O}$, $\ce{Tb2(C4H2O4)3.7H2O}$, and $\ce{GdTb(C4H2O4)3.7H2O}$ are $2.68 \times 10^{−2}$, $3.89 \times 10^{−2}$, and $3.18 \times 10^{−2}$ (in $\pu{emu mol−1 Oe−1}$), respectively. The calculated effective magnetic moments are in good agreement with the theoretical predictions on rare earth ions.


The magnetic properties of rare earth elements (REs) are due primarily to the electrons in the incomplete shell ($\mathrm{4f}$). In fact the magnetic moment per atom is determined by the number of electrons in the $\mathrm{4f}$ shell according to the Russell-Saunders coupling scheme. The derivation of the equation $(1)$ is partially explained in Ref.2:

$$\mu_\mathrm{eff} = g_{J} \sqrt{J(J+1)}\mu_\mathrm{B} \tag{1}$$

According to Ref.2:


  The Russell-Saunders coupling scheme is normally adopted to describe the $\mathrm{4f}$ magnetism of the REs. The spins of the individual electrons are coupled to form a total spin $S$ by the exchange interaction and the individual orbital angular momenta are coupled to form a total orbital momentum $L$ by the Coulomb interaction. The state with the lowest energy is found from Hund’s rules by maximizing $S$ and thereafter $L$. The total angular momentum $J$ is given by $J = |L ± S|$, where the minus sign is used for less than half filled shells and the plus sign for more than half filled shells. The total magnetic moment due to the spin angular momentum is $µ_S = 2µ_B\sqrt{S(S + 1)}$, where $µ_B$ is the Bohr magneton. The total moment due to the orbital angular momentum is $µ_L = µ_B\sqrt{L(L + 1)}$. These moments process around the direction of $J$, therefore the magnetic moment due to the total angular momentum looks slightly more complicated: $µ_J = g_J µ_B\sqrt{J(J + 1)}$, where $g_J$ is the Landé factor [12].


The sighted Reference ([12]) is actually Ref.3 here. According to Ref.3:
$$
J =
\begin{cases}
L-S,  &amp; \text{if $4F^n \le 7$} \\
L+S, &amp; \text{if $4F^n \gt 7$}
\end{cases}
$$

Note: The equation $(1)$ does not obey experimental values of $\mu$ found for $\mathrm{3d}$ elements (Rare-earth and metals really show "Spin"?). 

References:


B. Want, M. D. Shah, “Magnetic susceptibility measurements of pure and mixed gadolinium–terbium fumarate heptahydrate crystals,” Journal of Magnetism and Magnetic Materials 2016, 401, 391-393 (https://doi.org/10.1016/j.jmmm.2015.10.065).
I. L. M. Locht, Y. O. Kvashnin, D. C. M. Rodrigues, M. Pereiro, A. Bergman, L. Bergqvist, A. I. Lichtenstein, M. I. Katsnelson, A. Delin, A. B. Klautau, B. Johansson, I. Di Marco, O. Eriksson, “Standard model of the rare earths, analyzed from the Hubbard I approximation,” Phys. Rev. B 2016, 94, 085137 (https://doi.org/10.1103/PhysRevB.94.085137)(PDF).
Jens Jensen, Allan R. Mackintosh, In Rare earth magnetism: Structures and Excitations; The International Series of Monographs on Physics, J. Birman S. F. Edwards, C. H. Llewellyn, &amp; Smith M. Rees, General Eds.; Clarendon Press: Oxford, United Kingdom, 1991.