Total spin for lithium atom
Probably the easier way to explain it is to rely on the fact that if Li indeed had a ground state with $S = 3/2$, this would necessarily comprise four levels with $M_S = +3/2, +1/2, -1/2, -3/2$. This sort of splitting can be observed under a magnetic field, for example (the Zeeman effect).

Note that it's not possible for the Pauli principle to forbid $M_S = +3/2$ and $-3/2$ while simultaneously allowing $M_S = +1/2$ and $-1/2$. The Pauli principle doesn't mean that "this state can exist, but the electrons can't reach that state". It means that "this state simply cannot exist": because any state that violates the Pauli principle is equal to zero, which isn't a valid state. So it's not as if the $M_S = \pm 3/2$ states are there, and that the electrons simply choose to go into the $M_S = \pm 1/2$ states. The $\pm 3/2$ states are nonexistent. And consequently, because a $S = 3/2$ state necessitates the existence of $M_S = \pm 3/2$ states, it cannot exist either.

However, since that line of argument doesn't seem to fully convince you, you may want to go through the "exercise" of proving it. Unless I'm terribly wrong, this should be a completely foolproof (albeit terribly tedious) method. You can construct the full wavefunction for the Li atom as a Slater determinant (without loss of generality we choose $\alpha$ for the third spin):

$$\Psi = \frac{1}{\sqrt{6}} \begin{vmatrix}
1s(1)\alpha(1) &amp; 1s(1)\beta(1) &amp; 2s(1)\alpha(1) \\
1s(2)\alpha(2) &amp; 1s(2)\beta(2) &amp; 2s(2)\alpha(2) \\
1s(3)\alpha(3) &amp; 1s(2)\beta(3) &amp; 2s(2)\alpha(3) \\
\end{vmatrix}$$

so "all" we need to do is to apply $\hat{S}^2$ to this wavefunction and see what eigenvalue it returns. We know that

$$\hat{S}^2|\Psi\rangle = S(S+1)|\Psi\rangle$$

and so if $S = 1/2$ then the eigenvalue will be $3/4$; and if $S = 3/2$ the eigenvalue will be $15/4$. The $\hat{S}^2$ operator can be expanded as

$$\begin{align}
\hat{S}^2 &amp;= (\hat{S}_1 + \hat{S}_2 + \hat{S}_3)^2 \\
&amp;= \hat{S}_1^2 + \hat{S}_2^2 + \hat{S}_3^2 + 2(\hat{S}_1 \cdot \hat{S}_2) + 2(\hat{S}_1 \cdot \hat{S}_3) + 2(\hat{S}_2 \cdot \hat{S}_3) \\
&amp;= \hat{S}_1^2 + \hat{S}_2^2 + \hat{S}_3^2 + 2(\hat{S}_{1x} \hat{S}_{2x} + \hat{S}_{1y}\hat{S}_{2y} + \hat{S}_{1z}\hat{S}_{2z}) \\
&amp;\quad + 2(\hat{S}_{1x} \hat{S}_{3x} + \hat{S}_{1y}\hat{S}_{3y} + \hat{S}_{1z}\hat{S}_{3z}) + 2(\hat{S}_{2x} \hat{S}_{3x} + \hat{S}_{2y}\hat{S}_{3y} + \hat{S}_{2z}\hat{S}_{3z})
\end{align}$$

We already know what the individual operators $\hat{S}_i^2$, $\hat{S}_{ix}$, $\hat{S}_{iy}$, and $\hat{S}_{iz}$ do to the basis states $|\alpha(i)\rangle$ and $|\beta(i)\rangle$. Bearing in mind that operators don't act on different degrees of freedom, you can work out what this monster operator does to the Li wavefunction.

If you look online, there are examples of how to perform this calculation for a two-spin system (e.g. "prove that the singlet and triplet wavefunctions are eigenstates of $\hat{S}^2$"). The extension to a three-spin system is not difficult, but is extremely tedious, which is why I gave up on writing it out in full. But I am fairly certain that at the end of the day you will find the eigenvalue of $3/4$.