Unitary Transformation of the HF equations
You are going to want to check out Szabo and Ostlund page 143, they explain all the nitty-gritty for this matrix. The math itself is on page 22 and 23. But if you don't have a copy...

The $\mathbf{S}^{-1/2}$ comes from the need to orthogonalize your basis so that you get your equation (5) instead of the $\mathbf{FC}=\epsilon\mathbf{SC}$. It's one method of several to achieve this orthogonalization, and it's called symmetric orthogonalization. 

So how do you compute this function of a Hermitian matrix? Well, functions on diagonal matrices are super easy; you just apply the function to each diagonal element. For non-diagonal matrices, this is not true. More to your question, you can't just take one over the square root of each element of $\mathbf{S}$. But you can if it's diagonal. And you can make any Hermitian matrix diagonal via a unitary transformation.

The trick, then, is to diagonalize $\mathbf{S}$ first, apply the inverse square root to each eigenvalue, then transform back using the eigenvectors. So you're going to want to


Solve $\mathbf{SU} = \mathbf{sU}$. Where $\mathbf{s}$ are the eigenvalues of $\mathbf{S}$, and $\mathbf{U}$ are the eigenvectors.
For each $s$ in $\mathbf{s}$ do $s^{-1/2}$.
Transform back to original basis by $\mathbf{S}^{-1/2} = \mathbf{Us}^{-1/2}\mathbf{U}^{\dagger}$