What is the differences between normal equation and gradient descent for polynomial regression
You want to solve $X \times \theta = Y$. Actually, you have to find the parameters $\theta$. To find it, you should multiply both sides by $X^{-1}$ but it may have not an inverse. Consequently, you multiply each side by $X^t$ due to the fact that $X^tX$ has the inverse. After that, you multiply each side by $(X^tX)^{-1}$ which leads to a closed form equation for finding $\theta$. It should be mentioned that $X^tX$ does not always have an inverse. If you want to be sure it has it, you have to exploit data samples which are more than the number of features and they should not be linearly dependent otherwise the constructed matrix will not inverse and you will not be able to use it.