How does "linear algebraic" weight training function work? This answer shows that linear and polynomial function weights can be trained using this matrix operation:

$w = (X^TX)^{-1}X^Ty$

Therefore, algorithms such as gradient descent are not necessary for these functions. By my understanding, gradient descent for linear regression model finds perfect derivative for each weight so that cost function is minimal. 

Before asking for connections between gradient descent and equation above, let's separate the equation in smaller steps:

$X = [1,2,3,4,5,6,7,8,9,10,11,12]$

$y=[2.3,2.33,2.29,2.3,2.36,2.4,2.46,2.5,2.48,2.43,2.38,2.35]$

Let's turn $X$ vector to matrix by adding column of 1's which will be used to train the bias value:

\begin{bmatrix}
    1 &amp; 1 \\
    2 &amp; 1 \\
    3 &amp; 1 \\
    4 &amp; 1 \\
    5 &amp; 1 \\
    6 &amp; 1 \\
    7 &amp; 1 \\
    8 &amp; 1 \\
    9 &amp; 1 \\
    10 &amp; 1 \\
    11 &amp; 1 \\
    12 &amp; 1
  \end{bmatrix}


Transpose: $X^T$:

\begin{bmatrix}
     1 &amp; 2 &amp; 3 &amp; 4 &amp; 5 &amp; 6 &amp; 7 &amp; 9 &amp; 10 &amp; 11 &amp; 12 \\
     1 &amp; 1 &amp; 1 &amp; 1 &amp; 1 &amp; 1 &amp; 1 &amp; 1 &amp; 1 &amp; 1 &amp; 1 
  \end{bmatrix}
Matrix multiplication $X^TX$:

\begin{bmatrix}
     650 &amp; 78 \\
     78 &amp; 12
  \end{bmatrix}
Inverse matrix operation $(X^TX)^{-1}$:

\begin{bmatrix}
     0.00699301 &amp; -0.04545455 \\
     -0.04545455 &amp; 0.37878788
  \end{bmatrix}
Matrix multiplication $(X^TX)^{-1}X^T$:



array([[-0.03846154, -0.03146853, -0.02447552, -0.01748252, -0.01048951,
        -0.0034965 ,  0.0034965 ,  0.01048951,  0.01748252,  0.02447552,
         0.03146853,  0.03846154],
       [ 0.33333333,  0.28787879,  0.24242424,  0.1969697 ,  0.15151515,
         0.10606061,  0.06060606,  0.01515152, -0.03030303, -0.07575758,
        -0.12121212, -0.16666667]])




Matrix vector multiplication $(X^TX)^{-1}X^Ty$:

\begin{bmatrix}
     0.01174825 &amp; 2.30530303 
  \end{bmatrix}


This seems to be the perfect slope for minimizing cost. But I'm unable to understand how does it exactly work. 

It seems like that the algorithm such as gradient descent would've took extremely large amount of iterations to estimate perfect slope and bias value.

How/why does this equation exactly work? Is it somehow related to differentiation? Could it be compared to algorithms such as gradient descent? If so, how? Is it possible to use this equation with sigmoid functions?
Your goal is to find a $w$ such that

$$Xw \approx y$$

and way to model this problem is to minimize the objective function:

$$\min_w\|Xw-y\|^2.$$

Differentiating with respect to $w$ and equate it to zero gives us

$$2X^T(Xw-y)=0$$

$$X^TXw-X^Ty=0$$

$$X^TXw=X^Ty$$

if $X^TX$ is invertible, then we have 

$$w=(X^TX)^{-1}(X^Ty)$$

Remark:

We tend to avoid computing inverse and prefer gradient-based method. Complexity of the normal equation method is cubic.