Finding optimal weights for models I'm trying to implement an algorithm to find the minimal value of a function.

Before moving to sigmoid activation functions, i'm trying to understand linear regression.

Usually, a gradient descent algorithm is used to find an minimal value where the algorithm converges, but there are some other ways for linear models.

Say I have two vectors:

x=[1,2,3,4,5,6,7,8,9,10,11,12]

y=[2.3,2.33,2.29,2.3,2.36,2.4,2.46,2.5,2.48,2.43,2.38,2.35]




Between these points, I would like to add a linear separator with least squares.

Say I have some imperfect linear function:

$f(x)=0.026x+2.3$

As I know, there are two ways to find this:

$w = (X^TX)^{-1}X^Ty$

and a gradient descent algorithm:

$w^{new} = w^{old} - \nu \frac{dy}{dx}$

Although for linear models finding derivative is trivial, thus second method is not necessary. 

Now i've used the first equation on the vectors, in Python:

w = ((np.transpose(x)*x)**-1)*np.transpose(x)*y


Unfortunately, the output was irrelevant:

[ 2.3, 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ]


Then i've tried using the second method for 500 iterations, in Python:

for i in range(1,5000):
    x_old = x_new
    x_new = x_old - v*dydx
    print("x_new = {0} - {1}({2}) = {3}".format(x_old, v, dydx, x_new))


However, i'm not sure how to know when it reaches a convergence point.

How can I use these methods properly for linear models? And if so, how can they be used for more complex models such as logistic regression?
Also you have misunderstood the -1..
It's not the exponent rather the symbol of inverse of a Matrix 

Also use .T for Transpose...(a bit Pythonic Convenience)

Just to help you out,
Search np.linalg.inv

For your second Query,

Refer here

Just to add a little, you stop when you see that your loss isn't improving any more as such or is improving at the 5th decimal place..

Hope this helps..