Is this scheme correct for logistic regression with stochastic gradient descent I am implementing logistic regression with stochastic gradient descent, but it is not working as expected. I've tried many epochs and different learning rates $\alpha$ but the probability of belonging to the correct class oscillates around $0.5$. I have got two classes denoted as $0,1$ and two features. Is this scheme correct?: 

Initialize weights $w = [w_0, w_1, w_2]$ to some random number (I choose 1) and loop: 


Choose random training sample $x_i = [1, x_1, x_2]$. 
Calculate the probability of it belonging to class $1,$ 
$$P(\hat{y} = 1) = \frac{1}{1+e^{-w \cdot x_i}}$$
Update the weights according to 
$$w = w + \alpha x_i(y_i - P(\hat{y} = 1))$$ 


This scheme makes sense to me since the update to the weights would be big if the probability of belonging to class $1$ was low, if the correct class was $1$ but I can't verify it more than that.
Hard to say without more detail, but isn't your update wrong? you need to subtract rather than add the gradient. Unless alpha is negative, this is wrong.