How to understand incremental stochastic gradient algorithm and its implementation in logistic regression [updated]?
Gradient Descent

The idea of gradient descent is to traverse a function $f_{LR}(\textbf{w})$ and find a local maximum or minimum for a set a values $\textbf{w}$. Gradient descent is an iterative process. At each iteration you will evaluate the function given your current set of values. Then you will take the derivative of that function with respect to those values to see how much each value contributes to the slope of the function. We can then change the values in a proportionate way to move towards this optimal point. 

The gradient descent equation is described as

$\textbf{w}^{(k+1)} = \textbf{w}^{(k)} - \rho \frac{\partial f_{LR}(\textbf{w})}{\partial \textbf{w}^{(k)}}$

where $\rho$ is the learning rate, usually small number. A constant which determines the speed at which we want to change the values we are optimizing. 

Initializing the values

There's many ways to initialize these values. We can either set them all to zero, or set them randomly. 

Then you can take a random instance in your dataset $x_i$ and $y_i$ and compute the derivative 

$\frac{\partial f_{LR}(\textbf{w})}{\partial \textbf{w}^{(k)}} = \textbf{x}_i (-y_i \frac{e^{-y_i \textbf{x}_i \textbf{w}}}{1 + e^{-y_i \textbf{x}_i \textbf{w}}})$.

Once the compute this you can put the result into the gradient descent equation and update all your weights.

You then continue to go through this process until your weights $\textbf{w}$ converge to a value, or some other condition is met. You might want to limit your algorithm with some iteration counter to avoid infinite loops caused by weights that do not converge. 



Gradient Descent Code

Here is an example using gradient descent to train weights along the logistic regression loss function. 

import numpy as np

def update_weights(x_i, y_i, w):
   p = 0.8                  # The learning rate
   yhat = predict(x_i, w)   # The predicted target
   error = y_i - yhat       
   return w + p * (y_i - yhat) * x_i  # Update the weights

def predict(x_i, w):
   return 1/(1+np.exp(-1 * np.dot(w.T, x_i)))

def train_weights(x, y, verbose = 0):
   w = np.zeros((x.shape[1],))
   w_temp = np.zeros((x.shape[1],)) 

   epoch = 0
   while epoch &lt;= 1000:
      for i, x_i in enumerate(x):
         w = update_weights(x_i, y[i], w)

      pred = 1/(1+np.exp(-1 * np.dot(x, w)))
      error = np.sum(pred - y)

      pred[pred &lt; 0.5] = 0
      pred[pred &gt;= 0.5] = 1
      epoch += 1

      if verbose == 1:
         print('------------------------------------------------')
         print('Targets: ', y)
         print('Predictions: ', pred)
         print('Predictions: ', pred)
         print('------------------------------------------------')

      # Check if we have reach convergence
      if np.sum(np.abs(w_temp - w)) &lt; 0.001:
         print(epoch)
         print(error)
         return w
      else: 
         w_temp = w
  return w

# Create some artificial database
x = np.zeros((4,3))
y = np.zeros((4,)) 
x[0] = [1, 1, 1]
x[1] = [1, 2, 2]
x[2] = [1, 10, 10]
x[3] = [1, 11, 11]
y[0] = 0
y[1] = 0
y[2] = 1
y[3] = 1

print(x)
print(y)

# Train the weights
w = train_weights(x, y, 0)
print('Trained weights: ', w)

# Get predicitons
pred = 1/(1+np.exp(-1 * np.dot(x, w)))
error = np.sum(pred - y)   
pred[pred &lt; 0.5] = 0
pred[pred &gt;= 0.5] = 1
print('Predictions: ', pred)