Why is logistic regression not sigmoidal? The blue dots are the raw data and the line is my logistic regression. The line is quite straight and not sigmoidal as I would have expected. I suspect there is something wrong in my gradient descent equation but I don't understand the maths well enough to find the mistake.



This is the code I used to generate the logistic regression:

import numpy as np
import pandas as pd
import matplotlib 
import matplotlib.pyplot as plt
from scipy import stats


def logit(z):
    return (1/(1+np.exp(-z)))

def cost(data, weights):
    x_val = data[:,0:2]
    y_val = data[:,2]
    m = len(x_val)
    scores = np.dot(x_val, weights)
    cost = np.sum(y_val*scores - np.log(1 + np.exp(scores)))
    return cost

def gradient_descent(data, weights, learning_rate): 
    x_val = data[:,0:2]
    y_val = data[:,2]
    m = len(x_val)
    scores = np.dot(x_val, weights)
    weights -= (learning_rate/m)*np.dot(np.transpose(x_val),(logit(scores)-y_val))
    return weights 

dataset = pd.read_csv("/Users/An/Desktop/data/glass.csv") 
dataset.sort_values('Al', inplace=True) # sort by ascending "Al" values
dataset.head() 
dataset['binary'] = dataset.Type.map({1:0, 2:0, 3:0, 5:1, 6:1, 7:1})
ones = np.ones(len(dataset))
data = np.stack((ones,dataset["Al"], dataset["binary"]), axis=-1)

weights = np.random.rand(2)

def log_reg(data, weights):
    for i in range(9000):
        weights = gradient_descent(data, weights, 0.000001)
        loss = cost(data, weights)
    return weights

w = log_reg(data, weights)
if would suggest you implement the following formulas:


Sigmoid activation function


$$\sigma(x) = \frac{1}{1+e^{-x}}$$


Output (prediction) formula


$$\hat{y} = \sigma(w_1 x_1 + w_2 x_2 + b)$$


Error function


$$\text{Error}(y, \hat{y}) = - y \log(\hat{y}) - (1-y) \log(1-\hat{y})$$


The function that updates the weights


$$ w_i \longrightarrow w_i + \alpha (y - \hat{y}) x_i$$

$$ b \longrightarrow b + \alpha (y - \hat{y})$$

If you want the corresponding Python code for the above just let me know. I also have it but I guess it would be more useful for you (from the point of view of learning) to implement it yourself. In any case, if you want it, I can update my answer with the Python code.

Source code:

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def output_formula(features, weights, bias):
    return sigmoid(np.dot(features, weights) + bias)

def error_formula(y, output):
    return - y*np.log(output) - (1 - y) * np.log(1-output)

def update_weights(x, y, weights, bias, learnrate):
    output = output_formula(x, weights, bias)
    d_error = y - output
    weights += learnrate * d_error * x
    bias += learnrate * d_error
    return weights, bias