Stochastic gradient descent in logistic regression I am very new to machine learning and in my first project have stumbled across a lot of issues which I really want to get through.

I'm using logistic regression with R's glmnet package and alpha = 0 for ridge regression.

I'm using ridge regression actually since lasso deleted all my variables and gave very low area under curve (0.52) but with ridge there isn't much of a difference (0.61).

My dependent variable/output is probability of click, based on if there is a click or not in historical data.

The independent variables are state, city, device, user age, user gender, IP carrier, keyword, mobile manufacturer, ad template, browser version, browser family, OS version and OS family.

Of these, for prediction I'm using state, device, user age, user gender, IP carrier, browser version, browser family, OS version and OS family; I am not using keyword or template since we want to reject a user request before deep diving in our system and selecting a keyword or template. I am not using city because they are too many or mobile manufacturer because they are too few.

Is that okay or should I be using the rejected variables?

To start, I create a sparse matrix from my variables which are mapped against the column of clicks that have yes or no values.

After training the model, I save the coefficients and intercept. These are used for new incoming requests using the formula for logistic regression:


  


Where a is intercept, k is the ith coefficient and x is the ith variable value.

Is my approach correct so far?

Simple GLM in R (that is where there is no regularized regression, right?) gave me 0.56 AUC. With regularization I get 0.61 but there is no distinct threshold where we could say that above 0.xx its mostly ones and below it most zeros are covered; actually, the max probability that a click didn't happen is almost always greater than the max probability that a click happened.

So basically what should I do?

I have read how stochastic gradient descent is an effective technique in logit so how do I implement stochastic gradient descent in R? If it's not straightforward, is there a way to implement this system in Python? Is SGD implemented after generating a regularized logistic regression model or is it a different process altogether?

Also there is an algorithm called follow the regularized leader (FTRL) that is used in click-through rate prediction. Is there a sample code and use of FTRL that I could go through?
Stochastic gradient descent is a method of setting the parameters of the regressor; since the objective for logistic regression is convex (has only one maximum), this won't be an issue and SGD is generally only needed to improve convergence speed with masses of training data.

What your numbers suggest to me is that your features are not adequate to separate the classes. Consider adding extra features if you can think any any that are useful. You might also consider interactions and quadratic features in your original feature space.