title: Parameter estimation in logistic model by negative log-likelihood minimization - R
body: <p>I am currently attempting to estimate the parameters of a logistic regression model &quot;by hand&quot; on the <code>iris</code> dataset via minimisation of cross-entropy. Please note, when I say <code>iris</code> dataset, it has been changed such that there are only two classes - Setosa and Other. It was also normalised via the <code>scale</code> function:</p>
  <pre><code>library(dplyr)
  library(optimx)
  iriso &lt;- iris %&gt;%
    mutate(Species = ifelse(Species == &quot;setosa&quot;, &quot;setosa&quot;, &quot;other&quot;)) %&gt;%
    mutate(Species_n = ifelse(Species == &quot;setosa&quot;, 1, 0)) %&gt;%
    as.data.frame()
  
  iriso[,1:4] &lt;- scale(iriso[,1:4])
  </code></pre>
  <p>Based on my understanding: should everything be correct, when the optimisation is completed we should obtain the same set of parameters each time - no matter the starting point for the optimisation algorithm. However, the parameter estimates following optimisation are bouncing around:</p>
  <p>Some functions being defined:</p>
  <pre><code>X &lt;- model.matrix(~.,data=iriso[,1:4])
  Y &lt;- model.matrix(~0+Species_n,data=iriso)
  
  e &lt;- exp(1)
  sigmoid &lt;- function(y){
    1/(1 + e^-y)
  }
  
  #w is an array of weights. x is matrix of observations
  logistique &lt;- function(w, x){
    sigmoid(
      y = w[1]*x[,1] + w[2]*x[,2] + w[3]*x[,3] + w[4]*x[,4] + w[5]*x[,5]
      )
  }
  
  #y is obsrved values
  entropie &lt;- function(w, y, x){
    prob_pred &lt;- logistique(w = w, x = x)
  
    -sum(
      y*log(prob_pred) + (1-y)*log(1-prob_pred)
    ) 
  }
  
  </code></pre>
  <p>Optimisation step:</p>
  <pre><code>for(i in 1:5){
    w0 &lt;- rnorm(n = 5) #set of initial parameters
    optimx(par = w0, fn = entropie,  
           method = &quot;Nelder-Mead&quot;,
           y = iriso$Species_n, x = X) %&gt;%
      print()
  }
  </code></pre>
  <p>I can not seem to understand why I am not obtaining consistent answers. Is there something wrong with the code above? Is there a concept I am not aware of? Something I missed?</p>
  <p>Thanks.</p>