title: Why do multi-class logistic regression model coefficients change depending on class labels?
body: <p>I'm currently training a multi-class logistic regression model with scikit-learn and need the model coefficients for a subsequent feature selection pipeline. However, changing the class labels that should be predicted leads to different coefficients. So if I want to predict three classes with the labels 0, 1, 2 the results are different compared to naming them e.g. 2, 0, 1.</p>
  <p>To demonstrate this problem, I tested the example from the scikit-learn documentation:</p>
  <pre><code>from sklearn.datasets import load_iris
  from sklearn.linear_model import LogisticRegressionCV
  X, y = load_iris(return_X_y=True)
  clf = LogisticRegressionCV(cv=5, random_state=0).fit(X, y)
  </code></pre>
  <p>There are three classes in this example:</p>
  <pre><code>import numpy as np
  np.unique(y)
  &gt;&gt;&gt; array([0, 1, 2])
  </code></pre>
  <p>I renamed the class labels such that they have a different order and trained a second model with the renamed labels:</p>
  <pre><code>y_rename_dict = {0: 2, 1: 0, 2: 1}
  y1 = np.array([y_rename_dict[i] for i in y])
  clf1 = LogisticRegressionCV(cv=5, random_state=0).fit(X, y1)
  </code></pre>
  <p>Looking at the <code>coef_</code> values of both models reveals different results for each of the predicted classes (taking the different ordering of the coefficient arrays into account):</p>
  <pre><code>clf.coef_
  &gt;&gt;&gt; array([[-0.34022169,  2.46767758, -4.95392813, -2.48912258],
         [ 1.15417403,  0.09965468, -0.50367445, -2.92399758],
         [-0.81395234, -2.56733226,  5.45760258,  5.41312016]])
  clf1.coef_
  &gt;&gt;&gt; array([[ 1.16881994,  0.10139304, -0.50578412, -2.92007532],
         [-0.79861288, -2.56336652,  5.4577491 ,  5.41271211],
         [-0.37020706,  2.46197348, -4.95196498, -2.49263679]])
  </code></pre>
  <p>To show the magnitude of difference between these results, I took a look at the sum of the absolute values of the coefficients:</p>
  <pre><code>max(np.sum(np.abs(clf.coef_), axis=0) - np.sum(np.abs(clf1.coef_), axis=0))
  &gt;&gt;&gt; 0.007931465905081758
  </code></pre>
  <p>In this example, the difference is not that big but with increasing numbers of features and samples, the differences become more noticeable. I also tested the <code>SGDClassifier</code> model with 'log' loss but it also leads to different results depending on the order of the class labels.</p>
  <p>I was not able to find any explanation for this behavior. Is this a common problem of this model or is there maybe a solution to fix this issue?</p>