title: Keras and Sklearn logreg returning different results
body: <p>I'm comparing the results of a logistic regressor written in Keras to the default Sklearn Logreg. My input is one-dimensional. My output has two classes and I'm interested in the probability that the output belongs to the class 1.</p>
  
  <p>I'm expecting the results to be almost identical, but they are not even close.</p>
  
  <p>Here is how I generate my random data. Note that X_train, X_test are still vectors, I'm just using capital letters because I'm used to it. Also there is no need for scaling in this case.</p>
  
  <pre><code>X = np.linspace(0, 1, 10000)
  y = np.random.sample(X.shape)
  y = np.where(y&lt;X, 1, 0)
  </code></pre>
  
  <p>Here's cumsum of y plotted over X. Doing a regression here is not rocket science.</p>
  
  <p><a href="https://i.stack.imgur.com/1Ba18.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/1Ba18.png" alt="here&#39;s y plotted over x"></a></p>
  
  <p>I do a standard train-test-split:</p>
  
  <pre><code>X_train, X_test, y_train, y_test = train_test_split(X, y)
  X_train = X_train.reshape(-1,1)
  X_test = X_test.reshape(-1,1)
  </code></pre>
  
  <p>Next, I train a default logistic regressor:</p>
  
  <pre><code>from sklearn.linear_model import LogisticRegression
  sk_lr = LogisticRegression()
  sk_lr.fit(X_train, y_train)
  sklearn_logreg_result = sk_lr.predict_proba(X_test)[:,1]
  </code></pre>
  
  <p>And a logistic regressor that I write in Keras:</p>
  
  <pre><code>from keras.models import Sequential
  from keras.layers import Dense
  keras_lr = Sequential()
  keras_lr.add(Dense(1, activation='sigmoid', input_dim=1))
  keras_lr.compile(loss='mse', optimizer='sgd', metrics=['accuracy'])
  _ = keras_lr.fit(X_train, y_train, verbose=0)
  keras_lr_result = keras_lr.predict(X_test)[:,0]
  </code></pre>
  
  <p>And a hand-made solution:</p>
  
  <pre><code>pearson_corr = np.corrcoef(X_train.reshape(X_train.shape[0],), y_train)[0,1]
  b = pearson_corr * np.std(y_train) / np.std(X_train)
  a = np.mean(y_train) - b * np.mean(X_train)
  handmade_result = (a + b * X_test)[:,0]
  </code></pre>
  
  <p>I expect all three to deliver similar results, but here is what happens. This is a reliability diagram using 100 bins.</p>
  
  <p><a href="https://i.stack.imgur.com/1Zg0z.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/1Zg0z.png" alt="enter image description here"></a></p>
  
  <p>I have played around with loss functions and other parameters, but the Keras logreg stays roughly like this. What might be causing the problem here?</p>
  
  <p>edit: Using binary crossentropy is not the solution here, as shown by this plot (note that the input data has changed between the two plots).</p>
  
  <p><a href="https://i.stack.imgur.com/gZEjF.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/gZEjF.png" alt="enter image description here"></a></p>