title: inconsistent results between Keras and MLPClassifier sklearn
body: <p>I have been reading <a href="https://keras.io" rel="nofollow noreferrer">Keras</a> documentation to build my own MLP network that implements MLP backpropagation. I am familiar with <a href="http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html" rel="nofollow noreferrer">MLPClassifier</a> in sklearn but I want to learn <code>Keras</code> for deep learning. The following is the first attempt. The network has 3 layers of 1 input (features=64), 1 output and 1 hidden. The total is (64,64,1). The input is <code>numpy</code> matrix <code>X</code> of 125K samples (64 dim) and <code>y</code> is a 1D <code>numpy</code> binary class (1, -1):</p>
  
  <pre><code># Keras imports
  from keras.models import Sequential
  from sklearn.model_selection import train_test_split
  from keras.layers import Dense, Dropout, Activation
  from keras.initializers import RandomNormal, VarianceScaling, RandomUniform
  from keras.optimizers import SGD, Adam, Nadam, RMSprop
  
  # System imports
  import sys
  import os
  import numpy as np
  os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
  
  
  def train_model(X, y, num_streams, num_stages):
  
      '''
      STEP1: Initialize the Model
      '''
  
      tr_X, ts_X, tr_y, ts_y = train_test_split(X, y, train_size=.8)
      model = initialize_model(num_streams, num_stages)
  
      '''
      STEP2: Train the Model
      '''
      model.compile(loss='binary_crossentropy',
                    optimizer=Adam(lr=1e-3),
                    metrics=['accuracy'])
      model.fit(tr_X, tr_y,
                validation_data=(ts_X, ts_y),
                epochs=3,
                batch_size=200)
  
  
  def initialize_model(num_streams, num_stages):
  
      model = Sequential()
      hidden_units = 2 ** (num_streams + 1)
      # init = VarianceScaling(scale=5.0, mode='fan_in', distribution='normal')
      init_bound1 = np.sqrt(3.5 / ((num_stages + 1) + num_stages))
      init_bound2 = np.sqrt(3.5 / ((num_stages + 1) + hidden_units))
      init_bound3 = np.sqrt(3.5 / (hidden_units + 1))
      # drop_out = np.random.uniform(0, 1, 3)
  
      # This is the input layer (that's why you have to state input_dim value)
      model.add(Dense(num_stages,
                      input_dim=num_stages,
                      activation='relu',
                      kernel_initializer=RandomUniform(minval=-init_bound1, maxval=init_bound1)))
  
      model.add(Dense(hidden_units,
                      activation='relu',
                      kernel_initializer=RandomUniform(minval=-init_bound2, maxval=init_bound2)))
  
      # model.add(Dropout(drop_out[1]))
  
      # This is the output layer
      model.add(Dense(1,
                      activation='sigmoid',
                      kernel_initializer=RandomUniform(minval=-init_bound3, maxval=init_bound3)))
  
      return model
  </code></pre>
  
  <p>The problem is that I get 99% accuracy with the same dataset <code>X</code> and <code>y</code> when using <code>MLPClassifier sklearn</code>. However, Keras gives poor accuracy as seen below:</p>
  
  <pre><code>Train on 100000 samples, validate on 25000 samples
  Epoch 1/3
  100000/100000 [==============================] - 1s - loss: -0.5358 - acc: 0.0022 - val_loss: -0.7322 - val_acc: 0.0000e+00
  Epoch 2/3
  100000/100000 [==============================] - 1s - loss: -0.6353 - acc: 0.0000e+00 - val_loss: -0.7385 - val_acc: 0.0000e+00
  Epoch 3/3
  100000/100000 [==============================] - 1s - loss: -0.7720 - acc: 9.0000e-05 - val_loss: -0.9474 - val_acc: 5.2000e-04
  </code></pre>
  
  <p>I don't understand why? Am I missing something here? Any help is appreciated.</p>