title: Integrate Keras to SKLearn Pipeline?
body: <p>I have a sklearn pipeline performing feature engineering on heterogeneous data types (boolean, categorical, numeric, text) and wanted to try a neural network as my learning algorithm to fit the model.  I am running into some problems with the shape of the input data.  </p>
  
  <p>I am wondering if what I am trying to do is even possible and or if I should try a different approach?</p>
  
  <p>I have tried a couple different methods but am receiving these errors:</p>
  
  <ol>
  <li><p><code>Error when checking input: expected dense_22_input to have shape (11,) but got array with shape (30513,)</code> => I have 11 input features ... so I then tried converting my X and y to arrays and now get this error</p></li>
  <li><p><code>ValueError: Specifying the columns using strings is only supported for pandas DataFrames</code> => which I think is because of the <code>ColumnTransformer()</code> where I specify column names</p></li>
  </ol>
  
  <pre><code>print(X_train_OS.shape)
  print(y_train_OS.shape)
  
  (22354, 11)
  (22354,)
  </code></pre>
  
  <pre><code>from keras.models import Sequential
  from keras.layers import Dense
  from keras.wrappers.scikit_learn import KerasClassifier
  from keras.utils import to_categorical # OHE
  
  X_train_predictors = df_train_OS.drop("label", axis=1)
  X_train_predictors = X_train_predictors.values
  y_train_target = to_categorical(df_train_OS["label"])
  
  y_test_predictors = test_set.drop("label", axis=1)
  y_test_predictors = y_test_predictors.values
  y_test_target = to_categorical(test_set["label"])
  
  print(X_train_predictors.shape)
  print(y_train_target.shape)
  
  (22354, 11)
  (22354, 2)
  </code></pre>
  
  <pre><code>def keras_classifier_wrapper():
      clf = Sequential()
      clf.add(Dense(32, input_dim=11, activation='relu'))
      clf.add(Dense(2, activation='softmax'))
      clf.compile(loss='categorical_crossentropy', optimizer='adam', metrics=["accuracy"])
      return clf
  
  TOKENS_ALPHANUMERIC_HYPHEN = "[A-Za-z0-9\-]+(?=\\s+)"
  
  boolTransformer = Pipeline(steps=[
      ('bool', PandasDataFrameSelector(BOOL_FEATURES))])
  
  catTransformer = Pipeline(steps=[
      ('cat_imputer', SimpleImputer(strategy='constant', fill_value='missing')),
      ('cat_ohe', OneHotEncoder(handle_unknown='ignore'))])
  
  numTransformer = Pipeline(steps=[
      ('num_imputer', SimpleImputer(strategy='constant', fill_value=0)),
      ('num_scaler', StandardScaler())])
  
  textTransformer_0 = Pipeline(steps=[
      ('text_bow', CountVectorizer(lowercase=True,\
                                   token_pattern=TOKENS_ALPHANUMERIC_HYPHEN,\
                                   stop_words=stopwords))])
  
  textTransformer_1 = Pipeline(steps=[
      ('text_bow', CountVectorizer(lowercase=True,\
                                   token_pattern=TOKENS_ALPHANUMERIC_HYPHEN,\
                                   stop_words=stopwords))])
  
  FE = ColumnTransformer(
      transformers=[
          ('bool', boolTransformer, BOOL_FEATURES),
          ('cat', catTransformer, CAT_FEATURES),
          ('num', numTransformer, NUM_FEATURES),
          ('text0', textTransformer_0, TEXT_FEATURES[0]),
          ('text1', textTransformer_1, TEXT_FEATURES[1])])
  
  clf = KerasClassifier(keras_classifier_wrapper, epochs=100, batch_size=500, verbose=0)
  
  PL = Pipeline(steps=[('feature_engineer', FE),
                       ('keras_clf', clf)])
  
  PL.fit(X_train_predictors, y_train_target)
  #PL.fit(X_train_OS, y_train_OS)
  
  </code></pre>
  
  <p>I think I understand the problem here however not sure how to solve it.  If it is not possible to integrate sklearn ColumnTransformer+Pipeline into Keras model does Keras have a good way for dealing with fixed data types to feature engineer?  Thank you!</p>