title: plot_partial_dependence() from scikit-learn incorrectly raises NotFittedError for properly fitted models (e.g. KerasRegressor or LGBMClassifier)
body: <p>I'm trying to create partial dependence plots using <code>sklearn.inspection.plot_partial_dependence</code> on a model that I successfully built using keras and the keras sklearn wrapper utility (see code block below). The wrapped model successfully builds, it can use the fit method, and after fitting it can use the predict method with expected results. All indications are that it's a valid estimator. However, when I try to run plot_partial_dependence from sklearn.inspection, I get some error text implying that it is is <em>not</em> a valid estimator, even though I can demonstrate that it is.</p>
  
  <p>I have edited this to be more easily reproducible by using the sklearn example Boston housing data.</p>
  
  <pre><code>from sklearn.datasets import load_boston
  from sklearn.inspection import plot_partial_dependence, partial_dependence
  from keras.wrappers.scikit_learn import KerasRegressor
  import keras
  import tensorflow as tf
  import pandas as pd
  
  boston = load_boston()
  feature_names = boston.feature_names
  X = pd.DataFrame(boston.data, columns=boston.feature_names)
  y = boston.target
  mean = X.describe().transpose()['mean']
  std = X.describe().transpose()['std']
  X_norm = (X-mean)/std
  
  def build_model_small():
      model = keras.Sequential([
          keras.layers.Dense(64, activation='relu', input_shape=[len(X.keys())]),
          keras.layers.Dense(64, activation='relu'),
          keras.layers.Dense(1)
          ])
  
      optimizer = keras.optimizers.RMSprop(0.0005)
  
      model.compile(loss='mse',
                optimizer=optimizer,
                metrics=['mae', 'mse', 'mape'])
      return model
  
  
  kr = KerasRegressor(build_fn=build_model_small,verbose=0)
  kr.fit(X_norm,y, epochs=100, validation_split = 0.2)
  pdp_plot = plot_partial_dependence(kr,X_norm,feature_names)
  </code></pre>
  
  <p>Like I said, if I run <code>kr.predict(X.head(20))</code>, I get 20 predictions of <code>y</code> values for the first 20 rows of <code>X</code>, as one would expect from a valid estimator.</p>
  
  <p>But the error text I get from plot_partial_dependence is as follows:</p>
  
  <pre><code>Traceback (most recent call last):
    File "temp_ML_tf_sklearn_postproc.py", line 79, in &lt;module&gt;
      pdp_plot = plot_partial_dependence(kr,X,labels[:-1])
    File "/home/mymachine/anaconda3/lib/python3.7/site-packages/sklearn/inspection/_partial_dependence.py", line 678, in plot_partial_dependence
      for fxs in features)
    File "/home/mymachine/anaconda3/lib/python3.7/site-packages/joblib/parallel.py", line 921, in __call__
      if self.dispatch_one_batch(iterator):
    File "/home/mymachine/anaconda3/lib/python3.7/site-packages/joblib/parallel.py", line 759, in dispatch_one_batch
      self._dispatch(tasks)
    File "/home/mymachine/anaconda3/lib/python3.7/site-packages/joblib/parallel.py", line 716, in _dispatch
      job = self._backend.apply_async(batch, callback=cb)
    File "/home/mymachine/anaconda3/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 182, in apply_async
      result = ImmediateResult(func)
    File "/home/mymachine/anaconda3/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 549, in __init__
      self.results = batch()
    File "/home/mymachine/anaconda3/lib/python3.7/site-packages/joblib/parallel.py", line 225, in __call__
      for func, args, kwargs in self.items]
    File "/home/mymachine/anaconda3/lib/python3.7/site-packages/joblib/parallel.py", line 225, in &lt;listcomp&gt;
      for func, args, kwargs in self.items]
    File "/home/mymachine/anaconda3/lib/python3.7/site-packages/sklearn/inspection/_partial_dependence.py", line 307, in partial_dependence
      "'estimator' must be a fitted regressor or classifier."
  ValueError: 'estimator' must be a fitted regressor or classifier.
  </code></pre>
  
  <p>I looked at the source code for plot_partial_dependence, and it had the following to say.
  First, in the docstring, it says the first input <code>estimator</code> must be...</p>
  
  <blockquote>
  <pre><code>  A fitted estimator object implementing :term:`predict`,
      :term:`predict_proba`, or :term:`decision_function`.
      Multioutput-multiclass classifiers are not supported.
  </code></pre>
  </blockquote>
  
  <p>My estimator does in deed implement .predict.</p>
  
  <p>Second, the line called out in the errr traceback calls a checker that checks whether it is a regressor or classifier:</p>
  
  <pre><code>if not (is_classifier(estimator) or is_regressor(estimator)):
      raise ValueError(
          "'estimator' must be a fitted regressor or classifier."
      )
  </code></pre>
  
  <p>I looked into the source code of is_regressor(), and it is a one liner like so:</p>
  
  <pre><code>return getattr(estimator, "_estimator_type", None) == "regressor"
  </code></pre>
  
  <p>So I tried to sort of hack it by doing <code>setattr(mp,'_estimator_type','regressor')</code> and it just said <code>Attribute Error: can't set attribute</code>, so that's one cheap workaround that didn't work.</p>
  
  <p>I even tried the even hackier fix and temporarily commented out the offending check in the source of _partial_dependence.py (the if statement I copied above), and got the following error:</p>
  
  <pre><code>Traceback (most recent call last):
    File "temp_ML_tf_sklearn_postproc.py", line 79, in &lt;module&gt;
      pdp_plot = plot_partial_dependence(kr,X,labels[:-1])
    File "/home/billy/anaconda3/lib/python3.7/site-packages/sklearn/inspection/_partial_dependence.py", line 678, in plot_partial_dependence
      for fxs in features)
    File "/home/billy/anaconda3/lib/python3.7/site-packages/joblib/parallel.py", line 921, in __call__
      if self.dispatch_one_batch(iterator):
    File "/home/billy/anaconda3/lib/python3.7/site-packages/joblib/parallel.py", line 759, in dispatch_one_batch
      self._dispatch(tasks)
    File "/home/billy/anaconda3/lib/python3.7/site-packages/joblib/parallel.py", line 716, in _dispatch
      job = self._backend.apply_async(batch, callback=cb)
    File "/home/billy/anaconda3/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 182, in apply_async
      result = ImmediateResult(func)
    File "/home/billy/anaconda3/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 549, in __init__
      self.results = batch()
    File "/home/billy/anaconda3/lib/python3.7/site-packages/joblib/parallel.py", line 225, in __call__
      for func, args, kwargs in self.items]
    File "/home/billy/anaconda3/lib/python3.7/site-packages/joblib/parallel.py", line 225, in &lt;listcomp&gt;
      for func, args, kwargs in self.items]
    File "/home/billy/anaconda3/lib/python3.7/site-packages/sklearn/inspection/_partial_dependence.py", line 317, in partial_dependence
      check_is_fitted(est)
    File "/home/billy/anaconda3/lib/python3.7/site-packages/sklearn/utils/validation.py", line 967, in check_is_fitted
      raise NotFittedError(msg % {'name': type(estimator).__name__})
  sklearn.exceptions.NotFittedError: This KerasRegressor instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.
  </code></pre>
  
  <p>Which goes all the way back around to the problem of the sklearn functions not thinking this model is fitted, when it really is. Anyway at this point I decided not to try any more dangerous, hacky fixes of tinkering with the source code.</p>
  
  <p>I also tried passing <code>kr.fit(X,y,etc...)</code> directly in as the first argument of plot_partial_dependence. The computer spun up for a few minutes, indicating the fit was in fact running, but then I got the same error once it tried to run the partial dependence plot.</p>
  
  <p>One more rather confusing clue. I tried using the keras/sklearn wrapped pipeline in another sklearn function entirely, to see if it would work with any sklearn utilities at all. This time, I did:</p>
  
  <pre><code>from sklearn.model_selection import cross_validate
  cv_scores = cross_validate(kr,X_norm,y, cv=4, return_train_score=True, n_jobs=-1)`
  </code></pre>
  
  <p>and it worked! So I don't think there is something inherently wrong with my use of <code>keras.wrappers.scikit_learn.KerasRegressor</code>.</p>
  
  <p>This may just be a case where what I'm trying to do is an edge case not planned for in the plot_partial_dependence source code specifically and I'm out of luck, but I'm wondering if anyone else has seen issues like this and has a solution or work-around.</p>
  
  <p>I am using sklearn 0.22.1 and Python 3.7.3 (Anaconda) by the way. And to be clear, I have used plot_partial_dependence on sklearn-built models and even pipelines. This problem is only happening with the keras-based model. Many thanks for any input people may have.</p>
  
  <p>Edit:</p>
  
  <p>The previous version of this question involved building a pipeline with StandardScaler() and then the KerasRegressor wrapped object. Since then I discovered it happens even with just the KerasRegressor object, i.e. I have isolated the problem to that, not the pipeline. So as one commenter suggested, I took the pipeline part out of the question to make it simpler and more to the point. </p>