title: How to create a custom Python class to be used in Pipeline for dropping highly correlated features?
body: <p>I'm using the Sklearn <code>Pipeline</code> + <code>GridSearchCV</code> for the data-preprocessing/hyperparameter tuning of a Deep Neural Network (regression).
  For the preprocessing, I would need a custom class for dropping the highly correlated columns of the dataset. Here is my code (to be improved):</p>
  <pre class="lang-py prettyprint-override"><code>class MyDecorrelator():
     
     def __init__(self, threshold):
         self.threshold = threshold
     
     def fit(self, X, y=None):
         return self
     
     def transform(self, X, y = None):
         correlated_features = set()  # Set of all the names of correlated columns
         corr_matrix = X.corr()
         for i in range(len(corr_matrix.columns)):
             for j in range(i):
                 if abs(corr_matrix.iloc[i, j]) &gt; self.threshold: # we are interested in absolute coeff value
                     colname = corr_matrix.columns[i]  # getting the name of column
                     correlated_features.add(colname)
         return X.drop(labels=correlated_features, axis=1, inplace=True)
  
  
  
  def create_model(input_shape = 150, optimizer='adam', learn_rate=0.01, activation='relu', init='uniform', hidden_layers = 1, dropout = 0.5, hidden_size=64):
     # create model
     model = Sequential()
     model.add(Dense(input_shape, activation=activation, kernel_initializer=init, ))
     for i in range(hidden_layers):
         model.add(Dense(hidden_size, activation=activation))
         model.add(Dropout(dropout), )
     model.add(Dense(1, activation='linear'))
    # Compile model
     model.compile(loss='mean_absolute_error', optimizer=optimizer)
     return model
  
  
  estimator = Pipeline([
             ('scaler', MinMaxScaler(feature_range=(0.0, 1.0))),
             ('decorrelation', MyDecorrelator(0.9)),
             ('feature_selector', SelectKBest()),
             ('kr', KerasRegressor(build_fn = create_model))
             ], verbose = True)
  
  param_grid = [{
     'kr__optimizer': ['RMSprop', 'Adam'],
     'kr__epochs': [100, 300],
     #'kr__init': [ 'uniform', 'zeros', 'normal', ], 
     'kr__batch_size':[32, 128],
     'kr__learn_rate': [0.01, 0.1],
     'kr__activation': ['relu', 'sigmoid'],
     'kr__dropout': [0.9, 0.1],
     'kr__hidden_layers': [2, 3],
     'kr__hidden_size': [64, 128],
     'feature_selector__score_func': [mutual_info_regression],
     'feature_selector__k': [k],
     'kr__input_shape': [k]
  }
  
  for k in [50, 100] ]
  
  grid = HalvingGridSearchCV(estimator=estimator, param_grid=param_grid, n_jobs=-1, cv=KFold(n_splits = 5), verbose=10)
  </code></pre>
  <p>but, when I try to run the <code>grid.fit(X, Y)</code>, it gives the following error:</p>
  <pre><code>'MyDecorrelator' object has no attribute 'set_params'
  </code></pre>
  <p>Furthermore, if I try to change the first line to  <code>class MyDecorrelator(BaseEstimator):</code>, it says</p>
  <pre><code>AttributeError: 'numpy.ndarray' object has no attribute 'corr'
  </code></pre>
  <p>How to fix it?</p>
  <p><strong>Update:</strong>
  I have corrected using the solution by Comsavvy, but as a result i get a warning: <code>UserWarning: One or more of the test scores are non-finite: [nan nan nan ...]</code>. How can it happen? It worked without the decorrelation.</p>