title: Put customized functions in Sklearn pipeline
body: <p>In my classification scheme, there are several steps including:</p>
  <ol>
  <li>SMOTE (Synthetic Minority Over-sampling Technique)</li>
  <li>Fisher criteria for feature selection</li>
  <li>Standardization (Z-score normalisation)</li>
  <li>SVC (Support Vector Classifier)</li>
  </ol>
  <p>The main parameters to be tuned in the scheme above are percentile (2.) and hyperparameters for SVC (4.) and I want to go through grid search for tuning.</p>
  <p>The current solution builds a &quot;partial&quot; pipeline including step 3 and 4 in the scheme <code>clf = Pipeline([('normal',preprocessing.StandardScaler()),('svc',svm.SVC(class_weight='auto'))])</code>
  and breaks the scheme into two parts:</p>
  <ol>
  <li><p>Tune the percentile of features to keep through the first grid search</p>
  <pre class="lang-py prettyprint-override"><code>skf = StratifiedKFold(y)
  for train_ind, test_ind in skf:
      X_train, X_test, y_train, y_test = X[train_ind], X[test_ind], y[train_ind], y[test_ind]
      # SMOTE synthesizes the training data (we want to keep test data intact)
      X_train, y_train = SMOTE(X_train, y_train)
      for percentile in percentiles:
          # Fisher returns the indices of the selected features specified by the parameter 'percentile'
          selected_ind = Fisher(X_train, y_train, percentile) 
          X_train_selected, X_test_selected = X_train[selected_ind,:], X_test[selected_ind, :]
          model = clf.fit(X_train_selected, y_train)
          y_predict = model.predict(X_test_selected)
          f1 = f1_score(y_predict, y_test)
  </code></pre>
  <p>The f1 scores will be stored and then be averaged through all fold partitions for all percentiles, and the percentile with the best CV score is returned. The purpose of putting 'percentile for loop' as the inner loop is to allow fair competition as we have the same training data (including synthesized data) across all fold partitions for all percentiles.</p>
  </li>
  <li><p>After determining the percentile, tune the hyperparameters by second grid search</p>
  <pre class="lang-py prettyprint-override"><code>skf = StratifiedKFold(y)
  for train_ind, test_ind in skf:
      X_train, X_test, y_train, y_test = X[train_ind], X[test_ind], y[train_ind], y[test_ind]
      # SMOTE synthesizes the training data (we want to keep test data intact)
      X_train, y_train = SMOTE(X_train, y_train)
      for parameters in parameter_comb:
          # Select the features based on the tuned percentile
          selected_ind = Fisher(X_train, y_train, best_percentile) 
          X_train_selected, X_test_selected = X_train[selected_ind,:], X_test[selected_ind, :]
          clf.set_params(svc__C=parameters['C'], svc__gamma=parameters['gamma'])
          model = clf.fit(X_train_selected, y_train)
          y_predict = model.predict(X_test_selected)
          f1 = f1_score(y_predict, y_test)
  </code></pre>
  </li>
  </ol>
  <p>It is done in the very similar way, except we tune the hyperparamter for SVC rather than percentile of features to select.</p>
  <p>My questions are:</p>
  <ol>
  <li><p>In the current solution, I only involve 3. and 4. in the <code>clf</code> and do 1. and 2. kinda &quot;manually&quot; in two nested loop as described above. Is there any way to include all four steps in a pipeline and do the whole process at once?</p>
  </li>
  <li><p>If it is okay to keep the first nested loop, then is it possible (and how) to simplify the next nested loop using a single pipeline</p>
  <pre class="lang-py prettyprint-override"><code>clf_all = Pipeline([('smote', SMOTE()),
                      ('fisher', Fisher(percentile=best_percentile))
                      ('normal',preprocessing.StandardScaler()),
                      ('svc',svm.SVC(class_weight='auto'))]) 
  </code></pre>
  <p>and simply use <code>GridSearchCV(clf_all, parameter_comb)</code> for tuning?</p>
  <p>Please note that both <code>SMOTE</code> and <code>Fisher</code> (ranking criteria) have to be done only for the training data in each fold partition.</p>
  </li>
  </ol>
  <p>It would be so much appreciated for any comment.</p>
  <p><code>SMOTE</code> and <code>Fisher</code> are shown below:</p>
  <pre class="lang-py prettyprint-override"><code>def Fscore(X, y, percentile=None):
      X_pos, X_neg = X[y==1], X[y==0]
      X_mean = X.mean(axis=0)
      X_pos_mean, X_neg_mean = X_pos.mean(axis=0), X_neg.mean(axis=0)
      deno = (1.0/(shape(X_pos)[0]-1))*X_pos.var(axis=0) +(1.0/(shape(X_neg[0]-1))*X_neg.var(axis=0)
      num = (X_pos_mean - X_mean)**2 + (X_neg_mean - X_mean)**2
      F = num/deno
      sort_F = argsort(F)[::-1]
      n_feature = (float(percentile)/100)*shape(X)[1]
      ind_feature = sort_F[:ceil(n_feature)]
      return(ind_feature)
  </code></pre>
  <p><code>SMOTE</code> is from <a href="https://github.com/blacklab/nyan/blob/master/shared_modules/smote.py" rel="noreferrer">https://github.com/blacklab/nyan/blob/master/shared_modules/smote.py</a>, it returns the synthesized data. I modified it to return the original input data stacked with the synthesized data along with its labels and synthesized ones.</p>
  <pre class="lang-py prettyprint-override"><code>def smote(X, y):
      n_pos = sum(y==1), sum(y==0)
      n_syn = (n_neg-n_pos)/float(n_pos) 
      X_pos = X[y==1]
      X_syn = SMOTE(X_pos, int(round(n_syn))*100, 5)
      y_syn = np.ones(shape(X_syn)[0])
      X, y = np.vstack([X, X_syn]), np.concatenate([y, y_syn])
      return(X, y)
  </code></pre>