title: Apply sklearn preprocessing function by groups in a pipeline
body: <p>I am trying to apply a <code>sklearn</code> preprocessing function on a per group basis in a pipeline and then remove the grouping variable.  Below is some sample code  in which I would like to apply the <code>StandardScaler</code> per group based on the variable <code>group1</code> and then drop <code>group1</code>from the pipeline. What is the best way to do this?</p>
  <pre><code>import pandas as pd
  from random import choices
  from sklearn.datasets import load_iris
  from sklearn.model_selection import train_test_split
  from sklearn.pipeline import Pipeline
  from sklearn.feature_selection import VarianceThreshold
  from sklearn.preprocessing import StandardScaler
  from sklearn.preprocessing import OneHotEncoder
  from sklearn.decomposition import PCA
  from sklearn.compose import ColumnTransformer
  data = load_iris()
  df = pd.DataFrame(data['data'], columns=data['feature_names'])
  df['class'] = data['target']
  df['group1']=choices(['A','B','C'],k=150)
  df['group2']=choices(['on','off'],k=150)
  df[df.select_dtypes('object').columns.tolist()]=df.select_dtypes(['object']).apply(lambda x: x.astype('category'))
  
  y=df['class']
  X=df.drop('class',axis=1)
  
  X_train, X_test,Y_train, Y_test = train_test_split(X, y, test_size=0.10)
  num_attribs=X_train.select_dtypes(include='number').columns.tolist()
  cat_attribs=X_train.select_dtypes(include='category').columns.tolist()
  
  num_pipeline = Pipeline([
  #('imputer', Imputer(strategy=&quot;median&quot;)),
  ('nzv', VarianceThreshold()),
  ('std_scaler', StandardScaler()),
  ('pca', PCA(n_components=0.90, svd_solver='full')), # &lt;- apply this on a per group basis using 'group1 then drop group1'   
  ])
  cat_pipeline = Pipeline([
  ('one_hot', OneHotEncoder(sparse=False)),
  ])
  
  full_pipeline = ColumnTransformer([
          (&quot;num&quot;, num_pipeline, num_attribs),
          (&quot;cat&quot;, OneHotEncoder(), cat_attribs),
      ])
  
  X_train = full_pipeline.fit_transform(X_train)
  X_test = full_pipeline.transform(X_test)
  </code></pre>