title: XGBoost with GridSearchCV, Scaling, PCA, and Early-Stopping in sklearn Pipeline
body: <p>I want to combine a XGBoost model with input scaling and feature space reduction by PCA. In addition, the hyperparameters of the model as well as the number of components used in the PCA should be tuned using cross-validation. And to prevent the model from overfitting, early stopping should be added. </p>
  
  <p>For combining the various steps, I decided to use sklearn's <code>Pipeline</code> functionalities.</p>
  
  <p>At the beginning, I had some problems making sure, that the PCA is also applied to the validation set. But I think using <code>XGB__eval_set</code> makes the deal. </p>
  
  <p>The code is actually running without any errors, but seems to run forever (at some point the CPU usage of all cores goes down to zero but the processes continue to run for hours; had to kill the session at some point).</p>
  
  <pre><code>from sklearn.model_selection import GridSearchCV, train_test_split
  from sklearn.decomposition import PCA
  from sklearn.pipeline import Pipeline
  from sklearn.preprocessing import StandardScaler
  from xgboost import XGBRegressor   
  
  # Train / Test split
  X_train, X_test, y_train, y_test = train_test_split(X_with_features, y, test_size=0.2, random_state=123)
  
  # Train / Validation split
  X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=123)
  
  # Pipeline
  pipe = Pipeline(steps=[("Scale", StandardScaler()),
                         ("PCA", PCA()),
                         ("XGB", XGBRegressor())])
  
  # Hyper-parameter grid (Test only)
  grid_param_pipe = {'PCA__n_components': [5],
                     'XGB__n_estimators': [1000],
                     'XGB__max_depth': [3],
                     'XGB__reg_alpha': [0.1],
                     'XGB__reg_lambda': [0.1]}
  
  # Grid object
  grid_search_pipe = GridSearchCV(estimator=pipe,
                                  param_grid=grid_param_pipe,
                                  scoring="neg_mean_squared_error",
                                  cv=5,
                                  n_jobs=5,
                                  verbose=3)
  
  # Run CV
  grid_search_pipe.fit(X_train, y_train, XGB__early_stopping_rounds=10, XGB__eval_metric="rmse", XGB__eval_set=[[X_val, y_val]])
  </code></pre>