title: Plot xgboost eval metrics with bayesian optimization
body: <p>I'm using this piece of code to tune and train an XGBoost with Bayesian Optimization. I would like to plot the logloss against the epochs, but I haven't found a way to do it.</p>
  <p>Here's my XGBoost code:</p>
  <pre><code>def bayes_fun(parameters):
      '''
      Function that that sets paramters and performs cross-validation for Bayesian Optimisation
      '''
      parameters = parameters[0]                                # setting regressor parameters
      reg = xgb.XGBClassifier(objective = 'multi:softmax',       # objective function (target variable follows poisson distribution)
                                 num_class = 3,
                                 eval_metric = 'mlogloss',          # eval metric for poisson distribution
                                 base_score = 1,                    # specific to scoring 
                                 learning_rate = [8]               # learning rate
                                 random_state = 1234,               # set seed
                                 max_depth = int(parameters[0]),    # maximum depth trees can grow
                                 min_child_weight=parameters[1],    # amount of weight for a tree to produce a child
                                 gamma = parameters[2],             # controls minimum loss reduction
                                 reg_alpha = parameters[3],         # L1 regularisation term on weights
                                 reg_lambda = parameters[4],        # L2 regularisation term on weights
                                 subsample = parameters[5],         # ratio of the training instances
                                 colsample_bytree = parameters[6],  # ratio of number of features used when constructing a tree 
                                 colsample_bylevel = parameters[7],
                                 max_delta_step =parameters[9]) # ratio of number of features used at each node of each tree
      
      reg_param = reg.get_xgb_params() # getting parameters from regressor
      
      # cross validation
      bst = xgb.cv(params = reg_param,          # setting parameters
                   dtrain = DTrain,             # training data 
                   folds = folds_list,          # list of folds to use
                   num_boost_round = 9999,      # number of iterations
                   early_stopping_rounds = 30,  # early stopping (reduce overfitting)
                   verbose_eval = False)        # don\t show output text
      
      score = -100 * bst['test-mlogloss-mean'].iloc[-1] # if you are using negative log-likelyhood use minus
      return score
  
  # set parameter ranges
  domains = [{'name': 'max_depth', 'type': 'discrete', 'domain': (2, 10)},
             {'name': 'minchild', 'type': 'continuous', 'domain': (5, 500)},
             {'name': 'gamma', 'type': 'continuous', 'domain': (0, 1)},
             {'name': 'alpha', 'type': 'continuous', 'domain': (0, 10)},
             {'name': 'lmbd', 'type': 'continuous', 'domain': (0, 50)},
             {'name': 'subsample', 'type': 'continuous', 'domain': (0.50, 0.99)},
             {'name': 'colsample_bytree', 'type': 'continuous', 'domain': (0.40, 0.70)},
             {'name': 'colsample_bylevel', 'type': 'continuous', 'domain': (0.40, 0.70)},
             {'name': 'learning_rate', 'type': 'continuous', 'domain': (0,1)},
             {'name': 'max_delta_step', 'type': 'continuous', 'domain': (0,1)}]
  
  # Bayesian Optmisation (check package notes to adjust parameters to suit your data)
  optimizer = BayesianOptimization(f=bayes_fun, domain=domains, model_type='GP', acquisition_type='LCB', initial_design_type = 'latin', initial_design_numdata=5, exact_feval=True, maximize=False)
  
  # run Bayesian for 15 rounds
  optimizer.run_optimization(max_iter=200, verbosity=True)
  </code></pre>
  <p>Here's a code I found to get the logloss per epoch plot, but it's just with a simple <code>XGBClassifier()</code> call:</p>
  <pre><code>model.fit(X, data_nobands2.categoric, eval_metric=['merror','mlogloss'], eval_set=eval_s)
  
  results = model.evals_result()
  epochs = len(results['validation_0']['merror'])
  
  import matplotlib.pyplot as pyplot
  x_axis = range(0, epochs)
  # plot log loss
  fig, ax = pyplot.subplots()
  ax.plot(x_axis, results['validation_0']['mlogloss'], label='Train')
  ax.plot(x_axis, results['validation_1']['mlogloss'], label='Test')
  ax.legend()
  pyplot.ylabel('Log Loss')
  pyplot.title('XGBoost Log Loss')
  pyplot.show()
  </code></pre>
  <p>I think possibly part of the last code can be used for the bayesian optimization, but I don;t know how to do it. Has anyone done it, or something similar?</p>