title: XGBoost training fails if null values exists (setHandleInvalid "keep" exists for whole pipeline)
body: <p>I'm training an XGBoostRegressor model using Spark (Scala), and I've noticed that the number of predicted values is less than what was given to the model using model.transform(df).</p>
  
  <p>The problem is due to the fact there are (And should be, as per my use case) NULL values. I've handled those alongside the way by using setHandleInvalid in every phase I have (and specifically - stringIndexer, oneHotEncoder, vectorAssembler).</p>
  
  <p>But, still, if I'm using "keep", the model fails to train, but if I'm using "skip" (btw, only on the vectorAssembler), so the model managed to train, but just "discards" the records where even 1 field has a null.</p>
  
  <p>Tried tons of google but didn't really see any solution for it.</p>
  
  <p>Would appreciate anyone's input.</p>
  
  <p>Thanks in advance.</p>
  
  <p>Spark, Scala, XGBoost Docs, saw several PRs that didn't help, tried several strategies for dealing Null values but not even one succeeded.</p>
  
  <p>For keep case (where the train fails) -></p>
  
  <pre><code>  .setInputCol("country_code")
    .setOutputCol("country_code_indexed")
    .setHandleInvalid("keep")
  
  val oneHotEncoder = new OneHotEncoderEstimator()
  .setInputCol("user_country_code_indexed")
  .setOutputCol("user_country_oneHotEncoded")
  .setHandleInvalid("keep")
  
  val assembler =  new VectorAssembler()
    .setInputCols(trainUpdated.drop("label",
                                 "someCol1",
                                 "someCol2", 
                                 "country_code", 
                                 "country_code_indexed").columns)
    .setOutputCol("features")
    .setHandleInvalid("keep")
  
  val xgboostRegressor = new XGBoostRegressor(Map[String, Any](
    "num_round" -&gt; 100,
    "num_workers" -&gt; 10,  //num of instances * num of cores is the max.
    "objective" -&gt; "reg:linear",
    "eta" -&gt; 0.1,
    "gamma" -&gt; 0.5,
    "max_depth" -&gt; 6, 
    "early_stopping_rounds" -&gt; 9,
    "seed" -&gt; 1234,
    "lambda" -&gt; 0.4,
    "alpha" -&gt; 0.3,
    "colsample_bytree" -&gt; 0.6,
    "subsample" -&gt; 0.3
    ))
  </code></pre>
  
  <p>Then I get ->
  <code>ml.dmlc.xgboost4j.java.XGBoostError: XGBoostModel training failed</code></p>
  
  <p>Expected result - model trains <em>with</em> null values (as it's its default behavior...) and returns exact num of records as it had during train / test (fit / transform, same strategy for both).</p>