title: Explode array in apache spark Data Frame
body: <p>I am trying to flatten a schema of existing dataframe with nested fields. Structure of my dataframe is something like that:</p>
  
  <pre><code>root  
  |-- Id: long (nullable = true)  
  |-- Type: string (nullable = true)  
  |-- Uri: string (nullable = true)    
  |-- Type: array (nullable = true)  
  |    |-- element: string (containsNull = true)  
  |-- Gender: array (nullable = true)  
  |    |-- element: string (containsNull = true)
  </code></pre>
  
  <p>Type and gender can contain array of elements, one element or null value.
  I tried to use the following code:</p>
  
  <pre><code>var resDf = df.withColumn("FlatType", explode(df("Type")))
  </code></pre>
  
  <p>But as a result in a resulting data frame I loose rows for which I had null values for Type column. It means, for example, if I have 10 rows and in 7 rows type is null and in 3 type is not null, after I use explode in resulting data frame I have only three rows.</p>
  
  <p>How can I keep rows with null values but explode array of values?</p>
  
  <p>I found some kind of workaround but still stuck in one place. For standard types we can do the following:</p>
  
  <pre><code>def customExplode(df: DataFrame, field: String, colType: String): org.apache.spark.sql.Column = {
  var exploded = None: Option[org.apache.spark.sql.Column]
  colType.toLowerCase() match {
    case "string" =&gt; 
      val avoidNull = udf((column: Seq[String]) =&gt;
      if (column == null) Seq[String](null)
      else column)
      exploded = Some(explode(avoidNull(df(field))))
    case "boolean" =&gt; 
      val avoidNull = udf((xs: Seq[Boolean]) =&gt;
      if (xs == null) Seq[Boolean]()
      else xs)
      exploded = Some(explode(avoidNull(df(field))))
    case _ =&gt; exploded = Some(explode(df(field)))
  }
  exploded.get
  </code></pre>
  
  <p>}</p>
  
  <p>And after that just use it like this:</p>
  
  <pre><code>val explodedField = customExplode(resultDf, fieldName, fieldTypeMap(field))
  resultDf = resultDf.withColumn(newName, explodedField)
  </code></pre>
  
  <p>However, I have a problem for struct type for the following type of structure:</p>
  
  <pre><code> |-- Address: array (nullable = true)
   |    |-- element: struct (containsNull = true)
   |    |    |-- AddressType: array (nullable = true)
   |    |    |    |-- element: string (containsNull = true) 
   |    |    |-- DEA: array (nullable = true)
   |    |    |    |-- element: struct (containsNull = true)
   |    |    |    |    |-- Number: array (nullable = true)
   |    |    |    |    |    |-- element: string (containsNull = true)
   |    |    |    |    |-- ExpirationDate: array (nullable = true)
   |    |    |    |    |    |-- element: timestamp (containsNull = true)
   |    |    |    |    |-- Status: array (nullable = true)
   |    |    |    |    |    |-- element: string (containsNull = true)
  </code></pre>
  
  <p>How can we process that kind of schema when DEA is null?</p>
  
  <p>Thank you in advance.</p>
  
  <p>P.S. I tried to use Lateral views but result is the same.</p>