title: PySpark - Creating a UDF to Concatenate Two Columns of Lists Into a List of Lists
body: <p>Suppose I have a dataframe with columns <code>A, B, C, D, E</code>. Each of these can be comprised of a list of values or <code>null</code>. I would like to concatenate these values into a final column <code>F</code> that consists of a list of lists that ignores the <code>null</code> values and preserves the original columns' order.</p>
  <p>Ex input: <code>[a,b,c] | [b,c,d] | null | null | [z]</code>
  Ex output: <code>[[a,b,c], [b,c,d], [z]]</code></p>
  <p>Unfortunately, concat_ws flattens everything, so I believe I must use a UDF. Does anyone have a solution to this problem?</p>