title: Merge, Combine 2 column in spark dataframe
body: <p>I have 2 different dataframes and I was able to join them together based on g_id. Just like below:</p>
  <pre><code>df1 = dfx.join(df_gi, regexp_extract(trim(dfx.LOCATION), &quot;.*/GDocs/([0-9]{1,5})/.*&quot;, 1) == df_gi.g_id, &quot;inner&quot;)\
                  .select (dfx[&quot;*&quot;], df_gi[&quot;G_Number2&quot;])
  </code></pre>
  <p>Now, dfx daraframe has a column called G_Number1 and df_gi dataframe has a similar column called G_Number2, Both of these columns combined solves the missing pieces ... Meaning one column has some information and the other has some. Combining both together is the output needed.</p>
  <p><a href="https://i.stack.imgur.com/tKz3x.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/tKz3x.png" alt="enter image description here" /></a></p>
  <p>How can I achieve in pyspark?? I tried the concat function .. but i was way off.
  Thank you in advance.</p>