title: Pyspark: Merging lists which are in dataframe column
body: <p>I have a dataframe as shown below. I want to merge the lists if they have atleast one same value. It is okay to take any of the component number. For example, <strong>[1,2]</strong> and <strong>[1,4,9]</strong> has <strong>1</strong> as common value. So both will be merged to <strong>[1,2,4,9]</strong>. Now <strong>[1,2]</strong> has component number <strong>80</strong> and and <strong>[1,4,9]</strong> has component number <strong>30</strong>. For <strong>[1,2,4,9]</strong> it is okay to have any one of them as component number. In the example given below, I have considered <strong>30</strong>.</p>
  
  <p><img src="https://i.imgur.com/dHJMmoy.png" alt=""></p>
  
  <p>It is possible to have a solution using dataframe or rdd operation avoiding as much iteration as possible? Thanks.</p>