title: how to concat multiple columns in spark while getting the column names to be concatenated from another table (different for each row)
body: <p>I am trying to concat multiple columns in spark using concat function.</p>
  
  <p>For example below is the table for which I have to add new concatenated column</p>
  
  <pre><code>table - **t**
  +---+----+  
  | id|name|
  +---+----+  
  |  1|   a|  
  |  2|   b|
  +---+----+
  </code></pre>
  
  <p>and below is the table which has the information about which columns are to be concatenated for given id (for id 1 column id and name needs to be concatenated and for id 2 only id)</p>
  
  <pre><code>table - **r**
  +---+-------+
  | id|   att |
  +---+-------+
  |  1|id,name|
  |  2|   id  |
  +---+-------+
  </code></pre>
  
  <p>if I join the two tables and do something like below, I am able to concat but not based on the table r (as the new column is having 1,a for first row but for second row it should be 2 only)</p>
  
  <pre><code>t.withColumn("new",concat_ws(",",t.select("att").first.mkString.split(",").map(c =&gt; col(c)): _*)).show
  +---+----+-------+---+
  | id|name|  att  |new|
  +---+----+-------+---+
  |  1|   a|id,name|1,a|
  |  2|   b|  id   |2,b|
  +---+----+-------+---+
  </code></pre>
  
  <p>I have to apply filter before the select in the above query, but I am not sure how to do that in withColumn for each row.</p>
  
  <p>Something like below, if that is possible.</p>
  
  <pre><code>t.withColumn("new",concat_ws(",",t.**filter**("id="+this.id).select("att").first.mkString.split(",").map(c =&gt; col(c)): _*)).show
  </code></pre>
  
  <p>As it will require to filter each row based on the id.</p>
  
  <pre><code>scala&gt; t.filter("id=1").select("att").first.mkString.split(",").map(c =&gt; col(c))
  res90: Array[org.apache.spark.sql.Column] = Array(id, name)
  
  scala&gt; t.filter("id=2").select("att").first.mkString.split(",").map(c =&gt; col(c))
  res89: Array[org.apache.spark.sql.Column] = Array(id)
  </code></pre>
  
  <p>Below is the final required result.</p>
  
  <pre><code>+---+----+-------+---+
  | id|name|  att  |new|
  +---+----+-------+---+
  |  1|   a|id,name|1,a|
  |  2|   b|  id   |2  |
  +---+----+-------+---+
  </code></pre>