title: How to create a new column in a Spark DataFrame based on a second DataFrame (Java)?
body: <p>I have two Spark DataFrames where one of them has two cols, id and Tag. A second DataFrame has an id col, but missing the Tag. The first Dataframe is essentially a dictionary, each id appears once, while in the second DataFrame and id may appear several times. What I need is to create a new col in the second DataFrame that has the Tag as a function of the id in each row (in the second DataFrame). I think this can be done by converting to RDDs first ..etc, but I thought there must be a more elegant way using DataFrames (in Java). Example: given a <strong>df1 Row-> id: 0, Tag: "A"</strong>, a <strong>df2 Row1-> id: 0, Tag: null</strong>, a <strong>df2 Row2-> id: 0, Tag: "B"</strong>, I need to create a Tag col in the resulting DataFrame df3 equal to df1(id=0) = "A" IF df2 Tag was null, but keep original Tag if not null => resulting in <strong>df3 Row1-> id: 0, Tag: "A"</strong>, <strong>df3 Row2-> id: 0, Tag: "B"</strong>. Hope the example is clear.</p>
  
  <pre><code>|   ID  |   No.   |  Tag  | new Tag Col |
  |    1  |  10002  |   A   |      A      |
  |    2  |  10003  |   B   |      B      | 
  |    1  |  10004  | null  |      A      |
  |    2  |  10005  | null  |      B      |
  </code></pre>