title: Lookup in Spark dataframes
body: <p>I am using Spark 1.6 and I would like to know how to implement in lookup in the dataframes.</p>
  
  <p>I have two dataframes employee &amp; department.</p>
  
  <ul>
  <li><p>Employee Dataframe</p>
  
  <pre><code>-------------------
  Emp Id | Emp Name
  ------------------
  1 | john
  2 | David
  </code></pre></li>
  <li><p>Department Dataframe</p>
  
  <pre><code>--------------------
  Dept Id | Dept Name | Emp Id
  -----------------------------
  1 | Admin | 1
  2 | HR | 2
  </code></pre></li>
  </ul>
  
  <p>I would like to lookup emp id from the employee table to the department table and get the dept name. So, the resultset would be</p>
  
  <pre><code>Emp Id | Dept Name
  -------------------
  1 | Admin
  2 | HR
  </code></pre>
  
  <p>How do I implement this look up UDF feature in SPARK. I don't want to use JOIN on both the dataframes.</p>