title: How to pass the column of a second dataframe into a UDF in PySpark 1.6.1
body: <p>Here's what I'm trying to do. I want to perform a comparison between each entry of two columns in two different dataframes. The dataframes are shown below:</p>
  
  <pre><code>&gt;&gt;&gt; subject_df.show()
  +------+-------------+
  |USERID|     FULLNAME|
  +------+-------------+
  | 12345|  steve james|
  | 12346| steven smith|
  | 43212|bill dunnigan|
  +------+-------------+
  
  &gt;&gt;&gt; target_df.show()
  +------+-------------+
  |USERID|     FULLNAME|
  +------+-------------+
  |111123|  steve tyler|
  |422226|  linda smith|
  |123333|bill dunnigan|
  | 56453|  steve smith|
  +------+-------------+
  </code></pre>
  
  <p>Here is the logic I tried using:</p>
  
  <pre><code># CREATE FUNCTION    
  def string_match(subject, targets):
      for target in targets:
          &lt;logic&gt;
      return logic_result
  
  # CREATE UDF
  string_match_udf = udf(string_match, IntegerType())
  
  # APPLY UDF
  subject_df.select(subject_df.FULLNAME, string_match_udf(subject_df.FULLNAME, target_df.FULLNAME).alias("score"))
  </code></pre>
  
  <p>This is the error I get when running the code in a pyspark shell:</p>
  
  <pre><code>py4j.protocol.Py4JJavaError: An error occurred while calling o45.select.
  : java.lang.RuntimeException: Invalid PythonUDF PythonUDF#string_match(FULLNAME#2,FULLNAME#5), requires attributes from more than one child.
  </code></pre>
  
  <p>I think the root of my problem is trying to pass in the second column to the function. Should I be using RDDs? Keep in mind that the actual subject_df and target_df are both over 100,000 rows. I'm open to any advice.</p>