title: Adding column to pyspark dataframe based on conditions from other pyspark dataframes
body: <p>I am currently learning pyspark and currently working on adding columns to pyspark dataframes using multiple conditions.</p>
  
  <p>I have tried working with UDFs but getting some errors like:</p>
  
  <pre><code>TypeError: 'object' object has no attribute '__getitem__'
  
  from pyspark.sql.functions import udf, struct
  from pyspark.sql.types import IntegerType, StringType, FloatType
  from pyspark.sql.functions import pandas_udf, PandasUDFType
  #first dataframe (superset)    
  df1 = simple_example1
  #second dataframe
  df = diff_cols.dropna()
  
  def func(x,y):
      z = df1[(df1['a'] == x) &amp; (df1['b'] &lt;= (y+10000000000)) &amp; (df1['b'] &gt;= (y-10000000000))]
      z = z[(z["c"] ==1) | (z["d"] ==1)]
      z = z[(z["e"]!=0) | (z["f"]!=0) | (z["g"]!=0) | (z["h"]!=0)]
      return 1 if z.count() &gt; 3 else 0
  udf_func = udf(func, IntegerType())
  df = df.withColumn('status', udf_func(df['a'],df['b']))
  </code></pre>
  
  <p>what i am trying is as follow:
  1. for each row of df filter data from df1 where parameter a is equal to the parameter in df and parameter b should be in between b-10 to b+10
  2. then filter that data further with either c or d = 1
  3. then filter that data further if any of parameters from e f g h are non 0
  4. then count number of rows in the subset and assign 0/1
  5. return this 0/1 in status column of df</p>