title: PySpark - map with lambda function
body: <p>I'm facing an issue when mixing python map and lambda functions on a Spark environment.</p>
  
  <p>Given df1, my source dataframe:</p>
  
  <pre><code>Animals     | Food      | Home
  ----------------------------------
  Monkey      | Banana    | Jungle
  Dog         | Meat      | Garden
  Cat         | Fish      | House
  Elephant    | Banana    | Jungle
  Lion        | Meat      | Desert
  </code></pre>
  
  <p>I want to create another dataframe df2. It will contain two columns with a row per column of df1 (3 in my example).
  The first column would contain the name of df1 columns. The second column would contain an array of elements with the most occurrences (n=3 in the example below) and the count.</p>
  
  <pre><code>Column      | Content
  -----------------------------------------------------------
  Animals     | [("Cat", 1), ("Dog", 1), ("Elephant", 1)]
  Food        | [("Banana", 2), ("Meat", 2), ("Fish", 1)]
  Home        | [("Jungle", 2), ("Desert", 1), ("Garden", 1)]
  </code></pre>
  
  <p>I tried to do it with python list, map and lambda functions but I had conflicts with PySpark functions:</p>
  
  <pre><code>def transform(df1):
      # Number of entry to keep per row
      n = 3
      # Add a column for the count of occurence
      df1 = df1.withColumn("future_occurences", F.lit(1))
  
      df2 = df1.withColumn("Content",
          F.array(
              F.create_map(
                  lambda x: (x,
                      [
                          str(row[x]) for row in df1.groupBy(x).agg(
                              F.sum("future_occurences").alias("occurences")
                          ).orderBy(
                              F.desc("occurences")
                          ).select(x).limit(n).collect()
                      ]
                  ), df1.columns
              )
          )
      )
      return df2
  </code></pre>
  
  <p>The error is:</p>
  
  <pre><code>TypeError: Invalid argument, not a string or column: &lt;function &lt;lambda&gt; at 0x7fc844430410&gt; of type &lt;type 'function'&gt;. For column literals, use 'lit', 'array', 'struct' or 'create_map' function.
  </code></pre>
  
  <p>Any idea how to fix it?</p>
  
  <p>Thanks a lot!</p>