title: pyspark map type contains duplicate keys
body: <p>Could someone help me understand why the map type in pyspark could contain duplicate keys?</p>
  <p>An example would be:</p>
  <pre><code># generate a sample dataframe
  # the `field` column is an array of struct with value a and value b
  # the goal is to create a map from a -&gt; b 
  
  df = spark.createDataFrame([{
      'field': [Row(a=1, b=2), Row(a=1, b=3)],
  }])
  
  
  # above code would generate a dataframe like this
  +----------------+
  |           field|
  +----------------+
  |[[1, 2], [1, 3]]|
  +----------------+
  
  # with schema
  root
   |-- field: array (nullable = true)
   |    |-- element: struct (containsNull = true)
   |    |    |-- a: long (nullable = true)
   |    |    |-- b: long (nullable = true)
  
  </code></pre>
  <p>Then I applied <code>map_from_entries</code> on this dataframe, trying to collect unique <code>a-&gt;b</code> pairs. I was expecting the map to contain unique keys, that is <code>{1 -&gt; 3}</code> in this case. However, I'm getting {1 -&gt; 2, 1 -&gt; 3} before collecting. This contradict the common idea of a <code>map</code> type.</p>
  <pre><code>import pyspark.sql.functions as F
  df.select(F.map_from_entries(&quot;field&quot;))
  
  # the result is
  +-----------------------+
  |map_from_entries(field)|
  +-----------------------+
  |       [1 -&gt; 2, 1 -&gt; 3]|
  +-----------------------+
  </code></pre>
  <p>I also tried to apply <code>F.map_keys()</code> on this field and got <code>[1, 1]</code> as the result. However, when I collect this dataframe, I was able to get the result without duplicate keys:</p>
  <pre><code>df.select(F.map_from_entries(&quot;field&quot;)).collect()
  
  # result
  [Row(map_from_entries(field)={1: 3})]
  </code></pre>
  <p>This is causing some unexpected behavior in my spark job, and I would really appreciate if someone could help me understand why pyspark is behaving in this way. Is this a bug or by design?</p>