title: PySpark replace values below count threshold with values
body: <p>I'm trying to replace all values with occurence lower than set threshold with another value - Spark 2.1.0 , PySpark API and using DataFrame. </p>
  
  <p>When I test the function on an example (<code>df1</code> below), it works. But not on my real data. Both have the <code>dtype</code> attribute - string. With <code>df1</code>, I run the columns 'cat' and 'integers' in a for loop, which is what I want with the real data DF as well and again it works perfectly.</p>
  
  <pre><code>df1 = spark.createDataFrame([
      (0, "a","1"),
      (1, "b","1"),
      (2, "c","1"),
      (3, "a","1"),
      (4, "a","2"),
      (5, "c","2"),
      (6,"b","1"),
      (7,"b","1"),
  ], ["id", "cat","integer"])
  
  def cutoff(df,feat,threshold, otherclass='other'):
      if isinstance(threshold, float) and threshold&lt;1:
          threshold = str(int(threshold*df.count()))
  
      if isinstance(threshold,int):
          threshold = str(threshold)
  
      temp = df.groupBy(feat).count().orderBy('count')
      replace = temp.filter("count&lt;"+threshold).select(feat).rdd.map(lambda r:r[0]).collect()
      print "replacing ", replace,replace.__class__, " with ", otherclass, " subset ", feat
      df = df.replace(replace,otherclass,feat)
  
      return df
  </code></pre>
  
  <p>But when using real data (SQL import from Hive), I obtain </p>
  
  <pre><code>mydata (just a part):
  +---------------+-----+
  |   browser_name|count|
  +---------------+-----+
  |         Chrome| 2197|
  |             IE|  719|
  |        Firefox|  542|
  |  Mobile Safari|  370|
  |Android Browser|  361|
  |           Edge|  265|
  | Chrome WebView|  203|
  
  replacing  [u'Iron', u'UCBrowser', u'Puffin', u'Opera Mini', u'Yandex', u'Maxthon', u'Silk', u'Vivaldi', None, u'MIUI Browser', u'Chromium', u'WebKit', u'IEMobile', u'Facebook', u'Chrome WebView', u'Safari', u'Opera', u'Android Browser', u'Mobile Safari', u'Edge', u'IE', u'Firefox'] &lt;type 'list'&gt;  with  other  subset  browser_name
  
  
  Traceback (most recent call last):
    File "/tmp/zeppelin_pyspark-1704248642413819893.py", line 267, in &lt;module&gt;
      raise Exception(traceback.format_exc())
  Exception: Traceback (most recent call last):
    File "/tmp/zeppelin_pyspark-1704248642413819893.py", line 265, in &lt;module&gt;
      exec(code)
    File "&lt;stdin&gt;", line 12, in &lt;module&gt;
    File "&lt;stdin&gt;", line 10, in cutoff
    File "/usr/lib/spark/python/pyspark/sql/dataframe.py", line 1345, in replace
      self._jdf.na().replace(self._jseq(subset), self._jmap(rep_dict)), self.sql_ctx)
    File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
      answer, self.gateway_client, self.target_id, self.name)
    File "/usr/lib/spark/python/pyspark/sql/utils.py", line 63, in deco
      return f(*a, **kw)
    File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
      format(target_id, ".", name), value)
  Py4JJavaError: An error occurred while calling o4471.replace.
  : scala.MatchError: null
      at org.apache.spark.sql.DataFrameNaFunctions.replace0(DataFrameNaFunctions.scala:351)
      at org.apache.spark.sql.DataFrameNaFunctions.replace(DataFrameNaFunctions.scala:336)
      at sun.reflect.GeneratedMethodAccessor359.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
      at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
      at py4j.Gateway.invoke(Gateway.java:280)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.GatewayConnection.run(GatewayConnection.java:214)
      at java.lang.Thread.run(Thread.java:745)
  </code></pre>
  
  <p>So it complains about the <code>replace</code> function. But when I only run 
  <code>df.replace(['Chrome','Firefox'],'hovno','browser_name').show(10)</code> it works like a charm again (I even tried to input the list in unicode as the function generates it and its ok). So I'm wondering what did I do my <code>DF</code> in the function that it doesn't work? I understand <code>MatchError</code> as not being able to find the source values to replace, but they are surely there. </p>
  
  <p>Thanks a million!</p>