title: pyspark Drop rows in dataframe to only have X distinct values in one column
body: <p>So I have a dataframe with a column &quot;Category&quot; and it has over 12k distinct values, for sampling purposes I would like to get a small sample where there are only 1000 different values of this category column.</p>
  <p>Before I was doing:</p>
  <pre><code>small_distinct = df.select(&quot;category&quot;).distinct().limit(1000).rdd.flatMap(lambda x: x).collect()
  df = df.where(col(&quot;category&quot;).isin(small_distinct))
  </code></pre>
  <p>I know this is extremely inefficient as I'm doing a distinct of the category column and then casting it into a normal python list so I can use <code>isin()</code> filter.</p>
  <p>Is there any &quot;spark&quot; way of doing this? I thought maybe something with rollingoverwindows could do the job? But I cant get to solve it</p>
  <p>Thanks!</p>