title: Pyspark: Filter DF based on Array(String) length, or CountVectorizer count
body: <p>I have URL data aggregated into a string array. Of this form. [xyz.com,abc.com,efg.com] </p>
  
  <p>I eventually use a count vectorizer in pyspark to get it into a vector like (262144,[3,20,83721],[1.0,1.0,1.0]).</p>
  
  <p>Where the vector is saying out of 262144; there are 3 Urls present indexed at 3,20, and 83721 for a certain row. All this data is binary hence the array of 1's. </p>
  
  <p>I would like to filter and only used rows that contain a certain amount of entries. So if I say wanted to only use rows with 4 examples; I want the above row to be dropped. </p>
  
  <p>I am fine doing this filtering on the String Array of the Vector form returned by Count Vectorizer. </p>
  
  <p>In my data I have rows in tens of millions and am just not sure how to do it efficiently.</p>
  
  <p>Here is code from the docs  edited to produce an example</p>
  
  <pre><code>from pyspark.ml.feature import CountVectorizer
  
  # Input data: Each row is a bag of words with a ID.
  df = spark.createDataFrame([
      (0, "a".split(" ")),
      (1, "a b c".split(" "))
  ], ["id", "words"])
  
  # fit a CountVectorizerModel from the corpus.
  cv = CountVectorizer(inputCol="words", outputCol="features",         vocabSize=3)
  
  model = cv.fit(df)
  
  result = model.transform(df)
  </code></pre>
  
  <p>So here say we wanted only results that are of 2 length or higher. Again I do not mind doing this on the vector produced by CountVectorizer or the String array before that as long as it is efficient with the size of my data.</p>