title: How to model tf-idf spark
body: <p>I'm trying to re-write a code wrote (that it's in Python), but now in spark.</p>
  <pre><code>#pandas
  tfidf = TfidfVectorizer() 
  df_final = np.array(tfidf.fit_transform(df['sentence']).todense())
  </code></pre>
  <p>I read on spark documentation, is it necessary to use Tokenizer, HashingTF and then IDF to model tf-idf in PySpark?</p>
  <pre><code>#pyspark
  
  from pyspark.ml.feature import HashingTF, IDF, Tokenizer
  tokenizer = Tokenizer(inputCol = &quot;sentence&quot;, outputCol = &quot;words&quot;)
  wordsData = tokenizer.transform(df)
  hashingTF = HashingTF(inputCol = &quot;words&quot;, outputCol=&quot;rawFeatures&quot;, numFeatures = 20)
  
  tf = hashingTF.transform(wordsData)
  idf = IDF(inputCol = &quot;rawFeatures&quot;, outputCol = &quot;features&quot;)
  tf_idf = idf.fit(tf)
  
  df_final = tf_idf.transform(tf)
  </code></pre>