title: PySpark vs sklearn TFIDF
body: <p>I'm new to PySpark. I was playing around with the tfidf. Just wanted to check if they are giving same results. But they are not same. Here's what I did.</p>
  
  <pre><code># create the PySpark dataframe
  sentenceData = sqlContext.createDataFrame((
    (0.0, "Hi I heard about Spark"),
    (0.0, "I wish Java could use case classes"),
    (1.0, "Logistic regression models are neat")
  )).toDF("label", "sentence")
  
  # tokenize
  tokenizer = Tokenizer().setInputCol("sentence").setOutputCol("words")
  wordsData = tokenizer.transform(sentenceData)
  
  # vectorize
  vectorizer = CountVectorizer(inputCol='words', outputCol='vectorizer').fit(wordsData)
  wordsData = vectorizer.transform(wordsData)
  
  # calculate scores
  idf = IDF(inputCol="vectorizer", outputCol="tfidf_features")
  idf_model = idf.fit(wordsData)
  wordsData = idf_model.transform(wordsData)
  
  # dense the current response variable
  def to_dense(in_vec):
      return DenseVector(in_vec.toArray())
  to_dense_udf = udf(lambda x: to_dense(x), VectorUDT())
  
  # create dense vector
  wordsData = wordsData.withColumn("tfidf_features_dense", to_dense_udf('tfidf_features'))
  </code></pre>
  
  <p>I converted the PySpark df to pandas</p>
  
  <pre><code>wordsData_pandas = wordsData.toPandas()
  </code></pre>
  
  <p>and, then just calculating using sklearn's tfidf as following</p>
  
  <pre><code>def dummy_fun(doc):
      return doc
  
  # create sklearn tfidf
  tfidf = TfidfVectorizer(
      analyzer='word',
      tokenizer=dummy_fun,
      preprocessor=dummy_fun,
      token_pattern=None)  
  
  # transform and get idf scores
  feature_matrix = tfidf.fit_transform(wordsData_pandas.words)
  
  # create sklearn dtm matrix
  sklearn_tfifdf = pd.DataFrame(feature_matrix.toarray(), columns=tfidf.get_feature_names())
  
  # create PySpark dtm matrix
  spark_tfidf = pd.DataFrame([np.array(i) for i in wordsData_pandas.tfidf_features_dense], columns=vectorizer.vocabulary)
  </code></pre>
  
  <p>But unfortunately, I'm getting this for PySpark</p>
  
  <p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
  <div class="snippet-code">
  <pre class="snippet-code-html lang-html prettyprint-override"><code>&lt;table border="1" class="dataframe"&gt;  &lt;thead&gt;    &lt;tr style="text-align: right;"&gt;      &lt;th&gt;&lt;/th&gt;      &lt;th&gt;i&lt;/th&gt;      &lt;th&gt;are&lt;/th&gt;      &lt;th&gt;logistic&lt;/th&gt;      &lt;th&gt;case&lt;/th&gt;      &lt;th&gt;spark&lt;/th&gt;      &lt;th&gt;hi&lt;/th&gt;      &lt;th&gt;about&lt;/th&gt;      &lt;th&gt;neat&lt;/th&gt;      &lt;th&gt;could&lt;/th&gt;      &lt;th&gt;regression&lt;/th&gt;      &lt;th&gt;wish&lt;/th&gt;      &lt;th&gt;use&lt;/th&gt;      &lt;th&gt;heard&lt;/th&gt;      &lt;th&gt;classes&lt;/th&gt;      &lt;th&gt;java&lt;/th&gt;      &lt;th&gt;models&lt;/th&gt;    &lt;/tr&gt;  &lt;/thead&gt;  &lt;tbody&gt;    &lt;tr&gt;      &lt;th&gt;0&lt;/th&gt;      &lt;td&gt;0.287682&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.693147&lt;/td&gt;      &lt;td&gt;0.693147&lt;/td&gt;      &lt;td&gt;0.693147&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.693147&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;    &lt;/tr&gt;    &lt;tr&gt;      &lt;th&gt;1&lt;/th&gt;      &lt;td&gt;0.287682&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.693147&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.693147&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.693147&lt;/td&gt;      &lt;td&gt;0.693147&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.693147&lt;/td&gt;      &lt;td&gt;0.693147&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;    &lt;/tr&gt;    &lt;tr&gt;      &lt;th&gt;2&lt;/th&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.693147&lt;/td&gt;      &lt;td&gt;0.693147&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.693147&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.693147&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.693147&lt;/td&gt;    &lt;/tr&gt;  &lt;/tbody&gt;&lt;/table&gt;</code></pre>
  </div>
  </div>
  </p>
  
  <p>and this for sklearn,</p>
  
  <p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
  <div class="snippet-code">
  <pre class="snippet-code-html lang-html prettyprint-override"><code>&lt;table border="1" class="dataframe"&gt;  &lt;thead&gt;    &lt;tr style="text-align: right;"&gt;      &lt;th&gt;&lt;/th&gt;      &lt;th&gt;i&lt;/th&gt;      &lt;th&gt;are&lt;/th&gt;      &lt;th&gt;logistic&lt;/th&gt;      &lt;th&gt;case&lt;/th&gt;      &lt;th&gt;spark&lt;/th&gt;      &lt;th&gt;hi&lt;/th&gt;      &lt;th&gt;about&lt;/th&gt;      &lt;th&gt;neat&lt;/th&gt;      &lt;th&gt;could&lt;/th&gt;      &lt;th&gt;regression&lt;/th&gt;      &lt;th&gt;wish&lt;/th&gt;      &lt;th&gt;use&lt;/th&gt;      &lt;th&gt;heard&lt;/th&gt;      &lt;th&gt;classes&lt;/th&gt;      &lt;th&gt;java&lt;/th&gt;      &lt;th&gt;models&lt;/th&gt;    &lt;/tr&gt;  &lt;/thead&gt;  &lt;tbody&gt;    &lt;tr&gt;      &lt;th&gt;0&lt;/th&gt;      &lt;td&gt;0.355432&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.467351&lt;/td&gt;      &lt;td&gt;0.467351&lt;/td&gt;      &lt;td&gt;0.467351&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.467351&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;    &lt;/tr&gt;    &lt;tr&gt;      &lt;th&gt;1&lt;/th&gt;      &lt;td&gt;0.296520&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.389888&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.389888&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.389888&lt;/td&gt;      &lt;td&gt;0.389888&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.389888&lt;/td&gt;      &lt;td&gt;0.389888&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;    &lt;/tr&gt;    &lt;tr&gt;      &lt;th&gt;2&lt;/th&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.447214&lt;/td&gt;      &lt;td&gt;0.447214&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.447214&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.447214&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.000000&lt;/td&gt;      &lt;td&gt;0.447214&lt;/td&gt;    &lt;/tr&gt;  &lt;/tbody&gt;&lt;/table&gt;</code></pre>
  </div>
  </div>
  </p>
  
  <p>I did try out the <code>use_idf</code>, <code>smooth_idf</code> paramters. But none seem to make both same. What am I missing? Any help is appreciated. Thanks in advance.</p>