title: Latent Dirichlet allocation (LDA) in Spark - replicate model
body: <p>I want to save the LDA model from pyspark ml-clustering package and apply the model to the training &amp; test data-set after saving. However results diverge despite setting a seed. My code is the following:</p>
  
  <p>1) Import packages    </p>
  
  <pre><code>from pyspark.ml.clustering import LocalLDAModel, DistributedLDAModel
  from pyspark.ml.feature import CountVectorizer , IDF
  </code></pre>
  
  <p>2) Preparing the dataset</p>
  
  <pre><code>countVectors = CountVectorizer(inputCol="requester_instruction_words_filtered_complete", outputCol="raw_features", vocabSize=5000, minDF=10.0)
  cv_model = countVectors.fit(tokenized_stopwords_sample_df)
  result_tf = cv_model.transform(tokenized_stopwords_sample_df)
  vocabArray = cv_model.vocabulary
  idf = IDF(inputCol="raw_features", outputCol="features")
  idfModel = idf.fit(result_tf)
  result_tfidf = idfModel.transform(result_tf)
  result_tfidf = result_tfidf.withColumn("id", monotonically_increasing_id())    
  corpus = result_tfidf.select("id", "features")
  </code></pre>
  
  <p>3) Training the LDA model</p>
  
  <pre><code>lda = LDA(k=number_of_topics, maxIter=100, docConcentration = [alpha], topicConcentration = beta, seed = 123)
  model = lda.fit(corpus)
  model.save("LDA_model_saved")
  topics = model.describeTopics(words_in_topic)  
  topics_rdd = topics.rdd
  modelled_corpus = model.transform(corpus)
  </code></pre>
  
  <p>4)      Replicate the model</p>
  
  <pre><code>#Prepare the data set
  countVectors = CountVectorizer(inputCol="requester_instruction_words_filtered_complete", outputCol="raw_features", vocabSize=5000, minDF=10.0)
  cv_model = countVectors.fit(tokenized_stopwords_sample_df)
  result_tf = cv_model.transform(tokenized_stopwords_sample_df)
  vocabArray = cv_model.vocabulary
  idf = IDF(inputCol="raw_features", outputCol="features")
  idfModel = idf.fit(result_tf)
  result_tfidf = idfModel.transform(result_tf)   
  result_tfidf = result_tfidf.withColumn("id", monotonically_increasing_id())
  corpus_new = result_tfidf.select("id", "features")
  
  #Load the model to apply to new corpus
  newModel = LocalLDAModel.load("LDA_model_saved")
  topics_new = newModel.describeTopics(words_in_topic)  
  topics_rdd_new = topics_new.rdd
  modelled_corpus_new = newModel.transform(corpus_new)
  </code></pre>
  
  <p>The following results are different despite my assumption to be equal:
  <code>topics_rdd != topics_rdd_new</code> and <code>modelled_corpus != modelled_corpus_new</code> (also when inspecting the extracted topics they are different as well as the predicted classes on the dataset)</p>
  
  <p>So I find it really strange that the same model predicts different classes ("topics") on the same dataset, even though I set a seed in the model generation. Can someone with experience in replicating LDA models help?</p>
  
  <p>Thank you :)</p>