title: Spark collect()/count() never finishes while show() runs fast
body: <p>I'm running Spark locally on my Mac and there is a weird issue. Basically, I can output any number of rows using show() method of the DataFrame, however, when I try to use count() or collect() even on pretty small amounts of data, the Spark is getting stuck on that stage. And never finishes its job. I'm using gradle for building and running.</p>
  
  <p>When I run</p>
  
  <pre><code>./gradlew clean run
  </code></pre>
  
  <p>The program gets stuck at</p>
  
  <pre><code>&gt; Building 83% &gt; :run
  </code></pre>
  
  <p>What could cause this problem?
  Here is the code.</p>
  
  <pre class="lang-js prettyprint-override"><code>    val moviesRatingsDF = MongoSpark.load(sc).toDF().select("movieId", "userId","rating")
  
      val movieRatingsDF = moviesRatingsDF
        .groupBy("movieId")
        .pivot("userId")
        .max("rating")
        .na.fill(0)
  
      val ratingColumns = movieRatingsDF.columns.drop(1) // drop the name column
  
      val movieRatingsDS:Dataset[MovieRatingsVector] = movieRatingsDF
        .select( col("movieId").as("movie_id"), array(ratingColumns.map(x =&gt; col(x)): _*).as("ratings") )
        .as[MovieRatingsVector]
  
      val moviePairs = movieRatingsDS.withColumnRenamed("ratings", "ratings1")
        .withColumnRenamed("movie_id", "movie_id1")
        .crossJoin(movieRatingsDS.withColumnRenamed("ratings", "ratings2").withColumnRenamed("movie_id", "movie_id2"))
        .filter(col("movie_id1") &lt; col("movie_id2"))
  
      val movieSimilarities = moviePairs.map(row =&gt; {
        val ratings1 = sc.parallelize(row.getAs[Seq[Double]]("ratings1"))
        val ratings2 = sc.parallelize(row.getAs[Seq[Double]]("ratings2"))
        val corr:Double = Statistics.corr(ratings1, ratings2)
  
        MovieSimilarity(row.getAs[Long]("movie_id1"), row.getAs[Long]("movie_id2"), corr)
      }).cache()
  
  
      val collectedData = movieSimilarities.collect()
      println(collectedData.length)
  
      log.warn("I'm done") //never gets here
  
      close
  </code></pre>
  
  
  
  <p><a href="https://i.stack.imgur.com/HV95y.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/HV95y.png" alt="enter image description here"></a></p>