title: Check if pyspark dataframe is empty causing memory issues
body: <p>I have a table that has about 1 billion records. I run a query on it to essentially find duplicates. If the result of the query is 0 rows, there are no duplicates, otherwise there are. If there are duplicates, I want to write that table name to a text file.
  So what I am doing is </p>
  
  <pre><code>df = spark.sql("SELECT count(*) FROM table GROUP BY key1,key2,key3 HAVING count(*) &gt; 1 LIMIT 1)
  if df.count() &gt; 0:
      with open('duplicate_tables.txt','a') as file:
          file.write('\n' + table)
  </code></pre>
  
  <p>On the <code>df.count()</code> line, I get an error like <code>java.io.IOException: No space left on device</code>. Is this because <code>count()</code> is inefficient. I also get the same error when I try using </p>
  
  <pre><code>if len(df.head(1)) != 0:
  </code></pre>
  
  <p>In my query, I thought (hoped) that adding in the LIMIT 1 would help so it wouldn't have to go through hundreds and hundreds of rows, just check if it's empty or not. If I take out the count part, it works fine. </p>
  
  <p>I have seen a few ways to rewrite the count statement (I have gone through <a href="https://stackoverflow.com/questions/32707620/how-to-check-if-spark-dataframe-is-empty">How to check if spark dataframe is empty?</a>), but so far I haven't had any luck. </p>