title: pyspark: why is cache so slow
body: <p>when I run</p>
  
  <pre><code>import time
  start_time = time.time()
  print(df_join.count())
  end_time = time.time()
  print((end_time - start_time))
  </code></pre>
  
  <p>i get</p>
  
  <pre><code>25721
  19.099464416503906
  </code></pre>
  
  <p>when i run</p>
  
  <pre><code>start_time = time.time()
  df_join.cache()
  print(df_join.count())
  end_time = time.time()
  print((end_time - start_time))
  </code></pre>
  
  <p>it still running after 5 minutes. does it really take that long to cache 27 rows of data? about 15-20 columns wide, and the complexity is a url string. </p>
  
  <p><strong>EDIT 1:</strong>
  it turns out i have a column which its type is array of jsons. if I take that out everything works fine. unfortuantly, pyspark reads that as a string and i do not know how to tell it to be array of jsons</p>
  
  <p>How can I improve it?</p>