title: spark reading large file
body: <p>This may be a silly question. I want to make sure I understand this correctly.</p>
  
  <p>When you in a huge file <code>(400GB)</code> into a cluster, where the collective executor memory is only around <code>120GB</code>, Spark seems to read forever. It doesn't crash, nor does it start the first map job. </p>
  
  <p>What I think is happening is, Spark is reading thru the large file as streams, and start discarding the older lines when the executors run out of memory. This obvious can be a problem when execution of <code>.map</code> codes starts, as the executor jvm would be reading back the file from the beginning again. I am wondering though, whether Spark is somehow spilling the data onto the hard drive, similar to the shuffle spilling mechanism. </p>
  
  <p>Notice, I am not referring to the cache process. This has to do with the initial read using 
  <code>sc.textFile(filename)</code></p>