title: Spark job just hangs with large data
body: <p>I am trying to query from s3 (15 days of data). I tried querying them separately (each day) it works fine. It works fine for 14 days as well. But when I query 15 days the job keeps running forever (hangs) and the task # is not updating. </p>
  
  <p>My settings :</p>
  
  <p>I am using 51 node cluster r3.4x large with dynamic allocation and maximum resource turned on. </p>
  
  <p>All I am doing is =</p>
  
  <pre><code>val startTime="2017-11-21T08:00:00Z"
  val endTime="2017-12-05T08:00:00Z"
  
  val start = DateUtils.getLocalTimeStamp( startTime )
  val end = DateUtils.getLocalTimeStamp( endTime )
  
  val days: Int = Days.daysBetween( start, end ).getDays
  
  val files: Seq[String] = (0 to days)
        .map( start.plusDays )
        .map( d =&gt; s"$input_path${DateTimeFormat.forPattern( "yyyy/MM/dd" ).print( d )}/*/*" )
  
  sqlSession.sparkContext.textFile( files.mkString( "," ) ).count
  </code></pre>
  
  <p>When I run the same with 14 days, I got 197337380 (count) and I ran the 15th day separately and got 27676788. But when I query 15 days total the job hangs</p>
  
  <p>Update :</p>
  
  <p>The job works fine with :</p>
  
  <pre><code>  var df = sqlSession.createDataFrame(sc.emptyRDD[Row], schema)
  
      for(n &lt;- files ){
        val tempDF = sqlSession.read.schema( schema ).json(n)
        df = df(tempDF)
      }
  
  df.count
  </code></pre>
  
  <p>But can some one explain why it works now but not before ?</p>
  
  <p>UPDATE : After setting mapreduce.input.fileinputformat.split.minsize to 256 GB it works fine now. </p>