title: Unbounded table is spark structured streaming
body: <p>I'm starting to learn Spark and am having a difficult time understanding the rationality behind Structured Streaming in Spark. Structured streaming treats all the data arriving as an unbounded input table, wherein every new item in the data stream is treated as new row in the table. I have the following piece of code to read in incoming files to the <code>csvFolder</code>. </p>
  
  <pre><code>val spark = SparkSession.builder.appName("SimpleApp").getOrCreate()
  
  val csvSchema = new StructType().add("street", "string").add("city", "string")
  .add("zip", "string").add("state", "string").add("beds", "string")
  .add("baths", "string").add("sq__ft", "string").add("type", "string")
  .add("sale_date", "string").add("price", "string").add("latitude", "string")
  .add("longitude", "string")
  
  val streamingDF = spark.readStream.schema(csvSchema).csv("./csvFolder/")
  
  val query = streamingDF.writeStream
    .format("console")
    .start()
  </code></pre>
  
  <p>What happens if I dump a 1GB file to the folder. As per the specs, the streaming job is triggered every few milliseconds. If Spark encounters such a huge file in the next instant, won't it run out of memory while trying to load the file. Or does it automatically batch it? If yes, is this batching parameter configurable?</p>