title: How do you setup multiple Spark Streaming jobs with different batch durations?
body: <p>We are in the beginning phases of transforming the current data architecture of a large enterprise and I am currently building a Spark Streaming ETL framework in which we would connect all of our sources to destinations (source/destinations could be Kafka topics, Flume, HDFS, etc.) through transformations. This would look something like: </p>
  
  <p><code>SparkStreamingEtlManager.addEtl(Source, Transformation*, Destination)
  SparkStreamingEtlManager.streamEtl()
  streamingContext.start()</code></p>
  
  <p>The assumptions is that, since we should only have one SparkContext, we would deploy all of the ETL pipelines in one application/jar. </p>
  
  <p>The problem with this is that the batchDuration is an attribute of the context itself and not of the ReceiverInputDStream (Why is this?). Do we need to therefore have multiple Spark Clusters, or, allow for multiple SparkContexts and deploy multiple applications? Is there any other way to control the batch duration per receiver?</p>
  
  <p>Please let me know if any of my assumptions are naive or need to be rephrased. Thanks!</p>