title: how to repartition rdd based on data size
body: <p>I am working on spark streaming project which takes the data from Kafka applied some rules and save data in Hive. My problem is ingestion rate of data is not fixed. 60seconds may be 1 million message come or may be 1. I want to add repartition on Dstream. Because there is only 3 partition of Dstream, unable to process millions record within minute. repartition gives problem at the time of less than 20 records. It creates multiple small files in Hive. </p>
  
  <pre><code>dataStream.map(_._2).repartition(20)
  </code></pre>
  
  <p>My question is how I can rdd repartition based on rdd size. So that it can handle a single message or 1 million message.</p>