title: How to Persist an array in spark
body: <p>I'm comparing two tables to find out difference between them (i.e Source and destination), for that I'm loading those tables to memory and the comparison happens as expected in the machine of configuration 8GB memory and 4 cores but when comparing large amount of data the system hangs and runs out of memory, so I used <code>persist()</code> of storagelevel <code>DISK_ONLY</code></p>
  
  <p>the machine is capable of holding 100,000 rows in memory to store that to <code>DISK</code> at a time and do the rest comparison operations, I'm trying like below:</p>
  
  <pre><code>var partition = math.ceil(c / 100000.toFloat).toInt
  println(partition + " partition")
  var a = 1
  var data = spark.sparkContext.parallelize(Seq(""))
  var offset = 0
  for (s &lt;- a to partition) {
  
    val query = "(select * from destination LIMIT 100000 OFFSET " + offset + ") as src"
    data = data.union(spark.read.jdbc(url, query, connectionProperties).rdd.map(_.mkString(","))).persist(StorageLevel.DISK_ONLY)
    offset += 100000
  }
  val dest = data.collect.toArray
  val s = spark.sparkContext.parallelize(dest, 1).persist(StorageLevel.DISK_ONLY)
  </code></pre>
  
  <p>yes off-course I can use partition but the problem is I need to supply <code>Lowerbounds</code>,<code>Upperbounds</code>,<code>NumPartitions</code> dynamically for fetching <code>100,000</code>   I tried like:</p>
  
  <pre><code>val destination = spark.read.options(options).jdbc(options("url"), options("dbtable"), "EMPLOYEE_ID", 1, 22, 21, new java.util.Properties()).rdd.map(_.mkString(","))
  </code></pre>
  
  <p>it takes too much of time and storing those files into partitions though comparing operation is Iterative in nature its reading all the partitions for each and every step.</p>
  
  <p>Coming to the problem</p>
  
  <pre><code>val dest = data.collect.toArray
  val s = spark.sparkContext.parallelize(dest, 1).persist(StorageLevel.DISK_ONLY)
  </code></pre>
  
  <p>the above lines convert all the partitioned  RDD's to Array and parallelizing it to single partition so I don't want to iterate through all the partitions again and again. But <code>val dest = data.collect.toArray</code> can't able to convert some huge amount of lines because of shortage in memory and seems it won't allow to <code>Persist()</code> an array in spark.</p>
  
  <p>Is there is any way I can store and parallelize to one partition in <code>DISK</code>
  Sorry for being a noob.</p>
  
  <p>Thanks you..!</p>