title: Spark Scala Get Data Back from rdd.foreachPartition
body: <p>I have some code like this:</p>
  
  <pre><code>      println("\nBEGIN Last Revs Class: "+ distinctFileGidsRDD.getClass)
        val lastRevs = distinctFileGidsRDD.
          foreachPartition(iter =&gt; {
            SetupJDBC(jdbcDriver, jdbcUrl, jdbcUser, jdbcPassword)
            while(iter.hasNext) {
              val item = iter.next()
              //println(item(0))
              println("String: "+item(0).toString())
              val jsonStr = DB.readOnly { implicit session =&gt;
                sql"SELECT jsonStr FROM lasttail WHERE fileGId = ${item(0)}::varchar".
                  map { resultSet =&gt; resultSet.string(1) }.single.apply()
              }
              println("\nJSON: "+jsonStr)
            }
          })
        println("\nEND Last Revs Class: "+ lastRevs.getClass)
  </code></pre>
  
  <p>The code outputs (with heavy edits) something like:</p>
  
  <pre><code>BEGIN Last Revs Class: class org.apache.spark.rdd.MapPartitionsRDD
  String: 1fqhSXPE3GwrJ6SZzC65gJnBaB5_b7j3pWNSfqzU5FoM
  JSON: Some({"Struct":{"fileGid":"1fqhSXPE3GwrJ6SZzC65gJnBaB5_b7j3pWNSfqzU5FoM",... )
  String: 1eY2wxoVq17KGMUBzCZZ34J9gSNzF038grf5RP38DUxw
  JSON: Some({"Struct":{"fileGid":"1fqhSXPE3GwrJ6SZzC65gJnBaB5_b7j3pWNSfqzU5FoM",... )
  ...
  JSON: None()
  END Last Revs Class: void
  </code></pre>
  
  <p>QUESTION 1:
  How can I get the lastRevs value to be in a useful format like the JSON string/null or an option like Some / None?</p>
  
  <p>QUESTION 2: 
  My preference: IS there another way get at partitions data that an RDD-like format (rather than the iterator format)?</p>
  
  <pre><code>dstream.foreachRDD { (rdd, time) =&gt;
    rdd.foreachPartition { partitionIterator =&gt;
      val partitionId = TaskContext.get.partitionId()
      val uniqueId = generateUniqueId(time.milliseconds, partitionId)
      // use this uniqueId to transactionally commit the data in partitionIterator
    } 
  }
  </code></pre>
  
  <p>from <a href="http://spark.apache.org/docs/latest/streaming-programming-guide.html#performance-tuning" rel="noreferrer">http://spark.apache.org/docs/latest/streaming-programming-guide.html#performance-tuning</a></p>
  
  <p>QUESTION 3: Is the method of getting data that I am using a sane method (given I am following the link above)? (Put aside the fact that this is a scalikejdbc system JDBC right now. This is going to be a key, value store of some type other than this prototype.)</p>