title: Spark Streaming: NullPointerException inside foreachPartition
body: <p>I have a spark streaming job which reads from Kafka and does some comparisons with an existing table in Postgres before writing to Postrges again. This is what it looks like :</p>
  
  <pre><code>val message = KafkaUtils.createStream(...).map(_._2)
  
  message.foreachRDD( rdd =&gt; {
  
    if (!rdd.isEmpty){
      val kafkaDF = sqlContext.read.json(rdd)
      println("First")
  
      kafkaDF.foreachPartition(
        i =&gt;{
          val jdbcDF = sqlContext.read.format("jdbc").options(
            Map("url" -&gt; "jdbc:postgresql://...",
              "dbtable" -&gt; "table", "user" -&gt; "user", "password" -&gt; "pwd" )).load()
  
          createConnection()
          i.foreach(
            row =&gt;{
              println("Second")
              connection.sendToTable()
            }
          )
          closeConnection()
        }
      )
  </code></pre>
  
  <p>This code is giving me NullPointerException at the line <code>val jbdcDF = ...</code></p>
  
  <p>What am I doing wrong? Also, my log <code>"First"</code> works, but <code>"Second"</code> doesn't show up anywhere in the logs. I tried the entire code with <code>kafkaDF.collect().foreach(...)</code> and it works perfectly, but has very poor performance. I am looking to replace it with <code>foreachPartition</code>. </p>
  
  <p>Thanks</p>