title: Spark Java: How to move data from HTTP source to Couchbase sink?
body: <p>I've a <code>.gz</code> file available on a Web server that I want to consume in a streaming manner and insert the data into Couchbase. The <code>.gz</code> file has only one file in it, which in turn contains one JSON object per line.</p>
  
  <p>Since Spark doesn't have a HTTP receiver, I wrote one myself (shown below). I'm using <a href="https://developer.couchbase.com/documentation/server/4.6/connectors/spark-2.1/spark-intro.html" rel="nofollow noreferrer">Couchbase Spark connector</a> to do the insertion. However, when running, the job is not actually inserting anything. I've a suspicion that it is due to my inexperience with Spark and not knowing how to start and await termination. As you can see below, there are 2 places such calls can be made.</p>
  
  <p><strong>Receiver</strong>:</p>
  
  <pre><code>public class HttpReceiver extends Receiver&lt;String&gt; {
      private final String url;
  
      public HttpReceiver(String url) {
          super(MEMORY_AND_DISK());
          this.url = url;
      }
  
      @Override
      public void onStart() {
          new Thread(() -&gt; receive()).start();
      }
  
      private void receive() {
          try {
              HttpURLConnection conn = (HttpURLConnection) new URL(url).openConnection();
              conn.setAllowUserInteraction(false);
              conn.setInstanceFollowRedirects(true);
              conn.setRequestMethod("GET");
              conn.setReadTimeout(60 * 1000);
  
              InputStream gzipStream = new GZIPInputStream(conn.getInputStream());
              Reader decoder = new InputStreamReader(gzipStream, UTF_8);
              BufferedReader reader = new BufferedReader(decoder);
  
              String json = null;
              while (!isStopped() &amp;&amp; (json = reader.readLine()) != null) {
                  store(json);
              }
              reader.close();
              conn.disconnect();
          } catch (IOException e) {
              stop(e.getMessage(), e);
          }
      }
  
      @Override
      public void onStop() {
  
      }
  }
  </code></pre>
  
  <p><strong>Dataload</strong>:</p>
  
  <pre><code>public void load(String url) throws StreamingQueryException, InterruptedException {
          JavaStreamingContext ssc = new JavaStreamingContext(sc, new Duration(1000));
          JavaReceiverInputDStream&lt;String&gt; lines = ssc.receiverStream(new HttpReceiver(url));
  
          lines.foreachRDD(rdd -&gt;
                  sql.read().json(rdd)
                          .select(new Column("id"),
                                  new Column("name"),
                                  new Column("rating"),
                                  new Column("review_count"),
                                  new Column("hours"),
                                  new Column("attributes"))
                          .writeStream()
                          .option("idField", "id")
                          .format("com.couchbase.spark.sql")
                          .start()
  //                        .awaitTermination(sparkProperties.getTerminationTimeoutMillis())
          );
  
  //        ssc.start();
          ssc.awaitTerminationOrTimeout(sparkProperties.getTerminationTimeoutMillis());
  }
  </code></pre>
  
  <p>The commented lines show my confusion with starting and terminating the jobs. Also, feel free to comment regarding the receiver if there's something wrong with it or can be improved.</p>
  
  <p>Using Spark v2.1.0 with Java.</p>
  
  <p><strong>Edit 1</strong>:</p>
  
  <p>Also tried this implementation:</p>
  
  <pre><code>lines.foreachRDD(rdd -&gt;
            couchbaseWriter(sql.read().json(rdd)
                    .select(new Column("id"),
                            new Column("name"),
                            new Column("rating"),
                            new Column("review_count"),
                            new Column("hours"),
                            new Column("attributes"))
                    .write()
                    .option("idField", "id")
                    .format("com.couchbase.spark.sql"))
                    .couchbase()
    );
  
    ssc.start();
    ssc.awaitTermination();
  </code></pre>
  
  <p>But it throws <code>IllegalStateException: SparkContext has been shutdown</code></p>
  
  <pre><code>11004 [JobScheduler] ERROR org.apache.spark.streaming.scheduler.JobScheduler  - Error running job streaming job 1488664987000 ms.0
  java.lang.IllegalStateException: SparkContext has been shutdown
      at org.apache.spark.SparkContext.runJob(SparkContext.scala:1910)
      at org.apache.spark.SparkContext.runJob(SparkContext.scala:1981)
      at org.apache.spark.rdd.RDD$$anonfun$fold$1.apply(RDD.scala:1088)
      at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
      at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
      at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
      at org.apache.spark.rdd.RDD.fold(RDD.scala:1082)
      at org.apache.spark.sql.execution.datasources.json.InferSchema$.infer(InferSchema.scala:69)
  </code></pre>
  
  <p><strong>Edit 2</strong>:
  Turns out the error from edit 1 was caused by a <code>@PostDestruct</code> method where I was closing the context. I'm using Spring and the bean is supposed to be singleton, but somehow Spark is causing it to destroy before the job finishes. I've now removed the  <code>@PostDestruct</code> and made some changes; the following seems to be working but with open questions:</p>
  
  <pre><code>public void load(String dataDirURL, String format) throws StreamingQueryException, InterruptedException {
      JavaStreamingContext ssc = new JavaStreamingContext(sc, new Duration(1000));
      JavaReceiverInputDStream&lt;String&gt; lines = ssc.receiverStream(new HttpReceiver(dataDirURL));
  
      lines.foreachRDD(rdd -&gt; {
          try {
              Dataset&lt;Row&gt; select = sql.read().json(rdd)
                      .select("id", "name", "rating", "review_count", "hours", "attributes");
              couchbaseWriter(select.write()
                      .option("idField", "id")
                      .format(format))
                      .couchbase();
          } catch (Exception e) {
              // Time to time throws AnalysisException: cannot resolve '`id`' given input columns: [];
          }
      });
  
      ssc.start();
      ssc.awaitTerminationOrTimeout(sparkProperties.getTerminationTimeoutMillis());
  }
  </code></pre>
  
  <p>Open Questions:</p>
  
  <ol>
  <li>Time to time throws 
  <code>AnalysisException: cannot resolve '</code>id<code>' given input columns: [];</code>. Is this a problem with my receiver?</li>
  <li><p>When the document already exists, the task fails with the following exception. In my case, I'd simply like to overwrite the doc if present, not blow up.</p>
  
  <pre><code>Lost task 1.0 in stage 2.0 (TID 4, localhost, executor driver): com.couchbase.client.java.error.DocumentAlreadyExistsException
  at com.couchbase.client.java.CouchbaseAsyncBucket$13.call(CouchbaseAsyncBucket.java:475)
  </code></pre></li>
  </ol>