title: Whats the Efficient way to call http request and read inputstream in spark MapTask
body: <p>Please see the below code sample</p>
  
  <pre><code>JavaRDD&lt;String&gt; mapRDD = filteredRecords
              .map(new Function&lt;String, String&gt;() {
  
                  @Override
                  public String call(String url) throws Exception {
                      BufferedReader in = null;
                      URL formatURL = new URL((url.replaceAll("\"", ""))
                              .trim());
                      try {
                          HttpURLConnection con = (HttpURLConnection) formatURL
                                  .openConnection();
                          in = new BufferedReader(new InputStreamReader(con
                                  .getInputStream()));
  
                          return in.readLine();
                      } finally {
                          if (in != null) {
                              in.close();
                          }
                      }
                  }
              }); 
  </code></pre>
  
  <p>here url is http GET request. example </p>
  
  <pre><code>http://ip:port/cyb/test?event=movie&amp;id=604568837&amp;name=SID&amp;timestamp_secs=1460494800&amp;timestamp_millis=1461729600000&amp;back_up_id=676700166
  </code></pre>
  
  <p>This piece of code is very slow . IP and port are random and load is distributed so ip can have 20 different value with port so I dont see bottleneck .</p>
  
  <p>When I comment </p>
  
  <pre><code> in = new BufferedReader(new InputStreamReader(con
                              .getInputStream()));
  
                      return in.readLine();
  </code></pre>
  
  <p>The code is too fast. 
  NOTE: Input data to process is 10GB. Using spark to read from S3.</p>
  
  <p>is there anything wrong I am doing with BufferedReader or InputStreamReader any alternative . 
  I cant use foreach in spark as I have to get the response back from server and need to save JAVARdd as textFile on HDFS. </p>
  
  <p>if we use mappartition code something as below</p>
  
  <pre><code>JavaRDD&lt;String&gt; mapRDD = filteredRecords.mapPartitions(new FlatMapFunction&lt;Iterator&lt;String&gt;, String&gt;() {
  
          @Override
          public Iterable&lt;String&gt; call(Iterator&lt;String&gt; tuple) throws Exception {
  
              final List&lt;String&gt; rddList = new ArrayList&lt;String&gt;();
              Iterable&lt;String&gt; iterable = new Iterable&lt;String&gt;() {
  
                  @Override
                  public Iterator&lt;String&gt; iterator() {
                      return rddList.iterator();
                  }
              };
              while(tuple.hasNext()) {
                  URL formatURL = new URL((tuple.next().replaceAll("\"", ""))
                          .trim());
                  HttpURLConnection con = (HttpURLConnection) formatURL
                          .openConnection();
                  try(BufferedReader br = new BufferedReader(new InputStreamReader(con
                          .getInputStream()))) {
  
                      rddList.add(br.readLine());
  
                  } catch (IOException ex) {
                      return rddList;
                  }
              }
              return iterable;
          }
      }); 
  </code></pre>
  
  <p>here also for each record we are doing same .. isnt it ?</p>