title: Async urlfetch Http post on App engine using Future
body: <p>My goal is to rapidly make posts to a server from appengine(java).  I am attempting to do this using UrlFetchService.fetchAsync.  I have been basing my code after <a href="http://ikaisays.com/2010/06/" rel="nofollow noreferrer">this blog post</a>.  I have been able to make the request using the code below, however I get some strange behavior:</p>
  
  <pre><code>private void futureRequests() {
      URLFetchService fetcher = URLFetchServiceFactory.getURLFetchService();
      URL url = new URL("https://someserver.com");
  
      FetchOptions fetchOptions = FetchOptions.Builder.withDefaults();
      fetchOptions.doNotValidateCertificate();
      fetchOptions.setDeadline(60D);
  
      ArrayList&lt;Future&lt;HTTPResponse&gt;&gt; asyncResponses = new ArrayList&lt;Future&lt;HTTPResponse&gt;&gt;();
  
      for (int i = 0; i &lt; postDatas.size(); i++) {
  
          HTTPRequest request = new HTTPRequest(url, HTTPMethod.POST, fetchOptions);
          request.setPayload(postDatas.get(i).getBytes(UTF8));
          HTTPHeader header = new HTTPHeader("Content-Type", "application/x-www-form-urlencoded;charset=UTF-8");
          request.setHeader(header);
          header = new HTTPHeader("Content-Length", Integer.toString(postDatas.get(i).getBytes().length));
          request.setHeader(header);
          header = new HTTPHeader("Authorization", "auth=" + authToken);
          request.setHeader(header);
          Future&lt;HTTPResponse&gt; responseFuture = fetcher.fetchAsync(request);
          asyncResponses.add(responseFuture);
      }
  
      for (Future&lt;HTTPResponse&gt; future : asyncResponses) {
          HTTPResponse response;
          try {
              response = future.get();
              int responseCode = response.getResponseCode();
              resp.getWriter().println("response: " + responseCode);
              logger.warning("Response: " + responseCode);
  
          } catch (Exception e) {
          }
      }
  }
  </code></pre>
  
  <p>The strange behavior is that I get duplicate posts on the server, and according to my appstats page I use 10x-20x more urlFetches than what was added with the code above.  Below is my appstats screen:</p>
  
  <p><img src="https://i.stack.imgur.com/LvTLu.png" alt="Appstats screen shot"></p>
  
  <p>There are more urlFetch calls that could not fit on the screen.  It appears that the requests are still completing in a synchronous fashion(circled items), but there are many urlFetches that appear to go on at the same time.  My question is how am I getting all this calls to urlFetch when I only had 14 Future ??  Could the server be giving an error or 503 and urlFetch retrying until it goes through?  And how can I be getting 2 posts for each request??</p>
  
  <p>I understand that I could use the task queue to do asyc request, however I am dealing with a relatively low number of request(20-100) and the cold start time of ramping up another instance would probably make this not a good option for my situation.  Can anyone explain this behavior or have experience with this?</p>