title: Java: Optimizing an application using asynchronous programming
body: <p>I have to modify a dropwizard application to improve its running time. Basically, this application receives around 3 million URLs daily and downloads and parses them to detect for malicious content. The problem is that the application is only able to process 1 million URLs. When I looked at the application I found that it is making a lot of sequential calls. I would like some suggestion on how can I improve the application by making it asynchronous or other techniques.</p>
  
  <p>Required code is as follows:-</p>
  
  <pre><code>/* Scheduler */
  private long triggerDetection(String startDate, String endDate) {
  for (UrlRequest request : urlRequests) {
                  if (!validateRequests.isWhitelisted(request)) {
                      ContentDetectionClient.detectContent(request);
                  }
              }
  }
  
  /* Client */
  public void detectContent(UrlRequest urlRequest){
          Client client = new Client();
          URI uri = buildUrl(); /* It returns the URL of this dropwizard application's resource method provided below */
  
          ClientResponse response = client.resource(uri)
                  .type(MediaType.APPLICATION_JSON_TYPE)
                  .post(ClientResponse.class, urlRequest);
  
          Integer status = response.getStatus();
          if (status &gt;= 200 &amp;&amp; status &lt; 300) {
              log.info("Completed request for url: {}", urlRequest.getUrl());
  
          }else{
              log.error("request failed for url: {}", urlRequest.getUrl());
          }
      }
  
      private URI buildUrl() {
          return UriBuilder
                  .fromPath(uriConfiguration.getUrl())
                  .build();
      }
  
  /* Resource Method */
   @POST
      @Path("/pageDetection")
      @Consumes(MediaType.APPLICATION_JSON)
      @Produces(MediaType.APPLICATION_JSON)
      /**
       * Receives the url of the publisher, crawls the content of that url, applies a detector to check if the content is malicious.
       * @returns returns the probability of the page being malicious
       * @throws throws exception if the crawl call failed
       **/
      public DetectionScore detectContent(UrlRequest urlRequest) throws Exception {
  
          return contentAnalysisOrchestrator.detectContentPage(urlRequest);
      }
  
  /* Orchestrator */
  public DetectionScore detectContentPage(UrlRequest urlRequest) {
          try {
  
              Pair&lt;Integer, HtmlPage&gt; response =  crawler.rawLoad(urlRequest.getUrl());
              String content =   response.getValue().text();
  
              DetectionScore detectionScore = detector.getProbability(urlRequest.getUrl(), content);
              contentDetectionResultDao.insert(urlRequest.getAffiliateId(), urlRequest.getUrl(),detectionScore.getProbability()*1000,
                      detectionScore.getRecommendation(), urlRequest.getRequestsPerUrl(), -1, urlRequest.getCreatedAt() );
  
              return detectionScore;
  
          } catch (IOException e) {
              log.info("Error while analyzing the url : {}", e);
              throw new WebApplicationException(e, Response.Status.INTERNAL_SERVER_ERROR);
          }
      }
  </code></pre>
  
  <p>I was thinking of following approaches:-</p>
  
  <ul>
  <li><p>Instead of calling the dropwizard resource method via POST, I call <code>orchestrator.detectContent(urlRequest)</code> directly from scheduler.</p></li>
  <li><p>The orchestrator can return detectionScore and I'll store all the detectScores in a map/table and perform batch database insertion instead of individual insertion as in the present code.</p></li>
  </ul>
  
  <p>I would like some comments on the above approaches and possibly other techniques by which I can improve the running time. Also, I just read about Java asynchronous programming but can't seem to understand how to use it in the above code, so would like some help with this also.</p>
  
  <p>Thanks.</p>
  
  <p>Edit:
  I can think of two bottlenecks:</p>
  
  <ul>
  <li>Downloading of webpage</li>
  <li>Insertion of result in to the database (Database is located in another system)</li>
  <li>It seems that processing is performed 1 URL at a time</li>
  </ul>
  
  <p>The system has 8 GB of memory out of which 4 GB appears to be free</p>
  
  <pre><code>$ free -m
               total       used       free     shared    buffers     cached
  Mem:          7843       4496       3346          0        193       2339
  -/+ buffers/cache:       1964       5879 
  Swap:         1952        489       1463 
  </code></pre>
  
  <p>CPU usage is also minimal:</p>
  
  <pre><code>top - 13:31:19 up 19 days, 15:39,  3 users,  load average: 0.00, 0.00, 0.00
  Tasks: 215 total,   1 running, 214 sleeping,   0 stopped,   0 zombie
  Cpu(s):  0.5%us,  0.0%sy,  0.0%ni, 99.4%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
  Mem:   8031412k total,  4605196k used,  3426216k free,   198040k buffers
  Swap:  1999868k total,   501020k used,  1498848k free,  2395344k cached
  </code></pre>