title: Concurrent HTTP URL connections in Java
body: <p>I am developing a tool that grabs all the text files from a given folder. These text files contain bunch of http URL. The requirement is that I do a GET request to these URL's , log the response code and optionally print the input stream. 
  I have to perform this in multi-threaded fashion and the thread count should be configurable. So I am using TestNG and using data provider with parallel=true to set this up. Here is what I have so far. 
  Am I doing this right?</p>
  
  <p><strong>Edit#2</strong>
  The other issue I found is that when I run this tool for 90K URLs and with 100 threads..The requests are sent fast in the beginning but they slow down remarkably after a while.  What could be the issue here? </p>
  
  <p><strong>Edit#1</strong>
  I am able to run this tool for small number of records like a few thousands but I am not able it for a significantly high number of URLs(250K) and threads (2500). I am using ReportNG to generate reports. Here are the exceptions I get intermittently when I run my tool for 250K URLs and 2500 threads. </p>
  
  <pre><code>java.net.SocketException: Unexpected end of file from server
  at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
  at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
  at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
  at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
  at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
  at java.net.HttpURLConnection.getResponseCode(Unknown Source)
  at com.HTTPConn.doHttpUrlConnectionAction(HTTPConn.java:83)
  at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  at java.lang.reflect.Method.invoke(Unknown Source)
  at 
  
  org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
      at org.testng.internal.Invoker.invokeMethod(Invoker.java:673)
      at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:842)
      at org.testng.internal.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:73)
      at org.testng.internal.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:14) 
  
      [TestNG] Reporter org.uncommons.reportng.HTMLReporter@eee36c failed
  org.uncommons.reportng.ReportNGException: Failed generating HTML report.
      at org.uncommons.reportng.HTMLReporter.generateReport(HTMLReporter.java:118)
      at org.testng.TestNG.generateReports(TestNG.java:1053)
      at org.testng.TestNG.run(TestNG.java:1016)
      at com.TestNGRunner.main(TestNGRunner.java:83)
  Caused by: java.lang.NullPointerException
      at java.lang.String.compareTo(Unknown Source)
      at org.uncommons.reportng.TestResultComparator.compare(TestResultComparator.java:29)
      at org.uncommons.reportng.TestResultComparator.compare(TestResultComparator.java:25)
      at java.util.Collections.indexedBinarySearch(Unknown Source)
      at java.util.Collections.binarySearch(Unknown Source)
      at org.uncommons.reportng.HTMLReporter.sortByTestClass(HTMLReporter.java:229)
      at org.uncommons.reportng.HTMLReporter.createResults(HTMLReporter.java:180)
      at org.uncommons.reportng.HTMLReporter.generateReport(HTMLReporter.java:110)
      ... 3 more
  
  
  public class HTTPConn {
      Logger log;
      File folder;
      boolean isStreamRequired;
      @BeforeClass(alwaysRun=true)
      public void setup(){
          log = Logger.getLogger(HTTPConn.class);
          folder = new File("c:\path\to\folder\from\where\I\want\to\read\textfiles");
          isStreamRequired = false;
      }
  
      @DataProvider(name="getRecords", parallel=true)
      public Object[][] getURLsFromTheFile() throws IOException {
          try {
              File [] listFiles = folder.listFiles();
              int i=0;
              Object[][] result = null;
              ArrayList&lt;String&gt; totalList = new ArrayList&lt;String&gt;();          
              for(int ind=0;ind&lt;listFiles.length;ind++){
                  if(listFiles[ind].isFile()){
                      List&lt;String&gt; arrList =FileUtils.readLines(listFiles[ind]); 
                      totalList.addAll(arrList);//add files one by one
                  }
              }           
              result=new Object[totalList.size()][];                  
              for(String s:totalList){                        
               result[i]=new Object[]{s};
               i++;
           }              
           return result;
      } catch (Exception e) {
          // TODO Auto-generated catch block
          e.printStackTrace();
          return null;
      }
   }
      @Test(dataProvider = "getRecords")
      public void doHttpUrlConnectionAction(String desiredUrl){
           try {
                URL url = new URL(desiredUrl);
                HttpURLConnection connection = (HttpURLConnection) url.openConnection();
                connection.setRequestMethod("GET");             
                connection.connect();
                int responseCode = connection.getResponseCode();  
                log.info("\nResponse Code:"+ desiredUrl+"\n"+responseCode+"\n");        
                if(responseCode!=200)
                    Assert.fail("Response Code:"+responseCode);
                else
                    Reporter.log("Response Code:"+responseCode);
                // read the output from the server
                if(isStreamRequired) {
                   //read the input stream and print it to log
                    reader.close();               
              }//isStreamOutputRequired           
                connection.disconnect();           
          } catch (Exception e) {
              // TODO Auto-generated catch block
              e.printStackTrace();
          } 
      }
  }
  </code></pre>