title: Screen scraping in Java
body: <p>I'm trying to create an application, written in java, that uses my university class search function. I am using a simple http get request with the following code:</p>
  
  <pre><code>public static String GET_Request(String urlToRead) {
  
      java.net.CookieManager cm = new java.net.CookieManager();
      java.net.CookieHandler.setDefault(cm);
  
      URL url;
      HttpURLConnection conn;
      BufferedReader rd;
      String line;
      String result = "";
  
      try {
  
          url = new URL(urlToRead);     
          conn = (HttpURLConnection) url.openConnection();    
          conn.setRequestMethod("GET");
          rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
          while ((line = rd.readLine()) != null) {
             result += line;
      }
  
      rd.close();
  
      } 
      catch (Exception e) {
          e.printStackTrace();
      }
      return result;
  }
  </code></pre>
  
  <p>But it is not working.</p>
  
  <p>Here is the url I am trying to scrape:
  <a href="https://webapp4.asu.edu/catalog/classlist?c=TEMPE&amp;s=CSE&amp;n=100&amp;t=2141&amp;e=open&amp;hon=F" rel="nofollow">https://webapp4.asu.edu/catalog/classlist?c=TEMPE&amp;s=CSE&amp;n=100&amp;t=2141&amp;e=open&amp;hon=F</a></p>
  
  <p>I tried looking into jsoup but when I go to their try jsoup tab and fetch the url it is coming up with the same results as the get request is coming up with.</p>
  
  <p>The, repeated, failed results that I'm getting with the http get request and jsoup is that it is bring up the search page of the university but not the actual classes and information about if they are open or not.</p>
  
  <p>What I am ultimately looking for is a way to scrape the website that shows if the classes have open seats or not. Once I get the contents of the web page I could parse through it I'm just not getting any good results.</p>
  
  <p>Thanks!</p>