title: Web scraping with java and jsoup
body: <p>I am trying to scrape data from the following table.</p>
  <p><a href="https://finance.yahoo.com/quote/%5EVIX/history?p=%5EVIX" rel="nofollow noreferrer">Yahoo finance CBOE Volatility Index</a></p>
  <p>I am using jsoup for it.</p>
  <pre><code>        String url = &quot;https://finance.yahoo.com/quote/%5EVIX/history?p=%5EVIX&amp;guccounter=1&amp;guce_referrer=aHR0cHM6Ly9tYWlsLmdvb2dsZS5jb20v&amp;guce_referrer_sig=AQAAAKU5UXnZEhNK_s1k-l6fQ7l-jFaR2xghH5NOhaohsec-HThT1BaEsni-hUlysVCFWpzd4qa2OZ2YZtBDJNQqKw1Uh64_nppDI4RnzPnTgxDGta123-A_SbIBm4SA5B0xopHvDcl5A21esFvWceZnRJPk6ohtud7OGJpWcNLdADYT&quot;;
          Document doc = Jsoup.connect(url).get();
  
          Element table = doc.getElementById(&quot;mrt-node-Col1-1-HistoricalDataTable&quot;);
          Elements rows=table.select(&quot;tr&quot;);
  
          Elements first=rows.get(0).select(&quot;th,td&quot;);
          List&lt;String&gt;headers=new ArrayList&lt;&gt;();
  
          for(Element header:first)
              headers.add(header.text());
  
  
          List&lt;Map&lt;String,String&gt;&gt; listMap = new ArrayList&lt;Map&lt;String,String&gt;&gt;();
          for(int row=1;row&lt;rows.size()-1;row++) {
              Elements colVals = rows.get(row).select(&quot;th,td&quot;);
              int colCount = 0;
              Map&lt;String,String&gt; tuple = new LinkedHashMap&lt;String,String&gt;();
              for(Element colVal : colVals)
                  tuple.put(headers.get(colCount++), colVal.text());
              listMap.add(tuple);
          }
  </code></pre>
  <p>By this approach I am only getting the first 100 or some more rows. This is because it first loads that amount of rows and whenever we scroll to that position of the row, newer rows are loaded. I could not find any pagination and nothing helpful from the network calls.The data seems to be encoded in gif format(whenever there is a mouse event on scroll).</p>
  <p>I found a way around to use selenium web driver and fetch all the data. I was wondering is there any way to just use Jsoup to solve the issue.</p>