title: How to Web Scrape muliple JavaScript Rendered Pages with Ruby on Rails and Watir
body: <p>This is the page I am trying to get the results of </p>
  
  <pre><code>https://www.cargurus.com/Cars/inventorylisting/viewDetailsFilterViewInventoryListing.action?sourceContext=carGurusHomePage_false_0&amp;formSourceTag=104&amp;entitySelectingHelper.selectedEntity=m2&amp;zip=48126#resultsPage=1
  </code></pre>
  
  <p>Because the search results are rendered by JavaScript, I need to use the WebDriver Watir. I want to get the results of the first 5 pages. However, when I can only get the first page only and not other pages. How do I visit the other pages?</p>
  
  <pre><code>    $pageCount = 1
  
  5.times do  
                  page = $pageCount.to_s
                  browser = Watir::Browser.start("https://www.cargurus.com/Cars/inventorylisting/viewDetailsFilterViewInventoryListing.action?sourceContext=carGurusHomePage_false_0&amp;formSourceTag=104&amp;entitySelectingHelper.selectedEntity=m2&amp;zip=48126#resultsPage=#{page}", browser = :phantomjs)
                  #browser.link(:class, "nextPageElement").wait_until_present
                  doc = Nokogiri::HTML.parse(browser.html)
                      entries = doc.css('.cg-dealFinder-result-wrap')
                      entries.each do |entry|
                            title = entry.css('.cg-dealFinder-result-model&gt;span')[0].text.strip
                            link = entry.css('.cg-dealFinder-result-stats&gt;p&gt;span')[0].text.strip
                            newEntry = Entry.new(title: title, description:link)
                            if title.present? &amp;&amp; link.present? &amp;&amp; Entry.where(title: title, description: link).blank?
                               newEntry.save
                            end
                      end
                  $pageCount = $pageCount + 1
                  browser.close
          end
  </code></pre>
  
  <p>Btw I am able to successfully web scrape multiple for other websites that have non-JavaScript rendered page results.</p>