title: DuckDuckGo Scraper, can not go to next page
body: <p>I have written a Url scraping code that works on bing and Google and navigates fine through the pages.</p>
  <p>I am now tring to set it to work on duckduckgo.com. I have got it working so it scrapes from a single page. The ONLY problem I have is I CAN NOT workout how to get it to show more results, how to navigate. It only pulls results from the first page.</p>
  <p>Google and Bing have a next button which the code can navigate, however I can not work out how to do it for duckduckgo. I am stuck on this last bit. <strong>The rest is fine.</strong> <em>The search results, pages to navigate and 2x delays come from Sheet10</em></p>
  <p><strong>I DO NOT need the fully code re-written. All I need help on is how to navigate the pages.</strong></p>
  <pre><code>Private Sub duckduckgoScraper()
  '''DuckDuckGo URL SCRAPER
  Dim ie As Object
  Dim HTMLdoc As Object
  Dim nextPageElement As Object
  Dim div As Object
  Dim link As Object
  Dim url As String
  Dim pageNumber As Long
  Dim i As Long
  Dim myCounter As Long
  
  '''Takes search from Sheet10 to DuckDuckGo
      url = &quot;https://duckduckgo.com/?q=&quot; &amp; Replace(Worksheets(&quot;Sheet10&quot;).Range(&quot;G17&quot;).Value &amp; Range(&quot;H17&quot;).Value, &quot; &quot;, &quot;+&quot;)
  Set ie = CreateObject(&quot;InternetExplorer.Application&quot;)
      
  With ie
      .Visible = True
      .navigate url
  Do While .Busy Or .readyState &lt;&gt; 4
      DoEvents
  Loop
  End With
  
  Application.Wait Now + TimeSerial(0, 0, 5)
      Set HTMLdoc = ie.document
  
  '''Searches URLS and places them in Sheet called Sheet2  ROW 2 Column A
  With Sheets(&quot;Sheet2&quot;)
      pageNumber = 1
      i = 2
  Do
      For Each div In HTMLdoc.getElementsByTagName(&quot;div&quot;)
          If div.getAttribute(&quot;class&quot;) = &quot;result__body links_main links_deep&quot; Then
              Set link = div.getElementsByTagName(&quot;a&quot;)(0)
              .Cells(i, 1).Value = link.getAttribute(&quot;href&quot;)
              i = i + 1
          End If
      Next div
              
  '''Searches Number of Pages entered in Sheet10
      If pageNumber &gt;= Replace(Worksheets(&quot;Sheet10&quot;).Range(&quot;I17&quot;).Value, &quot; &quot;, &quot;+&quot;) Then Exit Do
          On Error Resume Next
  
  '''################################################################################################
  '''##########################  **I am stuck here, the rest is fine**  #############################
  '''################################################################################################
          Set nextPageElement = HTMLdoc.getElementByClassName(&quot;I NEED THIS BIT, I AM STUCK HERE&quot;)
      If nextPageElement Is Nothing Then Exit Do
   
         
  '''Scrolls Down the Browser
      ie.document.parentWindow.Scroll 0&amp;, 99999
  '''Random delay from Max number entered in Sheet10
      Application.Wait Now + TimeSerial(0, 0, Application.RandBetween(1, Worksheets(&quot;Sheet10&quot;).Range(&quot;J17&quot;).Value))
  '''Click the next page
          nextPageElement.Click 
      Do While ie.Busy Or ie.readyState &lt;&gt; 4
          DoEvents
      Loop
  '''Random delay from Max number entered in Sheet10
      Application.Wait Now + TimeSerial(0, 0, Application.RandBetween(1, Worksheets(&quot;Sheet10&quot;).Range(&quot;K17&quot;).Value))
          Set HTMLdoc = ie.document
  ''' Delete duplicates
      Sheet2.Columns(&quot;A&quot;).RemoveDuplicates Columns:=Array(1), Header:=xlYes
  ''' Delete Row If Blank
      Sheet2.Columns(&quot;A&quot;).SpecialCells(xlCellTypeBlanks).EntireRow.Delete
          pageNumber = pageNumber + 1
  myCounter = myCounter + 1
  Worksheets(&quot;Sheet10&quot;).Range(&quot;G6&quot;).Value = myCounter
      Loop
  End With
  
  '''Quite browser and clear
  ie.Quit
      Set ie = Nothing
      Set HTMLdoc = Nothing
      Set nextPageElement = Nothing
      Set div = Nothing
      Set link = Nothing
  
  ''' To stop the code early, change page number to 0, else code will finish when page number completed
  If Sheet10.Range(&quot;I17&quot;) = 0 Then
      Complete.Show
          Termination.Hide
  ElseIf Sheet10.Range(&quot;I17&quot;) &gt; 0 Then
      Complete.Show
  End If
  End Sub
  </code></pre>
  <p><strong>What I have tried So Far</strong>
  I have tried the bits in yellow,however I can not get it to work. The rld-1 will change to rdl-2 and 3 when the next button is clicked. This is the only bit i am now stuck on.
  <a href="https://i.stack.imgur.com/tSKIa.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/tSKIa.png" alt="enter image description here" /></a>
  <a href="https://i.stack.imgur.com/eSMo9.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/eSMo9.png" alt="enter image description here" /></a></p>
  <p>Please could someone advise. I think it may be due to it being JavaScript, but I have limited knowledge in Programing and have been stuck on this for  few days now.</p>
  <p>As always, thanks in advance</p>