title: Website Source Code not showing entire content that needs to be scrapped in rvest
body: <p>Hi I am new to stack overflow so would appreciate any help I can get, I am doing an NLP project where I am going to do some analysis on the works of some Urdu writes, the poems or shayaris of the writer are available on this link &quot;https://www.rekhta.org/poets/mirza-ghalib/couplets&quot; , but when I run the code I wrote for getting the text using rvest in r, it only gives me the first 50 blocks of text out of a desired 370, the rest it says are empty vectors, so I  looked at the page source code and can only see the code for the first 50 blocks of poetry, and none of the rest even though the rest are visible when you open the link scroll down below the 50th block.</p>
  <p>This is the code I am using</p>
  <pre><code>
  library(tidyverse)
  library(rvest)
  link&lt;- &quot;https://www.rekhta.org/poets/mirza-ghalib/couplets&quot;
  page &lt;- read_html(link)
  nos&lt;- (1:370)
  nos&lt;-paste(&quot;(&quot;,nos,&quot;)&quot;, sep=&quot;&quot;)
  nos
  nodes&lt;-paste(&quot;.sherSection:nth-child&quot;,nos,&quot; .c span&quot;, sep=&quot;&quot;)
  nodes
  nodes[51]
  shayari &lt;- page %&gt;% 
    html_nodes(&quot;.sherSection:nth-child(51) .c span&quot;) %&gt;% 
    html_text() %&gt;%
    paste(collapse = &quot;&quot;)
  shayari
  getshayari &lt;-function(node) { 
    shayari&lt;-page %&gt;% 
    html_nodes(node) %&gt;% 
    html_text() %&gt;%
    paste(collapse = &quot;&quot;)
    return(shayari)
    
  }
  g_shayaris = sapply(nodes, FUN = getshayari, USE.NAMES = FALSE)
  g_shayaris
  s&lt;-page %&gt;% html_nodes(&quot;.sherSection:nth-child(51) p:nth-child(1) span:nth-child(1)&quot;)  %&gt;% 
    html_text()
  s
  </code></pre>
  <p>I get the first 50 blocks of poetry but the rest are all empty, maybe I think this has something to do with how the website only loads the first 50 blocks first then the next 50 and so on and so forth, but I am very new to all of this so any help would be great.</p>
  <p>Hi, I have since tried using RSelenium to try and load the dynamic webpage but Selenium refuses to open the browser even after employing docker.</p>
  <pre><code>system('docker run --platform linux/amd64 -d -p 4445:4444 selenium/standalone-chrome')
  remDr &lt;- remoteDriver(remoteServerAddr = &quot;localhost&quot;, port = 4445L, browserName = &quot;chrome&quot;)
  remDr$open()
  remDr$navigate(&quot;https://www.rekhta.org/poets/mirza-ghalib/couplets&quot;)
  remDr$getTitle()
  rD &lt;- rsDriver(browser=&quot;chrome&quot;, port=4445L, chromever = &quot;102.0.5005.61&quot;)
  driver &lt;- rsDriver(remoteServerAddr = &quot;localhost&quot;, port=4445L, browser=c(&quot;firefox&quot;), version = &quot;latest&quot;)
  remDr &lt;- rD[[&quot;client&quot;]]
  remDr$navigate(&quot;https://www.rekhta.org/poets/mirza-ghalib/couplets&quot;)
  </code></pre>
  <p>But none of them work, I get something like this</p>
  <pre><code>&gt; remDr &lt;- remoteDriver(remoteServerAddr = &quot;localhost&quot;, port = 4445L, browserName = &quot;chrome&quot;)
  &gt; remDr$open()
  [1] &quot;Connecting to remote server&quot;
  $id
  [1] NA
  &gt; remDr$navigate(&quot;https://www.rekhta.org/poets/mirza-ghalib/couplets&quot;)
  NULL
  &gt; rD &lt;- rsDriver(browser=&quot;chrome&quot;, port=4445L, chromever = &quot;102.0.5005.61&quot;)
  checking Selenium Server versions:
  BEGIN: PREDOWNLOAD
  BEGIN: DOWNLOAD
  BEGIN: POSTDOWNLOAD
  checking chromedriver versions:
  BEGIN: PREDOWNLOAD
  BEGIN: DOWNLOAD
  BEGIN: POSTDOWNLOAD
  checking geckodriver versions:
  BEGIN: PREDOWNLOAD
  BEGIN: DOWNLOAD
  BEGIN: POSTDOWNLOAD
  checking phantomjs versions:
  BEGIN: PREDOWNLOAD
  BEGIN: DOWNLOAD
  BEGIN: POSTDOWNLOAD
  [1] &quot;Connecting to remote server&quot;
  $id
  [1] NA
  </code></pre>