title: R: scraping additional data after POST only works for first page
body: <p>I would like to scrape drug informations offered by the Swiss government for an University research project from:</p>
  
  <p><a href="http://www.spezialitaetenliste.ch/ShowPreparations.aspx?searchType=Substance&amp;searchValue=" rel="nofollow noreferrer">http://www.spezialitaetenliste.ch/ShowPreparations.aspx?searchType=Substance&amp;searchValue=</a></p>
  
  <p>The page does offer a robotx.txt file, however, it's content is freely available to the public and I assume that scraping this data is unprohibited. </p>
  
  <p>This is an update <a href="https://stackoverflow.com/questions/56048418/r-scraping-aspx-with-content-from-dopostback-script">of this question</a>, since I made some progress.</p>
  
  <p><strong>What I achieved so far</strong></p>
  
  <pre><code># opens the first results page 
  # opens the first link as a table at the end of the page
  
  library("rvest")
  library("dplyr")
  
  
  url &lt;- "http://www.spezialitaetenliste.ch/ShowPreparations.aspx?searchType=Substance&amp;searchValue="
  pgsession&lt;-html_session(url)
  pgform&lt;-html_form(pgsession)[[1]]
  
  page&lt;-rvest:::request_POST(pgsession,url,
                             body=list(
                               `ctl00$cphContent$gvwPreparations$ctl13$gvwpPreparations$txtPageNumber`=1,
                               `__VIEWSTATE`=pgform$fields$`__VIEWSTATE`$value,
                               `__VIEWSTATEGENERATOR`=pgform$fields$`__VIEWSTATEGENERATOR`$value,
                               `__VIEWSTATEENCRYPTED`=pgform$fields$`__VIEWSTATEENCRYPTED`$value,
                               `__EVENTVALIDATION`=pgform$fields$`__EVENTVALIDATION`$value,
                               `ctl00$cphContent$gvwPreparations$ctl13$gvwpPreparations$ddlPageSize`="10",
                               `__EVENTTARGET`="ctl00$cphContent$gvwPreparations$ctl02$ctl00",
                               `__EVENTARGUMENT`=""
  
                               ),
                             encode="form")
  </code></pre>
  
  <p>next: get the basic data</p>
  
  <pre><code># makes a table of all results of the first page
  
  read_html(page) %&gt;%
    html_nodes(xpath = '//*[@id="ctl00_cphContent_gvwPreparations"]') %&gt;%
    html_table(fill=TRUE) %&gt;% 
    bind_rows %&gt;%
    tibble()
  </code></pre>
  
  <p>next: get the additional data</p>
  
  <pre><code># gives the desired informations (=additional data) of the first drug (not yet very structured)
  
  read_html(page) %&gt;%
    html_nodes(xpath = '//*[@id="ctl00_cphContent_fvwPreparation"]') %&gt;%
    html_text 
  </code></pre>
  
  <p><strong>My Problem:</strong></p>
  
  <pre><code># if I open the second  search page
  
  page&lt;-rvest:::request_POST(pgsession,url,
                             body=list(
                               `ctl00$cphContent$gvwPreparations$ctl13$gvwpPreparations$txtPageNumber`=2,
                               `__VIEWSTATE`=pgform$fields$`__VIEWSTATE`$value,
                               `__VIEWSTATEGENERATOR`=pgform$fields$`__VIEWSTATEGENERATOR`$value,
                               `__VIEWSTATEENCRYPTED`=pgform$fields$`__VIEWSTATEENCRYPTED`$value,
                               `__EVENTVALIDATION`=pgform$fields$`__EVENTVALIDATION`$value,
                               `ctl00$cphContent$gvwPreparations$ctl13$gvwpPreparations$ddlPageSize`="10",
                               `__EVENTTARGET`="ctl00$cphContent$gvwPreparations$ctl02$ctl00",
                               `__EVENTARGUMENT`=""
  
                               ),
                             encode="form")
  </code></pre>
  
  <p>next: get the new basic data</p>
  
  <pre><code># I get easily a table with the new results
  
  read_html(page) %&gt;%
    html_nodes(xpath = '//*[@id="ctl00_cphContent_gvwPreparations"]') %&gt;%
    html_table(fill=TRUE) %&gt;% 
    bind_rows %&gt;%
    tibble()
  </code></pre>
  
  <p><strong>But if I try to get the new additional data, I get the results from page 1 again:</strong></p>
  
  <pre><code># does not give the desired output:
  
  read_html(page) %&gt;%
    html_nodes(xpath = '//*[@id="ctl00_cphContent_fvwPreparation"]') %&gt;%
    html_text 
  </code></pre>
  
  <p>What I am looking for: the detailed data of the first drug of page 2
  <a href="https://i.stack.imgur.com/wCfVH.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/wCfVH.png" alt="enter image description here"></a></p>
  
  <p><strong>Questions:</strong></p>
  
  <ul>
  <li>Why do I get duplicate results? Is it because of the <code>__VIEWSTATE</code> that might
  change during the new <code>request_POST</code> ?</li>
  <li>Is there any way to solve this problem?</li>
  <li>Is there any better way how to get the basic and additional data? If yes, how?</li>
  </ul>