title: Python's requests_html render() function does not allow to persist CSRF token - cannot login to website
body: <p>I need to use .render() to get the entire DOM content off of a website managed by a dynamic front-end framework (JS or Ajax e.g.). That website stores a CSRF token in a hidden input field as usual. When I specify use r.html.render(send_cookies_session=True) or r.html.render(), the dynamic part of the DOM is rendered and I capture the CSRF token, but after I logged in and try to crawl a route behind the login page, the CSRF token changes and I am redirected to the login page.</p>
  <p>Then when I use r.html.render(reload=False), the dynamic part of the DOM including the hidden div with the CSRF token is not rendered. I can obviously not login as I cannot capture the CSRF token.</p>
  <p>I have tried a lot of options for render(), but nothing seems to work.</p>
  <pre><code>   def login(self):
          payload = json.loads(self.options.login_params)
  
          with HTMLSession() as session:
              r = session.get(self.login_url)
  
              # need to turn off reload or else a new CSRF token is captured when the page is reloaded
              # render the html to capture the CSRF token
              r.html.render(send_cookies_session=True)
              # r.html.render(reload=False)
  
              html = BeautifulSoup(
                  r.html.html, &quot;html.parser&quot;)
  
              # save CSRF token to payload dict
              payload = self.add_csrf_to_payload(html, payload)
  
              # login with CSRF token from payload
              p = session.post(self.login_url, data=payload)
              html = BeautifulSoup(r.html.html, &quot;html.parser&quot;)
  
              # download HTML from target URL using CSRF =&gt; here it breaks down as the CSRF changes due to the render() function (unless set to reload=False)
              r = session.get(self.target_url)
              html = BeautifulSoup(r.html.html, &quot;html.parser&quot;)
              payload = self.add_csrf_to_payload(html, payload)
  
              self.session = session
              self.payload = payload
  
              return html
  </code></pre>