title: Executing javascript code to accept terms and open next page
body: <p>I would like to crawl a javacode rendered website that requires clicking on an 'Accept Terms' button to enter.  I am using Scrapy and Splash and have tried to execute the javascript code using the both splash endpoints 'render.html' and 'execute'.  In both cases the the output is the start page. Why dosent this work as expected?</p>
  
  <p>url = start page with the "Accept Terms" button.</p>
  
  <p>url/index.aspx = page I want to render.</p>
  
  <p>Using render.html:</p>
  
  <pre><code>yield scrapy.Request('url', self.parse, meta={ 'splash':
  {   'endpoint':'render.html','args': {'js_source':
  'document.getElementById("AcceptTerms").click();', 'html': 1, 'wait':
  0.5}}})
  </code></pre>
  
  <p>or by using execute and lua:</p>
  
  <pre><code>lua_source_string = 'function main(splash)
  splash:go("url/index.aspx")
  splash:wait(0.5)
  splash:runjs("document.getElementById(\'AcceptTerms\').click();")
  return splash:html() end'
  
  yield scrapy.Request('url', self.parse, meta={ 'splash': { 'endpoint':'execute','args': {'lua_source' : lua_source_string}}})
  </code></pre>
  
  <p>'url' is the page that is rendered.</p>
  
  <p>If I follow the example from <a href="http://blog.scrapinghub.com/2015/03/02/handling-javascript-in-scrapy-with-splash/" rel="nofollow">http://blog.scrapinghub.com/2015/03/02/handling-javascript-in-scrapy-with-splash/</a> and use the following lua string with jquery like so:</p>
  
  <pre><code>lua_source_string = 'function main(splash)
  splash:autoload("https://ajax.googleapis.com/ajax/libs/jquery/2.1.3/jquery.min.js")
  splash:go("url/index.aspx")
  splash:wait(0.5)
  splash:runjs("$(\'#AcceptTerms\').click();")
  return splash:html() end'
  </code></pre>
  
  <p>Or using the jquery code like so:</p>
  
  <pre><code>lua_source_string = 'function main(splash)
  splash:autoload("i/am/restricted/to/only/two/links/see/above/jquery.min.js")
  splash:go("url/index.aspx")
  splash:wait(0.5)
  splash:runjs("$(\'#AcceptTerms\').trigger(\'click\');")
  return splash:html() end'
  </code></pre>
  
  <p>I get the same results.  The rendered page is 'url'.</p>