title: How to crawl an Angular JS web app?
body: <p>I am trying to crawl a web app which has login as the first hurdle and is built on angular js. I was using scrapy and selenium to crawl the website but there is a problem with logging in. I have this post where I asked for help <a href="https://stackoverflow.com/questions/30926880/what-is-the-best-way-for-crawling-with-scrapy-and-selenium-angularjs-website">https://stackoverflow.com/questions/30926880/what-is-the-best-way-for-crawling-with-scrapy-and-selenium-angularjs-website</a> but I didn't get any help. The problem is when I use selenium(shown below) for reading the innerHTML of the body element it is giving me blank response. Why is it so? Is it because of HTTPS or because any other permission issues?</p>
  
  <p>Also if some one can help me with how to use both scrapy and selenium along with login and cookies to scrap a website. That will be really helpful.</p>
  
  <pre><code>def crawl_url(url, run_headless=True):
      if run_headless:
          display = Display(visible=0, size=(1024, 768))
          display.start()
  
      url = correct_url(url)
      browser = webdriver.PhantomJS(service_args=['--load-images=no'])
      login_url = "https://domain.com/login"
      browser.get(login_url)
      time.sleep(15)
      element = browser.find_element_by_xpath('/html/body')
      print element.get_attribute('innerHTML')
  </code></pre>