title: Scrapy Xpath construction producing empty brackets on dynamic site
body: <p>I am trying to create a spider via scrapy to crawl a website and extract all links for specific stores. Ultimately, the spider would then use those store links to extract pricing information. The site is designed to break down store information into States and Regions. I have been able to extract all sub links for the States/regions, but I cannot seem to extract the specific information for the stores, that information being the store links. </p>
  
  <p>I thought it had to do with the page not being loaded long enough for those store links to display. So I used selenium and delayed the timing by 20 seconds to allow appropriate time for the webpage to load. An example link would be <a href="https://weedmaps.com/dispensaries/in/united-states/colorado" rel="nofollow">https://weedmaps.com/dispensaries/in/united-states/colorado</a> and as you can see on the left hand side there is information for specific stores. A snippet using the inspect element yields the following HTML: </p>
  
  <pre><code>&lt;li class="ng-scope" ng-repeat="listing in listings" wm-listing-detail="" role="listing"&gt;
    &lt;div ng-class="{unpublished: !listing.published}"&gt;
       &lt;div class="listing feature-1 recreational dispensary" ng-     class="featureClassesFor(listing)" ng-click="setActiveMarker(listing)"&gt;
           &lt;img class="avatar" ng-src="https://d2kxqxnk1i5o9a.cloudfront.net/uploads/avatars/dispensaries/5566/square_10569095_10152344322971376_2924814837799131094_n.jpg" src="https://d2kxqxnk1i5o9a.cloudfront.net/uploads/avatars/dispensaries/5566/square_10569095_10152344322971376_2924814837799131094_n.jpg"/&gt;
           &lt;div&gt;
             &lt;div class="name hovers"&gt;
                &lt;a class="ng-binding" href="/dispensaries/organic-alternatives" role="listing url" ng-bind-html="listing.name"&gt;Organic Alternatives&lt;/a&gt;
             &lt;/div&gt;
           &lt;/div&gt;
           &lt;div class="address ng-binding"&gt;
           &lt;div class="wm_map_inactive_rating"&gt;
  </code></pre>
  
  <p>I am looking to extract the information following the @href.</p>
  
  <p>I have the following python code in the spider section of the scrapy project:</p>
  
  <pre><code>import scrapy
  from scrapy.spider import BaseSpider 
  from selenium import webdriver
  from selenium.webdriver.support.ui import WebDriverWait
  
  class scrapybotspider(scrapy.Spider):
      name = "scrapybot_spider"
      start_urls = ['https://weedmaps.com/dispensaries/in/united-states/colorado']
      delay = 20
  
      def __init__(self):
          self.driver = webdriver.Firefox()
  
      def parse(self, response):
          self.driver.get(response.url)
  
          while True:
              StateCountyURL = response.xpath("//*[@id='wm-listings']/div/li[2]/div/div/div[1]/div/@href").extract()
              print StateCountyURL
  </code></pre>
  
  <p>This produces empty brackets and I have tried a number of different xpath constructions.</p>