title: Using Selenium + Scrapy
body: <p>I'm trying to use scrapy with selenium to be able to interact with javascript and still have the powerful scraping framework that scrapy offers. I've written a script that visits <a href="http://www.iens.nl" rel="nofollow noreferrer">http://www.iens.nl</a>, enters "Amsterdam" in the search bar and then clicks on the search button succesfully. After clicking on the search button I want scrapy to retreive an element from the newly rendered page. Unfortunately scrapy doesn't return any values.</p>
  
  <p>This is what my code looks like:</p>
  
  <pre><code>from selenium import webdriver
  from scrapy.loader import ItemLoader
  from scrapy import Request
  from scrapy.crawler import CrawlerProcess
  from properties import PropertiesItem
  import scrapy
  
  
  class BasicSpider(scrapy.Spider):
      name = "basic"
      allowed_domains = ["web"]
      # Start on a property page
      start_urls = ['http://www.iens.nl']
  
      def __init__(self):
          chrome_path = '/Users/username/Documents/chromedriver'
          self.driver = webdriver.Chrome(chrome_path)
  
      def parse(self, response):
          self.driver.get(response.url)
          text_box = self.driver.find_element_by_xpath('//*[@id="searchText"]')
          submit_button = self.driver.find_element_by_xpath('//*[@id="button_search"]')
          text_box.send_keys("Amsterdam")
          submit_button.click()
  
          l = ItemLoader(item=PropertiesItem(), response=response)
          l.add_xpath('description', '//*[@id="results"]/ul/li[1]/div[2]/h3/a/')
  
          return l.load_item()
  
  
  process = CrawlerProcess()
  process.crawl(BasicSpider)
  process.start()
  </code></pre>
  
  <p>"properties" is another script that looks like this:</p>
  
  <pre><code>from scrapy.item import Item, Field
  
  class PropertiesItem(Item):
      # Primary fields
      description = Field()
  </code></pre>
  
  <p><strong>Q:</strong> How do I succesfully make scrapy find the element I call "description" by its xpath on the page selenium reached and return it as output?</p>
  
  <p>Thanks in advance!</p>