title: Passing Scrapy response URL to Selenium and then Selenium response back to Scrapy
body: <p>How to pass a Scrapy response url to Selenium and then selenium response back to Scrapy</p>
  
  <p>I have this Scrapy spider <strong>first.py</strong> : </p>
  
  <pre><code># -*- coding: utf-8 -*-
  import scrapy
  import re
  import json
  
  
  class FirstSpider(scrapy.Spider):
      name = "first"
      allowed_domains = ["someautosite.co.uk"]
      start_urls = (
          'http://www.someautosite.co.uk/some_specific_search_results',
      )
  
      def parse(self, response):
          for car_url in response.xpath('//article[contains(@class, "standard")]/div/div[2]/div[1]/h1/a/@href').extract():
              absoluteurl = response.urljoin(car_url)
              # yield {'URL': absoluteurl}
              yield scrapy.Request(absoluteurl, callback=self.parse_car)
  
      def parse_car(self, response):
          pattern = re.compile(r"var utag_data = ({.*?});", re.MULTILINE | re.DOTALL)
          utag_data = response.xpath('//script[contains(.,"var utag")]/text()').re(pattern)[0]
          utag_data_obj = json.loads(utag_data)
          # make = utag_data_obj['make']
          # model = utag_data_obj['model']
          # yield {'Make':utag_data_obj['make'],
          #        'model':utag_data_obj['model'],
          #        }
          # yield utag_data
          tel = response.xpath('//article/div[3]/section/div/div[@itemprop="telephone"]/text()').extract_first()
          # tel_json_str = '{"tel":"' + str(tel) + '"}'
          # tel_json_obj = json.loads(tel_json_str)
          # Combine 2 JSON objects into one:
          car_json = utag_data_obj.copy()
          car_json.update({"tel": tel})
          yield car_json
          quotations_url = response.xpath('/html/body/article/section/ul/li[2]/a/@href').extract_first()
          yield scrapy.Request(quotations_url, callback=self.parse_quotations)
  
      def parse_quotations(self, response):  # parse insurance quotation website link with selenium
          import filldata2
  </code></pre>
  
  <p>and then I have a Selenium <strong>filldata2.py</strong> module which tries to get a quotation for a car from a url link which is extracted in the parse_car method from the scrapy spider code above.</p>
  
  <p>Now the selenum module starts like this : </p>
  
  <pre><code>from selenium import webdriver
  
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait  # available since 2.4.0
  from selenium.webdriver.support import expected_conditions as EC  # available since 2.26.0
  from selenium.webdriver.common.keys import Keys
  import time
  import six
  from six.moves.configparser import SafeConfigParser
  
  regno = 'AA00AAA'
  mile = '15000'
  firstname = 'John'
  lastname = 'Smith'
  [...]
  
  def yesno(idul):
      idxpath = '//*[@id="{}"]'.format(idul)
      return idxpath
  
  
  def findid(idul):
      found = driver.find_element_by_id(idul)
      return found
  
  
  def clickyes(idul):
      idxpath = '//*[@id="{}"]'.format(idul)
      arg = '{}//span[contains(text(), "Yes")]'.format(idxpath)
      return driver.find_element_by_xpath(arg).click()
  
  
  def clickno(idul):
      idxpath = '//*[@id="{}"]'.format(idul)
      arg = '{}//span[contains(text(), "No")]'.format(idxpath)
      return driver.find_element_by_xpath(arg).click()
  
  
  def clickspan(idul):
      idxpath = '//*[@id="{}"]'.format(idul)
      arg = '{}//span[1]'.format(idxpath)
      driver.find_element_by_xpath(arg).click()
  
  
  class DivSelect(object):
      def __init__(self, idul, divtext):
          self.idul = idul
          self.divtext = divtext
          # exemplu: '//div[contains(text(), "Right Hand")]'
          # self.divulxpath = '//div[contains(text(), "{}")]'.format(self.divtext)
          self.idxpath = '//*[@id="{}"]'.format(self.idul)
  
      def findid(self):
          el = 'driver.find_element_by_id({})'.format(self.idul)
          return el
  
      @property
      def clicky(self):  # merge doar la selectare de divuri
          if len(str(self.divtext)) &gt;= 2 and not self.divtext.isdigit():
              arg = '{}//div[contains(text(), "{}")]'.format(self.idxpath, self.divtext)
          else:
              arg = '{}//div[{}]/label/div'.format(self.idxpath, self.divtext)
              print('driver.find_element_by_xpath("{}").click()'.format(arg))
          driver.find_element_by_xpath(arg).click()
  
  
  def printval(cee, cssid):
      def getval():
          val = driver.find_element_by_xpath('//*[@id="{}"]'.format(cssid)).get_attribute('value')
          if not val:
              val = input('Care e valoarea masinii:\n')
          driver.find_element_by_xpath('//*[@id="{}"]'.format(cssid)).click()
          fillin(cssid, val)
          time.sleep(2)
          # print(val)
          # assert isinstance(val, object)
          return val
  
      valoare = getval()
      if valoare.lower() == 'pret':
          print('{} estimat este : £ {} '.format(cee, valoare)) if valoare else 'Nu era nici un {}({}) estimat'.format(
              cee, cssid)
      else:
          print('{} estimat/a/e este : {} '.format(cee, valoare)) if valoare else 'Nu era nici un {}({}) estimat'.format(
              cee, cssid)
  
  
  def clickbutton(cssid):
      driver.find_element_by_xpath('//*[@id="{}"]'.format(cssid)).click()
  
  
  def fillin(cssid, var):
      return driver.find_element_by_id(str(cssid)).send_keys(var)
  
  
  def fillinsugestionbox(cssid, var):
      driver.find_element_by_id(str(cssid)).send_keys(var)
      return driver.find_element_by_xpath('//*[@id=\"{0}\"]'.format(cssid)).send_keys(Keys.RETURN)
  
  
  knowsRegistrationNumber = Yesno('knows-registration-number').clickyes
  
  # 1.2 Then please enter it here to get started:
  registrationNumber = driver.find_element_by_id('registration-number')
  registrationNumber.send_keys(regno)
  
  # 1.3 Find your vehicle find-vehicle-by-reg
  findVehicleByReg = driver.find_element_by_id('find-vehicle-by-reg')
  findVehicleByReg.click()
  time.sleep(1)
  
  # TODO : if no other variants
  # 1.3.1 multiple-vehicles-section : a select list with more options
  # multipleVehiclesSection = driver.find_element_by_id('multiple-vehicles-section')
  # multipleVehiclesSection.click()
  #     possible-vehicles : the select list id
  try:
      element = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "possible-vehicles")))
      possibleVehicles = driver.find_element_by_id('possible-vehicles')
      possibleVehicles.click()
      print('am asteptat destul')
      dropdown = possibleVehicles.find_elements_by_tag_name('option')
      print('Am selectat :\n      {} \n dintre urmatoarele:'.format(dropdown[1].text))
      for option in dropdown[1:]:
          print(option.text)
  
      if dropdown:
          dropdown[1].click()
  except:
      print('Elementul possible-vehicles nu e prezent')
  # finally:
  
  time.sleep(2)
  
  # //*[@id="has-tracker"] Yes/No
  hasTracker = Yesno('has-tracker').clickno
  
  # //*[@id="imported"] Yes/No
  imported = Yesno('imported').clickno
  
  # //*[@id="steering"] - 2 Divs
  # Choose from options :
  # Left Hand or # Right Hand
  steering = DivSelect('steering', 'Right Hand').clicky
  
  # TODO: vezi ce faci daca nu are pret setat. Pune tu unul
  # //*[@id="current-value"] - citeste valoarea
  # driver.find_element_by_xpath('//*[@id="current-value"]')
  
  printval('Pret', 'current-value')
  # print('Pretul estimat este : £ {} '.format(currentValue)) if  currentValue else 'Nu era nici un pret estimat'
  
  printval('scaune', 'numberOfSeats-dropdown')
  
  # //*[@id="has-modifications"]
  hasModifications = Yesno('has-modifications').clickno
  
  # clik next button
  # //*[@id="vehicle-lookup-next"]
  clickbutton('vehicle-lookup-next')
  time.sleep(1)
  # ============================================
  # 2. Vehicle usage                           |
  # ============================================
  # 2.1 When did you buy the car?
  # //*[@id="vehicle-usage"]//span[1]
  vehicleUsage = Yesno('vehicle-usage').clickspan  # I haven't bought this car yet
  
  # 2.2 What do you use your car for?
  # //*[@id="use-of-vehicle"]/ol/li[2]/div[2]/label/div/div[2]
  # //*[@id="use-of-vehicle"]//div[2]
  useOfVehicle = DivSelect('use-of-vehicle', '2').clicky  # Social, Domestic, Pleasure and Commuting (SDPC)
  
  # 2.3 What would you say your annual personal mileage is?
  # //*[@id="annual-mileage"]
  annualMileage = driver.find_element_by_id('annual-mileage')
  annualMileage.send_keys(mile)
  [...much more...]
  ...
  ...
  
  fillin('email', email)
  # Main telephone number
  # Let the insurance providers answer your queries
  # Let us keep you up to date
  # //*[@id="communication-options"]/ol/li[2]/div[4]/label/div/div[2]
  DivSelect('communication-options', 'Post').clicky
  # Please tick this box to confirm you have read and understood our website Terms and Conditions, \
  #   any assumptions we may have made and Your Rewards Terms and Conditions. \
  #   If you do not understand any items within this document please contact us.
  # //*[@id="contact-details"]/div/ol/li[6]/ol/li[2]/div[2]/label/span
  # Yesno('contact-details').clickspan - nu merge
  driver.find_element_by_xpath('//*[@id="contact-details"]/div/ol/li[6]/ol/li[2]/div[2]/label/span').click()
  # //*[@id="contact-details-next"]
  clickbutton('contact-details-next')
  driver.implicitly_wait(10)
  try:
      element = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.ID, "quotes")))
      print('element = ', element)
      try:
          """ 
          wait for loading bar to go away:
          """
          element2 = WebDriverWait(driver, 60).until(EC.invisibility_of_element_located((By.XPATH, '//*[@id="quotes-loading-container"]/div/div[1]')))
          print('element2 = ', element2)
      except:
          print('bara de loading inca este activa. butonul more details cu cotatii nu e vizibil')
  except:
      print('tabelul cu cotatii nu e vizibil')
  
  source_code = driver.find_element_by_id('quotes').get_attribute('innerHTML')
  # element.get_attribute('innerHTML')
  f = open('C:\\Users\\ZZZ\\PycharmProjects\\selenscrapy\\'+str(regno)+'.html', 'wb')
  f.write(source_code.encode('utf-8'))
  f.close()
  </code></pre>
  
  <p>I know the code is messy. I'm a python beginner and I'm playing with this code to scrap some cars from a car selling website and I try to get a insurance quotation for them from a different site. The link for the outer insurance quotation site (full of javascript, that's why i need Selenium webdriver) is a redirection link from the car selling site, because the 2 sites collaborate.
  Now, as I said before, this quotation url needs to be parsed by selenium, which I would like to keep it as on separate module file, maybe even 2 separate files, one with config and one with actions to be taken. </p>
  
  <p>How do I pass the insurance quotation URL obtained from scrapy FirstSpider parse_car() method to selenium module and the response of selenium script (which is called source_code in the second module above) back to scrapy in the FirstSpider parse_quotations() method.</p>
  
  <p>Thank you !</p>