title: Passing Selenium opened URL to Scrapy and scraping the data
body: <p>I've been trying to scrape Bioshock games from the steam store and save their name, price and link in a CSV file. I know how to do it just by using Scrapy, but I really want to know if there's a way to do it combining both Scrapy and Selenium. I want to use Selenium just to get rid of the age check gate that pops up on certain game store sites. </p>
  
  <p><a href="https://i.stack.imgur.com/keH0A.png" rel="nofollow noreferrer">Example of an age gate</a></p>
  
  <p><a href="https://i.stack.imgur.com/x0fuM.png" rel="nofollow noreferrer">Example of another age gate</a></p>
  
  <p>So I've managed to scrape games that don't have the age gate by using Scrapy and I've managed to bypass the age gates using Selenium. </p>
  
  <p>The problem I'm having is passing the game store site that Selenium opened by bypassing the age gate to Scrapy so it can crawl it. Since everything works fine on its own I came to the conclusion that the problem is that I don't know how to connect them.</p>
  
  <pre><code>def parse_product(self, response):
      product = ScrapesteamItem()
      sel = self.driver
      #Passing first age gate
      if '/agecheck/app/' in response.url:
          sel.get(response.url)
          select = Select(sel.find_element_by_xpath("""//*[@id="ageYear"]"""))
          select.select_by_visible_text("1900")
          sel.find_element_by_xpath("""//*[@id="agecheck_form"]/a""").click()
          #Pass Selenium newly opened site to Scrapy
      #Passing second age gate
      elif '/agecheck' in response.url:
          sel.get(response.url)
          sel.find_element_by_xpath("""//*[@id="app_agegate"]/div[3]/a[1]""").click()
          #Pass Selenium newly opened site to Scrapy
      #Scraping the data with scrapy
      else:
          name = response.css('.apphub_AppName ::text').extract()
          price = response.css('div.game_purchase_price ::text, div.discount_final_price ::text').extract()
          link = response.css('head &gt; link:nth-child(40) ::attr(href)').extract()
  
          for product in zip(name, price, link):
              scrapedInfo = {
                  'NAME' : product[0],
                  'PRICE' : product[1].rstrip().lstrip(),
                  'LINK' : product[2]
              }
  
              yield scrapedInfo
  </code></pre>
  
  <p>I hope someone will know how to do it (if it's even possible).</p>
  
  <p>P.S. I know there are much better ways to scrape Steam store, I know there's an API probably but before I go and learn that I would like to know if there's a way to do it like this even if it's sub-optimal.</p>