title: Scraping Amazon with Selenium - XPath Problem
body: <p>I'm doing a course project but the data I got from Amazon is missing products' names, prices and categories. Since I don't have AWS account for API, I decided to scrape this info based on ASIN (product ID), which I have. But I don't really know much about web scraping yet (XML structure, for example). The scraping part of the code is adapted from a functional forum scraping project, but it's not working here.</p>
  <p>I also tried BeautifulSoup, that I even found specifically from a similar Amazon project, but it didn't work either. Since Selenium is more versatile, I'd really prefer to learn this way. So, here's the code, with the not functional XPath:</p>
  <pre><code>from selenium import webdriver
  from random import randint
  
  asin_set = ['0151004714', '0380709473','0511189877', '0528881469', '0545105668', '0557348153', '0594033926', '0594296420', '0594450268', '0594451647', '0594459451', '0594481902', '059449771X']
  
  driver = webdriver.Chrome()
  list_of_dicts[:] = []
  print('This is gonna be LEGEN... wait for it:')
  for i in asin_set[:5]:
      url = f'https://www.amazon.com/gp/product/{i}'
      driver.get(url)
      product_info = {}
      product_info['asin'] = i
      try:
          name = driver.find_elements_by_xpath('//*[@id=&quot;' + x + '&quot;]')   #&lt;---
          product_info['name'] = name.text('productTitle')               #&lt;---
      except:
          product_info['name'] = 0
      try:
          price = driver.find_elements_by_xpath('//*[@id=&quot;' + x + '&quot;]')  #&lt;---
          product_info['price'] = price.text                             #&lt;---
      except:
          product_info['price'] = 0
      try:
          category = driver.find_elements_by_xpath('//*[@id=&quot;' + x + '&quot;]/ul/li[5]/span/a')        #&lt;---
          product_info['category'] = category.get_attribute('wayfinding-breadcrumbs_feature_div') #&lt;---
      except:
          product_info['category'] = 0
  
      list_of_dicts.append(product_info)  # Append scrape to dictionary
      print(str(len(list_of_dicts)) + ' . ', end='')   # print the current length of the scrapes
      sleep(randint(1,2))               # Sleep 1 or 2 seconds in bewteen scrapes
  print('DARY!')
  </code></pre>
  <p>The cell runs fine, the browser opens each page. But things are not being correctly accessed or stored, and the result I'm getting of list_of_dicts is this:</p>
  <pre><code>[{'asin': '0151004714', 'name': 0, 'price': 0, 'category': 0},
   {'asin': '0380709473', 'name': 0, 'price': 0, 'category': 0},
   {'asin': '0511189877', 'name': 0, 'price': 0, 'category': 0},
   {'asin': '0528881469', 'name': 0, 'price': 0, 'category': 0},
   {'asin': '0545105668', 'name': 0, 'price': 0, 'category': 0}]
  </code></pre>