title: Beautiful Soup fails to acquire prices from amazon at times
body: <p>In running a beautiful soup script, to acquire prices from amazon. I've stumbled across a problem where beautiful soup fails to acquire the prices at random often, in the form of empty lists in output.</p>
  <pre class="lang-py prettyprint-override"><code>def getAmazonPrice(productUrl):
      elems = []
      while elems == None or elems == []:
          headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36'}  # to make the server think its a web browser and not a bot
          res = requests.get(productUrl, headers=headers)
          res.raise_for_status()
  
          soup = bs4.BeautifulSoup(res.text, 'lxml')
          elems = soup.select('#mediaNoAccordion &gt; div.a-row &gt; div.a-column.a-span4.a-text-right.a-span-last &gt; span.a-size-medium.a-color-price.header-price')
          print(elems)
      return elems[0].text.strip()
  
  
  price = getAmazonPrice('https://www.amazon.com/Automate-Boring-Stuff-Python-2nd-ebook/dp/B07VSXS4NK/ref=sr_1_1?crid=30NW5VCV06ZMP&amp;dchild=1&amp;keywords=automate+the+boring+stuff+with+python&amp;qid=1586810720&amp;sprefix=automate+the+bo%2Caps%2C288&amp;sr=8-1')
  print('The price is ' + price)
  
  </code></pre>
  <p>Output:</p>
  <pre><code>[]
  []
  []
  []
  []
  []
  []
  []
  []
  []
  []
  []
  []
  []
  []
  []
  []
  []
  []
  [&lt;span class=&quot;a-size-medium a-color-price header-price&quot;&gt;
  
  
                          $26.58
  
  
  
          &lt;/span&gt;]
  The price is $26.58
  </code></pre>
  <p>I feel like the issue stems from amazon blocking my scraping tool.
  How would you use Beautiful Soup to scrape amazon?</p>