title: HTTP Error 404: Not Found - BeautifulSoup and Python
body: <p>I have a script to scrape a site but I keep getting an "urllib.error.HTTPError: HTTP Error 404: Not Found".  I have tried adding the user agent to the header and running the script and I still get the same error.  Here is my code </p>
  
  <pre><code>from urllib.request import urlopen, Request
  from bs4 import BeautifulSoup as soup
  import json
  
  atd_url = 'https://courses.lumenlearning.com/catalog/achievingthedream'
  
  #opening up connection and grabbing page
  res = Request(atd_url,headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36'})
  uClient = urlopen(res)
  page_html = uClient.read()
  uClient.close()
  
  #html parsing
  page_soup = soup(page_html, "html.parser")
  
  #grabs info for each textbook
  containers = page_soup.findAll("div",{"class":"book-info"})
  
  data = []
  for container in containers:
     item = {}
     item['type'] = "Course"
     item['title'] = container.h2.text
     item['author'] = container.p.text
     item['link'] = container.p.a["href"]
     item['source'] = "Achieving the Dream Courses"
     item['base_url'] = "https://courses.lumenlearning.com/catalog/achievingthedream"
     data.append(item) # add the item to the list
  
  with open("./json/atd-lumen.json", "w") as writeJSON:
     json.dump(data, writeJSON, ensure_ascii=False)
  </code></pre>
  
  <p>Here is the full error message I get every time I run the script</p>
  
  <pre><code>Traceback (most recent call last):
  File "atd-lumen.py", line 9, in &lt;module&gt;
  uClient = urlopen(res)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 223, in urlopen
  return opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 532, in open
  response = meth(req, response)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 642, in http_response
  'http', request, response, code, msg, hdrs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 570, in error
  return self._call_chain(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain
  result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 650, in http_error_default
  raise HTTPError(req.full_url, code, msg, hdrs, fp)
  urllib.error.HTTPError: HTTP Error 404: Not Found
  </code></pre>
  
  <p>Any suggestions on how to fix this issue?  It is a valid link when entered into a browser. </p>