title: How can I programmatically detect if a site prohibits screen scraping?
body: <p>I've written a small program that find all links on a web page and dumps them into a list, then loops through that list, opens each link and collects all links on those pages. It does this n layers deep. </p>
  
  <p>I'm using Beautiful Soup to screen scrape. One of the problems I've run into is that some pages don't allow screen scraping and crash the program. I've written a snippet of code that detects if a site doesn't allow scraping by looking for if the title tag of the page says "Access Denied". Sites that have this use CloudFare to block screen scraping.</p>
  
  <pre><code>&gt;     links = []
  &gt;     for link in links:
  &gt;             if not nltk.clean_html(str(connect_to_webpage(link).find_all('title')))[2:15]
  &gt; == 'Access denied': #check link title tag to see if site says 'Access Denied'
  &gt;                 links.append(link)
  </code></pre>
  
  <p>Apparently not everyone uses Cloud Fare though, so there are some instances it doesn't catch and the program crashes. Is there a universal condition I can add that will catch all instances of screen scraper blocking? What is the best way to go about this? Thanks.</p>