title: Retrieve all URLs to web pages on the same domain, but no media or anchors
body: <p>I'm new to Python, and extremely impressed by the amount of libraries at my disposal. I have a function already that uses Beautiful Soup to extract URLs from a site, but not all of them are relevant. I only want webpages (no media) on the same website (domain or subdomain, but no other domains). I'm trying to manually program around examples I run into, but I feel like I'm reinventing the wheel - surely this is a common problem in internet applications.</p>
  
  <p>Here's an example list of URLs that I might retrieve from a website, say <a href="http://example.com" rel="nofollow">http://example.com</a>, with markings for whether or not I want them and why. Hopefully this illustrates the issue.</p>
  
  <p>Good:</p>
  
  <ul>
  <li><code>example.com/page</code> - it links to another page on the same domain</li>
  <li><code>example.com/page.html</code> - has an filetype ending but it's an HTML page</li>
  <li><code>subdomain.example.com/page.html</code> - it's on the same site, though on a subdomain</li>
  <li><code>/about/us</code> - it's a relative link, so it doesn't have the domain it it, but it's implied</li>
  </ul>
  
  <p>Bad:</p>
  
  <ul>
  <li><code>otherexample.com/page</code> - bad, the domain doesn't match</li>
  <li><code>example.com/image.jpg</code> - bad, not an image and not a page</li>
  <li><code>/</code> - bad - sometimes there's just a slash in the "a" tag, but that's a reference to the page I'm already on</li>
  <li><code>#anchor</code> - this is also a relative link, but it's on the same page, so there's no need for it</li>
  </ul>
  
  <p>I've been writing cases in <code>if</code> statements for each of these...but there has to be a better way!</p>
  
  <hr>
  
  <p>Edit: Here's my current code, which returns nothing:</p>
  
  <pre><code>ignore_values = {"", "/"}
  def desired_links(href):
       # ignore if href is not set
       if not href:
           return False
  
       # ignore if it is just a link to the same page
       if href.startswith("#"):
           return False
  
       # skip ignored values
       if href in ignore_values:
           return False
  
  
   def explorePage(pageURL):
   #Get web page
       opener = urllib2.build_opener()
       opener.addheaders = [('User-Agent', 'Mozilla/5.0')]
       response = opener.open(pageURL)
       html = response.read()
  
       #Parse web page for links
       soup = BeautifulSoup(html, 'html.parser')
       links = [a["href"] for a in soup.find_all("a", href=desired_links)]
       for link in links:
           print(link)
  
       return
  
  
  
   def main():
       explorePage("http://xkcd.com")
  </code></pre>