python html parsing
Question: I have the following problem:

I would like to parse html files and get links from the html file. I can get
links with the following code:

    
    
    class MyHTMLParser(HTMLParser):
        links=[]
        def __init__(self,url):
            HTMLParser.__init__(self)
            self.url = url
    
        def handle_starttag(self, tag, attrs):
            try: 
                if tag == 'a':
                    for name, value in attrs:
                        if name == 'href':
                            if value[:5]=="http:":
                                self.links.append(value)
            except: 
                pass
    

But I dont want to get audio files, video files, etc. I only want to get html
links. How can I do that?


Answer: > I can check link ending and if it is particular format I can avoid appending
> that link to the list. Is there other way?

You could look at the `'Content-Type'` header:

    
    
    import urllib2
    url = 'http://stackoverflow.com/questions/13431060/python-html-parsing'
    req = urllib2.Request(url)
    req.get_method = lambda : 'HEAD'    
    response = urllib2.urlopen(req)
    content_type = response.headers.getheader('Content-Type')
    print(content_type)
    

yields

    
    
    text/html; charset=utf-8
    

* * *

Many thanks to @JonClements for `req.get_method = lambda : 'HEAD'`. More info
on this and alternate methods for sending a HEAD request can be found
[here](http://stackoverflow.com/q/107405/190597).

