Python BeautifulSoup Extract specific URLs
Question: Is it possible to get only specific URLs?

Like:

    
    
    <a href="http://www.iwashere.com/washere.html">next</a>
    <span class="class">...</span>
    <a href="http://www.heelo.com/hello.html">next</a>
    <span class="class">...</span>
    <a href="http://www.iwashere.com/wasnot.html">next</a>
    <span class="class">...</span>
    

Output should be only URLs from `http://www.iwashere.com/`

like, output URLs:

    
    
    http://www.iwashere.com/washere.html
    http://www.iwashere.com/wasnot.html
    

I did it by string logic. Is there any direct method using BeautifulSoup?


Answer: You can match multiple aspects, including using a regular expression for the
attribute value:

    
    
    import re
    soup.find_all('a', href=re.compile('http://www\.iwashere\.com/'))
    

which matches (for your example):

    
    
    [<a href="http://www.iwashere.com/washere.html">next</a>, <a href="http://www.iwashere.com/wasnot.html">next</a>]
    

so any `<a>` tag with a `href` attribute that has a value that starts with the
string `http://www.iwashere.com/`.

You can loop over the results and pick out just the `href` attribute:

    
    
    >>> for elem in soup.find_all('a', href=re.compile('http://www\.iwashere\.com/')):
    ...     print elem['href']
    ... 
    http://www.iwashere.com/washere.html
    http://www.iwashere.com/wasnot.html
    

To match all relative paths instead, use a negative look-ahead assertion that
tests if the value does _not_ start with a schem (e.g. `http:` or `mailto:`),
or a double slash (`//hostname/path`); any such value _must_ be a relative
path instead:

    
    
    soup.find_all('a', href=re.compile(r'^(?!(?:[a-zA-Z][a-zA-Z0-9+.-]*:|//))'))
    

