Unable to identify link class
Question: I am very new to Programming and Python and am trying out to code this simple
scraper to extract all the profile URLs of therapists from this page

[http://www.therapy-
directory.org.uk/search.php?search=Sheffield&services[23]=1&business_type[individual]=1&distance=40&uqs=626693](http://www.therapy-
directory.org.uk/search.php?search=Sheffield&services\[23\]=1&business_type\[individual\]=1&distance=40&uqs=626693)

    
    
    import requests
    from bs4 import BeautifulSoup
    
    def tru_crawler(max_pages):
       p = '&page='
       page = 1
       while page <= max_pages:
         url = 'http://www.therapy-directory.org.uk/search.php?search=Sheffield&distance=40&services[23]=on&services=23&business_type[individual]=on&uqs=626693' + p + str(page)
         code = requests.get(url)
         text = code.text
         soup = BeautifulSoup(text)
         for link in soup.findAll('a',{'member-summary':'h2'}):
            href = 'http://www.therapy-directory.org.uk' + link.get('href')
            yield href + '\n'
            print(href)
        page += 1
    

Now when I am running this code, I am getting nothing, primarily because the
soup.findall is empty.

The HTML of the profile link shows

    
    
    <div class="member-summary">
    <h2 class="">
     <a href="/therapists/julia-church?uqs=626693">Julia Church</a>
    </h2>
    

So I am not sure what additional parameters to pass in soup.findall('a') in
order to get the Profile URLs

Please help

Thanks

**Update -**

I ran the revised code and it returned a bunch of errors

Okay this time after it scraped page 1 it returned a bunch of errors

    
    
    Traceback (most recent call last):
    File "C:/Users/PB/PycharmProjects/crawler/crawler-revised.py", line    19,      enter code here`in <module>
    tru_crawler(3)
    File "C:/Users/PB/PycharmProjects/crawler/crawler-revised.py", line 9, in tru_crawler
    code = requests.get(url)
    File "C:\Python27\lib\requests\api.py", line 68, in get
    return request('get', url, **kwargs)
    File "C:\Python27\lib\requests\api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
    File "C:\Python27\lib\requests\sessions.py", line 464, in request
    resp = self.send(prep, **send_kwargs)
    File "C:\Python27\lib\requests\sessions.py", line 602, in send
    history = [resp for resp in gen] if allow_redirects else []
    File "C:\Python27\lib\requests\sessions.py", line 195, in resolve_redirects
    allow_redirects=False,
    File "C:\Python27\lib\requests\sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
    File "C:\Python27\lib\requests\adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
    requests.exceptions.ConnectionError: ('Connection aborted.',  BadStatusLine("''",))
    

What's going wrong here?


Answer: Currently, parameter of `findAll()` you have doesn't make sense. It reads:
find all `<a>` having `member-class` attribute equals "h2".

One possible way is using `select()` method passing [CSS
selector](http://www.crummy.com/software/BeautifulSoup/bs4/doc/#css-selectors)
as parameter :

    
    
    for link in soup.select('div.member-summary h2 a'):
        href = 'http://www.therapy-directory.org.uk' + link.get('href')
        yield href + '\n'
        print(href)
    

Above CSS selector reads: find `<div>` tag having class equals "member-
summary", then within that `<div>` find `<h2>` tag, then within that `<h2>`
find `<a>` tag.

Working example:

    
    
    import requests
    from bs4 import BeautifulSoup
    
    p = '&page='
    page = 1
    url = 'http://www.therapy-directory.org.uk/search.php?search=Sheffield&distance=40&services[23]=on&services=23&business_type[individual]=on&uqs=626693' + p + str(page)
    code = requests.get(url)
    text = code.text
    soup = BeautifulSoup(text)
    for link in soup.select('div.member-summary h2 a'):
        href = 'http://www.therapy-directory.org.uk' + link.get('href')
        print(href)
    

Output (trimmed, from total 26 links) :

    
    
    http://www.therapy-directory.org.uk/therapists/lesley-lister?uqs=626693
    http://www.therapy-directory.org.uk/therapists/fiona-jeffrey?uqs=626693
    http://www.therapy-directory.org.uk/therapists/ann-grant?uqs=626693
    .....
    .....
    http://www.therapy-directory.org.uk/therapists/jan-garbutt?uqs=626693
    

