Python crawler is not crawling all sites
Question: I'm trying to get links from websites through this code

    
    
    import requests
    from bs4 import BeautifulSoup
    def get_links(max_pages):
        page = 1
        while page <= max_pages:
            address = 'http://hamariweb.com/mobiles/nokia_mobile-phones1.aspx?Page=' + str(page)
            source_code = requests.get(address)
            plain_text = source_code.text
            soup = BeautifulSoup(plain_text)
            for link in soup.findAll('a', {'class': 'TextClass8pt'}):
                href = link.get("href")
                print(href)
    
            page += 1
    
    get_links(3)
    

and it's giving expected output. But when I tried this

    
    
    address = 'http://propakistani.pk/category/cellular/page/' + str(page)
    

for link in `soup.findAll('a', {'class': 'aa_art_hdng'}):`

It's showing this error

> TypeError: getresponse() got an unexpected keyword argument 'buffering'

I also tried another site but that time neighter it showed any output nor any
error. Why it's showing proper output with different sites? Is there any
problem with my code? Please help me. Thanks


Answer: there is **No Tag Match** with this condition `soup.findAll('a', {'class':
'TextClass8pt'})`

Try following

**Demo** :

    
    
    import requests
    from bs4 import BeautifulSoup
    
    def get_links(max_pages):
        page = 1
    
        while page <= max_pages:
            address = 'http://propakistani.pk/category/cellular/page/' + str(page)
            source_code = requests.get(address)
            plain_text = source_code.text
            soup = BeautifulSoup(plain_text)
            for link in soup.findAll('a'):
                href = link.get("href")
                print(href)
    
            page += 1
    
    get_links(3)
    

* * *

**Or**

There are `a` tag with class value `aa_loop_h2a` e.g. `<a class="aa_loop_h2a"
href="http://propakistani.pk/2015/04/20/mobile-data-usage-in-pakistan-
grows-600-during-2014/" title="Mobile Data Usage in Pakistan Grows 600% During
2014">Mobile Data Usage in Pakistan Grows 600% During 2014</a>`

So try with `soup.findAll('a', {'class': 'aa_loop_h2a'})` condition.

