BeautifulSoup is not scraping ALL anchor tags in a page with .find_all('a'). Am I overlooking something?
Question: Okay, so in a terminal, after importing and making the necessary objects--I
typed:

    
    
    for links in soup.find_all('a'):
         print(links.get('href'))
    

which gave me all the links on a wikipedia page (roughly 250). No problems.

However, in a program I am coding, I only receive about 60 links (and this is
scraping the same wikipedia page) and the ones I DO get are mostly not worth
anything. I double checked that I initialized both exactly the same--the only
difference is the names of variables. For reference, here is the function
where I setup the BS4 object, and grab the desired page:

    
    
    def get_site(hyperLink):
        userSite = urllib3.PoolManager()
        siteData = userSite.request("GET", hyperLink)
        bsd = BeautifulSoup(siteData.data)
        return bsd
    

Later, I grab the elements and append them to a list I will then manipulate:

    
    
    def find_urls(bsd, urls, currentNetloc):
        for links in bsd.find_all('a'):
            urls.append(links.get('href'))
        return urls
    

Other relevant info:

  * I am using Python 3.3
  * I am using urllib3, BeautifulSoup 4, and urlparse (from urllib)
  * I am working in PyCharm (for the actual program)
  * Using Lubuntu, if it matters.

After running a command line instance of python3 and importing "sys" I typed
and received:

    
    
    $ sys.executable
    '/usr/bin/python3'
    $ sys.path
    ['', '/usr/local/lib/python3.3/dist-packages/setuptools-1.1.5-py3.3.egg', '/usr/local/lib/python3.3/dist-packages/pip-1.4.1-py3.3.egg', '/usr/local/lib/python3.3/dist-packages/beautifulsoup4-4.3.2-py3.3.egg', '/usr/lib/python3.3', '/usr/lib/python3.3/plat-i386-linux-gnu', '/usr/lib/python3.3/lib-dynload', '/usr/local/lib/python3.3/dist-packages', '/usr/lib/python3/dist-packages']
    

After running these commands in a Pycharm project, I received exactly the same
results, with the exception that the directories containing my pycharm
projects were included in the list.


Answer: This is not my answer. I got it from
[here](http://fryandata.wordpress.com/2014/06/17/24/), which has helped me
before.

    
    
        from bs4 import BeautifulSoup
        import csv
    
        # Create .csv file with headers
        f=csv.writer(open("nyccMeetings.csv","w"))
        f.writerow(["Name", "Date", "Time", "Location", "Topic"])
    
        # Use python html parser to avoid truncation
        htmlContent = open("nyccMeetings.html")
        soup = BeautifulSoup(htmlContent,"html.parser")
    
        # Find each row
        rows = soup.find_all('tr')
        for tr in rows:
            cols = tr.find_all('td') # Find each column
            try:
                names = cols[0].get_text().encode('utf-8')
                date = cols[1].get_text().encode('utf-8')
                time = cols[2].get_text().encode('utf-8')
                location = cols[3].get_text().encode('utf-8')
                topic = cols[4].get_text().encode('utf-8')
           except:
                continue
           # Write to .csv file
           f.writerow([names, date, time, location, topic])
    

I think it would be useful to note some of the troubles I ran into while
writing this script:

Specify your parser. It is very important to specify the type of html parser
that BeautifulSoup will use to parse through the html tree form. The html file
that I read into Python was not formatted correctly so BeautifulSoup truncated
the html and I was only able to access about a quarter of the records. By
telling BeautifulSoup to explicitly use the built-in Python html parser, I was
able to avoid this issue and retrieve all records.

Encode to UTF-8. get_text() had some issues with encoding the text inside the
html tags. As such, I was unable to write data to the comma-delimited file. By
explicitly telling the program to encode to UTF-8, we avoid this issue
altogether.

