Webscraping from directory of HTML files using BS4 and python
Question: I have a website in which each person's details are stored in separate .HTML
file. So there are totally 100 person whose details are stored in 100
different .html files. But all have same HTML structure.

Here is the website link <http://www.coimbatore.com/doctors/home.htm>.

So if you see this website there are many categories and the `~all-
doctors.html~` files are in same directory.

<http://www.coimbatore.com/doctors/cardiology.htm>

has 5 doctors' link. If I click on any doctors name it will take to

<http://www.coimbatore.com/doctors/>**thatdoctorname**.htm. So all the files
are in the same directory /doctors/ If I am not wrong. So how do I scrape the
details of each doctor ?

I was planning to `wget` all the files from that
<http://www.coimbatore.com/doctors/> URL, save locally and merge as one
`whole.html` file using `join` function in LINUX. Is there any better way?

**UPDATE**

    
    
    letters = ['doctor1','doctor2'...]
    for i in range(30):
        try:
            page = urllib2.urlopen("http://www.coimbatore.com/doctors/{}.htm".format(letters[i]))
        except urllib2.HTTPError:
            continue
        else:
    


Answer: This code should get you started.

    
    
    import urllib2
    from bs4 import BeautifulSoup
    
    doctors = ['thomas']
    for doctor in doctors:
        try:
            page = urllib2.urlopen("http://www.coimbatore.com/doctors/{}.htm".format(doctor))
            soup = BeautifulSoup(page)
        except urllib2.HTTPError:
            continue
    
        rows = soup.find("table", cellspacing=0).find_all('tr')
    
        for row in rows:
            cols = row.find_all('td')
            print "%s: %s" % (cols[0].get_text().replace('\n', ' '), cols[1].get_text().replace('\n', ' '))
    

It has an output of

    
    
    Name of Doctor: Dr.Thomas Alexander
    Qualification: M.D (Internal Medicine), D.M. (Cardiology)
    Fellowship & Membership: Fellow of Indian College of Cardiology Associate Fellow
     of American College of Cardiology
    Address of Clinic / Visiting Hospitals: Kovai Medical Center and Hospital, P.B.N
    o.3209, Avanashi Road, Coimbatore-641 014
    Telephone Number: +91-422-827784
    Consulting Hours:  8am - 5pm
    Specialist in: Senior Consultant and Interventional Cardiologist
    

A few notes that you may wish to deal with differently. I replaced all
newlines (`\n`) with spaces because the code has weird line breaks like so:

    
    
    <td><b><font face="Arial,Helvetica"><font color="#0000FF"><font size=-1>Name
    of Doctor</font></font></font></b></td>
    

Notice that it forces the break between `Name` and `of`.

If you are attempting to make a CSV out of this, the script can be easily
modified to pull only the second cell on each row.

