Scrape 3 tables from multiple URLs with beautifulsoup
Question: I am fairly new to web scrapping and also to python. For my bachelor thesis I
need data from rivercruise ships. I was able to write the following code,
which is working on <http://www.cruiseshipschedule.com>.

There is also a second code which I wrote to get all the links for the ships I
am interested in. I kindly ask for help, to combine those two codes in order
to scrape the same 3 tables from all the links. This is my scraper, which
works without any errors.

    
    
        from mechanize import Browser
        from bs4 import BeautifulSoup
        import lxml
        import csv
    
        url1 = 'http://www.cruiseshipschedule.com/ama-waterways/ms-amabella-cruises/'
    
        mech = Browser()
        page1 = mech.open(url1)
        html = page1.read()
        soup1 = BeautifulSoup(html, "lxml")
    
        ship_in = soup1.h1
        ship_in = ship_in.text
        ship = u' '.join(ship_in.split())
        u' '.join(ship_in.split())
        ship = [ship]
    
        h21 = soup1.h2
        h22 = h21.findNext('h2')
        h23 = h22.findNext('h2')
        h24 = h23.findNext('h2')
        h25 = h24.findNext('h2')
        h_y1 = h22.text
        h_y2 = h23.text
        h_y3 = h24.text
        itinerary1_header = u' '.join(h_y1.split())
        u' '.join(h_y1.split())
        itinerary2_header = u' '.join(h_y2.split())
        u' '.join(h_y2.split())
        itinerary3_header = u' '.join(h_y3.split())
        u' '.join(h_y3.split())
    
        table_1 = soup1.findAll('table')[0]
        table_2 = soup1.findAll('table')[1]
        table_3 = soup1.findAll('table')[2]
    
        rows_1 = table_1.findAll("tr")
        rows_2 = table_2.findAll("tr")
        rows_3 = table_3.findAll("tr")
    
    
        for row_1 in rows_1:
                cells_1 = row_1.findAll('td')
    
                list_1 = table_1.findAll('li')
                decks = str(list_1[0].get_text()).split(':')
                cabin = str(list_1[1].get_text()).split(':')
                cabin_number = str(list_1[2].get_text()).split(':')
    
                list_key = ''.join(list(decks[0] + '|' + cabin[0] + '|' + cabin_number[0]))
                list_value = ''.join(list(decks[1] + '|' + cabin[1] + '|' + cabin_number[1]))
                list_key = list_key.split('|')
                list_value = list_value.split('|')
    
                try: #we are using "try" because the table is not well formatted. This allows the program to continue after encountering an error.
                        col1_1 = str(cells_1[0].get_text()).split('\n') # This structure isolate the item by its column in the table and converts it into a string.
                        col2_1 = str(cells_1[1].get_text()).split('\n')
                        col3_1  = str(cells_1[2].get_text()).split('\n')
                        col4_1  = str(cells_1[3].get_text()).split('\n')
                except:
                        continue #This tells the computer to move on to the next item after it encounters an error
    
                keys_1 = ['ship'] + col1_1 + col3_1 + list_key
                values_1 = ship + col2_1 + col4_1 + list_value
                dict_1 = dict(zip(keys_1, values_1))
    
                with open('Z:/Cruiseshipschedule/details/details_'+ ship_in + '.csv', 'wb') as f:  # Just use 'w' mode in 3.x
                        w = csv.DictWriter(f, dict_1.keys())
                        w.writeheader()
                        w.writerow(dict_1)
                        f.close()
    
        list_of_rows_2=[]
        for row_2 in rows_2:
                cells_2 = row_2.find_all("td")        
                list_of_cells_2 = [itinerary1_header]
    
                try: #we are using "try" because the table is not well formatted. This allows the program to continue after encountering an error.
                        date1_2 = str(cells_2[0].get_text()) # This structure isolate the item by its column in the table and converts it into a string.
                        itinerary2_2 = str(cells_2[1].get_text()).split('\n')
                        price3_2 = str(cells_2[2].get_text()).split('\n')
                        list_of_cells_2.append(date1_2)
                        list_of_cells_2.append(itinerary2_2)
                        list_of_cells_2.append(price3_2)
                except:
    
                        continue #This tells the computer to move on to the next item after it encounters an error
                list_of_cells_2.append(ship)
                list_of_rows_2.append(list_of_cells_2)
    
                outfile_2 = open('Z:/Cruiseshipschedule/itinerary1/itinerary1_'+ ship_in + '.csv', "wb")
                writer = csv.writer(outfile_2, delimiter='|')
                writer.writerows(list_of_rows_2)      
                outfile_2.close()
    
    
    
        list_of_rows_3=[]
        for row_3 in rows_3:
                cells_3 = row_3.find_all("td")        
                list_of_cells_3 = [itinerary2_header]
    
                try: #we are using "try" because the table is not well formatted. This allows the program to continue after encountering an error.
                        date1_3 = str(cells_3[0].get_text()) # This structure isolate the item by its column in the table and converts it into a string.
                        itinerary2_3 = str(cells_3[1].get_text()).split('\n')
                        price3_3 = str(cells_3[2].get_text()).split('\n')
                        list_of_cells_3.append(date1_3)
                        list_of_cells_3.append(itinerary2_3)
                        list_of_cells_3.append(price3_3)
                except:
    
                        continue #This tells the computer to move on to the next item after it encounters an error
                list_of_cells_3.append(ship)
                list_of_rows_3.append(list_of_cells_3)
    
                outfile_3 = open('Z:/Cruiseshipschedule/itinerary2/itinerary2_'+ ship_in + '.csv', "wb")
                writer = csv.writer(outfile_3, delimiter='|')
                writer.writerows(list_of_rows_3)
                outfile_3.close()        
    
        print "check out the data!"
    

Here is the second code - which also works and gets all links and stores them
in the `page_array` variable.

    
    
       from mechanize import Browser
       from bs4 import BeautifulSoup
       import lxml
       from lxml import html
       import csv
       import requests
    
    
       page_array = []
    
       mech = Browser()
       url = 'http://www.cruiseshipschedule.com/'
       page = mech.open(url)
       html = page.read()
       soup = BeautifulSoup(html, "lxml")
       table1 = soup.table #Ocean Cruise
       table2 = table1.findNext('table')#River Cruise
       pages = table2.findAll('a')
    
    
       for page in pages:
             page_array.append(page.get('href').replace('http://www.cruiseshipschedule.com/', ''))
    

What is a proper python method to combine those codes and get all the data I
need. Your help would be appreciated.

Edit:

    
    
    from mechanize import Browser
    from bs4 import BeautifulSoup
    import lxml
    from lxml import html
    import csv
    import requests
    
    
    page_array = []
    
    mech = Browser()
    url = 'http://www.cruiseshipschedule.com/'
    page = mech.open(url)
    html = page.read()
    soup = BeautifulSoup(html, "lxml")
    table1 = soup.table #Ocean Cruise
    table2 = table1.findNext('table')#River Cruise
    pages = table2.findAll('a')
    
    
    for page in pages:
         page_array.append(page.get('href'))
    
    
    for page in page_array:
    
        mech = Browser()
        page1 = mech.open(page)
        html = page1.read()
        soup1 = BeautifulSoup(html, "lxml")
    
        ship_in = soup1.h1
        ship_in = ship_in.text
        ship = u' '.join(ship_in.split())
        u' '.join(ship_in.split())
        ship = [ship]
    
        h21 = soup1.h2
        h22 = h21.findNext('h2')
        h23 = h22.findNext('h2')
        h24 = h23.findNext('h2')
        h25 = h24.findNext('h2')
        h_y1 = h22.text
        h_y2 = h23.text
        h_y3 = h24.text
        itinerary1_header = u' '.join(h_y1.split())
        u' '.join(h_y1.split())
        itinerary2_header = u' '.join(h_y2.split())
        u' '.join(h_y2.split())
        itinerary3_header = u' '.join(h_y3.split())
        u' '.join(h_y3.split())
    
        table_1 = soup1.findAll('table')[0]
        table_2 = soup1.findAll('table')[1]
        table_3 = soup1.findAll('table')[2]
    
        rows_1 = table_1.findAll("tr")
        rows_2 = table_2.findAll("tr")
        rows_3 = table_3.findAll("tr")
    
    
        for row_1 in rows_1:
                cells_1 = row_1.findAll('td')
    
                list_1 = table_1.findAll('li')
                decks = str(list_1[0].get_text()).split(':')
                cabin = str(list_1[1].get_text()).split(':')
                cabin_number = str(list_1[2].get_text()).split(':')
    
                list_key = ''.join(list(decks[0] + '|' + cabin[0] + '|' + cabin_number[0]))
                list_value = ''.join(list(decks[1] + '|' + cabin[1] + '|' + cabin_number[1]))
                list_key = list_key.split('|')
                list_value = list_value.split('|')
    
                try: #we are using "try" because the table is not well formatted. This allows the program to continue after encountering an error.
                        col1_1 = str(cells_1[0].get_text()).split('\n') # This structure isolate the item by its column in the table and converts it into a string.
                        col2_1 = str(cells_1[1].get_text()).split('\n')
                        col3_1  = str(cells_1[2].get_text()).split('\n')
                        col4_1  = str(cells_1[3].get_text()).split('\n')
                except:
                        continue #This tells the computer to move on to the next item after it encounters an error
    
                keys_1 = ['ship'] + col1_1 + col3_1 + list_key
                values_1 = ship + col2_1 + col4_1 + list_value
                dict_1 = dict(zip(keys_1, values_1))
    
                with open('Z:/Cruiseshipschedule/details/details_'+ ship_in + '.csv', 'wb') as f:  # Just use 'w' mode in 3.x
                        w = csv.DictWriter(f, dict_1.keys())
                        w.writeheader()
                        w.writerow(dict_1)
                        f.close()
    
        list_of_rows_2=[]
        for row_2 in rows_2:
                cells_2 = row_2.find_all("td")        
                list_of_cells_2 = [itinerary1_header]
    
                try: #we are using "try" because the table is not well formatted. This allows the program to continue after encountering an error.
                        date1_2 = str(cells_2[0].get_text()) # This structure isolate the item by its column in the table and converts it into a string.
                        itinerary2_2 = str(cells_2[1].get_text()).split('\n')
                        price3_2 = str(cells_2[2].get_text()).split('\n')
                        list_of_cells_2.append(date1_2)
                        list_of_cells_2.append(itinerary2_2)
                        list_of_cells_2.append(price3_2)
                except:
    
                        continue #This tells the computer to move on to the next item after it encounters an error
                list_of_cells_2.append(ship)
                list_of_rows_2.append(list_of_cells_2)
    
                outfile_2 = open('Z:/Cruiseshipschedule/itinerary1/itinerary1_'+ ship_in + '.csv', "wb")
                writer = csv.writer(outfile_2, delimiter='|')
                writer.writerows(list_of_rows_2)      
                outfile_2.close()
    
    
    
        list_of_rows_3=[]
        for row_3 in rows_3:
                cells_3 = row_3.find_all("td")        
                list_of_cells_3 = [itinerary2_header]
    
                try: #we are using "try" because the table is not well formatted. This allows the program to continue after encountering an error.
                        date1_3 = str(cells_3[0].get_text()) # This structure isolate the item by its column in the table and converts it into a string.
                        itinerary2_3 = str(cells_3[1].get_text()).split('\n')
                        price3_3 = str(cells_3[2].get_text()).split('\n')
                        list_of_cells_3.append(date1_3)
                        list_of_cells_3.append(itinerary2_3)
                        list_of_cells_3.append(price3_3)
                except:
    
                        continue #This tells the computer to move on to the next item after it encounters an error
                list_of_cells_3.append(ship)
                list_of_rows_3.append(list_of_cells_3)
    
                outfile_3 = open('Z:/Cruiseshipschedule/itinerary2/itinerary2_'+ ship_in + '.csv', "wb")
                writer = csv.writer(outfile_3, delimiter='|')
                writer.writerows(list_of_rows_3)
                outfile_3.close()        
    
        print "check out the data!"
    

This is now my edited version. The output are 3 .csv files. And then it throws
an error:

check out the data!

Traceback (most recent call last): File "C:/Python27/ship scraper editedt.py",
line 55, in table_3 = soup1.findAll('table')[2] IndexError: list index out of
range

However I do not get the error, when running only a single url (url1 =
'<http://www.cruiseshipschedule.com/ama-waterways/ms-amabella-cruises/>').

**Edit:**

    
    
       from mechanize import Browser
       from bs4 import BeautifulSoup
       import lxml
       from lxml import html
       import csv
       import requests
    
       base_url = 'http://www.cruiseshipschedule.com/'
    
       def get_links():
              links_array = []
              mech = Browser()
              mech.set_handle_robots(False)
              page = mech.open(base_url)
              html = page.read()
              soup = BeautifulSoup(html, "lxml")
              tables = soup.findAll('table') 
    
              for table in tables:
                  links = tables[1].findAll('a')
    
              for link in links:
                    links_array.append(link.get('href').replace('http://www.cruiseshipschedule.com/', ''))
    
              return links_array
    
    
       def get_headings(url):
           mech = Browser()
           mech.set_handle_robots(False)
           page = mech.open(url)
           html = page.read()
           soup = BeautifulSoup(html, "lxml")
           headings = soup.findAll('h2') 
           return headings
    
    
       get_links()
       urls = [base_url + link for link in get_links()]
    
       for url in urls:
           mech = Browser()
           mech.set_handle_robots(False)
    
           try:
               page = mech.open(url)
           except:
               continue
           html = page.read()
           soup = BeautifulSoup(html, "lxml")
    
           tables = soup.findAll('table')
    
           ship_in = soup.h1
           ship_in = ship_in.text
           ship = u' '.join(ship_in.split())
           u' '.join(ship_in.split())
           ship = [ship]
           try:
               details = tables[0]
           except:
               continue
           rows_1 = details.findAll("tr")  
    
           for row_1 in rows_1:
                   cells_1 = row_1.findAll('td')
                   try:
                          list_1 = details.findAll('li')
                          decks = list_1[0].text.encode('utf8').split(':')
                          cabin = list_1[1].text.encode('utf8').split(':')
                          cabin_number = list_1[2].text.encode('utf8').split(':')
                          list_key = ''.join(list(decks[0] + '|' + cabin[0] + '|' + cabin_number[0]))
                          list_value = ''.join(list(decks[1] + '|' + cabin[1] + '|' + cabin_number[1]))
                          list_key = list_key.split('|')
                          list_value = list_value.split('|')
    
                          try: 
                                  col1_1 = str(cells_1[0].get_text()).split('\n')
                                  col2_1 = str(cells_1[1].get_text()).split('\n')
                                  col3_1  = str(cells_1[2].get_text()).split('\n')
                                  col4_1  = str(cells_1[3].get_text()).split('\n')
                          except:
                                  continue
    
                          keys_1 = ['ship'] + col1_1 + col3_1 + list_key
                          values_1 = ship + col2_1 + col4_1 + list_value
                          dict_1 = dict(zip(keys_1, values_1))
                          with open('Z:/Cruiseshipschedule/details/details_'+ ship_in + '.csv', 'wb') as f:
                                 w = csv.DictWriter(f, dict_1.keys())
                                 w.writeheader()
                                 w.writerow(dict_1)
                                 f.close()                  
    
                   except:                   
                          if not list_1:
                                 list_of_rows_1=[]
                                 for row_1 in rows_1:
                                        cells_1 = row_1.findAll('td')
    
                                        try: 
                                                col1_1 = cells_1[0].text.encode('utf8').split(':') 
                                                col2_1 = cells_1[1].text.encode('utf8').split(':')
                                                col3_1 = cells_1[2].text.encode('utf8').split(':')
                                                col4_1 = cells_1[3].text.encode('utf8').split(':')
                                                list_of_cells_1.append(col1_1)
                                                list_of_cells_1.append(col2_1)
                                                list_of_cells_1.append(col3_1)
                                                list_of_cells_1.append(col4_1)
                                        except:
                                                continue 
                                        list_of_rows_1.append(list_of_cells_1)
    
                                        outfile_1 = open('Z:/Cruiseshipschedule/details/details_'+ ship_in + '.csv', "wb")
                                        writer = csv.writer(outfile_1, delimiter='|')
                                        writer.writerows(list_of_rows_1)      
                                        outfile_1.close()
                          else:
                              continue
           try:
                  itineray1 = tables[1]
                  rows_2 = itineray1.findAll("tr")
                  list_of_rows_2=[]
                  for row_2 in rows_2:
                          cells_2 = row_2.find_all("td")
                          list_of_cells_2 = [get_headings(url)[2].text]
    
                          try:
                                  date1_2 = str(cells_2[0].get_text())
                                  itinerary2_2 = str(cells_2[1].get_text()).split('\n')
                                  price3_2 = str(cells_2[2].get_text()).split('\n')
                                  list_of_cells_2.append(date1_2)
                                  list_of_cells_2.append(itinerary2_2)
                                  list_of_cells_2.append(price3_2)
                          except:
                                  continue
                          list_of_cells_2.append(ship)
                          list_of_rows_2.append(list_of_cells_2)
    
                          outfile_2 = open('Z:/Cruiseshipschedule/itinerary1/itinerary1_'+ ship_in + '.csv', "wb")
                          writer = csv.writer(outfile_2, delimiter='|')
                          writer.writerows(list_of_rows_2)      
                          outfile_2.close()      
    
           except:
                  continue
    
           try:
                  itineray2 = tables[2]
                  list_of_rows_3=[]
                  for row_3 in rows_3:
                          cells_3 = row_3.find_all("td")        
                          list_of_cells_3 = [get_headings(url)[3].text]
    
                          try:
                                  date1_3 = str(cells_3[0].get_text())
                                  itinerary2_3 = str(cells_3[1].get_text()).split('\n')
                                  price3_3 = str(cells_3[2].get_text()).split('\n')
                                  list_of_cells_3.append(date1_3)
                                  list_of_cells_3.append(itinerary2_3)
                                  list_of_cells_3.append(price3_3)
                          except:
    
                                  continue
                          list_of_cells_3.append(ship)
                          list_of_rows_3.append(list_of_cells_3)
    
                          outfile_3 = open('Z:/Cruiseshipschedule/itinerary2/itinerary2_'+ ship_in + '.csv', "wb")
                          writer = csv.writer(outfile_3, delimiter='|')
                          writer.writerows(list_of_rows_3)
                          outfile_3.close()
           except:
                  continue
           print "check out the data!"
    

Thanks for all the support! The code works - which means I get data. But
strange wise for some ships it takes only a few itineraries. I can't find the
mistake in my code. Python doesn't throw a error.

While scraping I saw that a few urls do not have exactly the same structure of
tables. That is why I put try and except to avoid stopping the script.

I would really appreciate some thoughts.


Answer: 
    #Second program here:
    ...
    ...
    page_array = [....]
    
    for page in page_array:
    
        mech = Browser()
        page1 = mech.open(page)
    
       #...The rest of the 1st program here
    

Another option would be to convert your second program to a function:

    
    
       from mechanize import Browser
       from bs4 import BeautifulSoup
       import lxml
       from lxml import html
       import csv
       import requests
    
    def get_links(url):
           links_array = []
    
           mech = Browser()
           #url = 'http://www.cruiseshipschedule.com/'
           page = mech.open(url)
           html = page.read()
           soup = BeautifulSoup(html, "lxml")
           table1 = soup.table #Ocean Cruise
           table2 = table1.findNext('table')#River Cruise
           links = table2.findAll('a')
    
    
           for link in links:
                 links_array.append(link.get('href').replace('http://www.cruiseshipschedule.com/', ''))
    
          return links_array  #<****HERE
    

Then in the first program, you would import the file that contains your second
program:

    
    
    import second_prog 
    
    url = 'http://www.cruiseshipschedule.com/ama-waterways/ms-amabella-cruises/'
    mech = Browser()
    
    for link in second_prog.get_links(url):
        page = mech.open(link)
        #Continue with first program here
    

And, if you want to keep everything in the same file, then you could move the
function in your second program into your first program--which would mean you
wouldn't need the import statement anymore.

**Edit:**

Does any of the following code have anything to do with your error:

    
    
    ship_in = soup1.h1
    ship_in = ship_in.text
    ship = u' '.join(ship_in.split())
    u' '.join(ship_in.split())
    ship = [ship]
    
    h21 = soup1.h2
    h22 = h21.findNext('h2')
    h23 = h22.findNext('h2')
    h24 = h23.findNext('h2')
    h25 = h24.findNext('h2')
    h_y1 = h22.text
    h_y2 = h23.text
    h_y3 = h24.text
    itinerary1_header = u' '.join(h_y1.split())
    u' '.join(h_y1.split())
    itinerary2_header = u' '.join(h_y2.split())
    u' '.join(h_y2.split())
    itinerary3_header = u' '.join(h_y3.split())
    u' '.join(h_y3.split())
    

Let's see:

    
    
    soup1 = BeautifulSoup(html, "lxml")
    
    #Code above here
    
    table_1 = soup1.findAll('table')[0]
    table_2 = soup1.findAll('table')[1]
    table_3 = soup1.findAll('table')[2]
    

Calculating table_1 does not reference anything that was removed.  
Calculating table_2 does not reference anything that was removed.  
Calculating table_3 does not reference anything that was removed.

So, you make a copy of your program, and you delete that whole section. Then
you try to figure out what went wrong. The process of deleting code to isolate
a problem is called _debugging_ code.

Next, why should you make BS go to the trouble--and the time--of searching
through the whole html page 3 times to get all the tables?

    
    
    table_1 = soup1.findAll('table')[0]
    table_2 = soup1.findAll('table')[1]
    table_3 = soup1.findAll('table')[2]
    

Every time you write `soup1.findAll('table')`, BS has to search through the
whole html page to find all the `<table>` tags.

Instead, you can search just once:

    
    
     tables = soup1.findAll('table')
    
     table1 = tables[0]
     table2 = tables[1]
     table3 = tables[2]
    

Making python retrieve an element from a list is very fast--much faster than
making BS search through an entire web page hunting for all the `<table>`
tags.

Next, whenever you find yourself writing variables with names:

    
    
     table1
     table2
     table3
    

and they only differ by a number, you need to STOP what you are doing--and use
a list instead. In this case, you already have a list: `tables`, and the
elements inside `tables` already have the names `tables[0], tables[1],
tables[2]`, etc., so you don't need to create the variables `table1, table2,
table3`. In fact, you don't even have to refer to the elements of `tables` by
the names `tables[0], tables[1], tables[2]`\--instead you can use a `for loop`
to step through all the tables:

    
    
    for table in tables:
    
        #Do something with the table variable
    

That has two advantages:

1) You don't have to write all the names `tables[0], tables[1], tables[2]` in
your code. What if you had to examine 1,000 tables? Are you really going to
write:

    
    
    tables[0] = ...
    tables[1] = ...
    ...
    ...
    <an hour later>
    tables[999] = ...
    

2) The second advantage of using a for loop is that you only have to write the
code that processes a table ONCE, and the `for loop` will apply the code to
each table in the `tables` list.

If `tables` has more table tags than you want to examine, then you can write:

    
    
    first_three = tables[:3]
    

But note that if tables only has two tables, then `first_three` will only
contain two tables. That _can_ be an advantage: a for loop doesn't need to
know how many table tags are in the tables list--a for loop will blindly
process all the elements in the list you give it, no matter how many tables
are in the list.

Finally, your error states:

    
    
    table_3 = soup1.findAll('table')[2] IndexError: list index out of range
    

That is the same error you would get if you did this:

    
    
    data = ['a', 'b']
    print(data[2])
    

That means that findAll() found less than 3 tables on the page. How do you fix
that? If the page doesn't contain three tables then it doesn't contain three
tables. You can process only the tables a page contains with a max of three,
like this:

    
    
    target_tables = tables[:3]
    
    for table in target_tables:
    
        #Do stuff here
    

