How to scrape multiple HTML tables with Beautiful Soup parser?
Question: sorry for the stupid question ... just started using python (but I love it).

**The problem:** I want to scrape data from the [center for documentation of
violism in syria](http://vdc-
sy.org/index.php/en/martyrs/1/c29ydGJ5PWEua2lsbGVkX2RhdGV8c29ydGRpcj1ERVNDfGFwcHJvdmVkPXZpc2libGV8c2hvdz0xfGV4dHJhZGlzcGxheT0wfA==).
currently I'm using this scraper to collect the data. the problem is that I
can access only one row instead of scraping all rows from the table. the
preferred output should look like

name status sex province area dateofdeath causeofdeath

    
    
    import urllib2
    from BeautifulSoup import BeautifulSoup
    f = open('syriawar.tsv', 'w')
    f.write("Row" + "\t" + "Data" + "\n")
    
    for x in range (0,249):
    
    
    syria = "file" + "\t" + str(x)
    print "fetching data ... " + syria
    
    
    url ='http://vdc-sy.org/index.php/en/martyrs/' + str(x) + '/c29ydGJ5PWEua2lsbGVkX2RhdGV8c29ydGRpcj1ERVNDfGFwcHJvdmVkPXZpc2libGV8c2hvdz0xfGV4dHJhZGlzcGxheT0wfA==' 
    
    page = urllib2.urlopen(url)
    soup = BeautifulSoup(page)
    
    sentence = soup.findAll('tr')[3].text
    
    words = sentence
    Data = str(words)
    
    f.write(str(x) + "\t" + Data + "\n" )
    
    f.close()
    


Answer: You need another layer of iteration. You should first call findAll('tr') to
get all the rows. Then remove the rows that are headers and empty and then
loop through the remaining rows and call .text on those elements to get the
text of the rows you want. Write each row to the file from within your inner
loop.

Here is the script fixed. Note that the utf-8 codec had to used because the
page contains unicode in the text. You should verify that this is getting
everything you want. The empty tags were causing Beautiful Soup some problems.

    
    
    import urllib2
    from bs4 import BeautifulSoup
    import codecs
    
    f = codecs.open('syriawar.tsv', 'w', 'utf-8')
    f.write("Row" + "\t" + "Data" + "\n")
    
    for x in range (0,249):
    
      syria = "file" + "\t" + str(x)
      print "fetching data ... " + syria
    
      url ='http://vdc-sy.org/index.php/en/martyrs/' + str(x) + '/c29ydGJ5PWEua2lsbGVkX2RhdGV8c29ydGRpcj1ERVNDfGFwcHJvdmVkPXZpc2libGV8c2hvdz0xfGV4dHJhZGlzcGxheT0wfA=='
    
      page = urllib2.urlopen(url)
      soup = BeautifulSoup(page)
    
      rows = soup.findAll('tr')
    
      i = 0;
      for row in rows[3:]:
         if i%2 == 0:
            f.write(str(i/2) + "\t" + row.text + "\n" )
         i += 1
    
    f.close()
    

Another spiffy way to do this is to use
[Scrapemark](http://arshaw.com/scrapemark/). It works great for tables and
lists.

