title: Python - Crawler to Extract Data from Tables
body: <p>I am trying to extract customer reviews section from Agoda (<a href="http://www.agoda.com/grand-hyatt-taipei/hotel/taipei-tw.html" rel="nofollow">http://www.agoda.com/grand-hyatt-taipei/hotel/taipei-tw.html</a>). The data I am interested are under the 'div id = "hotelreview-panel" ', which include the number of reviews made by different types of traveler (eg. Business traveler) and the corresponding KPI scales (eg. Value for Money) for each type of traveler.</p>
  
  <p>There are two problems I have:</p>
  
  <p>(1) I cannot reach the right table through the find function of BeautifulSoup. There exists the table class "customer-review-category-issues" but it kept returning none. </p>
  
  <pre><code>import requests
  import math
  import csv
  from bs4 import BeautifulSoup
  
  HotelNames = ['grand-hyatt-taipei']
  
  with open('agoda_hotel_reviews.csv', 'w') as csvfile:
      for iHotel in HotelNames: 
          url = "http://www.agoda.com/"+iHotel+"/hotel/taipei-tw.html"
          res = requests.get(url)
          soup = BeautifulSoup(res.text, 'html.parser')
  
          table_review = soup.find("table", {"class" : "customer-review-category-issues"})
          record_rev = []
  
          for row in table_review.findAll('tr'):
              col = row.findAll('td')
              rev_issue = col[1].string.split('\n').strip()[0]
              rev_count = col[1].string.split('\n').strip()[1]
              record_rev.extend([rev_issue], [rev_count])
  
       filewriter = csv.writer(csvfile, delimiter='|', lineterminator='\n')
       filewriter.writerow(record_rev)
  </code></pre>
  
  <p>(2) How do I extract the KPI when I switch to different traveler types so my returning list become something like [All reviews, 35, 8.1, 9.2, 9.0, 9.1, 9.1, 8.3, Business travelers, 10, 7.8, 8.6, 8.4, 8.6, 8.6, 7.2], that is [Traveler Type, # of Review, KPI 1 (value for money), KPI 2 (location), ... KPI 6]?</p>