<p>I am trying to scrape an html table and save its data in a database. What strategies/solutions have you found to be helpful in approaching this program.</p>

<p>I'm most comfortable with Java and PHP but really a solution in any language would be helpful.</p>

<p>EDIT: For more detail, the UTA (Salt Lake's Bus system) provides bus schedules on its website. Each schedule appears in a table that has stations in the header and times of departure in the rows. I would like to go through the schedules and save the information in the table in a form that I can then query. </p>

<p>Here's the <a href="http://www.rideuta.com/ridingUTA/schedules/routeSchedules.aspx" rel="nofollow noreferrer">starting point</a> for the schedules </p>

[{'answer_id': 73213, 'author': 'Matej', 'author_id': 11457, 'author_profile': 'https://Stackoverflow.com/users/11457', 'pm_score': 2, 'selected': False, 'text': '<p>There is a nice book about this topic: <a href="http://oreilly.com/catalog/9780596005771/" rel="nofollow noreferrer">Spidering Hacks by Kevin Hemenway and Tara Calishain</a>. </p>\n'}, {'answer_id': 73216, 'author': 'David Cumps', 'author_id': 4329, 'author_profile': 'https://Stackoverflow.com/users/4329', 'pm_score': 3, 'selected': True, 'text': '<p>It all depends on how properly your HTML to scrape is? If it\'s valid XHTML, you can simply use some XPath queries on it to get whatever you want.</p>\n\n<p>Example of xpath in php: <a href="http://blogoscoped.com/archive/2004_06_23_index.html#108802750834787821" rel="nofollow noreferrer">http://blogoscoped.com/archive/2004_06_23_index.html#108802750834787821</a></p>\n\n<p>A helper class to scrape a table into an array: <a href="http://www.tgreer.com/class_http_php.html" rel="nofollow noreferrer">http://www.tgreer.com/class_http_php.html</a></p>\n'}, {'answer_id': 73217, 'author': 'Gilligan', 'author_id': 12356, 'author_profile': 'https://Stackoverflow.com/users/12356', 'pm_score': 1, 'selected': False, 'text': '<p>I have tried screen-scraping before, but I found it to be very brittle, especially with dynamically-generated code.\nI found a third-party DOM-parser and used it to navigate the source code with Regex-like matching patterns in order to find the data I needed.</p>\n\n<p>I suggested trying to find out if the owners of the site have a published API (often Web Services) for retrieving data from their system. If not, then good luck to you.</p>\n'}, {'answer_id': 73236, 'author': 'Petey', 'author_id': 2059, 'author_profile': 'https://Stackoverflow.com/users/2059', 'pm_score': 2, 'selected': False, 'text': "<p>I've found that scripting languages are generally better suited for doing such tasks. I personally prefer Python, but PHP will work as well. Chopping, mincing and parsing strings in Java is just too much work.</p>\n"}, {'answer_id': 73321, 'author': 'pianohacker', 'author_id': 10956, 'author_profile': 'https://Stackoverflow.com/users/10956', 'pm_score': 1, 'selected': False, 'text': '<p>This would be by far the easiest with Perl, and the following CPAN modules:</p>\n\n<ul>\n<li><a href="http://metacpan.org/pod/HTML::Parser" rel="nofollow noreferrer">http://metacpan.org/pod/HTML::Parser</a></li>\n<li><a href="http://metacpan.org/pod/LWP" rel="nofollow noreferrer">http://metacpan.org/pod/LWP</a></li>\n<li><a href="http://metacpan.org/pod/DBD/mysql" rel="nofollow noreferrer">http://metacpan.org/pod/DBD/mysql</a></li>\n<li><a href="http://metacpan.org/pod/DBI.pm" rel="nofollow noreferrer">http://metacpan.org/pod/DBI.pm</a></li>\n</ul>\n\n<p>CPAN being the main distribution mechanism for Perl modules, and accessible by running the following shell command, for example:</p>\n\n<p><code>\n # cpan HTML::Parser\n</code></p>\n\n<p>If you\'re on Windows, things will be more interesting, but you can still do it: <a href="http://www.perlmonks.org/?node_id=583586" rel="nofollow noreferrer">http://www.perlmonks.org/?node_id=583586</a></p>\n'}, {'answer_id': 73505, 'author': 'cjm', 'author_id': 8355, 'author_profile': 'https://Stackoverflow.com/users/8355', 'pm_score': 1, 'selected': False, 'text': '<p>pianohacker overlooked the <a href="http://metacpan.org/pod/HTML::TableExtract" rel="nofollow noreferrer">HTML::TableExtract</a> module, which was designed for exactly this sort of thing.  You\'d still need <a href="http://metacpan.org/pod/LWP" rel="nofollow noreferrer">LWP</a> to retrieve the table.</p>\n'}, {'answer_id': 279752, 'author': 'Thorvaldur', 'author_id': 35781, 'author_profile': 'https://Stackoverflow.com/users/35781', 'pm_score': 1, 'selected': False, 'text': '<p>If what you want is a form a csv table then you can use this:\nusing python:</p>\n\n<p>for example imagine you want to scrape forex quotes in csv form from some site like: <a href="http://www.oanda.com/convert/fxhistory?date_fmt=us&amp;date=11/10/08&amp;date1=01/01/08&amp;exch=USD&amp;exch2=USD&amp;expr=AUD&amp;expr2=AUD&amp;lang=en&amp;margin_fixed=0&amp;format=CSV&amp;redirected=1" rel="nofollow noreferrer">fxoanda</a></p>\n\n<p>then...</p>\n\n<pre><code>from BeautifulSoup import BeautifulSoup\nimport urllib,string,csv,sys,os\nfrom string import replace\n\ndate_s = \'&amp;date1=01/01/08\'\ndate_f = \'&amp;date=11/10/08\'\nfx_url = \'http://www.oanda.com/convert/fxhistory?date_fmt=us\'\nfx_url_end = \'&amp;lang=en&amp;margin_fixed=0&amp;format=CSV&amp;redirected=1\'\ncur1,cur2 = \'USD\',\'AUD\'\nfx_url = fx_url + date_f + date_s + \'&amp;exch=\' + cur1 +\'&amp;exch2=\' + cur1\nfx_url = fx_url +\'&amp;expr=\' + cur2 +  \'&amp;expr2=\' + cur2 + fx_url_end\ndata = urllib.urlopen(fx_url).read()\nsoup = BeautifulSoup(data)\ndata = str(soup.findAll(\'pre\', limit=1))\ndata = replace(data,\'[&lt;pre&gt;\',\'\')\ndata = replace(data,\'&lt;/pre&gt;]\',\'\')\nfile_location = \'/Users/location_edit_this\'\nfile_name = file_location + \'usd_aus.csv\'\nfile = open(file_name,"w")\nfile.write(data)\nfile.close()\n</code></pre>\n\n<p>once you have it in this form you can convert the data to any form you like.</p>\n'}, {'answer_id': 4850867, 'author': 'immutabl', 'author_id': 2671514, 'author_profile': 'https://Stackoverflow.com/users/2671514', 'pm_score': 1, 'selected': False, 'text': "<p>At the risk of starting a shitstorm here on SO, I'd suggest that if the format of the table never changes, you could just about get away with using Regularexpressions to parse and capture the content you need.</p>\n"}]
['https://Stackoverflow.com/questions/73168', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/3274/']