Web servers commonly generate directory indexes listing the files within a directory and additional metadata.  There is variation in the content and formatting of these pages, but generally have a strong common resemblance.  There are common operations like identifying the most recent file in a directory according to the listed timestamp or downloading the file with latest version.  These involve common steps of downloading the html index, extracting filenames, pattern-matching, date parsing, sorting, etc...  I typically hack together a minimal solution in bash using wget, grep, awk, sort, and date, but I'm curious if anyone has worked on generalized solutions to problems involving html file indexes.Are there any (preferably command-line) utilities for processing/parsing these html files?Obviously it is a non-trivial task and can't be performed perfectly by in all cases, but a solution that avoids re-implementing the wheel for 90% of all cases is certainly within the realm of possibility.
Tools for parsing automatically-generated HTTP indexes?
wget