Best way to store scraped web pages?
Simply put you can create an &quot;index data base&quot; in the following format:
ID | URL | Timestamp | link_to_file | other_metadata

You could even store the actual file, instead of just the link with most databases.
However the simplest way might be the best here. This &quot;index db&quot; would be created automatically in the scraping process and serve as navigation and check to avoid double scraping, etc.