Show HN: Scrape-reduce – A simple way to scrape websites
This is a tiny little project I made to scrape some websites. Not sure if it&#x27;s a novel idea, but I&#x27;ve found that splitting scraping into 2 steps is a good way to iterate quickly and safely.<p>1. &quot;scrape&quot; is when you actually download the HTML file for a page. This should be done very cautiously as you might be blocked by IP or captcha&#x27;s if you read a webpage too often. This project caches each page you request, so it can be cancelled and resumed later if necessary.<p>2. &quot;reduce&quot; - in this context - is simply processing the downloaded HTML to get what you want. This project provides a full DOM to query for the text on the page that you&#x27;re looking for, then save it to another file, or whatever you want to do.<p>By splitting it up into this &quot;pipeline&quot; we get a very simple &quot;scrape.ts&quot; file for fetching the HTML. And a very simple &quot;reduce.ts&quot; for processing the HTML.<p>Let me know what you think!