title: Problems while web scraping with golang
body: <p>I'm creating a web scraper with golang and I just wanted to ask some questions about how most of them work. For example, how does Googlebot not use a lot of bandwidth when scraping because you have to go to each URL to get data and that can be thousands of URLs so not only will that take bandwidth, it will also take a lot of time. I'm building a web scraper and I'm experiencing these issues and I wanted to ask what the best way of fixing these issues is. <a href="https://github.com/hackermondev/cosmic" rel="nofollow noreferrer">https://github.com/hackermondev/cosmic</a></p>