title: optimize web scraping using wget
body: <p>I am using wget to download a huge list of web pages(around 70,000).I am forced to put a sleep of around 2 seconds in between successive wget.This takes a huge amount of time.Something like 70 days.What I would like to do is to use proxies so that I can significantly speed up the process.I am using a simple bash script for this process.Any suggestions and comments are appreciated.</p>