title: wget Vs urlretrieve of python
body: <p>I have a task to download Gbs of data from a website. The data is in form of .gz files, each file being 45mb in size.</p>
  
  <p>The easy way to get the files is use "wget -r -np -A files url". This will donwload data in a recursive format and mirrors the website. The donwload rate is very high 4mb/sec.</p>
  
  <p>But, just to play around I was also using python to build my urlparser.</p>
  
  <p>Downloading via Python's urlretrieve is damm slow, possible 4 times as slow as wget. The download rate is 500kb/sec. I use HTMLParser for parsing the href tags.</p>
  
  <p>I am not sure why is this happening. Are there any settings for this.</p>
  
  <p>Thanks</p>