title: Downloading files in a loop with Python mechanize
body: <p>I am having trouble downloading multiple files in a loop with Python mechanize. I am also utilizing Beautiful Soup 4. The documentation for either package does not seem to have the answers.</p>
  
  <p>Here is my code - please skip down to the actual loop. I included everything for reference:</p>
  
  <pre><code>import mechanize, cookielib, os, time
  from bs4 import BeautifulSoup
  
  
  fcList = ['abandoned mine land inventory points', 'abandoned mine land inventory polygons', \
            'abandoned mine land inventory sites', 'coal mining operations', 'coal pillar location-mining', \
            'industrial mineral mining operations', 'longwall mining panels', 'mine drainage treatment/land recycling project locations', \
            'mined out areas', 'residual waste operations', 'underground mining permit']
  
  dlLink = 'FTP Download'
  dloadPath = 'C:\\Users\\SomeGuy\\Downloads'
  
  # Browser
  br = mechanize.Browser()
  
  # Cookie Jar
  cj = cookielib.LWPCookieJar()
  br.set_cookiejar(cj)
  
  # Select the first (index zero) form
  br.select_form(nr=0)
  
  # Input form data
  br.form['Keyword']='mining'
  br.submit()
  html = br.response().read()
  
  # Pass html to beautiful soup for parse
  soup = BeautifulSoup(html)
  htmlinks = soup.findAll("a")
  
  # Find links with desired text
  for htmlink in htmlinks:
      string = str(htmlink.string)
      if string.lower() in fcList:
          print "Matched link!", string + ". attempting download...\n"
          try:
              req = br.click_link(text = string)
              br.open(req)
              print "URL: " + str(br.geturl)
              html = br.response().read()
              soup = BeautifulSoup(html)
              the_tag = soup.find('a', text=dlLink)
              fileURL = the_tag.get('href')
              print fileURL
              # attempt download
              fnam = string.replace(" ", "_")
              fnam = fnam.replace("/", "_")
              f = br.retrieve(fileURL, os.path.join(dloadPath, fnam + ".zip"))
              print f + "\n"
              br.back()
          except:
              print "An unknown error occurred."
  </code></pre>
  
  <p>Output:</p>
  
  <pre><code>&gt;&gt;&gt; 
  Matched link! Abandoned Mine Land Inventory Points. attempting download...
  
  URL: &lt;bound method Browser.geturl of &lt;mechanize._mechanize.Browser instance at 0x02D9D7B0&gt;&gt;
  http://www.pasda.psu.edu/data/dep/AMLInventoryPoints2013_04.zip
  An unknown error occurred.
  Matched link! Abandoned Mine Land Inventory Polygons. attempting download...
  
  An unknown error occurred.
  Matched link! Abandoned Mine Land Inventory Sites. attempting download...
  
  An unknown error occurred.
  Matched link! Coal Mining Operations. attempting download...
  
  An unknown error occurred.
  Matched link! Coal Pillar Location-Mining. attempting download...
  
  An unknown error occurred.
  Matched link! Industrial Mineral Mining Operations. attempting download...
  
  An unknown error occurred.
  Matched link! Longwall Mining Panels. attempting download...
  
  An unknown error occurred.
  Matched link! Mine Drainage Treatment/Land Recycling Project Locations. attempting     download...
  
  An unknown error occurred.
  Matched link! Mined Out Areas. attempting download...
  
  An unknown error occurred.
  Matched link! Residual Waste Operations. attempting download...
  
  An unknown error occurred.
  Matched link! Underground Mining Permit. attempting download...
  
  An unknown error occurred.
  &gt;&gt;&gt; 
  </code></pre>
  
  <p>I believe the problem could be due to having no wait time between downloads. This code will download the very first file in the loop successfully, no matter which one I choose. Or perhaps it's some other bug I am not aware of- I just downloaded mechanize and beautifulsoup yesterday!</p>