how to properly use try/except/with inside functions and main
Question: I am a relative python newbie and I am getting confused with how to properly
handle exceptions. Apologies for the dumb question.

In my main() I iterate through a list of dates and for each date I call a
function, which downloads a csv file from a public web server. I want to
properly catch exceptions for obvious reasons but especially because I do not
know when the files of interest will be available for download. My program
will execute as part of a cron job and will attempt to download these files
every 3 hours if available.

What I want is to download the first file in the list of dates and if that
results in a 404 then the program shouldn't proceed to the next file because
the assumption is if the oldest date in the list is not available then none of
the others that come after it will be available either.

I have the following python pseudo code. I have try/except blocks inside the
function that attempts to download the files but if an exception occurred
inside the function how do I properly handle it in the main() so I can make
decisions whether to proceed to the next date or not. The reason why I created
a function to perform the download is because I want to re-use that code later
on in the same main() block for other file types.

    
    
    def main():
    ...
    ...
    # datelist is a list of date objects
        for date in datelist:
            download_file(date)
    
    def download_file(date):
        date_string = str(date.year) + str(date.strftime('%m')) + str(date.strftime('%d'))
        request = HTTP_WEB_PREFIX+ date_string + FILE_SUFFIX
        try: 
            response = urllib2.urlopen(request)
        except urllib2.HTTPError, e:
            print "HTTPError = " + str(e)
        except urllib2.URLError, e:
            print "URLError = " + str(e)
        except httplib.HTTPException, e:
            print "HTTPException = " + str(e)  
        except IOError:
            print "IOError = " + str(e)
        except Exception:
            import traceback
            print "Generic exception: " + traceback.format_exc()
        else: 
            print "No problem downloading %s - continue..." % (response)
            try: 
                with open(TMP_DOWNLOAD_DIRECTORY + response, 'wb') as f:
            except IOError:
                print "IOError = " + str(e)
            else:
                f.write(response.read())
            f.close()
    


Answer: The key concept here is, if you can fix the problem, you should trap the
exception; if you can't, it's the caller's problem to deal with. In this case,
the downloader can't fix things if the file isn't there, so it should bubble
up its exceptions to the caller; the caller should know to stop the loop if
there's an exception.

So let's move all the exception handling out of the function into the loop,
and fix it so it craps out if there's a failure downloading the file, as the
spec requires:

    
    
    for date in datelist:
            date_string = str(date.year) + 
                          str(date.strftime('%m')) + 
                          str(date.strftime('%d'))
        try:
            download_file(date_string)
        except:
            e = sys.exc_info()[0]
            print ( "Error downloading for date %s: %s" % (date_string, e) )
            break
    

`download_file` should now, unless you want to put in retries or something
like that, simply not trap the exceptions at all. Since you've decoded the
date as you like in the caller, that code can come out of `download_file` as
well, giving the much simpler

    
    
    def download_file(date_string):
        request = HTTP_WEB_PREFIX + date_string + FILE_SUFFIX
        response = urllib2.urlopen(request) 
        print "No problem downloading %s - continue..." % (response)
        with open(TMP_DOWNLOAD_DIRECTORY + response, 'wb') as f:
            f.write(response.read())
            f.close()
    

I would suggest that the `print` statement is superfluous, but that if you
really want it, using `logger` is a more flexible way forward, as that will
allow you to turn it on or off as you prefer later by changing a config file
instead of the code.

