Recursive Error Handling in Python with a limited amount of tries I have a class method that is set up to make requests to an API and I'm having issues handling the possible errors that could arise by trying to communicate with that server. The idea is if an error were to pop up, then it would call on itself until either it no longer needs to (because it worked), or it reaches a limit I set up for it; in this case, I would want it to just simply raise an exception that would be caught somewhere else in the code. What I don't know is how to properly return the response if it crapped out the first time it made the call and had to call on itself from the except block.

I'm not 100% clear if this actually gets the job done. Additionally, I dont know if this is the best way to do it. I know a possible suggestion is to make error handling more specific, given the requests library myriad exceptions, but I figured that any exception should just be handled by trying X amount of time and then quitting if it doesn't work.

def getRequest(self, endpoint, attempts):
        baseUrl = self.baseUrl
        url = baseUrl + endpoint
        if self.expirationT &lt;= datetime.datetime.now():
            self.token, self.expirationT = self.auth(client_id, client_secret)
        else:
            pass
        try:
            response = requests.get(url, auth = BearerAuth(self.token))
            response.raise_for_status()
        except:
            if attempts &lt; 20:
                time.sleep(3)
                response = self.getRequest(endpoint, attempts + 1)
                return response
            else:
                raise Exception
        else:
            return response
I know a possible suggestion is to make error handling more specific, given the requests library myriad exceptions, but I figured that any exception should just be handled by trying X amount of time and then quitting if it doesn't work.


I disagree here. Not just any exception should be handled. It's possible to have typo'd a bug into your code inside the try, and you definitely don't want try masking a bug.

If you check the source (or documentation), you'll see that the requests exceptions all seem to inherit from RequestException. If you really want to handle every possible request exception the same, I would catch the base class RequestException instead. I still don't think this is a good idea though without doing any logging. There may very well be a RequestException that gets thrown at some point which indicates that you accidentally gave the request bad data, not that there was a problem with the request being carried out using good data. I'd check the docs for the methods used and figure out what exact exceptions you want to retry on.



This also doesn't need to be recursion. In this case, nothing bad will likely happen because you have a limit of 20 retries, which isn't enough to exhaust the stack any sane case. If you ever increase that limit up to 1000 though, you may run into real problems.

I think this could be done pretty easily using a while True loop. The first two lines of the function seem to be essentially constants, so they don't need to be recomputed every time. Everything under those lines though can be stuck in a loop.

def getRequest(self, endpoint, max_attempts=20, retry_delay=3):
    baseUrl = self.baseUrl
    url = baseUrl + endpoint

    attempts = 0
    while True:
        if self.expirationT &lt;= datetime.datetime.now():
            self.token, self.expirationT = self.auth(client_id, client_secret)

        try:
            response = requests.get(url, auth=BearerAuth(self.token))
            response.raise_for_status()
            return response

        except requests.RequestException as e:
            attempts += 1

            if attempts &lt; max_attempts:
                time.sleep(retry_delay)

            else:
                raise RuntimeError("Max number of retires met.")

                # Or to preserve in the trace the original problem that caused the error:
                # raise RuntimeError("Max number of retires met.") from e


Things to note:


To retry now, instead of manually recursing, I'm just letting control fall out of the except so that the while can restart again.
Instead of attempts being a parameter, I just made it a local variable which is incremented inside of the except.
I'm throwing a more specialized exception with an informative error message. Throwing the generic Exception makes life more difficult for the users of your code. Ideally, they should be able to pick and choose what exceptions they handle and when. Throwing Exception though forces them to catch your errors. RuntimeError isn't really the best exception here, but I couldn't think of a good built-in one for this purpose. You may want to make a custom exception for this case:

class TooManyRetries(Exception):
    pass

. . .

raise TooManyRetries("Max number of retires met.")

I got rid of the else: pass. That isn't necessary.
You had two "magic numbers": 20 and 3 to mean the max number of attempts and the retry delay. I don't think it's a good idea to have those hard coded though. What if you want to change either at some point? You'd have to edit the code. I made them parameters of the function, defaulting to the values that you had. If you don't specify them, the behavior will be as you had before, but now they can be easily changed as needed.