title: Firebase Cloud Function Repeatedly Fails Due to Quota Error
body: <p>I have a Firebase Cloud Function that is triggered by an <code>onCreate</code> at a path in my Realtime database. Yesterday, after doing some basic testing, I began to get "quota exceeded" errors in the Cloud Functions Log. What was troubling though, is that the error kept coming, first every 2 seconds or so, then ramping up to every 8 seconds or so, for about 1.5 hours.</p>
  
  <p>Here's a small segment of the trouble:</p>
  
  <p><a href="https://i.stack.imgur.com/nH7Dg.png" rel="noreferrer"><img src="https://i.stack.imgur.com/nH7Dg.png" alt="Errors. Errors. More Errors. *Sigh*"></a></p>
  
  <p>I took a look at the documentation covering <a href="https://firebase.google.com/docs/functions/retries" rel="noreferrer">Retrying Asynchronous Functions</a> and it seems clear that in most cases a function will stop executing and the event will be discarded if an error occurs. The triggering event did not seem to be getting cleared in my case, perhaps because the error I was getting was coming from outside of my function. Here's the full error text:</p>
  
  <blockquote>
    <p>Error: quota exceeded (DNS resolutions : per 100 seconds); to increase quotas, enable billing in your project at <a href="https://console.cloud.google.com/billing?project=myproject" rel="noreferrer">https://console.cloud.google.com/billing?project=myproject</a>. Function cannot be executed.</p>
  </blockquote>
  
  <p><strong>Yes, I realize that a quota issue is easily solved by moving to the Blaze plan, which I will be doing soon, but before I do, I'd like to understand how to handle this case should it happen in the future.</strong> I had to re-deploy my functions to get the Error to stop happening. After studying the docs, it sounds as though I also could have deleted the function to stop the error, but neither re-deploying or deleting functions seems like a great path to take once my app reaches production.</p>
  
  <p>My thought is that I must have been stuck in a retry loop internal to the Firebase Cloud function service. None of my logging was being hit (some of which would happen on function start, and some during a function run), plus the message says "Function cannot be executed", which to me means that the Error happened before my code was ever executed. Another function which is triggered by the same event was logging an identical Error.</p>
  
  <p>So, to anyone in the know, my questions are as follows:</p>
  
  <ol>
  <li>For cases where functions seem to be stuck in a loop, is redeployment or deletion the ONLY path to recovery? What other possible approaches might I have taken?</li>
  <li>Is there a way I can handle an Error that is coming from outside my implementation? Can an Error like this be raised by the service or do Errors ONLY originate in my code? My guess is that I'm limited to monitoring the logs for such failures, but perhaps there are other possible actions that can be taken to make my service more robust.</li>
  </ol>
  
  <p>Finally, a few further notes to address follow-up questions some may have:</p>
  
  <ul>
  <li><p>Yes, the repeating Error did prevent any new invocations of the function in question to take place. Other functions could still be executed, except for the one other that is triggered by the same event.</p></li>
  <li><p>Yes, I can recreate the behavior to an extent. If I set the DNS Resolutions quota per 100 seconds to a low value (5, for example), and do a few executions, I get the same error, repeating every 8 seconds or so. Oddly though, when I recreate in this manner, it recovers on its own after about 10-20 throws, seemingly around the time the 100 seconds has elapsed, which makes sense. In the case of the original incident, the errors repeated for more than an hour, at which point I decided to re-deploy, stopping them.</p></li>
  <li><p>Yes, I do see the most recent <code>Function execution took ### ms, finished with status 'ok'</code> message in my logs before the Errors start rolling in. After the Errors are done repeating, I do see a the last-in triggering data get processed by the function successfully. This is what caused me to wonder if it was the triggering event that was causing Cloud Functions to keep trying.</p></li>
  <li><p>No, my function does not write to a location that would retrigger itself.</p></li>
  <li><p>Yes, I realize I could easily resolve this by <strong>moving to the Blaze plan</strong> to get a much higher quota. <strong>I will be doing that.</strong> First though, I'd like to understand the mechanics of what has gone wrong so I can make any available improvements.</p></li>
  </ul>