title: Errors saving data to Google Datastore
body: <p>We are trying to save streaming data to Google Datastore using this repo as the basis: <a href="https://github.com/meraki/google-cloud-scanning-api" rel="nofollow noreferrer">https://github.com/meraki/google-cloud-scanning-api</a>. After the update to Meraki 3.0, we had to refactor the code to reflect the new data structure. However, we are now getting errors in Cloud Functions during the save process:</p>
  <pre><code>ERROR SAVING: Error: 4 DEADLINE_EXCEEDED: Deadline Exceeded
  at Object.exports.createStatusError (/workspace/node_modules/grpc/src/common.js:91:15)
  at Object.onReceiveStatus (/workspace/node_modules/grpc/src/client_interceptors.js:1209:28)
  at InterceptingListener._callNext (/workspace/node_modules/grpc/src/client_interceptors.js:568:42)
  at InterceptingListener.onReceiveStatus (/workspace/node_modules/grpc/src/client_interceptors.js:618:8)
  at callback (/workspace/node_modules/grpc/src/client_interceptors.js:847:24)
  at callbackTrampoline (internal/async_hooks.js:126:14) {
  code: 4,
  metadata: Metadata { _internal_repr: {}, flags: 0 },
  details: 'Deadline Exceeded'
  }
  </code></pre>
  <p>and</p>
  <pre><code>ERROR SAVING: Error: 14 UNAVAILABLE: Received RST_STREAM with error code 7
  at Object.exports.createStatusError (/workspace/node_modules/grpc/src/common.js:91:15)
  at Object.onReceiveStatus (/workspace/node_modules/grpc/src/client_interceptors.js:1209:28)
  at InterceptingListener._callNext (/workspace/node_modules/grpc/src/client_interceptors.js:568:42)
  at InterceptingListener.onReceiveStatus (/workspace/node_modules/grpc/src/client_interceptors.js:618:8)
  at callback (/workspace/node_modules/grpc/src/client_interceptors.js:847:24)
  at callbackTrampoline (internal/async_hooks.js:126:14) {
  code: 14,
  metadata: Metadata { _internal_repr: {}, flags: 0 },
  details: 'Received RST_STREAM with error code 7'
  }
  </code></pre>
  <p>We think this is a rate error. Our metrics show that we went from about 4.7 invocations per second to 0.03 invocations per second. We think we are ingesting more data in shorter bursts, causing it to exceed the rate limits. We tried to implement a retry (pulled from here: <a href="https://cloud.google.com/datastore/docs/concepts/transactions#uses_for_transactions" rel="nofollow noreferrer">https://cloud.google.com/datastore/docs/concepts/transactions#uses_for_transactions</a>) but we still see the error on some saves.</p>
  <p>Our retry function:</p>
  <pre><code>async function addObservationWithRetry(entry, type) {
      const maxTries = 5;
      async function tryRequest(currentAttempt, delay) {
          try {
              await addObservation(entry, type);
          } catch (err) {
              console.log(`Timeout, attempt ${currentAttempt}`);
              if (currentAttempt &lt;= maxTries) {
                  // Use exponential backoff
                  setTimeout(async () =&gt; {
                      await tryRequest(currentAttempt + 1, delay * 2);
                  }, delay);
              }
              throw err;
          }
      }
      await tryRequest(1, 100);
  }
  </code></pre>
  <p>Calling the retry function:</p>
  <pre><code>observations.forEach(function(entry) {
      addObservationWithRetry(entry, &quot;event&quot;);
  });
  </code></pre>
  <p>Adding an observation:</p>
  <pre><code>function addObservation(obs, type) {
      let entities = [];
      const locations = obs.locations;
      const entity = {
          key: &quot;null&quot;,
          data: [&lt;dictionaries for all variables&gt;],
      };
  
      locations.forEach(function(loc) {
          entity.data[x].value = &lt;different values that are the same for each location&gt;;
  
          loc.rssiRecords.forEach(function(record) {
              const observationKey = datastore.key(type);
              entity.key = observationKey;
              entity.data[y].value = &lt;different values that are the same for each rssi record&gt;;
              entities.push(entity);
  
          });
      });
  
      if (entities.length &gt; 0) {
          datastore
              .save(entities) // Push array of entities
              .then(() =&gt; {})
              .catch(err =&gt; {
                  console.error('ERROR SAVING:', err);
              });
      }
  }
  </code></pre>