title: Job to call external API every hour and perform tasks for ~10000 rows individually
body: <p>I'm currently looking at designing a system that needs to basically run a job every hour, but for roughly 10,000 rows. Each one of those rows would then need to call an external api and do some other bits in terms of analytics.</p>
  <p>I'm currently trying to work out the best way to achieve this, but I've not had to do anything like this before so would appreciate and advice or guidance anyone has. I'm primarily used to GCP so I've focussed my ideas on the tooling available there (This will also most likely be done in a JS/Node environment).</p>
  <p>My initial thoughts on the design are as follows.</p>
  <ul>
  <li>Use Cloud Scheduler to create a job that runs every hour</li>
  <li>Cloud scheduler triggers a cloud function</li>
  <li>Cloud function retrieves all necessary rows and creates a pub/sub message in a topic for each row.</li>
  <li>Cloud pub sub then triggers another cloud function that calls the external API and performs other tasks for that row and writes the data back where it needs to go.</li>
  <li>Fin</li>
  </ul>
  <p>My reasoning for having the first function add items to a queue is obviously a cloud function is limited by execution time and memory, so I didn't think it prudent to have one function try and process all rows by itself. My assumption that pub/sub would trigger a new instance of the function each time rather than overwrite the first?</p>
  <p>I think I can in theory batch some of the external API calls, maybe upto around 20 at a time, so I don't know if that would/should have an impact on the above design.</p>
  <p>I obviously want this to cost as little as possible as well, so I don't know whether having an app engine instance do this would be better? But then I also don't know if I'd run into memory and timeout issues there.</p>
  <p>A thought that's occurred to me as I write this is whether I could batch the batches as it were. Coming from a JS background I could create all the batch API calls and execute them in a Promise.all() call. Again not sure on impact on memory and performance with that so I guess I would need to test it.</p>
  <p>Does anyone notice any gaping holes in the above or would there be a better solution to this?</p>
  <p>Thanks</p>