title: How to schedule hundreds of thousands of tasks?
body: <p>We have hundreds of thousands of tasks that need to be run at a variety of arbitrary intervals, some every hour, some every day, and so on. The tasks are resource intensive and need to be distributed across many machines.</p>
  
  <p>Right now tasks are stored in a database with an "execute at this time" timestamp. To find tasks that need to be executed, we query the database for jobs that are due to be executed, then update the timestamps when the task is complete. Naturally this leads to a substantial write load on the database.</p>
  
  <p>As far as I can tell, we are looking for something to release tasks into a queue at a set interval. (Workers could then request tasks from that queue.) </p>
  
  <p>What is the best way to schedule recurring tasks at scale?</p>
  
  <p>For what it's worth we're largely using Python, although we have no problems using components (RabbitMQ?) written in other languages.</p>
  
  <p><strong>UPDATE:</strong> Right now we have about 350,000 tasks that run every half hour or so, with some variation. 350,000 tasks * 48 times per day is 16,800,000 tasks executed per day.</p>
  
  <p><strong>UPDATE 2:</strong> There are no dependencies. The tasks do not have to be executed in order and do not rely on previous results.</p>