title: How to schedule a periodic Celery task per Django model instance?
body: <p>I have a bunch of Feed objects in my database, and I'm trying to get each Feed to be updated every hour.  My issue here is that I need to make sure there aren't any duplicate updates -- it needs to happen no more than once an hour, but I also don't want feeds waiting two hours for an update. (It's okay if it happens every hour +/- a few minutes, but twice in a few minutes is bad.)</p>
  
  <p>I'm using Django and Celery with Amazon SQS as a broker.  I have the feed update code set up as a Celery task, but I'm failing to find a way to prevent duplicates while remaining compatible with Celery running on multiple nodes.</p>
  
  <p>My current solution is to add a <code>last_update_scheduled</code> attribute to the Feed model and run the following task every 5 minutes (pseudo-code):</p>
  
  <pre><code>threshold = datetime.now() - timedelta(seconds=3600)
  for f in Feed.objects.filter(Q(last_update_scheduled__lt = threshold) |
                               Q(last_update_scheduled = None)):
      updateFeed.delay(f)
      f.last_update_scheduled = now
      f.save()
  </code></pre>
  
  <p>This is susceptible to a number of synchronization issues.  For example, if my task queues get backed up, this task could run twice at the same time, causing duplicate updates.  I've seen some solutions for this (like <a href="http://ask.github.com/celery/cookbook/tasks.html" rel="nofollow noreferrer">Celery's recipe</a> and <a href="https://stackoverflow.com/questions/4095940/running-unique-tasks-with-celery">an adaptation on Stack Overflow</a>), but the memcached solution isn't reliable, e.g. duplicates could happen when restarting memcached or if it happens to run out of memory and purge old data.  Not to mention I'd hate to have to add memcached to my production configuration just for a simple lock.</p>
  
  <p>In a perfect world, I'd like to be able to say:</p>
  
  <pre><code>@modelTask(Feed, run_every=3600)
  def updateFeed(feed):
      # do something expensive
  </code></pre>
  
  <p>But so far my imagination fails me on how to implement that decorator.</p>