title: How to design a distributed job scheduler?
body: <p>I want to design a job scheduler cluster, which contains several hosts to do cron job scheduling. For example, a job which needs <code>run every 5 minutes</code> is submitted to the cluster, the cluster should point out which host to fire next run, making sure:</p>
  
  <blockquote>
    <ol>
    <li>Disaster tolerance: if not all of the hosts are down, the job should be fired successfully.</li>
    <li>Validity: only one host to fire next job run.</li>
    </ol>
  </blockquote>
  
  <p>Due to disaster tolerance, job cannot bind to a specific host. One way is all the hosts polling a DB table(certainly with lock), this guaranteed only one host gets the next job run. Since it often locks table, is there any better design?</p>