title: Web crawlers overloading site
body: <p>We have a problem with a number of our sites where Yahoo, Google, Yandex, Bing Ahrefs and others all index the site at the same time which kills the website.</p>
  
  <p>I've configured fail2ban to block the source IPs, but these are forever changing so not ideal.  I have also tried using robots.txt but this makes little difference.</p>
  
  <p>We have tried putting the site behind cloudflare, but again this makes little difference and all we can do is block the source IPs.</p>
  
  <p>What else can I do?  </p>
  
  <p>Currently we're monitoring the site with Nagios, which restarts nginx when the site becomes un-responsive, but this seems far from ideal.  </p>
  
  <p>Ubuntu server running nginx</p>
  
  <p>Robots.txt file is here:-</p>
  
  <pre><code>User-agent: *
  Disallow: /
  </code></pre>
  
  <p>Posting here in case there is anything that I can get our developers to try.</p>
  
  <p>Thanks</p>