title: Facebook Crawler Bot Crashing Site
body: <p>Did Facebook just implement some web crawler? My website has been crashing a couple times over the past few days, severely overloaded by IPs that I've traced back to Facebook.</p>
  
  <p>I have tried googling around but can't find any definitive resource regarding controling Facebook's crawler bot via robots.txt. There is a reference on adding the following:</p>
  
  <p>User-agent: facebookexternalhit/1.1
  Crawl-delay: 5</p>
  
  <p>User-agent: facebookexternalhit/1.0
  Crawl-delay: 5</p>
  
  <p>User-agent: facebookexternalhit/*
  Crawl-delay: 5</p>
  
  <p>But I can't find any specific reference on whether Facebook bot respects the robots.txt. According to older sources, Facebook "does not crawl your site". But this is definitely false, as my server logs showed them crawling my site from a dozen+ IPs from the range of 69.171.237.0/24 and 69.171.229.115/24 at the rate of many pages each second.</p>
  
  <p>And I can't find any literature on this. I suspect it is something new that FB just implemented over the past few days, due to my server never crashing previously. </p>
  
  <p>Can someone please advice?</p>