title: How to reduce the CPU utilization when google and bing crawls on magento site?
body: <p>I have been facing a weird situation for a while now and need a guidance regarding this.</p>
  
  <p><strong>Problem:</strong>
  <em>Since last two days we were experiencing very slow website as compared to what it was when we launched the server . We thought it was temperory issue . But now , it has gone dead slow &amp; a page takes atleast 3 mins to load. I also checked that the CPU uitlization somehow reached 100% and believe that the crawling might doing this.</em></p>
  
  <p>We are using some third party to do our SEO and google dynamic remarketing and advertising of our magento website. i firmly believe that these things needs to crawl my website for indexing over the search engine.</p>
  
  <p>I have seen that google and bing crawls our website regularly. You may call it google bot and bing bot and suddenly it has seen a largest spike.</p>
  
  <p><strong>Have a look at the screenshot:</strong>
  <a href="https://www.dropbox.com/s/2c4u04rhtbi99j0/Screenshot%202015-11-14%2014.16.41.png?dl=0" rel="nofollow">https://www.dropbox.com/s/2c4u04rhtbi99j0/Screenshot%202015-11-14%2014.16.41.png?dl=0</a> </p>
  
  <p>With the largest spike being caused by bing and google at the same time and the smaller ones appear to only be google bot.</p>
  
  <p>So i just had a quick question regarding this?</p>
  
  <p><em>Do you guys think that if a bot IP is whitelisted, will we have a problem with the SEO and google advertising and dynamic remarketing, because then it will not allow that IP to crawl on our website???</em></p>
  
  <p><em>Is this a spam or the bots crawling our store which is causing store response time to reduce which can impact search engine ranking and conversions on our store??</em></p>
  
  <p><em>Can a large instance type of AWS will help us solving our CPU-usage problem??</em>
  <strong>Note</strong>: We are already using <strong>m3.large</strong> instance type.</p>