Site Down for few minutes after clearing cache
Have you installed "Memcache" module ? If no then : 

First of all, check mysql slow query log. Also install "Memcache" module which will help you to reduce mysql server load.

Please refer this link for Drupal Memcache installation.