Drupal database loading slow
I think the problem is how your views are cached - if you say it is sometimes slow, it is most probably because those times the cache is beeing saved.
If the content doesnt change often, you might just prolongue the interval. Another option is to use Views content cache.
But basically if you have enough visitors (and enough people hovering the images :), they will only wait from time to time ...