drupal 7 on virtual server with debian linux - inconsistent response time
500 MB is not much, unfortunately :)
and the default debian settings for Apache and MySQL are prepared for bigger servers. You should make sure the server doesnt swap.

Probably the most important setting for that is in the part mpm_prefork_module in the apache configuration (/etc/apache2/apache2.conf). You will want to have less Startservers, MinSpareServers and MaxSpareservers.

Installing APC helps, too (and is relatively easy). Even though you have to sacrifice some RAM, it speeds up the loading and reduces the RAM needed for every pageload. With 500MB, you will want to use APC only as opcode. Make sure you dont use much more stuff than 1 installation of drupal, or set APC so that other stuff doesnt get cached (the default settings caches every php file you use, thus probably pushing the important stuff away).

Enable the css + js aggregation - apart from saving roundtrips for the client, it also saves your server from serving extra files.

Also take into account that drupal does some caching internally and the times when it takes longer may be because of that - it is useful to "prime" the caches ie by crawling all pages before you start measuring.

2bits have great articles about optimizing for drupal.