Varnish causes page load time to increase
I cannot comment yet as a new user so here goes my response.

For starters 1 second to 1.3 seconds is not all that much of a difference, and you dont seem to have an overabundance of misses so its doing its thing. Run some tests that simulate a lot of page requests on the same content and watch the hits/misses.  Also did you test again without Varnish on the same site?

One other thing is to be sure you've properly sized the Varnish storage to match your servers available resources.  This being off too much can drag the system for obvious reasons. 

Any reason your not using Drupal 7?  Pressflow 7 is available now too.

There is also the option to use Boost + Varnish.
http://www.trellon.com/content/blog/boosted-varnish-high-performance-caching-made-easy