Varnish and page cache
As answered by @dicix, Drupal will not return the correct Cache-Control headers to Varnish without "Page Cache" being enabled in the Drupal Performance configuration.

The problem I ran into with having both the Drupal Page Cache and Varnish at the same time is that it potentially increases the delay before stale content is refreshed to a maximum of both TTLs. For example, having a max-age of 5min could result in the stale content sticking around and being served for up to 10 minutes (or 15 minutes if Varnish doesn't reset the cache-control header given to the browser to something less).

My solution was to set "Cache pages for anonymous users" to be on in the Drupal Performance settings with a reasonable "Expiration of cached pages" of 5min. I then added the following to my settings.php to prevent Drupal from actually caching any page content:

// Use the fake cache for pages since we are caching anonymous HTML with Varnish
if (!class_exists('DrupalFakeCache')) {
  $conf['cache_backends'][] = 'includes/cache-install.inc';
}
$conf['cache_class_cache_page'] = 'DrupalFakeCache';


The DrupalFakeCache just provides stubs that don't store anything or return anything.

Reference: Suppress caching &#40;for development&#41; or to use an external page cache