Configuring Drupal 7 settings.php with both Memcache and Varnish
I got it a little bit later. It was just the array format being different in the two sources I copies and pasted from. The problem line was:

#$conf['cache_backends'][] = array('sites/all/modules/contrib/varnish/varnish.cache.inc');


It should be:

#$conf['cache_backends'][] = 'sites/all/modules/contrib/varnish/varnish.cache.inc';