How do I get CiviCRM working with varnish on wordpress?
A big part of the pages in civi are not public, so as you might serve different pages to every user, they can't be cached.

But I think your problem is elsewhere, if you are "alone" on your admin pages, it should not be slow, even without varnish

Have a look at the demo pages, are you experiencing the same speed on your server? if not, there is something wrong in your install, and I'd fix that, so you do not need varnish

X+