Varnish doesn't cache first hit at all, only second hit
It looks like a normal behaviour : 


first url hit : Varnish puts the page in cache with some id
only following hits will serve the cached id instead of querying the backend. 


If you want anonymous users to be always served by varnish, you need to implement a cache warmer logic.

Setup a sitemap. Then create a cron task that will parse the sitemap with curl for instance,  and run that cron task at a frequency synced with your cache expiry time...