SQL dumping all pages from buffer cache every few minutes
Although you seem to have solved the issue yourself, here is a summary of the relevant information surrounding the solution.

Server Memory Server Configuration Options

Microsoft write in their article Server Memory Server Configuration Options(Microsoft | SQL Docs) for the section Setting the memory options manually

(emphasis mine)  


  Also, setting a min_server_memory value is essential in a virtualized environment to ensure memory pressure from the underlying host does not attempt to deallocate memory from the buffer pool on a guest SQL Server virtual machine (VM) beyond what is needed for acceptable performance.  


The section regarding Lock Pages in Memory (same document) has an equalling compelling paragraph which reads as:

(emphasis mine)  


  This Windows policy determines which accounts can use a process to keep data in physical memory, preventing the system from paging the data to virtual memory on disk. Locking pages in memory may keep the server responsive when paging memory to disk occurs. The Lock Pages in Memory option is set to ON in instances of SQL Server Standard edition and higher when the account with privileges to run sqlservr.exe has been granted the Windows Lock Pages in Memory (LPIM) user right.  


The section LPIM goes on to explain that:

(emphasis mine)


  Setting this option does not affect SQL Server dynamic memory management, allowing it to expand or shrink at the request of other memory clerks. When using the Lock Pages in Memory user right it is recommended to set an upper limit for max server memory as detailed above. 


... and in an important comment that:

(emphasis mine) 


  Setting this option should only be used when necessary, namely if there are signs that sqlservr process is being paged out. In this case, error 17890 will be reported in the Errorlog, resembling the below example:  

A significant part of sql server process memory has been paged out. 
This may result in a performance degradation. Duration: #### seconds. 
Working set (KB): ####, committed (KB): ####, memory utilization: ##%.  

  
  Starting with SQL Server 2012 (11.x), trace flag 845 is not needed for Standard Edition to use Locked Pages.


Solution

Based on the above findings and your observations the solution to your problem would be to configure the following settings: 


min_server_memory (5-10 GB?) based on your comment:


  About 8-9GB of data in the cache at the moment, seems to be stable.  


...and Microsoft's recommendation of setting a min_server_memory.
max_server_memory (20-32 GB) based on your observation: 


  It seems to suggest that 20GB is enough for the workload on this box. I'm happy leaving it with 32GB for now but I'd really like to get to the bottom of this so I can better configure the VMs/SQL.  


...and Microsoft's recommendation of setting a max_server_memory.
Lock Pages in Memory : Enabled for SQL Server Service Account.
Based on the ERRORLOG entry of your SQL Server instance you mentioned and Microsoft's reference in the article.  


  Setting this option should only be used when necessary, namely if there are signs that sqlservr process is being paged out.



Before You Continue...

(One of) the benefit(s) of having a virtualised environment is that resources can/should be shared, and possibly even over-commited. However, turning on Lock Pages In Memory (LPIM) could have a negative impact on your Hyper-V environment, if your hardware is hosting multiple instances. An over-commitment of RAM could exhaust other instances. 

Before you consider switching all the levers, start off with settings 1. and 2. and if fine tuning these memory settings doesn't work, then consider turning on LPIM if you have enough hardware.