Would SQL Server in a Cluster failover if it is running out of RAM I have a SQL Server 2008 R2 in a Cluster in an active-passive mode.
The server is running out of RAM (both physical and pagefile).
In the instance level settings there is not Max RAM set (set to the maximum number).

Would the instance failover?
Starting with SQL Server 2012 the cluster will have a more comprehensive health check which allows for a possible resource based health condition using the SP_Server_diagnostics procedure. https://msdn.microsoft.com/en-us/library/ff878233.aspx

More on that failover policy can be read about here: https://msdn.microsoft.com/en-us/library/ff878664.aspx#monitor

Unfortunately though SQL Server 2008R2 and earlier doesn't have much of a sophisticated cluster health check for the SQL Service. Simply the health check will just issue the following command "SELECT @@Servername" and if it returns in a timely fashion the health check passes. 

I would look instead though at why is the server under memory pressure and setting the max memory setting in the SQL Server instance putting a cap on that memory foot print.