How to grant more memory to run R processes in SQL Server 2016 R Services
You'll want to use the resource governor.  Here is a good blog.

Enable Resource Governor (do some research on this):

ALTER RESOURCE GOVERNOR RECONFIGURE;  


This will tell you how much % of your memory R can use.

SELECT * FROM sys.resource_governor_external_resource_pools


Sample query to change values from the link:

-- Default value
ALTER EXTERNAL RESOURCE POOL [default] 
WITH (AFFINITY CPU = AUTO)
GO

CREATE EXTERNAL RESOURCE POOL RService_Resource_Pool  
WITH (  
     MAX_CPU_PERCENT = 10  
    ,MAX_MEMORY_PERCENT = 5
);  

ALTER RESOURCE POOL [default] WITH (max_memory_percent = 60, max_cpu_percent=90);  
ALTER EXTERNAL RESOURCE POOL [default] WITH (max_memory_percent = 40, max_cpu_percent=10);  
ALTER RESOURCE GOVERNOR reconfigure;


Don't forget to run reconfigure again:

ALTER RESOURCE GOVERNOR RECONFIGURE;