Failed to create appdomain "master.sys[runtime].2"
I finally understand why your server uses only 3Gb of RAM, you have 32-bit version of SQL Server: SQL 2012 (SP1) - 11.0.3128.0 Intel(x86) and it just cannot use more than 4Gb.

You should use 64-bit version of the Server to resolve your problem.

In previous versions (until 2012) it seems -g parameter could be useful:
SQL Server 2012 Memory


  -g startup parameter
  
  We used the -g startup option to change the default value of a region
  in SQL Server user address space known as "Memory-To-Reserve". This
  region was also known as "memory-to-leave or MTL.  The
  "Memory-To-Reserve" (or) -g configuration option are relevant only for
  a 32-bit instance of SQL Server.
  
  Multi pages allocation and CLR was part of Mem-to-reserve (-g)  in In
  previous SQL Server versions until SQL Server 2008 R2 , From Denali
  they are part of BPOOL (Controlled by Max server memory)  So you may
  have to remove –g if you have set it to give space for multipage
  allocator or CLR in earlier versions and migrating to Denali now.


But since you are on 2012 it seems the only way is to install SQL Server x64