Increase memory limit when server is stopped
Open SQL Server Configuration Manager
In the SQL Server Services pane, locate the instance that has the problem
Right-click that instance and select Properties
On the Startup Parameters tab, type -f, click Add, then click Apply
Right-click that instance and select Start (or Restart)


This will start the instance in minimal configuration mode (single user, no CHECKPOINT, and no startup procedures are run), which should allow you enough resources to connect and fix what you broke.