SQL Server 2008 R2 Resource Managment
" Occasionally it will creep up approaching 100% RAM usage; this has
  been causing a lot of problems"


That's often a red herring, as SQL Server will use all of RAM (but will give it back under OS memory pressure). 

Rule of thumb leave 2GB or 5-10% of total physical memory free (Max Memory setting) whichever is larger.

Glenn Berry has a good resource here:


Suggested Max Memory Settings for SQL Server 2005/2008


Also:


A Sys Admins Guide to Microsoft SQL Server Memory 
Wow… An online calculator to misconfigure your SQL Server memory!
How much memory does my SQL Server actually need?
Troubleshooting SQL Server: A Guide for the Accidental DBA
Server Memory Options