SQL Server 2008: how much RAM memory should SQL Server use in a 8GB RAM server?
Question: Considering scenario, that we have 8GB and we also have IIS hosted in this server (which I believe is a bad idea), in how much RAM should SQL Server use? 


The answer to the question "how much memory SQL Server would use" is, as much as it can. For that reason its always advisable to put appropriate value for max server memory so that buffer pool is restricted.

Please don't use task manager to get SQL Server memory utilization it wont give correct value in case when SQL Server service account has locked pages in memory(LPIM) privilege. To find how much RAM SQL Server is using please use below query

select
(physical_memory_in_use_kb/1024)Memory_usedby_Sqlserver_MB,
(locked_page_allocations_kb/1024 )Locked_pages_used_Sqlserver_MB,
(total_virtual_address_space_kb/1024 )Total_VAS_in_MB,
process_physical_memory_low,
process_virtual_memory_low
from sys. dm_os_process_memory



  Where can I configure the amount of RAM that SQL Server should really use without impacting Windows, IIS and any other running services?


You can do that in Sp_configure using below query. 

sp_configure 'show advanced options',1
go
reconfigure 
go
sp_configure 'max server memory (MB)' , xxx --value in MB which you need to set
go
reconfigure.


You can fix max server memory using the query but remember, SQL Server can still use memory outside buffer pool for third party DLL's and extended stored procs. This can lead SQL Server to use memory more that what is restricted in buffer pool

Please read this article. It would help you in setting starting value for max server memory. You should use below counters to set correct value for max server memory and monitor SQL Server memory usage.

SQLServer:Buffer Manager--Page Life Expectancy(PLE): PLE shows for how long page remain in buffer pool. The longer it stays the better it is. Its common misconception to take 300 as a baseline for PLE.   But it is not,I read it from Jonathan Kehayias book( troubleshooting SQL Server) that this value was baseline when SQL Server was of 2000 version and max RAM one could see was from 4-6 G. Now with 200G or RAM coming into picture this value is not correct. He also gave the formula( tentative) how to calculate it. Take the base counter value of 300 presented by most resources, and then determine a multiple of this value based on the configured buffer cache size, which is the 'max server memory' sp_ configure option in SQL Server, divided by 4 GB.   So, for a server with 32 GB allocated to the buffer pool, the PLE value should be at least (32/4)*300 = 2400. So far this has done good to me so I would recommend you to use it.  

SQLServer:Buffer Manager--CheckpointPages/sec: Checkpoint pages /sec counter is important to know about memory pressure because if buffer cache is low then lots of new pages needs to be brought into and flushed out from buffer pool,  due to load checkpoint's work will increase and will start flushing out dirty pages very frequently. If this counter is high then your SQL Server buffer pool is not able to cope up with requests coming and we need to increase it by increasing buffer pool memory or by increasing physical RAM and then making adequate changes in Buffer pool size. Technically this value should be low if you are looking at line graph in perfmon this value should always touch base for stable system.  

SQLServer:Buffer Manager--Freepages: This value should not be less you always want to see high value for it. 

SQLServer:Memory Manager--Memory Grants Pending: If you see memory grants pending in buffer pool your server is facing SQL Server memory crunch and increasing memory would be a good idea. For memory grants please read this article

SQLServer:memory Manager--Target Server Memory: This is amount of memory SQL Server is trying to acquire.

SQLServer:memory Manager--Total Server memory This is current memory SQL Server has acquired.