Assign maximum memory limit for a particular database in sql server
There is no way to limit the amount of memory a specific database takes.  SQL Server uses a least used / first evicted from memory policy (basically).