Limit the RAM memory usage in MongoDB
If you are running on Linux, you can use control groups to limit MongoDB memory as shown in the following article:

Easy Steps to Limit Mongodb Memory Usage by Ramakanta Sahoo

On Windows, a similar technique using the Windows System Resource Manager is described in:

Limit MongoDB memory use on Windows without Virtualization by Simon Green