Does it make sense to limit Mongo RAM?
No. If it starts to swap, it means that MongoDB needs more RAM than the system has available. Limiting the RAM used would make the situation worse.

MongoDB uses only up to a certain limit (95% of physical RAM, iirc), which might lead to the wrong assumption that there is free RAM available and it must be a problem caused by MongoDB. Quite the contrary: MongoDB keeps the system usable for administrative tasks.

Solution: increase the physical RAM (scale up) or start sharding (scale out).