Is it recommended to monthly planned restart MongoDB production server to clean non-mapped virtual memory? We have a replica set of production mongoDB 3.6.10 , large volume of 5.5 TB , over 10,000 connections anytime , applications read both from primary and secondary servers , we see almost every month a crash in the primary server while every month we see increase in the non-mapped virtual memory.
Is it recommended to do a planned restart every month in order to clean the non-mapped virtual memory and to avoid server crashes?
Thanks
alon
Is it recommended to do a planned restart every month in order to clean the non-mapped virtual memory and to avoid server crashes? 


No, scheduled restarts are not a recommended practice. You should investigate and address the source of your issue rather than restarting your server periodically.

If your deployment is correctly configured, MongoDB shouldn't crash when out of memory. Restarting your mongod processes will temporarily free up memory, but it is expected that a database server will try to take advantage of available memory as needed for your workload.

If you are running Linux, one likely cause for unexpected process shutdown is the Out-of-Memory (OOM) Killer which looks for processes using a lot of memory (like a database server) and shuts them down if there is insufficient memory or swap space. If the OOM Killer decides to kill mongod, the telltale log message is normally in your system logs  -- try grepping for "oom-killer" or "killed process" in /var/log/messages. See Taming the OOM Killer for more background if this sounds potentially applicable to your environment. 

If your overall data size is 5.5TB, I assume there is also general contention for server resources unless you have a relatively small working set or very large host instances.

One source of non-mapped memory usage is your current connection count. mongod allocates up to 1MB per connection, so 10,000+ connections is 10GB+ of RAM. Other consumers of RAM are server-side processing such as aggregation queries, in-memory sorts, Map/Reduce, and JavaScript evaluation. Allocated non-mapped virtual memory or memory outside of the WiredTiger cache will normally be freed when no longer needed

The MongoDB Production Notes and Operations Checklist include some advice that is worth reviewing for applicability to your environment.