title: Mongo DB Out Of Memory Crashes
body: <p>I'm running a MongoDB 3.4 instance with WiredTiger that 64 GB of RAM. Recently, I've been encountering frequent of out of memory conditions, which require a database restart. </p>
  
  <p>Below is what an increase in virtual memory looks like from mongostat. All of sudden the virtual memory increases and the server crashes. I've tried to go through the logs to look for issues, but haven't yet identified unique queries or actions that seem to precede virtual memory increase and the crash. </p>
  
  <pre><code>insert query update delete getmore command % dirty % used flushes vsize   res qr|qw ar|aw netIn netOut conn set    repl                      time
      *0    34     *0     *0       5    85|0     0.0   80.1       1 61.5G 59.5G   0|0   5|0 13.2k  5.09m  279 server  PRI 2019-09-23T14:20:36-04:00
       1    11     *0     *0       3    43|0     0.0   80.0       0 61.5G 60.1G   0|0   5|0 7.65k   223k  279 server  PRI 2019-09-23T14:20:37-04:00
      *0     5     *0     *0       0    46|0     0.0   80.1       0 61.9G 60.8G   0|0   5|0 4.03k   161k  279 server  PRI 2019-09-23T14:20:38-04:00
      *0    10     *0     *0       0    39|0     0.0   80.1       0 62.7G 61.5G   0|0   5|0 3.64k   128k  279 server  PRI 2019-09-23T14:20:39-04:00
      *0    18     *0     *0       0    51|0     0.0   80.0       0 63.0G 61.9G   0|0  24|0 5.85k  33.2k  279 server  PRI 2019-09-23T14:20:41-04:00
      *0    *0     *0     *0       0    25|0     0.0   80.0       0 63.0G 61.9G   0|0   6|0 2.10k   109k  279 server  PRI 2019-09-23T14:20:43-04:00
      *0     8     *0     *0       0    28|0     0.0   80.0       0 63.0G 61.9G   0|0   7|0 2.87k   118k  279 server  PRI 2019-09-23T14:20:45-04:00
      *0    32     *0     *0       2    98|0     0.0   80.3       0 63.2G 61.9G   0|0   6|0 11.6k   366k  279 server  PRI 2019-09-23T14:20:46-04:00
      *0    *0     *0     *0       1    18|0     0.0   80.2       0 63.4G 61.8G   0|0   5|0 1.93k  4.34m  279 server  PRI 2019-09-23T14:20:47-04:00
      *0     8     *0     *0       0    61|0     0.0   80.3       0 63.4G 61.8G   0|0   5|0 4.57k   325k  279 server  PRI 2019-09-23T14:20:48-04:00
  insert query update delete getmore command % dirty % used flushes vsize   res qr|qw ar|aw netIn netOut conn set repl                      time
       1    18      2     *0       3    40|0     0.0   80.4       0 63.6G 61.8G   0|0   6|0 22.5k   219k  279 server  PRI 2019-09-23T14:20:49-04:00
      *0    21     *0     *0       2    29|0     0.0   80.4       0 63.7G 61.8G   0|0   6|0 5.67k   573k  279 server  PRI 2019-09-23T14:20:50-04:00
      *0    17     *0     *0       0    45|0     0.0   80.2       0 63.8G 61.8G   0|0   5|0 5.53k  4.06m  279 server  PRI 2019-09-23T14:20:51-04:00
       1     8      1     *0       4    23|0     0.0   80.2       0 63.9G 61.8G   0|0   5|0 87.7k   341k  279 server  PRI 2019-09-23T14:20:52-04:00
       1    23     *0     *0       4    44|0     0.0   80.3       0 64.1G 61.8G   0|0   6|0 10.2k  4.56m  279 server  PRI 2019-09-23T14:20:53-04:00
      *0    *0     *0     *0       0     8|0     0.0   80.4       0 64.2G 61.8G   0|0   5|0  569b   122k  279 server  PRI 2019-09-23T14:20:54-04:00
      *0    12     *0     *0       0    30|0     0.0   80.2       0 64.3G 61.8G   0|0   5|0 3.61k  92.2k  279 server  PRI 2019-09-23T14:20:55-04:00
      *0     7     *0     *0       0    62|0     0.0   80.2       0 64.4G 61.8G   0|0   5|0 4.69k   113k  279 server  PRI 2019-09-23T14:20:56-04:00
      *0     1     *0     *0       0    29|0     0.0   80.2       0 64.5G 61.8G   0|0   5|0 2.59k  91.3k  279 server  PRI 2019-09-23T14:20:58-04:00
      *0     2     *0     *0       1    54|0     0.0   80.0       0 64.7G 61.7G   0|0   5|0 3.89k   104k  279 server  PRI 2019-09-23T14:20:59-04:00
  insert query update delete getmore command % dirty % used flushes vsize   res qr|qw ar|aw netIn netOut conn server  repl                      time
      *0     2     *0     *0       0    25|0     0.0   80.2       0 64.8G 61.7G   0|0   7|0 2.10k  86.5k  279 server  PRI 2019-09-23T14:21:00-04:00
       1    40      1     *0       0    69|0     0.0   80.4       0 65.0G 61.8G   0|0   7|2 14.5k   174k  279 server  PRI 2019-09-23T14:21:01-04:00
      *0     1     *0     *0       3    14|0     0.0   80.6       0 65.1G 61.8G   0|0   6|0 2.89k  80.5k  279 server  PRI 2019-09-23T14:21:02-04:00
      *0     2     *0     *0       0    18|0     0.0   80.8       0 65.2G 61.8G   0|0   6|0 1.51k  87.8k  279 server  PRI 2019-09-23T14:21:03-04:00
      *0     2     *0     *0       0     8|0     0.0   80.9       0 65.4G 61.8G   0|0   6|0  811b   126k  279 server  PRI 2019-09-23T14:21:04-04:00
      *0    16     *0     *0       0    45|0     0.0   80.9       0 65.5G 61.8G   0|0   6|0 5.02k   105k  279 server  PRI 2019-09-23T14:21:06-04:00
      *0     4     *0     *0       0    49|0     0.0   80.9       0 65.6G 61.8G   0|0   6|0 3.70k   101k  279 server  PRI 2019-09-23T14:21:07-04:00
      *0     4     *0     *0       1    51|0     0.0   80.9       0 65.8G 61.8G   0|0   6|0 4.52k   388k  279 server  PRI 2019-09-23T14:21:08-04:00
      *0     2      1     *0       2    33|0     0.0   81.0       0 65.9G 61.8G   0|0   6|0 84.9k   256k  279 server  PRI 2019-09-23T14:21:09-04:00
      *0    *0     *0     *0       0    20|0     0.0   81.0       0 66.0G 61.8G   0|0   6|0 1.32k  82.3k  279 server  PRI 2019-09-23T14:21:10-04:00
  insert query update delete getmore command % dirty % used flushes vsize   res qr|qw ar|aw netIn netOut conn set     repl                      time
      *0    21     *0     *0       0    41|0     0.0   81.0       0 66.1G 61.8G   0|0   6|0 5.66k   106k  279 server  PRI 2019-09-23T14:21:11-04:00
      *0    *0     *0     *0       0    11|0     0.0   81.1       0 66.3G 61.8G   0|0   6|0  818b  77.0k  279 server  PRI 2019-09-23T14:21:12-04:00
      *0    *0     *0     *0       0    18|0     0.0   81.1       0 66.4G 61.8G   0|0   6|0 1.21k  81.5k  279 server  PRI 2019-09-23T14:21:13-04:00
      *0    *0     *0     *0       1    15|0     0.0   81.0       0 66.5G 61.7G   0|0   6|0 2.05k  77.0k  279 server  PRI 2019-09-23T14:21:14-04:00
      *0     1     *0     *0       0    29|0     0.0   80.9       0 66.6G 61.8G   0|0   6|0 1.89k  87.6k  279 server  PRI 2019-09-23T14:21:15-04:00
      *0    20     *0     *0       0    72|0     0.0   80.9       0 66.8G 61.8G   0|0   6|0 7.59k   173k  279 server  PRI 2019-09-23T14:21:16-04:00
       4     5     *0     *0       6    38|0     0.0   80.8       0 66.8G 61.6G   0|0   6|0 20.2k   119k  279 server  PRI 2019-09-23T14:21:17-04:00
      *0    *0     *0     *0       0    42|0     0.0   81.0       0 67.0G 61.7G   0|0   6|0 2.45k  96.6k  279 server  PRI 2019-09-23T14:21:18-04:00
      *0    *0     *0     *0       0    17|0     0.0   80.9       0 67.1G 61.7G   0|0   6|0 1.15k  80.6k  279 server  PRI 2019-09-23T14:21:20-04:00
      *0    15     *0     *0       0    13|0     0.0   80.1       0 67.2G 61.7G   0|0   6|0 3.01k  83.2k  279 server  PRI 2019-09-23T14:21:21-04:00
  insert query update delete getmore command % dirty % used flushes vsize   res qr|qw ar|aw netIn netOut conn set     repl                      time
      *0     3     *0     *0       0    42|0     0.0   80.0       0 67.2G 61.7G   0|0   6|0 3.16k  96.3k  279 server  PRI 2019-09-23T14:21:22-04:00
      *0    *0     *0     *0       1    20|0     0.0   80.2       0 67.5G 61.8G   0|0   6|0 2.13k  81.7k  279 server  PRI 2019-09-23T14:21:23-04:00
      *0     2      1     *0       0    13|0     0.0   80.2       0 67.6G 61.8G   0|0   6|1 58.9k  90.9k  279 server  PRI 2019-09-23T14:21:24-04:00
      *0     1     *0     *0       2    29|0     0.0   80.3       0 67.7G 61.8G   0|0   6|0 3.74k   249k  279 server  PRI 2019-09-23T14:21:25-04:00
       3    29      2     *0       7    90|0     0.0   80.2       0 67.8G 61.7G   0|0   8|0 49.8k   494k  279 server  PRI 2019-09-23T14:21:26-04:00
       3    10      1     *0      12    40|0     0.0   80.2       0 68.0G 61.8G   0|0   8|0 18.2k  1.36m  279 server  PRI 2019-09-23T14:21:27-04:00
      *0     2     *0     *0       0    54|0     0.0   80.4       0 68.1G 61.7G   0|0   8|0 3.42k   308k  279 server  PRI 2019-09-23T14:21:28-04:00
      *0    *0     *0     *0       0    24|0     0.0   80.6       0 68.2G 61.7G   0|0   9|0 1.84k  81.0k  279 server  PRI 2019-09-23T14:21:29-04:00
      *0    12     *0     *0       0    20|0     0.0   80.8       0 68.4G 61.8G   0|0   9|0 3.26k   107k  279 server  PRI 2019-09-23T14:21:30-04:00
      *0     7     *0     *0       0    47|0     0.0   81.0       0 68.5G 61.7G   0|0   8|0 4.26k   128k  279 server  PRI 2019-09-23T14:21:32-04:00
  insert query update delete getmore command % dirty % used flushes vsize   res qr|qw ar|aw netIn netOut conn set     repl                      time
      *0    *0     *0     *0       1    15|0     0.0   81.2       0 68.6G 61.7G   0|0   8|0 1.77k  79.8k  279 server  PRI 2019-09-23T14:21:33-04:00
      *0    *0     *0     *0       0    15|0     0.0   81.4       0 68.7G 61.7G   0|0   8|0 1.07k  78.2k  279 server  PRI 2019-09-23T14:21:34-04:00
      *0    *0     *0     *0       0     8|0     0.0   81.6       0 68.9G 61.7G   0|0   6|0  545b  5.90m  279 server  PRI 2019-09-23T14:21:35-04:00
      *0    22     *0     *0       0    64|0     0.0   81.8       1 69.0G 61.7G   0|0   8|0 7.73k   120k  279 server  PRI 2019-09-23T14:21:36-04:00
      *0     1     *0     *0       1    39|0     0.0   81.9       0 69.1G 61.7G   0|0   8|0 3.59k  95.0k  279 server  PRI 2019-09-23T14:21:37-04:00
      *0     1     *0     *0       0    46|0     0.0   82.1       0 69.3G 61.8G   0|0   8|0 3.12k   201k  279 server  PRI 2019-09-23T14:21:38-04:00
      *0     2     *0     *0       0    22|0     0.0   82.2       0 69.4G 61.8G   0|0   8|0 2.09k  86.7k  279 server  PRI 2019-09-23T14:21:39-04:00
      *0    13     *0     *0       0    18|0     0.0   82.3       0 69.5G 61.8G   0|0   7|0 3.03k   135k  279 server  PRI 2019-09-23T14:21:40-04:00
      *0     7     *0     *0       0    51|0     0.0   82.5       0 69.7G 61.8G   0|0   7|0 4.56k   107k  280 server  PRI 2019-09-23T14:21:42-04:00
      *0    *0     *0     *0       1    18|0     0.0   82.5       0 69.7G 61.6G   0|0   7|0 2.25k  81.0k  279 server  PRI 2019-09-23T14:21:43-04:00
  insert query update delete getmore command % dirty % used flushes vsize   res qr|qw ar|aw netIn netOut conn set     repl                      time
      *0     5     *0     *0       0    28|0     0.0   82.7       0 69.9G 61.7G   0|0   9|0 2.78k   157k  279 server  PRI 2019-09-23T14:21:44-04:00
      *0    21     *0     *0       1    48|0     0.0   82.9       0 70.1G 61.8G   0|0   9|0 7.43k   230k  280 server  PRI 2019-09-23T14:21:45-04:00
      *0     8     *0     *0       2    71|0     0.0   83.0       0 70.2G 61.7G   0|0   9|0 7.69k   164k  280 server  PRI 2019-09-23T14:21:47-04:00
      *0     2     *0     *0       1    27|0     0.0   83.2       0 70.4G 61.8G   0|0   9|0 2.13k   580k  280 server  PRI 2019-09-23T14:21:48-04:00
      *0    *0     *0     *0       0    42|0     0.0   83.3       0 70.6G 61.8G   0|0   9|0 2.43k   102k  280 server  PRI 2019-09-23T14:21:49-04:00
      *0    *0     *0     *0       0    16|0     0.0   83.2       0 70.6G 61.8G   0|0   8|0 1.09k  4.04m  280 server  PRI 2019-09-23T14:21:50-04:00
       1    23     *0     *0       0    29|0     0.0   83.3       0 70.8G 61.8G   0|0   9|1 6.38k   101k  281 server  PRI 2019-09-23T14:21:51-04:00
     EOF
  no reachable servers
  </code></pre>
  
  <p>I also see virtual memory increases that sometimes do not result in crashes. However, after the virtual memory increases and doesn't go back down and stays at its high value. </p>
  
  <pre><code>    *0     97     *0     *0      10     62|0     0.0   80.1       0 49.1G 47.0G   0|0   3|0 29.2k  1.05m  281 server  PRI 2019-09-24T11:43:35-04:00
      *0     56     *0     *0       2     17|0     0.0   80.2       0 49.3G 48.0G   0|0   3|0 14.7k  1.31m  281 server  PRI 2019-09-24T11:43:36-04:00
      *0     94     *0     *0       5     20|0     0.0   80.1       0 50.1G 48.8G   0|0   3|0 26.6k   872k  281 server  PRI 2019-09-24T11:43:37-04:00
      *0     46     *0     *0       8     30|0     0.0   80.1       0 50.4G 49.1G   0|0   4|0 14.8k   738k  281 server  PRI 2019-09-24T11:43:38-04:00
      *0     61     *0     *0       6     36|0     0.0   80.2       0 50.8G 49.6G   0|0   3|0 20.1k  1.53m  281 server  PRI 2019-09-24T11:43:39-04:00
      *0     74     *0     *0       1     34|0     0.0   80.0       0 51.2G 49.9G   0|0   3|0 19.5k   621k  281 server  PRI 2019-09-24T11:43:40-04:00
      *0    145     *0     *0       5     33|0     0.0   80.0       0 51.5G 50.3G   0|0   3|0 40.6k  2.31m  281 server  PRI 2019-09-24T11:43:41-04:00
  insert  query update delete getmore  command % dirty % used flushes vsize   res qr|qw ar|aw netIn netOut conn         set repl                      time
      *0     30     *0     *0      10     23|0     0.0   80.1       0 51.9G 50.6G   0|0   4|0 10.3k   821k  281 server  PRI 2019-09-24T11:43:42-04:00
      *0     66     *0     *0      12      6|0     0.0   80.0       0 52.4G 51.1G   0|0   3|0 19.0k  1.03m  281 server  PRI 2019-09-24T11:43:43-04:00
      *0     62     *0     *0      11     19|0     0.0   80.2       0 52.8G 51.6G   0|0   3|0 19.1k  1.09m  281 server  PRI 2019-09-24T11:43:45-04:00
      *0     58     *0     *0       9     47|0     0.0   80.2       0 53.5G 52.2G   0|0   3|0 17.9k  1.57m  281 server  PRI 2019-09-24T11:43:46-04:00
       3    139      3     *0      10     38|0     0.0   80.2       0 54.2G 52.9G   0|0   3|0  103k   876k  281 server  PRI 2019-09-24T11:43:47-04:00
      *0     79     *0     *0       4     20|0     0.0   80.4       0 54.9G 53.5G   0|0   4|0 23.1k  2.25m  281 server  PRI 2019-09-24T11:43:48-04:00
      *0     46     *0     *0       6     25|0     0.0   80.2       0 55.3G 54.0G   0|0   4|0 14.2k  4.06m  281 server  PRI 2019-09-24T11:43:49-04:00
      *0     58     *0     *0       6     49|0     0.0   80.3       0 55.7G 54.4G   0|0   3|0 18.4k   902k  281 server  PRI 2019-09-24T11:43:50-04:00
      *0     82     *0     *0      10     34|0     0.0   80.0       0 56.3G 55.0G   0|0   3|0 22.5k   997k  281 server  PRI 2019-09-24T11:43:51-04:00
      *0     43     *0     *0       7     35|0     0.0   80.0       0 56.7G 55.4G   0|0   3|0 15.5k  2.27m  281 server  PRI 2019-09-24T11:43:52-04:00
  insert  query update delete getmore  command % dirty % used flushes vsize   res qr|qw ar|aw netIn netOut conn         set repl                      time
      *0     54     *0     *0       6     25|0     0.0   80.1       0 57.1G 55.8G   0|0   3|0 17.1k  6.94m  281 server  PRI 2019-09-24T11:43:53-04:00
      *0     72     *0     *0       4      9|0     0.0   80.2       0 57.6G 56.3G   0|0   4|0 19.6k   808k  281 server  PRI 2019-09-24T11:43:54-04:00
      *0     64     *0     *0       9     62|0     0.0   80.2       0 58.0G 56.7G   0|0   4|0 21.4k   975k  281 server  PRI 2019-09-24T11:43:55-04:00
      *0     43     *0     *0       5     17|0     0.0   80.2       0 58.5G 57.2G   0|0   3|0 11.4k  2.94m  281 server  PRI 2019-09-24T11:43:56-04:00
      *0     75     *0     *0       7     18|0     0.0   80.1       0 58.9G 57.6G   0|0   3|0 23.5k  1.44m  281 server  PRI 2019-09-24T11:43:57-04:00
      *0    128     *0     *0       7     26|0     0.0   80.2       0 59.5G 58.2G   0|0   3|0 37.6k  1.44m  281 server  PRI 2019-09-24T11:43:58-04:00
      *0     64     *0     *0       1     37|0     0.0   79.8       0 59.9G 58.6G   0|0   3|0 18.0k  1.01m  281 server  PRI 2019-09-24T11:43:59-04:00
       1     83     *0     *0       7     47|0     0.0   80.0       0 60.4G 59.1G   0|0   4|0 27.5k  4.80m  281 server  PRI 2019-09-24T11:44:00-04:00
      *0     75     *0     *0       6     42|0     0.0   80.0       1 60.7G 59.4G   0|0   4|0 21.7k   835k  281 server  PRI 2019-09-24T11:44:01-04:00
      *0     31     *0     *0       8     19|0     0.0   80.1       0 61.1G 59.8G   0|0   3|0 10.7k  1.66m  281 server  PRI 2019-09-24T11:44:02-04:00
  insert  query update delete getmore  command % dirty % used flushes vsize   res qr|qw ar|aw netIn netOut conn         set repl                      time
      *0     26     *0     *0       4     13|0     0.0   80.1       0 61.3G 60.0G   0|0   4|0 8.07k   341k  281 server  PRI 2019-09-24T11:44:03-04:00
      *0     42     *0     *0       4     17|0     0.0   80.0       0 61.5G 60.3G   0|0   4|0 12.3k   467k  281 server  PRI 2019-09-24T11:44:04-04:00
       1     59     *0     *0       7     55|0     0.0   80.0       0 61.9G 60.6G   0|0   5|0 20.7k  1.30m  281 server  PRI 2019-09-24T11:44:05-04:00
      *0     43     *0     *0       6     25|0     0.0   80.0       0 62.5G 61.1G   0|0   4|0 12.4k  3.48m  281 server  PRI 2019-09-24T11:44:06-04:00
      *0     52     *0     *0      10     24|0     0.0   80.0       0 62.8G 61.4G   0|0   3|0 16.1k  1.56m  281 server  PRI 2019-09-24T11:44:07-04:00
      *0     29     *0     *0       4     32|0     0.0   80.0       0 62.8G 61.4G   0|0   3|0 10.9k  1.35m  281 server  PRI 2019-09-24T11:44:09-04:00
      *0     44     *0     *0       7     36|0     0.0   80.0       0 62.8G 61.4G   0|0   4|0 14.2k   776k  281 server  PRI 2019-09-24T11:44:10-04:00
      *0    135     *0     *0       4     38|0     0.0   80.0       0 62.8G 61.4G   0|0   3|0 36.2k   824k  281 server  PRI 2019-09-24T11:44:11-04:00
      *0     36     *0     *0       1     33|0     0.0   80.0       0 62.8G 61.4G   0|0   4|0 12.3k  1.33m  281 server  PRI 2019-09-24T11:44:12-04:00
      *0     54     *0     *0       7     19|0     0.0   80.0       0 62.8G 61.4G   0|0   2|0 15.9k   737k  279 server  PRI 2019-09-24T11:44:13-04:00
  insert  query update delete getmore  command % dirty % used flushes vsize   res qr|qw ar|aw netIn netOut conn         set repl                      time
      *0     60     *0     *0       4      7|0     0.0   80.0       0 62.8G 61.4G   0|0   2|0 16.5k   798k  279 server  PRI 2019-09-24T11:44:14-04:00
      *0     72     *0     *0       4     39|0     0.0   80.0       0 62.8G 61.4G   0|0   1|0 21.3k   835k  279 server  PRI 2019-09-24T11:44:15-04:00
      *0     47     *0     *0       5     31|0     0.0   80.0       0 62.8G 61.4G   0|0   2|0 14.1k  1.28m  279 server  PRI 2019-09-24T11:44:16-04:00
      *0     68     *0     *0       3     14|0     0.0   80.0       0 62.8G 61.4G   0|0   1|0 19.0k   851k  279 server  PRI 2019-09-24T11:44:17-04:00
      *0     52     *0     *0       7     20|0     0.0   80.0       0 62.8G 61.4G   0|0   2|0 15.4k   993k  279 server  PRI 2019-09-24T11:44:18-04:00
      *0     59     *0     *0       6     21|0     0.0   80.0       0 62.8G 61.4G   0|0   1|0 19.5k  1.23m  279 server  PRI 2019-09-24T11:44:19-04:00
      *0     61     *0     *0       4     55|0     0.0   80.0       0 62.8G 61.4G   0|0   1|0 19.4k   839k  279 server  PRI 2019-09-24T11:44:20-04:00
      *0    334     *0     *0       0     21|0     0.0   80.0       0 62.8G 61.4G   0|0   1|0 86.4k   727k  279 server  PRI 2019-09-24T11:44:21-04:00
      *0     25     *0     *0       2     28|0     0.0   80.1       0 62.8G 61.4G   0|0   1|0 9.11k  1.43m  279 server  PRI 2019-09-24T11:44:22-04:00
      *0     48     *0     *0       8     20|0     0.0   79.9       0 62.8G 61.4G   0|0   1|0 15.6k   707k  279 server  PRI 2019-09-24T11:44:23-04:00
  </code></pre>
  
  <p>What is the best practice for diagnosing the cause of these virtual memory increases? How come the virtual memory increases and doesn't subsequently decrease? What would cause that? What should I look for in the logs and other places? </p>