title: mongo 4.4 crashes with too many open files
body: <p>we suddenly got MongoDB servers crashing on docker container <code>mongo:4.4</code> with this error:</p>
  <pre><code>couldn't open [/proc/1/stat] Too many open files in system&quot;,&quot;file&quot;:&quot;src/mongo/util/processinfo_linux.cpp&quot;,&quot;line&quot;:78}
  </code></pre>
  <p>you can see the full log here:</p>
  <pre><code>[36mdb_1              |[0m {&quot;t&quot;:{&quot;$date&quot;:&quot;2020-12-01T12:47:43.080+00:00&quot;},&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;COMMAND&quot;,  &quot;id&quot;:51803,   &quot;ctx&quot;:&quot;conn60&quot;,&quot;msg&quot;:&quot;Slow query&quot;,&quot;attr&quot;:{&quot;type&quot;:&quot;command&quot;,&quot;ns&quot;:&quot;myDB.user&quot;,&quot;command&quot;:{&quot;aggregate&quot;:&quot;user&quot;,&quot;pipeline&quot;:[{&quot;$match&quot;:{&quot;__t&quot;:&quot;user&quot;}},{&quot;$unwind&quot;:{&quot;path&quot;:&quot;$security.tokens.learner&quot;,&quot;preserveNullAndEmptyArrays&quot;:true}},{&quot;$unwind&quot;:{&quot;path&quot;:&quot;$security.tokens.cp&quot;,&quot;preserveNullAndEmptyArrays&quot;:true}},{&quot;$unwind&quot;:{&quot;path&quot;:&quot;$security.tokens.admin&quot;,&quot;preserveNullAndEmptyArrays&quot;:true}},{&quot;$match&quot;:{&quot;$or&quot;:[{&quot;$or&quot;:[{&quot;$and&quot;:[{&quot;trash.status&quot;:false},{&quot;$or&quot;:[{&quot;learner.lock.status&quot;:false},{&quot;learner.lock.status&quot;:{&quot;$exists&quot;:false}}]},{&quot;learner.suspend.status&quot;:false},{&quot;security.tokens.learner.token&quot;:&quot;asdjasdjkasdasdbashd&quot;},{&quot;security.tokens.learner.expireDate&quot;:{&quot;$gte&quot;:{&quot;$date&quot;:&quot;2020-12-01T12:47:42.977Z&quot;}}},{&quot;shared.roles.learner&quot;:true}]},{&quot;$and&quot;:[{&quot;security.tokens.learner.token&quot;:&quot;asdjasdjkasdasdbashd&quot;},{&quot;security.tokens.learner.expireDate&quot;:{&quot;$gte&quot;:{&quot;$date&quot;:&quot;2020-12-01T12:47:42.977Z&quot;}}},{&quot;$or&quot;:[{&quot;learner.lock.status&quot;:false},{&quot;learner.lock.status&quot;:{&quot;$exists&quot;:false}}]},{&quot;learner.suspend.status&quot;:false},{&quot;$or&quot;:[{&quot;facebook.registered&quot;:true},{&quot;google.registered&quot;:true},{&quot;linkedin.registered&quot;:true}]}]}]},{&quot;$and&quot;:[{&quot;trash.status&quot;:false},{&quot;$or&quot;:[{&quot;cp.lock.status&quot;:false},{&quot;cp.lock.status&quot;:{&quot;$exists&quot;:false}}]},{&quot;cp.suspend.status&quot;:false},{&quot;security.tokens.cp.token&quot;:&quot;asdjasdjkasdasdbashd&quot;},{&quot;security.tokens.cp.expireDate&quot;:{&quot;$gte&quot;:{&quot;$date&quot;:&quot;2020-12-01T12:47:42.977Z&quot;}}},{&quot;shared.roles.cp&quot;:true}]},{&quot;$and&quot;:[{&quot;trash.status&quot;:false},{&quot;$or&quot;:[{&quot;admin.lock.status&quot;:false},{&quot;admin.lock.status&quot;:{&quot;$exists&quot;:false}}]},{&quot;admin.suspend.status&quot;:false},{&quot;security.tokens.admin.token&quot;:&quot;asdjasdjkasdasdbashd&quot;},{&quot;security.tokens.admin.expireDate&quot;:{&quot;$gte&quot;:{&quot;$date&quot;:&quot;2020-12-01T12:47:42.977Z&quot;}}},{&quot;shared.roles.admin&quot;:true}]}]}}],&quot;cursor&quot;:{},&quot;$db&quot;:&quot;myDB&quot;},&quot;planSummary&quot;:&quot;COLLSCAN&quot;,&quot;keysExamined&quot;:0,&quot;docsExamined&quot;:1569,&quot;cursorExhausted&quot;:true,&quot;numYields&quot;:2,&quot;nreturned&quot;:1,&quot;queryHash&quot;:&quot;167A82D6&quot;,&quot;planCacheKey&quot;:&quot;C324EA6F&quot;,&quot;reslen&quot;:4753,&quot;locks&quot;:{&quot;ReplicationStateTransition&quot;:{&quot;acquireCount&quot;:{&quot;w&quot;:5}},&quot;Global&quot;:{&quot;acquireCount&quot;:{&quot;r&quot;:5}},&quot;Database&quot;:{&quot;acquireCount&quot;:{&quot;r&quot;:5}},&quot;Collection&quot;:{&quot;acquireCount&quot;:{&quot;r&quot;:5}},&quot;Mutex&quot;:{&quot;acquireCount&quot;:{&quot;r&quot;:3}}},&quot;storage&quot;:{},&quot;protocol&quot;:&quot;op_msg&quot;,&quot;durationMillis&quot;:101}}
  [36mdb_1              |[0m {&quot;t&quot;:{&quot;$date&quot;:&quot;2020-12-01T12:58:25.000+00:00&quot;},&quot;s&quot;:&quot;E&quot;,  &quot;c&quot;:&quot;-&quot;,        &quot;id&quot;:23077,   &quot;ctx&quot;:&quot;ftdc&quot;,&quot;msg&quot;:&quot;Assertion&quot;,&quot;attr&quot;:{&quot;error&quot;:&quot;Location13538: couldn't open [/proc/1/stat] Too many open files in system&quot;,&quot;file&quot;:&quot;src/mongo/util/processinfo_linux.cpp&quot;,&quot;line&quot;:78}}
  [36mdb_1              |[0m {&quot;t&quot;:{&quot;$date&quot;:&quot;2020-12-01T12:58:25.339+00:00&quot;},&quot;s&quot;:&quot;E&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:22435,   &quot;ctx&quot;:&quot;thread68&quot;,&quot;msg&quot;:&quot;WiredTiger error&quot;,&quot;attr&quot;:{&quot;error&quot;:23,&quot;message&quot;:&quot;[1606827505:339548][1:0x7f0bbd7e2700], log-server: __directory_list_worker, 46: /data/db/journal: directory-list: opendir: Too many open files in system&quot;}}
  [36mdb_1              |[0m {&quot;t&quot;:{&quot;$date&quot;:&quot;2020-12-01T12:58:25.339+00:00&quot;},&quot;s&quot;:&quot;E&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:22435,   &quot;ctx&quot;:&quot;thread68&quot;,&quot;msg&quot;:&quot;WiredTiger error&quot;,&quot;attr&quot;:{&quot;error&quot;:23,&quot;message&quot;:&quot;[1606827505:339753][1:0x7f0bbd7e2700], log-server: __log_prealloc_once, 505: log pre-alloc server error: Too many open files in system&quot;}}
  [36mdb_1              |[0m {&quot;t&quot;:{&quot;$date&quot;:&quot;2020-12-01T12:58:25.339+00:00&quot;},&quot;s&quot;:&quot;E&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:22435,   &quot;ctx&quot;:&quot;thread68&quot;,&quot;msg&quot;:&quot;WiredTiger error&quot;,&quot;attr&quot;:{&quot;error&quot;:23,&quot;message&quot;:&quot;[1606827505:339773][1:0x7f0bbd7e2700], log-server: __log_server, 961: log server error: Too many open files in system&quot;}}
  [36mdb_1              |[0m {&quot;t&quot;:{&quot;$date&quot;:&quot;2020-12-01T12:58:25.339+00:00&quot;},&quot;s&quot;:&quot;E&quot;,  &quot;c&quot;:&quot;STORAGE&quot;,  &quot;id&quot;:22435,   &quot;ctx&quot;:&quot;thread68&quot;,&quot;msg&quot;:&quot;WiredTiger error&quot;,&quot;attr&quot;:{&quot;error&quot;:-31804,&quot;message&quot;:&quot;[1606827505:339785][1:0x7f0bbd7e2700], log-server: __log_server, 961: the process must exit and restart: WT_PANIC: WiredTiger library panic&quot;}}
  [36mdb_1              |[0m {&quot;t&quot;:{&quot;$date&quot;:&quot;2020-12-01T12:58:25.339+00:00&quot;},&quot;s&quot;:&quot;F&quot;,  &quot;c&quot;:&quot;-&quot;,        &quot;id&quot;:23089,   &quot;ctx&quot;:&quot;thread68&quot;,&quot;msg&quot;:&quot;Fatal assertion&quot;,&quot;attr&quot;:{&quot;msgid&quot;:50853,&quot;file&quot;:&quot;src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp&quot;,&quot;line&quot;:520}}
  [36mdb_1              |[0m {&quot;t&quot;:{&quot;$date&quot;:&quot;2020-12-01T12:58:25.339+00:00&quot;},&quot;s&quot;:&quot;F&quot;,  &quot;c&quot;:&quot;-&quot;,        &quot;id&quot;:23090,   &quot;ctx&quot;:&quot;thread68&quot;,&quot;msg&quot;:&quot;\n\n***aborting after fassert() failure\n\n&quot;}
  [36mdb_1              |[0m {&quot;t&quot;:{&quot;$date&quot;:&quot;2020-12-01T12:58:25.339+00:00&quot;},&quot;s&quot;:&quot;F&quot;,  &quot;c&quot;:&quot;CONTROL&quot;,  &quot;id&quot;:4757800, &quot;ctx&quot;:&quot;thread68&quot;,&quot;msg&quot;:&quot;Writing fatal message&quot;,&quot;attr&quot;:{&quot;message&quot;:&quot;Got signal: 6 (Aborted).\n&quot;}}
  [36mdb_1              |[0m {&quot;t&quot;:{&quot;$date&quot;:&quot;2020-12-01T12:58:25.350+00:00&quot;},&quot;s&quot;:&quot;E&quot;,  &quot;c&quot;:&quot;CONTROL&quot;,  &quot;id&quot;:31430,   &quot;ctx&quot;:&quot;thread68&quot;,&quot;msg&quot;:&quot;Error collecting stack trace&quot;,&quot;attr&quot;:{&quot;error&quot;:&quot;unw_get_proc_name(55CF8D21B921): unspecified (general) error\nerror: unw_step: unspecified (general) error\nunw_get_proc_name(55CF8D21B921): unspecified (general) error\nerror: unw_step: unspecified (general) error\n&quot;}}
  </code></pre>
  <p>this is the result <code>ulimit -a</code> inside the mongo container:</p>
  <pre><code>$ ulimit -a
  core file size          (blocks, -c) unlimited
  data seg size           (kbytes, -d) unlimited
  scheduling priority             (-e) 0
  file size               (blocks, -f) unlimited
  pending signals                 (-i) 31806
  max locked memory       (kbytes, -l) 16384
  max memory size         (kbytes, -m) unlimited
  open files                      (-n) 90000
  pipe size            (512 bytes, -p) 8
  POSIX message queues     (bytes, -q) 819200
  real-time priority              (-r) 0
  stack size              (kbytes, -s) 8192
  cpu time               (seconds, -t) unlimited
  max user processes              (-u) unlimited
  virtual memory          (kbytes, -v) unlimited
  file locks                      (-x) unlimited
  </code></pre>
  <p>How do I fix this?</p>