Why mongod.service is killed by my system? I am using MongoDB 4.4 version on ubuntu 20.4, using 2 mongo instances on the same server. I encountered this problem:
Sep 03 10:45:25 mlnode systemd[1]: Stopping MongoDB Database Server...
Sep 03 10:45:25 mlnode systemd[1]: mongod.service: Succeeded.
Sep 03 10:45:25 mlnode systemd[1]: Stopped MongoDB Database Server.
Sep 03 10:45:25 mlnode systemd[1]: Started MongoDB Database Server.
Sep 03 10:55:46 mlnode systemd[1]: Stopping MongoDB Database Server...
Sep 03 10:55:46 mlnode systemd[1]: mongod.service: Succeeded.
Sep 03 10:55:46 mlnode systemd[1]: Stopped MongoDB Database Server.
Sep 03 10:55:46 mlnode systemd[1]: Started MongoDB Database Server.
Sep 03 11:05:05 mlnode systemd[1]: Stopping MongoDB Database Server...
Sep 03 11:05:05 mlnode systemd[1]: mongod.service: Succeeded.
Sep 03 11:05:05 mlnode systemd[1]: Stopped MongoDB Database Server.
Sep 03 11:05:05 mlnode systemd[1]: Started MongoDB Database Server.
Sep 03 11:13:20 mlnode systemd[1]: /lib/systemd/system/mongod.service:11: PIDFile= references a path below legacy directory /var/run/, updating /var/run/mongodb/mongod.pid → /run/mongodb/mongod.pid; please update the unit file accordingly.
Sep 03 11:25:47 mlnode systemd[1]: /lib/systemd/system/mongod.service:11: PIDFile= references a path below legacy directory /var/run/, updating /var/run/mongodb/mongod.pid → /run/mongodb/mongod.pid; please update the unit file accordingly.
Sep 03 11:56:47 mlnode systemd[1]: mongod.service: Main process exited, code=killed, status=6/ABRT
Sep 03 11:56:47 mlnode systemd[1]: mongod.service: Failed with result 'signal'

Wondering why the system is killing mongod.service, I don't have any clue. Then I checked my mongo logs:
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-09-03T11:56:46.379+05:30&quot;},&quot;s&quot;:&quot;F&quot;,  &quot;c&quot;:&quot;-&quot;,        &quot;id&quot;:23083,   &quot;ctx&quot;:&quot;conn10155&quot;,&quot;msg&quot;:&quot;Invariant failure&quot;,&quot;attr&quot;:{&quot;expr&quot;:&quot;ret&quot;,&quot;error&quot;:&quot;UnknownError: -31803: WT_NOTFOUND: item not found&quot;,&quot;file&quot;:&quot;src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp&quot;,&quot;line&quot;:1598}}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-09-03T11:56:46.379+05:30&quot;},&quot;s&quot;:&quot;F&quot;,  &quot;c&quot;:&quot;-&quot;,        &quot;id&quot;:23084,   &quot;ctx&quot;:&quot;conn10155&quot;,&quot;msg&quot;:&quot;\n\n***aborting after invariant() failure\n\n&quot;}
{&quot;t&quot;:{&quot;$date&quot;:&quot;2020-09-03T11:56:46.379+05:30&quot;},&quot;s&quot;:&quot;F&quot;,  &quot;c&quot;:&quot;CONTROL&quot;,  &quot;id&quot;:4757800, &quot;ctx&quot;:&quot;conn10155&quot;,&quot;msg&quot;:&quot;Writing fatal message&quot;,&quot;attr&quot;:{&quot;message&quot;:&quot;Got signal: 6 (Aborted).\n&quot;}}
Systemd is changing the PIDFile path at startup to /run/mongodb/mongod.pid.  However, the mongod doesn't consult the unit file, it uses its own config, usually /etc/mongod.conf.
Update both the unit file and the mongod.conf to use &quot;/run/mongodb/mongod.pid&quot; as the PIDFile.
Since mongod forks, systemd is expecting to see the pidfile created so that it knows the process has started.  That would use signal 15, it isn't the main problem, though.
Signal 6 occurs when the mongod itself calls abort.
The error calls out line 1598, which is:
Status WiredTigerRecordStore::updateRecord(OperationContext* opCtx,
                                           const RecordId&amp; id,
                                           const char* data,
                                           int len) {
    dassert(opCtx-&gt;lockState()-&gt;isWriteLocked());
    invariant(opCtx-&gt;lockState()-&gt;inAWriteUnitOfWork() || opCtx-&gt;lockState()-&gt;isNoop());

    WiredTigerCursor curwrap(_uri, _tableId, true, opCtx);
    curwrap.assertInActiveTxn();
    WT_CURSOR* c = curwrap.get();
    invariant(c);
    setKey(c, id);
    int ret = wiredTigerPrepareConflictRetry(opCtx, [&amp;] { return c-&gt;search(c); });
    invariantWTOK(ret);  &lt;&lt;--- Line 1598


The WiredTiger docs provide an description of WT_NOTFOUND:

#define WT_NOTFOUND
Item not found.
This error indicates an operation did not find a value to return. This includes cursor search and other operations where no record matched the cursor's search key such as WT_CURSOR::update or WT_CURSOR::remove.

The provided 3 log lines unfortunately don't give us any real clues as to what wasn't found.  It could have been a missing or corrupt data file, a damaged entry in the journal, or a truncated record.
What I suspect happened is the PIDFile mismatch led to systemd causing an unclean shutdown that left one or more files corrupt.
To fix this:

fix the PIDFile configuration in both the unit file and the mongod.conf so this doesn't happen again
make a backup of the dbpath
remove all of the files in the dbpath
attempt to start mongodb

If it is able to startup with the corrected PIDFile settings and a clean data directory, you can decide from there whether to restore the backup and attempt to repair the corrupt files, or just start over with an empty db.