title: MongoDB databases disappeared
body: <p>My entire mongo database seems to be gone (this is for my thesis which is due in a month so kind of desperate here).</p>
  
  <p>I'm running some code fetching tweets (it's been working for a month without any edits) and writing to mongo using pymongo and I get this error:</p>
  
  <pre><code>Traceback (most recent call last):
     File "linguistic_feats.py", line 100, in &lt;module&gt;
  write_all_user_tweets(conn, dataset)
     File "linguistic_feats.py", line 89, in write_all_user_tweets
  [get_all_tweets(user, conn) for user in parse]
     File "linguistic_feats.py", line 89, in &lt;listcomp&gt;
  [get_all_tweets(user, conn) for user in parse]
     File "/Users/zeerakw/Documents/Education/Master/Projects/Gender/collect.py", line 58, in get_all_tweets
  [conn.write_user_collection(tweet) for tweet in  all_tweets]
     File "/Users/zeerakw/Documents/Education/Master/Projects/Gender/collect.py", line 58, in &lt;listcomp&gt;
  [conn.write_user_collection(tweet) for tweet in  all_tweets]
     File "/Users/zeerakw/Documents/Education/Master/Projects/Gender/db.py", line 49, in write_user_collection
  table.update({'_id': tweet['id_str']}, {'$setOnInsert': tweet}, upsert = True)
     File "/usr/local/lib/python3.5/site-packages/pymongo/collection.py", line 2230, in update
  with self._socket_for_writes() as sock_info:
     File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/contextlib.py", line 59, in __enter__
  return next(self.gen)
     File "/usr/local/lib/python3.5/site-packages/pymongo/mongo_client.py", line 712, in _get_socket
  server = self._get_topology().select_server(selector)
     File "/usr/local/lib/python3.5/site-packages/pymongo/topology.py", line 141, in select_server
  address))
     File "/usr/local/lib/python3.5/site-packages/pymongo/topology.py", line 117, in select_servers
  self._error_message(selector))
  pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 61] Connection refused
  </code></pre>
  
  <p>I tried to access the mongo console but receive:</p>
  
  <pre><code>MongoDB shell version: 3.2.0
  connecting to: test
  2016-06-24T03:02:05.622+0200 W NETWORK  [thread1] Failed to connect to 127.0.0.1:27017 after 5000 milliseconds, giving up.
  2016-06-24T03:02:05.623+0200 E QUERY    [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
  connect@src/mongo/shell/mongo.js:224:14
  @(connect):1:6
  
  exception: connect failed
  </code></pre>
  
  <p>Then I called <code>mongod</code> manually, which allowed me to access the console but now instead of having multiple <code>show dbs</code> simply gives:</p>
  
  <pre><code>$ mongo
  MongoDB shell version: 3.2.0
  connecting to: test
  Server has startup warnings:
  2016-06-24T03:07:43.638+0200 I CONTROL  [initandlisten]
  2016-06-24T03:07:43.638+0200 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
  &gt; use thesis 
  &gt; show dbs
  local  0.000GB
  &gt; show collections
  &gt;
  </code></pre>
  
  <p>Log file:</p>
  
  <pre><code>2016-06-24T01:17:51.074+0200 E STORAGE  [thread1] WiredTiger (24) [1466723871:73125][473:0x70000020a000], log-server: /usr/local/var/mongodb/journal: opendir: Too many open files
  2016-06-24T01:17:51.074+0200 E STORAGE  [thread1] WiredTiger (24) [1466723871:74536][473:0x70000020a000], log-server: dirlist journal prefix WiredTigerPreplog: Too many open files
  2016-06-24T01:17:51.074+0200 E STORAGE  [thread1] WiredTiger (24) [1466723871:74568][473:0x70000020a000], log-server: log pre-alloc server error: Too many open files
   2016-06-24T01:17:51.074+0200 E STORAGE  [thread1] WiredTiger (24) [1466723871:74588][473:0x70000020a000], log-server: log server error: Too many open files
   2016-06-24T01:17:51.078+0200 W FTDC     [ftdc] Uncaught exception in 'FileNotOpen Failed to open interim file /usr/local/var/mongodb/diagnostic.data/metrics.interim.temp' in full-time diagnostic data capture subsystem. Shutting down the full-time diagnostic data capture subsystem.
   2016-06-24T01:18:14.124+0200 E STORAGE  [thread2] WiredTiger (24) [1466723894:124125][473:0x700000393000], file:WiredTiger.wt, WT_SESSION.checkpoint: WiredTiger.turtle: fopen: Too many open files
   2016-06-24T01:18:14.127+0200 E STORAGE  [thread2] WiredTiger (24) [1466723894:126992][473:0x700000393000], checkpoint-server: checkpoint server error: Too many open files
   2016-06-24T01:18:14.127+0200 E STORAGE  [thread2] WiredTiger (-31804) [1466723894:127084][473:0x700000393000], checkpoint-server: the process must exit and restart: WT_PANIC: WiredTiger library panic
  2016-06-24T01:18:14.127+0200 I -        [thread2] Fatal Assertion 28558
   2016-06-24T01:18:14.127+0200 I -        [thread2] 
  
  ***aborting after fassert() failure
  
  
  2016-06-24T01:18:14.163+0200 F -        [thread2] Got signal: 6 (Abort trap: 6).
  
  0x10788d449 0x10788cddf 0x7fff8deb952a 0x7fff94e7ba26 0x7fff9ce4e6e7 0x10782d53a 0x1076679a1 0x10800e381 0x10800e4c9 0x10800ea74 0x107f9ac5d 0x7fff98d5199d 0x7fff98d5191a 0x7fff98d4f351
  ----- BEGIN BACKTRACE -----
  {
    "backtrace": [
      {
        "b": "106FA5000",
    "o": "8E8449"
  },
  {
    "b": "106FA5000",
    "o": "8E7DDF"
  },
  {
    "b": "7FFF8DEB7000",
    "o": "252A"
  },
  {
    "b": "7FFF94E79000",
    "o": "2A26"
  },
  {
    "b": "7FFF9CDF0000",
    "o": "5E6E7"
  },
  {
    "b": "106FA5000",
    "o": "88853A"
  },
  {
    "b": "106FA5000",
    "o": "6C29A1"
  },
  {
    "b": "106FA5000",
    "o": "1069381"
  },
  {
    "b": "106FA5000",
    "o": "10694C9"
  },
  {
    "b": "106FA5000",
    "o": "1069A74"
  },
  {
    "b": "106FA5000",
    "o": "FF5C5D"
  },
  {
    "b": "7FFF98D4E000",
    "o": "399D"
  },
  {
    "b": "7FFF98D4E000",
    "o": "391A"
  },
  {
    "b": "7FFF98D4E000",
    "o": "1351"
  }
    ],
    "processInfo": {
  "mongodbVersion": "3.2.0",
  "gitVersion": "45d947729a0315accb6d4f15a6b06be6d9c19fe7",
  "compiledModules": [
  
  ],
  "uname": {
    "sysname": "Darwin",
    "release": "15.5.0",
    "version": "Darwin Kernel Version 15.5.0: Tue Apr 19 18:36:36 PDT 2016; root:xnu-3248.50.21~8\/RELEASE_X86_64",
    "machine": "x86_64"
  },
  "somap": [
    {
      "path": "\/usr\/local\/opt\/mongodb\/bin\/mongod",
      "machType": 2,
      "b": "106FA5000",
      "buildId": "A553ECE2755B341A850927E8DEE071E3"
    },
    {
      "path": "\/usr\/lib\/libSystem.B.dylib",
      "machType": 6,
      "b": "7FFF92227000",
      "buildId": "C5D09FE1CC70383EAC2718602F2EDEC4"
    },
    {
      "path": "\/usr\/lib\/libc++.1.dylib",
      "machType": 6,
      "b": "7FFF9112A000",
      "buildId": "8FC3D139805534989AC56467CB7F4D14"
    },
    {
      "path": "\/usr\/lib\/system\/libcache.dylib",
      "machType": 6,
      "b": "7FFF9ACA9000",
      "buildId": "9548AAE92AB735259ECEA2A7C4688447"
    },
    {
      "path": "\/usr\/lib\/system\/libcommonCrypto.dylib",
      "machType": 6,
      "b": "7FFF8B009000",
      "buildId": "9373226134B43914B7A290A81A182DBA"
    },
    {
      "path": "\/usr\/lib\/system\/libcompiler_rt.dylib",
      "machType": 6,
      "b": "7FFF9DD3D000",
      "buildId": "A13ECF69F59F38AE86097B731450FBCD"
    },
    {
      "path": "\/usr\/lib\/system\/libcopyfile.dylib",
      "machType": 6,
      "b": "7FFF8E288000",
      "buildId": "A48637BCF3F234F2BB684C65FD012832"
    },
    {
      "path": "\/usr\/lib\/system\/libcorecrypto.dylib",
      "machType": 6,
      "b": "7FFF9F808000",
      "buildId": "B5C05FD7A540345A87BF8E41848A3C17"
    },
    {
      "path": "\/usr\/lib\/system\/libdispatch.dylib",
      "machType": 6,
      "b": "7FFF8BEA7000",
      "buildId": "C749985761A53D7DA5EA65DCC8C3DF92"
    },
    {
      "path": "\/usr\/lib\/system\/libdyld.dylib",
      "machType": 6,
      "b": "7FFF8B915000",
      "buildId": "CC088C2AD40733E7A6B6B06E0D4AD999"
    },
    {
      "path": "\/usr\/lib\/system\/libkeymgr.dylib",
      "machType": 6,
      "b": "7FFF9B28B000",
      "buildId": "8371CE545FDD3CE9B3DFE98C761B6FE0"
    },
    {
      "path": "\/usr\/lib\/system\/liblaunch.dylib",
      "machType": 6,
      "b": "7FFF92532000",
      "buildId": "834ED60551143641AA4DECF31B801C50"
    },
    {
      "path": "\/usr\/lib\/system\/libmacho.dylib",
      "machType": 6,
      "b": "7FFF92499000",
      "buildId": "318264FA58F139D882851F6254EE410E"
    },
    {
      "path": "\/usr\/lib\/system\/libquarantine.dylib",
      "machType": 6,
      "b": "7FFF8C27F000",
      "buildId": "0F4169F00C843A25B3AEE47B3586D908"
    },
    {
      "path": "\/usr\/lib\/system\/libremovefile.dylib",
      "machType": 6,
      "b": "7FFF8DEC0000",
      "buildId": "552EF39E14D7363E90594565AC2F894E"
    },
    {
      "path": "\/usr\/lib\/system\/libsystem_asl.dylib",
      "machType": 6,
      "b": "7FFF8AFB5000",
      "buildId": "41F8E11F1BD03F1DBA3AAA1577ED98A9"
    },
    {
      "path": "\/usr\/lib\/system\/libsystem_blocks.dylib",
      "machType": 6,
      "b": "7FFF91908000",
      "buildId": "1244D9D5F6AA35BBB30786851C24B8E5"
    },
    {
      "path": "\/usr\/lib\/system\/libsystem_c.dylib",
      "machType": 6,
      "b": "7FFF9CDF0000",
      "buildId": "B552D565B7983B9BAE63F623B42A5F01"
    },
    {
      "path": "\/usr\/lib\/system\/libsystem_configuration.dylib",
      "machType": 6,
      "b": "7FFF8E876000",
      "buildId": "3DEB7DF9680437E1BC830166882FF0FF"
    },
    {
      "path": "\/usr\/lib\/system\/libsystem_coreservices.dylib",
      "machType": 6,
      "b": "7FFF8B8FE000",
      "buildId": "1B3F5AFCFFCD3ECB8B9A5538366FB20D"
    },
    {
      "path": "\/usr\/lib\/system\/libsystem_coretls.dylib",
      "machType": 6,
      "b": "7FFF9CDBC000",
      "buildId": "C90DAE384082381CA1852A6A8B677628"
    },
    {
      "path": "\/usr\/lib\/system\/libsystem_dnssd.dylib",
      "machType": 6,
      "b": "7FFF92CDD000",
      "buildId": "4D10E12B59B5386F82DA326F18028F0A"
    },
    {
      "path": "\/usr\/lib\/system\/libsystem_info.dylib",
      "machType": 6,
      "b": "7FFF98C85000",
      "buildId": "FAA9226D64DE3769A6D86CABA4B7FF4D"
    },
    {
      "path": "\/usr\/lib\/system\/libsystem_kernel.dylib",
      "machType": 6,
      "b": "7FFF98CDF000",
      "buildId": "78E54D59D2B03F549A4A0A68D671F253"
    },
    {
      "path": "\/usr\/lib\/system\/libsystem_m.dylib",
      "machType": 6,
      "b": "7FFF9F7D7000",
      "buildId": "08E1A4B264483DFEA58CACC7335BE7E4"
    },
    {
      "path": "\/usr\/lib\/system\/libsystem_malloc.dylib",
      "machType": 6,
      "b": "7FFF94E79000",
      "buildId": "5748E8B2F81C34C68B13456213127678"
    },
    {
      "path": "\/usr\/lib\/system\/libsystem_network.dylib",
      "machType": 6,
      "b": "7FFF9A5E8000",
      "buildId": "B52DAB7392DC3DA7B9F4B899D66445C1"
    },
    {
      "path": "\/usr\/lib\/system\/libsystem_networkextension.dylib",
      "machType": 6,
      "b": "7FFF97679000",
      "buildId": "66095DC7653938F295EE458F15F6D014"
    },
    {
      "path": "\/usr\/lib\/system\/libsystem_notify.dylib",
      "machType": 6,
      "b": "7FFF9942C000",
      "buildId": "D48BDE340F7E34CAA0FFC578E39987CC"
    },
    {
      "path": "\/usr\/lib\/system\/libsystem_platform.dylib",
      "machType": 6,
      "b": "7FFF8DEB7000",
      "buildId": "29A905EF67773C3382B06C3A88C4BA15"
    },
    {
      "path": "\/usr\/lib\/system\/libsystem_pthread.dylib",
      "machType": 6,
      "b": "7FFF98D4E000",
      "buildId": "3DD1EF4C1D1B3ABF8CC6B3B1CEEE9559"
    },
    {
      "path": "\/usr\/lib\/system\/libsystem_sandbox.dylib",
      "machType": 6,
      "b": "7FFF921C2000",
      "buildId": "150A9D3DF69E32F78C7B8E72CAAFF7E4"
    },
    {
      "path": "\/usr\/lib\/system\/libsystem_secinit.dylib",
      "machType": 6,
      "b": "7FFF95103000",
      "buildId": "32B1A8C6DC843F4FB8CE9A52B47C3E6B"
    },
    {
      "path": "\/usr\/lib\/system\/libsystem_trace.dylib",
      "machType": 6,
      "b": "7FFF993CB000",
      "buildId": "F00E92E4DBDA3749B5B30C3FBBABA1CB"
    },
    {
      "path": "\/usr\/lib\/system\/libunc.dylib",
      "machType": 6,
      "b": "7FFF9ECDF000",
      "buildId": "DDB1E947C77533B8B46163E5EB698F0E"
    },
    {
      "path": "\/usr\/lib\/system\/libunwind.dylib",
      "machType": 6,
      "b": "7FFF8B125000",
      "buildId": "F6EB48E54D12359AAB54C937FBBE9043"
    },
    {
      "path": "\/usr\/lib\/system\/libxpc.dylib",
      "machType": 6,
      "b": "7FFFA10B6000",
          "buildId": "54D1328E054E3DAA89E2375722F9D18F"
        },
        {
          "path": "\/usr\/lib\/libobjc.A.dylib",
          "machType": 6,
          "b": "7FFF9E13F000",
          "buildId": "7489D2D61EFD3414B18D2AECCCC90286"
        },
        {
          "path": "\/usr\/lib\/libauto.dylib",
          "machType": 6,
          "b": "7FFF9B6C1000",
          "buildId": "999E610F41FC32A3ADCA5EC049B65DFB"
        },
        {
          "path": "\/usr\/lib\/libc++abi.dylib",
          "machType": 6,
          "b": "7FFF8E88B000",
          "buildId": "DCCC81773D0935BC97842A04FEC4C71B"
        },
        {
          "path": "\/usr\/lib\/libDiagnosticMessagesClient.dylib",
          "machType": 6,
          "b": "7FFF96E9C000",
          "buildId": "4243B6B421E9355B9C5A95A216233B96"
        }
      ]
    }
  }
  </code></pre>
  
  <p>[UPDATE]</p>
  
  <p>Now the daemon won't even start anymore.</p>
  
  <pre><code>2016-06-24T10:05:32.986+0200 I -        [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
  2016-06-24T10:05:32.986+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=9G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
  2016-06-24T10:05:33.226+0200 E STORAGE  [initandlisten] WiredTiger (-31803) [1466755533:226568][6493:0x7fff75e89000], txn-recover: Recovery failed: WT_NOTFOUND: item not found
  2016-06-24T10:05:33.248+0200 I -        [initandlisten] Assertion: 28595:-31803: WT_NOTFOUND: item not found
  2016-06-24T10:05:33.249+0200 I STORAGE  [initandlisten] exception in initAndListen: 28595 -31803: WT_NOTFOUND: item not found, terminating
  2016-06-24T10:05:33.249+0200 I CONTROL  [initandlisten] dbexit:  rc: 100
  </code></pre>