Sharding state recovery process found document - contains wrong list of config servers I've been stuck on this issue for the most part of the day, feel like I am hitting a wall. I hope someone more knowledgeable can help me.

Historical Setup


Mongo Shard (4x replica sets)
Config Servers (3x)
Mongos (2x)


This setup got migrated to a different DC. Process involved setting up servers in new DC, connecting them as members and subsequently shutting down instances in the original DC.

Migration got completed successfully, without any issues.

The Problem

From time to time, Mongo instance fails to start as it is trying to connect to config servers in the OLD DC.

Log line from mongod:

SHARDING [rsSync] Sharding state recovery process found document { _id: "minOpTimeRecovery", configsvrConnectionString: "config_cluster/mongo-config-1:27017,mongo-config-2:27017,mongo-config-3:27017", shardName: "shard_cluster", minOpTime: { ts: Timestamp 1536309187000|2, t: 26457 }, minOpTimeUpdaters: 1 }


Problem is, those mongo-config-1, mongo-config-2 and mongo-config-3 are domains from OLD DC. They are not in the new Mongos instance's config file.

I've found the following entries in Mongo PRIMARY's db.system.version


  { "_id" : "authSchema", "currentVersion" : 5 }
  
  { "_id" : "minOpTimeRecovery", "configsvrConnectionString" : "config_cluster/mongo-config-1:27017,mongo-config-2:27017,mongo-config-3:27017", "shardName" : "shard_cluster", "minOpTime" : { "ts" : Timestamp(1536309187, 2), "t" : NumberLong(26457) }, "minOpTimeUpdaters" : 1 }


My question is:


Is it safe to delete {"_id": "minOpTimeRecovery"} document?
If not, what is the safest way to resolve this?
As per below documentation you can delete the document.

https://docs.mongodb.com/manual/tutorial/restore-sharded-cluster/#remove-the-minoptimerecovery-document-from-the-admin-system-versions-collection