Delayed replication false alerts and migration delays I am running mongo 2.6.1 on a 3-shard cluster. 

Configuration is: 3 data nodes and 2 arbiters. 

The one data node is hidden and delayed for 3 hours (slavedelay).

I have hidden the node in order to start/stop during backup and the 3 hours delay is for safety during new deployments. 

The issue that I am facing is: During the weekend there are no write operations on the cluster (that is the nature of application). On Monday morning when the writes begin, the monitoring system reports that the delayed data node is behind for more than 40 hours. 

I guess that is because the oplog that has no records during the weekend and when a record gets in on Monday, it needs 3 hours to reach the delayed slave. 

Please share your opinion on the matter, is my thoughts correct here or should I check on a different direction?
The way I understand it, the hidden secondary will always be behind by the number of seconds defined by slaveDelay. In your case, at any moment in the day, the hidden secondary will always be behind by three hours from the last write to the oplog. If there is no more write activity, then the hidden secondary will be completely caught up in three hours. You can check the status of replication in Mongo directly by connecting to each node and running rs.printReplicationInfo()

I configured a 3 sharded cluster using a hidden secondary that is exactly 180 seconds behind, using 2.6.3

Here is the configuration of the two secondaries, the last is hidden and 180 seconds behind. 

{
        "_id" : 1,
        "host" : "myhost:27001"
},
{
        "_id" : 2,
        "host" : "myhost:27002",
        "priority" : 0,
        "slaveDelay" : 180,
        "hidden" : true
},


Priority is set to 0 to keep the hidden secondary from participating in elections.

Started inserting documents on the shard and am now looking at the oplog using db.printReplicationInfo(). 

MongoDB's documentation on db.printReplicationInfo()


  If run on a secondary, the method calls
  db.printSlaveReplicationInfo(). See db.printSlaveReplicationInfo()


Visible Secondary mongo --port 27001

shard2:SECONDARY&gt; db.printReplicationInfo()
configured oplog size:   50MB
log length start to end: 592secs (0.16hrs)
oplog first event time:  Tue Aug 26 2014 08:46:45 GMT+0000 (UTC)
oplog last event time:   Tue Aug 26 2014 08:56:37 GMT+0000 (UTC)
now:                     Tue Aug 26 2014 08:58:45 GMT+0000 (UTC)


Hidden Secondary mongo --port 27002

shard2:SECONDARY&gt; db.printReplicationInfo()
configured oplog size:   50MB
log length start to end: 722secs (0.2hrs)
oplog first event time:  Tue Aug 26 2014 08:43:46 GMT+0000 (UTC)
oplog last event time:   Tue Aug 26 2014 08:55:48 GMT+0000 (UTC)
now:                     Tue Aug 26 2014 08:58:48 GMT+0000 (UTC)


180 seconds later, the hidden secondary catches up mongo --port 27002

shard2:SECONDARY&gt; db.printReplicationInfo()
configured oplog size:   50MB
log length start to end: 592secs (0.16hrs)
oplog first event time:  Tue Aug 26 2014 08:46:45 GMT+0000 (UTC)
oplog last event time:   Tue Aug 26 2014 08:56:37 GMT+0000 (UTC)
now:                     Tue Aug 26 2014 09:02:18 GMT+0000 (UTC)


So whether you are writing to the oplog or not, the secondary will continue to keep up by N seconds as defined in slaveDelay. If your monitoring tool is saying 40 hours, verify first that you are getting the same information using mongo's replicationInfo commands.

db.getReplicationInfo()

db.printReplicationInfo()

rs.printReplicationInfo()

The output of db.printReplicationInfo() is identical to that of rs.printReplicationInfo()

Update

http://docs.mongodb.org/manual/core/replica-set-delayed-member/

Concerning your question about delayed chunk migrations

Sharding


  In sharded clusters, delayed members have limited utility when the
  balancer is enabled. Because delayed members replicate chunk
  migrations with a delay, the state of delayed members in a sharded
  cluster are not useful for recovering to a previous state of the
  sharded cluster if any migrations occur during the delay window.


If I understand your comment correctly.

If secondaryThrottle were set to false, the balancer could continue to migrate chunks without waiting for a delayed member. I'm not sure that's 100% accurate, but that's how I interpreted their documentation about secondaryThrottle and delayed members.