Mongodb Secondary Node not recovering
Link in the error message exactly explain what happened.


  A replica set member becomes “stale” when its replication process
  falls so far behind that the primary overwrites oplog entries the
  member has not yet replicated. The member cannot catch up and becomes
  “stale.” When this occurs, you must completely resynchronize the
  member by removing its data and performing an initial sync.


To avoid this in future:


You need to investigate why secondary fall behind so much. Possible
more writes then normally expected.
Your oplog size might not be set up correctly. Once your secondary is
behind than the first entry in oplog it will never catch up as it has
no way to get those transactions.