What does "Safe mode: Warning: The network does not appear to fully agree! Some miners appear to be experiencing issues." mean? My bitcoind client that runs on testnet has started returning a lot of errors today. It has been doing so for a few of hours (as of 2014-08-29 9:50 UTC).

When I call it via RPC, I get the following response on ALL requests

{
  "result":null,
  "error": {
    "code":-2,
    "message":"Safe mode: Warning: The network does not appear to fully agree! Some miners appear to be experiencing issues."},
    "id":"1"
  }
}


It doesn't seem to be downloading any new blocks. It seems that I'm not the only one who's affected. As of now, http://tbtc.blockr.io/ is also stuck on blocks that are almost 5 hours old.

The version of bitcoind I use is 0.9.2.1, and the debug log says:

2014-08-29 05:11:59 ProcessBlock: ORPHAN BLOCK 0, prev=0000000000000e819b253066739275290fb4274b580bf62fbd279c518e6e2ebd
2014-08-29 05:11:59 ProcessBlock: ORPHAN BLOCK 1, prev=0000000000000e70d898fa12f328b9ee2d97912e9a495fc80eaeeb781096cadf
2014-08-29 05:11:59 ProcessBlock: ORPHAN BLOCK 2, prev=000000000000256194246bdabb62d3b50e2c309217258b17c543d13ec839367c
2014-08-29 05:12:00 CheckForkWarningConditions: Warning: Large valid fork found
  forking the chain at height 277766 (000000009cf297bc2a9610af823b49fc1d98e001239e99204c3c410e1ad3fe54)
  lasting to height 277767 (0000000000000e819b253066739275290fb4274b580bf62fbd279c518e6e2ebd).
Chain state database corruption likely.
2014-08-29 05:12:00 UpdateTip: new best=00000000735de11454d7de10a71a14e9695135f59b70a4eb906b1107df842270  height=277770  log2_work=61.959615  tx=2398741  date=2014-08-29 05:20:26 progress=1.000001
2014-08-29 05:12:00 UpdateTip: new best=00000000e9cc8d1bea3894d11cd4bacf4099dac3b644cb4f39b4ba3ad89ed2f8  height=277769  log2_work=61.959615  tx=2398732  date=2014-08-29 05:00:09 progress=0.999999
2014-08-29 05:12:00 UpdateTip: new best=00000000d553cd36cc43b3d1255ec395c05e1d4ad730178cd987e960931399ae  height=277768  log2_work=61.959615  tx=2398721  date=2014-08-29 04:39:40 progress=0.999997
2014-08-29 05:12:00 UpdateTip: new best=0000000000000f46520ca51be43443905945d2db5b513f35c2283c2d95a41077  height=277767  log2_work=61.959615  tx=2398683  date=2014-08-29 04:19:31 progress=0.999995
2014-08-29 05:12:00 UpdateTip: new best=000000009cf297bc2a9610af823b49fc1d98e001239e99204c3c410e1ad3fe54  height=277766  log2_work=61.959004  tx=2398667  date=2014-08-29 04:16:35 progress=0.999995
2014-08-29 05:12:00 UpdateTip: new best=0000000000000e819b253066739275290fb4274b580bf62fbd279c518e6e2ebd  height=277767  log2_work=61.959615  tx=2398670  date=2014-08-29 04:15:08 progress=0.999995
2014-08-29 05:12:00 UpdateTip: new best=0000000000000e70d898fa12f328b9ee2d97912e9a495fc80eaeeb781096cadf  height=277768  log2_work=61.960226  tx=2398674  date=2014-08-29 04:24:20 progress=0.999996
2014-08-29 05:12:00 CheckForkWarningConditions: Warning: Large valid fork found
  forking the chain at height 277766 (000000009cf297bc2a9610af823b49fc1d98e001239e99204c3c410e1ad3fe54)
  lasting to height 277767 (0000000000000e819b253066739275290fb4274b580bf62fbd279c518e6e2ebd).
Chain state database corruption likely.


What does this message mean? How is a fork of a single block a "large valid fork"? Why is this that this seems to completely freeze bitcoind? How can I fix it?
When Bitcoin detects that there is something seriously wrong with the network it gets into safe mode, disabling the majority of the methods exposed via its RPC API (only the okSafeMode methods will still work). It does that by filling up the error JSON node with an error code and the the generic error message you received so that the response will not be parseable:



The reasons that push it into safe mode can be a large valid/invalid fork, an invalid longest chain, etc.

Try adding these lines to your bitcoin.conf

checkblocks=10
checklevel=1


then restart bitcoind as well as your app and see if the issue is resolved. If not, try reindexing the blocks on your disk (via the -reindex param).

Also, consider using the -alertnotify hook to get custom notifications and/or to shut down your bitcoin apps every time this happens.