In what scenarios can the blockchain size decrease for bitcoin? I am running a private bitcoin network for which I changed the target time between two blocks to 12 seconds and the difficulty adjustment to 25 blocks interval. I ran the network for about 4 hours with 50 nodes. In one of the node's logs I observed that the blockchain height increased up to a maximum of 181 and then started decreasing, all the way to 38. what could be an explanation for such a strange behaviour.

Please refer to the log below:

2015-11-04 01:58:47 receive version message: /Satoshi:0.11.99/: version 70011, blocks=181, us=0.0.0.0:0, peer=2, peeraddr=127.0.0.1:44117
2015-11-04 01:58:47 UpdateTip: new best=0000005265ca4ce01ad0d06f45cf475bf303de3d64e942c5cf1177e00f346c78  height=180  log2_work=37.083283  tx=30941  date=2015-11-04 01:53:17 progress=1.000000  cache=0.0MiB(1tx)
2015-11-04 01:58:47 UpdateTip: new best=00000052a34cedf3c5ddbeb46d36644654523db855c4cce984d2623e840dd219  height=179  log2_work=37.082953  tx=30940  date=2015-11-04 01:53:10 progress=1.000000  cache=0.0MiB(2tx)
2015-11-04 01:58:47 UpdateTip: new best=00000030fd7652affb883f05fe0c98e7fe3fbc3cfd74808e061ed05ec61c22e6  height=178  log2_work=37.082623  tx=30939  date=2015-11-04 01:52:55 progress=1.000000  cache=0.0MiB(3tx)
2015-11-04 01:58:47 AddToWallet c32bcbd8102c602a5e71ee717232e204435f331dce6fbfb9eb5d552698faa95b
2015-11-04 01:58:47 AddToWallet 1c91517aeadd12bcbcfdf4a1423b671d405543ae9abfbd87078969ce1971663f
2015-11-04 01:58:47 AddToWallet b11f9c2e3b1ab3d3983da63783bb95903d89405243d0716ea88272a9261b7a33
It looks like this part of the code is what was adding all those UpdateTip messages:

https://github.com/bitcoin/bitcoin/blob/8fe30fb4d130532d4a0e4c9d143f03e1b85a749e/src/main.cpp#L2234

Essentially, bitcoind thinks another chain with more cumulative work exists on your 4-node network, though from the logs, it's hard to tell how much lower it goes. You'll have to run it for longer in order to find out.